File writing monitoring method and system

Abstract

The present invention discloses a file writing monitoring method and system, which includes obtaining original file writing event data; using a differentiated sampling strategy to generate weighted sampling data according to the file type and system resource status; detecting anomalies through a local anomaly scoring method and outputting real-time anomaly detection results; constructing a directed acyclic graph between each level of the system to analyze the anomaly propagation path; using a matrix decomposition method to identify writing behavior patterns; and dynamically optimizing monitoring parameters in combination with the system resource status. The present invention realizes intelligent monitoring and anomaly diagnosis of file writing behavior, reduces monitoring overhead, and improves the accuracy and real-time performance of anomaly detection.

Description

Technical Field

[0001] The invention belongs to the field of performance monitoring, and in particular to a file writing monitoring method and system. Background Art

[0002] In industrial scenarios such as electricity and gas, embedded devices need to continuously record important operating data, operation logs, and configuration information. These data are of great value for equipment operation and maintenance, fault diagnosis, and safety supervision. Especially in key equipment such as smart meters and gas monitoring terminals, the reliability of data writing is directly related to the accuracy and integrity of metering data. In other words, in the fields of electricity, gas, and environmental monitoring, smart power equipment, gas equipment, and various environmental monitoring equipment need to efficiently and accurately transmit and store key data. Since such devices usually use Flash storage media, which has the characteristics of limited write life and limited storage space, in the process of storing key data, if there are abnormalities such as timeouts or failures, data loss will occur, thereby affecting the performance of the entire system and the accuracy of business data. Therefore, establishing an accurate file write monitoring mechanism is of great significance to extending the service life of equipment, ensuring data security, and improving system reliability.

[0003] At present, file write monitoring in embedded devices mainly adopts simple judgment methods based on thresholds and periodic file system checks. Some solutions set hook functions at the file system layer to count the number of writes and the amount of data; other solutions implement write logging at the application layer and discover abnormal behavior by analyzing the logs; there are also solutions that use the fsck tool to check the integrity of the file system regularly. Although these traditional methods are simple to implement and consume less system resources, their monitoring capabilities are limited and it is difficult to meet the stringent requirements of industrial sites. Traditional data write process monitoring methods mainly rely on application logging in user space and some basic kernel tools. These methods are usually limited in function and can only observe possible delays in the system, but cannot clearly identify the specific stage of the delay and the specific cause of the delay. At the same time, it is also impossible to confirm whether the data is successfully written to the disk.

[0004] The existing technical solutions have the following specific problems: First, in resource-constrained embedded environments, the traditional fixed-frequency sampling strategy cannot balance monitoring accuracy and system overhead, and is prone to excessive occupation of CPU and memory resources; second, due to the lack of a mechanism to identify file types, it is impossible to distinguish the importance of metering data, log files, and configuration files, resulting in unreasonable allocation of monitoring resources; third, the existing anomaly detection methods cannot effectively handle the unique writing mode of Flash media, such as performance fluctuations caused by wear leveling and garbage collection; fourth, in terms of file system anomaly propagation analysis, the existing solutions only focus on single-level faults and cannot track the complete anomaly link from the application layer to the physical device layer; finally, the lack of in-depth analysis capabilities for write behavior makes it difficult to predict the remaining life of Flash blocks and potential write anomalies. These technical problems significantly affect the operating stability and data reliability of embedded devices. Summary of the invention

[0005] The purpose of the invention is to provide a file writing monitoring method and system, in order to solve at least one technical problem existing in the prior art.

[0006] The technical solution, a file writing monitoring method, comprises the following steps:

[0007] S1. Obtain original file write event data, file type information and system resource data, and use a differentiated sampling method to generate weighted sampling data; based on the weighted sampling data, extract multi-dimensional features to form a feature vector set; standardize the feature vector set, and analyze the feature correlation within a preset sliding time window to obtain pre-processed multi-level monitoring data; wherein the file type information includes ordinary files, log files, configuration files and temporary files; the original file write event data includes the timestamp of the write operation, the file descriptor, the write data size, the write strategy and the write location; the system resource data includes the CPU usage, the available size of the file system cache and the I / O queue length;

[0008] S2. Based on the pre-processed multi-level monitoring data, a local anomaly scoring method is used to detect anomalies and generate real-time anomaly detection results; at the same time, feature compression processing is performed on the pre-processed multi-level monitoring data to generate compressed time series feature data;

[0009] S3. Based on the real-time anomaly detection results, a directed acyclic graph between the various levels of the system is constructed to generate a system state diagram; based on the system state diagram, the anomaly propagation path is analyzed, the propagation probability is calculated, and the anomaly propagation path data and propagation probability matrix are obtained; combined with the anomaly propagation path data and the propagation probability matrix, the Bayesian inference method is used to generate the root cause probability distribution data;

[0010] S4. Based on the compressed time series feature data, construct a Hankel matrix to generate a structured feature matrix; perform singular value decomposition on the structured feature matrix, extract the main feature components, and output write behavior pattern data and time series feature data; based on the write behavior pattern data and time series feature data, calculate performance indicators, identify system bottlenecks, and generate performance bottleneck indicator data;

[0011] S5. Based on the system resource data, calculate the resource utilization and generate resource evaluation index data; combine the write behavior pattern data, performance bottleneck index data and resource evaluation index data to construct an optimization objective function, solve the optimal monitoring parameters, and output the optimized monitoring parameter data.

[0012] File write monitoring system, including:

[0013] at least one processor; and,

[0014] a memory communicatively connected to at least one of the processors; wherein,

[0015] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the file writing monitoring method.

[0016] Beneficial effects: The present invention can accurately analyze the abnormal propagation path and root cause, reduce monitoring overhead, realize accurate identification of write behavior patterns and adaptive adjustment of monitoring strategies; can timely discover and locate system anomalies, and can dynamically optimize monitoring strategies according to system status, thereby improving system reliability and performance, and providing strong support for the operation and maintenance management of large-scale distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the method of the present invention.

[0018] Figure 2 This is a flow chart of step S1 of the present invention.

[0019] Figure 3 This is a flow chart of step S2 of the present invention.

[0020] Figure 4 This is a flow chart of step S3 of the present invention.

[0021] Figure 5 This is a flow chart of step S4 of the present invention.

[0022] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, a file writing monitoring method is provided, comprising the following steps:

[0024] S1. By dynamically loading the eBPF program into the kernel, probe points are set at key functions of the file system, page cache, common block layer, scheduling layer, device driver layer and physical device layer; the original file write event data, file type information and system resource data are obtained, and a differentiated sampling method is used to generate weighted sampling data; based on the weighted sampling data, multi-dimensional features are extracted to form a feature vector set; the feature vector set is standardized, and the feature correlation is analyzed within a preset sliding time window to obtain pre-processed multi-level monitoring data; the file type information includes ordinary files, log files, configuration files and temporary files; the original file write event data includes the timestamp of the write operation, the file descriptor, the write data size, the write strategy and the write location; the system resource data includes CPU usage, the available file system cache size and the I / O queue length;

[0025] S2. Based on the pre-processed multi-level monitoring data, a local anomaly scoring method is used to detect anomalies and generate real-time anomaly detection results; at the same time, feature compression processing is performed on the pre-processed multi-level monitoring data to generate compressed time series feature data;

[0026] S3. Based on the real-time anomaly detection results, a directed acyclic graph between the various levels of the system is constructed to generate a system state diagram; based on the system state diagram, the anomaly propagation path is analyzed, the propagation probability is calculated, and the anomaly propagation path data and propagation probability matrix are obtained; combined with the anomaly propagation path data and the propagation probability matrix, the Bayesian inference method is used to generate the root cause probability distribution data;

[0027] S4. Based on the compressed time series feature data, construct a Hankel matrix to generate a structured feature matrix; perform singular value decomposition on the structured feature matrix, extract the main feature components, and output write behavior pattern data and time series feature data; based on the write behavior pattern data and time series feature data, calculate performance indicators, identify system bottlenecks, and generate performance bottleneck indicator data;

[0028] S5. Based on the system resource data, calculate the resource utilization and generate resource evaluation index data; combine the write behavior pattern data, performance bottleneck index data and resource evaluation index data to construct an optimization objective function, solve the optimal monitoring parameters, and output the optimized monitoring parameter data.

[0029] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0030] S11, receiving original file write event data, reading file type information, and collecting system resource data in real time; based on the original file write event data, file type information and system resource data, extracting the importance of the file type and the current system resource status, calculating the sampling weight value, and generating weighted sampling data;

[0031] S12, based on the weighted sampling data, calculate the write delay time and the I / O queue waiting time; measure the data write size and the cache occupied space; record the CPU load value, the file system cache usage ratio and the current length of the I / O queue; combine the write delay time, the I / O queue waiting time, the data write size, the cache occupied space, the CPU load value, the file system cache usage ratio and the current length of the I / O queue to form a feature vector set;

[0032] S13, reading historical characteristic extreme value data from pre-stored historical data; based on the historical characteristic extreme value data, performing standardization processing on each value in the characteristic vector set to obtain a standardized characteristic vector;

[0033] S14. Based on the current system resource status, a sliding time window is constructed; based on the standardized feature vector, the correlation coefficient between each feature in the sliding time window is calculated to generate a feature correlation matrix; the standardized feature vector and the feature correlation matrix are combined to form pre-processed multi-level monitoring data.

[0034] In one embodiment of the present application, the original file write event data and the file type identifier T are obtained, and the sampling weight is calculated: w = f(T, S), where T is the file type and S is the system resource status, and the weighted sampling data D is output. Based on the weighted sampling data D, the following features are extracted: time feature t: write delay, queue waiting time; spatial feature s: write size, cache occupancy; state feature c: CPU load, file system cache usage, I / O queue length; construct a feature vector F = [t, s, c], and output a feature vector set F. Based on the feature vector set F, Min-Max standardization is applied to each dimension: x' = (x-min) / (max-min), and a standardized feature vector F' is output, where x' is the standardized feature value, x is the original feature value, min is the minimum value of the feature in all samples, and max is the maximum value of the feature in all samples. Based on the standardized feature vector F', a sliding time window W is constructed, and the window size is dynamically adjusted; the feature correlation matrix C in the time window is calculated; and the pre-processed multi-level monitoring data M is output, including the standardized feature vector F' and the feature correlation matrix C.

[0035] In another embodiment of the present application, the system implements differentiated data collection and preprocessing based on the extended Berkeley Packet Filter (eBPF) technology. By dynamically loading the eBPF program into the kernel, setting probe points at key functions of the file system, page cache, common block layer, scheduling layer, device driver layer, and physical device layer, the original file write event data is collected; the collected data is stored in eBPF Maps to achieve efficient data transmission and generate preprocessed multi-level monitoring data.

[0036] The system uses eBPF iterator to scan the original file write event data and obtain file metadata information from the virtual file system (VFS) layer; attaches kernel probes (kprobe) to key functions such as vfs_write and ext4_file_write_iter to collect file names, extensions, inode information and access timestamps, and stores these data in the HASH type eBPF Map to generate file feature vectors; reads file operation history records based on the ring buffer mechanism, calculates file access frequency, and outputs file activity data; stores the file feature vector and file activity data in an ARRAY type eBPF Map to generate file type feature data.

[0037] The system obtains CPU usage data, file system cache usage data and I / O load data in real time by attaching eBPF programs to CPU, file system cache, I / O scheduler and I / O queue related functions; uses eBPF ring buffer to record the system status of the last N sampling points, uses exponential smoothing to predict short-term trends, and generates resource trend data; obtains system call delay distribution based on performance events (perf events), calculates system stability indicators, and outputs system stability data; stores resource trend data and system stability data in eBPF Map of PERCPU type to generate system status feature data.

[0038] Read file type feature data and system status feature data from eBPF Maps; retrieve historical data through the BPF auxiliary function bpf_map_lookup_elem, calculate the file importance score, and generate the file weight coefficient; calculate the resource pressure coefficient based on the system load status data, and output the resource weight coefficient; combine the file weight coefficient and the resource weight coefficient and update them to the eBPF Map through bpf_map_update_elem to obtain the comprehensive weight data.

[0039] The original file write event data and comprehensive weight data are read from Maps through the eBPF program; the sampling interval is dynamically adjusted using the BPF timer to generate a sampling time series; the function return value and execution time are obtained through the kernel return probe (kretprobe) at the sampling point, and the feature sampling data is output; the BPF auxiliary function is used for data filtering, and finally the weighted sampling data is generated and stored in the eBPF Maps.

[0040] In another embodiment of the present application, when an application requests to write file data through a system call (such as write()), the system call triggers an interrupt, causing control to be transferred from user space to kernel space. In kernel space, the process of writing files to disk is usually divided into six logical layers, namely the virtual file system layer, the file system layer (including the file system cache), the general block layer, the I / O scheduling layer, the device driver layer, and the physical device layer. This embodiment divides these six logical layers into four stages for monitoring: the file system processing and page cache management stage, the dirty page write-back stage, the I / O request scheduling stage, and the device driver and physical device writing stage.

[0041] (1) File system processing and cache management stage.

[0042] At this stage, the virtual file system (VFS) layer first receives and processes write requests from applications, converting them into operations that can be executed by a specific file system (such as ext4). Next, the specific file system is responsible for updating the metadata of the file and allocating the required data blocks. In order to improve performance, the file data is first written to the page cache and marked as a "dirty page". Once the data is successfully written to the page cache, the system will immediately return a confirmation of the successful write, but the data has not yet been actually written to the disk. At this stage, this embodiment collects the time interval from each file entering the kernel space to being written to the page cache, the amount of data written each time, and various abnormal situations that may occur during the data writing process to the page cache, such as insufficient cache space, the file already exists, or read and write errors. These performance data are helpful in evaluating the system's write load and potential performance bottlenecks. In addition, this embodiment also collects key indicators of the page cache system, such as cache hit rate, number of dirty pages, and cache memory size, so as to monitor the performance of the page cache system in real time. If the cache system performance cannot meet the current needs, the write-back strategy, cache size and other related parameters can be dynamically adjusted to adapt to the changing workload, thereby optimizing the overall performance.

[0043] (2) Dirty page write-back phase.

[0044] At this stage, the Linux kernel will periodically activate the write-back thread according to the current status of the page cache, trigger the system call sync or other preset conditions. These write-back threads are responsible for refreshing the dirty pages in the page cache to the disk to ensure the persistence of the data. During the write-back process, the dirty pages are converted into I / O requests through the general block layer, and then these requests are sent to the I / O scheduling layer for further processing. At this stage, this embodiment collects the time interval from the start of the write-back task to the generation of block I / O requests (bio requests) for dirty pages, sorting and merging these requests, and finally packaging them into I / O requests (requests) and inserting them into the queue. At the same time, this embodiment also monitors various abnormal situations that may occur during the write-back process, such as disk failure, insufficient resources, file system errors, or insufficient permissions.

[0045] (3) I / O request scheduling phase.

[0046] At this stage, the optimized I / O requests are inserted into the queue. These I / O requests will be inserted into the scheduling queue according to the scheduling strategy (such as: elevator algorithm, completely fair queue, etc.), or directly inserted into the software queue, or directly inserted into the hardware queue. When a certain I / O request arrives at the front end of the hardware queue, it will be distributed to the corresponding device driver. At this stage, this embodiment collects the time interval from the I / O request being inserted into the queue until it is distributed to the driver. If this delay is high, it indicates that the I / O scheduler processes the request slowly or the device driver processes the request slowly. This embodiment also monitors the number of requests in the I / O request queue in real time, and the change in queue length can help identify the I / O load situation and whether there is a potential bottleneck.

[0047] (4) Device driver and physical device writing phase.

[0048] At this stage, after receiving the I / O requests, the device driver parses these requests, communicates with the disk controller through the hardware interface, instructs the disk controller to execute the parsed requests, and finally writes the data to the disk. At this stage, this embodiment will record the entire time interval from when the I / O request is distributed to the driver until the data is actually written to the disk, as well as various abnormal situations that may occur during the process of writing data to the disk, such as insufficient disk space, operation timeout, or device non-existence. This embodiment will also collect the throughput of the disk, that is, the amount of data written per second, which can reflect the write speed of the disk. If a long time interval or a low throughput is observed, it indicates that the disk performance is insufficient or the processing efficiency of the driver is low.

[0049] The above four stages constitute a complete file storage process. Each link is crucial. Any problem in any step may cause file writing delay or failure. Therefore, ensuring efficient execution of each stage is the key to achieving reliable data storage.

[0050] This embodiment implements full-stack, low-overhead monitoring of file write behavior by implanting eBPF probes at core file system functions, and reduces the system overhead of data collection through differentiated sampling strategies. Specifically, by dynamically loading the eBPF program into the kernel and setting probe points at key functions of the file system, page cache, common block layer, scheduling layer, device driver layer, and physical device layer, three-dimensional monitoring of the entire life cycle of file writes is achieved; a differentiated sampling strategy driven by file type information is adopted, and different sampling frequencies are used for files of different importance, thereby reducing system overhead while ensuring monitoring accuracy; an efficient data transmission channel is established between kernel state and user state through the eBPF Maps mechanism, avoiding frequent context switching overhead; and combining the sliding time window technology to analyze feature correlation, which not only ensures the temporal continuity of the data, but also achieves a rapid response to sudden performance anomalies. This embodiment not only ensures the integrity and accuracy of the data, but also achieves the optimal balance between monitoring overhead and accuracy.

[0051] According to one aspect of the present application, step S11 is further:

[0052] S111. The system receives original file write event data and extracts metadata information of the file, including file name, extension, creation time and modification time; uses semantic analysis method to process file name and path information and generates file feature vector; calculates file activity based on file access mode and modification frequency and outputs file activity data; combines file feature vector and file activity data to generate file type feature data.

[0053] S112. The system acquires system resource data including CPU usage data, file system cache usage data and I / O load data in real time; constructs a resource status time series, uses exponential smoothing method to predict short-term resource trends, and generates resource trend data; calculates system stability indicators based on resource usage volatility, and outputs system stability data; combines resource trend data and system stability data to generate system status characteristic data.

[0054] S113. Import file type feature data and system status feature data; use fuzzy reasoning method to calculate file importance score and generate file weight coefficient; calculate resource pressure coefficient based on system load status and output resource weight coefficient; perform weighted combination of file weight coefficient and resource weight coefficient to obtain comprehensive weight data.

[0055] S114, receiving the original file write event data and the comprehensive weight data; setting the sampling interval according to the comprehensive weight value to generate a sampling time series; extracting the write event features at the sampling time point and outputting the feature sampling data; performing noise filtering on the feature sampling data and finally generating weighted sampling data.

[0056] In one embodiment of the present application, a file activity calculation method is as follows: file activity score A = θ·Ft +φ·Mt + ψ·Rt; wherein Ft = Σ(Δi·ti) / T is a time decay factor, ti is a time interval between the i-th access and the current access, T is an observation period, and Δi is a time decay coefficient; Mt = (m1·w1 + m2·w2 + ... + mn·wn) / N is a modification frequency factor, mi is the number of each type of modification operation, and wi is a corresponding operation weight; Rt = (s1 + s2 + ... +sk) / (k·S) is a space occupancy factor, si is the size of the data written for the i-th time, S is the storage block size, and k is the number of writes; θ, φ, ψ are balance coefficients and θ+φ+ψ=1, which are used to adjust the relative importance of each factor.

[0057] Exponential smoothing prediction method: resource trend prediction value P(t+1) = α·Xt + (1-α)·Pt + β·(Xt-X(t-1)); where Xt is the current observation value, Pt is the previous prediction value, α is the smoothing coefficient (0<α<1), and β is the trend correction coefficient; system stability index S = 1 / (1+σ 2 )·exp(-λ·|dX / dt|); where σ 2 is the prediction error variance, dX / dt is the rate of change of resource utilization, and λ is the sensitivity parameter.

[0058] Fuzzy reasoning weight calculation method: comprehensive weight W = α·Wf + β·Ws; where Wf = Σ(μi·wi) / Σμi is the file weight coefficient, μi is the membership of the i-th file type, and wi is the preset type weight value; Ws = exp(-λ·L)·exp(-γ·M)·exp(-η·I) is the system state weight coefficient, L is the CPU load rate, M is the memory usage rate, I is the I / O queue length, λ, γ, η are attenuation coefficients; α and β are balance coefficients and α+β=1, which are used to adjust the relative importance of file features and system status; the membership μi is calculated by the triangular fuzzy membership function: μi = max(0, min((x-ai) / (bi-ai), (ci-x) / (ci-bi))), where x is the input feature value, ai, bi, ci are the parameters of the fuzzy set.

[0059] This embodiment realizes an accurate differentiated data sampling strategy by combining file type feature extraction and system status assessment. Specifically, based on file metadata information and semantic analysis methods, the type characteristics and importance of files are accurately identified; by analyzing file access patterns and modification frequencies, the file activity is calculated to achieve a quantitative assessment of file importance; an exponential smoothing prediction method is used to analyze system resource trends and accurately grasp the law of system load changes; combined with fuzzy reasoning technology, the importance of files and system load status are comprehensively considered to generate reasonable sampling weights; through adaptive sampling interval control, it ensures intensive monitoring of important files while avoiding excessive pressure on the system. This embodiment not only reduces the system overhead of data collection, but also ensures the representativeness and integrity of the monitoring data, providing a high-quality data foundation for subsequent anomaly detection and analysis.

[0060] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0061] S21. Based on the preprocessed multi-level monitoring data, the mean and standard deviation of each dimension are calculated to construct an adaptive threshold; based on the preprocessed multi-level monitoring data, the deviation value of each dimension is calculated and weighted to obtain a weighted deviation value; the weighted deviation values ​​of each dimension are summarized and compared with the adaptive threshold to output a real-time anomaly detection result;

[0062] S22. Generate a random projection matrix based on the preprocessed multi-level monitoring data; multiply the multi-level monitoring data with the random projection matrix to obtain preliminary compression features; extract feature vectors based on the preliminary compression features; multiply the preliminary compression features with the feature vector transpose matrix to obtain compressed time series feature data.

[0063] In one embodiment of the present application, based on the preprocessed multi-level monitoring data M, a local anomaly score is constructed: s = Σ(wi×di), where di is the deviation of dimension i and wi is the weight; an adaptive threshold is applied: θ = μ + k×σ, where μ is the mean, σ is the standard deviation, and k is the dynamic coefficient; and a real-time anomaly detection result R is output. Based on the preprocessed multi-level monitoring data M, a random projection is applied: Y = M×R, where R is a random matrix, and the main features are retained: P = Y×VT, where V is a feature vector; and a compressed time series feature P is output.

[0064] This embodiment achieves efficient real-time anomaly detection and data dimension reduction by combining local anomaly scoring and feature compression technology. Specifically, by adopting an adaptive threshold mechanism, the system can dynamically adjust the anomaly judgment standard according to the load characteristics of different time periods, effectively reducing the false alarm rate; by calculating the mean and standard deviation of each dimension, and setting adaptive thresholds based on these statistical values, anomaly detection is more in line with actual business scenarios; weighted processing is performed on each dimension to highlight the influence of important features and improve the accuracy of anomaly detection; random projection methods are used for feature compression, which reduces storage and computing overhead while retaining the main features of the data; combined with the feature vector transpose matrix, effective dimensionality reduction of high-dimensional data is achieved, which not only maintains the main features of the data, but also reduces the computational complexity of subsequent processing. This embodiment improves the real-time response capability of the system while ensuring accuracy.

[0065] According to one aspect of the present application, step S21 is further:

[0066] S211. Based on the preprocessed multi-level monitoring data, the sliding window method is used to segment the data sequence to generate a time window sequence; based on the time window sequence, the mean, variance, skewness and kurtosis of the data in each time window are calculated to obtain statistical characteristic data; based on the preprocessed multi-level monitoring data, the adaptive kernel density estimation method is used to fit the data distribution to generate data distribution characteristics; combining the statistical characteristic data and the data distribution characteristics, a characteristic statistical report is generated;

[0067] S212, based on the data distribution characteristics of the characteristic statistical report, calculate the data fluctuation range and generate fluctuation range data; based on the fluctuation range data, use the exponential weighted moving average method to calculate the dynamic reference value and obtain the reference value data; obtain and generate the tolerance coefficient based on the current load state of the system; combine the reference value data, the fluctuation range data and the tolerance coefficient to output the adaptive threshold data;

[0068] S213, based on the pre-processed multi-level monitoring data, calculate the deviation between the characteristics of each dimension and the benchmark value to generate original deviation data; use wavelet transform to perform noise reduction on the original deviation data and output noise reduction deviation data; based on the feature correlation matrix, calculate the importance score of each dimension and generate dimension weight data; multiply the noise reduction deviation data by the dimension weight data to obtain weighted deviation data;

[0069] S214. Based on the weighted deviation data and the adaptive threshold data, a kernel function is used to map the multidimensional deviation to a unified metric space to generate a normalized anomaly score; based on the normalized anomaly score, a time series correlation analysis is performed to calculate the duration of continuous anomalies and generate anomaly persistence data; based on the normalized anomaly score and the anomaly persistence data, a multi-level decision rule is used to generate real-time anomaly detection results.

[0070] In one embodiment of the present application, the statistical feature calculation method is as follows: a multidimensional statistical feature vector F = [μ, σ, γ1, γ2, H]; wherein μ = Σxi / n is the mean, σ = sqrt(Σ(xi-μ) 2 / n) is the standard deviation, γ1 = Σ(xi-μ) 3 / (n·σ 3 ) is the skewness, γ2 = Σ(xi-μ) 4 / (n·σ 4 )-3 is the kurtosis, H = -Σ(pi·log2(pi)) is the information entropy; pi is obtained by kernel density estimation: pi = Σ(K((x-xi) / h)) / (n·h), K is the Gaussian kernel function, and h is the bandwidth parameter.

[0071] Adaptive threshold calculation method: dynamic threshold θt = μt + k·σt·(1+ω·Lt); where μt is the moving average at time t, σt is the dynamic standard deviation, k is the reference coefficient, Lt is the system load index, and ω is the load adjustment factor; moving average μt = α·xt + (1-α)·μt-1, dynamic standard deviation σt = sqrt(β·(xt-μt) 2 + (1-β)·σt-1 2 ), where α and β are smoothing parameters; Lt = (cpu_util + mem_util + io_util) / 3 is the normalized system load, where pu_util is the CPU utilization, mem_util is the memory utilization, and io_util is the I / O utilization.

[0072] Wavelet transform denoising method: the denoised coefficient aj,k = sign(dj,k)·(|dj,k|-λj·σj)+; where dj,k is the wavelet coefficient at position k in layer j, λj is the threshold parameter and λj = sqrt(2·log(N))·(1+α·j), α is the level adjustment factor, σj is the standard deviation of the noise in layer j; (x)+ means taking the positive operation max(x,0); the final reconstructed signal f(t) = Σ(aj,k·ψj,k(t)), ψj,k(t) is the wavelet basis function.

[0073] This embodiment achieves accurate real-time anomaly detection through multi-dimensional feature statistical analysis and adaptive threshold adjustment. Specifically, the sliding window technology is used to segment the multi-level monitoring data to ensure the time series continuity of data analysis; by calculating statistical features such as mean, variance, skewness and kurtosis, the distribution characteristics of the data are comprehensively characterized; the adaptive kernel density estimation method is used to accurately fit the probability distribution of the data, avoiding the limitations of the parameter distribution assumption; the abnormal judgment threshold is dynamically adjusted based on the data distribution characteristics, which improves the adaptability of anomaly detection; noise reduction processing is performed through wavelet transform to effectively eliminate the influence of random fluctuations in the data; and the weights of each dimension are determined in combination with feature correlation analysis to achieve accurate positioning of anomalies. This embodiment not only ensures the accuracy of detection, but also achieves a millisecond-level response speed, and can promptly detect abnormal behavior in the system.

[0074] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0075] S31, setting the file system layer, page cache layer, general block layer, scheduling layer, device driver layer and physical device layer as graph nodes, extracting the abnormal association relationship between the layers based on the graph nodes and the real-time abnormality detection results, constructing directed edges, and forming a system state graph;

[0076] S32, based on the system state diagram, calculating the shortest path between each node in the diagram to obtain abnormal propagation path data; based on the abnormal propagation path data, counting the propagation frequency of the abnormality between each node, calculating the propagation probability between nodes, and generating a propagation probability matrix;

[0077] S33. Read the root cause prior probability data from the pre-stored historical data, calculate the posterior probability of each root cause based on the root cause prior probability data, the abnormal propagation path data and the propagation probability matrix, and output the root cause probability distribution data.

[0078] In one embodiment of the present application, based on the real-time anomaly detection result R, a directed acyclic graph G(V, E) is constructed: V: {file system, page cache, general block layer, scheduling layer, device driver layer, physical device layer}; E: inter-layer anomaly propagation relationship; the system state graph G is output. Based on the system state graph G, the Dijkstra algorithm is applied to calculate the shortest anomaly propagation path, and the propagation probability matrix Q is constructed: qij represents the probability of anomaly propagation from node i to j, and the anomaly propagation path L and the propagation probability matrix Q are output. Based on the anomaly propagation path L and the propagation probability matrix Q, Bayesian inference is applied: P(Ci|E) = P(E|Ci)×P(Ci) / P(E), where Ci is a possible root cause, E is an observed anomaly, P( ) represents probability, and P(Ci|E) represents the probability of root cause Ci when anomaly E is observed; the root cause probability distribution B is output.

[0079] This embodiment achieves accurate analysis of the abnormal propagation path and root cause location by constructing a directed acyclic graph between the various levels of the system and combining it with the Bayesian inference method. Specifically, a directed acyclic graph is constructed based on the abnormal association relationship between the file system, page cache, general block layer, scheduling layer, device driver layer and physical device layer, which intuitively shows the dependency relationship between the various components of the system; by analyzing the propagation characteristics of abnormalities between different levels, a propagation probability matrix is ​​constructed to quantitatively describe the propagation law of abnormalities; combined with the prior probability of the root cause in the historical data, the Bayesian inference method is used to calculate the posterior probability of each possible root cause, which takes into account both the currently observed abnormal phenomena and the historical experience data; the shortest path algorithm is used to quickly locate the key path of abnormal propagation, thereby improving the efficiency of root cause location. This embodiment can not only accurately track the propagation path of the abnormality, but also quickly locate the root cause of the problem, shortening the time for problem location and resolution.

[0080] According to one aspect of the present application, step S32 is further:

[0081] S321. The system receives the system state diagram, uses a depth-first search algorithm to traverse all nodes in the diagram, and generates a node access sequence; calculates the weight of the edge based on the temporal association between nodes, and outputs the edge weight data; performs reachability analysis on each edge to generate path reachability data; combines the node access sequence and path reachability data, and outputs the initial path set.

[0082] S322. Import the initial path set, calculate the time delay of each path, and generate path delay data; use the minimum spanning tree algorithm to extract the key propagation path and output the key path data; analyze the aggregation effect of abnormal propagation based on the path overlap and generate path aggregation data; combine the key path data with the path aggregation data and output the optimized path set.

[0083] S323, read the optimized path set, count the transfer frequency of anomalies between nodes, and generate anomaly transfer frequency data; use exponential smoothing method to calculate the time distribution of anomaly propagation between nodes, and output propagation delay data; calculate the conditional probability between nodes based on historical anomaly records, and generate conditional probability data; combine the anomaly transfer frequency data and conditional probability data, and output propagation probability data.

[0084] S324. Receive the optimized path set and propagation probability data, calculate the credibility score of the path, and generate path credibility data; use the shortest path algorithm to screen the main path of anomaly propagation and output the main propagation path data; analyze the anomaly propagation range based on the connectivity of the graph and generate propagation range data; perform time series decomposition on the main propagation path data, determine the time sequence of anomaly propagation, and generate propagation time series data; combine the propagation time series data and the propagation range data to construct a spatiotemporal evolution model of anomaly propagation and output anomaly evolution data; fuse the anomaly evolution data with the path credibility data to generate anomaly propagation path data including the propagation path, propagation time series and credibility; construct a probability transfer matrix between nodes based on the anomaly propagation path data and output the propagation probability matrix.

[0085] In one embodiment of the present application, the edge weight calculation method is: edge weight W(i, j) = α·T(i, j) + β·C(i, j) + γ·D(i, j); wherein T(i, j) = exp(-Δt / τ) is the timing association weight, Δt is the time interval between the abnormal occurrence of nodes i and j, and τ is the time decay parameter; C(i, j) = |Cor(Xi, Xj)| is the performance indicator correlation weight, Xi and Xj are the performance indicator sequences of the nodes, and Cor is the Pearson correlation coefficient; D(i, j) = 1 / (1+d(i, j)) is the topological distance weight, and d(i, j) is the shortest path length between nodes; α, β, γ are weight coefficients and α+β+γ=1.

[0086] Path aggregation calculation method: Path aggregation coefficient PC = Σ(wi·oi) / Σwi; where oi is the overlap of path i, oi = Σ(cj·nj) / (L·N), cj is the number of times node j passes through, nj is the number of paths adjacent to node j, L is the path length, and N is the total number of paths; wi is the path weight, wi = exp(-λ·Li)·exp(-μ·Ti), Li is the path length, Ti is the cumulative delay of the path, and λ and μ are attenuation parameters.

[0087] Propagation probability calculation method: conditional propagation probability P(j|i) = α·Pf(j|i) + β·Ph(j|i) + γ·Pt(j|i); where Pf(j|i) = nij / ni is the frequency probability, nij is the number of times the anomaly propagates from i to j, and ni is the total number of anomalies at node i; Ph(j|i) = exp(-hij / h0) is the historical propagation probability, hij is the historical average propagation delay, and h0 is the reference delay; Pt(j|i) = 1 / (1+exp(-(wij-w0) / σ)) is the topological propagation probability, wij is the edge weight between nodes, w0 is the threshold parameter, and σ is the shape parameter; α, β, γ are mixed weights and α+β+γ=1.

[0088] This embodiment achieves accurate tracking of abnormal propagation paths and quantitative description of propagation rules through graph theory algorithms and probability model analysis. Specifically, the system state graph is traversed based on the depth-first search algorithm to ensure the integrity of the path analysis; the time characteristics of abnormal propagation are accurately characterized by calculating the time-series correlation weights of the edges; the minimum spanning tree algorithm is used to extract key propagation paths, reducing the complexity of the analysis; the aggregation effect of abnormalities is analyzed based on path overlap, revealing the spatial characteristics of abnormal propagation; the propagation delay distribution is analyzed through the exponential smoothing method to achieve an accurate estimation of the propagation speed; the conditional probability is calculated in combination with historical abnormal records to construct an accurate propagation probability model. This embodiment can not only accurately track the diffusion path of the anomaly, but also predict the propagation trend of the anomaly, providing an important basis for the risk prevention and control of the system.

[0089] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0090] S41, reorganizing the compressed time series feature data in time order, constructing a multi-row and multi-column Hankel matrix, and obtaining a structured feature matrix;

[0091] S42, calculating the singular values ​​and the corresponding left and right singular vectors of the structured feature matrix, and calculating the cumulative contribution rate based on the singular values; reading the threshold from the pre-stored feature contribution rate threshold data, selecting the singular values ​​and their corresponding singular vectors whose cumulative contribution rate reaches the threshold, and generating the write behavior pattern data and the timing feature data;

[0092] S43. Based on the write behavior pattern data and the timing feature data, calculate the system performance index value, identify the local extreme value points in the index sequence, and generate performance bottleneck index data.

[0093] In one embodiment of the present application, based on the compressed time series feature P, a Hankel matrix H is constructed to capture the time series pattern and output a structured feature matrix H. Based on the structured feature matrix H, a singular value decomposition is applied: H = U × Σ × V T , select important singular values: retain the components with cumulative contribution rate reaching 95%; output the write behavior pattern U and timing feature V. Based on the write behavior pattern U and timing feature V, calculate the performance index: I = f(U, V), identify the key bottleneck point: find the local extreme point in I; output the performance bottleneck index K.

[0094] This embodiment realizes the pattern recognition of file writing behavior and the precise location of performance bottlenecks by constructing a Hankel matrix and performing singular value decomposition, combined with an adaptive threshold selection mechanism. Specifically, the Hankel matrix is ​​constructed based on the compressed time series feature data, which effectively captures the periodic patterns and long-term dependencies in the data sequence; the main feature components are extracted through singular value decomposition, which realizes the dimensionality reduction representation of the writing behavior pattern and reduces the computational complexity of subsequent analysis; the feature contribution rate threshold adaptive selection algorithm is adopted to ensure that the cumulative contribution rate of the selected features reaches the preset threshold, and the main feature information of the data is retained while reducing the dimension; based on the extracted writing behavior pattern and time series features, a multi-dimensional performance indicator system is constructed, and the system bottleneck is accurately located by identifying the local extreme points in the performance indicator sequence. This embodiment can not only effectively identify typical writing behavior patterns, but also timely discover performance bottlenecks, providing an accurate guidance basis for system optimization.

[0095] According to one aspect of the present application, step S42 is further:

[0096] S421. The system receives a structured feature matrix, divides the large-scale matrix into sub-matrices using a block processing method, and generates matrix block data; calculates local singular values ​​for each sub-matrix and outputs local singular value data; uses a parallel computing method to merge local singular values ​​and generate a global singular value sequence; based on the numerical distribution characteristics of the singular value sequence, outputs singular value distribution data.

[0097] S422, import singular value distribution data, calculate the energy contribution of each singular value, and generate energy distribution data; use an adaptive segmentation method to segment the energy distribution and output energy segmentation data; determine the truncation position based on the cumulative energy ratio and generate truncation threshold data; combine the energy distribution data and the truncation threshold data to output a feature selection scheme.

[0098] S423, read the structured feature matrix and feature selection scheme, calculate the left singular vector corresponding to the main singular value, and generate the left vector data; use the orthogonal transformation to optimize the sparsity of the left singular vector, and output the optimized left vector; calculate the right singular vector corresponding to the main singular value, and generate the right vector data; regularize the right singular vector, and output the normalized right vector.

[0099] S424. Receive the optimized left vector and the normalized right vector, map the left vector to the behavior feature space using projection transformation, and generate behavior feature data; perform time series reconstruction on the right vector and output time series pattern data; calculate pattern similarity based on the metric of the feature space and generate pattern similarity data; combine the behavior feature data and time series pattern data into write behavior pattern data and time series feature data, respectively.

[0100] In one embodiment of the present application, the block processing method is specifically as follows: large-scale matrix block M = [M11, M12; M21, M22], where M11, M12, M21, and M22 are four sub-blocks of the matrix M respectively; local SVD calculation U1Σ1V1' = svd(M11), U2Σ2V2' = svd(M22), where U1Σ1V1' is the singular value decomposition result of the matrix M11; U1 is the left singular vector matrix; Σ1 is the singular value diagonal matrix; V1' is the transpose of the right singular vector matrix; U2 is Σ2V2' is the singular value decomposition result of the matrix M22; update matrix Q = M21V1Σ1 -1 , R = M12'U2Σ2 -1 ; Calculate the incidence matrix K = I - Q'Q -RR', where I is the identity matrix and ' represents the transpose; the decomposition of K is UkΣkVk' = svd(K); the final singular value Σ = diag(Σ1, Σk, Σ2), the left singular vector U = [U1, -U1Q; U2R, U2]·Uk, and the right singular vector V = [V1, -V1Q'; V2R', V2]·Vk.

[0101] This embodiment realizes efficient identification and feature extraction of file writing behavior patterns through adaptive singular value decomposition and energy distribution analysis. Specifically, a block matrix processing method is used to divide large-scale matrices, which reduces memory overhead; the calculation process of singular values ​​is accelerated through parallel computing technology, which improves processing efficiency; based on the energy distribution characteristics of singular values, an adaptive segmentation method is used to determine the optimal truncation position, which realizes the automatic selection of feature dimensions; the sparsity of the left singular vector is optimized through orthogonal transformation, which improves the interpretability of feature representation; the right singular vector is regularized to enhance the generalization ability of the model; the features are mapped to the behavior space through projection transformation, which realizes the accurate characterization of the writing behavior pattern; combined with the timing reconstruction technology, the time evolution characteristics of the behavior pattern are retained. This embodiment can not only effectively extract the essential characteristics of file writing behavior, but also realize efficient data compression, providing a reliable feature basis for subsequent performance analysis.

[0102] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0103] S51. Based on the system resource data, calculate the utilization rate of each resource and generate resource evaluation index data;

[0104] S52. Based on the write behavior pattern data, performance bottleneck indicator data and resource evaluation indicator data, an optimization objective function including monitoring overhead and monitoring accuracy is constructed; based on the optimization objective function, a gradient descent method is used to solve the optimal parameters to obtain optimized monitoring parameter data.

[0105] In one embodiment of the present application, based on the system resource status S, the resource utilization vector is calculated: r = [CPU utilization, memory usage, I / O load], and the resource evaluation index A is output. Based on the write behavior pattern U, the performance bottleneck index K and the resource evaluation index A, an optimization objective function is constructed: min f(x) = α×monitoring overhead + β×monitoring accuracy, and the gradient descent method is used to solve the optimal monitoring parameters, and the optimized monitoring parameters O are output.

[0106] This embodiment achieves adaptive optimization of monitoring strategies and optimal balance of resource utilization through real-time resource status evaluation and multi-objective optimization methods. Specifically, based on the real-time collected CPU usage, file system cache usage and I / O load data, a multi-dimensional resource evaluation index system is constructed to achieve a comprehensive evaluation of the system resource status; combined with the write behavior pattern and performance bottleneck indicators, a multi-objective optimization function including monitoring overhead and monitoring accuracy is constructed, and the dynamic optimization of the monitoring strategy is achieved by weighing the monitoring accuracy and system overhead; the gradient descent method is used to solve the optimal monitoring parameters to ensure the convergence and stability of the optimization process; by dynamically adjusting the sampling frequency and feature extraction strategy, the on-demand allocation of monitoring resources is achieved, and the resource utilization efficiency is improved. This embodiment not only ensures the accuracy and comprehensiveness of monitoring, but also minimizes the impact of monitoring on system performance.

[0107] According to one aspect of the present application, step S52 is further:

[0108] S521. Based on the write behavior pattern data, performance bottleneck indicator data and resource evaluation indicator data, the principal component analysis method is used to extract key influencing factors and generate influencing factor data; based on the influencing factor data, the interaction strength between the influencing factors is calculated and the factor association data is output; based on the factor association data, the lag effect of each factor is analyzed through time series correlation to generate time lag characteristic data; based on the influencing factor data and the time lag characteristic data, the target constraint conditions are generated;

[0109] S522. Based on the target constraint conditions, a trade-off function between monitoring cost and accuracy is constructed to generate trade-off function data; based on the trade-off function data, a constrained optimization model is constructed using the Lagrange multiplier method to obtain optimization model data; the value range of the optimization parameter is obtained and set based on the current state of the system to generate parameter range data; the optimization model data and the parameter range data are combined to output the optimization problem description;

[0110] S523. Based on the description of the optimization problem, a gradient descent method is used to calculate the first-order derivative to generate gradient data; based on the gradient data, a parameter value is updated through an adaptive step size strategy, and a parameter iteration sequence is output; based on the parameter iteration sequence, a parameter convergence index is calculated to generate convergence data; based on the parameter iteration sequence and the convergence data, an optimization solution result is obtained;

[0111] S524. Based on the optimization solution results, the cross-validation method is used to evaluate the stability of the parameters and generate parameter stability data; based on the optimization solution results, the performance indicators under different parameter combinations are calculated and the performance evaluation data are output; based on the parameter stability data and the performance evaluation data, the optimal parameter combination is screened and a candidate parameter set is generated; the candidate parameters are verified in actual scenarios and finally the optimized monitoring parameter data is output.

[0112] In one embodiment of the present application, the influencing factor correlation analysis method is as follows: multidimensional influencing factor I(t) = W·F(t) + λ·▽F(t) + μ·∫F(τ)dτ; where F(t) is the feature vector at time t [f1(t), f2(t), ..., fn(t)], and W is the feature weight matrix, which is obtained by least squares optimization: W = (F'F) -1 ·F'Y, Y is the performance indicator sequence; ▽F(t) is the characteristic change rate, indicating the short-term impact; ∫F(τ)dτ is the characteristic cumulative value, indicating the long-term impact; λ and μ are the balance coefficients; the time lag correlation R(τ) = Σ(I(t)·Y(t+τ)) / sqrt(Σ(I(t) 2 )·Σ(Y(t+τ) 2 )), τ is the time lag parameter.

[0113] This embodiment realizes intelligent optimization of monitoring strategies and efficient allocation of resources through multi-objective optimization and adaptive adjustment of parameters. Specifically, based on the principal component analysis method, key influencing factors are extracted from write behavior patterns, performance bottleneck indicators and resource evaluation data, and the main influencing dimensions of system performance are accurately grasped; by calculating the interaction strength and lag effect between influencing factors, a complete performance impact chain is constructed; the Lagrange multiplier method is used to establish a constrained optimization model, and the trade-off problem between monitoring overhead and accuracy is converted into a solvable mathematical form; the parameter value range is dynamically adjusted based on the current state of the system to ensure the practicality of the optimization results; the convergence speed of parameter iteration is improved through the adaptive step size strategy; combined with cross-validation to evaluate parameter stability, the reliability of the optimization results is ensured. This embodiment not only ensures the monitoring effect, but also minimizes the system overhead and achieves the optimal configuration of monitoring resources.

[0114] In one embodiment of the present application, by assigning a unique identifier to the write operation of each file, the flow and processing of data are tracked at different stages. Starting from the time the file enters the kernel space, its related metadata, data block identifier, runtime data and other information are recorded. In the subsequent page cache, dirty page writeback, I / O scheduling, device driver and disk writing stages, the changes and processing status of these data are continuously tracked to locate performance bottlenecks and analyze resource utilization efficiency. A comprehensive and unified performance monitoring view is provided, which can comprehensively monitor the entire path of the file from entering the kernel space to being saved to the underlying storage device, and display the performance of each stage. When the I / O request fails to write to the disk successfully, it can quickly locate which request has an exception at which stage, resulting in the final write failure. At the same time, the view also displays the results of the comprehensive analysis of the system to help users fully understand the performance and resource utilization of the system during the file writing process.

[0115] According to one aspect of the present application, it also includes:

[0116] S6. Calculate the system health index based on the abnormal propagation path data, root cause probability distribution data, write behavior pattern data and monitoring parameter data, and generate a system status assessment report; combine the system status assessment report and performance bottleneck indicator data, and generate optimization suggestion data through decision tree analysis;

[0117] S61. Based on the abnormal propagation path data, the root cause probability distribution data, the write behavior pattern data and the monitoring parameter data, the system health index is comprehensively calculated to generate a system status assessment report;

[0118] S62. Based on the system status assessment report and performance bottleneck indicator data, extract relevant strategies from a pre-stored optimization strategy library; based on the relevant strategies, generate optimization suggestion data through decision tree analysis.

[0119] In one embodiment of the present application, based on the abnormal propagation path L, the root cause probability distribution B, the write behavior pattern U and the monitoring parameter data O, the system health index is calculated: h = g(L, B, U, O), where g( ) represents a function, and the system status evaluation report E is output. Based on the system status evaluation report E and the performance bottleneck indicator K, a decision tree algorithm is used to generate an optimization suggestion list, and the optimization suggestion Y is output.

[0120] This embodiment can comprehensively calculate the system health index by combining the abnormal propagation path data, the root cause probability distribution data, the write behavior pattern data and the monitoring parameter data, accurately reflect the current state and potential problems of the system, and provide a reliable basis for subsequent optimization. Based on the calculated system health index, a detailed system status assessment report is generated, which not only includes the health status of the current system, but also reveals the anomalies and bottlenecks in the system, which helps the system administrator to quickly understand the overall operation of the system and take timely measures. Combined with the system status assessment report and the performance bottleneck indicator data, the optimization suggestions are generated through decision tree analysis, which can automatically select the most appropriate optimization strategy according to the pre-stored optimization strategy library, which not only improves the accuracy and pertinence of the optimization suggestions, but also reduces the workload of manual analysis. Through continuous monitoring of the system health index and the implementation of optimization suggestions, the stability and performance of the system can be improved. Especially under high load and complex environments, it can effectively reduce system failures and performance bottlenecks and improve user experience. This embodiment provides an efficient and accurate system evaluation and optimization method through multi-dimensional data analysis and intelligent decision tree algorithm, which can improve the stability and performance of the system.

[0121] According to one aspect of the present application, step S62 is further:

[0122] S621. Based on the system status assessment report and the performance bottleneck indicator data, the performance problems are classified using the hierarchical analysis method to generate problem classification data; based on the problem classification data, the severity scores of various types of problems are calculated to obtain problem severity data; based on the problem classification data and the problem severity data, time series analysis is performed to determine the frequency of problem occurrence and generate problem frequency data; based on the problem classification data and the problem frequency data, a problem feature vector is generated;

[0123] S622. Import the problem feature vector, retrieve relevant optimization solutions from the pre-stored optimization strategy library, and generate an initial strategy set; calculate the matching degree of each optimization solution with the current problem, and output strategy matching data; evaluate the feasibility of the strategy based on the historical optimization effect, and generate strategy feasibility data; combine the strategy matching data and strategy feasibility data, and output the candidate strategy set.

[0124] S623, read the candidate strategy set, analyze the resource overhead of each optimization strategy, and generate strategy overhead data; predict the performance improvement effect after the strategy is implemented, and output the effect prediction data; calculate the input-output ratio of the strategy, and generate strategy benefit data; based on multi-objective trade-off analysis, output the strategy evaluation results.

[0125] S624. Receive strategy evaluation results, use heuristic algorithms to combine multiple optimization strategies, and generate strategy combination solutions; evaluate the implementation order of different strategy combinations, and output execution sequence data; perform feasibility verification based on system capacity and resource constraints, and generate verification result data; integrate the strategy combination solution and execution sequence data into an executable optimization solution, and finally output optimization recommendation data.

[0126] This embodiment realizes the intelligent generation of system optimization suggestions and the reasonable arrangement of execution plans through multi-level analysis and strategy optimization selection. Specifically, the hierarchical analysis method is used to classify performance problems and establish a hierarchical representation of system problems; by calculating the severity and frequency of problems, a quantitative evaluation of problem priorities is achieved; the feasibility of strategies is evaluated based on historical optimization effects to avoid the execution of invalid optimization plans; a heuristic algorithm is used to combine multiple optimization strategies to achieve overall coordination of optimization plans; by evaluating the resource overhead and performance improvement effect of the strategy, the input-output ratio is accurately calculated; the feasibility is verified in combination with system capacity and resource constraints to ensure the practicality of the optimization suggestions; a multi-objective evaluation method is used to sort different strategy combinations to generate the optimal execution sequence. This embodiment can not only provide accurate optimization suggestions for system problems, but also ensure the effective implementation of optimization measures and improve the overall performance of the system.

[0127] In another embodiment of the present application, step S2 can also be: the system imports pre-processed multi-level monitoring data, detects anomalies through a local anomaly scoring method, and simultaneously performs spatiotemporal feature analysis to generate anomaly feature data and time series feature data; combines the anomaly features and time series features to output real-time anomaly detection results and compressed time series feature data. Specifically:

[0128] S2a, receiving the pre-processed multi-level monitoring data, calculating the local anomaly scores in the time dimension and the space dimension, and generating a preliminary anomaly score; dynamically adjusting the anomaly judgment threshold based on the current state of the system, and outputting the adaptive threshold data; comparing the preliminary anomaly score with the adaptive threshold, and generating anomaly feature data.

[0129] S2b, import the pre-processed multi-level monitoring data and abnormal feature data, extract the time series features and spatial features, and generate feature matrix data; use the adaptive feature selection algorithm to output the key feature data; perform dimensionality reduction and compression processing on the key features to generate time series feature data.

[0130] S2c, read the abnormal feature data and time series feature data, calculate the feature correlation, and generate feature association data; analyze the abnormal propagation characteristics based on the feature correlation, and output the propagation feature data; combine the abnormal features and propagation features to generate real-time anomaly detection results; construct a compressed representation based on the time series features, and output the compressed time series feature data.

[0131] In another embodiment of the present application, step S3 can also be: the system receives real-time anomaly detection results, compressed time series feature data and propagation feature data, constructs a directed acyclic graph integrating anomaly propagation and performance features, and generates an enhanced system state diagram; based on the enhanced system state diagram, analyzes the anomaly propagation path, calculates the propagation probability, and outputs the anomaly propagation path data and the propagation probability matrix; combines the anomaly propagation path data and the propagation probability matrix, and uses an enhanced Bayesian inference method to generate root cause probability distribution data. Specifically:

[0132] S3a, import real-time anomaly detection results, compressed timing feature data and propagation feature data, build a multi-level state diagram, and generate an initial state diagram; enhance node attributes based on compressed timing features and output node feature data; optimize edge weights based on propagation features and generate edge weight data; integrate node feature data and edge weight data, and output an enhanced system state diagram.

[0133] S3b, read the enhanced system state diagram, apply the graph analysis algorithm to calculate the abnormal propagation path, and generate the initial propagation path; evaluate the path credibility based on the node characteristics, and output the path credibility data; calculate the propagation probability based on the edge weight and generate the initial probability data; integrate the path credibility and propagation probability, and output the abnormal propagation path data and the propagation probability matrix.

[0134] S3c, receiving abnormal propagation path data, propagation probability matrix and node feature data, constructing an enhanced Bayesian network model, generating Bayesian network data; calculating conditional probability distribution, outputting probability distribution data; performing posterior probability inference based on node features, generating root cause probability distribution data.

[0135] In another embodiment of the present application, step S5 can also be: the system receives real-time anomaly detection results, root cause probability distribution data and compressed time series feature data, combines the CPU usage data, file system cache usage data and I / O load data collected by the system in real time, builds a resource status assessment model, and generates resource assessment indicator data; based on multi-source data fusion analysis, outputs the optimization objective function; solves the optimization problem, and generates optimized monitoring parameter data. Specifically:

[0136] S5a, import real-time anomaly detection results, root cause probability distribution data and compressed time series feature data, build an anomaly-resource association model, and generate association model data; receive real-time collected system resource data, calculate multi-dimensional resource utilization, and output resource utilization data; analyze resource anomaly relationships based on the association model and generate resource evaluation indicator data.

[0137] S5b, read resource assessment index data, root cause probability distribution data and compressed time series feature data, build a multi-objective optimization model, generate optimization model data; based on resource constraints and monitoring accuracy requirements, set optimization goals and output optimization objective functions.

[0138] S5c, receiving the optimization objective function, using the adaptive optimization algorithm to solve the optimal parameters, and generating parameter optimization data; evaluating the stability of the optimization results, and outputting stability evaluation data; determining the final parameters in combination with the stability evaluation, and generating optimized monitoring parameter data.

[0139] In one embodiment of the present application, based on the eBPF technology, key performance data of four stages in the file writing process are collected, analyzed and monitored in real time:

[0140] First, the unified performance monitoring program performs parameter control in user space, selects the object to be monitored, issues collection instructions, sets monitoring strategies and transmits them to kernel space. Then, the kernel space eBPF program is loaded into the kernel using the bpf() system call so that it can run in kernel space. The eBPF program contains the data collection logic required for execution in the kernel. At each stage of the file writing process, data is collected by loading and attaching the eBPF program to the corresponding mount point in the kernel (i.e., mounted to the kernel function), and the collected data is stored in the eBPF Maps. The unified performance monitoring program in user space collects performance data from the eBPF Maps and analyzes it, and finally outputs the analysis results to the terminal character interface.

[0141] Frequent data collection will affect the overall performance of the Linux system, including increasing CPU load, consuming memory resources, reducing I / O performance, etc. Therefore, this embodiment divides the data collection process into four subroutines. Each subroutine can run independently, and its operation and stop can be flexibly controlled by parameters, and a monitoring strategy can be set for each subroutine.

[0142] In the virtual file system, there are many available data structures that can provide key information such as file name, file size, and file type. This information can be used to filter out irrelevant files. Therefore, the file write function of the virtual file system layer, such as the vfs_write function, is used as the entry mount point. The exit mount point can select a specific file system function, such as the ext4_file_write_iter function, to filter out I / O operations of other file types. By calculating the time interval between the entry mount point and the exit mount point, the delay of the file write cache is obtained. If there is an abnormality in the file write cache, the specific cause of the failure is captured and analyzed at the exit point.

[0143] In order to monitor the performance of the page cache system, the eBPF program is dynamically mounted to key functions such as marking dirty pages, accessing pages, adding pages to the page cache, and writing back dirty pages, and real-time statistics are performed. At the same time, the memory usage of the page cache is monitored. In this way, the present embodiment can collect information about cache hit rate, cache miss number, hit number, and number of dirty pages, so as to comprehensively evaluate and analyze the performance of the page cache system.

[0144] In order to monitor the performance of the write-back process, statistics are collected from the time when the write-back thread is awakened and starts writing back dirty pages. A specific file system function (such as ext4_writepages) is selected as the entry mount point, which can effectively filter out the I / O write-back operations of other file systems. Through this mount point, detailed dirty page information and write-back task information can be obtained, including key data such as the inode to which the dirty page belongs, the number of dirty pages written back this time, the reason for triggering the write-back, and the write-back method used. The exit can select a static mount point (such as ext4_writepages_result). By calculating the time interval between the entry mount point and the exit mount point, the delay of generating an I / O request for the write-back dirty page and inserting it into the I / O queue is obtained. If an abnormal situation occurs during the write-back process, the specific cause of the failure is captured and analyzed at the exit point.

[0145] The I / O multi-queue mechanism of different Linux kernel versions is different, and there are many functions that insert I / O requests into the queue, which makes statistics more complicated. To simplify this process, a static mount point (such as block_rq_insert) can be selected as the entry point to track the insertion of I / O requests. However, this mount point may miss those I / O requests that bypass the queue. Therefore, you can track in the function that bypasses the queue (such as blk_mq_request_bypass_insert) to ensure that the insertion of I / O requests is fully covered. The exit mount point can be selected from the function that takes the I / O request out of the queue and distributes it to the driver (such as blk_mq_start_request). By calculating the time interval between the entry mount point and the exit mount point, the delay of I / O request scheduling is obtained.

[0146] In order to monitor the performance of the device driver and the physical device, the entry mount point can be selected in the function where the I / O request is dispatched to the driver (for example, blk_mq_start_request). After receiving the I / O request, the device driver will parse these requests and communicate with the disk controller through the hardware interface to instruct the disk controller to write the data to the disk. The exit mount point can select the function after the disk writing is completed (for example, blk_update_request) to record the tracking information when the I / O request is completed. By calculating the time interval between the entry mount point and the exit mount point, the delay of the I / O request to write to the disk is obtained. The device number, sector number, amount of data written, etc. where the data is stored can also be obtained. If an abnormal situation occurs during the disk writing process, the specific cause of the failure is captured and analyzed at the exit point.

[0147] The length of the I / O queue needs to be selected from the functions of inserting I / O requests into the queue and taking out I / O requests from the queue. Different Linux kernel versions use different I / O queue mechanisms. The function of inserting I / O requests into the queue is used as the entry mount point, and the function of taking out I / O requests from the queue is used as the exit mount point for dynamic statistics.

[0148] This embodiment uses eBPF technology to implement an efficient method for monitoring the process of writing files to disk. It can be deployed at runtime without modifying the kernel source code, and is particularly suitable for embedded systems with limited memory and processing power. This embodiment can track the processing flow of each I / O request in the process of writing files to disk. Once an I / O request is abnormal during the process of writing to disk, the system can quickly locate the problem file and its specific stage, thereby greatly shortening the troubleshooting time. By comprehensively analyzing the tracking data, this embodiment can quickly locate the system's performance bottlenecks and evaluate resource utilization efficiency, thereby enhancing the stability and reliability of the system. In addition, the design of the monitoring framework simplifies maintenance work.

[0149] According to one aspect of the present application, a file writing monitoring system includes:

[0150] at least one processor; and,

[0151] a memory communicatively connected to at least one of the processors; wherein,

[0152] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the file writing monitoring method described in any one of the above embodiments.

[0153] The present invention realizes intelligent monitoring and abnormal diagnosis of file writing behavior by constructing a complete closed-loop system from data collection, anomaly detection, propagation analysis to optimization feedback. A multi-level monitoring architecture is adopted to realize full-stack monitoring from the file system to the physical device layer; the monitoring overhead is reduced through differential sampling and feature compression technology; based on graph theory and probabilistic statistical methods, the abnormal propagation path and root cause are accurately analyzed; combined with matrix decomposition and deep learning technology, the accurate identification of writing behavior patterns is realized; through multi-objective optimization methods, the adaptive adjustment of monitoring strategies is realized. The present invention can timely detect and locate system anomalies, and can also dynamically optimize monitoring strategies according to system status, improve the reliability and performance of the system, and provide strong support for the operation and maintenance management of large-scale distributed systems.

[0154] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A file writing monitoring method, characterized in that: The steps include: S1. Obtaining original file write event data, file type information and system resource data, using a differentiated sampling method to generate weighted sampling data; based on the weighted sampling data, extracting multidimensional features to form a feature vector set; Standardize the feature vector set and analyze the feature correlation within the preset sliding time window to obtain preprocessed multi-level monitoring data; The file type information includes common files, log files, configuration files, and temporary files; the original file write event data includes the timestamp of the write operation, the file descriptor, the write data size, the write policy, and the write location; the system resource data includes the CPU usage, the available file system cache size, and the I / O queue length; S2. Based on the pre-processed multi-level monitoring data, a local anomaly scoring method is used to detect anomalies and generate real-time anomaly detection results; at the same time, feature compression processing is performed on the pre-processed multi-level monitoring data to generate compressed time series feature data; S3. Based on the real-time anomaly detection results, a directed acyclic graph between the various levels of the system is constructed to generate a system state diagram; based on the system state diagram, the anomaly propagation path is analyzed, the propagation probability is calculated, and the anomaly propagation path data and propagation probability matrix are obtained; combined with the anomaly propagation path data and the propagation probability matrix, the Bayesian inference method is used to generate the root cause probability distribution data; S4, constructing Hankel matrix based on compressed time series feature data to generate structured feature matrix; Perform singular value decomposition on the structured feature matrix, extract the main feature components, and output the write behavior pattern data and timing feature data; based on the write behavior pattern data and timing feature data, calculate the performance index, identify the system bottleneck, and generate performance bottleneck index data; S5. Based on the system resource data, calculate the resource utilization and generate resource evaluation index data; combine the write behavior pattern data, performance bottleneck index data and resource evaluation index data to construct an optimization objective function, solve the optimal monitoring parameters, and output the optimized monitoring parameter data.

2. The file writing monitoring method according to claim 1, characterized in that: Step S1 is further as follows: S11, receiving original file write event data, reading file type information at the same time, and collecting system resource data in real time; based on the original file write event data, file type information and system resource data, extracting the importance of the file type and the current system resource status, calculating the sampling weight value, and generating weighted sampling data; S12, based on the weighted sampling data, calculate the write delay time and the I / O queue waiting time; measure the data write size and the cache occupied space; record the CPU load value, the file system cache usage ratio and the current length of the I / O queue; combine the write delay time, the I / O queue waiting time, the data write size, the cache occupied space, the CPU load value, the file system cache usage ratio and the current length of the I / O queue to form a feature vector set; S13, reading historical characteristic extreme value data from pre-stored historical data; based on the historical characteristic extreme value data, performing standardization processing on each value in the characteristic vector set to obtain a standardized characteristic vector; S14. Based on the current system resource status, a sliding time window is constructed; based on the standardized feature vector, the correlation coefficient between each feature in the sliding time window is calculated to generate a feature correlation matrix; The standardized feature vectors and feature correlation matrices are combined to form preprocessed multi-level monitoring data.

3. The file writing monitoring method according to claim 2, characterized in that: Step S2 is further as follows: S21. Based on the preprocessed multi-level monitoring data, the mean and standard deviation of each dimension are calculated to construct an adaptive threshold; based on the preprocessed multi-level monitoring data, the deviation value of each dimension is calculated and weighted to obtain a weighted deviation value; the weighted deviation values ​​of each dimension are summarized and compared with the adaptive threshold to output a real-time anomaly detection result; S22, generating a random projection matrix based on the preprocessed multi-level monitoring data; multiplying the multi-level monitoring data with the random projection matrix to obtain preliminary compression features; Based on the preliminary compression features, feature vectors are extracted; the preliminary compression features are multiplied by the feature vector transpose matrix to obtain compressed time series feature data.

4. The file writing monitoring method according to claim 3, characterized in that: Step S3 is further as follows: S31, setting the file system layer, page cache layer, general block layer, scheduling layer, device driver layer and physical device layer as graph nodes, extracting the abnormal association relationship between the layers based on the graph nodes and the real-time abnormality detection results, constructing directed edges, and forming a system state graph; S32, based on the system state diagram, calculating the shortest path between each node in the diagram to obtain abnormal propagation path data; based on the abnormal propagation path data, counting the propagation frequency of the abnormality between each node, calculating the propagation probability between nodes, and generating a propagation probability matrix; S33. Read the root cause prior probability data from the pre-stored historical data, calculate the posterior probability of each root cause based on the root cause prior probability data, the abnormal propagation path data and the propagation probability matrix, and output the root cause probability distribution data.

5. The file writing monitoring method according to claim 4, characterized in that: Step S4 is further as follows: S41, reorganizing the compressed time series feature data in time order, constructing a multi-row and multi-column Hankel matrix, and obtaining a structured feature matrix; S42, calculating the singular values ​​and the corresponding left and right singular vectors of the structured feature matrix, and calculating the cumulative contribution rate based on the singular values; reading the threshold from the pre-stored feature contribution rate threshold data, selecting the singular values ​​and their corresponding singular vectors whose cumulative contribution rate reaches the threshold, and generating write behavior pattern data and timing feature data; S43. Based on the write behavior pattern data and the timing feature data, calculate the system performance index value, identify the local extreme value points in the index sequence, and generate performance bottleneck index data.

6. The file writing monitoring method according to claim 5, characterized in that: Step S5 is further as follows: S51. Based on the system resource data, calculate the utilization rate of each resource and generate resource evaluation index data; S52. Based on the write behavior pattern data, performance bottleneck indicator data and resource evaluation indicator data, an optimization objective function including monitoring overhead and monitoring accuracy is constructed; based on the optimization objective function, a gradient descent method is used to solve the optimal parameters to obtain optimized monitoring parameter data.

7. The file writing monitoring method according to claim 6, characterized in that: Also includes: S6. Calculate the system health index based on the abnormal propagation path data, root cause probability distribution data, write behavior pattern data and monitoring parameter data, and generate a system status assessment report; combine the system status assessment report and performance bottleneck indicator data, and generate optimization suggestion data through decision tree analysis; S61. Based on the abnormal propagation path data, the root cause probability distribution data, the write behavior pattern data and the monitoring parameter data, the system health index is comprehensively calculated to generate a system status assessment report; S62, extracting relevant strategies from a pre-stored optimization strategy library based on the system status evaluation report and performance bottleneck indicator data; Based on relevant strategies, optimization suggestion data is generated through decision tree analysis.

8. The file writing monitoring method according to claim 6, characterized in that: Step S52 is further as follows: S521. Based on the write behavior pattern data, performance bottleneck indicator data, and resource evaluation indicator data, a principal component analysis method is used to extract key influencing factors and generate influencing factor data; based on the influencing factor data, the interaction strength between the influencing factors is calculated and factor correlation data is output; Based on the factor association data, the lag effect of each factor is analyzed through time series correlation to generate time lag characteristic data; based on the influencing factor data and time lag characteristic data, the target constraint conditions are generated; S522. Based on the target constraint conditions, a trade-off function between monitoring cost and accuracy is constructed to generate trade-off function data; based on the trade-off function data, a constrained optimization model is constructed using the Lagrange multiplier method to obtain optimization model data; and based on the current state of the system, a value range of the optimization parameter is obtained to generate parameter range data. Combine the optimization model data and parameter range data to output the optimization problem description; S523. Based on the description of the optimization problem, a gradient descent method is used to calculate the first-order derivative and generate gradient data; Based on the gradient data, the parameter value is updated through the adaptive step size strategy, and the parameter iteration sequence is output; Based on the parameter iteration sequence, the parameter convergence index is calculated and the convergence data is generated; Based on the parameter iteration sequence and convergence data, the optimization solution is obtained; S524, based on the optimization solution, using a cross-validation method to evaluate the stability of the parameters and generate parameter stability data; Based on the optimization solution results, calculate the performance indicators under different parameter combinations and output performance evaluation data; Based on parameter stability data and performance evaluation data, the optimal parameter combination is screened to generate a candidate parameter set; The candidate parameters are verified in actual scenarios and the optimized monitoring parameter data is finally output.

9. The file writing monitoring method according to claim 6, characterized in that: Step S21 is further as follows: S211. Based on the preprocessed multi-level monitoring data, the sliding window method is used to segment the data sequence to generate a time window sequence; based on the time window sequence, the mean, variance, skewness and kurtosis of the data in each time window are calculated to obtain statistical characteristic data; Based on the preprocessed multi-level monitoring data, the adaptive kernel density estimation method is used to fit the data distribution and generate data distribution characteristics; Combine statistical feature data and data distribution characteristics to generate feature statistical reports; S212. Calculate the data fluctuation range based on the data distribution characteristics of the characteristic statistical report and generate fluctuation range data; Based on the fluctuation range data, the dynamic benchmark value is calculated using the exponentially weighted moving average method to obtain the benchmark value data; Obtain and generate a tolerance coefficient based on the current load state of the system; combine the baseline value data, the fluctuation range data and the tolerance coefficient to output the adaptive threshold data; S213, based on the pre-processed multi-level monitoring data, calculate the deviation between the characteristics of each dimension and the benchmark value to generate original deviation data; use wavelet transform to perform noise reduction on the original deviation data and output noise reduction deviation data; based on the feature correlation matrix, calculate the importance score of each dimension and generate dimension weight data; multiply the noise reduction deviation data by the dimension weight data to obtain weighted deviation data; S214, based on the weighted deviation data and the adaptive threshold data, using a kernel function to map the multidimensional deviation to a unified metric space to generate a normalized anomaly score; Based on the normalized anomaly score, time series correlation analysis is performed to calculate the duration of continuous anomalies and generate anomaly persistence data; Based on the normalized anomaly score and anomaly persistence data, multi-level decision rules are adopted to generate real-time anomaly detection results.

10. A file writing monitoring system, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the file writing monitoring method described in any one of claims 1 to 9.

Images