Client read-write behavior profiling method, electronic device, and storage medium

Abstract

The application provides a client read-write behavior profiling method, an electronic device and a storage medium. The method is applied to a management node of a distributed file system. The method comprises the following steps: acquiring first behavior data of a client at a query time based on a global index query interface of the distributed file system; determining second behavior data of a time period corresponding to two query times based on the first behavior data of the client at the two query times; and performing read-write behavior profiling based on the second behavior data of the client at multiple time periods. The method, the electronic device and the storage medium provided by the application can deploy the client read-write behavior profiling function with zero invasion, the client read-write behavior profiling is completely decoupled from the I / O path of the client, no I / O path overhead is required, multi-dimensional characterization of the read-write behavior of the client can be realized, and more intuitive and global information can be provided for the analysis of the client behavior in a complex business scenario.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a client read / write behavior profiling method, electronic device, and storage medium. Background Technology

[0002] In distributed file systems, a standardized performance metric monitoring system is typically configured to periodically monitor performance metrics such as the cumulative number of read / write operations and the cumulative number of read / write bytes for each client.

[0003] However, the above performance metrics can only provide administrators with a snapshot of the client's instantaneous performance and cannot meet the needs of complex business scenarios for client behavior analysis. Summary of the Invention

[0004] This invention provides a client read / write behavior profiling method, electronic device, and storage medium to address the shortcomings of related technologies where instantaneous performance snapshots for client monitoring cannot meet complex analysis needs.

[0005] This invention provides a client read / write behavior profiling method, applied to the management node of a distributed file system, the method comprising: Based on the global metric query interface of the distributed file system, the first row of data of the client at the time of query is obtained; Based on the first row of data from the client at two query times, determine the second row of data for the time period corresponding to the two query times; Based on the client's second behavior data across multiple time periods, a read / write behavior profile is created.

[0006] According to a client read / write behavior profiling method provided by the present invention, the step of profiling read / write behavior based on second behavior data of the client in multiple time periods includes: Based on the operation increment value of each operation of the client in the second behavior data of the multiple time periods, at least one of the following is determined: operation type distribution information, operation time period distribution information, throughput, and single operation data volume of the client. Based on at least one of the following: the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, a read and write behavior profile of the client is constructed.

[0007] According to a client read / write behavior profiling method provided by the present invention, the step of constructing client read / write behavior profiling data based on at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume includes: Based on at least one of the client's operation latency information and cache efficiency information, and at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, construct the client's read and write behavior profile data; The operation latency information is determined based on the latency data of each operation in the first row of data from the query time closest to the current time; the cache efficiency information is determined based on the cache hit rate in the first row of data from the query time closest to the current time.

[0008] A client read / write behavior profiling method provided by the present invention further includes: The second row of data is stored in the circular cache area corresponding to the client. The capacity of the ring buffer is determined based on the total duration of the multiple time periods and the duration of each individual time period.

[0009] According to a client read / write behavior profiling method provided by the present invention, the step of determining second behavior data corresponding to the two query times based on first behavior data of the client at two query times includes: Based on the cumulative operation value of each operation in the first row of data of the client at the two query times, calculate the operation increment value of each operation in the time period corresponding to the two query times; If the operation increment value is non-negative, the operation increment value of each operation is recorded in the second row of data of the time period corresponding to the two query times; If the operation increment value is negative, the operation increment value of each operation in the second row of data corresponding to the two query times is set to empty.

[0010] A client read / write behavior profiling method provided by the present invention further includes: Receive profile control commands based on external access interfaces; In response to the profile control command, display all or part of the read and write behavior profile data of one or more clients, or display the first or second behavior data of one or more clients, or reset the read and write behavior profile data of one or more clients.

[0011] A client read / write behavior profiling method provided by the present invention further includes: The client's identifier and the date corresponding to the read / write behavior profile data are used as keys, and the summary sequence of the read / write behavior profile data is used as the value for key-value storage.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the client read / write behavior profiling method as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the client read / write behavior profiling method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the client read / write behavior profiling method as described above.

[0015] The client read / write behavior profiling method, electronic device, and storage medium provided by this invention run entirely on the management node of the distributed file system and obtain first-behavior data by leveraging the existing global indicator query interface of the management node. Therefore, the client read / write behavior profiling function can be deployed with zero intrusion, and the client read / write behavior profiling is completely decoupled from the client I / O path, eliminating I / O path overhead. Compared to related technologies that only provide instantaneous snapshots, this invention obtains second-behavior data by performing differential calculations on the first-behavior data and aggregates second-behavior data from multiple time periods, transforming a single cumulative value into read / write behavior profiling data that includes a time dimension. This enables multi-dimensional characterization of client read / write behavior, and by continuously tracking second-behavior data across multiple time periods, it can effectively depict the changing trends of client read / write behavior, providing more intuitive and comprehensive information for client behavior analysis in complex business scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts of the client read / write behavior profiling method provided by the present invention.

[0018] Figure 2 This is a flowchart illustrating the method for storing second-row data provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the architecture of the distributed file system provided by the present invention.

[0020] Figure 4This is the second flowchart of the client read / write behavior profiling method provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the client read / write behavior profiling device provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] All actions involving the acquisition of signal information or data in this invention are carried out in accordance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0025] With the development of distributed storage technology, the operation and maintenance management of distributed file systems has become increasingly important. In order to monitor the system's operating status in real time, current distributed file systems typically have performance indicator monitoring systems in place.

[0026] Taking the distributed file system CephFS (Ceph File System) as an example, after connecting to the metadata server (MDS), the client periodically reports its performance metrics to the metadata server in the form of client metric messages MClientMetrics. These performance metrics are processed by the metadata server's metric handler MetricsHandler, and then aggregated by the metadata server's metric aggregator MetricAggregator to form the management node metric information MMDSMetrics, which is finally exposed externally through the statistics module of the management node (Manager, MGR). This statistics module can be stats / fs / perf_stats.py.

[0027] Based on this, you can view the performance metrics of each client in real time by executing the Ceph file system performance statistics command ceph fs perf stats, or by using the Ceph file system real-time monitoring tool cephfs-top.

[0028] For any client, performance metrics may include total_read_ops, total_read_size, total_write_ops, total_write_size, read latency statistics, write latency statistics, metadata latency statistics, cache hit rate, number of open files and inodes (index nodes), etc.

[0029] While the aforementioned performance metrics are standard features of distributed file systems and can be queried through standard interfaces, existing technologies still have the following shortcomings in actual operation and maintenance scenarios: Firstly, current standard interfaces only provide instantaneous snapshots or cumulative totals, lacking time-based behavioral analysis. Specifically, standard interfaces primarily return monotonically increasing cumulative values, preventing administrators from directly understanding client activity patterns over specific time periods. To understand client temporal behavioral characteristics, administrators must write complex scripts to periodically collect and post-process data, a cumbersome process lacking a unified analytical standard.

[0030] Secondly, the aforementioned standard interface only provides total statistics and lacks analysis of operation type distribution. Specifically, although the standard interface can provide independent counts for read, write, and metadata operations, it does not automatically calculate the proportion of each type of operation in the total load, nor does it automatically classify clients. Administrators must manually calculate the proportions to determine the client load characteristics.

[0031] Thirdly, the aforementioned standard interfaces cannot directly provide, or do not provide directly, real-time changes, average values, and peak values ​​of throughput. Specifically, the standard interfaces only provide cumulative byte counts; a single sampling point cannot reflect the instantaneous I / O (Input / Output) rate. Administrators need to obtain data from at least two different sampling points and manually perform differential calculations to derive the throughput over that period. This reliance on manual post-processing means the system cannot directly provide the client's average and peak throughput in a single query.

[0032] To address this issue, this invention provides a client read / write behavior profiling method, which is applied to the management node of a distributed file system. Here, a distributed file system refers to a system that allows files to be shared across multiple hosts over a network, such as CephFS, Lustre, and GlusterFS. The management node is the core component in a distributed file system responsible for monitoring, managing, and maintaining the cluster state; specifically, in CephFS, it is the MGR node.

[0033] Figure 1This is a flowchart illustrating the client read / write behavior profiling method provided by the present invention, as follows: Figure 1 As shown, the method includes: Step 110: Based on the global metric query interface of the distributed file system, obtain the first row of data of the client at the time of query.

[0034] Specifically, the global metrics query interface refers to a software interface built into the management node that can obtain performance metrics from each client accessing the distributed file system. For example, in CephFS, the global metrics query interface can be the standard performance metrics query interface provided by the stats module built into MGR.

[0035] The management node can query the first row of data from clients accessing the distributed file system at any given time by calling the global metrics query interface. It should be noted that querying the first row of client data via the global metrics query interface is a function supported and capable of being implemented by the global metrics query interface itself.

[0036] The query time refers to the point in time when the management node actively initiates the retrieval of the first line of data. The management node can periodically trigger the retrieval of the first line of data according to a preset query interval. For example, the query interval can be 5 seconds, 10 seconds, or other durations.

[0037] The first row of data refers to the client's performance metrics data directly obtained from the global metrics query interface at the time of query. The first row of data is typically presented as a cumulative value or a snapshot. For example, the first row of data may include at least one of the following: cumulative read operations (read_ops), cumulative read bytes (read_bytes), cumulative write operations (write_ops), cumulative write bytes (write_bytes), cumulative metadata operations (metadata_ops), average read latency (microseconds) (read_lat_us), average write latency (microseconds) (write_lat_us), and average metadata latency (microseconds) (meta_lat_us).

[0038] It is understandable that, in the case of multiple clients, obtaining the first row of data of each client at the time of query based on the global indicator query interface can also be understood as polling the first row of data of each client at the time of query based on the global indicator query interface.

[0039] Step 120: Based on the first line data of the client at the two query times, determine the second line data of the time period corresponding to the two query times.

[0040] Specifically, since the first row of data usually reflects the cumulative value that has been increasing since the client started, it cannot directly reflect the read and write behavior in a specific period of time. Therefore, for any client, the first row of data of the client at two query times can be differentially calculated.

[0041] The two query times can be adjacent. The time period corresponding to the two query times refers to the time interval defined by the preceding query time as the start time and the following query time as the end time. The duration of this time period is the time interval between the two query times. For example, if the two adjacent query times are 12:00:00 and 12:00:05, then the time period corresponding to the two query times is [12:00:00, 12:00:05], and the duration of this time period is 5 seconds.

[0042] Performing differential calculations on the first row of data from the client at both query times can be understood as performing differential calculations on the cumulative values ​​of the client at each of the two query times, thereby obtaining the incremental value of the client within the corresponding time period at each of the two query times. The incremental value within this time period can then be included in the second row of data for the corresponding time period at each of the two query times.

[0043] Understandably, compared to the first line of data reflecting a single point in time, the second line of data reflecting a period of time is usually represented as an incremental value. For example, by subtracting the cumulative read operations of the previous query from the cumulative read operations of the later query, the incremental value of read operations within that period can be obtained. In this way, static cumulative snapshots can be converted into dynamic incremental data for a given period, thereby identifying the client's actual read and write behavior within that period.

[0044] Step 130: Based on the client's second behavior data over multiple time periods, create a read / write behavior profile.

[0045] Specifically, for any client, secondary behavioral data can be obtained across multiple time periods, such as the client's secondary behavioral data for all time periods within a 24-hour period. Based on this, read / write behavior profiles can be created from the client's secondary behavioral data across these multiple time periods. Specifically, this involves aggregating, statistically analyzing, and performing analysis on the client's secondary behavioral data across multiple time periods to characterize the client's read / write behavior features and patterns, thus obtaining the client's read / write behavior profile data. Here, read / write behavior profile data can be understood as the result of profiling the client's read / write behavior. This profile data can reflect the client's read / write behavior characteristics from multiple dimensions, such as the distribution of client operation types, the distribution of operation time periods, average / peak throughput, the size of data per operation, and latency levels.

[0046] The method provided in this invention runs entirely on the management node of the distributed file system and obtains the first-line data by leveraging the existing global indicator query interface of the management node. Therefore, it allows for zero-intrusion deployment of client read / write behavior profiling functionality, and the client read / write behavior profiling is completely decoupled from the client I / O path, eliminating I / O path overhead. Compared to related technologies that only provide instantaneous snapshots, this method obtains the second-line data by performing differential calculations on the first-line data and aggregates the second-line data from multiple time periods, thereby transforming a single cumulative value into read / write behavior profiling data that includes a time dimension. This enables multi-dimensional characterization of client read / write behavior, and by continuously tracking the second-line data across multiple time periods, it effectively depicts the changing trends of client read / write behavior, providing more intuitive and comprehensive information for client behavior analysis in complex business scenarios.

[0047] Based on the above embodiments, step 130, which involves creating a read / write behavior profile based on the client's second behavior data across multiple time periods, includes: Based on the operation increment value of each operation of the client in the second behavior data of the multiple time periods, at least one of the following is determined: operation type distribution information, operation time period distribution information, throughput, and single operation data volume of the client. Based on at least one of the following: the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, a read and write behavior profile of the client is constructed.

[0048] Specifically, the first row of data at two query times can be differentially calculated to obtain the incremental value for the corresponding time period between the two query times. This incremental value can include the operation increment value for each operation, which can include at least one of read operations, write operations, and metadata operations. The operation increment values ​​correspond to read operation increment value `read_ops_delta`, read byte increment value `read_bytes_delta`, write operation increment value `write_ops_delta`, write byte increment value `write_bytes_delta`, and metadata operation increment value `metadata_ops_delta`, etc.

[0049] By aggregating the incremental values ​​of the above operations in the second behavior data of the client across multiple time periods, the following behavioral features can be calculated: The operation type distribution information reflects the client's load characteristics across different operation types. Specifically, by aggregating all incremental values ​​of read, write, and metadata operations within each time period, the proportion of each type of operation in the total number of operations can be calculated. For example, the proportions of read, write, and metadata operations can be obtained. Furthermore, operation type classification rules based on the proportions of each type of operation can be pre-defined. Therefore, based on the calculated proportions of each type of operation, the client's load type can be automatically classified.

[0050] For example, if the proportion of read operations is greater than or equal to a threshold, the client is classified as read-heavy; if the proportion of write operations is greater than or equal to a threshold, the client is classified as write-heavy; if the proportion of metadata operations is greater than or equal to a threshold, the client is classified as metadata-heavy; if the proportion of read operations is greater than the proportion of write operations, and the proportion of read operations is less than a threshold, the client is classified as read-biased; if the proportion of write operations is greater than the proportion of read operations, and the proportion of write operations is less than a threshold, the client is classified as write-biased; if the proportion of read operations and the proportion of write operations are close, the client is classified as balanced; if the total number of operations is 0, the client is classified as idle.

[0051] It is understandable that the operation type distribution information obtained thereby may include at least one of the following: read operation percentage, write operation percentage, metadata operation percentage, and load type.

[0052] Operation time period distribution information is used to identify the client's time activity patterns. Specifically, the operation increment values ​​of various operations in the second row of data for each time period can be accumulated according to a preset time unit, thereby obtaining an operation distribution array under each time unit. For example, the time unit can be hours. By accumulating the operation increment values ​​of different time periods according to the time unit, the operation volume of various operations under each hour can be obtained, that is, the operation volume distribution in hourly units. As another example, the operation increment values ​​of the second row of data for each time period can be accumulated according to the number of hours to which they belong, thereby obtaining a 24-slot activity distribution array, that is, the client's 24-hour activity distribution information.

[0053] This allows us to calculate the peak and active periods for various types of operations. For example, the hour with the highest combined total of the three types of operations can be taken as the peak hour, and the hours in which the total number of operations exceeds the peak by 10% can be listed as active periods.

[0054] For example, the operation time distribution information of a typical office scenario client can reflect the active period as [8,9,10,...,17], with the peak period as 14, that is, the active period of this client is concentrated during the daytime; while the operation time distribution information of a nighttime batch job client can reflect the active period as [1,2,3,4,5], that is, the active period of this client is concentrated in the early morning.

[0055] Throughput is used to measure the data transfer rate of a client. Specifically, average throughput and peak throughput can be calculated using the read / write byte increment values ​​and the duration of each time period. Average throughput is obtained by dividing the sum of the read / write byte increment values ​​across all time periods by the total duration of all time periods; peak throughput is the maximum throughput of a single time period. The throughput of a single time period is obtained by dividing the read / write byte increment value of that time period by the duration of that time period.

[0056] The data volume per single operation is used to estimate the average I / O size of the client. Specifically, the average data volume per read / write operation can be calculated by using the sum of the read / write byte increments across all time periods and the sum of the read / write operation increments across all time periods. The data volume per single operation helps administrators quickly determine whether the client primarily performs small file operations or large file operations. For example, a single operation data volume of less than 64KB typically indicates that the client primarily performs small file operations, while a single operation data volume greater than 1MB indicates that the client primarily performs large file operations.

[0057] The data volume per operation can include the average data volume of read operations and / or the average data volume of write operations. The average data volume of read operations can be obtained by dividing the sum of the read byte increments across all time periods by the sum of the read operation increments across all time periods; the average data volume of write operations can be obtained by dividing the sum of the write byte increments across all time periods by the sum of the write operation increments across all time periods.

[0058] After completing the calculation of behavioral characteristics in each of the above dimensions, at least one of the above operation type distribution information, operation time period distribution information, throughput, and single operation data volume can be assembled to form structured read and write behavior profile data.

[0059] In this embodiment of the invention, second behavior data from multiple time periods is aggregated to obtain multi-dimensional read and write behavior profile data such as operation type distribution information, operation time period distribution information, throughput, and single operation data volume. Administrators can quickly obtain the load characteristics and activity cycle of the client without manually performing differential calculations and ratio calculations, which greatly reduces the workload of operation and maintenance analysis.

[0060] Based on any of the above embodiments, in step 130, constructing the client's read / write behavior profile data based on at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume includes: Based on at least one of the client's operation latency information and cache efficiency information, and at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, construct the client's read and write behavior profile data; The operation latency information is determined based on the latency data of each operation in the first row of data from the query time closest to the current time; the cache efficiency information is determined based on the cache hit rate in the first row of data from the query time closest to the current time.

[0061] Specifically, in order to provide a more comprehensive picture of the client's read and write behavior, operation latency information and / or cache efficiency information that reflect the current real-time performance status can be added to the read and write behavior profile data.

[0062] The operation latency information is determined based on the latency data of each operation in the first row of data from the query time closest to the current time. Operation latency information can include read operation latency data, write operation latency data, and metadata operation latency data. The latency data can be sliding statistics calculated by the client itself, specifically including the latency mean and latency standard deviation.

[0063] Specifically, in a distributed file system, the latency data in the first row of data reported by the client is itself a sliding statistic calculated by the client. Therefore, there is no need to perform differential calculations over multiple time periods for the latency data. The management node can directly extract the latency data of read operations, write operations, and metadata operations from the latest acquired first row of data as the operation latency information for profiling the client's read and write behavior, thereby directly reflecting the client's current latency level.

[0064] Cache efficiency information is determined based on the cache hit rate of the first row of data from the query time most recently. Cache efficiency information may include Cap (Capability) hit rate and / or Dentry lease (directory entry lease) hit rate. Cache efficiency information can be used to alert administrators whether the client may have an excessively large working set or an unfriendly access pattern.

[0065] Specifically, similar to delayed data, in a distributed file system, the cache hit rate in the first row of data reported by the client is itself a sliding statistic calculated by the client. Therefore, there is no need to perform differential calculations over multiple time periods for the cache hit rate. The management node can directly extract the cache hit rate, Dentry-lease hit rate, and other cache hit rates from the latest acquired first row of data as cache efficiency information for profiling the client's read and write behavior. The Cap hit rate can be obtained by dividing the Cap hit count by the total number of Cap hits and misses; the Dentry Lease hit rate can be obtained by dividing the Dentry Lease hit count by the total number of Dentry Lease hits and misses.

[0066] Based on this, the real-time acquired operation latency information and / or cache efficiency information can be used together to construct read and write behavior profile data.

[0067] In this embodiment of the invention, client read and write behavior profile data is constructed based on the aggregation of second behavior data from multiple time periods, as well as the latency data and / or cache hit rate in the latest queried first behavior data. This further improves the comprehensiveness of the read and write behavior profile, thereby providing more comprehensive and reliable data support for performance tuning and anomaly investigation of the distributed file system.

[0068] Based on any of the above embodiments, the client read / write behavior profiling method further includes: The second row of data is stored in the circular cache area corresponding to the client. The capacity of the ring buffer is determined based on the total duration of the multiple time periods and the duration of each individual time period.

[0069] Specifically, a circular buffer is a fixed-capacity, First-In-First-Out (FIFO) data structure. The circular buffer can be a fixed-capacity double-ended queue (deque with maxlen), meaning that when a new second row of data is stored in the circular buffer, if the buffer is already full, the oldest second row of data stored in the buffer can be automatically evicted, thus providing buffer space for the new second row of data.

[0070] In the management node of the distributed file system, an independent circular buffer can be maintained for each client to store the second row of data obtained from each query and calculation. When newly generated second row of data is appended to the end of the circular buffer and the buffer is full, the oldest second row of data can be automatically evicted. Thus, the circular buffer always caches the second row of data from multiple time periods that have been most recently queried and calculated.

[0071] Furthermore, the capacity of the ring buffer is determined based on the total duration of the multiple time periods and the duration of each individual time period. Specifically, the total duration of the multiple time periods refers to the retention period of the second behavior data required for reading and writing behavior profiling as configured by the system, for example, 24 hours; the duration of each individual time period refers to the time interval between management nodes initiating behavior data queries, for example, 5 seconds. In order to store the second behavior data of the multiple time periods required for reading and writing behavior profiling of clients, the capacity of the ring buffer can be calculated using the following formula: Capacity = Total duration of multiple time slots / Duration of a single time slot For example, capacity = 24 hours × 3600 seconds / 5 seconds = 17280 entries.

[0072] It is understandable that the capacity referred to here is the number of rows of data in the second row that the circular buffer can cache, or it can be understood as the number of time periods corresponding to the second row of data that the circular buffer can store.

[0073] In this embodiment of the invention, by introducing a configurable ring buffer to store the second row of data, the memory usage required for the read and write behavior profile under the sliding window is controlled and has a clear upper bound. This effectively prevents the risk of the second row of data expanding infinitely over time and causing memory overflow of the management node, thus ensuring the stability of the distributed file system management service.

[0074] Based on any of the above embodiments, step 120, determining the second line data corresponding to the two query times based on the first line data of the client at the two query times, includes: Based on the cumulative operation value of each operation in the first row of data of the client at the two query times, calculate the operation increment value of each operation in the time period corresponding to the two query times; If the operation increment value is non-negative, the operation increment value of each operation is recorded in the second row of data of the time period corresponding to the two query times; If the operation increment value is negative, the operation increment value of each operation in the second row of data corresponding to the two query times is set to empty.

[0075] Specifically, since the first row of data returned by the global metric query interface of the distributed file system is usually a monotonically increasing metric, the management node can save the first row of data obtained from the previous query and the first row of data obtained from the current query. By subtracting the cumulative operation value from the cumulative operation value of the first row of data from the previous query from the cumulative operation value of each operation in the current query's first row of data, the operation increment value for the most recent time period is obtained.

[0076] Based on this, a judgment can be made on the calculated operation increment value, specifically whether the operation increment value is negative.

[0077] Under normal operating conditions of a distributed file system, the cumulative value monotonically increases, meaning the cumulative value of the current query will always be greater than or equal to the cumulative value of the previous query. The non-negative operation increment value indicates that the cumulative count is increasing normally. In this case, the operation increment value of each operation can be recorded in the second row of data corresponding to the two query times, and this second row of data can be appended to the circular buffer.

[0078] When a client reconnects, experiences a network disconnection, or a component restarts, the cumulative value it reports to the metadata server will reset from zero. In this case, the cumulative operation value for the current query will be less than the cumulative operation value for the previous query, resulting in a negative calculated operation increment. In other words, a negative operation increment indicates an abnormal event of a counter reset. In this situation, the calculated operation increment value can be discarded, the operation increment value in the second row of data can be set to null, and the cumulative operation value obtained from the current query can be used as the new baseline for the next query.

[0079] In this embodiment of the invention, by determining whether the operation increment value is negative, events where the cumulative value returns to zero due to client reconnection or network fluctuations are filtered out. This effectively avoids the pollution of read and write behavior profile data with false peaks or negative throughput caused by abnormal jumps, and ensures the accuracy and high fidelity of read and write behavior profile data.

[0080] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the second-row data storage method provided by the present invention, as shown below. Figure 2 As shown, to store the second row of data for each time period on the client, the following steps can be performed: First, start a background polling thread. This background polling thread is used to periodically query the first row of data for each client at each query time through the global metric query interface.

[0081] After the background polling thread starts, it waits for the polling interval. Here, the polling interval is the interval between two queries. The polling interval is configurable.

[0082] After the polling interval is reached, the global metric query interface is called to retrieve the first row of data. This first row of data can be specific to each client.

[0083] Then, each client is iterated through to determine the first row of data for each client at the time of the query. For example, the first row of data for a client, ClientSnapshot, may include the following information: Timestamp: Query time read_ops: Total number of read operations read_bytes: Total number of bytes read write_ops: Total number of write operations write_bytes: Total number of bytes written metadata_ops: Cumulative metadata operands read_lat_us: Average read latency (microseconds) write_lat_us: Average write latency (microseconds) meta_lat_us: Average metadata latency (microseconds).

[0084] Based on this, we can determine for each client whether it has the first row of data from the previous query. That is, whether the client's first row of data at the time of the previous query is recorded.

[0085] If a difference exists, a difference calculation is performed on the first row of data at the current query time and the first row of data at the previous query time to obtain the increment value for each operation. Then, it is determined whether the increment value is negative. If it is negative, the increment value is discarded, the time period corresponding to the difference calculation is marked, and the resulting second row of data with its increment value set to zero is appended to the circular buffer corresponding to the client. If it is non-negative, the time period corresponding to the difference calculation is marked, and the resulting second row of data is appended to the circular buffer corresponding to the client. The total duration of the time period corresponding to the second row of data that can be stored in the circular buffer is configurable.

[0086] Here, the second line of data in a client, ClientDelta, may include the following information: timestamp: query time wall_hour: The hour corresponding to the query time (0-23) interval_sec: The time interval (in seconds) between two queries. read_ops_delta: Read operation increment value read_bytes_delta: Increment value of read bytes write_ops_delta: Write operation increment value write_bytes_delta: Write byte increment value metadata_ops_delta: Incremental value for metadata operations read_lat_us: Current average read latency write_lat_us: Current average write latency meta_lat_us: The current average latency of metadata.

[0087] After this, the first row of data at the current query time is considered the data from the previous query. Additionally, expired clients are cleaned up. Then, the polling interval is returned.

[0088] If it doesn't exist—that is, if the first row of data is retrieved for the first time by the client—then the first row of data retrieved here can be saved, and the first row of data at the time of this query can be regarded as the data from the previous query. Additionally, expired clients are cleaned up. Then, the polling interval is returned.

[0089] This process repeats continuously, querying the first row of data for each client at each query time, and then updating the second row of data for each client at each query time for that period.

[0090] Based on any of the above embodiments, the client read / write behavior profiling method further includes: Receive profile control commands based on external access interfaces; In response to the profile control command, display all or part of the read and write behavior profile data of one or more clients, or display the first or second behavior data of one or more clients, or reset the read and write behavior profile data of one or more clients.

[0091] Specifically, the external access interface is an interface provided to administrators for accessing and managing client read and write behavior profile data. The external access interface can be a command-line interface or a REST (Representational State Transfer) API (Application Programming Interface).

[0092] Administrators can issue profile control commands through external access interfaces. These profile control commands can specifically be commands to display / clear / reset the read / write behavior profiles of specified clients, or commands to display profiles of specified / all clients and their load types, or commands to display second-order data of specified / all clients, etc. This embodiment of the invention does not impose specific limitations on these commands.

[0093] Accordingly, the management node can receive profile control commands through an external access interface.

[0094] For example, the ceph fs client profile command can be received via the command-line interface.<client_id> This command is used to display the identifier.<client_id> The command `ceph fs client profile` can be used to profile the read and write behavior of a specified client; alternatively, it can be used via the command-line interface to receive the profile control command `ceph fs client profile list`, which lists a summary of all clients and their load categories; or it can be used via the command-line interface to receive the profile control command `ceph fs clientprofile reset`.<client_id> This command is used to clear the identifier.<client_id> A profile of the read and write behavior of a specified client; or you can receive profile control commands via the command-line interface: ceph fs client profiletimeseries<client_id> This command is used to return the identifier as<client_id> The second row of data is arranged in chronological order by time period for the specified client.

[0095] For example, you can receive the profile control command ` / api / fs / client-profiles` via the REST API GET method. This command lists all profiled clients. Alternatively, you can receive the profile control command ` / api / fs / client-profiles / ` via the REST API GET method. <id>This command is used to obtain the identifier as<client_id> This provides a profile of the read and write behavior of a specified client; alternatively, it allows receiving profile control commands via the REST API GET method: / api / fs / client-profiles / <id> / timeseries, this command is used to retrieve the identifier as<client_id> The second row of data for the specified client is arranged chronologically by time period; or profile control commands can be received via the REST API GET method at / api / fs / client-profiles / <id>This command is used to reset the identifier to<client_id> A profile of the read and write behavior of a specified client.

[0096] Based on this, responses can be made to profile control commands, thereby displaying all or part of the read / write behavior profile data of one or more clients, or displaying the first or second behavior data of one or more clients, or resetting the read / write behavior profile data of one or more clients. Here, "one or more clients" can refer to the one or more clients specified by the profile control command, or it can refer to all clients accessing the distributed file system; this embodiment of the invention does not specifically limit this.

[0097] In this embodiment of the invention, by providing a standardized external access interface, a unified batch view that provides a comprehensive overview is provided to the operation and maintenance personnel. Administrators do not need to query and manually calculate the data of each client one by one, and can quickly identify abnormal load clients in the cluster, which greatly improves the observability and operation and maintenance efficiency of the distributed file system.

[0098] Based on any of the above embodiments, the client read / write behavior profiling method further includes: The client's identifier and the date corresponding to the read / write behavior profile data are used as keys, and the summary sequence of the read / write behavior profile data is used as the value for key-value storage.

[0099] Specifically, management nodes can use key-value storage to persistently store client read and write behavior profile data.

[0100] Here, the key is a unique identifier. In this embodiment of the invention, the key can be a combination of the client's identifier and the date corresponding to the read / write behavior profile data, for example, it can be represented as: cephfs_profiler / portrait / <client_id> / <date>, where<client_id> This refers to the client's identifier, and <Date> refers to the date corresponding to the read / write behavior profile data.

[0101] The value can be a summary sequence of read and write behavior profile data, such as a string in JSON serialization format of the read and write behavior profile data.

[0102] Serializing and storing read / write behavior profiles for each client in key-value format enables the persistence of these profiles. For example, assuming each client's read / write behavior is profiled daily, only one profile is generated per client per day. If key-value storage is used for these profiles, each client generates only a minimal amount of data per day.

[0103] Furthermore, the distributed file system can be configured to automatically clean up expired data, for example, retaining it for 30 days by default. When the management node restarts or recovers from a failure, it can automatically read the serialized data saved above from the key-value store, thereby restoring the historical profiles of each client.

[0104] In this embodiment of the invention, low-overhead key-value persistent storage prevents the loss of historical data caused by restarting the management node, and supports administrators to perform retrospective analysis spanning several days or weeks, thereby enabling timely detection of long-term load migration or hidden abnormal changes.

[0105] Based on any of the above embodiments Figure 3 This is a schematic diagram of the architecture of the distributed file system provided by the present invention, such as... Figure 3 As shown, the management node 310 is connected to the metadata server 320 and the administrator operation and maintenance tool 330.

[0106] The management node 310 internally includes a stats module 311 and a read / write behavior profiling module 312. The stats module 311 is an inherent module in the management node 310. This module can aggregate performance metrics reported by each client from the metadata server 320 through a global metric query interface, thereby outputting the first behavior data of each client. The read / write behavior profiling module 312 is a newly added module in the management node, decoupled from the stats module. The read / write behavior profiling module 312 mainly consists of the following core components: The Client Profiler (313) is responsible for maintaining the background polling thread. It obtains the first row of data for each client at each query time by calling the global metric query interface, performs differential calculations on the first row of data from adjacent query times to obtain the second row of data for each time period, and maintains a circular buffer for each client to store the second row of data for each time period. Furthermore, the Client Profiler (313) applies the calculated incremental value after the differential calculation to perform counter reset detection. The PortraitBuilder 314 is responsible for performing multi-dimensional calculations on second-behavior data across multiple time periods, including at least one of the following: operation type distribution information, operation time period distribution information, throughput, single operation data volume, operation latency information, and cache efficiency information.

[0107] External access interface 315 (CLI / REST API) is used to provide a command line interface (CLI) and a representational state transition interface (REST API) to receive external profile control commands input by the administrator operation and maintenance tool 330, and output client read and write behavior profile data to the administrator operation and maintenance tool 330.

[0108] In this embodiment of the invention, the client read / write behavior profiling is entirely implemented by the management node, requiring no modification to the client code, metadata server code, new network message types, or changes to existing data paths. It can be hot-loaded and enabled with a single startup command, without restarting any cluster components or recompiling any C++ code, achieving zero-intrusion deployment of the read / write behavior profiling.

[0109] Furthermore, the implementation of client read / write behavior profiling does not add any logic to the client I / O path, and does not increase the latency of any I / O operations. All calculations are completed in the background thread of the management node, completely decoupled from the I / O data path, achieving zero overhead for hot paths.

[0110] Figure 4 This is the second flowchart of the client read / write behavior profiling method provided by the present invention, as follows: Figure 4 As shown, the method includes the following steps: After obtaining the profile control command based on the external access interface, the circular buffer corresponding to the client specified by the profile control command can be determined, and then the second behavior data of the specified client in multiple time periods can be loaded from the circular buffer.

[0111] Subsequently, for the specified client, the second row of data for all time periods is traversed. Based on this, aggregation and derived metric calculations are performed on the second row of data for all time periods.

[0112] The aggregated content may include: accumulating the incremental values ​​of read / write / metadata operations, accumulating the incremental values ​​of read / write bytes; accumulating the above incremental values ​​by hour into the activity distribution array of 24 slots; calculating the throughput; recording the peak value; and accumulating the time interval.

[0113] Calculating derived metrics can include using the following formula: Read-to-write ratio = (Incremental value of read operations) / (Incremental value of read operations + Incremental value of write operations); Average throughput = Total byte increment / Total time interval; Data volume per operation = Total byte increment / Total operation increment; In addition, peak hours are indexes of the hours with the highest total operation volume; active periods are lists of hours where the total operation volume exceeds 10% of the peak.

[0114] Afterward, load classification can be performed. Specifically, the client's load type can be determined based on a preset threshold. For example, the threshold can be set to 0.7, or it can be set to other values.

[0115] Finally, the complete read / write behavior profile data is assembled and returned through an external access interface. The output read / write behavior profile data can include information such as client identity information, operation type distribution information, operation time period distribution information, throughput, single operation data volume, operation latency information, and cache efficiency information.

[0116] For example, read / write behavior profile data may include the following information: #Identity Information client_id: Unique identifier for the client fs_name: Distributed file system name hostname: Client hostname mount_point: Mount point path first_seen: the time of the first query last_seen: the most recent query time profile_window_hours: Profile time window (hours) #Operation type distribution information total_read_ops: Total number of read operations total_write_ops: Total number of write operations total_metadata_ops: Total number of metadata operations rw_ratio: Read-to-write ratio workload_classification: workload type #Throughput avg_read_throughput_MBps: Average read throughput (MB / s) avg_write_throughput_MBps: Average write throughput (MB / s) peak_read_throughput_MBps: Peak read throughput (MB / s) peak_write_throughput_MBps: Peak write throughput (MB / s) #Data volume per operation avg_read_size_KB: Average read I / O size (KB) avg_write_size_KB: Average write I / O size (KB) #Operation delay information avg_read_lat_ms: Average read latency (ms) avg_write_lat_ms: Average write latency (ms) avg_metadata_lat_ms: Average metadata latency (ms) #Operation time period distribution information hourly_activity: 24-hour activity distribution .peak_hour: Peak hour .active_hours: List of active hours .read_ops_by_hour: Hourly distribution of read operations .write_ops_by_hour: Hourly write operation distribution .metadata_ops_by_hour: Hourly distribution of metadata operations #Cache efficiency information cap_hit_ratio: Cap hit rate dentry_lease_hit_ratio: Dentry lease hit rate In this embodiment of the invention, profile dimensions are derived, including operation type distribution information, operation time period distribution information, throughput, single operation data volume, operation latency information, and cache efficiency information, eliminating the need for administrators to manually calculate and organize them.

[0117] The client read / write behavior profiling device provided by the present invention is described below. The client read / write behavior profiling device described below can be referred to in correspondence with the client read / write behavior profiling method described above.

[0118] Figure 5 This is a schematic diagram of the client read / write behavior profiling device provided by the present invention, as shown below. Figure 5 As shown, this device is used in the management node of a distributed file system, and the device includes: The query unit 510 is used to obtain the first row of data of the client at the time of query based on the global indicator query interface of the distributed file system. The differential unit 520 is used to determine the second row data corresponding to the two query times based on the first row data of the client at the two query times; The profiling unit 530 is used to create a profiling of read and write behavior based on the client's second behavior data over multiple time periods.

[0119] The apparatus provided in this invention runs entirely on the management node of the distributed file system and obtains the first-line data by leveraging the existing global indicator query interface of the management node. Therefore, it allows for zero-intrusion deployment of client read / write behavior profiling functionality, and the client read / write behavior profiling is completely decoupled from the client I / O path, eliminating I / O path overhead. Compared to related technologies that only provide instantaneous snapshots, this invention obtains the second-line data by performing differential calculations on the first-line data and aggregates the second-line data from multiple time periods, transforming a single cumulative value into read / write behavior profiling data that includes a time dimension. This enables multi-dimensional characterization of client read / write behavior, and by continuously tracking the second-line data across multiple time periods, it effectively depicts the changing trends of client read / write behavior, providing more intuitive and comprehensive information for client behavior analysis in complex business scenarios.

[0120] Based on any of the above embodiments, the image-making unit is specifically used for: Based on the operation increment value of each operation of the client in the second behavior data of the multiple time periods, at least one of the following is determined: operation type distribution information, operation time period distribution information, throughput, and single operation data volume of the client. Based on at least one of the following: the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, a read and write behavior profile of the client is constructed.

[0121] Based on any of the above embodiments, the image-making unit is specifically used for: Based on at least one of the client's operation latency information and cache efficiency information, and at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, construct the client's read and write behavior profile data; The operation latency information is determined based on the latency data of each operation in the first row of data from the query time closest to the current time; the cache efficiency information is determined based on the cache hit rate in the first row of data from the query time closest to the current time.

[0122] Based on any of the above embodiments, the device further includes a data buffer unit, used for: The second row of data is stored in the circular cache area corresponding to the client. The capacity of the ring buffer is determined based on the total duration of the multiple time periods and the duration of each individual time period.

[0123] Based on any of the above embodiments, the differential unit is specifically used for: Based on the cumulative operation value of each operation in the first row of data of the client at the two query times, calculate the operation increment value of each operation in the time period corresponding to the two query times; If the operation increment value is non-negative, the operation increment value of each operation is recorded in the second row of data of the time period corresponding to the two query times; If the operation increment value is negative, the operation increment value of each operation in the second row of data corresponding to the two query times is set to empty.

[0124] Based on any of the above embodiments, the device further includes an interaction unit, used for: Receive profile control commands based on external access interfaces; In response to the profile control command, display all or part of the read and write behavior profile data of one or more clients, or display the first or second behavior data of one or more clients, or reset the read and write behavior profile data of one or more clients.

[0125] Based on any of the above embodiments, the device further includes a storage unit for: The client's identifier and the date corresponding to the read / write behavior profile data are used as keys, and the summary sequence of the read / write behavior profile data is used as the value for key-value storage.

[0126] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a client read / write behavior profiling method, which includes: Based on the global metric query interface of the distributed file system, the first row of data of the client at the time of query is obtained; Based on the first row of data from the client at two query times, determine the second row of data for the time period corresponding to the two query times; Based on the client's second behavior data across multiple time periods, a read / write behavior profile is created.

[0127] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the client read / write behavior profiling method provided by the above methods, the method comprising: Based on the global metric query interface of the distributed file system, the first row of data of the client at the time of query is obtained; Based on the first row of data from the client at two query times, determine the second row of data for the time period corresponding to the two query times; Based on the client's second behavior data across multiple time periods, a read / write behavior profile is created.

[0129] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the client read / write behavior profiling method provided by the above methods, the method comprising: Based on the global metric query interface of the distributed file system, the first row of data of the client at the time of query is obtained; Based on the first row of data from the client at two query times, determine the second row of data for the time period corresponding to the two query times; Based on the client's second behavior data across multiple time periods, a read / write behavior profile is created.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / id> < / id> < / id>

Claims

1. A method for profiling client read / write behavior, characterized in that, The method, applied to the management node of a distributed file system, includes: Based on the global metric query interface of the distributed file system, the first row of data of the client at the time of query is obtained; Based on the first row data of the client at two query times, determine the second row data of the time period corresponding to the two query times; Based on the client's second behavior data across multiple time periods, a read / write behavior profile is created.

2. The client read / write behavior profiling method according to claim 1, characterized in that, The process of creating a read / write behavior profile based on the client's second behavioral data across multiple time periods includes: Based on the operation increment value of each operation of the client in the second behavior data of the multiple time periods, at least one of the following is determined: operation type distribution information, operation time period distribution information, throughput, and single operation data volume of the client. Based on at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, a read and write behavior profile of the client is constructed.

3. The client read / write behavior profiling method according to claim 2, characterized in that, The process of constructing a client's read / write behavior profile based on at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume includes: Based on at least one of the client's operation latency information and cache efficiency information, and at least one of the client's operation type distribution information, operation time period distribution information, throughput, and single operation data volume, construct the client's read and write behavior profile data; The operation latency information is determined based on the latency data of each operation in the first row of data of the query time closest to the current time; the cache efficiency information is determined based on the cache hit rate in the first row of data of the query time closest to the current time.

4. The client read / write behavior profiling method according to any one of claims 1 to 3, characterized in that, Also includes: The second row of data is stored in the circular cache area corresponding to the client. The capacity of the ring buffer is determined based on the total duration of the multiple time periods and the duration of each individual time period.

5. The client read / write behavior profiling method according to any one of claims 1 to 3, characterized in that, The step of determining the second set of data corresponding to the two query times based on the first set of data from the client at the two query times includes: Based on the cumulative operation value of each operation in the first row of data of the client at the two query times, calculate the operation increment value of each operation in the time period corresponding to the two query times; If the operation increment value is non-negative, the operation increment value of each operation is recorded in the second row of data of the time period corresponding to the two query times; If the operation increment value is negative, the operation increment value of each operation in the second row of data corresponding to the two query times is set to empty.

6. The client read / write behavior profiling method according to any one of claims 1 to 3, characterized in that, Also includes: Receive profile control commands based on external access interfaces; In response to the profile control command, display all or part of the read and write behavior profile data of one or more clients, or display the first or second behavior data of one or more clients, or reset the read and write behavior profile data of one or more clients.

7. The client read / write behavior profiling method according to any one of claims 1 to 3, characterized in that, Also includes: The client's identifier and the date corresponding to the read / write behavior profile data are used as keys, and the summary sequence of the read / write behavior profile data is used as the value for key-value storage.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the client read / write behavior profiling method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the client read / write behavior profiling method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the client read / write behavior profiling method as described in any one of claims 1 to 7.

Images