Read-write mode recognition and model training

Abstract

Provided in the embodiments of the present application are a read-write mode recognition method, a model training method and apparatus, and a device and a medium. The read-write mode recognition method comprises: acquiring a read-write log sequence of a target storage device within a specified time range; inputting into a siamese network model a low-dimensional matrix corresponding to the read-write log sequence and a plurality of reference read-write modes included in a read-write mode library, and calculating the similarity between a read-write mode within the specified time range and the plurality of reference read-write modes, so as to obtain a plurality of similarity values; and on the basis of magnitude comparison results of the plurality of similarity values, selecting one of the plurality of reference read-write modes as a target read-write mode of the target storage device within the specified time range.

Description

[0001] Reading and writing pattern recognition and model training technology fields

[0002]

[0001] The present application relates to the field of computer technology, and more particularly to read-write pattern recognition and model training.

[0003]

[0002] When a storage device (which may be a local storage device) performs a working status detection, it is usually determined whether there is a problem with the storage device based on some common read / write mode indicators, such as the maximum number of IOPS per second (MaxIOPS) and the average number of IOPS per second (AvglOPS).

[0004]

[0003] In the related art, the work of determining the read and write mode of a storage device mainly relies on clustering judgment based on statistical indicators of some basic performance data, or setting threshold judgment methods. However, the accuracy of the judgment results of these methods is relatively low. It may even happen that different judgment results are obtained when the same basic performance data is input. Therefore, a solution that can improve the accuracy of storage device read and write mode recognition is needed.

[0005]

[0004] The various embodiments of the present application provide read-write pattern recognition, model training methods, devices, equipment and media.

[0006]

[0005] In a first aspect, in one embodiment of the present application, a read / write pattern recognition method is provided. The method comprises: obtaining a read / write log sequence of a target storage device within a specified time range; inputting a low-dimensional matrix corresponding to the read / write log sequence and a plurality of reference read / write patterns contained in a read / write pattern library into a twin network model, calculating the similarity between the read / write pattern within the specified time range and the plurality of reference read / write patterns, and obtaining a plurality of similarity values; and selecting, based on a comparison result of the plurality of similarity values, from the plurality of reference read / write patterns as a target read / write pattern of the target storage device within the specified time range.

[0007] In a second aspect, one embodiment of the present application provides a model training method. The method includes: constructing training samples using pattern vectors corresponding to multiple reference read and write patterns; inputting the training samples into a to-be-trained model comprising at least two sub-networks; and when the inter-vector distance between output vectors of multiple sub-networks that input the same training sample is less than a distance threshold, indicating that a twin network model has been trained.

[0008]

[0007] In a third aspect, in one embodiment of the present application, a model training device is provided, which includes: an acquisition module for acquiring a read-write log sequence of a target storage device within a specified time range; a calculation module for inputting a low-dimensional matrix corresponding to the read-write log sequence and a plurality of reference read-write patterns contained in a read-write pattern library into a twin network model, calculating the similarity between the read-write pattern of the specified time range and the plurality of reference read-write patterns, and obtaining a plurality of similarity values; a selection module for selecting one from the plurality of reference read-write patterns as a target read-write pattern of the target storage device within the specified time range based on a size comparison result of the plurality of similarity values.

[0009]

[0008] In a fourth aspect, in one embodiment of the present application, an electronic device is provided, comprising a memory and a processor; wherein the memory is used to store a program; and the processor is coupled to the memory and is used to execute the program stored in the memory, so as to implement the method described in the first aspect or the method described in the second aspect.

[0010]

[0009] In a fifth aspect, in one embodiment of the present application, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect, or executes the method described in the second aspect.

[0011]

[0010] In a sixth aspect, in one embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the method described in the first aspect or is used to implement the method described in the second aspect.

[0012]

[0011] The technical solution provided by the embodiments of the present application obtains a read / write log sequence of the target storage device within a specified time range when determining the operating status of the storage device. Generally, the log sequence is obtained within a continuous period of time, which can fully reflect the temporal correlation between each continuous data in the data set. Since the target storage device has a variety of read / write modes, a plurality of reference read / write modes are preset in the read / write mode library. Furthermore, a twin network model is used to determine which reference read / write mode the target storage device's read / write mode within the specified time range is more similar to. Specifically, the read / write log sequence and the reference read / write mode are respectively input into the twin network model. The twin network model determines the similarity between the read / write mode within the specified time range and the plurality of reference read / write modes, and calculates a similarity value. Furthermore, the comparison results of the plurality of similarity values ​​are compared. Based on the comparison results, the target object mode corresponding to the target storage device within the current specified time range is determined. The above solution eliminates the need to directly calculate or determine read / write patterns based solely on a dataset. Instead, it uses the twin network model to determine the similarity between the target storage device's read / write patterns and the reference read / write patterns within a specified time range. This approach focuses on the differences between the twin network model inputs, resulting in better generalization capabilities and more accurate read / write pattern determination results.

[0013]

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] FIG1 is a schematic diagram of a read-write pattern recognition system according to an embodiment of the present application;

[0015] FIG2 is a schematic flow chart of a read-write pattern recognition method according to an embodiment of the present application;

[0016] FIG3 is a schematic diagram of a process flow of a dimensionality reduction processing method provided in an embodiment of the present application;

[0017] FIG4 is a schematic diagram of a process flow of a dimensionality reduction process illustrated in an embodiment of the present application;

[0018]

[0017] Figure 5a is a schematic diagram of a twin network model structure provided in an embodiment of the present application;

[0019]

[0018] Figure 5b is a schematic diagram of another twin network model structure provided in an embodiment of the present application;

[0020]

[0019] FIG6 is a schematic diagram of the structure of a storage device group provided in an embodiment of the present application;

[0021] FIG7 is a flow chart of a model training method according to an embodiment of the present application;

[0022]

[0021] FIG8a is a schematic diagram of a read-write mode recognition device provided in an embodiment of the present application;

[0023]

[0022] Figure 8b is a schematic diagram of a model training device provided in an embodiment of the present application;

[0024]

[0023] FIG9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025]

[0024] In order to enable people skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0026] Some processes described in the specification, claims, and figures of this application include multiple operations that appear in a specific order. These operations may be executed in a different order than the order in which they appear herein or in parallel. Operation sequence numbers, such as 101 and 102, are merely used to distinguish between different operations and do not represent any specific order of execution. Furthermore, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that terms such as "first" and "second" are used herein to distinguish between different messages, devices, modules, etc., and do not represent a sequential order or limit the "first" and "second" to different types. Furthermore, the embodiments described below represent only a portion of the embodiments of this application, and are not exhaustive. All other embodiments derived by those skilled in the art based on the embodiments of this application without inventive effort are considered within the scope of protection of this application.

[0027] In storage device applications, the read / write pattern (IO pattern) of the storage device can reflect the performance of the storage device in the current time, such as whether there are performance issues such as slow read / write speed and long response time. In related technologies, some common indicators are usually used to determine whether a storage device has performance issues. For example, the maximum number of IOPS (Maximum Input / Output Operations Per Second, MaxIOPS) and the average number of IOPS per second are used.

[0028] (Average Input / Output Operations Per Second, AvglOPS) and other indicators are used to judge the performance of storage devices when they are located on different machines, or to compare the performance results of different application requirements when running on the same storage device. Then, the execution indicators and performance comparison results are used to determine whether there are problems with the storage device. If a machine learning model is used to judge the performance of a storage device, the accuracy is relatively low or recognition errors occur when using traditional machine learning models to accurately identify the performance or working mode of the storage device due to the large number of indicators involved and the high similarity between indicators of different performances. Therefore, there is a need for a solution that can accurately judge the read and write mode of a storage device without relying entirely on the accuracy of the model input data or the accuracy of the feature extraction of the machine learning model. Terminology Explanation

[0029] Read / Write Pattern IO Pattern: IO behavior, a signal pattern. It can be understood as the pattern or behavior of input / output operations performed on a storage device (such as a Solid State Drive (SSD)). It has a significant impact on the performance of the storage device. IO Pattern affects the garbage collection (GC) data layout within the storage device, indirectly affecting the amount of data moved during the GC process. The amount of data moved during the GC process determines the backend traffic or workload of the storage device. When the IO Pattern is fully sequential, it is considered the most friendly mode for GC within the storage device. In this mode, the write amplification factor (the ratio between the data actually written to the storage device and the data requested to be written) is close to 1, thereby achieving better performance. The fully sequential IO Pattern reduces the need for data rearrangement during the GC process. On the other hand, when the IO Pattern is random in small blocks, it will result in more GC data movement, thereby reducing performance. Small random 10 patterns increase the write amplification factor and require more data movement during garbage collection, which may negatively impact performance.

[0030] It should be noted that a fully sequential IO pattern means that during input / output operations, data is read or written completely sequentially, without skipping or random access. This sequential IO pattern is generally more efficient because storage devices (such as hard disks) can more effectively utilize cache and prefetch mechanisms during sequential access, reducing seek time and rotational delay, thereby increasing data transmission speed. A small-block random IO pattern means that during input / output operations, data is read or written in small blocks, and the locations of these small blocks on the storage medium are random. This IO pattern generally means that data access is decentralized, without a fixed order or pattern.

[0031] Storage devices: These can generally be categorized as local storage devices and cloud storage devices. Cloud storage devices are standalone computing devices or virtual machine (VM) images used to move data from a local site to a remote cloud storage service provider. These devices can store primary data, backup data, and archived data. They can be proprietary vendor servers with pre-installed software or virtual devices installed on commodity hardware. [Illegible]: Disk access, with each access being recorded as an Illegible record.

[0032]

[0030] 10. OTrace: 10 log sequences.

[0033]

[0031] MaxIOPS: Maximum number of IOPS per second.

[0034] AvglOPS: average number of OOPs per second.

[0035]

[0033] tsfresh: a time series feature extraction tool.

[0036] Cosine distance: This can also be called cosine similarity. In geometry, the cosine of the angle can be used to measure the difference in the directions of two vectors. This concept is used in machine learning to measure the difference between sample vectors.

[0037]

[0035] Siamese Network Model: A Siamese Network is a neural network structure used to learn similarity or distance metrics between samples. It consists of two or more subnetworks that share parameters and have the same structure and weights. The core concept of a Siamese Network is to process input samples using subnetworks with shared weights and map them into a shared feature space. In this way, the position or representation of the input sample in the feature space can be used to measure the similarity or distance between samples. Specifically, for each input sample, the Siamese Network inputs it into two subnetworks for processing. These two subnetworks share the same parameters and weights, so they perform the same feature extraction operation on the input sample. Ultimately, each subnetwork maps the input sample into the feature space, obtaining a corresponding feature vector. In the feature space, various similarity metrics (such as Euclidean distance and cosine similarity) can be used to calculate the similarity or distance between samples. In this way, the Siamese Network can determine the degree of similarity between input samples by comparing the distance or similarity between feature vectors.

[0038] To address the above-mentioned technical problems, the present application proposes a read / write pattern recognition solution. FIG1 is a schematic diagram of a read / write pattern recognition system provided by an embodiment of the present application. As shown in FIG1 , a target storage device provides continuous read / write records within a specified time range. The resulting read / write records are converted (e.g., via blktrace) into an analyzable text format, 1OTrace (read / write log sequence). This read / write log sequence is then input into a trained read / write pattern recognition model. In the Siamese network model, the entire read / write log sequence 1OTrace is then subjected to dimensionality reduction (embedding) to map it into a vector matrix. The resulting vector matrix is ​​then fed into the Siamese network model and multiple reference read / write pattern datasets in a read / write pattern library for recognition and scoring. A discriminator is then used to perform similarity determination. The maximum similarity value obtained after analysis is used as the similarity of the closest 10 target read / write patterns. The reference read-write mode can then be used as the target read-write mode.

[0039]

[0037] The technical solution implemented in the present application will be explained below in conjunction with specific embodiments.

[0038] FIG2 is a flow chart of a read / write pattern recognition method provided in an embodiment of the present application. The method can be applied to a server (including a local server and a cloud server). As can be seen from FIG2, the method specifically includes the following steps 201 to 203.

[0040]

[0039] 201: Obtain a read and write log sequence of a target storage device within a specified time range.

[0041]

[0040] 202: The low-dimensional matrix corresponding to the read-write log sequence and the multiple reference read-write patterns contained in the read-write pattern library are respectively input into the twin network model, and the similarity between the read-write pattern of the specified time range and the multiple reference read-write patterns is calculated to obtain multiple similarity values.

[0042]

[0041] 203: According to the comparison results of the multiple similarity values, select one from the multiple reference read-write patterns as the target read-write pattern of the target storage device in the specified time range.

[0043] The target storage device referred to herein may be a local storage device or a cloud storage device. It should be noted that the target storage device is in a continuously operating state, meaning that the read / write mode of the target storage device may change dynamically in real time. Therefore, when determining the read / write mode of the target storage device, it is necessary to clearly define the time range within which the read / write mode is determined. In other words, the read / write mode may be different within different time ranges.

[0044]

[0043] As mentioned above, there are many types of read / write patterns (IO patterns), and the read / write pattern needs to be determined based on the read / write log sequence analysis. For example, common sequential random access models include the following.

[0045] Large Block Sequential: This mode reads or writes data sequentially in larger blocks (e.g., consecutive sectors or pages). Large Block Sequential mode can improve data transfer efficiency and reduce access latency to storage devices.

[0046]

[0045] Mixed Sequential: This mode refers to a combination of sequential and random access. In mixed sequential mode, there may be some continuous data access, but there may also be some random access patterns.

[0047] Random: This mode refers to reading or writing data in a random order, without obvious order. Random mode usually leads to higher access latency and greater data fragmentation.

[0048] Sequential Read / Random Write: This mode reads data sequentially but writes data randomly. This mode is common in certain application scenarios, such as logging, where read operations are more frequent and write operations are randomly distributed.

[0049] Random Read / Sequential Write: This mode reads data randomly but writes it sequentially. This mode is common in certain database applications, where reads are random and writes are sequential.

[0050]

[0049] In addition, there are other non-sequential read and write modes, such as the following read and write modes.

[0051]

[0050] Frequent read after write (Frequent Read After Write) mode refers to reading a piece of data multiple times immediately after writing it. This type of behavior can easily cause relatively high read latency.

[0052]

[0051] The frequent write after write (Frequent Write After Write) mode refers to the continuous duplication of the same data. This type of behavior actually has a great impact on the performance of the storage device and increases latency.

[0053]

[0052] High Frequent Access mode refers to a period of time when I / O data surges, resulting in high IOPS. This behavior can easily cause I / O queuing, leading to slow I / O and impacting access speed.

[0053] High Throughput Access mode refers to a period of time when a disk processes a large number of data read and write operations. If the disk's I / O capacity is insufficient, the disk's I / O performance limit can be exceeded, impacting the disk's I / O speed and stability.

[0054] It should be noted that the present solution can recognize a variety of read / write pattern types. The various read / write patterns in the above embodiments are provided for illustrative purposes only and do not constitute a limitation on the technical solution of the present application. In actual applications, users can add various types of reference read / write patterns (including but not limited to the sequence-dependent read / write patterns and non-sequence-dependent read / write patterns proposed in the above embodiments) to the read / write pattern library as needed, and the read / write pattern library can be dynamically updated as needed.

[0055] In order to facilitate the subsequent use of the twin network model to better determine the read and write patterns of the target storage device within a specified time range, it is necessary to establish a read and write pattern library. The read and write pattern library pre-stores multiple pattern vectors of reference read and write patterns. For example, these may be large block sequential pattern vectors, mixed sequential pattern vectors, random pattern vectors, read-after-write pattern vectors, write-after-write pattern vectors, high-frequency access pattern vectors, and so on. The read and write pattern library can be maintained as needed, that is, the pattern vectors in the read and write pattern library can be added or reduced as needed.

[0056] In the present application, the simultaneous comparison of an input read / write log sequence and a reference read / write pattern is used as an example for illustration. Multiple similarity values ​​are obtained through multiple similarity calculations. In practical applications, the input read / write log sequence can be compared with multiple reference patterns simultaneously to obtain multiple similarity values. In the twin network model, the two subnetworks for receiving the read / write log sequence and the reference read / write pattern are identical subnetworks with the same model parameters.

[0057] After obtaining the similarity values ​​through the above scheme, the magnitude relationship of multiple similarity values ​​will be further compared. Here, the judgment threshold corresponding to the similarity value may not be set, but the reference read-write mode corresponding to the maximum similarity value is selected as the target read-write mode of the target storage device within the specified time range. When performing similarity calculation, the similarity is calculated using the twin network model, that is, only the similarity between the twin network model input data needs to be considered, without considering the input data type, quantity, or model extraction feature accuracy. In other words, the influence of various factors on the input data can be eliminated, and the input data can be directly compared for similarity, thereby more accurately judging the target read-write mode of the target storage device within the specified time range.

[0058]

[0058] In addition, the technical solution of the present application obtains a read / write log sequence within a specified time range, which may be a log sequence of a single read / write mode, or a log sequence involving two different read / write modes, but the time lengths occupied by these two different read / write modes are different. If the traditional method is adopted, it is easy to obtain an erroneous read / write mode judgment result. The technical solution of the present application uses a twin network model to perform read / write mode similarity calculation. When performing feature processing on the input read / write log sequence, the influence of the temporal sequence in the collected read / write log sequence on the read / write mode judgment result is fully considered. By performing similarity comparison with a reference read / write pattern, the influence of the read / write log sequence with a longer time length can be fully reflected, and thus a reference read / write pattern similar to the read / write log sequence with a longer time length can be accurately calculated.

[0059] In one or more embodiments of the present application, FIG3 is a flow chart illustrating a dimensionality reduction method provided in an embodiment of the present application. As shown in FIG3 , the method specifically includes the following steps: After obtaining a read / write log sequence of a target storage device within a specified time range, the method further includes: 301: converting the read / write log sequence into a data set comprising a time element, a spatial element, and an operation type; 302: performing dimensionality reduction processing based on the data set to obtain a low-dimensional matrix. The time element includes an access start timestamp and an access end timestamp; the spatial element includes an access address offset and an access address size.

[0060] As described in step 302, dimensionality reduction processing is performed on the data set to obtain a low-dimensional matrix, specifically including: 3021: Establishing a timing matrix based on the operation type, access start timestamp, access end timestamp, access address offset, and access address size within the specified time range; 3022: Performing data statistical processing on the timing matrix to generate a vector matrix containing timing sequences; 3023: Performing timing feature extraction on the vector matrix to obtain the low-dimensional matrix that preserves the timing relationship.

[0060]

[0061] Each storage device has its own operational IO record. To facilitate subsequent processing, the acquired IO record is converted into text format, and the time element, spatial element, and operation type contained therein are extracted. It should be noted that each storage device has its own corresponding IO record. Generally speaking, to analyze different storage devices, it is necessary to obtain the corresponding IO records for each. Of course, if multiple storage devices perform the same IO task, one of the multiple associated storage devices can be selected as the target storage device.

[0061]

[0062] The time element mentioned here can be understood as the time sequence of accessing the storage space, the access start timestamp (submit_time), and the access end timestamp (end_time); the space element can be understood as the storage space, and the specific storage space indicators can be the access address offset (offset) and the access address size (length).

[0062]

[0063] Specifically, the operation type (op) indicates the type of operation, such as read or write. The operation type describes whether the operation reads data from the storage device or writes data to the storage device.

[0063]

[0064] Access address offset (offset): Indicates the location on the storage device that the operation accesses. The offset indicates the location from the beginning of the storage device where the operation reads or writes data.

[0064]

[0065] Access address size (length): Indicates the size of the data read or written by the operation. The address size indicates the amount of data involved in the operation, which can be bytes, blocks, or other units.

[0065]

[0066] Access start timestamp ( submit_time ): Indicates the start time of the 10 operation. The start timestamp records the time point when the 10 operation is initiated, usually expressed in system clock or other time units.

[0066]

[0067] Access end timestamp (end_time): Indicates the end time of the 10 operation. The end timestamp records the time point when the 10 operation is completed, that is, the end time of the data read or write operation.

[0067]

[0068] In step 302, the dataset is subjected to dimensionality reduction processing using an embedding method. FIG4 is a flow chart illustrating the dimensionality reduction process according to an embodiment of the present application. As can be seen from FIG4, a time series consisting of five tuples (op, offset, lengths submit_time> end_time) (that is, the read-write log sequence mentioned above) cannot be directly analyzed and recognized by a neural network, so an embedding strategy is required. Here, a 10-to-Vector (10 to 5 columns) mapping method is designed to map it to a low-dimensional matrix. Generally speaking, the value of N is usually

[0068]

[0069] Four features are extracted from lOTrace data as follows.

[0069]

[0070] First, load the lOTrace data: Read and load the lOTrace data into the parsing environment to parse the data and obtain the necessary fields, such as the five-tuple operation type, offset, size, submission time, and end time. Perform necessary preprocessing on the lOTrace data, such as data cleaning, filtering, or standardization, depending on the specific requirements of the feature extraction process. After preprocessing, the extracted features include the following.

[0070]

[0071] I0PS series: Calculates the number of 10 operations per second and forms a time series.

[0071]

[0072] BPS series: Calculates the number of bytes per second and forms a time series.

[0073] Sequential access ratio series: Calculate the ratio of sequential access 10 operations per second and form a time series.

[0072]

[0074] Read-write ratio series ( rw_ratio series ): Calculates the ratio of read and write operations per second and forms a time series.

[0073]

[0075] Implement the algorithm or calculation to extract features from the lOTrace data based on the defined feature extraction method. For each time step, calculate the IOPS, BPS, sequential access ratio, and read / write ratio and form them into an M x 4 vector matrix. Store the extracted features in an appropriate data structure or use them directly for further analysis, visualization, or modeling. The extracted feature vector matrix will contain the IOPS, BPS, sequential access ratio, and read / write ratio for each time step.

[0074]

[0076] Dimensionality reduction is then performed based on the extracted features. For example, the tsfresh time series feature extraction tool can extract 30 different types of time series features (Statistical Features), including maximum, mean, standard deviation, and kurtosis. Mapping the entire time series access sequence into a 30 x 4 vector matrix reduces the complexity of the overall analysis and preserves the time series relationships through second-order dimensionality reduction. The specific steps are as follows.

[0075]

[0077] Prepare the overall time series access sequence, including the IOPS sequence, BPS sequence, sequential access ratio sequence, and read / write ratio sequence. Use the time series feature extraction function in the tsfresh tool library to extract features from the time series access sequence. You can select the feature type to be extracted, such as maximum, mean, standard deviation, and kurtosis, totaling 30 different feature types. For each feature type, use the extraction function to calculate the corresponding eigenvalue. For example, for the maximum feature, calculate the maximum value of the entire time series access sequence. Sequentially place the eigenvalues ​​of each feature type in a 30 x 4 vector matrix. Each row represents a feature type, and each column represents IOPS, BPS, sequential access ratio, and read / write ratio. This feature extraction process results in a 30 x 4 vector matrix, where each row represents a feature type, and each column represents IOPS, BPS, sequential access ratio, and read / write ratio. This dimensionality reduction method reduces the complexity of the overall analysis and preserves temporal relationships, ensuring that the extracted features still contain temporal information.

[0076]

[0078] As an alternative, in addition to the dimensionality reduction schemes described above, principal component analysis (PCA) and linear discriminant analysis (LDA) can also be used to achieve dimensionality reduction. Taking PCA as an example, before dimensionality reduction, a time series matrix is ​​constructed from the quintuples. For each element in each quintuple, the time series matrix can be constructed with the numerical features as columns and the time as rows. If the data is arranged in time series, each row represents the eigenvalue at a time point. Before dimensionality reduction, the data is usually normalized to eliminate dimensional differences between different features. Normalization typically involves calculating the mean and standard deviation of each feature, then subtracting the mean from each eigenvalue and dividing it by the standard deviation. Next, the covariance matrix is ​​calculated. The covariance matrix is ​​a key component of PCA and measures the correlation between different features. By calculating the covariance matrix of the normalized data, the linear relationship between features can be understood. The eigenvectors and eigenvalues ​​of the covariance matrix are then calculated. The eigenvectors represent the main direction of change in the data, while the eigenvalues ​​measure the amount of change in each direction. The eigenvectors with the largest eigenvalues ​​are selected as principal components. Finally, the original data is projected onto these principal components to obtain the reduced-dimensional data. This is usually achieved by calculating the dot product between the original data and each principal component. The reduced-dimensional data retains the main variation patterns in the original data, but with a significantly reduced dimensionality.

[0077]

[0079] In an optional solution, performing data statistical processing based on the time series matrix to generate a vector matrix containing a time series sequence includes: performing data statistics based on the time series matrix to obtain the time series, where the time series includes: a read and write number sequence, a byte number sequence, a sequential access read and write ratio sequence, and a unit time read and write ratio sequence; bucketing the time series to extract a feature vector corresponding to each bucket; and vertically splicing the feature vectors based on the time sequence of the specified time range to obtain the vector matrix.

[0078]

[0080] As mentioned above, the original N * 5 time series data needs to be converted into an M * 4 vector matrix. In an alternative solution, statistical methods and bucketing can be used to illustrate how to perform this conversion. First, assume there is an original N * 5 time series data matrix, where each row represents a time step and each column represents a feature (for example, IOPS, BPS, sequential access ratio, read / write ratio, etc.). When vertically concatenating the four sequences, create an empty M * 4 matrix to store the concatenated results. Place the IOPS sequence in the first column of the matrix. Ensure that the length of the IOPS sequence matches M. If the length does not match, interpolation or truncation may be required. Place the BPS sequence in the second column of the matrix. Similarly, ensure that the length of the BPS sequence matches M. Place the sequential access ratio sequence in the third column of the matrix, with a length matching M.

[0079]

[0081] Finally, place the read / write ratio sequence in the fourth column of the matrix, with a length matching M. This vertical concatenation results in an M x 4 vector matrix, where each row represents a time step and each column represents a feature sequence (IOPS, BPS, sequential access ratio, and read / write ratio). For each feature, statistical calculations and bucketing are required. This involves calculating statistical metrics such as maximum, mean, standard deviation, and kurtosis, and bucketing the raw data into different intervals. For example, for each feature, statistical methods can be used to calculate metrics such as maximum, mean, standard deviation, and kurtosis. These metrics provide information about the distribution and variation of the feature.

[0080]

[0082] Next, the raw data for each feature is bucketed. Bucketing divides continuous numerical data into discrete intervals. Different bucketing methods can be selected based on data distribution and requirements, such as equal-width bucketing or equal-frequency bucketing. After bucketing, statistical metrics such as the amount of data, mean, and standard deviation for each feature in each bucket can be calculated. These statistical metrics can be used as new features and placed at corresponding positions in the M x 4 vector matrix.

[0081]

[0083] Using this statistical method and bucketing, we can convert the original N * 5 time series data into an M * 4 vector matrix, where each row represents a time step and each column represents a feature. This conversion can help reduce the dimensionality and complexity of the data and provide statistical information about the distribution and variation of the time series data.

[0082]

[0084] In one or more embodiments of the present application, the method for constructing the preset read-write pattern library includes: collecting historical read-write log sequences of multiple storage devices including the target storage device; classifying the historical read-write log sequences into read-write patterns, and marking the classification results as reference read-write patterns; establishing a read-write pattern cluster consisting of the read-write log sequences based on the reference read-write patterns; and establishing a read-write pattern library containing pattern vectors based on multiple read-write pattern clusters.

[0083]

[0085] In practical applications, the specific method for building a read / write pattern library is as follows: Collect IQTrace (read / write log sequence) data for known I / O pattern types (i.e., read / write pattern types). This data should include a variety of patterns, such as sequential reads, random writes, and mixed read / writes. The data used to build the pattern library often comes from historical read / write log sequences of multiple real storage devices (preferably including the target storage device). Of course, if the data is incomplete, simulations or experiments can also be used to generate historical read / write log sequences.

[0084]

[0086] Each piece of collected lOTrace data needs to be labeled with a corresponding reference read / write pattern based on its behavior. For example, lOTrace data that is read sequentially can be labeled as "sequential read," lOTrace data that is written randomly can be labeled as "random write," and so on. This process can be performed manually or through an automated algorithm.

[0085]

[0087] Features are extracted from each labeled lOTrace data entry. Features can include the number of IOTrace operations, byte count, access pattern, and read / write ratio. These features help describe the characteristics and behavior of IOTrace patterns. Furthermore, clustering algorithms are used to analyze the labeled lOTrace data. Clustering algorithms group lOTrace data with similar features into the same category. Commonly used clustering algorithms include K-means clustering, hierarchical clustering, and density clustering. Clustering analysis can group similar lOTrace data together to form IOTrace pattern clusters (i.e., read / write pattern clusters).

[0086]

[0088] Based on the results of cluster analysis, lOTrace data belonging to the same cluster is grouped into corresponding 10-pattern templates. Each 10-pattern template represents a 10-pattern cluster and contains representative lOTrace data and characteristic information for that cluster. The 10-pattern templates (i.e., reference read and write patterns) can be saved as structured data, such as a database or pattern vector.

[0087]

[0089] Through the above steps, a library of read / write patterns can be established. This library can be used for subsequent pattern recognition and classification tasks when input into the twin network model. When new lOTrace data requires pattern classification, its features can be extracted and compared with the reference read / write patterns in the library to find the best matching reference pattern for classification and recognition. The library can help understand and analyze the characteristics, behavior, and performance of read / write patterns, providing a reference for performance optimization and adjustment.

[0088]

[0090] There are multiple ways to calculate similarity using the twin network model. The following describes each of them in detail.

[0089]

[0091] In one or more embodiments of the present application, the step of inputting the read / write log sequence and the multiple reference read / write patterns into a twin network model, calculating the similarity between the read / write log sequence and the multiple reference read / write patterns, and obtaining multiple similarity values ​​includes: inputting the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork; including the first subnetwork and the second subnetwork in the twin network model; outputting a first output vector corresponding to the read / write log sequence through the first subnetwork; obtaining second output vectors corresponding to the multiple pre-stored reference read / write patterns; and performing similarity calculation on the first output vector and the second output vector to obtain the similarity value.

[0090]

[0092] Among them, the obtaining of the second output vectors corresponding to the multiple pre-stored reference read-write patterns includes: inputting the pattern vectors corresponding to the selected multiple reference read-write patterns into the second sub-network respectively; saving the second output vectors corresponding to the multiple reference read-write patterns output by the second sub-network, so as to provide the pre-stored multiple second output vectors when the first sub-network outputs the first output vector.

[0091]

[0093] FIG5a is a schematic diagram of a twin network model structure provided in an embodiment of the present application. FIG5a illustrates that, as an implementable solution, each time similarity is calculated, the low-dimensional matrix corresponding to the read / write log sequence (vector 1 in FIG5a ) is simply input into the first subnetwork (e.g., including the convolutional layers conv1 and conv2 and the fully connected layers dens1 and dens2 shown in FIG5a ). The second output vector (G(V2)) corresponding to the reference read / write pattern (vector2 in FIG5a ) is pre-calculated and stored in a designated storage space. The first subnetwork then performs feature processing and outputs the first output vector (G(V1)) corresponding to the read / write log sequence. The first output vector and the second output vector correspond to each other, that is, the first output vector and the second output vector obtained after feature extraction by the first sub-network and the second sub-network having the same model parameters (for example, including the convolutional layers conv1 and conv2 and the fully connected layers dens1 and dens2 shown in FIG5a ) have feature vectors with the same dimension, the same numerical range, and the same distribution.

[0092]

[0094] If the first output vector and the second output vector are relatively consistent, they can be directly provided to the discriminator (that is, the loss function) for similarity calculation. If the consistency is poor, normalization is required. In practical applications, the cosine similarity between the first output vector and the second output vector can be calculated. When the cosine similarity is greater than the similarity threshold (this threshold can be set as needed), the consistency is considered poor; otherwise, the consistency is considered good. As shown in Figure 5a, the two output vectors G(v1) and G(v2) are obtained. The two vectors are fed into the discriminator for distance metric calculation. Here, the distance metric is used.<G(Vl), G(V2)> Also known as the loss function, Cosine similarity is used here as the distance metric.

[0093]

[0095] Specifically, in a twin network model, the two sub-network models each output two output vectors G(v1) and G(v2). These two vectors can be used to calculate the similarity or distance between them. Typically, when calculating the similarity of output vectors, cosine similarity can be used as a distance metric.

[0094]

[0096] To calculate the similarity between the output vectors G(v1) and G(v2), the following steps can be performed.

[0095]

[0097] First, the output vectors G(v1) and G(v2) are normalized. Normalization can, for example, normalize the length of the vectors to 1, making them more comparable in similarity calculations. A common normalization method is to divide the vector by its L2 norm (Euclidean norm). After normalization, the similarity of the output vectors is calculated using cosine similarity as a distance metric. Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them. Cosine similarity ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity between the two vectors and values ​​closer to -1 indicating less similarity between the two vectors. The formula for calculating cosine similarity is as follows:

[0096]

[0098] Where x and y represent the normalized output vectors G(v1) and G(v2), respectively.

[0097]

[0099] After calculating the similarity, the cosine similarity formula can be used to calculate the similarity between the output vectors G(v1) and G(v2). A large result (for example, greater than the similarity threshold) indicates that the first and second output vectors are similar. A small result (for example, less than the similarity threshold) indicates that the first and second output vectors are dissimilar. In other words, a larger cosine similarity value indicates that the two vectors are more similar, while a smaller cosine similarity value indicates that the two vectors are less similar. By minimizing this loss function, the model can learn to map similar input samples to similar feature representations, thereby achieving training and optimization of the Siamese network model. The similarity between the output vectors G(v1) and G(v2) can be calculated by performing cosine similarity calculation on the normalized vectors. This similarity value can be used as a distance metric or part of a loss function to measure the similarity between two input samples in the feature space. That is, when identifying the read / write pattern of the target storage device, only the difference between the read / write sequence within the specified time range and the reference read / write pattern needs to be considered. The first and second sub-networks are identical and executed simultaneously. This effectively avoids interference caused by sub-network differences and reduces the tedious process of manual feature extraction and selection, as well as the adverse effects caused by inconsistencies in feature extraction.

[0098]

[0100] In practical applications, to improve computational efficiency, the second output vectors can be pre-stored. For example, a read / write pattern library can pre-store three different reference read / write patterns and their corresponding pattern vectors. The calculation results corresponding to these three reference read / write patterns, that is, the three second output vectors, can be stored together. Of course, these can be stored together in the read / write pattern library or separately. When similarity calculation is required, there is no need to repeatedly calculate the second output vector. After obtaining the first output vector, the similarity can be directly compared between the first output vector and the three second output vectors. This saves the time of repeatedly calculating the second output vector.

[0101] It should be noted that if any one or more parameters in the three reference read / write modes change, the write operation needs to recalculate the second output vectors for that one or more reference read / write modes, and the recalculated results replace the previous second output vectors. Alternatively, if subnetwork parameters change, the three aforementioned second output vectors need to be recalculated.

[0099]

[0102] Furthermore, the read / write pattern library needs to be dynamically updated based on actual needs. For example, when comparing multiple similarity values ​​with a preset similarity threshold, if all similarity values ​​are less than the threshold, the read / write pattern corresponding to the specified time range is determined to be the new read / write pattern. This new read / write pattern is then added to the read / write pattern library as a new reference read / write pattern.

[0100]

[0103] If all similarity values ​​are less than the similarity threshold during the similarity comparison, the read / write pattern corresponding to the specified time range is a new read / write pattern that differs from any existing reference read / write pattern in the existing read / write pattern library. Therefore, this new read / write pattern needs to be added to the current read / write pattern library as a reference. Timely updates to the read / write pattern library based on similarity comparison results can better meet read / write pattern recognition requirements. Furthermore, the dynamic update method based on these different read / write pattern libraries provides this solution with enhanced adaptability.

[0101]

[0104] The following example illustrates another implementation of similarity calculation using a twin network model. Inputting the read / write log sequence and multiple reference read / write patterns into the twin network model, calculating the similarity between the read / write log sequence and the multiple reference read / write patterns, and obtaining multiple similarity values ​​includes: inputting the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork to obtain a first output vector; inputting the pattern vectors corresponding to the selected multiple reference read / write patterns into a second subnetwork to obtain a second output vector; and inputting the first and second output vectors into a discriminator to perform similarity calculations to obtain the similarity values.

[0102]

[0105] Figure 5b is a schematic diagram of another twin network model structure provided in an embodiment of the present application. As shown in Figure 5b, as an implementable solution, each time similarity is calculated, the read / write log sequence and the pre-stored reference read / write pattern can be input into the first sub-network and the second sub-network, respectively. The first sub-network then performs the calculations. The first sub-network outputs a first output vector corresponding to the read / write log sequence, and the second sub-network outputs a second output vector corresponding to the reference read / write pattern. The first and second output vectors correspond, meaning that the first and second output vectors, obtained after feature extraction using the first and second sub-networks, which have identical model parameters, have feature vectors with the same dimensions, numerical range, and distribution.

[0103]

[0106] If the first output vector and the second output vector have good consistency, they can be directly provided to the discriminator (that is, the loss function) for similarity calculation. If the consistency is not good, normalization processing is required first. As shown in Figure 5b, the two output vectors G(v1) and G(v2) are obtained. The two vectors enter the discriminator for distance metric calculation. Here, Distance<G(Vl), G(V2)> Also known as a loss function, Cosine similarity is used as the distance metric here. The specific calculation process can be found in the embodiment corresponding to FIG5a, and will not be repeated here.

[0104]

[0107] In practical applications, it's necessary to compare the similarity of read / write log sequences within a specified time range with multiple reference read / write patterns. This requires inputting a low-dimensional matrix corresponding to a single read / write log sequence into the first subnetwork, and inputting pattern vectors corresponding to multiple reference read / write patterns into the second subnetwork. Similarity is then calculated between a first output vector and multiple second output vectors. To improve computational efficiency, multiple identical second subnetworks (also known as third, fourth, and so on) can be established. When establishing multiple second subnetworks, multiple discriminators can also be provided. Generally, the number of second subnetworks can be the same as the number of discriminators. This eliminates the need for queuing when calculating similarity, allowing parallel computations.

[0105]

[0108] In the above solution, when calculating similarity, the reference read / write pattern and the acquired read / write log sequence are input into the model and calculated separately, enabling real-time similarity calculation. In particular, when the reference read / write pattern is updated, or when the parameters of the first and second sub-networks are updated, the second output vector will also change. Therefore, the second sub-network must be re-input to obtain the latest second output vector. This meets the need for read / write pattern judgment during iterative updates of sub-networks or reference read / write patterns.

[0106]

[0109] In one or more embodiments of the present application, obtaining a read and write log sequence of a target storage device within a specified time range includes: grouping storage devices in a system including multiple storage devices according to grouping constraints; wherein the grouping constraints include: at least one of user name, disk type, and life cycle; selecting any storage device from any group as the target storage device; and obtaining a read and write log sequence of the target storage device within the specified time range.

[0107]

[0110] In practical applications, some scenarios may require a large amount of storage space, meaning that multiple storage devices may perform the same storage task. For example, in a primary-backup storage scenario, two sets of storage devices store the same content. This means that the storage system composed of the primary and backup storage devices meets the same set of constraints. The grouping constraints here include at least one of: user name, disk type, and lifecycle. In other words, multiple storage devices with the same user name, disk type, and lifecycle can be grouped together. This means that the read and write patterns of the multiple storage devices in the group may be identical or similar. When analyzing the performance of storage devices, it is not necessary to analyze all storage devices in the group; only a subset of them can be selectively analyzed. For example, if a group contains 10 storage devices and one is selected as the target storage device, only the read and write pattern recognition of that device needs to be performed, significantly reducing the computational complexity of pattern recognition compared to traditional methods. The greater the number of independent storage devices in the group, the greater the efficiency improvement. Therefore, the above solution can effectively reduce the workload of read and write pattern recognition and performance analysis.

[0108]

[0111] To facilitate understanding, the following will be illustrated using specific embodiments. Figure 6 is a schematic diagram of the storage device grouping structure provided in an embodiment of the present application. As can be seen from Figure 6, there are two groups of storage devices. The first group has a user name of A1, a disk type of A2, and a lifecycle of A3; the second group has a user name of B1, a disk type of B2, and a declaration period of B3. Assume that there are 10 disks in the system, 4 of which have a user name of A1, a disk type of A2, and a lifecycle of A3, while the remaining 6 have a user name of B1, a disk type of B2, and a declaration period of B3. Therefore, these 10 disks can be grouped according to the grouping constraints.

[0109]

[0112] After grouping, you can further select a disk from each of the first and second groups for read / write pattern analysis. To improve read / write pattern recognition efficiency, you can select a target storage device (i.e., a disk) from each of the first and second groups. Random selection can be used. Alternatively, if multiple disks in a group have a coding order, the order of the coding order can be used.

[0110]

[0113] After determining the target storage device, a continuous read and write log sequence is obtained from the storage space according to the specified time range.

[0111]

[0114] The above solution enables rapid read / write pattern recognition and storage device performance analysis for systems containing numerous storage devices, improving the efficiency of storage device performance analysis. Furthermore, using the twin network model for read / write pattern recognition offers the following advantages.

[0115] Learning a Similarity Metric: The Siamese Network aims to learn a similarity metric rather than directly performing classification. This allows the Siamese Network to better capture the similarities and differences between samples, rather than focusing solely on the categories to which they belong. This ability to learn a similarity metric is crucial in 10-pattern recognition, as 10-patterns are often diverse and sometimes difficult to clearly define. The Siamese Network can better adapt to this complexity by learning a similarity metric between samples.

[0112]

[0116] Training with a small number of samples: Siamese networks can be effectively trained with a small number of samples. Because they focus on the differences between pairs of samples rather than classifying individual samples, they do not require a large amount of labeled data for training. This is particularly beneficial for 10-pattern recognition, as obtaining large amounts of labeled 10-pattern data can be difficult and expensive. Siamese networks can learn effective similarity metrics using a small amount of labeled data, thus achieving low data costs.

[0113]

[0117] Robustness and Generalization: Siamese networks exhibit good robustness and generalization when dealing with unseen samples. By learning a similarity metric between samples, Siamese networks quantify similarity as a distance metric and learn the similarity between samples through optimization during training. This enables Siamese networks to better adapt to diverse data distributions and pattern variations, and to handle complex pattern recognition tasks.

[0114]

[0118] Automatic Feature Extraction: The Siamese Network uses a shared convolutional neural network (CNN) for feature extraction, eliminating the need for manual feature extraction and selection. The neural network automatically learns the most discriminative features from raw data without relying on prior knowledge from domain experts. This enables the Siamese Network to better capture features relevant to the 10 patterns, improving pattern recognition accuracy.

[0115]

[0119] In summary, the Siamese Network offers advantages over other machine learning models for calculating similarity when recognizing 10-patterns, including learning similarity metrics, training with a small number of samples, robustness and generalization capabilities, and automatic feature extraction. These advantages make the Siamese Network a powerful tool for 10-pattern recognition tasks.

[0116]

[0120] Based on the same example, an embodiment of the present application further provides a model training method. FIG7 is a flow chart of a model training method provided in an embodiment of the present application. As shown in FIG7 , the method includes the following steps 701 to 703.

[0117]

[0121] 701: Construct training samples using pattern vectors corresponding to a plurality of reference reading and writing patterns.

[0118]

[0122] 702: Input the training samples into the to-be-trained models of at least two sub-networks respectively.

[0119]

[0123] 703: When the inter-vector distance between the output vectors of multiple sub-networks that input the same training sample is less than the distance threshold, it means that the training obtains a twin network model.

[0120]

[0124] The Siamese Network model is a special neural network structure that is primarily used to learn similarity metrics between input data. The training process is as follows.

[0121]

[0125] Assume that a Siamese Network has been built that contains two sub-networks with the same structure and shared weights. These sub-networks are usually convolutional neural networks (CNNs) or recurrent neural networks (RNNs). These sub-networks receive different input data and generate corresponding feature vectors.

[0122]

[0126] Generating training samples: Generating training samples is a key step in Siamese Network training. Training samples must be presented in pairs, such as a pair of consecutive historical read / write log sequences. For each training batch, multiple pairs of samples are randomly selected. These pairs can be positive (similar or similar) or negative (different or dissimilar). Maintaining a balanced number of positive and negative samples is crucial for training stability.

[0123]

[0127] Training samples are input into the Siamese Network, and corresponding feature vectors are generated by the subnetworks. The distance between the two feature vectors is then calculated (such as Euclidean distance or cosine similarity). This distance between the feature vectors serves as the loss function. The loss function should be designed so that when the two input samples are similar, the smaller the distance, the smaller the loss; when the two input samples are dissimilar, the larger the distance, the larger the loss. Common loss functions include contrastive loss and triplet loss. The gradient of the loss function with respect to the network parameters is calculated using the backpropagation algorithm, and the parameters are updated using optimization methods such as gradient descent. During the update process, since the subnetworks share weights, their parameters are updated simultaneously. This step is repeated until the preset number of training rounds is reached or other stopping conditions are met. During training, hyperparameters and optimization strategies can be adjusted based on the performance on the validation set.

[0124]

[0128] For Siamese Network models, after N rounds of training, the model's performance can be evaluated by calculating the loss function. During training, the model continuously adjusts parameters to minimize the loss function, minimizing the distance between the feature vectors of similar pairs of samples and maximizing the distance between the feature vectors of dissimilar pairs of samples.

[0125]

[0129] As the loss function is minimized, the model gradually learns effective feature representations, bringing the feature vectors of similar pairs closer together and those of dissimilar pairs farther apart. After training is complete, the trained model can be used to calculate the similarity between two input samples.

[0126]

[0130] A common method for calculating similarity is cosine similarity. Cosine similarity measures the angle between two vectors, with values ​​ranging from -1 to 1. The closer the cosine similarity is to 1, the more similar the two vectors are; the closer it is to -1, the less similar they are. By applying cosine similarity to the feature vectors output by the model, we can obtain the similarity value between the two input samples as the output.

[0127]

[0131] Therefore, the output of a Siamese Network model can be a similarity value between two input samples, where a higher similarity value indicates that the two samples are more similar, and a lower similarity value indicates that the two samples are less similar. This output can be used for tasks such as metric learning and similarity comparison.

[0128]

[0132] Based on the same idea, an embodiment of the present application also provides a read-write pattern recognition device. FIG8a is a schematic diagram of a read-write pattern recognition device provided in an embodiment of the present application. FIG8a also shows that the device includes an acquisition module 801 and a calculation module 803.

[0129]

[0133] The acquisition module 801 is used to obtain the read and write log sequence of the target storage device within a specified time range.

[0130]

[0134] The calculation module 803 is used to input the low-dimensional matrix corresponding to the read-write log sequence and the multiple reference read-write patterns contained in the read-write pattern library into the twin network model, calculate the similarity between the read-write pattern in the specified time range and the multiple reference read-write patterns, and obtain multiple similarity values.

[0131]

[0135] Optionally, the device further includes a selection module 802, configured to select a plurality of reference reading and writing patterns from a preset reading and writing pattern library.

[0132]

[0136] The selection module 802 is configured to select one of the reference read / write patterns as the target read / write pattern of the target storage device within the specified time range according to a comparison result of the multiple similarity values.

[0133]

[0137] Optionally, the calculation module 803 is configured to convert the read and write log sequence into a data set including a time element, a space element, and an operation type; and perform dimensionality reduction processing based on the data set to obtain a low-dimensional matrix.

[0138] Optionally, the time elements include: an access start timestamp and an access end timestamp; the space elements include: an access address offset and an access address size; a calculation module 803 is configured to establish a timing matrix based on the operation type, the access start timestamp, the access end timestamp, the access address offset, and the access address size within the specified time range; perform data statistical processing based on the timing matrix to generate a vector matrix containing a timing sequence; and perform timing feature extraction on the vector matrix to obtain the low-dimensional matrix that retains the timing relationship.

[0134]

[0139] The calculation module 803 is configured to perform data statistics based on the time series matrix to obtain the time series, where the time series includes: a read and write number sequence, a byte number sequence, a sequential access read and write ratio sequence, and a unit time read and write ratio sequence; bucket the time series to extract a feature vector corresponding to each bucket; and vertically concatenate the feature vectors based on the time sequence of the specified time range to obtain the vector matrix.

[0135]

[0140] Optionally, the apparatus further includes a construction module 804, configured to collect historical read / write log sequences of multiple storage devices including the target storage device; perform read / write pattern classification processing on the historical read / write log sequences, and mark the classification results as reference read / write patterns; establish a read / write pattern cluster consisting of the read / write log sequences based on the reference read / write patterns; and establish a read / write pattern library containing pattern vectors based on the multiple read / write pattern clusters.

[0136]

[0141] A calculation module 803 is configured to input the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork; the twin network model includes the first subnetwork and the second subnetwork; output a first output vector corresponding to the read / write log sequence through the first subnetwork; obtain second output vectors corresponding to a plurality of pre-stored reference read / write patterns; and perform similarity calculation on the first output vector and the second output vector to obtain the similarity value.

[0137]

[0142] An acquisition module 801 is configured to input pattern vectors corresponding to the selected multiple reference read / write patterns into the second sub-network, respectively; and store second output vectors corresponding to the multiple reference read / write patterns output by the second sub-network, respectively, so as to provide the pre-stored multiple second output vectors when the first sub-network outputs the first output vector.

[0138]

[0143] The calculation module 803 is configured to input the low-dimensional matrix obtained based on the read / write log sequence into a first sub-network to obtain a first output vector; input the pattern vectors corresponding to the selected multiple reference read / write patterns into a second sub-network to obtain a second output vector; and input the first output vector and the second output vector into a discriminator for similarity calculation to obtain the similarity value.

[0139]

[0144] An acquisition module 801 is configured to group storage devices in a system including multiple storage devices according to grouping constraints, wherein the grouping constraints include at least one of user name, disk type, and lifecycle; select any storage device from any group as the target storage device; and obtain a read and write log sequence for the target storage device within a specified time range.

[0140]

[0145] Optionally, the construction module 804 is further configured to, when comparing the multiple similarity values ​​with a preset similarity threshold, determine that the read-write pattern corresponding to the specified time range is a new read-write pattern if the multiple similarity values ​​are all less than the similarity threshold; and update the new read-write pattern into the read-write pattern library as a newly added reference read-write pattern.

[0141]

[0146] Based on the same idea, an embodiment of the present application also provides a model training device. FIG8b is a schematic diagram of a model training device provided in an embodiment of the present application. FIG8b also shows that the device includes a construction module 811, an input module 812, and an output module 813.

[0142]

[0147] The construction module 811 is used to construct training samples using pattern vectors corresponding to multiple reference reading and writing patterns.

[0148] The input module 812 is used to input the training samples into the to-be-trained models of the at least two sub-networks respectively.

[0143]

[0149] The output module 813 is configured to indicate that a twin network model has been obtained through training when the inter-vector distance between the output vectors of multiple sub-networks that input the same training sample is less than a distance threshold.

[0144]

[0150] One embodiment of the present application further provides an electronic device. The electronic device is a master node electronic device in a computing unit. FIG9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a memory 901, a processor 902, and a communication component 903.

[0145]

[0151] The memory 901 is used to store programs; the processor 902 is coupled to the memory and is used to execute the program stored in the memory, so as to: obtain a read and write log sequence of a target storage device within a specified time range; input a low-dimensional matrix corresponding to the read and write log sequence and a plurality of reference read and write patterns contained in a read and write pattern library into a twin network model respectively, calculate the similarity between the read and write pattern of the specified time range and the plurality of reference read and write patterns, and obtain a plurality of similarity values; and select one from the plurality of reference read and write patterns as a target read and write pattern of the target storage device within the specified time range according to a comparison result of the plurality of similarity values.

[0146]

[0152] The processor 902 is further configured to convert the read and write log sequence into a data set including a time element, a space element, and an operation type; and perform dimensionality reduction processing based on the data set to obtain a low-dimensional matrix.

[0147]

[0153] The time elements include: an access start timestamp and an access end timestamp; the space elements include: an access address offset and an access address size; the processor 902 is further configured to establish a timing matrix based on the operation type, the access start timestamp, the access end timestamp, the access address offset, and the access address size within the specified time range; and perform data statistical processing based on the timing matrix to generate a vector matrix containing a timing sequence.

[0148]

[0154] Performing time series feature extraction on the vector matrix to obtain the low-dimensional matrix retaining the time series relationship.

[0149]

[0155] Processor 902 is further configured to perform data statistics based on the timing matrix to obtain the time series, where the time series includes: a read and write number sequence, a byte number sequence, a sequential access read and write ratio sequence, and a unit time read and write ratio sequence; bucket the time series to extract feature vectors corresponding to each bucket; and vertically splice the feature vectors based on the time sequence of the specified time range to obtain the vector matrix.

[0150]

[0156] Processor 902 is further configured to collect historical read and write log sequences of multiple storage devices including the target storage device; perform read and write pattern classification processing on the historical read and write log sequences, and mark the classification results as reference read and write patterns; establish a read and write pattern cluster consisting of the read and write log sequences based on the reference read and write patterns; and establish a read and write pattern library containing pattern vectors based on multiple read and write pattern clusters.

[0151]

[0157] Processor 902 is further configured to input the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork; include the first subnetwork and the second subnetwork in the twin network model; output a first output vector corresponding to the read / write log sequence through the first subnetwork; obtain second output vectors corresponding to a plurality of pre-stored reference read / write patterns; and perform similarity calculation on the first output vector and the second output vector to obtain the similarity value.

[0152]

[0158] The processor 902 is further configured to input the pattern vectors corresponding to the selected multiple reference read / write patterns into the second sub-network, respectively; and save the second output vectors corresponding to the multiple reference read / write patterns output by the second sub-network, respectively, so as to provide the pre-stored multiple second output vectors when the first sub-network outputs the first output vector.

[0153]

[0159] Processor 902 is further configured to input the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork to obtain a first output vector; input the pattern vectors corresponding to the selected multiple reference read / write patterns into a second subnetwork to obtain a second output vector; and input the first output vector and the second output vector into a discriminator for similarity calculation to obtain the similarity value.

[0154]

[0160] The processor 902 is further configured to group storage devices in a system including multiple storage devices according to grouping constraints, wherein the grouping constraints include at least one of a user name, a disk type, and a life cycle; select any one storage device from any group as the target storage device; and obtain a read and write log sequence for the target storage device within a specified time range.

[0155]

[0161] The processor 902 is also used to compare multiple similarity values ​​with a preset similarity threshold. If the multiple similarity values ​​are all greater than the similarity threshold, the read-write mode corresponding to the specified time range is determined to be a new read-write mode; and the new read-write mode is updated as a newly added reference read-write mode to the read-write mode library.

[0156]

[0162] During model training, the processor 902 is further configured to construct training samples using pattern vectors corresponding to a plurality of reference read and write patterns; input the training samples into the to-be-trained model comprising at least two sub-networks; and when the inter-vector distance between the output vectors of the plurality of sub-networks inputting the same training sample is less than a distance threshold, it indicates that a twin network model has been obtained through training.

[0157]

[0163] The memory 901 may be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM).

[0158] (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory

[0159] (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Magnetic or Optical Disk.

[0160]

[0164] Furthermore, the processor 902 in this embodiment may specifically be a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0161]

[0165] When executing the program in the memory, the processor 902 may implement other functions in addition to the above functions. For details, please refer to the description of the previous embodiments. Furthermore, as shown in FIG9 , the electronic device also includes other components such as a power supply component 904.

[0162]

[0166] An embodiment of the present application further provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the corresponding embodiment of FIG. 2 or FIG. 7 .

[0163]

[0167] Based on the above embodiment, when determining the operating status of a storage device, the read and write log sequence of the target storage device within a specified time range is obtained. Generally, the log sequence is obtained over a continuous period of time, which can fully reflect the temporal correlation between each continuous data point in the data set. Because the target storage device has a variety of read and write modes, multiple reference read and write modes are preset in the read and write mode library. Furthermore, the twin network model is used to determine which reference read and write mode the target storage device's read and write mode within the specified time range is most similar to. Specifically, the read and write log sequence and the reference read and write modes are respectively input into the twin network model. The twin network model determines the similarity between the read and write mode within the specified time range and the multiple reference read and write modes, and calculates a similarity value. Furthermore, the comparison results of the multiple similarity values ​​are compared. Based on the comparison results, the target object mode corresponding to the target storage device within the current specified time range is determined. This approach eliminates the need to directly calculate or determine read / write patterns based solely on a dataset. Instead, it uses the twin network model to determine the similarity between the target storage device's read / write patterns and a reference read / write pattern within a specified timeframe. This approach focuses on the differences between the twin network model's inputs, resulting in better generalization and more accurate read / write pattern determination.

[0164]

[0168] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0165]

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the solutions in this embodiment. Persons of ordinary skill in the art can understand and implement the solutions without inventive effort.

[0166]

[0170] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or alternatively, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or portions thereof.

[0167]

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

Claims 1. A method for recognizing a read / write pattern, comprising: Obtain a read and write log sequence of a target storage device within a specified time range; input a low-dimensional matrix corresponding to the read and write log sequence and a plurality of reference read and write patterns contained in a read and write pattern library into a twin network model respectively, calculate the similarity between the read and write pattern of the specified time range and the plurality of reference read and write patterns, and obtain a plurality of similarity values; based on a size comparison result of the plurality of similarity values, select one from the plurality of reference read and write patterns as the target read and write pattern of the target storage device within the specified time range.

2. The method according to claim 1, wherein: After obtaining the read and write log sequence of the target storage device within a specified time range, the method further includes: converting the read and write log sequence into a data set including a time element, a space element, and an operation type; and performing dimensionality reduction processing on the data set to obtain a low-dimensional matrix.

3. The method according to claim 2, wherein: The time elements include: an access start timestamp and an access end timestamp; the spatial elements include: an access address offset and an access address size; the dimensionality reduction processing based on the data set to obtain a low-dimensional matrix includes: establishing a timing matrix based on the operation type, access start timestamp, access end timestamp, access address offset and access address size within the specified time range; performing data statistical processing based on the timing matrix to generate a vector matrix containing a timing sequence; performing timing feature extraction on the vector matrix to obtain the low-dimensional matrix that retains the timing relationship.

4. The method according to claim 3, wherein: The performing of data statistical processing based on the time series matrix to generate a vector matrix containing a time series sequence includes: performing data statistics based on the time series matrix to obtain the time series, the time series including: a read and write number sequence, a byte number sequence, a sequential access read and write ratio sequence, and a unit time read and write ratio sequence; performing bucketing processing on the time series to extract a feature vector corresponding to each bucket; and performing vertical splicing processing on the feature vectors based on the time sequence of the specified time range to obtain the vector matrix.

5. The method according to claim 1, wherein: The method for constructing the read-write pattern library includes: collecting historical read-write log sequences of multiple storage devices including the target storage device; classifying the read-write patterns of the historical read-write log sequences and marking the classification results as reference read-write patterns; establishing read-write pattern clusters consisting of the read-write log sequences based on the reference read-write patterns; and establishing a read-write pattern library containing pattern vectors based on multiple read-write pattern clusters.

6. The method according to any one of claims 2 to 4, wherein: Inputting the read / write log sequence and the plurality of reference read / write patterns into a twin network model, calculating the similarity between the read / write log sequence and the plurality of reference read / write patterns, and obtaining a plurality of similarity values, comprises: inputting the low-dimensional matrix obtained based on the read / write log sequence into a first subnetwork; including the first subnetwork and the second subnetwork in the twin network model; outputting a first output vector corresponding to the read / write log sequence through the first subnetwork; obtaining pre-stored second output vectors corresponding to the plurality of reference read / write patterns; Perform similarity calculation on the first output vector and the second output vector to obtain the similarity value.

7. The method according to claim 6, wherein: The obtaining of the second output vectors corresponding to the plurality of pre-stored reference read-write patterns includes: inputting the pattern vectors corresponding to the plurality of selected reference read-write patterns into the second sub-network; and saving the second output vectors corresponding to the plurality of reference read-write patterns output by the second sub-network, so as to provide the plurality of pre-stored second output vectors when the first sub-network outputs the first output vector.

8. The method according to any one of claims 2 to 4, wherein: The step of inputting the read-write log sequence and the multiple reference read-write patterns into the twin network model respectively, calculating the similarity between the read-write log sequence and the multiple reference read-write patterns, and obtaining multiple similarity values ​​includes: inputting the low-dimensional matrix obtained based on the read-write log sequence into the first sub-network to obtain a first output vector; inputting the pattern vectors corresponding to the selected multiple reference read-write patterns into the second sub-network to obtain a second output vector; and inputting the first output vector and the second output vector into the discriminator for similarity calculation to obtain the similarity value.

9. The method according to claim 1, wherein: Obtaining a read and write log sequence of a target storage device within a specified time range includes: grouping storage devices in a system including multiple storage devices according to grouping constraints; wherein the grouping constraints include at least one of user name, disk type, and life cycle; selecting any storage device from any group as the target storage device; and obtaining a read and write log sequence of the target storage device within the specified time range.

10. The method according to claim 1, further comprising: When comparing multiple similarity values ​​with a preset similarity threshold, if the multiple similarity values ​​are all less than the similarity threshold, the read-write mode corresponding to the specified time range is determined to be a new read-write mode; the new read-write mode is updated as a new reference read-write mode to the read-write mode library.

11. A model training method, comprising: Construct training samples using pattern vectors corresponding to multiple reference reading and writing patterns; The training samples are respectively input into the to-be-trained model comprising at least two sub-networks; when the inter-vector distance between the output vectors of the multiple sub-networks inputting the same training samples is less than a distance threshold, it indicates that a twin network model is obtained through training.

12. A read / write pattern recognition device, comprising: An acquisition module is used to obtain the read and write log sequence of the target storage device within a specified time range; a calculation module, configured to input the low-dimensional matrix corresponding to the read / write log sequence and a plurality of reference read / write patterns contained in a read / write pattern library into a twin network model, calculate similarities between the read / write pattern in the specified time range and the plurality of reference read / write patterns, and obtain a plurality of similarity values; The selection module is configured to select one from the plurality of reference read / write patterns as the target read / write pattern of the target storage device within the specified time range according to a comparison result of the plurality of similarity values.

13. An electronic device comprising a memory and a processor; wherein: The memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory, so as to implement the method according to any one of claims 1 to 10, or to implement the method according to claim 11. method.

14. A non-transitory machine-readable storage medium storing executable code, wherein: When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 10, or is used to implement the method according to claim 11.

15. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented, or the steps of the method according to claim 11 are implemented.

Images