Method, apparatus and computer program product for anomaly detection of storage system

Abstract

The invention relates to a method, a device and a computer program product for anomaly detection of a storage system. The method includes obtaining a plurality of attribute values for a plurality of volume data block entries. The method further includes clustering a first portion of the plurality of volume data block entries into a congeneric pool based on the plurality of attribute values of the plurality of volume data block entries. In addition, the method also includes determining a second portion of the plurality of volume data block entries that is not clustered to the similar pool as an abnormal entry. By means of the mode, the storage system can automatically screen out the volume data block entries possibly having exceptions, so that the fault diagnosis time can be saved, the system fault solving efficiency is improved, and the service quality is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data storage, and more specifically to methods, apparatus and computer program products for anomaly detection in storage systems. Background Technology

[0002] Various anomalies can frequently occur in storage systems, including hardware failures, software errors, and data corruption. For example, file system corruption can render data unreadable or unwriteable, software or firmware defects can affect the normal operation of storage devices, and software errors or hardware failures can cause data stored on the disk to become inconsistent or corrupted.

[0003] System logs typically record information associated with hardware failures and software errors, making them an important resource for diagnosing storage system anomalies. When a storage system malfunctions, system administrators can access log records before or after the anomaly occurred and then look for error or warning codes. In some cases, if error or warning codes cannot directly pinpoint the location of the anomaly, engineers with specialized knowledge are needed to further analyze the detailed log content to locate the problem. Summary of the Invention

[0004] In a first aspect of the embodiments of this disclosure, a method for anomaly detection in a storage system is provided. The method includes obtaining multiple attribute values ​​of multiple volume data block entries. The method further includes clustering a first portion of the multiple volume data block entries into a similar pool based on the multiple attribute values. Furthermore, the method includes identifying a second portion of the multiple volume data block entries that are not clustered into the similar pool as anomaly entries.

[0005] In a second aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a memory coupled to at least one processor and having instructions stored thereon, the instructions causing the electronic device to perform an action when executed by the at least one processor. The action includes acquiring multiple attribute values ​​of multiple volume data block entries. The action further includes clustering a first portion of the multiple volume data block entries into a similar pool based on the multiple attribute values ​​of the multiple volume data block entries. Furthermore, the action includes identifying a second portion of the multiple volume data block entries that are not clustered into the similar pool as anomalous entries.

[0006] In a third aspect of the embodiments of this disclosure, a computer program product is provided, tangibly stored on a non-volatile computer-readable medium and including machine-executable instructions that, when executed, cause a machine to implement a method for anomaly detection in a storage system. The method includes obtaining multiple attribute values ​​of multiple volume data block entries. The method further includes clustering a first portion of the multiple volume data block entries into a similar pool based on the multiple attribute values. Furthermore, the method includes identifying a second portion of the multiple volume data block entries that are not clustered into the similar pool as anomaly entries.

[0007] In a fourth aspect of embodiments of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a method for anomaly detection in a storage system. The method includes acquiring multiple attribute values ​​of multiple volume data block entries. The method further includes clustering a first portion of the multiple volume data block entries into a similar pool based on the multiple attribute values. Furthermore, the method includes identifying a second portion of the multiple volume data block entries that are not clustered into the same pool as anomaly entries.

[0008] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example storage system in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method for anomaly detection in a storage system according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of an example architecture for anomaly detection according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating an example of real-time clustering of volume data block entries to determine anomalous entries according to some embodiments of the present disclosure is shown;

[0014] Figure 5A schematic diagram illustrates an example process for determining a pool of similar clusters in the case of real-time clustering, according to some embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram illustrates an example process for determining outlier entries after determining a pool of similar clusters in the case of real-time clustering, according to some embodiments of the present disclosure;

[0016] Figure 7 A schematic diagram illustrates an example process for grouping volume data block entries in the case of background clustering, according to some embodiments of the present disclosure;

[0017] Figure 8 The illustration shows an example process, according to some embodiments of the present disclosure, of adding a portion of volume block entries from a previous set of volume block entries to another set of volume block entries in the case of background clustering to determine anomalous entries; and

[0018] Figure 9 A block diagram of a device that can implement several embodiments of the present disclosure is shown. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit implementations may also be included below.

[0021] As mentioned above, storage systems, whether in a laboratory or at a customer's site, are prone to various system anomalies every day. If these anomalies are not detected and addressed promptly, they can lead to significant losses. For example, data loss, system downtime, or sharp performance degradation can severely impact an enterprise's operational efficiency and service quality. Detecting system anomalies and finding their root causes is a challenge for system administrators and engineers, typically involving complex technical analysis and tedious debugging. In some cases, engineers need to add more debugging trace code to the system or reproduce the problem in a specific test environment to more accurately pinpoint the issue.

[0022] In related technologies, storage systems can record a large amount of operation logs, which record every operation and state change of the system in detail. When an anomaly occurs, engineers need to carefully analyze these log files to diagnose the root cause of the problem. This analysis process typically involves searching for error codes, warning messages, records of failed operations, and any non-standard system responses. For example, if a file system suddenly becomes unresponsive to user queries, engineers may need to examine the file system's log files for any abnormal records related to input / output operations. Through these records, engineers can understand whether the problem is caused by resource locking, data corruption, or a software defect. However, locating the anomaly from a large amount of log records not only requires engineers to have a high level of technical skill, but the process is also very time-consuming.

[0023] Therefore, embodiments of this disclosure provide a scheme for anomaly detection in a storage system. In this scheme, volume data block entries associated with physical data blocks in the storage system, along with multiple attribute values ​​for each volume data block entry, can be obtained. Then, the scheme can perform a clustering operation on multiple volume data block entries based on these attribute values. Most of these volume data block entries can be clustered into the same cluster, also referred to herein as a homogeneous pool. Entries not clustered into the homogeneous pool can be identified as anomalous entries. Engineers can analyze these anomalous entries to determine the location of problems or failures.

[0024] In this way, the storage system can automatically filter out volume data block entries that may be abnormal. System administrators or engineers only need to analyze a small number of these filtered abnormal entries to determine whether a fault has occurred near the corresponding physical data block. This saves time spent analyzing logs and debugging the system, improves the efficiency of anomaly location, and thus improves the efficiency of troubleshooting and service quality.

[0025] Figure 1A schematic diagram of an example storage system 100 in which several embodiments of the present disclosure may be implemented is shown. The storage system may include multiple independent redundant array of hard disks (RAID) stripes, each RAID stripe corresponding to a physical large block. The physical large block can be used to actually store data, and each physical large block may include multiple physical data blocks. The storage system may also include multiple virtual volume large blocks, each volume large block being associated with multiple physical large blocks, and each volume large block may include multiple volume data block entries, where each volume data block entry may correspond to a physical data block.

[0026] like Figure 1 As shown, the storage system 100 includes RAID stripes 102 and 104. RAID stripe 102 corresponds to physical large data block 110, and RAID stripe 104 corresponds to physical large data block 120. Physical large data block 110 includes physical data blocks 112, 114, and 116, and physical large data block 120 includes data blocks 122, 124, and 126. The storage system 100 also includes a volume data block 130, which includes volume data block entries 131, 132, 133, 134, 135, and 136. Volume data block entry 131 corresponds to physical data block 112, volume data block entry 132 corresponds to physical data block 114, volume data block entry 133 corresponds to physical data block 116, volume data block entry 134 corresponds to physical data block 122, volume data block entry 135 corresponds to physical data block 124, and volume data block entry 136 corresponds to physical data block 126.

[0027] In storage system 100, a volume data block entry can have multiple attributes. Taking volume data block entry 131 as an example, it has attributes 140-1, 140-2, ..., 140-N (collectively referred to as attribute 140). For example, attribute 140 may include, but is not limited to, the location of the corresponding physical data block within the physical large data block, the offset of the corresponding physical data block within the physical large data block, the size of the compressed physical data block, the data checksum, the reference count of the physical data block, the type of the volume data block entry, and the subtype of the volume data block entry. Multiple volume data block entries can have the same attribute, but for the same attribute, multiple volume data block entries can have different values.

[0028] In the solutions provided by the embodiments of this disclosure, the storage system 100 can cluster volume data block entries (e.g., including but not limited to volume data block entries 131, 132, 133, 134, 135, and 136) based on attribute 140. Since the attribute values ​​of most normal volume data block entries have relatively minor differences, while the attribute values ​​of abnormal volume data block entries may vary significantly compared to normal volume data block entries, after clustering these volume data block entries based on attribute values, most volume data block entries will be clustered into the same cluster (i.e., pool 142), and volume data block entries not clustered into pool 142 can be identified as abnormal entries. The storage system 100 can provide these abnormal entries to the system administrator to provide clues and hints for locating system faults. For example, if volume data block entry 131 is not clustered into the same pool 142, the storage system 100 can indicate that volume data block entry 131 is an abnormal entry, and that the corresponding physical data block 112 and the physical data blocks near physical data block 112 may be faulty.

[0029] In this way, storage system 100 can automatically filter out volume data block entries that may be abnormal. System administrators only need to analyze a small number of the filtered abnormal entries to determine whether a fault has occurred near the corresponding physical data block. This saves time in fault diagnosis, thereby improving the efficiency of resolving system faults and enhancing service quality.

[0030] Figure 2 A flowchart of a method 200 for anomaly detection in a storage system according to some embodiments of the present disclosure is shown. Method 200 may, for example, be derived from... Figure 1 The storage system 100 or the control unit of the storage system 100 performs the operation. For example... Figure 2 As shown in box 202, method 200 can obtain multiple attribute values ​​for multiple volume data block entries. For example, in... Figure 1In the storage system 100 shown, volume data block entries 131, 132, 133, 134, 135, and 136 correspond to physical data blocks 112, 114, 116, 122, 124, and 126, respectively. Each volume data block entry includes multiple attributes 140, and the same attribute 140 of multiple volume data block entries can have different values. For example, suppose attribute 140-1 represents the data block entry type, which is an enumerated data structure, and its value can be 0, 1, 2, etc. In this case, volume data block entries 131, 132, 133, 134, 135, and 136 all have attribute 140-1 (i.e., data block entry type), but the value of attribute 140-1 for volume data block entry 131 can be, for example, 0, the value of attribute 140-1 for volume data block entry 132 can be, for example, 1, and so on. Storage system 100 can retrieve the values ​​of multiple attributes 140 for volume data block entries 131, 132, 133, 134, 135, and 136. In some embodiments, a vector can be used to represent the multiple attribute values ​​of a volume data block entry.

[0031] In box 204, method 200 can cluster a first portion of multiple volume block entries into a homogeneous pool based on multiple attribute values ​​of multiple volume block entries. Since the differences between attribute values ​​of most normal volume block entries are relatively gradual, while the attribute values ​​of abnormal volume block entries may vary significantly compared to normal volume block entries, after clustering these volume block entries based on attribute values, most volume block entries will be clustered into the same cluster (also called the homogeneous pool), and these volume block entries are referred to as the first portion. For example, in... Figure 1 In the storage system 100 shown, these volume data block entries can be clustered based on the values ​​of multiple attributes 140 of volume data block entries 131, 132, 133, 134, 135, and 136. For example... Figure 1 As shown, after clustering, volume data block entries 132, 133, 134, 135, and 136 are clustered into homogeneous pool 142, indicating that the attribute values ​​of these volume data block entries are similar. In other words, the distance between the attribute value vectors of multiple volume data block entries clustered into homogeneous pool 142 is minimized. In some embodiments, homogeneous pools are determined based on computer algorithms during the clustering process. In some embodiments, homogeneous pools are determined based on the size of the resulting clusters after clustering multiple volume data block entries. For example, after clustering is completed, the cluster comprising the majority of volume data block entries can be identified as a homogeneous pool.

[0032] In box 206, method 200 can identify a second portion of multiple volume block entries that have not been clustered into the same pool as outliers. After clustering, if a subset of volume block entries (also referred to as the second portion) are not clustered into the same pool, their attribute values ​​differ significantly from those of the volume block entries in the same pool; in other words, their attribute value vectors are significantly different from the attribute value vectors of the volume block entries in the same pool. Therefore, these volume block entries that have not been clustered into the same pool can be identified as outliers. For example, in... Figure 1 In the storage system 100 shown, if volume data block entry 131 is not clustered into the same pool 142, it can be identified as an anomalous entry, indicating that the corresponding physical data block 112 and the physical data blocks near physical data block 112 may be faulty. In some embodiments, the storage system 100 can provide volume data block entry 131 to the system administrator to provide the system administrator with hints and clues for locating system faults.

[0033] In this way, the storage system can automatically filter out volume data block entries that may be abnormal. System administrators only need to analyze a small number of the filtered abnormal entries to determine whether a fault has occurred near the corresponding physical data block. This saves time in fault diagnosis, thereby improving the efficiency of resolving system faults and enhancing service quality.

[0034] Figure 3 A schematic diagram of an example architecture 300 for anomaly detection according to some embodiments of the present disclosure is shown. Figure 3 As shown, after the storage system performs an input / output (I / O) operation 302, architecture 300 can determine multiple volume data block entries 304-1, 304-2, ..., 304-N (collectively referred to as volume data block entries 304) associated with the I / O operation 302. For example, architecture 300 can determine which physical data blocks the I / O operation 302 operates on, and then determine the volume data block entries 304 corresponding to these physical data blocks. It should be understood that volume data block entries 304 can come from the same large volume data block (e.g., such as...). Figure 1 The volume data block 130 shown can also come from multiple volume data blocks. Then, the architecture 300 can obtain the attribute values ​​of multiple volume data block entries 304.

[0035] In some cases, the differences between multiple values ​​of the same attribute are small, but different values ​​can indicate completely different states or patterns. For example, for Boolean attributes, the difference between 0 and 1 is small, but can indicate completely different states. For enumeration attributes, the differences between attribute values ​​such as 1, 2, 3, etc., are small, but can indicate completely different patterns. If these attribute values ​​are not specially processed, volume data block entries with different attribute values ​​are likely to be clustered together, leading to an increased false negative rate for outlier entries. Therefore, in some embodiments, the attribute mapping module 306 can specially process the values ​​of attributes with different data types to increase the differences between different attribute values, thereby improving the sensitivity of the clustering operation and reducing the false negative rate.

[0036] In some embodiments, a plurality of volume data block entries include two volume data block entries (also referred to as a first volume data block entry and a second volume data block entry) that have different values ​​for a certain attribute. In these embodiments, the attribute mapping module 306 can convert the value of the attribute (also referred to as a first value) of the first volume data block entry into a new value (also referred to as a third value) based on the type of the attribute. Furthermore, the attribute mapping module 306 can also convert the value of the attribute (also referred to as a second value) of the second volume data block entry into a new value (also referred to as a fourth value) based on the type of the attribute, wherein the difference between the two converted attribute values ​​(i.e., the third value and the fourth value) is greater than the difference between the two original attribute values ​​(i.e., the first value and the second value).

[0037] In some embodiments, for properties of enumeration type, the original property value can be transformed using the following equation (1):

[0038] y = 10 x (1)

[0039] Where x represents the original attribute value and y represents the transformed attribute value.

[0040] In some embodiments, for Boolean type attributes, the original attribute value can be converted using the following equation (2):

[0041] y = 100 x (2)

[0042] Where x represents the original attribute value and y represents the transformed attribute value.

[0043] In some embodiments, for numeric attributes, the original attribute value can be converted using the following equation (3):

[0044]

[0045] Where x represents the original attribute value, e represents the natural constant, and y represents the transformed attribute value.

[0046] In some embodiments, for attributes of types other than those described above, the original attribute values ​​may remain unchanged. For example, suppose volume data block entries 304-1 and 304-2 have an enumerated attribute "Type", an enumerated attribute "Entry Status", a numeric attribute "Virtual Reference Count", a numeric attribute "Leaf Reference Count", a Boolean attribute "Movement Status", and a numeric attribute "Number of Virtual Data Blocks".

[0047] In this example, assuming volume block entry 304-1 has a "Type" value of 2, an "Entry Status" value of 1, a "Virtual Reference Count" value of 0, a "Leaf Reference Count" value of 0, a "Movement Status" value of 0, and a "Number of Virtual Blocks" value of 0, then the values ​​of these attributes for volume block entry 304-1 can be represented by a vector as [2,1,0,0,0,0]. Similarly, assuming volume block entry 304-2 has a "Type" value of 1, an "Entry Status" value of 0, a "Virtual Reference Count" value of 1, a "Leaf Reference Count" value of 1, a "Movement Status" value of 1, and a "Number of Virtual Blocks" value of 1, then the values ​​of these attributes for volume block entry 304-2 can be represented by a vector as [1,0,1,1,1,1]. In this case, volume block entries 304-1 and 304-2 represent two completely different entries, but the distance between their attribute value vectors is small, making it possible for them to be clustered into the same cluster.

[0048] After transforming the attribute values ​​of volume data block entries 304-1 and 304-2 using equations (1) to (3) above, the attribute value of volume data block entry 304-1 can be transformed into [100,10,1000,1000,1,1000], and the attribute value of volume data block entry 304-2 can be transformed into [10,1,1,1,100,1]. In this way, the difference between attribute values ​​can be increased, thereby improving the sensitivity of clustering operations and reducing the false negative rate.

[0049] like Figure 3As shown, after the attribute mapping module 306 transforms the attribute values ​​of the volume data block entries 304, the real-time clustering module 308, the background clustering module 310, and the standard clustering module 312 can be used to cluster the volume data block entries 304. When the number of volume data block entries to be clustered is large, the efficiency of the clustering algorithm may decrease. The real-time clustering module 308 can cluster incremental entries in real time when volume data block entries are received, the background clustering module 310 can periodically group and cluster a large number of volume data block entries, and the standard clustering module 312 can cluster all volume data block entries when the number of volume data block entries is small. In this way, different clustering processes can be executed for different data volumes and timeliness requirements, thereby improving the efficiency of the clustering process and increasing the clustering accuracy.

[0050] Figure 4 A schematic diagram of example 400 is shown, illustrating real-time clustering of volume data block entries to identify anomalous entries according to some embodiments of the present disclosure. Figure 4 As shown, in Example 400, volume data block entries 404-1, 404-2, ..., 404-N (collectively referred to as volume data block entries 404) are included. Attribute mapping module 406 (e.g., as...) Figure 3 The attribute mapping module 306 in the illustrated architecture 300 can transform the attribute values ​​of volume data block entries 404. Then, the real-time clustering module 408 can construct and determine a homogeneous pool 410 in real-time during the receipt of volume data block entries 404. After the homogeneous pool 410 is determined, the real-time clustering module 408 can determine whether subsequent volume data block entries can be clustered into the homogeneous pool 410 based on their attribute values. If subsequent volume data block entries cannot be clustered into the homogeneous pool 410, the real-time clustering module 408 can place them into a heterogeneous pool 412, where volume data block entries can be identified as anomalous entries.

[0051] In some embodiments, during the real-time clustering process, one volume block entry from a plurality of volume block entries (also referred to as a first volume block entry) can be identified as an anchor entry, and candidate homogeneous pools can then be created based on the anchor entry. Homogeneous pools can then be determined based on the candidate homogeneous pools and a subset of the volume block entries from the plurality of volume block entries (also referred to as a third subset). After determining the homogeneous pools, subsequent entries from the plurality of volume block entries (also referred to as a fourth subset) can be clustered into the homogeneous pools based on multiple attribute values. In some embodiments, when determining homogeneous pools based on the candidate homogeneous pools and a subset of the volume block entries from the plurality of volume block entries, an original density (also referred to as a first density) of the candidate homogeneous pools can be determined, the original density indicating the differences between the plurality of volume block entries in the candidate homogeneous pools. A new density (also referred to as a second density) of the candidate homogeneous pools can then be determined if a new volume block entry (also referred to as a second volume block entry) is added to the candidate homogeneous pools. The density change of the candidate homogeneous pools can then be determined based on the original density and the new density, and homogeneous pools can be determined based on this density change. In some embodiments, if the density change is greater than a predetermined density change threshold, the new volume data block entry can be added to the candidate pool of similar entries. If the number of volume data block entries in the candidate pool of similar entries is greater than a predetermined number threshold, the candidate pool of similar entries can be determined as a pool of similar entries.

[0052] Figure 5 A schematic diagram of an example process 500 for determining homogeneous pools in the case of real-time clustering, according to some embodiments of the present disclosure, is shown. Figure 5 As shown, in process 500, the density ρ for volume data block entries can be defined using the following equation (4). vlb :

[0053]

[0054] Among them, D vlb This represents the Euclidean distance from the volume block entry to the anchor entry in the same pool. Euclidean distance is the "straight-line" distance between two points in Euclidean space. Both the attribute value vector of the anchor entry and the attribute value vector of the volume block entry can be represented as n-dimensional vectors with n attribute values. The Euclidean distance can be the square root of the sum of the squares of the differences in each dimension of the n-dimensional space, representing the shortest straight-line distance from the volume block entry to the anchor entry in that n-dimensional space. After the attribute values ​​are transformed by the attribute mapping module, the minimum non-zero Euclidean distance between two volume block entries is 1, meaning the maximum density of the volume block entry is 1. For volume block entries with the same attribute values, the density can be set to 10 to better distinguish them from cases where two volume block entries have different attribute values.

[0055] Furthermore, in process 500, the density ρ of similar pools can also be defined using the following equation (5). p :

[0056]

[0057] Where N represents the number of volume data block entries in the same pool.

[0058] Then, the density change ρ of the same pool after adding a new volume data block entry can be defined using the following equation (6). Δ :

[0059]

[0060] In process 500, the real-time clustering module (e.g., Figure 3 The real-time clustering module 308 can sequentially receive volume data block entries 506, 508, and 510. The real-time clustering module can determine the first received volume data block entry 506 as the anchor entry. When volume data block entry 508 is received, its distance D from volume data block entry 506 can be determined. vlb2 And based on D vlb2 To determine the density ρ of volume data block entry 508. vlb2 At this point, the candidate pool 502 includes volume data block entries 506 and 508, and the density ρ of volume data block entry 508 can be adjusted. vlb2 The density ρ of the current candidate pool 502 is determined. p .

[0061] In process 500, N can be used. as To represent the number of volume data block entries in candidate pool 502, and to use N an This represents the number of volume data block entries in heterogeneous pool 504. Additionally, T can be used... n To represent a predetermined number threshold, and to use T Δ This is used to represent the predetermined density change threshold.

[0062] When the real-time clustering module receives a new volume data block entry 510, it can determine its distance D to the anchor entry (i.e., volume data block entry 506). vlb3 And based on distance D vlb3 To determine the density ρ of volume data block entry 510. vlb3 Then, it can be based on density ρ vlb3 To calculate the density change ρ of candidate pool 502 if volume data block entry 510 is added to candidate pool 502. Δ If the density changes ρ ΔLess than the predetermined density change threshold T Δ Then, volume data block entry 510 can be added to candidate pool 502, and the number N of volume data block entries in candidate pool 502 can be updated. as If the density changes ρ Δ Greater than the density change threshold T Δ And the density ρ of candidate pool 502 p Greater than density ρ vlb3 Then, volume data block entry 510 can be added to heterogeneous pool 504, and the number N of volume data block entries in heterogeneous pool 504 can be updated. an .

[0063] If the number N of volume data block entries in candidate pool 502 is... as Greater than the predetermined threshold T n Then, candidate pool 502 can be determined as a pool of the same type (i.e., it can be considered that the pool of the same type is ready) for clustering of subsequent volume block entries. In some embodiments, if the number N of volume block entries in heterogeneous pool 504 is less than the number of pools of the same type before the pool of the same type is ready, an If the number of volume block entries exceeds another predetermined threshold, it can be assumed that the volume block entries in the current candidate pool 502 are dissimilar to most entries. Therefore, the candidate pool 502 and the heterogeneous pool 504 can be cleared, and new anchor entries can be determined. This process can then be repeated until the candidate pool 502 is determined as a pool of similar volume blocks. In this way, pools of similar volume blocks can be determined in real time while new volume block entries are being received, without waiting for a large number of volume block entries to be received before performing clustering operations. This improves the efficiency of determining pools of similar volume blocks.

[0064] In some embodiments, after determining the homogeneous pool, the density change of the homogeneous pool can be determined if subsequent volume block entries (also referred to as third volume block entries) from multiple volume block entries are added to the homogeneous pool. If the density change of the homogeneous pool is greater than a predetermined density change threshold, the volume block entry can be identified as an anomalous entry. In some embodiments, after determining the homogeneous pool, the number of subsequent volume block entries clustered into the homogeneous pool (also referred to as the first number) and the number of volume block entries not clustered into the homogeneous pool (also referred to as the second number) can be determined. If the ratio of the first number and the second number is less than a predetermined ratio threshold, a new homogeneous pool can be re-established by selecting a new anchor entry.

[0065] Figure 6 A schematic diagram of an example process 600 for determining outlier entries after determining a pool of similar clusters in the case of real-time clustering, according to some embodiments of the present disclosure, is shown. Figure 6As shown, in process 600, the homogeneous pool 602 is a prepared homogeneous pool (e.g., the homogeneous pool determined via process 500), which includes volume data block entries 606, 608, and 610, wherein volume data block entry 606 is an anchor entry. Upon receiving volume data block entry 612, the distance from volume data block entry 612 to the anchor entry (i.e., volume data block entry 606) and the density change of homogeneous pool 602 if volume data block entry 612 is added to homogeneous pool 602 can be determined. If the density change is greater than a predetermined density change threshold, volume data block entry 612 can be added to heterogeneous pool 604.

[0066] In some embodiments, the real-time clustering module can determine the number of volume block entries added to the heterogeneous pool 604 after the homogeneous pool 602 is determined. If this number exceeds a predetermined threshold, it can be considered that the volume block entries in the previously determined homogeneous pool 602 may be dissimilar to most volume block entries, and a new homogeneous pool can be re-established by selecting new anchor entries. In this way, erroneous homogeneous pools can be discarded in a timely manner and correct homogeneous pools can be re-established, thereby improving the accuracy of real-time clustering.

[0067] In some embodiments, entries for volume data blocks that should not be added to the same pool 602 can also be determined after the same pool 602 is determined. If, after the same pool 602 is determined, the ratio of the number of entries added to the same pool 602 to the number of entries added to the different pool 604 is less than a predetermined ratio threshold, it can be considered that the volume data block entries in the previously determined same pool 602 may be dissimilar to most volume data block entries, and a new same pool can be re-established by selecting new anchor entries. In this way, the accuracy of determining whether the same pool 602 is correctly or incorrectly established can be improved.

[0068] In some cases, to find missed anomalous entries, background clustering can be performed periodically on a large number of volume block entries over a past period. During background clustering, in some embodiments, the total number of multiple volume block entries can be determined. If the total number exceeds a predetermined total number threshold (also called a first predetermined total number threshold), the multiple volume block entries can be split into a first group of volume block entries and a second group of volume block entries. Anomalous entries in the first group of volume block entries and anomalous entries in the second group of volume block entries can then be detected in parallel. In some embodiments, when determining anomalous entries in the first group of volume block entries, the first group of volume block entries can be split into a third group of volume block entries and a fourth group of volume block entries. Anomalous entries in the third group of volume block entries and anomalous entries in the fourth group of volume block entries can then be detected serially.

[0069] Figure 7A schematic diagram of an example process 700 for grouping volume data block entries in the case of background clustering, according to some embodiments of the present disclosure, is shown. Figure 7 As shown, in process 700, background clustering needs to be performed on the set of volume data block entries 702 corresponding to 1GB physical data blocks. Assuming each physical data block is 4KB in size, the set of volume data block entries 702 will contain approximately 250,000 volume data block entries. Performing clustering on such a large number of entries would be very time-consuming. Therefore, if the size of the set of volume data block entries 702 exceeds a predetermined total number threshold, it can be split into subsets 704, 706, 708, and 710, and each subset can, for example, correspond to a 256MB physical data block. Then, clustering operations can be performed in parallel on subsets 704, 706, 708, and 710, which improves the execution efficiency of the clustering algorithm and saves time spent executing the clustering algorithm.

[0070] like Figure 7 As shown, process 700 can further split each of subsets 704, 706, 708, and 710. For example, process 700 can split subset 706 into subsets 712-1, 712-2, ..., 712-N (collectively referred to as subset 712), where each subset 712 may correspond to a 32MB physical data block, for example. Then, a clustering algorithm can be performed on each subset 712 separately. However, while performing clustering algorithms on these subsets 712 can improve the execution efficiency of the algorithm, it may reduce the accuracy and realism of the clustering. Therefore, process 700 can perform clustering algorithms serially on multiple subsets 712 and adjust the volume data block entries in subsequent subsets based on the clustering results of previous subsets, thereby realizing a dynamically adjustable clustering process.

[0071] like Figure 7 As shown, process 700 can first perform a clustering algorithm on subset 712-1 and determine the set of anomalous entries 714-1. If the set of anomalous entries 714-1 meets predetermined conditions, it can be considered that the accuracy of the set of anomalous entries 714-1 may be problematic. Then, process 700 can try to find the false positives or false negatives by adjusting the volume data block entries in the next subset 712-2. Then, process 700 can determine the set of anomalous entries 714-1, 714-2, ..., 714-N as the set of anomalous entries for subset 706, and determine the set of anomalous entries 704, 706, 708, and 710 as the set of anomalous entries for volume data block entry set 702.

[0072] In some embodiments, when serially detecting anomalous entries in the third group of volume block entries and anomalous entries in the fourth group of volume block entries, the number of anomalous entries in the third group of volume block entries can be determined. If the number of anomalous entries in the third group of volume block entries is less than a predetermined anomalous number threshold, a portion of the entries in the third group of volume block entries can be added to the fourth group of volume block entries. Then, anomalous entries in the updated fourth group of volume block entries can be detected.

[0073] Figure 8 A schematic diagram of an example process 800, according to some embodiments of the present disclosure, is shown, in the case of background clustering, to add a portion of a previous set of volume block entries to another set of volume block entries to determine anomalous entries. As shown in FIG800, process 800 includes subsets 812-1, 812-2, ..., 812-N of volume block entries (collectively referred to as subset 812, for example, could be...). Figure 7 The process 800 first performs a clustering algorithm on subset 812-1 to identify outliers. If the number of outliers in subset 812-1 is less than a predetermined outlier threshold (e.g., 1, 3, 5, etc.), to verify that entries in the same pool of subset 812-1 are not outliers relative to other subsets, the process 800 can add a portion 802 (e.g., 2MB) of subset 812-1 to the next subset 812-2. At this point, subset 812-2 includes volume data block entries corresponding to 34MB of physical data blocks, where 2MB comes from subset 812-1.

[0074] Then, process 800 can perform a clustering algorithm on subset 812-2. After clustering subset 812-2, if outlier entries are detected in portion 802, it indicates that subset 812-1 may contain a large number of outlier entries. At this point, the volume data block entries in subset 812-1 can be located by using the detected outlier entries in portion 802, thereby further confirming whether the entries previously identified as normal entries in subset 812-1 are actually outlier entries.

[0075] This approach reduces the amount of data involved in each clustering algorithm, thereby improving the execution efficiency of the clustering algorithm. Furthermore, it allows for the verification of clustering results, thus improving the accuracy of anomaly detection.

[0076] In some cases, when the total number of volume data block entries to be inspected is small (e.g., in the case of functional testing), standard clustering can be performed on these entries. When performing standard clustering, the density-based spatial clustering (DBSCAN) algorithm can be used to cluster the volume data block entries.

[0077] In standard clustering, one volume block entry can be considered a neighborhood of another volume block entry, and the maximum distance between these two entries can be controlled by the parameter `Epsilon`, which defines the radius of the neighborhood around each point. After transforming the attribute values ​​of the volume block entries, the minimum distance between two different volume block entries with different attribute meanings is 10, so `Epsilon` can be set to 10 for the first attempt. The number of samples (or total weight) in the neighborhood of the anchor entry considered as the core point can be controlled by the parameter `MIN_SAMPLES`, and can be set to twice the number of attributes of the volume block entries used for clustering. These two key parameters can be adjusted in different ways to obtain different clustering results.

[0078] Using the DBSCAN algorithm to perform standard clustering offers several advantages. For example, DBSCAN can identify clusters of arbitrary shapes, making it excellent for complex datasets and capable of discovering non-linearly distributed clusters. Furthermore, due to its density-based clustering approach, DBSCAN performs well on datasets with significant variations in data density, effectively distinguishing between high-density and low-density regions to form clusters. In addition, DBSCAN effectively identifies and handles noisy data and outliers (i.e., anomalies). This is because the algorithm only defines sufficiently high-density regions as clusters (i.e., pooled clusters), while low-density points are labeled as noise. In this way, standard clustering can achieve higher accuracy results within an acceptable timeframe.

[0079] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. For example, Figure 1 The control unit of the storage system 100 shown can be as follows: Figure 9 The example device 900 is shown. As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0080] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0082] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0083] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0085] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for anomaly detection in a storage system: Retrieve multiple attribute values ​​for multiple volume data block entries; Based on the multiple attribute values ​​of the multiple volume data block entries, the first portion of the multiple volume data block entries is clustered into a pool of the same type; as well as The second portion of the multiple volume data block entries that are not clustered into the same pool is identified as anomaly entries.

2. The method of claim 1, wherein the plurality of attribute values ​​of the plurality of volume data block entries include a first value of an attribute of a first volume data block entry and a second value of an attribute of a second volume data block entry, and clustering the first portion of the plurality of volume data block entries into the homogeneous pool based on the plurality of attribute values ​​of the plurality of volume data block entries includes: The first value is converted into a third value based on the type of the attribute. The second value is converted into a fourth value based on the type of the attribute, wherein the difference between the third value and the fourth value is greater than the difference between the first value and the second value; and Based on the third and fourth values, one or both of the first volume data block entries and the second volume data block entries are clustered into the same pool.

3. The method of claim 1, wherein clustering the first portion of the plurality of volume data block entries into the homogeneous pool based on the plurality of attribute values ​​of the plurality of volume data block entries comprises: The first volume data block entry among the plurality of volume data block entries is identified as the anchor entry; Create a candidate pool of similar entries based on the anchor entries; The same pool is determined based on the candidate similar pools and the third portion of the plurality of volume data block entries; as well as The fourth part is clustered into the same pool based on multiple attribute values ​​of the fourth part in the plurality of volume data block entries, wherein the first part includes at least a portion of the third part and the fourth part.

4. The method of claim 3, wherein determining the homogeneous pool based on the third portion of the candidate homogeneous pool and the plurality of volume data block entries comprises: Determine a first density for the candidate pool of similar volumes, the first density indicating the differences between multiple volume data block entries in the candidate pool of similar volumes; Determine the second density of the candidate pool if the second volume data block entry from the third part is added to the candidate pool; Based on the first density and the second density of the candidate pool, determine the density change of the candidate pool if the second volume data block entry is added to the candidate pool; and The pool of the same type is determined based on the density change of the candidate pool of the same type.

5. The method of claim 4, wherein determining the similar pool based on the density change of the candidate similar pool comprises: In response to the density change being greater than a predetermined density change threshold, the second volume data block entry is added to the candidate pool of the same type; as well as In response to the number of volume data block entries in the candidate pool being greater than a predetermined threshold, the candidate pool is determined as the pool.

6. The method of claim 3, wherein determining the second portion of the plurality of volume data block entries that are not clustered into the same pool as anomalous entries comprises: After determining the homogeneous pool, determine the density change of the homogeneous pool if a third volume data block entry from the plurality of volume data block entries is added to the homogeneous pool; and In response to the density change in the same pool exceeding a predetermined density change threshold, the third volume data block entry is identified as an anomalous entry.

7. The method according to claim 6, further comprising: After determining the clustered pool, determine a first number of volume data block entries subsequently clustered into the clustered pool and a second number of volume data block entries not clustered into the clustered pool; and In response to the ratio of the first number to the second number being less than a predetermined ratio threshold, a new pool of the same type is re-established by selecting a new anchor entry.

8. The method according to claim 1, further comprising: Determine the total number of the plurality of volume data block entries; In response to the total number being greater than a first predetermined total number threshold, the plurality of volume data block entries are split into a first group of volume data block entries and a second group of volume data block entries; as well as Abnormal entries in the first group of volume data block entries and abnormal entries in the second group of volume data block entries are detected in parallel.

9. The method of claim 8, wherein detecting anomalous entries in the first set of volume data block entries and anomalous entries in the second set of volume data block entries in parallel comprises: The first group of volume data block entries is split into a third group of volume data block entries and a fourth group of volume data block entries; as well as Abnormal entries in the third group of volume data block entries and abnormal entries in the fourth group of volume data block entries are detected serially.

10. The method of claim 9, wherein serially detecting anomalous entries in the third group of volume data block entries and anomalous entries in the fourth group of volume data block entries comprises: Determine the number of abnormal entries in the third group of volume data block entries; In response to the fact that the number of abnormal entries in the third group of volume data block entries is less than a predetermined abnormal number threshold, a portion of the entries in the third group of volume data block entries are added to the fourth group of volume data block entries; as well as Detect abnormal entries in the updated fourth group of volume data block entries.

11. An electronic device, comprising: At least one processor; as well as A memory, coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the electronic device to perform actions, including: Retrieve multiple attribute values ​​for multiple volume data block entries; Based on the multiple attribute values ​​of the multiple volume data block entries, a first portion of the multiple volume data block entries is clustered into a similar pool; and The second portion of the multiple volume data block entries that are not clustered into the same pool is identified as anomaly entries.

12. The apparatus of claim 11, wherein the plurality of attribute values ​​of the plurality of volume block entries include a first value of an attribute of a first volume block entry and a second value of an attribute of a second volume block entry, and clustering the first portion of the plurality of volume block entries into the homogeneous pool based on the plurality of attribute values ​​of the plurality of volume block entries includes: The first value is converted into a third value based on the type of the attribute. The second value is converted into a fourth value based on the type of the attribute, wherein the difference between the third value and the fourth value is greater than the difference between the first value and the second value; and Based on the third and fourth values, one or both of the first volume data block entries and the second volume data block entries are clustered into the same pool.

13. The apparatus of claim 11, wherein clustering the first portion of the plurality of volume block entries into the homogeneous pool based on the plurality of attribute values ​​of the plurality of volume block entries comprises: The first volume data block entry among the plurality of volume data block entries is identified as the anchor entry; Create a candidate pool of similar entries based on the anchor entries; The same pool is determined based on the candidate similar pools and the third portion of the plurality of volume data block entries; as well as The fourth part is clustered into the same pool based on multiple attribute values ​​of the fourth part in the plurality of volume data block entries, wherein the first part includes at least a portion of the third part and the fourth part.

14. The apparatus of claim 13, wherein determining the homogeneous pool based on the third portion of the candidate homogeneous pool and the plurality of volume data block entries comprises: Determine a first density for the candidate pool of similar volumes, the first density indicating the differences between multiple volume data block entries in the candidate pool of similar volumes; Determine the second density of the candidate pool if the second volume data block entry from the third part is added to the candidate pool; Based on the first density and the second density of the candidate pool, determine the density change of the candidate pool if the second volume data block entry is added to the candidate pool; and The pool of the same type is determined based on the density change of the candidate pool of the same type.

15. The apparatus of claim 14, wherein determining the similar pool based on the density change of the candidate similar pool comprises: In response to the density change being greater than a predetermined density change threshold, the second volume data block entry is added to the candidate pool of the same type; as well as In response to the number of volume data block entries in the candidate pool being greater than a predetermined threshold, the candidate pool is determined as the pool.

16. The apparatus of claim 13, wherein identifying the second portion of the plurality of volume data block entries that are not clustered into the same pool as anomalous entries comprises: After determining the homogeneous pool, determine the density change of the homogeneous pool if a third volume data block entry from the plurality of volume data block entries is added to the homogeneous pool; and In response to the density change in the same pool exceeding a predetermined density change threshold, the third volume data block entry is identified as an anomalous entry.

17. The device according to claim 16, further comprising: After determining the clustered pool, determine a first number of volume data block entries subsequently clustered into the clustered pool and a second number of volume data block entries not clustered into the clustered pool; and In response to the ratio of the first number to the second number being less than a predetermined ratio threshold, a new pool of the same type is re-established by selecting a new anchor entry.

18. The device according to claim 11, wherein the action further includes: Determine the total number of the plurality of volume data block entries; In response to the total number being greater than a first predetermined total number threshold, the plurality of volume data block entries are split into a first group of volume data block entries and a second group of volume data block entries; as well as Abnormal entries in the first group of volume data block entries and abnormal entries in the second group of volume data block entries are detected in parallel.

19. The apparatus of claim 18, wherein detecting anomalous entries in the first set of volume data block entries and anomalous entries in the second set of volume data block entries in parallel comprises: The first group of volume data block entries is split into a third group of volume data block entries and a fourth group of volume data block entries; as well as Abnormal entries in the third group of volume data block entries and abnormal entries in the fourth group of volume data block entries are detected serially.

20. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to: Retrieve multiple attribute values ​​for multiple volume data block entries; Based on the multiple attribute values ​​of the multiple volume data block entries, the first portion of the multiple volume data block entries is clustered into a pool of the same type; as well as The second portion of the multiple volume data block entries that are not clustered into the same pool is identified as anomaly entries.

Images