Method for identifying faulty storage device and related device

Abstract

The embodiment of the invention provides a method for identifying a faulty storage device, which comprises the following steps: executing a first collection task, the first collection task is to collect log data of a full storage device, and the full storage device comprises one or more storage devices in each of a plurality of servers. The health degree of each storage device in the full storage devices is determined according to the log data collected by the first collection task, the first type of storage devices are determined from the full storage devices according to the health degree of each storage device, and the health degree of the first type of storage devices is lower than the first health value. And executing a second collection task which is used for collecting log data of the first type of storage devices, and determining the storage devices with faults in the first type of storage devices according to the log data collected by the second collection task.

Description

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office on July 7, 2023, with application number 202310833333.1, and invention name “A hardware fault alarm method, device and computing device cluster”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a method for identifying a faulty storage device and related devices. Background Art

[0003] Nowadays, memory failures occur frequently in large-scale data centers, causing a decrease in the stability and reliability of servers and even the entire IT infrastructure, which has a negative impact on business service level agreements (SLAs). Memory failures are the main cause of server crashes, and whether memory failures can be identified in a timely manner has become one of the important issues that the industry needs to study and solve in the cloud computing era.

[0004] Existing technologies periodically obtain log data of all memories in a data center and identify memory failures based on these log data. However, for large-scale data centers, the number of memories is often hundreds of thousands, resulting in a large amount of time spent on each round of data acquisition. It usually takes only a short time from the occurrence of a memory failure to causing a server crash, so it may not be possible to identify a memory failure before the server crashes. Summary of the invention

[0005] The present application provides a method for identifying a faulty storage device and related devices, which are used to effectively avoid the situation where a faulty storage device cannot be identified in time.

[0006] The first aspect of the present application provides a method for identifying a faulty storage device:

[0007] A data center includes multiple servers, each of which includes one or more storage devices. In addition, a fault identification device is provided in the data center, and the method is applied to the fault identification device. The process of the method includes: executing a first collection task, the first collection task is to collect log data of a full storage device, and the full storage device includes one or more storage devices in each server of a plurality of servers. Determine the health of each storage device in the full storage device according to the log data collected by the first collection task. Determine a first type of storage device from the full storage device according to the health of each storage device, and the health of the first type of storage device is lower than the first health value. Execute a second collection task, the second collection task is to collect log data of the first type of storage device. Determine a storage device with a fault in the first type of storage device according to the log data collected by the second collection task.

[0008] In the present application, the first type of storage devices can be understood as storage devices with a higher risk of failure, and their number accounts for a smaller proportion. These storage devices are screened out based on the health of the full amount of storage devices, so that log data can be collected from these storage devices in a targeted manner. This can greatly shorten the time required for data collection and avoid the situation where faulty storage devices cannot be identified in a timely manner.

[0009] In a possible implementation, a second type of storage device is determined from all storage devices according to the health of each storage device, the health of the second type of storage device is less than a second health value, the second health value is less than the first health value, and the first type of storage device is a storage device whose health is between the second health value and the first health value. An alarm is issued for the second type of storage device.

[0010] In the present application, the second type of storage device can be understood as a storage device with a very high risk of failure. Therefore, after screening out this part of the storage devices, an alarm can be directly issued to improve the efficiency of the alarm.

[0011] In a possible implementation, the log data collected by the first collection task is input into a health algorithm model to obtain a confidence level corresponding to each storage device, where the confidence level is used to indicate the health level of the storage device.

[0012] In a possible implementation, the first acquisition task occupies the first data acquisition link, and the second acquisition task occupies the second data acquisition link, and the data will be suppressed when the first data acquisition link collects data. The method also includes: executing a third acquisition task in the second data acquisition link, and the third acquisition task is to collect log data of the full storage device. Execute the first inspection task, and the first inspection task is to inspect the log data collected by the third acquisition task to determine the storage device with faults in the full storage device. When the first type of storage device is determined, the operation of suspending the third acquisition task is triggered and the second acquisition task is executed. When the second acquisition task is completed, continue to execute the third acquisition task.

[0013] In this application, the log data of all storage devices will also be inspected, and the third collection task and the second collection task occupy the same data collection link, thereby reducing the required resources and improving the comprehensiveness of faulty storage device identification. Since the data will be suppressed when the first data collection link collects data, the time consumption of the first collection task can be greatly reduced, so that the first type of storage device can be determined more quickly, triggering the execution of the second collection task, and ensuring that the faulty storage device can be identified in time.

[0014] In a possible implementation, when the second collection task is completed, the operation of pausing the first inspection task is triggered and the second inspection task is executed, and the second inspection task is to inspect the log data collected by the second collection task to determine the storage device with faults in the first type of storage device. When the second inspection task is completed, the first inspection task is continued to be executed.

[0015] In the present application, the second inspection task takes precedence over the first inspection task, further improving the efficiency of identifying faulty storage devices.

[0016] In a possible implementation, a third type of storage device is determined from all storage devices according to the health level of each storage device, the health level of the third type of storage device is lower than a third health value, and the third health value is greater than the first health value. Log data of the third type of storage device is selected from the log data collected by the first collection task. The log data of the third type of storage device is input into a fault classification prediction algorithm model to determine a storage device with a fault in the third type of storage device.

[0017] In the present application, storage devices whose health level is greater than or equal to the third health value can be understood as storage devices that have basically no risk of failure. Correspondingly, third-type storage devices can be understood as storage devices that have a risk of failure. The log data of these storage devices are screened out from the log data collected by the first collection task for fault prediction, which not only improves the comprehensiveness of faulty storage device identification, but also avoids unnecessary overhead.

[0018] In a possible implementation manner, the storage device is a storage medium.

[0019] In a second aspect of the present application, a fault identification device is provided, which is applied to a data center, wherein the data center includes a plurality of servers, each of which includes one or more storage devices. The fault identification device includes a collection module and a determination module.

[0020] The collection module is used to execute a first collection task, where the first collection task is to collect log data of a full storage device, where the full storage device includes one or more storage devices in each server of a plurality of servers.

[0021] The determination module is used to determine the health of each storage device in the full storage device according to the log data collected by the first collection task.

[0022] The determination module is further used to determine a first type of storage device from all storage devices according to the health level of each storage device, and the health level of the first type of storage device is lower than the first health value.

[0023] The collection module is further used to execute a second collection task, where the second collection task is to collect log data from the first type of storage device.

[0024] The determination module is further used to determine a faulty storage device in the first type of storage device according to the log data collected by the second collection task.

[0025] In a possible implementation, the fault identification device further includes an alarm module.

[0026] The determination module is also used to determine a second type of storage device from the full amount of storage devices according to the health of each storage device, the health of the second type of storage device is less than a second health value, the second health value is less than the first health value, and the first type of storage device is a storage device whose health is between the second health value and the first health value.

[0027] The alarm module is used to generate an alarm for the second type of storage device.

[0028] In one possible implementation,

[0029] The determination module is specifically used to input the log data collected by the first collection task into the health degree algorithm model to obtain the confidence degree corresponding to each storage device, and the confidence degree is used to indicate the health degree of the storage device.

[0030] In a possible implementation, the first acquisition task occupies a first data acquisition link, and the second acquisition task occupies a second data acquisition link. When the first data acquisition link acquires data, data is suppressed.

[0031] The collection module is also used to execute a third collection task in the second data collection link, where the third collection task is to collect log data of the full storage device.

[0032] The determination module is further used to execute a first inspection task, where the first inspection task is to inspect the log data collected by the third collection task to determine a storage device with a fault in the full storage device.

[0033] The collection module is specifically used to trigger the operation of pausing the third collection task and executing the second collection task when the first type storage device is determined.

[0034] The collection module is also used to continue to execute the third collection task when the second collection task is completed.

[0035] In one possible implementation,

[0036] The determination module is specifically used to trigger the suspension of the first inspection task and execute the second inspection task when the second collection task is completed. The second inspection task is to inspect the log data collected by the second collection task to determine the storage device with faults in the first type of storage device.

[0037] The determination module is also used to continue executing the first inspection task when the second inspection task is completed.

[0038] In one possible implementation,

[0039] The determination module is further used to determine a third type of storage device from all storage devices according to the health level of each storage device, the health level of the third type of storage device is lower than a third health value, and the third health value is greater than the first health value.

[0040] The collection module is further used to select log data of the third type of storage device from the log data collected by the first collection task.

[0041] The determination module is further used to input the log data of the third type storage device into the fault classification prediction algorithm model to determine the storage device with faults in the third type storage device.

[0042] In a possible implementation manner, the storage device is a storage medium.

[0043] A third aspect of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method of the first aspect mentioned above.

[0044] A fourth aspect of the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method of the aforementioned first aspect.

[0045] The fifth aspect of the present application provides a computer program product, which, when instructions are executed by a computing device cluster, enables the computing device cluster to execute the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1a This is a schematic diagram of the time distribution from the first CE to the first UCE;

[0047] Figure 1b A schematic diagram of the scenario used in this application;

[0048] Figure 2 A schematic diagram of a flow chart of a method for identifying a faulty storage device in the present application;

[0049] Figure 3 Another flowchart of the method for identifying a faulty storage device in the present application is shown;

[0050] Figure 4 A schematic diagram of the priority control of target collection orders in this application;

[0051] Figure 5 Another flowchart of the method for identifying a faulty storage device in the present application is shown;

[0052] Figure 6 Another flowchart of the method for identifying a faulty storage device in the present application is shown;

[0053] Figure 7 It is a structural schematic diagram of the fault identification device in this application;

[0054] Figure 8 A schematic diagram of the structure of the computing device in this application;

[0055] Fig. 9 A schematic diagram of a computing device cluster in this application;

[0056] Fig.10 Another schematic diagram of a computing device cluster in this application. DETAILED DESCRIPTION

[0057] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] To facilitate understanding of this application, the concepts involved in this application are introduced below:

[0060] Correctable Errors (CE): Correctable errors are errors that can be detected and corrected by the server platform. These are usually single-bit errors, but based on the CPU and memory configuration, they can also be certain types of multi-bit errors (corrected by advanced ECC). Correctable errors can be caused by both soft and hard errors and will not disrupt the operation of the server.

[0061] Uncorrectable Errors (UCE): Uncorrectable errors are multi-bit errors that cannot be corrected by the server platform. These errors can be caused by any combination of soft or hard errors, but are usually caused by multiple hard errors.

[0062] Fault classification prediction algorithm model: It can collect and clean the fault logs of storage devices to obtain the corresponding fault performance information of each storage device, including but not limited to the time of fault occurrence, the system address of the fault occurrence, the number of fault occurrences, etc. By constructing the time and space characteristics of the fault performance information, it uses deep learning or rules to predict whether the storage device will fail.

[0063] Health degree algorithm model: After cleaning and constructing the features of the fault logs of the storage device before the uncorrectable fault occurs, the storage device is modeled using features of different time scales and spatial scales, and the confidence level of the health degree of the storage device is output, thereby determining the health degree of the storage device.

[0064] Inspection: When an uncorrectable error occurs in a storage device, the relevant fault register information will be stored. Therefore, the fault can be determined based on the register information to determine whether the storage device is faulty. Since the log data records the errors that have occurred, the inspection can directly determine whether the storage device is faulty based on the log data.

[0065] Nowadays, memory failures are frequent in large-scale data centers, causing a decline in the stability and reliability of servers and even the entire IT infrastructure, which has a negative impact on SLA. Memory failures are the main cause of server crashes. Whether memory failures can be identified in a timely manner has become one of the important issues that the industry needs to study and solve in the cloud computing era. Traditional memory failure prediction systems predict memory reliability based on system logs such as memory and hard disks. However, large data centers often include hundreds of thousands of memory and other basic hardware facilities, and there is a lot of delay in collecting system logs. See Figure 1a , Figure 1a The figure shows the time distribution from the first CE to the first UCE of the memory. It can be seen that nearly 40% of UCEs occur within one day after the first CE. If the collection frequency of memory logs is less than one day, nearly 40% of memory errors cannot be predicted in advance. Therefore, improving the timeliness of log collection is of great significance for the early prediction of memory failures.

[0066] See also Figure 1b The method for identifying a faulty storage device in the present application can be applied to Figure 1b In the system architecture shown. A data center includes multiple servers, each of which includes a network card, a processor, and a storage device such as a memory bar or a disk that communicates through a bus. In order to avoid server crashes caused by storage device failures, the data center is equipped with a fault identification device, which is configured to communicate with each server in the data center, thereby collecting log data related to the storage device generated by each server, and determining the storage device with a fault (hereinafter referred to as the faulty storage device) based on the log data, so as to perform subsequent fault recovery.

[0067] See also Figure 2 The following is a detailed description of the process of the method for identifying a faulty storage device in the present application:

[0068] 201. Execute a first collection task, where the first collection task is to collect log data from a full storage device, where the full storage device includes one or more storage devices in each of a plurality of servers.

[0069] See also Figure 3, the fault identification device is provided with two data collection links for collecting log data related to storage devices generated by the server, and a data collection and processing platform for processing the log data collected by the data collection link. Among them, the data collection and processing platform 1 periodically collects the log data of each storage device on each server in the data center through the data collection link 1 (after completing one round of collection, the next round of collection is immediately started). For the convenience of subsequent description, this type of log data is recorded as full log data. The data collection and processing platform 1 reports the full log data to the deep inspection system, and the deep inspection system inspects these log data to determine the faulty storage device. The list of faulty storage devices will be reported to the inspection alarm platform, and the inspection alarm platform will generate corresponding alarms. It should be understood that since the data collection link 1 will periodically collect the full log data, the deep inspection system will also periodically inspect the full log data it has collected.

[0070] Data collection and processing platform 2 also periodically collects full log data through data collection link 2. Since there is a large amount of redundant data in the log data that is not required by the algorithm, such as intermediate data stored in registers and information of other storage devices, data collection and processing platform 2 will perform log preprocessing on the full log data, extract useful information, and only retain data related to the algorithm; in addition, data collection and processing platform 2 performs feature extraction on the preprocessed full data, converts text information into the data format required by the algorithm, such as numerical information, matrix information, etc., that is, feature engineering. After performing log preprocessing and feature engineering on the full log data, data collection and processing platform 2 reports it to the fault algorithm platform.

[0071] 202. Determine the health of each storage device in the full storage device according to the log data collected by the first collection task;

[0072] The fault algorithm platform is configured with a health algorithm model. The log data of the storage device is input into the model. The model will output the corresponding confidence level, which is used to indicate the health level of the storage device. In one case, the higher the confidence level, the healthier the storage device. For example, the confidence value can be from 0 to 100, where the larger the value, the healthier the storage device. Of course, the confidence value can also be set to be inversely proportional to the health level of the storage device, which is not limited here. The fault algorithm platform inputs the aforementioned full amount of log data into the health algorithm model to obtain the health level of each storage device.

[0073] 203. Determine a first type of storage device from all storage devices according to the health level of each storage device, the health level of the first type of storage device being lower than a first health value;

[0074] The fault algorithm platform determines that the storage device with a health level lower than the first health value is a first type of storage device. The first health value may be, for example, a corresponding confidence level of 30. The first type of storage device can be understood as a storage device with a higher risk of failure, and the number accounts for a relatively small proportion. The fault algorithm platform will generate a target collection form, which includes all the first type of storage devices determined above. Specifically, the target collection form may include the inherent attributes of the first type of storage device, such as the serial number of the storage device, through which the server where the first type of storage device is located and information such as the IP address can be found. In addition, the log data collected by the server also contains the inherent attributes of the storage device, which can be matched with the information in the target collection form.

[0075] 204. Execute a second collection task, where the second collection task is to collect log data from the first type of storage device;

[0076] After generating the target collection form, the fault algorithm platform sends the target collection form to the data collection and processing platform 1. Figure 4 In a possible implementation, the data collection and processing platform 1 includes a log collection control system, a priority controller and a data collector. The target collection list from the fault algorithm platform will be sent to the log collection control system. The log collection control system performs priority control on the priority controller and sets the storage device in the target collection list to the highest collection priority. The priority controller issues instructions to the data collector, instructing the data collector to give priority to collecting log data from the storage devices in the target collection list. Based on the above settings, when the data collection and processing platform 1 receives the target collection list, it will first suspend the collection of the full amount of log data, and switch to giving priority to collecting the log data of the first type of storage device in the target collection list through the data collection link 1. After completing the collection of the log data of the first type of storage device, the data collection and processing platform 1 will continue to collect the full amount of log data.

[0077] 205. Determine a faulty storage device in the first type of storage devices according to the log data collected by the second collection task.

[0078] After the log data collection of the first type of storage device is completed, the data collection and processing platform 1 reports the log data of the first type of storage device to the deep inspection system. At this time, if the deep inspection system is inspecting the full amount of log data collected previously, the deep inspection system can first suspend the inspection of the full amount of log data, and switch to prioritize the inspection of the log data of the first type of storage device, and then continue the inspection of the full amount of log data; or, the deep inspection system can also complete the inspection of the full amount of log data first, and then inspect the log data of the first type of storage device.

[0079] In a possible implementation, the fault algorithm platform may also directly determine the second type storage device as a faulty storage device and report it to the prediction and alarm platform. The second type storage device is a storage device whose health level is lower than the second health value. The second health value is lower than the first health value mentioned above. For example, it may correspond to a confidence level of 10. The second type storage device may be understood as a storage device with a very high risk of failure. Of course, the aforementioned first type storage device may also only refer to a storage device that is lower than the first health value but higher than the second health value.

[0080] In addition, the fault algorithm platform is also configured with a fault classification prediction algorithm model. After the full amount of log data is input into the fault classification prediction algorithm model, the model outputs the result of whether each storage device is a faulty storage device. The list of faulty storage devices will be reported to the prediction alarm platform, and the prediction alarm platform will generate corresponding alarms. Of course, the fault algorithm platform can also determine the third type of storage device, which is a storage device with a health level lower than the third health value, and the third health value is greater than the aforementioned first health value, for example, it can correspond to a confidence level of 60. Storage devices with a health level higher than or equal to the third health value can be understood as storage devices with a lower risk of failure, and correspondingly, the third type of storage devices can be understood as storage devices with a risk of failure. The fault algorithm platform can select the log data of the third type of storage device from the full amount of log data, and input it into the fault classification prediction algorithm model, thereby determining the faulty storage device and reporting it to the prediction alarm platform. Based on this, the amount of data that needs to be processed by the fault classification prediction algorithm model can be reduced, thereby saving resources and time.

[0081] Of course, in a possible implementation, after the full amount of log data is input into the health algorithm model in the aforementioned step 202, the health algorithm model can also convert the confidence into different labels to indicate the health level after obtaining the confidence level. The different labels include, for example, "slightly unhealthy", "moderately unhealthy" and "very unhealthy", where the storage device labeled "slightly unhealthy" corresponds to the aforementioned third type of storage device; the storage device labeled "moderately unhealthy" corresponds to the aforementioned first type of storage device; the storage device labeled "very unhealthy" corresponds to the aforementioned second type of storage device.

[0082] The above introduces the processing process of the fault algorithm platform on the full log data collected by the data collection link 2. It should be understood that since the data collection link 2 will periodically collect the full log data, the fault algorithm platform will also periodically perform similar processing on the collected full log data as mentioned above, and the details will not be repeated here.

[0083] In the above process, log data will not be suppressed when data collection link 1 collects data, but log data will be suppressed when data collection link 2 collects data. For example, only a part of the large amount of log data generated by the same storage device in a short period of time will be collected. Therefore, data collection link 2 can collect the full amount of log data faster, so that the fault algorithm platform can generate target collection orders at a high frequency and send them to the data collection and processing platform 1. And because the number of first-type storage devices is relatively small, the log data collection and inspection of the first-type storage devices can be completed quickly. It is not difficult to see that compared with the inspection of the full amount of log data, the inspection of the first-type storage devices has been greatly improved in frequency, so it can effectively avoid the situation where the faulty storage device cannot be identified in time.

[0084] Another process of the method for identifying a faulty storage device in the present application is introduced below:

[0085] A01. Execute a first collection task, where the first collection task is to collect log data from a full storage device, where the full storage device includes one or more storage devices in each of a plurality of servers;

[0086] See also Figure 5 , compared to Figure 3The fault identification device shown in the present embodiment newly adds a data collection link 3, a data collection and processing platform 3, and a deep inspection system 2. Similarly, the data collection and processing platform 1 periodically collects the full amount of log data through the data collection link 1, and reports it to the deep inspection system 1, which inspects the log data to identify the faulty storage device. The list of faulty storage devices will be reported to the inspection and alarm platform, which will generate corresponding alarms. It should be understood that since the data collection link 1 will periodically collect the full amount of log data, the deep inspection system 1 will also periodically inspect the full amount of log data it has collected.

[0087] The data collection and processing platform 2 also periodically collects the full amount of log data through the data collection link 2, and reports the full amount of log data to the fault algorithm platform after log preprocessing and feature engineering.

[0088] A02. Determine the health of each storage device in the full storage device according to the log data collected by the first collection task;

[0089] The fault algorithm platform is configured with a health algorithm model. The fault algorithm platform inputs the aforementioned full log data into the health algorithm model to obtain the health of each storage device.

[0090] A03. Determine a first type of storage device from all storage devices according to the health level of each storage device, where the health level of the first type of storage device is lower than a first health value;

[0091] The fault algorithm platform determines the storage devices whose health level is lower than the first health value as first type storage devices, and the fault algorithm platform generates a target collection list, which includes all the first type storage devices determined above.

[0092] A04. Execute a second collection task, where the second collection task is to collect log data from the first type of storage device;

[0093] After generating the target collection form, the fault algorithm platform sends the target collection form to the data collection and processing platform 3. After receiving the target collection form, the data collection and processing platform 3 periodically collects log data of the first type storage device through the data collection link 3.

[0094] A05. Determine a faulty storage device in the first type of storage devices according to the log data collected by the second collection task.

[0095] After the collection is completed, the data collection and processing platform 3 uploads the collected log data of the first type of storage devices to the deep inspection system 2, and the deep inspection system 2 inspects the log data to determine the faulty storage devices in the first type of storage devices.

[0096] In a possible implementation, the fault algorithm platform may also directly determine the second type storage device as a faulty storage device and report it to the prediction and alarm platform. The second type storage device is a storage device whose health level is lower than the second health value, and the second health value is lower than the first health value mentioned above. Of course, the aforementioned first type storage device may also only refer to a storage device that is lower than the first health value but higher than the second health value.

[0097] In addition, the fault algorithm platform is also configured with a fault classification prediction algorithm model. After the full amount of log data is input into the fault classification prediction algorithm model, the model outputs the result of whether each storage device is a faulty storage device. The list of faulty storage devices will be reported to the prediction alarm platform, and the prediction alarm platform will generate corresponding alarms. Of course, the fault algorithm platform can also determine the third type of storage device, which is a storage device with a health level lower than the third health value, and the third health value is greater than the aforementioned first health value. The fault algorithm platform can select the log data of the third type of storage device from the full amount of log data, and input it into the fault classification prediction algorithm model, thereby determining the faulty storage device and reporting it to the prediction alarm platform. Based on this, the amount of data that needs to be processed by the fault classification prediction algorithm model can be reduced, thereby saving resources and time.

[0098] The above describes the process of the fault algorithm platform processing the full amount of log data collected by the data collection link 2. It should be understood that since the data collection link 2 will periodically collect the full amount of log data, the fault algorithm platform will also periodically process the collected full amount of log data in a similar manner to the above, and the details will not be repeated here. When the fault algorithm platform generates a target collection form again and sends it to the data collection and processing platform 3, the data collection and processing platform 3 periodically collects the log data of the first type of storage device according to this target collection form to determine the faulty storage device.

[0099] The above describes the application of the target collection form to patrol alarms. In another possible implementation, the target collection form can also be applied to predictive alarms, which is described in detail below:

[0100] B01. Execute a first collection task, where the first collection task is to collect log data from a full storage device, where the full storage device includes one or more storage devices in each of a plurality of servers;

[0101] See also Figure 6 In this embodiment, the fault identification device is provided with two data collection links for collecting log data related to the storage device generated by the server, and a data collection and processing platform for processing the log data collected by the data collection link. Among them, the data collection and processing platform 2 periodically collects the full amount of log data through the data collection link 2, and reports it to the fault algorithm platform 2 after log preprocessing and feature engineering of the full amount of log data.

[0102] B02. Determine the health of each storage device in the full storage device according to the log data collected by the first collection task;

[0103] The fault algorithm platform 2 is configured with a health degree algorithm model. The fault algorithm platform 2 inputs the aforementioned full log data into the health degree algorithm model to obtain the health degree of each storage device.

[0104] B03. Determine a first type of storage device from all storage devices according to the health level of each storage device, where the health level of the first type of storage device is lower than a first health value;

[0105] The fault algorithm platform 2 determines the storage devices whose health level is lower than the first health value as the first type of storage devices, and the fault algorithm platform 2 generates a target collection list, which includes all the first type of storage devices determined above.

[0106] B04, executing a second collection task, where the second collection task is to collect log data from the first type of storage device;

[0107] After generating the target collection form, the fault algorithm platform 2 sends the target collection form to the data collection and processing platform 1. After receiving the target collection form, the data collection and processing platform 1 periodically collects log data of the first type storage device through the data collection link 1.

[0108] B05. Determine a faulty storage device in the first type of storage devices according to the log data collected by the second collection task.

[0109] After the collection is completed, the data collection and processing platform 1 uploads the collected log data of the first type of storage device to the fault algorithm platform 1. The fault algorithm platform 1 is configured with a storage device fault classification prediction algorithm model, and the fault algorithm platform 1 inputs the log data of the first type of storage device into the model to determine the faulty storage device and report it to the prediction alarm platform.

[0110] Fault algorithm platform 2 is also configured with a storage device fault classification prediction algorithm model. The difference is that fault algorithm platform 2 will input the full log data into the model to determine the faulty storage device and report it to the prediction alarm platform.

[0111] In the above process, fault algorithm platform 2 is responsible for detecting the full amount of log data to ensure the comprehensiveness of the prediction alarm; and the fault algorithm platform is responsible for detecting the log data of the first type of storage device to improve the pertinence and efficiency of the prediction alarm, and effectively avoid the situation where the faulty storage device cannot be identified in time.

[0112] The method in this application is introduced above, and the fault identification device in this application is introduced below:

[0113] See also Figure 7 , the fault identification device 700 in the present application includes a collection module 701 and a determination module 702 .

[0114] The collection module 701 is used to execute a first collection task, where the first collection task is to collect log data from a full storage device, where the full storage device includes one or more storage devices in each of a plurality of servers.

[0115] The determination module 702 is used to determine the health of each storage device in the full storage device according to the log data collected by the first collection task.

[0116] The determination module 702 is further configured to determine a first type of storage device from all storage devices according to the health level of each storage device, wherein the health level of the first type of storage device is lower than a first health value.

[0117] The collection module 701 is further used to execute a second collection task, where the second collection task is to collect log data from the first type of storage device.

[0118] The determination module 702 is further configured to determine a faulty storage device in the first type of storage devices according to the log data collected by the second collection task.

[0119] In a possible implementation, the fault identification device further includes an alarm module 703 .

[0120] Determination module 702 is also used to determine a second type of storage device from all storage devices based on the health of each storage device, the health of the second type of storage device is less than the second health value, the second health value is less than the first health value, and the first type of storage device is a storage device whose health is between the second health value and the first health value.

[0121] The alarm module 703 is used to generate an alarm for the second type storage device.

[0122] In one possible implementation,

[0123] The determination module 702 is specifically used to input the log data collected by the first collection task into the health degree algorithm model to obtain the confidence degree corresponding to each storage device, and the confidence degree is used to indicate the health degree of the storage device.

[0124] In a possible implementation, the first acquisition task occupies a first data acquisition link, and the second acquisition task occupies a second data acquisition link. When the first data acquisition link acquires data, data is suppressed.

[0125] The collection module 701 is further used to execute a third collection task in the second data collection link, where the third collection task is to collect log data of the full storage device.

[0126] The determination module 702 is further configured to execute a first inspection task, wherein the first inspection task is to inspect the log data collected by the third collection task to determine a storage device having a fault in the full storage device.

[0127] The acquisition module 701 is specifically configured to trigger the operation of pausing the third acquisition task and executing the second acquisition task when the first type storage device is determined.

[0128] The collection module 701 is further configured to continue executing the third collection task when the second collection task is completed.

[0129] In one possible implementation,

[0130] The determination module 702 is specifically used to trigger the suspension of the first inspection task and execute the second inspection task when the second collection task is completed. The second inspection task is to inspect the log data collected by the second collection task to determine the storage device with faults in the first type of storage devices.

[0131] The determination module 702 is further configured to continue executing the first inspection task when the second inspection task is completed.

[0132] In one possible implementation,

[0133] The determination module 702 is further configured to determine a third type of storage device from all storage devices according to the health level of each storage device, wherein the health level of the third type of storage device is lower than a third health value, and the third health value is greater than the first health value.

[0134] The collection module 701 is further configured to select log data of a third type of storage device from the log data collected by the first collection task.

[0135] The determination module 702 is further configured to input the log data of the third type storage device into the fault classification prediction algorithm model to determine the storage device with faults in the third type storage device.

[0136] In a possible implementation manner, the storage device is a storage medium.

[0137] Among them, the acquisition module, the determination module and the alarm module can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the acquisition module is introduced below by taking the acquisition module as an example. Similarly, the implementation of the determination module and the alarm module can refer to the implementation of the acquisition module.

[0138] As an example of a software functional unit, the acquisition module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0139] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0140] As an example of a hardware functional unit, the acquisition module may include at least one computing device, such as a server, etc. Alternatively, the acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0141] The multiple computing devices included in the acquisition module can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0142] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the method for identifying a faulty storage device in the present application, the determination module can be used to execute any step in the method for identifying a faulty storage device in the present application, and the alarm module can be used to execute any step in the method for identifying a faulty storage device in the present application. The steps that the acquisition module, the determination module, and the alarm module are responsible for implementing can be specified as needed. The full functions of the fault identification device are realized by respectively implementing different steps in the method for identifying a faulty storage device in the present application through the acquisition module, the determination module, and the alarm module.

[0143] The present application also provides a computing device 100. Figure 8 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0144] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 104 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0145] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0146] The memory 106 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0147] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module, determination module, and alarm module, thereby implementing the method for identifying a faulty storage device in the present application. That is, the memory 106 stores instructions for executing the method for identifying a faulty storage device in the present application.

[0148] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0149] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0150] like Fig. 9 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the method for identifying a faulty storage device in the present application.

[0151] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also respectively store some instructions for executing the method for identifying a faulty storage device in the present application. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the method for identifying a faulty storage device in the present application.

[0152] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the fault identification device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules among the collection module, the determination module and the alarm module.

[0153] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the acquisition module. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the determination module and the alarm module.

[0154] Fig.10 The connection mode between the computing device clusters shown may be that considering that the method for identifying a faulty storage device provided in the present application requires the collection of a large amount of log data, it is considered that the functions implemented by the collection module are handed over to the computing device 100A for execution.

[0155] It should be understood that Fig.10The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0156] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the method for identifying a faulty storage device in the present application.

[0157] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the method for identifying a faulty storage device in the present application.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying a faulty storage device, applied to a fault identification device of a data center, wherein the data center further comprises a plurality of servers, each of the plurality of servers comprises one or more storage devices, characterized in that: The method comprises: Executing a first collection task, where the first collection task is to collect log data of a full storage device, where the full storage device includes the one or more storage devices in each of the multiple servers; Determining the health of each storage device in the full storage device according to the log data collected by the first collection task; Determine a first type of storage device from the full amount of storage devices according to the health level of each storage device, the health level of the first type of storage device being lower than a first health value; Executing a second collection task, where the second collection task is to collect log data from the first type of storage device; A faulty storage device in the first type of storage devices is determined according to the log data collected by the second collection task.

2. The method according to claim 1, characterized in that The method further comprises: Determine a second type of storage device from the full amount of storage devices according to the health level of each storage device, the health level of the second type of storage device is less than a second health value, the second health value is less than the first health value, and the first type of storage device is a storage device whose health level is between the second health value and the first health value; An alarm is issued for the second type of storage device.

3. The method according to claim 1 or 2, characterized in that: The determining the health of each storage device in the full storage device according to the log data collected by the first collection task includes: The log data collected by the first collection task is input into a health degree algorithm model to obtain a confidence degree corresponding to each storage device, where the confidence degree is used to indicate the health degree of the storage device.

4. The method according to any one of claims 1 to 3, characterized in that The first acquisition task occupies a first data acquisition link, and the second acquisition task occupies a second data acquisition link, and the data is suppressed when the first data acquisition link acquires data; The method further comprises: Executing a third collection task in the second data collection link, wherein the third collection task is to collect log data of the full storage device; Executing a first inspection task, wherein the first inspection task is to inspect the log data collected by the third collection task to determine a storage device with a fault in the full storage device; The performing of the second acquisition task comprises: When the first type of storage device is determined, triggering the operation of pausing the third acquisition task and executing the second acquisition task; The method further comprises: When the second collection task is completed, continue to execute the third collection task.

5. The method according to claim 4, characterized in that The step of determining, according to the log data collected by the second collection task, a storage device having a fault in the first type of storage device comprises: When the second collection task is completed, triggering the operation of pausing the first inspection task and executing the second inspection task, the second inspection task is to inspect the log data collected by the second collection task to determine the storage device with fault in the first type of storage device; The method further comprises: When the second inspection task is completed, continue to execute the first inspection task.

6. The method according to claim 5, characterized in that The method further comprises: Determine a third type of storage device from the full amount of storage devices according to the health level of each storage device, the health level of the third type of storage device is lower than a third health value, and the third health value is greater than the first health value; Selecting the log data of the third type of storage device from the log data collected by the first collection task; The log data of the third type storage device is input into a fault classification prediction algorithm model to determine a storage device with faults in the third type storage device.

7. The method according to any one of claims 1 to 6, characterized in that The storage device is a memory stick or a disk.

8. A fault identification device, characterized in that: Applied to a data center, the data center includes a plurality of servers, each of which includes one or more storage devices, and the apparatus includes a collection module and a determination module; The collection module is used to perform a first collection task, wherein the first collection task is to collect log data of a full storage device, wherein the full storage device includes the one or more storage devices in each of the multiple servers; The determination module is used to determine the health of each storage device in the full storage device according to the log data collected by the first collection task; The determination module is further configured to determine a first type of storage device from the full amount of storage devices according to the health level of each storage device, the health level of the first type of storage device being lower than a first health value; The collection module is further used to execute a second collection task, where the second collection task is to collect log data from the first type of storage device; The determination module is further configured to determine a faulty storage device in the first type of storage device based on the log data collected by the second collection task.

9. The device according to claim 8, characterized in that The device also includes an alarm module; The determination module is further configured to determine a second type of storage device from the full storage devices according to the health level of each storage device, the health level of the second type of storage device being less than a second health value, the second health value being less than the first health value, and the first type of storage device being a storage device whose health level is between the second health value and the first health value; The alarm module is used to generate an alarm for the second type storage device.

10. The device according to claim 8 or 9, characterized in that The determination module is specifically used to input the log data collected by the first collection task into the health algorithm model to obtain the confidence level corresponding to each storage device, and the confidence level is used to indicate the health level of the storage device.

11. The device according to any one of claims 8 to 10, characterized in that The first acquisition task occupies a first data acquisition link, and the second acquisition task occupies a second data acquisition link, and the data is suppressed when the first data acquisition link acquires data; The collection module is further used to perform a third collection task in the second data collection link, where the third collection task is to collect log data of the full storage device; The determination module is further used to execute a first inspection task, wherein the first inspection task is to inspect the log data collected by the third collection task to determine a storage device with a fault in the full storage device; The acquisition module is specifically configured to trigger the operation of pausing the third acquisition task and executing the second acquisition task when the first type storage device is determined; The acquisition module is further configured to continue executing the third acquisition task when the second acquisition task is completed.

12. The device according to claim 11, characterized in that The determination module is specifically configured to trigger, when the second collection task is completed, the operation of suspending the first inspection task and executing the second inspection task, wherein the second inspection task is to inspect the log data collected by the second collection task to determine the storage device with fault in the first type of storage device; The determining module is further configured to continue executing the first inspection task when the second inspection task is completed.

13. The device according to claim 12, characterized in that The determination module is further configured to determine a third type of storage device from the full amount of storage devices according to the health level of each storage device, the health level of the third type of storage device being lower than a third health value, and the third health value being greater than the first health value; The collection module is further used to select the log data of the third type of storage device from the log data collected by the first collection task; The determination module is further configured to input the log data of the third type storage device into a fault classification prediction algorithm model to determine a faulty storage device in the third type storage device.

14. The device according to any one of claims 8 to 13, characterized in that The storage device is a memory stick or a disk.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method as claimed in any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 7.

Images