A training method of a fault diagnosis model and a fault diagnosis method of a storage system

Abstract

The embodiment of the application provides a kind of training method of fault diagnosis model and the fault diagnosis method of storage system, it is related to fault detection technical field, the training method of fault diagnosis model includes: obtaining the sample log generated in the time period when sample fault occurs in sample storage system;First prediction information is obtained by processing sample log using general diagnostic large language model in fault diagnosis model;Based on the first prediction information and the first reference information obtained, the general diagnostic large language model is model tuned, and the general diagnostic large language model trained is obtained;Second prediction information is obtained by processing sample log using expert diagnostic large language model corresponding to the actual fault category of sample fault in fault diagnosis model;Second prediction information and the second reference information are used to model tuning expert diagnostic large language model, and the expert diagnostic large language model trained is obtained.The efficiency of storage system fault diagnosis can be improved by the present scheme.

Description

Technical Field

[0001] This application relates to the field of fault detection technology, and in particular to a training method for a fault diagnosis model and a fault diagnosis method for a storage system. Background Technology

[0002] Current storage systems often use flash memory as the storage medium to store data. These systems face complex failure risks, including physical media degradation, communication link anomalies, firmware anomalies, and power supply interference. These problems can lead to data loss, performance degradation, or even system crashes.

[0003] To ensure data security and system reliability, fault diagnosis of storage systems is necessary to determine the type and cause of faults, enabling timely remediation of defects. However, manual fault diagnosis is limited by the technical skills and experience of personnel, making it difficult and inefficient, especially in large-scale deployment scenarios (such as data centers), where manual diagnosis of storage system faults is challenging and inaccurate.

[0004] Therefore, improving the efficiency of fault diagnosis in storage systems is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method for training a fault diagnosis model and a method for diagnosing faults in a storage system, so as to improve the efficiency of fault diagnosis in storage systems. The specific technical solution is as follows:

[0006] This application provides a method for training a fault diagnosis model, the method comprising:

[0007] Obtain the logs generated during the period when the sample storage system experienced a sample failure, and use them as sample logs;

[0008] The sample logs are processed using a general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; wherein, the first prediction information represents the reasoning process by which the general diagnostic language model uses the sample logs to determine the fault category to which the fault in the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0009] Based on the obtained first prediction information and first reference information, the model parameters of the general diagnostic language model to be trained are tuned to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using the sample log.

[0010] The sample log is processed using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information; wherein, the obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample log to determine the cause of the failure of the sample storage system;

[0011] Based on the obtained second prediction information and second reference information, the expert diagnostic large language model is tuned to obtain a trained expert diagnostic large language model; wherein, the second reference information represents the actual reasoning process of using the sample log to determine the actual cause of the sample failure.

[0012] In one embodiment, the logs generated by the sample storage system during the time period in which the sample failure occurs include: log logs, SMART logs at multiple times during the time period, and core dump logs at multiple times during the time period.

[0013] The sample logs are processed using the general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information, including:

[0014] Based on the differences between the core dump logs at every two adjacent moments in the sample log, and / or the differences between the SMART logs at every two adjacent moments, sample event information representing the events generated by the sample storage system during the time period in which the sample failure occurred is determined.

[0015] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the first sample combined logs;

[0016] The sample event information and the first sample combined log are input into the general diagnostic language model of the initial structure in the fault diagnosis model to obtain the first prediction information.

[0017] And / or,

[0018] The sample logs are processed using the expert diagnostic large language model to be trained, corresponding to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information, including:

[0019] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the second sample combined logs;

[0020] The second sample combined log is input into the expert diagnostic large language model to be trained in the fault diagnosis model to obtain the second prediction information.

[0021] In one embodiment, before processing the sample logs using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information, the method further includes:

[0022] Acquire natural language samples; wherein the natural language samples contain text in the field of storage system technology;

[0023] The initial structure of the large language model is pre-trained using the natural language samples to obtain the general diagnostic large language model to be trained and the expert diagnostic large language model to be trained in the fault diagnosis model.

[0024] In one embodiment, the step of tuning the parameters of the general diagnostic language model to be trained based on the obtained first prediction information and first reference information to obtain the trained general diagnostic language model includes:

[0025] Calculate a model score that represents the similarity between the obtained first predicted information and the first reference information;

[0026] Based on the obtained model score, the model parameters of the general diagnostic large language model to be trained are tuned until the obtained model score reaches the maximum value, thus obtaining the trained general diagnostic large language model.

[0027] And / or,

[0028] Based on the obtained second prediction information and second reference information, the expert diagnosis large language model is subjected to parameter tuning to obtain a trained expert diagnosis large language model, including:

[0029] Calculate a model score that represents the degree of similarity between the obtained second predicted information and the second reference information;

[0030] Based on the obtained model score, the parameter tuning of the expert diagnostic large language model is carried out until the obtained model score reaches the maximum value, thus obtaining the trained expert diagnostic large language model.

[0031] In one embodiment, the sample storage system is a flash-based storage system; the fault categories include at least one of the following categories:

[0032] Physical medium abnormality, communication abnormality, firmware abnormality, and power supply interference.

[0033] This application embodiment also provides a fault diagnosis method for a storage system, the method comprising:

[0034] Obtain the logs generated by the storage system to be tested within the testing period, and use them as logs to be processed;

[0035] The log to be processed is processed using a general diagnostic language model in the fault diagnosis model to obtain a first diagnostic result; wherein, the first diagnostic result includes: the fault category to which the faults that occurred in the storage system to be tested during the time period to be tested belong; the fault diagnosis model also includes an expert diagnostic language model;

[0036] The expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model is used to process the log to be processed to obtain a second diagnostic result; wherein, the second diagnostic result includes: the reasons that caused the storage system to be tested to fail during the time period to be tested;

[0037] The fault diagnosis model is trained using the training method described above.

[0038] In one embodiment, the first diagnostic result further includes: the reasoning process by which the general diagnostic big language model determines the fault category;

[0039] And / or, the second diagnostic result also includes: the reasoning process by which the expert diagnostic big language model determines the cause of the failure.

[0040] In one embodiment, the logs generated by the storage system under test during the test period include: log logs, SMART logs at multiple times during the test period, and core dump logs at multiple times during the test period.

[0041] The process of using a general diagnostic language model in the fault diagnosis model to process the log to be processed yields a first diagnostic result, including:

[0042] Based on the differences between the core dump logs at every two adjacent moments in the log to be processed, and / or the differences between the SMART logs at every two adjacent moments, the event information to be processed representing the events generated by the storage system to be tested during the time period to be tested is determined.

[0043] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the first combined log to be processed.

[0044] The event information to be processed and the combined log to be processed are input into the general diagnostic language model in the fault diagnosis model to obtain the first diagnostic result.

[0045] And / or,

[0046] Using the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, the log to be processed is processed to obtain a second diagnostic result, including:

[0047] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the second combined log to be processed.

[0048] The second combined log to be processed is input into the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model to obtain the second diagnostic result.

[0049] This application embodiment also provides a training device for a fault diagnosis model, the device comprising:

[0050] The sample log acquisition module is used to acquire logs generated during the period when a sample failure occurs in the sample storage system, and use them as sample logs.

[0051] The first processing module is used to process the sample logs using a general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; wherein, the obtained first prediction information represents: the reasoning process by which the general diagnostic language model uses the sample logs to determine the fault category to which the fault occurring in the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0052] The first parameter tuning module is used to perform model parameter tuning on the general diagnostic language model to be trained based on the obtained first prediction information and first reference information, so as to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using the sample log.

[0053] The second processing module is used to process the sample log using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information; wherein, the obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample log to determine the cause of the failure of the sample storage system.

[0054] The second parameter tuning module is used to perform parameter tuning on the expert diagnostic large language model based on the obtained second prediction information and second reference information to obtain the trained expert diagnostic large language model; wherein, the second reference information represents the actual reasoning process of using the sample log to determine the actual cause of the sample failure.

[0055] In one embodiment, the logs generated by the sample storage system during the time period in which the sample failure occurs include: log logs, SMART logs at multiple times during the time period, and core dump logs at multiple times during the time period.

[0056] The first processing module includes:

[0057] The sample event determination submodule is used to determine sample event information representing events generated by the sample storage system during the time period in which the sample failure occurred, based on the differences between the core dump logs at every two adjacent moments in the sample log and / or the differences between the SMART logs at every two adjacent moments.

[0058] The first sample combination submodule is used to sort and combine the acquired log logs and core dump logs in chronological order to obtain the first sample combined log.

[0059] The first input submodule is used to input the sample event information and the first sample combination log into the general diagnostic language model of the initial structure in the fault diagnosis model to obtain the first prediction information.

[0060] And / or,

[0061] The second processing module includes:

[0062] The second sample combination submodule is used to sort and combine the acquired log logs and core dump logs in chronological order to obtain the second sample combined log.

[0063] The second input submodule is used to input the second sample combination log into the expert diagnostic large language model to be trained in the fault diagnosis model to obtain the second prediction information.

[0064] In one embodiment, the apparatus further includes:

[0065] The natural language sample acquisition module is used to acquire natural language samples before the first processing module processes the sample logs using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information; wherein, the natural language samples contain text in the field of storage system technology.

[0066] The pre-training module is used to pre-train the large language model of the initial structure using the natural language samples to obtain the general diagnostic large language model to be trained and the expert diagnostic large language model to be trained in the fault diagnosis model.

[0067] In one embodiment, the first parameter tuning module includes:

[0068] The first score calculation submodule is used to calculate the model score, which represents the degree of similarity between the obtained first predicted information and the first reference information.

[0069] The first parameter tuning submodule is used to tune the parameters of the general diagnostic language model to be trained based on the obtained model score until the obtained model score reaches the maximum value, thus obtaining the trained general diagnostic language model.

[0070] And / or,

[0071] The second parameter tuning module includes:

[0072] The second score calculation submodule is used to calculate the model score, which represents the degree of similarity between the obtained second prediction information and the second reference information.

[0073] The second parameter tuning submodule is used to tune the parameters of the expert diagnostic large language model based on the obtained model score until the obtained model score reaches the maximum value, thus obtaining the trained expert diagnostic large language model.

[0074] In one embodiment, the sample storage system is a flash-based storage system; the fault categories include at least one of the following categories:

[0075] Physical medium abnormality, communication abnormality, firmware abnormality, and power supply interference.

[0076] This application embodiment also provides a fault diagnosis device for a storage system, the device comprising:

[0077] The pending log acquisition module is used to acquire the logs generated by the storage system to be tested within the testing time period, and use them as pending logs.

[0078] The first diagnostic module is used to process the log to be processed using a general diagnostic language model in the fault diagnosis model to obtain a first diagnostic result; wherein, the first diagnostic result includes: the fault category to which the fault that occurred in the storage system to be tested during the time period to be tested belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0079] The second diagnostic module is used to process the log to be processed using the expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, and to obtain a second diagnostic result; wherein, the second diagnostic result includes: the reasons that caused the storage system to be tested to fail during the time period to be tested;

[0080] The fault diagnosis model is trained using the training method described above.

[0081] In one embodiment, the first diagnostic result further includes: the reasoning process by which the general diagnostic big language model determines the fault category; and / or, the second diagnostic result further includes: the reasoning process by which the expert diagnostic big language model determines the cause of the fault.

[0082] In one embodiment, the logs generated by the storage system under test during the test period include: log logs, SMART logs at multiple times during the test period, and core dump logs at multiple times during the test period.

[0083] The first diagnostic module includes:

[0084] The event information determination submodule is used to determine the event information to be processed, representing the events generated by the storage system to be detected during the time period, based on the differences between the core dump logs at every two adjacent moments in the log to be processed, and / or the differences between the SMART logs at every two adjacent moments.

[0085] The first combination submodule is used to sort and combine the acquired logs and core dump logs in chronological order to obtain the first combined log to be processed.

[0086] The first diagnostic submodule is used to input the event information to be processed and the combined log to be processed into the general diagnostic language model in the fault diagnosis model to obtain the first diagnostic result.

[0087] And / or,

[0088] The second diagnostic module includes:

[0089] The second combination submodule is used to sort and combine the acquired logs and core dump logs in chronological order to obtain the second combined log to be processed.

[0090] The second diagnostic submodule is used to input the second combined log to be processed into the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, and obtain the second diagnostic result.

[0091] This application also provides an electronic device, including:

[0092] Memory, used to store computer programs;

[0093] The processor, when executing a program stored in memory, implements the training method for the aforementioned fault diagnosis model or the fault diagnosis method for the storage system.

[0094] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described training method for the fault diagnosis model or the fault diagnosis method for the storage system.

[0095] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the training method of any of the fault diagnosis models described above or the fault diagnosis method of the storage system.

[0096] Beneficial effects of the embodiments in this application:

[0097] The fault diagnosis method for a storage system provided in this application obtains logs generated during the time period when a sample fault occurs in the sample storage system as sample logs; processes the sample logs using a general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; the first prediction information represents the reasoning process by which the general diagnostic language model uses the sample logs to determine the fault category to which the fault occurs in the sample storage system; finally, based on the obtained first prediction information and first reference information, the general diagnostic language model to be trained is tuned to obtain a trained general diagnostic language model; in addition, this solution also processes the sample logs using an expert diagnostic language model to be trained corresponding to the actual fault category of the sample fault in the fault diagnosis model to obtain second prediction information; the second prediction information represents the reasoning process by which the expert diagnostic language model uses the sample logs to determine the cause of the fault in the sample storage system; and then, based on the obtained second prediction information and second reference information, the expert diagnostic language model is tuned to obtain a trained expert diagnostic language model.

[0098] In this scheme, through the above training process, the general diagnostic language model can learn the correlation between the storage system's logs and the fault categories to which the faults belong; the expert diagnostic language model corresponding to a fault category can learn the correlation between the storage system's logs and the causes of faults belonging to that fault category. Subsequently, by using the trained general diagnostic language model and expert diagnostic language model, the fault category and cause of the faults occurring in the storage system can be diagnosed, without the need for manual fault diagnosis of the storage system. Therefore, this scheme can improve the efficiency of fault diagnosis of the storage system.

[0099] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0100] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0101] Figure 1 This is a schematic diagram of the first process for training a fault diagnosis model provided in an embodiment of this application;

[0102] Figure 2 This is a schematic diagram of the time distribution of the three types of logs in the embodiments of this application;

[0103] Figure 3 This is a schematic diagram of a second process for training a fault diagnosis model provided in an embodiment of this application;

[0104] Figure 4 This is a schematic diagram of the structure of the large language model in the embodiments of this application;

[0105] Figure 5 A flowchart illustrating the fault diagnosis method for a storage system provided in an embodiment of this application;

[0106] Figure 6 This is a schematic diagram of the overall process of an embodiment of this application;

[0107] Figure 7 A schematic diagram of the structure of the training device for the fault diagnosis model provided in the embodiments of this application;

[0108] Figure 8 This is a schematic diagram of the structure of the fault diagnosis device for the storage system provided in the embodiments of this application;

[0109] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0110] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0111] To improve the efficiency of fault diagnosis in storage systems, embodiments of this application provide a method for training a fault diagnosis model and a method for diagnosing faults in storage systems; both methods can be applied to electronic devices capable of data processing. The method for training the fault diagnosis model includes the following steps:

[0112] Obtain the logs generated during the period when the sample storage system experienced a sample failure, and use them as sample logs;

[0113] The sample logs are processed using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information; wherein, the first prediction information obtained represents: the reasoning process of the general diagnostic language model using the sample logs to determine the fault category to which the fault of the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0114] Based on the obtained first prediction information and first reference information, the model parameters of the general diagnostic language model to be trained are tuned to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using sample logs;

[0115] The sample logs are processed using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain the second prediction information. The obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample logs to determine the cause of the failure in the sample storage system.

[0116] Based on the obtained second prediction information and second reference information, the model parameters of the expert diagnosis big language model are tuned to obtain the trained expert diagnosis big language model; wherein, the second reference information represents the actual reasoning process of using sample logs to determine the actual cause of sample failure.

[0117] In this embodiment, through the above training process, the general diagnostic language model can learn the correlation between the storage system's logs and the fault categories to which the faults belong; the expert diagnostic language model corresponding to a fault category can learn the correlation between the storage system's logs and the causes of faults belonging to that fault category. Subsequently, by using the trained general diagnostic language model and expert diagnostic language model, the fault category and cause of the faults occurring in the storage system can be diagnosed, without the need for manual fault diagnosis of the storage system. Therefore, this solution can improve the efficiency of fault diagnosis of the storage system.

[0118] The training method of the fault diagnosis model provided in the embodiments of this application is described below with reference to the accompanying drawings. Figure 1 As shown, the method includes the following steps:

[0119] S101, Obtain the logs generated during the time period when the sample storage system experienced a sample failure, and use them as sample logs;

[0120] The sample storage system can be any storage system that has experienced a failure. In this embodiment, the sample storage system can refer to a flash-based storage system, that is, the sample storage system includes one or more solid-state drives (SSDs) using flash memory as the storage medium. The type of flash memory can be NAND (Not AND) flash memory. Alternatively, the sample storage system can also be a storage system based on a hard disk drive (HDD), etc.

[0121] During the operation of a storage system, the system records its operating status through logs. The main types of logs include: Log logs, SMART (Self-Monitoring, Analysis, and Reporting Technology) logs, and coredump logs.

[0122] The Log records the working process of the storage system in a time-stream manner. For example, it records each read, write, and erase operation of each block in the storage system, as well as the corresponding execution time. In addition, it can also record the execution time and results of each firmware operation in the storage system.

[0123] The SMART log records statistical information about the storage system at a specific moment, such as the total amount of writes and the total number of data errors. SMART log generation can be triggered by specific events, such as the storage system generating a SMART log each time it boots up, or when the storage system receives a SMART log generation command, which can be issued by technicians when troubleshooting is required.

[0124] The core dump log is generated by the firmware and can record the hardware status of the storage system periodically or when the system crashes. This includes information such as the storage system's memory status, flash memory status, register status, stack pointer, and power supply voltage.

[0125] The distribution of generation times for the above three types of logs can be as follows: Figure 2 As shown, the generation frequency of Log logs, core dump logs, and SMART logs decreases sequentially.

[0126] Sample logs may include at least one of the three types of logs mentioned above.

[0127] The time period during which a sample failure occurs in the sample storage system can include a specified time period before the failure and a specified time period after the failure. In practice, all logs and core dump logs within the time period during which the sample failure occurs can be obtained, along with the SMART logs from an adjacent time point before and after the failure, which will be used as the sample logs.

[0128] The sample faults can be faults detected in advance for the sample storage system. There can be multiple sample faults, specifically including sample faults of various fault categories, and each fault category can include sample faults with multiple causes. Therefore, in this embodiment, steps S101-S105 for training the fault diagnosis model can be performed separately for each sample fault.

[0129] In practice, a sufficient number of data samples can be collected and made to follow a certain paradigm to facilitate the learning and understanding of large language models. For example, a fault sample can be represented by a data structure with the following paradigm:

[0130] Data Sample: [Smart Information, Log, Core Dump Log, Fault Information, Inference Process]; where fault information includes the fault category of the sample fault and the cause of the sample fault. The inference process includes the inference process for the fault category and the inference process for the cause of the fault. In this embodiment, the fault category can be regarded as a general category of fault, while the cause of the fault can be regarded as a specific category of fault. For example, the fault category output by the general diagnostic large language model can be: NAND anomaly, and the cause of the fault can be: NAND word line failure. Since NAND word line failure is a cause of NAND anomaly, NAND word line failure can also be regarded as a specific category of NAND anomaly. The cause of the fault can also be specified to the location of the defect, such as which storage block in the storage system has failed, or which algorithm in the firmware has an anomaly, etc.

[0131] The dataset obtained after collecting N samples in this way can be denoted as:

[0132] Data_Set = {Data_Sample} i ;i=1,...,N}.

[0133] S102, the sample logs are processed using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information;

[0134] The first prediction information obtained represents the reasoning process by which the general diagnostic language model uses sample logs to determine the fault category to which the faults occurring in the sample storage system belong; the fault diagnosis model also includes an expert diagnostic language model.

[0135] Understandably, the fault category is determined during the reasoning process, and therefore the first predicted information includes the determined fault category. Of course, the general diagnostic language model can also additionally output information representing the fault category separately.

[0136] A fault diagnosis model can include a general diagnostic language model and multiple expert diagnostic language models. Each expert diagnostic language model corresponds to a specific fault category, meaning each model infers the cause of a fault only within that category. For example, pre-collected sample data can be categorized into N fault categories. After training the general diagnostic language model using this sample data, the model can identify these N fault categories. Therefore, a fault diagnosis model can contain a maximum of N different expert diagnostic language models. Of course, a fault diagnosis model can also contain only one expert diagnostic language model.

[0137] The diagnostic language models to be trained in the fault diagnosis model (general diagnostic language model and expert diagnostic language model) can be any pre-trained language model with natural language understanding capabilities. These language models are typically deep learning models based on the Transformer architecture, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer). Compared to traditional machine learning or LSTM (Long Short-Term Memory) based deep learning methods for fault diagnosis, Transformer-based language models can better understand and process long text sequences. Without needing to consider comprehensive feature extraction from logs, these models can perform fault diagnosis directly from logs, demonstrating strong adaptability and making them highly suitable for log analysis in solid-state storage systems. Therefore, they can significantly improve the efficiency and accuracy of fault diagnosis.

[0138] The diagnostic large language models to be trained can be the same large language model. Different training methods can be used to fine-tune the large language model to obtain different diagnostic large language models.

[0139] In this step, the three types of logs obtained above can be concatenated to form a text sequence. This text sequence can record information such as the state of the physical medium, firmware execution status, and power supply voltage, which can be effectively applied to fault diagnosis. The concatenated text is then input into the general diagnostic language model to be trained. Simultaneously, prompts instructing the general diagnostic language model to use logs for fault diagnosis can also be input to obtain the first predicted information output by the model. This first predicted information can be a thought chain representing the reasoning process; specifically, it can be in text form or a vectorized form of text.

[0140] The general diagnostic language model in this embodiment and the expert diagnostic language model mentioned later are actually both AI Agents (artificial intelligence agents). Therefore, the general diagnostic language model and the expert diagnostic language model can also be called general Agent and expert Agent, respectively.

[0141] Fault categories can include at least one of the following categories:

[0142] The first category is physical media anomalies: Physical media refers to the medium used to store data. In the case of NAND flash memory, physical media anomalies can include NAND anomalies. Since NAND typically uses SRAM (Static Random-Access Memory) or DRAM (Dynamic Random Access Memory) as cache, physical media anomalies can also include SRAM or DRAM anomalies. Due to the complex physical and electrical characteristics of NAND, factors such as PE (Program / Erase), DR (Data Retention), RD (Read Disturbance), and CT (Changes of Temperature) can alter the data reliability of NAND storage cells, leading to physical media anomalies. This typically manifests as an increase in the number of bit flips, and in severe cases (such as excessive bit flips exceeding the data error correction capability of the storage system), data errors can occur. SRAM and DRAM are also affected by electrical interference, temperature changes, cosmic rays, etc., causing bit flips, which can also lead to data errors in severe cases.

[0143] The second category is communication anomalies: the communication links between the solid-state drive and the host, and between the flash memory controller and the NAND, may become abnormal due to interference, resulting in data transmission errors and thus causing system problems.

[0144] The third category is firmware anomalies: When there are logical flaws in the algorithms within the firmware, it can lead to storage system anomalies under certain circumstances. These firmware algorithms are the ones the storage system relies on to process data. For example, a logical flaw in the IO (Input / Output) flow control algorithm may result in insufficient free blocks in the storage system. A logical flaw in the wear-leveling algorithm may cause some storage blocks to be excessively worn down.

[0145] The fourth category is power supply interference: power supply interference can cause electronic components in the storage system to deviate from their normal operating state. For example, flash memory controllers and NAND flash memory may produce incorrect outputs when the voltage is abnormal, which means they may output incorrect data.

[0146] S103, Based on the obtained first prediction information and first reference information, the model parameters of the general diagnostic language model to be trained are tuned to obtain the trained general diagnostic language model.

[0147] The first reference information refers to the actual reasoning process of determining the actual fault category of a sample fault using sample logs.

[0148] The first reference information can be set by technicians for sample faults to represent the reasoning process by which technicians determine the fault based on the sample logs; it can be in text form. For example, the first reference information in one scenario might include the following:

[0149] ① A program fail occurred on block1, indicating that the NAND flash memory has failed; ② The PE (write / erase cycles) of this block is 4000, which exceeds the nominal maximum lifespan (e.g., 3500), indicating that the NAND flash memory has been overused, and the program fail is a normal phenomenon; ③ Checking the status of other blocks, we found that PE ∈ [3300, 3400], indicating that the other blocks are experiencing normal wear and tear, and only this block is abnormal. Therefore, the fault category is most likely firmware abnormality.

[0150] The general diagnostic language model to be trained is then tuned, i.e., fine-tuned. This may include the following steps:

[0151] Step A1: Calculate the model score, which represents the degree of similarity between the obtained first predicted information and the first reference information;

[0152] This embodiment can use reinforcement learning training methods to tune the parameters of the general diagnostic large language model. The first predicted information output by the general diagnostic large language model can be a chain-of-thought representing the reasoning process. This chain-of-thought can contain multiple steps, so the similarity between each step and the first reference information can be calculated as the score of that step. For example, the sine distance or cosine distance can be calculated, and then the sum of the scores of each step is used as the model score.

[0153] In one implementation, an Actor-Critic architecture can be used to train the general diagnostic language model. Here, "Actor" refers to the general diagnostic language model, and "Critic" is the scoring model, hereinafter referred to as the first scoring model. The first scoring model can be pre-trained and can score each step of the thought chain output by the general diagnostic language model. The higher the similarity between each step output by the general diagnostic language model and the first reference information, the higher the sum of the scores for each step; the lower the similarity, the lower the sum of the scores for each step. The model score of the general diagnostic language model is the sum of the scores given by the first scoring model for each step output by the general diagnostic language model.

[0154] The first scoring model can be trained using supervised learning. For example, multiple training samples can be pre-set for training the first scoring model. Each training sample includes a reasoning process for inferring fault categories using the logs of the storage system and its corresponding score. Technicians can evaluate the accuracy of each reasoning process, obtaining a score representing its accuracy; that is, the higher the accuracy, the higher the score for that reasoning process. In this way, by training the first scoring model using the pre-set training samples, the first scoring model can learn how to score various reasoning processes.

[0155] Step A2: Based on the obtained model score, perform model parameter tuning on the general diagnostic language model to be trained, and return to execute the step of using the general diagnostic language model to be trained in the fault diagnosis model to process the sample logs and obtain the first prediction information, until the obtained model score reaches the maximum value, and the trained general diagnostic language model is obtained.

[0156] The general diagnostic language model is tuned with the goal of maximizing its score. After each tuning, the model is used again to process the sample logs, and the score is recalculated. After multiple rounds of tuning, the general diagnostic language model will learn the reasoning process for the fault category of the sample, and thus be able to determine the fault category based on the logs.

[0157] S104. The sample logs are processed using the expert diagnostic language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain the second prediction information.

[0158] The second prediction information obtained represents the reasoning process by which the expert diagnostic language model uses sample logs to determine the cause of the failure in the sample storage system.

[0159] The process of training the expert diagnostic language model in steps S104-S105 and the process of training the general diagnostic language model in steps S102-S103 do not necessarily have a specific order.

[0160] Understandably, the reasoning process for determining the cause of a sample failure inherently includes the identified cause of the failure. Furthermore, the expert diagnostic large language model can also additionally output information indicating the cause of the failure separately.

[0161] Because solid-state drives (SSDs) contain multiple complex modules, they exhibit a wide variety of fault types. Using a single model for fault diagnosis would require the model to identify various issues, including NAND faults, communication faults, and firmware faults, placing very high demands on the model. Given a fixed model architecture, improving accuracy often necessitates increasing model parameters or enhancing training data quality, leading to higher training costs. Therefore, to achieve better fault diagnosis results while maintaining cost control, this embodiment employs a multi-model architecture (i.e., a general diagnostic language model combined with expert diagnostic language models). The general diagnostic language model determines the approximate fault category, while each expert diagnostic language model is responsible for a single domain to identify the specific fault cause. For example, expert diagnostic language model 1 analyzes NAND anomalies, expert diagnostic language model 2 analyzes communication anomalies, and expert diagnostic language model 3 analyzes firmware anomalies, etc. In this way, the general diagnostic language model can quickly identify the category, and then the expert diagnostic models can accurately identify the fault cause, balancing diagnostic efficiency and accuracy.

[0162] S105, Based on the obtained second prediction information and second reference information, the model parameters of the expert diagnosis big language model are tuned to obtain the trained expert diagnosis big language model.

[0163] The second reference information refers to the actual reasoning process of using sample logs to determine the actual cause of sample failure.

[0164] The second reference information can also be set by technicians for sample faults. For example, when the actual fault category has been determined to be a firmware anomaly, the second reference information in one scenario may include the following:

[0165] ① Upon checking the write / erase process, it was found that this block had been frequently worn down recently, while the wear of other blocks was even, indicating that the wear leveling algorithm had failed; ② Upon checking the data status of this block, it was found that after each data was written to this block, it was quickly updated by the user, and its effective data dropped to 0 before it was reclaimed; ③ Upon checking the wear leveling algorithm, it was found that during the implementation of the algorithm, blocks with 0 effective data were directly given the highest reclamation priority, without considering the wear status of the block. Therefore, it was determined that the cause of the failure was the failure of the wear leveling algorithm.

[0166] Understandably, current large language models already possess the ability to understand algorithm code. Therefore, the code of each algorithm in the firmware can be pre-input into the expert diagnostic large language model used for analyzing firmware anomalies. When the expert diagnostic large language model determines that an algorithm needs to be checked, its code can then be examined.

[0167] This step involves fine-tuning the expert diagnostic large language model. This process is similar to step S103 above, and may also include the following steps:

[0168] Step B1: Calculate the model score, which represents the degree of similarity between the obtained second predicted information and the second reference information;

[0169] Similarly, an Actor-Critic architecture can be used to train this expert diagnostic language model. Here, "Actor" refers to the expert diagnostic language model, and "Critic" is the scoring model, hereinafter referred to as the second scoring model for clarity. The second scoring model can be pre-trained and can score each step of the thought chain output by the expert diagnostic language model. The higher the similarity between each step output by the expert diagnostic language model and the second reference information, the higher the sum of the scores for each step; conversely, the lower the similarity, the lower the sum of the scores. The model score of the expert diagnostic language model is the sum of the scores given by the second scoring model to each step output by the expert diagnostic language model.

[0170] The second scoring model can also be trained using supervised learning. For example, multiple training samples can be pre-set for training the second scoring model. Each training sample includes a reasoning process that uses the storage system's logs to infer the cause of a failure and its corresponding score. Technicians can evaluate the accuracy of each reasoning process, obtaining a score representing its accuracy; that is, the higher the accuracy, the higher the score for that reasoning process. By training the second scoring model using these pre-set training samples, the model can ultimately learn how to score various reasoning processes.

[0171] Step B2: Based on the obtained model score, perform model parameter tuning on the expert diagnostic big language model, and return to execute the step of using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to process the sample logs until the obtained model score reaches the maximum value, thus obtaining the trained expert diagnostic big language model.

[0172] The optimization objective is to maximize the model score of the expert diagnostic language model. After each round of parameter tuning, the model is used again to process the sample logs, and the model score is recalculated. After multiple rounds of parameter tuning, the expert diagnostic language model will be able to learn the reasoning process for the cause of the fault in the sample.

[0173] In this embodiment, through the above training process, the general diagnostic language model can learn the correlation between the storage system's logs and the fault categories to which the faults belong; the expert diagnostic language model corresponding to a fault category can learn the correlation between the storage system's logs and the causes of faults belonging to that fault category. Subsequently, by using the trained general diagnostic language model and expert diagnostic language model, the fault category and cause of the faults occurring in the storage system can be diagnosed, without the need for manual fault diagnosis of the storage system. Therefore, this solution can improve the efficiency of fault diagnosis of the storage system.

[0174] Furthermore, this embodiment uses a multi-model, phased processing architecture to sequentially determine the fault category and the cause of the fault, which can improve the accuracy of fault diagnosis while reducing the training cost of the model.

[0175] In one embodiment of this application, as Figure 3 As shown, before processing the sample logs using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information in step S102 above, the method may further include the following steps:

[0176] S301, Obtain natural language samples;

[0177] The natural language samples include text related to the field of storage system technology.

[0178] Understandably, steps S301-S302 can be executed before step S101 above, which retrieves the logs generated during the time period when the sample storage system experienced a sample failure. Natural language samples can be obtained from books or web pages related to the technology of storage systems, and may include user manuals, explanations of storage system principles, background information, and future prospects of storage systems.

[0179] S302, using natural language samples to pre-train the large language model of the initial structure, to obtain the general diagnostic large language model to be trained and the expert diagnostic large language model to be trained in the fault diagnosis model.

[0180] In one implementation method, the large language model in this embodiment can be as follows: Figure 4 The structure shown includes:

[0181] Tokenization layers: These are used to convert input text into quantifiable numbers; the input text is obtained based on SMART logs, log logs, and core dump logs.

[0182] Word embedding layers: These are used to convert the aforementioned quantized numbers into vectors. For example, they can be 768-dimensional vectors.

[0183] Transformer layer: Used to transform the input, it is the core of the large language model. This embodiment can use 16 Transformer layers, i.e. Figure 4 Transformer layers 1 to 16 are used. Of course, the number of Transformer layers can also be other values, but this embodiment does not limit the specific number.

[0184] Output layer: Used to output the final result, which may include the reasoning process and fault categories.

[0185] This embodiment does not limit the specific pre-training method. For example, it can be pre-trained using a masked language model (MLM).

[0186] In this embodiment, by pre-training the initial large language model using text from the storage system technology field, the large language model can learn natural language understanding capabilities while simultaneously understanding the natural language of the storage system technology field. This allows the pre-trained large language model to be used as both a general diagnostic large language model and an expert diagnostic large language model to be trained, further improving the efficiency of subsequent training of these models and enhancing the accuracy of fault diagnosis using them.

[0187] In one embodiment of this application, the logs generated by the sample storage system during the time period in which a sample failure occurs include: log logs, SMART logs at multiple times during the time period, and core dump logs at multiple times during the time period.

[0188] The above process of the sample logs using the general diagnostic language model to be trained in the fault diagnosis model yields the first prediction information, including:

[0189] Step C1: Based on the differences between the core dump logs at every two adjacent moments in the sample log, and / or the differences between the SMART logs at every two adjacent moments, determine the sample event information representing the events generated by the sample storage system during the period in which the sample failure occurred.

[0190] The events generated by a storage system within a certain time period can be reflected by the changes in the state of the storage system within that time period. Therefore, the event information of the events generated by the storage system within a certain time period can specifically be information representing the changes in the state of the storage system within that time period.

[0191] Since both the core dump log and the SMART log record the instantaneous state of the storage system, by comparing the differences between the core dump logs at two adjacent moments and / or the differences between the SMART logs at two adjacent moments, we can determine the state changes of the storage system between these two moments, and thus determine the events that occurred within those two moments. By summarizing the events at two adjacent moments, we can obtain the events that occurred in the storage system throughout the entire time period.

[0192] For example, if the SMART log at the previous moment records: NAND: program fail cnt = 0 (0 write failures); and the SMART log at the next moment records: NAND: program fail cnt = 1 (1 write failure), then we can determine that the event generated by the storage system is: a write failure occurred in NAND. In practice, a simple program can be written to determine the events generated by the storage system. For example, the correspondence between log differences and events can be predefined. Accordingly, after determining the differences between logs, the event that occurred can be determined based on this correspondence.

[0193] When the physical medium of the storage system is NAND, the events that can be determined based on the differences between the core dump logs at two adjacent moments, and / or the differences between the SMART logs at two adjacent moments, may include, but are not limited to, the following:

[0194] (1) NAND function abnormalities, including: NAND read failure, NAND write failure, NAND erase failure, etc.

[0195] (2) SRAM or DRAM malfunction, including SRAM or DRAM read failure, etc.

[0196] (3) Transmission link anomalies, including: transmission anomalies between the main controller (i.e., flash controller) and NAND, such as DDR (Double Data Rate) transmission anomalies; and interface transmission anomalies between the hard drive and the host, such as SATA (Serial ATA) or PCIe (peripheral component interconnect express) transmission anomalies.

[0197] (4) Firmware anomalies, including: insufficient free blocks, wear leveling failure, null pointer reference, etc.

[0198] (5) Power supply abnormality, including: voltage exceeding the limit, etc.

[0199] Step C2: Sort and combine the acquired logs and core dump logs in chronological order to obtain the first sample combined logs;

[0200] This involves combining the acquired log files and core dump logs into a text sequence in chronological order. Before combining the log files and core dump logs, preprocessing can be performed to streamline the sample log combination and improve the efficiency of subsequent model training and inference. For example, irrelevant data or outliers can be removed from the log files and core dump logs. Specific preprocessing methods can be set according to actual conditions and requirements. Alternatively, in one implementation, the acquired log files, SMART logs, and core dump logs can be sorted and combined in chronological order to obtain the first sample combined log.

[0201] Step C3: Input the sample event information and the first sample combined log into the general diagnostic language model of the initial structure in the fault diagnosis model to obtain the first prediction information;

[0202] In other words, the general diagnostic big language model in this embodiment actually processes the sample event information and the first sample combination log obtained from the sample log.

[0203] By first determining the sample event information representing the events generated by the sample storage system during the time period when the sample failure occurred, based on the differences between the core dump logs at every two adjacent moments in the sample logs, and / or the differences between the SMART logs at every two adjacent moments, we can initially extract information useful for fault diagnosis from the sample logs. By sorting and combining the acquired logs and core dump logs in chronological order, it is easier for the diagnostic large language model to analyze the logs in chronological order. Therefore, this processing can improve information utilization and thus improve the efficiency of model training and inference.

[0204] The above-mentioned fault diagnosis model uses the expert diagnostic large language model to be trained, corresponding to the actual fault category of the sample fault, to process the sample logs and obtain the second prediction information, including:

[0205] Step D1: Sort and combine the acquired logs and core dump logs in chronological order to obtain the second sample combined logs;

[0206] For the same sample failure, the first sample combined log and the second sample combined log can contain the same content.

[0207] Similarly, the acquired logs, SMART logs, and core dump logs can be sorted and combined in chronological order to obtain a second sample combined log.

[0208] Step D2: Input the second sample combined log into the expert diagnostic large language model to be trained in the fault diagnosis model to obtain the second prediction information.

[0209] In other words, in this embodiment, the expert diagnostic big language model actually processes the second sample combination log obtained based on the sample log.

[0210] Since the events generated by the storage system are helpful in determining the fault category, and the information contained in the log and core dump log is sufficient to analyze the specific cause, in order to improve the efficiency of model training and inference, the log obtained by combining the log and core dump log can be input into the expert diagnostic big language model to obtain the second prediction information output by the expert diagnostic big language model.

[0211] In this embodiment, based on the differences between the core dump logs at every two adjacent time points in the sample log, and / or the differences between the SMART logs at every two adjacent time points, sample event information representing events generated by the sample storage system during the time period in which the sample failure occurred is determined. The acquired log logs and core dump logs are sorted and combined in chronological order to obtain the first sample combined log, which can improve the efficiency of training a general diagnostic language model. The acquired log logs and core dump logs are sorted and combined in chronological order to obtain the second sample combined log, which can improve the efficiency of training an expert diagnostic language model.

[0212] This application also provides a method for fault diagnosis of a storage system, such as... Figure 5 As shown, the method includes:

[0213] S501: Obtain the logs generated by the storage system to be tested within the testing period, and use them as logs to be processed.

[0214] The storage system under test can be a flash-based storage system or a hard disk drive-based storage system, etc. The steps in this embodiment can be executed periodically to periodically check whether the storage system has failed. Alternatively, the steps in this embodiment can be executed when the storage system under test experiences an anomaly, such as when the storage system under test crashes, to facilitate timely fault repair of the storage system under test.

[0215] The logs generated by the storage system under test during the test period may include: Log logs, SMART logs, and core dump logs generated by the storage system under test during the test period.

[0216] S502 uses the general diagnostic language model in the fault diagnosis model to process the log to be processed and obtain the first diagnostic result.

[0217] The first diagnostic result includes: the fault category of the fault that occurred in the storage system under test during the test period; the fault diagnosis model also includes an expert diagnosis language model.

[0218] Fault categories may include at least one of the following: physical medium abnormality, communication abnormality, firmware abnormality, and power interference.

[0219] S503 uses the expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model to process the log to be processed and obtain the second diagnostic result.

[0220] The second diagnostic result includes: the reasons why the storage system under test failed during the test period.

[0221] In this application embodiment, the fault diagnosis model can be the fault diagnosis model trained in the above-mentioned fault diagnosis model training method, specifically it can be trained based on the above-mentioned fault diagnosis model training method.

[0222] In one implementation, the three types of logs obtained can be concatenated to form a text sequence, which is then input into the general diagnostic language model and the corresponding expert diagnostic language model. Specifically, the general diagnostic language model processes the logs to be processed in a manner similar to that used in step S102 above; the expert diagnostic language model processes the logs to be processed in a manner similar to that used in step S104 above, and will not be elaborated further here.

[0223] The identified cause of the failure can be used to inform technicians, who can then repair the storage system under test based on the cause of the failure.

[0224] In one implementation, the general diagnostic language model can output only the fault category, while the expert diagnostic language model can output only the cause of the fault. In another implementation, the first diagnostic result may further include the reasoning process by which the general diagnostic language model determines the fault category, and the second diagnostic result may further include the reasoning process by which the expert diagnostic language model determines the cause of the fault. Outputting the reasoning process for determining the fault category and cause facilitates subsequent fault repair of the storage system. The reasoning process can be a thought process chain.

[0225] In this embodiment, through the above training process, the general diagnostic language model can learn the correlation between the storage system's logs and the fault categories to which the faults belong; the expert diagnostic language model corresponding to a fault category can learn the correlation between the storage system's logs and the causes of faults belonging to that fault category. Subsequently, by using the trained general diagnostic language model and expert diagnostic language model, the fault category and cause of the faults occurring in the storage system under test can be diagnosed based on the logs generated by the storage system under test during the test period, without the need for manual fault diagnosis of the storage system under test. Therefore, this solution can improve the efficiency of fault diagnosis of storage systems.

[0226] Similarly, in one embodiment, the logs generated by the storage system under test during the test period include: log logs, SMART logs at multiple times during the test period, and core dump logs at multiple times during the test period.

[0227] The log to be processed is obtained by using the general diagnostic language model in the fault diagnosis model, including:

[0228] Based on the differences between the core dump logs at every two adjacent moments in the log to be processed, and / or the differences between the SMART logs at every two adjacent moments, determine the event information to be processed that represents the events generated by the storage system to be tested during the time period to be tested.

[0229] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the first combined log to be processed.

[0230] Of course, in one implementation, the acquired log, SMART log, and core dump log can also be sorted and combined in chronological order to obtain the first combined log to be processed.

[0231] Input the event information to be processed and the combined log to be processed into the general diagnostic language model in the fault diagnosis model to obtain the first diagnostic result;

[0232] And / or,

[0233] Using the expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, the log to be processed is processed to obtain the second diagnostic result, including:

[0234] The acquired logs and core dump logs are sorted and combined in chronological order to obtain the second combined log to be processed.

[0235] The second combined log to be processed and the first combined log to be processed can be the same data.

[0236] Similarly, the acquired logs, SMART logs, and core dump logs can be sorted and combined in chronological order to obtain a second set of combined logs to be processed.

[0237] The second combined log to be processed is input into the expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model to obtain the second diagnostic result.

[0238] The overall process of this application embodiment can be described as follows: Figure 6 As shown, it includes two stages: model training and fault diagnosis.

[0239] During the model training phase, natural language samples 10 are first acquired for large language model pre-training (S601), that is, the initial structure of the large language model is pre-trained using natural language samples to obtain the general diagnostic large language model and the expert diagnostic large language model to be trained in the fault diagnosis model. Then, fault samples 20 are acquired for large language model fine-tuning (S602) to obtain the trained fault diagnosis model 30, including the trained general agent 301 (i.e., the general diagnostic large language model) and the trained multiple expert agents 302 (i.e., the expert diagnostic large language model).

[0240] During the fault diagnosis phase, the SMART log 40, log log 50, and core dump log 60 generated by the storage system under test during the test period are first acquired. The general agent 301 in the fault diagnosis model 30 processes the SMART log 40, log log 50, and core dump log 60 to determine the fault category of the storage system under test. Then, an expert agent 302 is selected from the fault diagnosis model 30 based on the fault category; that is, the expert diagnostic language model corresponding to the determined fault category is selected. This expert agent 302 is then used to determine the cause of the fault in the storage system under test. Afterwards, defect repair can be performed based on the determined cause of the fault (S603).

[0241] For fault diagnosis methods, steps similar to those for training fault diagnosis models can be found in the relevant descriptions of the training method embodiments above.

[0242] This application also provides a training device for a fault diagnosis model, such as... Figure 7 As shown, the device includes:

[0243] The sample log acquisition module 701 is used to acquire logs generated during the time period when a sample failure occurs in the sample storage system, and use them as sample logs.

[0244] The first processing module 702 is used to process the sample log using the general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; wherein, the obtained first prediction information represents: the reasoning process of the general diagnostic language model using the sample log to determine the fault category to which the fault occurring in the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0245] The first parameter tuning module 703 is used to perform model parameter tuning on the general diagnostic language model to be trained based on the obtained first prediction information and first reference information, so as to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using the sample log.

[0246] The second processing module 704 is used to process the sample log using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information; wherein, the obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample log to determine the cause of the failure of the sample storage system.

[0247] The second parameter tuning module 705 is used to perform parameter tuning on the expert diagnosis big language model based on the obtained second prediction information and second reference information to obtain the trained expert diagnosis big language model; wherein, the second reference information represents the actual reasoning process of using the sample log to determine the actual cause of the sample failure.

[0248] In one embodiment, the logs generated by the sample storage system during the time period in which the sample failure occurs include: log logs, SMART logs at multiple times during the time period, and core dump logs at multiple times during the time period.

[0249] The first processing module 702 includes:

[0250] The sample event determination submodule is used to determine sample event information representing events generated by the sample storage system during the time period in which the sample failure occurred, based on the differences between the core dump logs at every two adjacent moments in the sample log and / or the differences between the SMART logs at every two adjacent moments.

[0251] The first sample combination submodule is used to sort and combine the acquired log logs and core dump logs in chronological order to obtain the first sample combined log.

[0252] The first input submodule is used to input the sample event information and the first sample combination log into the general diagnostic language model of the initial structure in the fault diagnosis model to obtain the first prediction information.

[0253] In one embodiment, the second processing module 704 includes:

[0254] The second sample combination submodule is used to sort and combine the acquired log logs and core dump logs in chronological order to obtain the second sample combined log.

[0255] The second input submodule is used to input the second sample combination log into the expert diagnostic large language model to be trained in the fault diagnosis model to obtain the second prediction information.

[0256] In one embodiment, the apparatus further includes:

[0257] The natural language sample acquisition module is used to acquire natural language samples before the first processing module 702 processes the sample logs using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information; wherein, the natural language samples contain text in the field of storage system technology.

[0258] The pre-training module is used to pre-train the large language model of the initial structure using the natural language samples to obtain the general diagnostic large language model to be trained and the expert diagnostic large language model to be trained in the fault diagnosis model.

[0259] In one embodiment, the first parameter tuning module 703 includes:

[0260] The first score calculation submodule is used to calculate the model score, which represents the degree of similarity between the obtained first predicted information and the first reference information.

[0261] The first parameter tuning submodule is used to tune the parameters of the general diagnostic language model to be trained based on the obtained model score until the obtained model score reaches the maximum value, thus obtaining the trained general diagnostic language model.

[0262] In one embodiment, the second parameter tuning module 705 includes:

[0263] The second score calculation submodule is used to calculate the model score, which represents the degree of similarity between the obtained second prediction information and the second reference information.

[0264] The second parameter tuning submodule is used to tune the parameters of the expert diagnostic large language model based on the obtained model score until the obtained model score reaches the maximum value, thus obtaining the trained expert diagnostic large language model.

[0265] In one embodiment, the sample storage system is a flash-based storage system; the fault categories include at least one of the following categories:

[0266] Physical medium abnormality, communication abnormality, firmware abnormality, and power supply interference.

[0267] This application also provides a fault diagnosis device for a storage system, such as... Figure 8 As shown, the device includes:

[0268] The pending log acquisition module 801 is used to acquire the logs generated by the storage system to be tested within the testing time period, and use them as pending logs.

[0269] The first diagnostic module 802 is used to process the log to be processed using a general diagnostic language model in the fault diagnosis model to obtain a first diagnostic result; wherein, the first diagnostic result includes: the fault category to which the fault that occurred in the storage system to be tested during the time period to be tested belongs; the fault diagnosis model also includes an expert diagnostic language model;

[0270] The second diagnostic module 803 is used to process the log to be processed using the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, and obtain a second diagnostic result; wherein, the second diagnostic result includes: the reasons that caused the storage system to be tested to fail during the time period to be tested;

[0271] The fault diagnosis model is trained using the training method described above.

[0272] In one embodiment, the first diagnostic result further includes: the reasoning process by which the general diagnostic big language model determines the fault category; and / or, the second diagnostic result further includes: the reasoning process by which the expert diagnostic big language model determines the cause of the fault.

[0273] In one embodiment, the logs generated by the storage system under test during the test period include: log logs, SMART logs at multiple times during the test period, and core dump logs at multiple times during the test period.

[0274] The first diagnostic module 802 includes:

[0275] The event information determination submodule is used to determine the event information to be processed, representing the events generated by the storage system to be detected during the time period, based on the differences between the core dump logs at every two adjacent moments in the log to be processed, and / or the differences between the SMART logs at every two adjacent moments.

[0276] The first combination submodule is used to sort and combine the acquired logs and core dump logs in chronological order to obtain the first combined log to be processed.

[0277] The first diagnostic submodule is used to input the event information to be processed and the combined log to be processed into the general diagnostic language model in the fault diagnosis model to obtain the first diagnostic result.

[0278] In one embodiment, the second diagnostic module 803 includes:

[0279] The second combination submodule is used to sort and combine the acquired logs and core dump logs in chronological order to obtain the second combined log to be processed.

[0280] The second diagnostic submodule is used to input the second combined log to be processed into the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, and obtain the second diagnostic result.

[0281] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0282] Memory 901 is used to store computer programs;

[0283] When the processor 902 executes the program stored in the memory 901, it implements the steps of the above-mentioned fault diagnosis model training method or the fault diagnosis method of the storage system.

[0284] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0285] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0286] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0287] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0288] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0289] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the training method for any of the above-described fault diagnosis models or the fault diagnosis method for the storage system.

[0290] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training method of any of the fault diagnosis models in the above embodiments or the fault diagnosis method of the storage system.

[0291] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0292] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0293] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0294] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A training method for a fault diagnosis model, characterized in that, The method includes: Obtain the logs generated during the period when the sample storage system experienced a sample failure, and use them as sample logs; The sample logs are processed using a general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; wherein, the first prediction information represents the reasoning process by which the general diagnostic language model uses the sample logs to determine the fault category to which the fault in the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model; Based on the obtained first prediction information and first reference information, the model parameters of the general diagnostic language model to be trained are tuned to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using the sample log. The sample log is processed using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information; wherein, the obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample log to determine the cause of the failure of the sample storage system; Based on the obtained second prediction information and second reference information, the expert diagnostic large language model is tuned to obtain a trained expert diagnostic large language model; wherein, the second reference information represents the actual reasoning process of using the sample log to determine the actual cause of the sample failure.

2. The method according to claim 1, characterized in that, The logs generated by the sample storage system during the time period in which the sample failure occurred include: log logs, SMART logs at multiple times during that time period, and core dump logs at multiple times during that time period. The sample logs are processed using the general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information, including: Based on the differences between the core dump logs at every two adjacent moments in the sample log, and / or the differences between the SMART logs at every two adjacent moments, sample event information representing the events generated by the sample storage system during the time period in which the sample failure occurred is determined. The acquired logs and core dump logs are sorted and combined in chronological order to obtain the first sample combined logs; The sample event information and the first sample combined log are input into the general diagnostic language model of the initial structure in the fault diagnosis model to obtain the first prediction information. And / or, The sample logs are processed using the expert diagnostic large language model to be trained, corresponding to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information, including: The acquired logs and core dump logs are sorted and combined in chronological order to obtain the second sample combined logs; The second sample combined log is input into the expert diagnostic large language model to be trained in the fault diagnosis model to obtain the second prediction information.

3. The method according to claim 1, characterized in that, Before processing the sample logs using the general diagnostic language model to be trained in the fault diagnosis model to obtain the first prediction information, the method further includes: Acquire natural language samples; wherein the natural language samples contain text in the field of storage system technology; The initial structure of the large language model is pre-trained using the natural language samples to obtain the general diagnostic large language model to be trained and the expert diagnostic large language model to be trained in the fault diagnosis model.

4. The method according to claim 1, characterized in that, Based on the obtained first prediction information and first reference information, the general diagnostic language model to be trained is subjected to parameter tuning to obtain a trained general diagnostic language model, including: Calculate a model score that represents the similarity between the obtained first predicted information and the first reference information; Based on the obtained model score, the model parameters of the general diagnostic large language model to be trained are tuned until the obtained model score reaches the maximum value, thus obtaining the trained general diagnostic large language model. And / or, Based on the obtained second prediction information and second reference information, the expert diagnosis large language model is subjected to parameter tuning to obtain a trained expert diagnosis large language model, including: Calculate a model score that represents the degree of similarity between the obtained second predicted information and the second reference information; Based on the obtained model score, the parameter tuning of the expert diagnostic large language model is carried out until the obtained model score reaches the maximum value, thus obtaining the trained expert diagnostic large language model.

5. The method according to any one of claims 1-4, characterized in that, The sample storage system is a flash memory-based storage system; the fault categories include at least one of the following categories: Physical medium abnormality, communication abnormality, firmware abnormality, and power supply interference.

6. A fault diagnosis method for a storage system, characterized in that, The method includes: Obtain the logs generated by the storage system to be tested within the testing period, and use them as logs to be processed; The log to be processed is processed using a general diagnostic language model in the fault diagnosis model to obtain a first diagnostic result; wherein, the first diagnostic result includes: the fault category to which the faults that occurred in the storage system to be tested during the time period to be tested belong; the fault diagnosis model also includes an expert diagnostic language model; The expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model is used to process the log to be processed to obtain a second diagnostic result; wherein, the second diagnostic result includes: the reasons that caused the storage system to be tested to fail during the time period to be tested; The fault diagnosis model is trained based on the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that, The first diagnostic result also includes: the reasoning process by which the general diagnostic big language model determines the fault category; And / or, the second diagnostic result also includes: the reasoning process by which the expert diagnostic big language model determines the cause of the failure.

8. The method according to claim 6, characterized in that, The logs generated by the storage system under test during the test period include: log logs, SMART logs at multiple times during the test period, and core dump logs at multiple times during the test period. The process of using a general diagnostic language model in the fault diagnosis model to process the log to be processed yields a first diagnostic result, including: Based on the differences between the core dump logs at every two adjacent moments in the log to be processed, and / or the differences between the SMART logs at every two adjacent moments, the event information to be processed representing the events generated by the storage system to be tested during the time period to be tested is determined. The acquired logs and core dump logs are sorted and combined in chronological order to obtain the first combined log to be processed. The event information to be processed and the combined log to be processed are input into the general diagnostic language model in the fault diagnosis model to obtain the first diagnostic result. And / or, Using the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, the log to be processed is processed to obtain a second diagnostic result, including: The acquired logs and core dump logs are sorted and combined in chronological order to obtain the second combined log to be processed. The second combined log to be processed is input into the expert diagnostic big language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model to obtain the second diagnostic result.

9. A training device for a fault diagnosis model, characterized in that, The device includes: The sample log acquisition module is used to acquire logs generated during the period when a sample failure occurs in the sample storage system, and use them as sample logs. The first processing module is used to process the sample logs using a general diagnostic language model to be trained in the fault diagnosis model to obtain first prediction information; wherein, the obtained first prediction information represents: the reasoning process by which the general diagnostic language model uses the sample logs to determine the fault category to which the fault occurring in the sample storage system belongs; the fault diagnosis model also includes an expert diagnostic language model; The first parameter tuning module is used to perform model parameter tuning on the general diagnostic language model to be trained based on the obtained first prediction information and first reference information, so as to obtain the trained general diagnostic language model; wherein, the first reference information represents the actual reasoning process of determining the actual fault category of the sample fault using the sample log. The second processing module is used to process the sample log using the expert diagnostic big language model to be trained, which corresponds to the actual fault category of the sample fault in the fault diagnosis model, to obtain second prediction information; wherein, the obtained second prediction information represents the reasoning process by which the expert diagnostic big language model uses the sample log to determine the cause of the failure of the sample storage system. The second parameter tuning module is used to perform parameter tuning on the expert diagnostic large language model based on the obtained second prediction information and second reference information to obtain the trained expert diagnostic large language model; wherein, the second reference information represents the actual reasoning process of using the sample log to determine the actual cause of the sample failure.

10. A fault diagnosis device for a storage system, characterized in that, The device includes: The pending log acquisition module is used to acquire the logs generated by the storage system to be tested within the testing time period, and use them as pending logs. The first diagnostic module is used to process the log to be processed using a general diagnostic language model in the fault diagnosis model to obtain a first diagnostic result; wherein, the first diagnostic result includes: the fault category to which the fault that occurred in the storage system to be tested during the time period to be tested belongs; the fault diagnosis model also includes an expert diagnostic language model; The second diagnostic module is used to process the log to be processed using the expert diagnostic language model corresponding to the fault category represented by the first diagnostic result in the fault diagnosis model, and to obtain a second diagnostic result; wherein, the second diagnostic result includes: the reasons that caused the storage system to be tested to fail during the time period to be tested; The fault diagnosis model is trained based on the method described in any one of claims 1-5.

11. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-5 or any one of claims 6-8.

Images