A device fault processing method and device based on bcache, equipment and storage medium

Abstract

The application discloses a kind of based on bcache's equipment failure processing method, device, equipment and storage medium, it is related to computer technical field, including: the preset high-speed storage device in government affair cloud platform storage architecture is determined as cache device, and the preset low-speed storage device is determined as back-end device, when the first target error appears in cache device, the multi-dimensional index based on the first hardware health index and software state index of cache device is constructed, and whether the fault type of cache device is target fault type is judged using preset state machine;If it is target fault type, stop the read-write operation of cache device;If the second target error appears in back-end device, the second hardware health index and error log of back-end device are based on, and health state result is determined using preset AI model, and preventive treatment and fault handling are carried out using health state result. Identify a variety of bcache's fault scene and distinguish processing fault type.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a device fault processing method and device based on bcache, apparatus and storage medium. BACKGROUND

[0002] Bcache (block layer cache) is a block device caching mechanism provided by Linux kernel. Due to the hierarchical design of bcache, a data access may generate multiple read and write operations, and under a large amount of read and write pressure, IO failure caused by software logic and hardware failure is inevitable. When a fault occurs when bcache accesses a cache disk, all dirty data will be synchronized back to the back-end hard disk, and when the synchronization is completed and the cache disk group is not used, the bcache will automatically unload the faulty cache disk; the back-end device is switched to a cache-free mode for continuous operation, thereby improving the availability of bcache, and providing basic IO services in the case of cache disk failure, although the performance will be significantly degraded. When the reference count cannot be cleared within a certain time, it means that the dirty data is not completely flushed or the device is still in use, in which case the bcache device will be disabled, and the problem will be exposed to the user for processing, at this time the access to the bcache will fail. However, only relying on the reference count to determine whether to automatically unload will cause premature unloading and loss of dirty data, or IO stuck and unable to unload but no stop device to cause business accumulation; moreover, the existing fault processing scheme adopts unified processing logic and does not classify faults; in particular, the integrity of the dirty data flushing process is not checked, and the partial flushing failure or data damage scenario cannot be identified.

[0003] From the above, how to identify various bcache fault scenarios and distinguish the fault types is a problem to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a device fault processing method and device based on bcache, apparatus and storage medium, which can identify various bcache fault scenarios and distinguish the fault types. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a device fault processing method based on bcache, comprising:

[0006] The preset high-speed storage device in the government affair cloud platform storage architecture is determined as a cache device, and the preset low-speed storage device in the government affair cloud platform storage architecture is determined as a backend device; when a first target error occurs in the cache device, a multi-dimensional index is constructed based on a first hardware health index and a software state index of the cache device; whether the failure type of the cache device is a target failure type is determined based on the multi-dimensional index and by using a preset state machine; the target failure type is an unrecoverable failure type; and the government affair cloud platform storage architecture is a storage architecture integrated with bcache.

[0007] If the failure type of the cache device is the target failure type, the read-write operation of the cache device is stopped, and the cache device is marked as a device to be repaired.

[0008] If a second target error occurs in the backend device, a health state result is determined based on a second hardware health index and an error log of the backend device and by using a preset AI model, and corresponding preventive processing operation and failure processing operation are performed by using the health state result.

[0009] Optionally, the preset high-speed storage device in the government affair cloud platform storage architecture is determined as a cache device, and the preset low-speed storage device in the government affair cloud platform storage architecture is determined as a backend device; when a first target error occurs in the cache device, a multi-dimensional index is constructed based on a first hardware health index and a software state index of the cache device, including:

[0010] The preset high-speed storage device in the government affair cloud platform storage architecture is determined as a cache device, and the preset low-speed storage device in the government affair cloud platform storage architecture is determined as a backend device; the preset high-speed storage device is a solid state disk; and the preset low-speed storage device is a mechanical hard disk.

[0011] When a first target error occurs in the cache device, a multi-dimensional index is constructed based on a first hardware health index and a software state index of the cache device; the first target error includes a data insertion error, a read-write index error, a write-in first target block error, a read dirty data error, a data cleaning error and a data migration error; the first target block is a region of the cache device used for caching metadata; the dirty data is data that has been written into the cache device but has not been synchronized to the backend device; and the dirty data includes the metadata and business data.

[0012] Optionally, when a first target error occurs in the cache device, a multi-dimensional index is constructed based on a first hardware health index and a software state index of the cache device, whether the failure type of the cache device is a target failure type is determined based on the multi-dimensional index and by using a preset state machine, including:

[0013] acquiring, based on a first SMART technology of the cache device, a first hardware health index including a wear level, a number of damaged storage units and an IO response time when the cache device has a first target error;

[0014] determining, based on performance data of the cache device in an IO operation, a software state index including a read / write failure frequency and a throughput fluctuation;

[0015] constructing a multi-dimensional index based on the first hardware health index and the software state index, and determining, based on the multi-dimensional index and by using a finite state automaton, whether a failure type of the cache device is a target failure type.

[0016] Optionally, after determining, based on the multi-dimensional index and by using a preset state machine, whether the failure type of the cache device is the target failure type, the method further includes:

[0017] if the failure type of the cache device is not the target failure type, performing a flush operation on dirty data based on an exponential backoff retry strategy and a target timeout threshold, and reducing an IO performance of the cache device during the flush operation.

[0018] Optionally, if the backend device has a second target error, determining a health state result based on a second hardware health index and an error log of the backend device and by using a preset AI model, includes:

[0019] if the backend device has the second target error, determining a second hardware health index based on a second SMART technology of the backend device; the second target error includes a data synchronization error, a write error to a second target block, a cache read error, a cache miss error and a data discard error; the second target block is a region of the backend device used for caching the metadata;

[0020] determining an error log by using a distribution of error types, an error occurrence frequency and an error recovery success rate of the backend device in a preset time period, and determining a health score based on the second hardware health index and the error log and by using a preset AI model.

[0021] Optionally, the method further includes:

[0022] determining whether the health score exceeds a target score threshold;

[0023] if the health score exceeds the target score threshold, indicating that the backend device has a potential failure trend, performing a flush operation on the dirty data based on a preset flush time, and then reducing an IO performance of the backend device.

[0024] Optionally, after judging whether the health score exceeds the target score threshold, the method further comprises:

[0025] If the health score does not exceed the target score threshold, it is indicated that the backend device has a serious failure trend, the IO request for the backend device is stopped, a corresponding device fault code is returned, and the data in the backend device is migrated to a backup device.

[0026] In a second aspect, the present application provides a device fault processing apparatus based on bcache, comprising:

[0027] A fault type judgment module is configured to determine a preset high-speed storage device in a government cloud platform storage architecture as a cache device, determine a preset low-speed storage device in the government cloud platform storage architecture as a backend device, when a first target error occurs in the cache device, construct a multi-dimensional index based on a first hardware health index and a software state index of the cache device, and judge whether the fault type of the cache device is a target fault type based on the multi-dimensional index and by using a preset state machine; the target fault type is an unrecoverable fault type; the government cloud platform storage architecture is a storage architecture integrated with bcache;

[0028] A device marking module is configured to, if the fault type of the cache device is the target fault type, stop read and write operations of the cache device, and mark the cache device as a device to be repaired.

[0029] A fault processing module is configured to, if a second target error occurs in the backend device, determine a health state result based on a second hardware health index and an error log of the backend device and by using a preset AI model, and perform corresponding preventive processing operations and fault processing operations by using the health state result.

[0030] In a third aspect, the present application provides an electronic device, comprising:

[0031] A memory is configured to save a computer program.

[0032] A processor is configured to execute the computer program to implement the foregoing device fault processing method based on bcache.

[0033] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, wherein the computer program is executed by a processor to implement the foregoing device fault processing method based on bcache.

[0034] The preset high-speed storage device in the government affair cloud platform storage architecture is determined as a cache device, and the preset low-speed storage device in the government affair cloud platform storage architecture is determined as a backend device. When a first target error occurs in the cache device, a multi-dimensional index is constructed based on a first hardware health index and a software state index of the cache device, and whether the failure type of the cache device is a target failure type is determined based on the multi-dimensional index and by using a preset state machine. The target failure type is an unrecoverable failure type. The government affair cloud platform storage architecture is a storage architecture integrated with bcache. If the failure type of the cache device is the target failure type, the read-write operation of the cache device is stopped, and the cache device is marked as a device to be repaired. When a second target error occurs in the backend device, a health state result is determined based on a second hardware health index and an error log of the backend device and by using a preset AI model, and corresponding preventive processing operation and failure processing operation are performed based on the health state result.

[0035] As can be seen from the above, when the cache device in the government affair cloud platform storage architecture has an unrecoverable failure, the application can immediately stop the service, prevent the failure from spreading, and prevent damaged data read from the failed device from causing business exceptions. By using the multi-dimensional index and the preset state machine, it can be determined whether the failure of the cache device is an unrecoverable failure, and the failure type is classified and processed. When the backend device has a potential risk, the health state result of the backend device is determined based on the second hardware health index and the error log and by using the AI model. Based on the health state result, the backend device can be subjected to failure prevention processing to avoid data loss caused by sudden failure. If the health state result indicates that the backend device has failed, failure processing operation is performed. In this way, whether the cache device or the backend device, if a failure occurs, the targeted processing of the failure can be performed in a timely manner, and automatic unloading and other situations can be avoided to ensure data security. For the backend device failure, preventive processing can also be performed to reduce business pressure, and the frequency of active repair of sudden failures can be effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor based on the provided drawings.

[0037] Figure 1 A flowchart of a device failure processing method based on bcache is disclosed in the present application.

[0038] Figure 2A specific bcache-based device failure processing method flowchart disclosed by the present application;

[0039] Figure 3 A cache device error code processing diagram provided by the present application;

[0040] Figure 4 A traditional bcache exception processing diagram provided by the present application;

[0041] Figure 5 A bcache-based device failure processing device structure diagram disclosed by the present application;

[0042] Figure 6 An electronic device structure diagram disclosed by the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] At present, when a failure occurs when bcache accesses a cache disk, all dirty data will be first synchronized back to the backend hard disk, and when the synchronization is completed and the cache disk group is not used, the bcache will automatically unload the failed cache disk; the backend device is switched to a cache-free mode to continue running, thereby improving the availability of the bcache, and providing basic IO services in the case of cache disk failure, although the performance will be significantly degraded. When the reference count cannot be cleared within a certain time, it indicates that the dirty data is not completely flushed or the device is still in use, in which case the bcache device will be disabled, and the problem will be exposed to the user for processing, at this time, the access to the bcache will fail. Relying only on the reference count to determine whether to automatically unload will cause premature unloading and loss of dirty data, or IO stuck and unable to unload, but no stop device to cause business accumulation; moreover, the existing failure processing scheme adopts unified processing logic and does not classify failures; in particular, the completeness of the dirty data flushing process is not checked, and the partial flushing failure or data damage scenario cannot be identified. Therefore, the present application provides a bcache-based device failure processing method, which can prevent the backend device from failing based on the health status result, avoid data loss caused by sudden failure, and perform failure processing operation if the health status result indicates that the backend device has failed, so as to ensure that the business is not affected, and the frequency of active repair of sudden failure can be effectively reduced.

[0045] Reference Figure 1As shown in the figure, an embodiment of the present invention discloses a device fault handling method based on bcache, including:

[0046] Step S11: The preset high-speed storage device in the government cloud platform storage architecture is identified as a cache device, and the preset low-speed storage device in the government cloud platform storage architecture is identified as a backend device. When the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the first hardware health indicator and software status indicator of the cache device. Based on the multi-dimensional indicator and using a preset state machine, it is determined whether the fault type of the cache device is the target fault type. The target fault type is an unrecoverable fault type. The government cloud platform storage architecture is a storage architecture integrating bcache.

[0047] In this embodiment, a preset high-speed storage device in the government cloud platform storage architecture is designated as a cache device, and a preset low-speed storage device in the same architecture is designated as a backend device. When the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator. Specifically, the process involves designating a preset high-speed storage device as a cache device and a preset low-speed storage device as a backend device within the government cloud platform's storage architecture. When the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator. This includes: designating a preset high-speed storage device as a cache device and a preset low-speed storage device as a backend device; the preset high-speed storage device is a solid-state drive (SSD); the preset low-speed storage device is a hard disk drive (HDD); when the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator; the first target error includes data insertion error, read / write index error, write to the first target block error, read dirty data error, data cleanup error, and data migration error; the first target block is the area on the cache device used to cache metadata; the dirty data is data that has been written to the cache device but not synchronized to the backend device; the dirty data includes the metadata and business data.

[0048] It can be understood that when the first target error occurs in the cache device, the first hardware health indicators including wear level, number of damaged storage units and IO response time are obtained based on the first SMART technology (Self-Monitoring Analysis and Reporting Technology) of the cache device, then the performance data of the cache device for IO operation is counted, and the software state indicators including read / write failure frequency and throughput fluctuation are determined based on the counting result, and the multi-dimensional indicators are constructed by using the first hardware health indicators and the software state indicators. Specifically, when the first target error occurs in the cache device, the first hardware health indicators and the software state indicators of the cache device are used to construct multi-dimensional indicators, whether the fault type of the cache device is the target fault type is judged based on the multi-dimensional indicators and by using the preset state machine, including: when the first target error occurs in the cache device, the first hardware health indicators including wear level, number of damaged storage units and IO response time are obtained based on the first SMART technology of the cache device; the software state indicators including read / write failure frequency and throughput fluctuation are determined based on the performance data of the cache device for IO operation; the multi-dimensional indicators are constructed based on the first hardware health indicators and the software state indicators, and whether the fault type of the cache device is the target fault type is judged based on the multi-dimensional indicators and by using the finite state automaton.

[0049] In this embodiment, whether the fault type of the cache device is a target fault type that cannot be recovered is determined based on the multi-dimensional index and using a finite state automaton. The target fault type that cannot be recovered includes physical damage of a chip of the cache device. In a specific implementation, if the number of damaged storage units in the multi-dimensional index exceeds a target damage threshold and the read-write failure frequency is lower than a target frequency threshold, the fault type of the cache device is determined as the target fault type that cannot be recovered. If the fault type of the cache device is not the target fault type, the fault type of the cache device is represented as a temporary fault type that can be recovered, and a first attempt to flush dirty data is performed, a timeout time is set to 100 ms, if the first flushing fails, a second timeout time is set to 200 ms, if the second flushing fails, a third timeout time is set to 400 ms, the timeout time is exponentially increased until a target timeout threshold is reached, and during the flushing process, the cache service is not interrupted but the performance is reduced, for example, 100 IO requests are processed at the same time originally, and now only 50 IO requests are processed, the remaining resources are preferentially guaranteed for dirty data flushing, data safety is ensured, and business availability is maintained. Specifically, after determining whether the fault type of the cache device is the target fault type based on the multi-dimensional index and using the preset state machine, the method further includes: if the fault type of the cache device is not the target fault type, flushing dirty data based on an exponential backoff retry strategy and a target timeout threshold, and reducing IO performance of the cache device during the flushing process.

[0050] In step S12, if the fault type of the cache device is the target fault type, the read-write operation of the cache device is stopped, and the cache device is marked as a device to be repaired.

[0051] In this embodiment, if the fault type of the cache device is the target fault type, the read-write operation of the cache device is immediately stopped, all read-write requests for the cache device are stopped, and then the cache device is marked as a device to be repaired. The cache device is detected, and a corresponding device recovery strategy is determined based on the detection result.

[0052] In step S13, if the second target error occurs in the backend device, a health state result is determined based on the second hardware health index and the error log of the backend device and using a preset AI model, and corresponding preventive processing operations and fault processing operations are performed using the health state result.

[0053] In the embodiment, when the second target error occurs in the backend device, a second hardware health index is determined based on a second SMART technology of the backend device; the second target error includes a data synchronization error, a second target block writing error, a cache reading error, a cache miss error and a data discard error; the data synchronization error is an error occurring when dirty data is synchronized to the backend device; the second target block writing error is an error occurring when a target block is written to the backend device; the cache reading error is an error occurring when data is read from the cache device; the cache miss error includes an error occurring when data is read from the backend device; and the data discard error is an error occurring when the storage space of the backend device is recovered. An error log is determined based on the error type distribution, error occurrence frequency and error recovery success rate of the backend device within a preset time period, and a health trend of the backend device is analyzed based on the second hardware health index and the error log and by using a preset AI model to obtain a corresponding health score.

[0054] Specifically, when the second target error occurs in the backend device, a health state result is determined based on the second hardware health index and the error log of the backend device and by using a preset AI model, including: when the second target error occurs in the backend device, a second hardware health index is determined based on a second SMART technology of the backend device; the second target error includes a data synchronization error, a second target block writing error, a cache reading error, a cache miss error and a data discard error; the second target block is an area of the backend device used to cache the metadata; an error log is determined based on the error type distribution, error occurrence frequency and error recovery success rate of the backend device within a preset time period, and a health score is determined based on the second hardware health index and the error log and by using a preset AI model.

[0055] It can be understood that after obtaining the health score, it is judged whether the health score exceeds a target score threshold. If the health score exceeds the target score threshold and is less than a target potential threshold, it is indicated that the backend device has a potential failure trend and needs preventive treatment. For example, if the second hardware health index is high but not failed, the dirty data in the cache device is synchronized to the backend device in advance to reduce the amount of dirty data in subsequent failures, and then the cache parameters can be adjusted. For example, the data is synchronized when the cache is full to 50%, instead of 80%, to reduce the write pressure of the backend device. The target score threshold can be adjusted according to actual conditions, which is not limited here. Specifically, the health state result is used to perform corresponding preventive treatment operations and failure treatment operations, including: judging whether the health score exceeds a target score threshold; if the health score exceeds the target score threshold, it is indicated that the backend device has a potential failure trend, the dirty data is written based on a preset writing time, and then the IO performance of the backend device is reduced.

[0056] In this embodiment, if the health score does not exceed the target score threshold, it is indicated that the backend device has a serious failure trend, all IO requests for the backend device are blocked, and a corresponding device fault code is returned. The data in the backend device is migrated to a backup device through a distributed storage mechanism, and the original backend device is marked as a fault state after migration. In one specific embodiment, the health score is 0 to 100 points, of which 60 to 80 points are a potential failure trend, and less than 60 points are a serious failure trend. Specifically, after judging whether the health score exceeds the target score threshold, it also includes: if the health score does not exceed the target score threshold, it is indicated that the backend device has a serious failure trend, the IO request for the backend device is stopped, and a corresponding device fault code is returned. Then, the data in the backend device is migrated to a backup device.

[0057] As can be seen from the above, when the cache device in the government affair cloud platform storage architecture has an unrecoverable fault, the application will immediately stop service, prevent the fault from spreading, and prevent damaged data of the fault device from causing business exceptions; the multi-dimensional index and the preset state machine can accurately distinguish whether the fault of the cache device is an unrecoverable fault, and the fault type is classified and processed; when the backend device has a potential risk, the health state result of the backend device is determined based on the second hardware health index and the error log and by using an AI model, the backend device is prevented from fault based on the health state result, sudden faults are avoided to cause data loss, and if the health state result indicates that the backend device has a fault, a fault processing operation is performed. In this way, whether the cache device or the backend device, if a fault occurs, the fault can be processed in a targeted manner in a timely manner, and automatic unloading and the like do not occur, so as to ensure data security, the backend device fault can also be prevented, the business pressure is reduced, and the frequency of active repair of sudden faults can be effectively reduced.

[0058] As can be seen from the above embodiment, the application determines the fault type of the device based on the multi-dimensional index, and performs corresponding fault processing, so the process of determining the fault type of the device based on the multi-dimensional index is described.

[0059] Referring to Figure 2 As shown in the figure, the embodiment of the application discloses a specific device fault processing method based on bcache, which comprises the following steps:

[0060] In the embodiment, when the cache device has a first target error, the first SMART technology of the cache device is used to obtain a first hardware health index comprising wear degree, damaged storage unit quantity and IO response time, then the performance data of the cache device for IO operation is counted, and a software state index comprising read / write failure frequency, throughput fluctuation and cache index matching result is determined based on the counting result, and the first hardware health index and the software state index are used to construct a multi-dimensional index; the first target error comprises data insertion error, read / write index error, writing a first target block error, reading dirty data error, data cleaning error and data migration error; the first target block is a region of the cache device for caching metadata; the dirty data is data that has been written into the cache device but has not been synchronized to the backend device; the dirty data comprises the metadata and business data.

[0061] It can be understood that after obtaining the multi-dimensional index, it is determined whether the fault type of the cache device is a target fault type that cannot be recovered based on the multi-dimensional index and using a preset state machine. If the fault type of the cache device is the target fault type, the read and write operations of the cache device are immediately stopped, all read and write requests for the cache device are stopped, and then the cache device is marked as a device to be repaired, the cache device is detected, and a corresponding device recovery strategy is determined based on the detection result. Figure 3 A processing cache device error code schematic diagram provided for the embodiment is provided. For the bch_cache_set_error field in the cache device that appears an error, the original processing logic at the - mark is that the cache device is directly logged out, and the modified logic below is that the use of the device is first stopped, and the IO operation is disabled to avoid the loss of dirty data. If the fault type of the cache device is not the target fault type, it indicates that the fault type of the cache device is a temporary fault type that can be recovered, dirty data is flushed based on an exponential backoff retry strategy and a target timeout threshold, and the IO performance of the cache device is reduced during the flushing process.

[0062] Further, if a second target error occurs in the backend device, a second hardware health index is determined based on the second SMART technology of the backend device; the second target error includes a data synchronization error, a second target block write error, a cache read error, a cache miss error and a data discard error; the data synchronization error is an error that occurs when dirty data is synchronized to the backend device. An error log is determined using the error type distribution, error occurrence frequency and error recovery success rate of the backend device in a preset time period, the health trend of the backend device is predicted based on the second hardware health index and the error log and using a preset AI model to obtain a corresponding health score. Then it is determined whether the health score exceeds a target score threshold. If the health score exceeds the target score threshold and is less than a target potential threshold, it indicates that the backend device has a potential fault trend and needs preventive treatment, the dirty data in the cache device is synchronized to the backend device in advance, and the cache parameters are adjusted to reduce the write pressure of the backend device; if the health score does not exceed the target score threshold, it indicates that the backend device has a serious fault trend, the IO request for the backend device is stopped, and a corresponding device fault code is returned, and then the data in the backend device is migrated to a backup device.

[0063] In the embodiment, Figure 4A traditional bcache exception processing schematic diagram provided for the embodiment can be seen, the traditional method does not carry out fault classification, if the cache device fails, only through reference count is judged, if reference count can be zero, separates the cache device, if the reference count can not be zero, stops the bcache service.But the defect of traditional scheme is, if due to software defects and other reasons, lead to reference count zero in advance, but actually dirty data has not been completely synchronized to the backend device, the cache device is separated at this time and will cause dirty data loss, and the backend device error directly stops the bcache service, without any preventive measures, excessive sacrifice availability.The device fault processing scheme based on bcache proposed in the embodiment, when the cache device is not the target fault type, forces to flush dirty data, avoids the problem of dirty data loss caused by reference count misjudgment, and the cache service does not stop, only the performance is reduced, and when fault determination is carried out, the multi-dimensional index determined by the hardware health index and the software health index can comprehensively determine the fault of the device, so that the fault processing is more intelligent and accurate.And when the backend device fault is predicted, the device that will soon fail can be found in advance, and corresponding preventive treatment is carried out, and the business interruption risk is reduced.

[0064] As can be seen from the above, the present application carries out fault determination based on multi-dimensional index, which can reduce misjudgment, forces to flush when the cache device is not the target fault type, only reduces the IO performance, ensures data security, and maintains the continuity of government cloud platform business;If the cache device is a non-recoverable fault, the service is directly stopped to ensure data security;When the backend device fails, preventive treatment and fault handling can be carried out in advance to avoid the loss caused by directly stopping the service, complete data protection before the fault breaks out, and reduce the business interruption risk.

[0065] Correspondingly, referring to Figure 5 The present application also provides a device fault processing apparatus based on bcache, which comprises:

[0066] The fault type determination module 11 is used for determining a preset high-speed storage device in the government cloud platform storage architecture as a cache device, and determining a preset low-speed storage device in the government cloud platform storage architecture as a backend device, when the cache device appears a first target error, constructing a multi-dimensional index based on the first hardware health index and software state index of the cache device, and determining whether the fault type of the cache device is a target fault type based on the multi-dimensional index and using a preset state machine;The target fault type is a non-recoverable fault type;The government cloud platform storage architecture is a storage architecture integrated with bcache;

[0067] The device marking module 12 is configured to stop read-write operations of the cache device and mark the cache device as a device to be repaired if the fault type of the cache device is a target fault type.

[0068] The fault processing module 13 is configured to determine a health state result based on the second hardware health index and the error log of the backend device and by using a preset AI model when the second target error occurs in the backend device, and perform corresponding preventive processing operations and fault processing operations by using the health state result.

[0069] In some embodiments, the fault type judgment module 11 can specifically include:

[0070] The device determination unit is configured to determine a preset high-speed storage device in a government affair cloud platform storage architecture as a cache device, and determine a preset low-speed storage device in the government affair cloud platform storage architecture as a backend device.

[0071] The index construction unit is configured to construct a multi-dimensional index based on a first hardware health index and a software state index of the cache device when a first target error occurs in the cache device.

[0072] In some embodiments, the device marking module 12 can specifically include:

[0073] The first index acquisition unit is configured to acquire, based on a first SMART technology, a first hardware health index including a wear level, a number of damaged storage units, and an IO response time when a first target error occurs in the cache device.

[0074] The state index determination unit is configured to determine a software state index including a read-write failure frequency and a throughput fluctuation based on performance data of an IO operation of the cache device.

[0075] The type judgment unit is configured to construct a multi-dimensional index based on the first hardware health index and the software state index, and determine whether the fault type of the cache device is a target fault type based on the multi-dimensional index and by using a finite state automaton.

[0076] In some embodiments, the device fault processing apparatus based on bcache can further include:

[0077] The data flushing unit is configured to flush dirty data based on an exponential backoff retry strategy and a target timeout threshold, and reduce IO performance of the cache device during the flushing process if the fault type of the cache device is not the target fault type.

[0078] In some embodiments, the fault processing module 13 can specifically include:

[0079] a second index determination unit, configured to determine a second hardware health index based on a second SMART technology of the backend device if the second target error occurs in the backend device;

[0080] a health score determination unit, configured to determine an error log by using error type distribution, error occurrence frequency and error recovery success rate of the backend device in a preset time period, and determine a health score based on the second hardware health index and the error log and by using a preset AI model.

[0081] In some embodiments, the fault processing module 13 can specifically include:

[0082] a score judgment unit, configured to judge whether the health score exceeds a target score threshold;

[0083] a performance reduction unit, configured to, if the health score exceeds the target score threshold, represent that the backend device has a potential fault trend, perform a flash on the dirty data based on a preset flash time, and then reduce IO performance of the backend device.

[0084] In some embodiments, the device fault processing apparatus based on bcache can further specifically include:

[0085] a data migration unit, configured to, if the health score does not exceed the target score threshold, represent that the backend device has a serious fault trend, stop an IO request for the backend device, return a corresponding device fault code, and then migrate data in the backend device to a backup device.

[0086] Further, the embodiment of the present application further discloses an electronic device, Figure 6 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the device fault processing method based on bcache disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0087] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application requirements, which will not be specifically limited herein.

[0088] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0089] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the device fault processing method based on the bcache executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0090] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the device fault processing method based on the bcache disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0091] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0092] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0093] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0094] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply or require any such actual relationship or order. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0095] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described by applying specific examples; the above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A device fault handling method based on bcache, characterized in that, include: The preset high-speed storage device in the government cloud platform storage architecture is identified as a cache device, and the preset low-speed storage device in the government cloud platform storage architecture is identified as a back-end device. When the cache device encounters a first target error, a multi-dimensional indicator is constructed based on the first hardware health indicator and software status indicator of the cache device. Based on the multi-dimensional indicator and using a preset state machine, it is determined whether the fault type of the cache device is the target fault type. The target fault type is an unrecoverable fault type; the government cloud platform storage architecture is a storage architecture integrating bcache; If the fault type of the cache device is the target fault type, then stop the read and write operations of the cache device and mark the cache device as a device to be repaired; If the backend device encounters a second target error, the health status result is determined based on the second hardware health indicator and error log of the backend device and using a preset AI model. The corresponding preventive processing operation and fault handling operation are then performed using the health status result. Specifically, the process involves designating a preset high-speed storage device as a cache device and a preset low-speed storage device as a backend device within the government cloud platform's storage architecture. When a first target error occurs in the cache device, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator. This includes: designating a preset high-speed storage device as a cache device and a preset low-speed storage device as a backend device; the preset high-speed storage device is a solid-state drive (SSD); the preset low-speed storage device is a hard disk drive (HDD); when a first target error occurs in the cache device, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator; the first target error includes data insertion error, read / write index error, write to the first target block error, read dirty data error, data cleanup error, and data migration error; the first target block is the area on the cache device used to cache metadata; the dirty data is data that has been written to the cache device but not synchronized to the backend device; the dirty data includes the metadata and business data. The step of determining whether the fault type of the cache device is the target fault type based on the multi-dimensional indicators and using a preset state machine further includes: if the fault type of the cache device is not the target fault type, then the dirty data is flushed based on the exponential backoff retry strategy and the target timeout threshold, and the IO performance of the cache device is reduced during the flushing process. Specifically, if the backend device experiences a second target error, determining the health status result based on the backend device's second hardware health indicator and error log using a preset AI model includes: determining the second hardware health indicator based on the backend device's second SMART technology; the second target error includes data synchronization error, write to the second target block error, cache read error, cache miss error, and data discard error; the second target block is the area on the backend device used to cache the metadata; determining the error log using the backend device's error type distribution, error frequency, and error recovery success rate within a preset time period; and determining a health score based on the second hardware health indicator and the error log using a preset AI model. The step of using the health status result to perform corresponding preventive processing and fault handling operations includes: determining whether the health score exceeds a target score threshold; if the health score exceeds the target score threshold, it indicates that the backend device has a potential fault trend, flushing the dirty data based on a preset flushing time, and then reducing the IO performance of the backend device.

2. The device fault handling method based on bcache according to claim 1, characterized in that, When the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator. Based on this multi-dimensional indicator and using a preset state machine, it is determined whether the cache device's fault type matches the target fault type, including: When the cache device encounters a first target error, a first hardware health indicator including wear and tear, number of damaged storage units, and IO response time is obtained based on the first SMART technology of the cache device. The performance data for I / O operations based on the cache device includes software status indicators such as read / write failure frequency and throughput fluctuations. A multi-dimensional indicator is constructed based on the first hardware health indicator and the software status indicator. Based on the multi-dimensional indicator and using a finite state automaton, it is determined whether the fault type of the cache device is the target fault type.

3. The device fault handling method based on bcache according to claim 1, characterized in that, After determining whether the health score exceeds the target score threshold, the method further includes: If the health score does not exceed the target score threshold, it indicates that the backend device has a serious failure trend. In this case, I / O requests to the backend device are stopped, the corresponding device fault code is returned, and the data in the backend device is migrated to the backup device.

4. A device fault handling device based on bcache, characterized in that, include: The fault type judgment module is used to identify the preset high-speed storage device in the government cloud platform storage architecture as a cache device and the preset low-speed storage device in the government cloud platform storage architecture as a back-end device. When the cache device has a first target error, it constructs a multi-dimensional indicator based on the first hardware health indicator and software status indicator of the cache device, and judges whether the fault type of the cache device is the target fault type based on the multi-dimensional indicator and the preset state machine. The target fault type is an unrecoverable fault type; the government cloud platform storage architecture is a storage architecture integrating bcache; The device marking module is used to stop the read and write operations of the cache device and mark the cache device as a device to be repaired if the fault type of the cache device is the target fault type. The fault handling module is used to determine the health status result based on the second hardware health indicator and error log of the backend device and using a preset AI model when the backend device encounters a second target error, and to perform corresponding preventive handling operations and fault handling operations using the health status result. Specifically, the fault type determination module is used to identify a preset high-speed storage device in the government cloud platform storage architecture as a cache device and a preset low-speed storage device in the same architecture as a backend device. The preset high-speed storage device is a solid-state drive (SSD); the preset low-speed storage device is a hard disk drive (HDD). When the cache device experiences a first target error, a multi-dimensional indicator is constructed based on the cache device's first hardware health indicator and software status indicator. The first target error includes data insertion errors, read / write index errors, write to the first target block errors, read dirty data errors, data cleanup errors, and data migration errors. The first target block is the area on the cache device used to cache metadata. Dirty data refers to data that has been written to the cache device but not synchronized to the backend device; the dirty data includes the metadata and business data. Specifically, the bcache-based device fault handling device is further used to flush dirty data based on an exponential backoff retry strategy and a target timeout threshold if the fault type of the cache device is not the target fault type, and to reduce the IO performance of the cache device during the flushing process. Specifically, the fault handling module is used to determine a second hardware health indicator based on the second SMART technology of the backend device when a second target error occurs in the backend device; the second target error includes data synchronization error, write to the second target block error, cache read error, cache miss error, and data discard error; the second target block is the area of ​​the backend device used to cache the metadata; the module determines the error log by using the error type distribution, error frequency, and error recovery success rate of the backend device within a preset time period; and determines a health score based on the second hardware health indicator and the error log using a preset AI model. Specifically, the fault handling module is used to determine whether the health score exceeds the target score threshold; if the health score exceeds the target score threshold, it indicates that the backend device has a potential fault trend, and the dirty data is flushed based on a preset flushing time, thereby reducing the IO performance of the backend device.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the bcache-based device fault handling method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the bcache-based device fault handling method as described in any one of claims 1 to 3.

Images