Intelligent operation and maintenance method and device for distributed file system, equipment and medium

Abstract

The invention discloses an intelligent operation and maintenance method, device and equipment of a distributed file system and a medium, is applied to an intelligent operation and maintenance system and relates to the technical field of computers, and the method comprises the steps of collecting performance index data of a distributed file system cluster, and preprocessing the performance index data to obtain target performance index data; inputting the target performance index data to a preset intelligent data analysis platform, and analyzing the target performance index data through the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster; and if the distributed file system cluster has the exception, generating a target operation and maintenance strategy corresponding to the exception according to a preset intelligent data analysis platform, and executing the target operation and maintenance strategy to complete operation and maintenance of the distributed file system cluster. Therefore, real-time monitoring and anomaly detection of the cluster state can be realized, a corresponding maintenance strategy is automatically executed, and the accuracy and response speed of operation and maintenance work are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an intelligent operation and maintenance method, device, equipment and medium for a distributed file system. Background Art

[0002] With the widespread adoption of Ceph (Controlled Replication Under Scalable Hashing, a distributed file system), its management and maintenance have become increasingly important. Managing and maintaining large-scale Ceph clusters is no easy task and presents a host of challenges. First, performance bottlenecks are a key factor limiting the effectiveness of Ceph clusters. When processing massive amounts of data, ensuring efficient data read and write operations and avoiding overall performance degradation caused by I / O (Input / Output) latency is a pressing issue. Furthermore, hardware failures are a significant factor affecting the stable operation of Ceph clusters. Because Ceph clusters typically consist of hundreds or thousands of nodes, the failure of any single node can threaten the data integrity and service continuity of the entire cluster.

[0003] Traditional operations and maintenance methods, such as manual configuration, monitoring, and troubleshooting, often make it difficult to quickly identify the root cause of problems and implement effective countermeasures. More importantly, manual operations are prone to errors, especially in complex distributed environments. Mistakes can lead to serious consequences, such as data loss or service interruptions. Summary of the Invention

[0004] In view of this, the present invention aims to provide a distributed file system intelligent operation and maintenance method, apparatus, device, and medium that can achieve real-time monitoring of cluster status and anomaly detection, and automatically execute corresponding maintenance strategies, thereby improving the accuracy and response speed of operation and maintenance work. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses an intelligent operation and maintenance method for a distributed file system, which is applied to an intelligent operation and maintenance system, including:

[0006] Collecting performance indicator data of a distributed file system cluster and preprocessing the performance indicator data to obtain target performance indicator data;

[0007] Inputting the target performance indicator data into a preset intelligent data analysis platform, and analyzing the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an anomaly in the distributed file system cluster;

[0008] If an abnormality exists in the distributed file system cluster, a target operation and maintenance strategy corresponding to the abnormality is generated according to the preset intelligent data analysis platform, and the operation and maintenance of the distributed file system cluster is completed by executing the target operation and maintenance strategy.

[0009] Optionally, collecting performance indicator data of a distributed file system cluster and preprocessing the performance indicator data to obtain target performance indicator data includes:

[0010] Collect data from each node in the distributed file system cluster using a data collection agent according to a preset data collection period to obtain hardware performance data and system performance data corresponding to each node;

[0011] The hardware performance data and the system performance data are uploaded to a local data cleaning backend system, so that the hardware performance data and the system performance data are pre-processed by the data cleaning backend system to obtain target performance indicator data.

[0012] Optionally, preprocessing the hardware performance data and the system performance data by the data cleaning backend system to obtain target performance indicator data includes:

[0013] Detecting abnormal values ​​and duplicate values ​​in the hardware performance data and the system performance data, and removing the abnormal values ​​and the duplicate values ​​from the hardware performance data and the system performance data to obtain first preprocessed data;

[0014] Missing values ​​are filled in the first preprocessed data, and the obtained filled data is standardized to obtain target performance indicator data.

[0015] Optionally, inputting the target performance indicator data into a preset intelligent data analysis platform, and analyzing the target performance indicator data by the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster, includes:

[0016] The target performance indicator data is input into a preset intelligent data analysis platform, and the current operation and maintenance requirements are determined, so that the intelligent data analysis platform selects a corresponding anomaly detection model based on the current operation and maintenance requirements, and performs an anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements through the anomaly detection model.

[0017] Optionally, selecting a corresponding anomaly detection model based on the current operation and maintenance requirements, and performing anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements using the anomaly detection model, includes:

[0018] If the current operation and maintenance requirement is a bad disk analysis requirement, determining a target anomaly detection model corresponding to the bad disk analysis requirement, and determining a sampling time difference and a sampling time interval corresponding to the disk performance data in the target performance indicator data by using the target anomaly detection model;

[0019] Determining the disk busyness based on the ratio of the sampling time difference to the sampling time interval;

[0020] If the disk busyness is greater than a preset busyness threshold within a preset time interval, the disk is determined to be a bad disk.

[0021] Optionally, if an abnormality exists in the distributed file system cluster, generating a target operation and maintenance strategy corresponding to the abnormality according to the preset intelligent data analysis platform, and completing the operation and maintenance of the distributed file system cluster by executing the target operation and maintenance strategy, including:

[0022] If the anomaly in the distributed file system cluster is a load balancing anomaly, the preset data scheduling strategy is used as the anomaly operation and maintenance strategy, and the data distribution is adjusted by executing the preset data scheduling strategy to improve cluster utilization;

[0023] If the abnormality in the distributed file system cluster is a hard disk abnormality, the preset fault hard disk offline policy is used as the abnormal operation and maintenance policy, and the bad disk is automatically offlined by executing the preset fault hard disk offline policy;

[0024] If the abnormality existing in the distributed file system cluster is a performance abnormality caused by a snapshot, matching a target repair action corresponding to the snapshot, and completing the operation and maintenance of the distributed file system cluster by executing the target repair action;

[0025] If the abnormality existing in the distributed file system cluster is a performance abnormality caused by backfilling, cluster resource allocation is adjusted based on the current cluster state to complete operation and maintenance of the distributed file system cluster.

[0026] Optionally, the intelligent operation and maintenance method of the distributed file system further includes:

[0027] Record the abnormal recovery status of the distributed file system cluster after the target operation and maintenance strategy is executed, and feed back the abnormal recovery status to the preset intelligent data analysis platform so that the preset intelligent data analysis platform can be updated based on the abnormal recovery status.

[0028] In a second aspect, the present application discloses an intelligent operation and maintenance device for a distributed file system, which is applied to an intelligent operation and maintenance system, including:

[0029] A data collection module is used to collect performance index data of the distributed file system cluster and pre-process the performance index data to obtain target performance index data;

[0030] an abnormality determination module, configured to input the target performance indicator data into a preset intelligent data analysis platform, and analyze the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster;

[0031] The intelligent operation and maintenance module is used to generate a target operation and maintenance strategy corresponding to the abnormality according to the preset intelligent data analysis platform if there is an abnormality in the distributed file system cluster, and complete the operation and maintenance of the distributed file system cluster by executing the target operation and maintenance strategy.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the intelligent operation and maintenance method of the distributed file system as described above.

[0035] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned intelligent operation and maintenance method of a distributed file system.

[0036] In the present application, performance indicator data of a distributed file system cluster can be collected, and the performance indicator data can be preprocessed to obtain target performance indicator data; the target performance indicator data can be input into a preset intelligent data analysis platform, and the target performance indicator data can be analyzed by the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster; if there is an abnormality in the distributed file system cluster, a target operation and maintenance strategy corresponding to the abnormality is generated according to the preset intelligent data analysis platform, and the operation and maintenance of the distributed file system cluster is completed by executing the target operation and maintenance strategy.

[0037] It can be seen that the method of the present application can be used to pre-process the performance indicator data of the collected distributed file system cluster, and the target performance indicator data obtained after pre-processing can be input into the preset intelligent data analysis platform, and the target performance indicator data can be analyzed by the preset intelligent data analysis platform to determine whether there is an anomaly in the distributed file system cluster; and in the case of an anomaly, the preset intelligent data analysis platform generates an operation and maintenance strategy corresponding to the anomaly, and finally the intelligent operation and maintenance of the distributed file system is realized by executing the generated operation and maintenance strategy. In this way, the data analysis technology based on artificial intelligence can be integrated into the daily management of the Ceph cluster, thereby not only realizing real-time monitoring and anomaly detection of the cluster status, but also predictive analysis based on historical data, early warning of potential risks, and automatic execution of corresponding maintenance strategies, which greatly improves the accuracy and response speed of operation and maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of an intelligent operation and maintenance method for a distributed file system disclosed in this application;

[0040] Figure 2 A schematic diagram of data collection and analysis disclosed in this application;

[0041] Figure 3 A data preprocessing diagram disclosed in this application;

[0042] Figure 4 This is a schematic diagram of intelligent operation and maintenance interaction disclosed in this application;

[0043] Figure 5 This is a schematic diagram of the structure of an intelligent operation and maintenance device for a distributed file system disclosed in this application;

[0044] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] Traditional operations and maintenance methods, such as manual configuration, monitoring, and troubleshooting, often make it difficult to quickly identify the root cause of problems and implement effective countermeasures. More importantly, manual operations are prone to errors, especially in complex distributed environments. Mistakes can lead to serious consequences, such as data loss or service interruptions.

[0047] In order to overcome the above technical problems, the present application discloses an intelligent operation and maintenance method, device, equipment and medium for a distributed file system, which can realize real-time monitoring and anomaly detection of cluster status, and automatically execute corresponding maintenance strategies, thereby improving the accuracy and response speed of operation and maintenance work.

[0048] See also Figure 1 As shown, an embodiment of the present invention discloses an intelligent operation and maintenance method for a distributed file system, which is applied to an intelligent operation and maintenance system, including:

[0049] Step S11: collecting performance indicator data of the distributed file system cluster and preprocessing the performance indicator data to obtain target performance indicator data.

[0050] In this embodiment, it is necessary to collect data from the Ceph distributed file system cluster and pre-process the data. Specifically, it is necessary to collect data from each node in the distributed file system cluster through a data collection agent according to a preset data collection cycle to obtain the hardware performance data and system performance data corresponding to each node. It should be noted that, Figure 2As shown, each node in a distributed file system cluster corresponds to a data collection agent. The agent regularly collects hardware performance data from each node, such as disk I / O operation rate and disk read / write latency, according to a preset cycle. It also collects system performance data for each node, such as network traffic, CPU (Central Processing Unit) utilization, and network packet loss rate. Disk I / O monitoring allows us to understand the rate and frequency of data reads and writes, assessing the load on the storage system. Disk read / write latency provides important information about data access latency, helping us understand application responsiveness. CPU utilization helps identify whether computing resources are being fully and effectively utilized, or whether bottlenecks are limiting system performance. This continuous data collection mechanism ensures that any changes or trends that may affect system stability are captured promptly, providing a solid foundation for subsequent analysis.

[0051] Furthermore, it is necessary to upload the hardware performance data and system performance data to the local data cleaning backend system Data wash sever, so that the hardware performance data and system performance data can be pre-processed by the data cleaning backend system to obtain the target performance indicator data. Figure 3 As shown, the data cleaning backend system, Data Wash Server, needs to detect outliers and duplicate values ​​in the hardware performance data and system performance data, remove outliers and duplicate values ​​from the hardware performance data and system performance data to obtain first preprocessed data; perform missing value filling on the first preprocessed data, and standardize the obtained filled data to obtain target performance indicator data. It should be noted that after collecting the hardware performance data and system performance data, the collected data needs to be uploaded to the data cleaning backend system. The data cleaning backend system then detects outliers and duplicate values ​​in the hardware performance data and system performance data, removes outliers and duplicate values ​​from the hardware performance data and system performance data to obtain first preprocessed data. Furthermore, the data cleaning backend system fills missing values ​​in the first preprocessed data and standardizes the obtained filled data to obtain target performance indicator data. This layered processing mechanism not only improves the efficiency of the entire data processing process but also enhances data quality, providing a solid foundation for subsequent advanced data analysis.

[0052] Step S12: input the target performance indicator data into a preset intelligent data analysis platform, and analyze the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster.

[0053] In this embodiment, the target performance indicator data needs to be input into the preset intelligent data analysis platform so that the preset intelligent data analysis platform can determine whether there are data anomalies in the input data. Specifically, the target performance indicator data needs to be input into the preset intelligent data analysis platform and the current operation and maintenance requirements are determined so that the intelligent data analysis platform can select the corresponding anomaly detection model based on the current operation and maintenance requirements, and perform anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements through the anomaly detection model. Figure 4 As shown, the pre-configured intelligent data analysis platform analyzes target performance indicator data and then selects a machine learning model based on the specific problem. It's important to note that the appropriate machine learning model should be selected based on the nature of the problem, such as a time series prediction model for trend analysis or a classification model for anomaly detection. This allows for automated selection of the appropriate anomaly detection model based on current O&M requirements, ensuring that anomaly analysis results are more tailored to actual O&M scenarios and improving alert accuracy and actionability.

[0054] Step S13: If there is an abnormality in the distributed file system cluster, a target operation and maintenance strategy corresponding to the abnormality is generated according to the preset intelligent data analysis platform, and the operation and maintenance of the distributed file system cluster is completed by executing the target operation and maintenance strategy.

[0055] In this embodiment, if an anomaly exists in the distributed file system cluster, it is necessary to generate a corresponding operation and maintenance strategy through the preset intelligent data analysis platform, and complete the operation and maintenance of the distributed file system cluster by executing the generated operation and maintenance strategy. Specifically, the generation of the operation and maintenance strategy can be divided into several aspects. First, if the anomaly existing in the distributed file system cluster is a load balancing anomaly, the preset data scheduling strategy is used as the abnormal operation and maintenance strategy, and the data distribution is adjusted by executing the preset data scheduling strategy to improve cluster utilization. Specifically, when a load imbalance is detected in a storage pool, the system will generate a preset data scheduling strategy for data migration as the abnormal operation and maintenance strategy, and then automatically trigger the data migration plan according to the preset data scheduling strategy to balance the capacity between each node, so that the cluster utilization rate is maximized.

[0056] Secondly, if the anomaly in the distributed file system cluster is a hard disk anomaly, the preset faulty hard disk offline policy will be used as the abnormal operation and maintenance policy, and the bad disk will be automatically offlined by executing the preset faulty hard disk offline policy. Specifically, if an OSD (Object Storage Daemon) is found to be down, it means that the OSD daemon has stopped running or cannot be accessed and communicated normally by other components in the cluster. Therefore, it is necessary to combine the dmesg log of the storage node where the OSD is located to make a judgment. If a hard disk is found to be faulty, the preset faulty hard disk offline policy will be used as the abnormal operation and maintenance policy, and the automatic offline process will be started immediately to avoid service interruptions caused by human processing delays. This automated response mechanism greatly reduces the need for human intervention and improves the availability and efficiency of the system.

[0057] Third, if the anomaly in the distributed file system cluster is caused by a snapshot, the target repair action corresponding to the snapshot is matched and executed to complete the operation and maintenance of the distributed file system cluster. Specifically, if a slow operations alarm is detected in the system, the health of the placement group (PG) in the current cluster is first determined. The PG status is then analyzed. If the slow performance is caused by a snapshot, the target repair action corresponding to the snapshot is matched, triggering the snapshot processing logic and automatically and promptly compacting the problematic OSD, ensuring a timely response and shortening the time for real-time optimization.

[0058] Fourthly, if the anomaly in the distributed file system cluster is a performance anomaly caused by backfilling, cluster resource allocation is adjusted based on the current cluster state to complete the operation and maintenance of the distributed file system cluster. Specifically, if the cluster state shows slow performance caused by backfilling, the backfill-related logic is triggered to lower the backfill value of the disk experiencing performance degradation, reducing cluster pressure and eliminating the impact on users.

[0059] It should be noted that the system supports the input of personalized operation and maintenance plans and corresponding operations for personalized scenarios. For example, users can describe a certain operation and maintenance scenario based on their actual operation and maintenance experience, and then write an automatic processing plan for the current operation and maintenance scenario. When the system detects the corresponding status of the Ceph cluster and the corresponding changes in the cluster status, it will automatically respond and perform corresponding operations.

[0060] It's also important to note that after executing the target O&M strategy, the distributed file system cluster's recovery status is recorded and fed back to the pre-set intelligent data analysis platform, allowing it to update itself based on the recovery status. This allows the system to self-learn, gradually improving its decision-making logic and response strategies, ensuring continuous improvement in the overall performance and stability of the Ceph cluster over time.

[0061] In this embodiment, the collected performance indicator data of the distributed file system cluster can be preprocessed, and the target performance indicator data obtained after preprocessing can be input into a preset intelligent data analysis platform. The target performance indicator data can be analyzed by the preset intelligent data analysis platform to determine whether there are any anomalies in the distributed file system cluster; and when an anomaly occurs, the preset intelligent data analysis platform generates an operation and maintenance strategy corresponding to the anomaly, and finally, the generated operation and maintenance strategy is executed to achieve intelligent operation and maintenance of the distributed file system. In this way, on the one hand, artificial intelligence-based data analysis technology can be integrated into the daily management of the Ceph cluster, thereby not only achieving real-time monitoring and anomaly detection of the cluster status, but also predictive analysis based on historical data, early warning of potential risks, and automatic execution of corresponding maintenance strategies, greatly improving the accuracy and response speed of operation and maintenance work. On the other hand, in this application, necessary maintenance operations, such as cluster balancing adjustment, data pool automatic balancing, and faulty disk replacement, are executed through an automated response mechanism, effectively reducing the risks caused by human intervention. On the other hand, the results of the execution are fed back to the system, forming a closed-loop control, enabling the system to self-learn and continuously optimize operation and maintenance strategies. This mechanism helps to continuously improve the overall performance and stability of the Ceph cluster, while also providing a solid foundation for future technology upgrades.

[0062] As a preferred embodiment, if the current operation and maintenance demand is a bad disk analysis demand, the target anomaly detection model corresponding to the bad disk analysis demand is determined, and the sampling time difference and sampling time interval corresponding to the disk performance data in the target performance indicator data are determined by the target anomaly detection model. Specifically, the time difference between two samples and the time interval between two samples can be determined. The disk busyness is then determined based on the ratio of the sampling time difference to the sampling time interval, wherein the calculation formula for the busyness is as follows: the time difference between two samples / the time interval between two samples*100%. If the disk busyness is greater than the preset busyness threshold within the preset time interval, the disk is determined to be a bad disk. For example, if the busy percentage of the current disk continues to be 100% and lasts for more than 3 minutes, even if there is no log information in the operating system, the current disk is considered to be a bad disk, and the osd must be decisively taken offline and logically deleted, otherwise this bad disk will affect the I / O of the entire storage cluster.

[0063] As a preferred embodiment, a combination of eBPF (extended Berkeley Packet Filter), kernel events, and the NVMe (NVM Express) native interface can be used for data collection, achieving real-time, low-intrusive, and high-precision performance. eBPF technology can establish a bridge between user and kernel modes, dynamically inserting code into the kernel to collect block-layer I / O request events in real time without modifying kernel code. By tracking kernel events, the Linux kernel's tracepoint or kprobe mechanism can be utilized to monitor key block-layer events, providing underlying data support for storage performance optimization and anomaly troubleshooting. NVMe native commands can be directly called through the NVMe native interface to obtain hardware-level health status, temperature, error counts, and other information. Data is obtained directly from the device firmware, avoiding interference from intermediate layers and effectively improving data collection efficiency.

[0064] See also Figure 5 As shown, an embodiment of the present invention discloses an intelligent operation and maintenance device for a distributed file system, which is applied to an intelligent operation and maintenance system, including:

[0065] The data collection module 11 is used to collect performance index data of the distributed file system cluster and pre-process the performance index data to obtain target performance index data;

[0066] Anomaly determination module 12, configured to input the target performance indicator data into a preset intelligent data analysis platform, and analyze the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an anomaly in the distributed file system cluster;

[0067] The intelligent operation and maintenance module 13 is used to generate a target operation and maintenance strategy corresponding to the abnormality according to the preset intelligent data analysis platform if there is an abnormality in the distributed file system cluster, and complete the operation and maintenance of the distributed file system cluster by executing the target operation and maintenance strategy.

[0068] In this embodiment, the performance indicator data of the collected distributed file system cluster can be preprocessed, and the target performance indicator data obtained after preprocessing can be input into a preset intelligent data analysis platform, and the target performance indicator data can be analyzed by the preset intelligent data analysis platform to determine whether there are any abnormalities in the distributed file system cluster; and in the event of an abnormality, an operation and maintenance strategy corresponding to the abnormality can be generated by the preset intelligent data analysis platform, and finally the intelligent operation and maintenance of the distributed file system can be realized by executing the generated operation and maintenance strategy. In this way, artificial intelligence-based data analysis technology can be integrated into the daily management of the Ceph cluster, thereby not only realizing real-time monitoring and anomaly detection of the cluster status, but also predictive analysis based on historical data, early warning of potential risks, and automatic execution of corresponding maintenance strategies, greatly improving the accuracy and response speed of operation and maintenance work.

[0069] In some embodiments, the data acquisition module 11 may specifically include:

[0070] The data collection submodule is used to collect data from each node in the distributed file system cluster through a data collection agent according to a preset data collection period to obtain hardware performance data and system performance data corresponding to each node;

[0071] The data preprocessing submodule is used to upload the hardware performance data and the system performance data to a local data cleaning backend system so as to preprocess the hardware performance data and the system performance data through the data cleaning backend system to obtain target performance indicator data.

[0072] In some embodiments, the data preprocessing submodule may specifically include:

[0073] a data elimination unit, configured to detect abnormal values ​​and duplicate values ​​in the hardware performance data and the system performance data, and remove the abnormal values ​​and the duplicate values ​​from the hardware performance data and the system performance data to obtain first preprocessed data;

[0074] The data standardization unit is used to fill missing values ​​in the first preprocessed data and standardize the obtained filled data to obtain target performance indicator data.

[0075] In some embodiments, the abnormality determination module 12 may specifically include:

[0076] The anomaly analysis submodule is used to input the target performance indicator data into a preset intelligent data analysis platform and determine the current operation and maintenance requirements so that the intelligent data analysis platform selects a corresponding anomaly detection model based on the current operation and maintenance requirements, and performs an anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements through the anomaly detection model.

[0077] In some embodiments, the abnormality analysis submodule may specifically include:

[0078] a sampling data determining unit configured to, if the current operation and maintenance requirement is a bad disk analysis requirement, determine a target anomaly detection model corresponding to the bad disk analysis requirement, and determine, by using the target anomaly detection model, a sampling time difference and a sampling time interval corresponding to the disk performance data in the target performance indicator data;

[0079] a disk parameter determination unit, configured to determine a disk busyness based on a ratio of the sampling time difference to the sampling time interval;

[0080] The bad disk determination unit is configured to determine that the disk is a bad disk if the busyness of the disk is greater than a preset busyness threshold within a preset time interval.

[0081] In some embodiments, the intelligent operation and maintenance module 13 may specifically include:

[0082] A first intelligent operation and maintenance unit is configured to, if the anomaly existing in the distributed file system cluster is a load balancing anomaly, use a preset data scheduling strategy as an abnormal operation and maintenance strategy, and adjust data distribution by executing the preset data scheduling strategy to improve cluster utilization;

[0083] a second intelligent operation and maintenance unit configured to, if the abnormality existing in the distributed file system cluster is a hard disk abnormality, use a preset faulty hard disk offline policy as the abnormal operation and maintenance policy, and automatically offline the bad disk by executing the preset faulty hard disk offline policy;

[0084] a third intelligent operation and maintenance unit, configured to, if the anomaly existing in the distributed file system cluster is a performance anomaly caused by a snapshot, match a target repair action corresponding to the snapshot, and complete operation and maintenance of the distributed file system cluster by executing the target repair action;

[0085] The fourth intelligent operation and maintenance unit is configured to adjust cluster resource allocation based on a current cluster state to complete operation and maintenance of the distributed file system cluster if the abnormality existing in the distributed file system cluster is a performance abnormality caused by backfilling.

[0086] In some embodiments, the intelligent operation and maintenance device for the distributed file system may further include:

[0087] The platform update unit is used to record the abnormal recovery status of the distributed file system cluster after the execution of the target operation and maintenance strategy, and feed back the abnormal recovery status to the preset intelligent data analysis platform so that the preset intelligent data analysis platform can be updated based on the abnormal recovery status.

[0088] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0089] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the intelligent operation and maintenance method of the distributed file system disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0091] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0092] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to implement the intelligent operation and maintenance method of the distributed file system executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to perform other specific tasks.

[0093] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned intelligent operation and maintenance method for a distributed file system. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0095] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0097] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0098] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An intelligent operation and maintenance method for a distributed file system, characterized in that: Applied to intelligent operation and maintenance systems, including: Collecting performance indicator data of a distributed file system cluster and preprocessing the performance indicator data to obtain target performance indicator data; Inputting the target performance indicator data into a preset intelligent data analysis platform, and analyzing the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an anomaly in the distributed file system cluster; If an abnormality exists in the distributed file system cluster, a target operation and maintenance strategy corresponding to the abnormality is generated according to the preset intelligent data analysis platform, and the operation and maintenance of the distributed file system cluster is completed by executing the target operation and maintenance strategy.

2. The intelligent operation and maintenance method of a distributed file system according to claim 1, characterized in that: The collecting of performance indicator data of the distributed file system cluster and preprocessing the performance indicator data to obtain target performance indicator data includes: Collect data from each node in the distributed file system cluster using a data collection agent according to a preset data collection period to obtain hardware performance data and system performance data corresponding to each node; The hardware performance data and the system performance data are uploaded to a local data cleaning backend system, so that the hardware performance data and the system performance data are pre-processed by the data cleaning backend system to obtain target performance indicator data.

3. The intelligent operation and maintenance method of a distributed file system according to claim 2, characterized in that: The preprocessing of the hardware performance data and the system performance data by the data cleaning backend system to obtain target performance indicator data includes: Detecting abnormal values ​​and duplicate values ​​in the hardware performance data and the system performance data, and removing the abnormal values ​​and the duplicate values ​​from the hardware performance data and the system performance data to obtain first preprocessed data; Missing values ​​are filled in the first preprocessed data, and the obtained filled data is standardized to obtain target performance indicator data.

4. The intelligent operation and maintenance method of a distributed file system according to claim 1, characterized in that: Inputting the target performance indicator data into a preset intelligent data analysis platform, and analyzing the target performance indicator data by the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster, includes: The target performance indicator data is input into a preset intelligent data analysis platform, and the current operation and maintenance requirements are determined, so that the intelligent data analysis platform selects a corresponding anomaly detection model based on the current operation and maintenance requirements, and performs an anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements through the anomaly detection model.

5. The intelligent operation and maintenance method of a distributed file system according to claim 4, characterized in that: The selecting a corresponding anomaly detection model based on the current operation and maintenance requirements, and performing an anomaly analysis on the target performance indicator data according to the current operation and maintenance requirements by using the anomaly detection model, includes: If the current operation and maintenance requirement is a bad disk analysis requirement, determining a target anomaly detection model corresponding to the bad disk analysis requirement, and determining a sampling time difference and a sampling time interval corresponding to the disk performance data in the target performance indicator data by using the target anomaly detection model; Determining the disk busyness based on the ratio of the sampling time difference to the sampling time interval; If the disk busyness is greater than a preset busyness threshold within a preset time interval, the disk is determined to be a bad disk.

6. The intelligent operation and maintenance method of a distributed file system according to claim 5, characterized in that: If an abnormality exists in the distributed file system cluster, a target operation and maintenance strategy corresponding to the abnormality is generated according to the preset intelligent data analysis platform, and the operation and maintenance of the distributed file system cluster is completed by executing the target operation and maintenance strategy, including: If the anomaly in the distributed file system cluster is a load balancing anomaly, the preset data scheduling strategy is used as the anomaly operation and maintenance strategy, and the data distribution is adjusted by executing the preset data scheduling strategy to improve cluster utilization; If the abnormality in the distributed file system cluster is a hard disk abnormality, the preset fault hard disk offline policy is used as the abnormal operation and maintenance policy, and the bad disk is automatically offlined by executing the preset fault hard disk offline policy; If the abnormality existing in the distributed file system cluster is a performance abnormality caused by a snapshot, matching a target repair action corresponding to the snapshot, and completing the operation and maintenance of the distributed file system cluster by executing the target repair action; If the abnormality existing in the distributed file system cluster is a performance abnormality caused by backfilling, cluster resource allocation is adjusted based on the current cluster state to complete operation and maintenance of the distributed file system cluster.

7. The intelligent operation and maintenance method of a distributed file system according to any one of claims 1 to 6, characterized in that: Also includes: Record the abnormal recovery status of the distributed file system cluster after the target operation and maintenance strategy is executed, and feed back the abnormal recovery status to the preset intelligent data analysis platform so that the preset intelligent data analysis platform can be updated based on the abnormal recovery status.

8. An intelligent operation and maintenance device for a distributed file system, characterized in that: Applied to intelligent operation and maintenance systems, including: A data collection module is used to collect performance index data of the distributed file system cluster and pre-process the performance index data to obtain target performance index data; an abnormality determination module, configured to input the target performance indicator data into a preset intelligent data analysis platform, and analyze the target performance indicator data through the preset intelligent data analysis platform to determine whether there is an abnormality in the distributed file system cluster; The intelligent operation and maintenance module is used to generate a target operation and maintenance strategy corresponding to the abnormality according to the preset intelligent data analysis platform if there is an abnormality in the distributed file system cluster, and complete the operation and maintenance of the distributed file system cluster by executing the target operation and maintenance strategy.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the intelligent operation and maintenance method of a distributed file system according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the intelligent operation and maintenance method of the distributed file system according to any one of claims 1 to 7 is implemented.

Images