Configuration data management method, system and device for big data cluster

Abstract

The invention provides a configuration data management method, system and device for big data clusters. The method comprises the following steps: collecting configuration data of a plurality of big data clusters through a distributed collector; storing the configuration data through a data processing background, and performing intelligent early warning and automatic restoration on the configuration data through an alarm rule; and performing service configuration and display of the configuration data on a visual platform. Through automatic management of the system, the management efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of big data clusters, and in particular to a configuration data management method, system and device for a big data cluster. Background Art

[0002] With the development of the Internet and big data and technological progress, server clusters such as Kafka, Hadoop and Spark have emerged, further promoting the progress of Internet big data. As the number of server clusters continues to increase, in order to ensure the high availability and performance of the system, it is often necessary to deploy a large number of redundant services in these clusters.

[0003] The big data components used within an enterprise are not limited to a single one. Hadoop, Spark, Hive, Flink clusters, etc. are all used simultaneously. These massive redundant configuration files are distributed across different nodes, lack configuration consistency verification, are scattered and complex to manage, and have high maintenance costs. They are also prone to configuration inconsistencies and difficulty in locating faults. If too much reliance is placed on manual management of these massive configuration files, it will lead to inefficiencies. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a configuration data management method, system and device for a big data cluster.

[0005] In a first aspect, the present application provides a configuration data management method for a big data cluster, the method comprising:

[0006] Collect configuration data of multiple big data clusters through distributed collectors;

[0007] The configuration data is stored in the data processing background, and intelligent early warning and automatic repair of the configuration data are performed through alarm rules;

[0008] Business configuration and the configuration data are displayed on the visualization platform.

[0009] Optionally, performing intelligent early warning on the configuration data through an alarm rule includes:

[0010] Determining target configuration data currently to be analyzed in the configuration data;

[0011] According to the attributes of the target configuration data, the target configuration data is compared with the corresponding historical configuration data or standard configuration data to determine whether there is changed data;

[0012] If there is any changed data, intelligent warning will be issued.

[0013] Optionally, automatically repairing the configuration data by using an alarm rule includes:

[0014] Acquire preset business configuration items from the visualization platform;

[0015] When it is detected that the automatic repair item in the business configuration item is turned on, if there is changed data after comparing the target configuration data with the corresponding standard configuration data, the automatic repair mechanism is triggered;

[0016] The target configuration data is replaced with the standard configuration data based on the automatic repair mechanism.

[0017] Optionally, the method further comprises:

[0018] The data processing background records the historical versions of all configuration data and the changed data in each version.

[0019] Optionally, the method further comprises:

[0020] Using the warning information corresponding to all configuration data and change data recorded in the data processing background as samples, the initial warning model is trained;

[0021] If the change data and the warning result output by the initial warning model are consistent with the change data and the warning information in the sample, it is determined that the training of the initial warning model is completed and the target warning model is obtained;

[0022] The latest configuration data is input into the target early warning model to obtain the change data output by the target early warning model and the early warning information corresponding to the change data.

[0023] Optionally, collecting configuration data of multiple big data clusters through a distributed collector includes:

[0024] Collecting all configuration data in the multiple big data clusters regularly through distributed collectors; and

[0025] When a hash value of a configuration file of a target big data cluster is detected to be changed through a distributed collector, configuration data of the changed configuration file is collected.

[0026] Optionally, collecting configuration data of multiple big data clusters through a distributed collector includes:

[0027] The distributed collector calls the API interface of the data processing background to obtain a collection template, wherein each collection template corresponds to a data format of a type of configuration file;

[0028] The configuration data in the corresponding configuration file is collected through the collection template, and the format of the configuration data is converted into a unified format that can be parsed by the data processing background.

[0029] In a second aspect, the present application provides a configuration data management system for a big data cluster, the system comprising:

[0030] Distributed collector, used to collect configuration data of multiple big data clusters;

[0031] A data processing backend, connected to the distributed collector, for storing the configuration data and performing intelligent early warning and automatic repair of the configuration data through alarm rules;

[0032] A visualization platform is connected to the data processing backend and is used for business configuration and display of the configuration data.

[0033] In a third aspect, the present application provides a configuration data management device for a big data cluster, the device comprising:

[0034] The collection module is used to collect the configuration data of multiple big data clusters through distributed collectors;

[0035] An early warning and repair module is used to store the configuration data through a data processing background, and to perform intelligent early warning and automatic repair on the configuration data through alarm rules;

[0036] The display module is used to display the business configuration and the configuration data on the visualization platform.

[0037] In a fourth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0038] Memory, used to store computer programs;

[0039] The processor is used to implement any of the steps of the configuration data management method for a big data cluster when executing a program stored in the memory.

[0040] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the configuration data management methods for a big data cluster are implemented.

[0041] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0042] In the method provided in the embodiment of the present application, the distributed collector is responsible for the timed collection of configuration data in the big data cluster, the data processing background stores the collected data, and realizes intelligent early warning and automatic repair functions through customized rule verification, and then the visualization platform is responsible for the display of business data and supports business configuration functions. In this way, complex and changeable configuration files can be visualized, and messy configurations can be analyzed and processed, errors can be predicted and discovered in advance, and failures caused by inconsistent configurations can be avoided. Compared with the inefficiency caused by manual management of complex and massive configuration data, this application improves management efficiency through system automation management. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A schematic diagram of a configuration data management system for a big data cluster provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of a configuration data management method for a big data cluster provided in an embodiment of the present application;

[0047] Figure 3 A flow chart of the interaction between the distributed collector and the data background system provided in the embodiment of the present application;

[0048] Figure 4 A processing flow chart of the data background system provided in the embodiment of the present application;

[0049] Figure 5 A functional schematic diagram of the main components of the system provided in the embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a configuration data management device for a big data cluster provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0054] In order to solve the problems mentioned in the background technology, according to one aspect of an embodiment of the present application, an embodiment of a configuration data management method for a big data cluster is provided.

[0055] Optionally, in the embodiment of the present application, the configuration data management method of the big data cluster can be applied to Figure 1 The hardware environment shown in FIG. 1 is composed of a server 101, a data processing backend 103 and a visualization platform 105. Figure 1 As shown, each server belongs to a node in the big data cluster. The distributed collector is deployed on each node of each big data cluster to collect the configuration data of multiple big data clusters and send the configuration data to the data processing background through the message queue MQ (Message Queue). The data processing background saves, processes and analyzes the configuration data, and can also perform intelligent early warning and automatic repair of the configuration data through alarm rules. The data processing analysis and automatic repair of the data processing background depend on the specific business configuration in the visualization platform. Finally, the visualization platform can display the configuration data to facilitate users to manage and view the configuration data. The visualization platform can use computers, mobile phones, tablets and other devices.

[0056] A configuration data management method for a big data cluster in an embodiment of the present application may be executed by the data processing background 103 or by the server 101 .

[0057] The present application embodiment provides a configuration data management method for a big data cluster, which is applied to a configuration data management system. The following will be combined with a specific implementation method to describe in detail a configuration data management method for a big data cluster provided by the present application embodiment. Figure 2 As shown, the specific steps are as follows:

[0058] Step 201: Collect configuration data of multiple big data clusters through a distributed collector.

[0059] Step 202: The configuration data is stored in the data processing background, and intelligent early warning and automatic repair of the configuration data are performed through alarm rules.

[0060] Step 203: Display the business configuration and configuration data on the visualization platform.

[0061] A big data cluster is a system consisting of multiple computers or servers used to store and process large-scale data sets. These computers are usually connected together through a network and run specialized software to coordinate and distribute data processing tasks. A single computer or server that makes up a big data cluster is called a node. In an enterprise, it is a common practice to use multiple big data clusters such as Hadoop, Spark, Hive, and Flink at the same time to support a variety of different data processing and analysis needs. However, this may lead to redundant configuration files and dispersion on different nodes, making configuration management complicated.

[0062] Based on the above problems, this application sets up a distributed collector, each of which is deployed on each node of each big data cluster to collect configuration data on the node, and then transmit the collected configuration data to the data processing background through MQ. The distributed collector can realize parallel collection of configuration data and improve data collection efficiency.

[0063] The configuration data collected by the distributed collector includes hardware configuration, software configuration, network configuration and security configuration. The configuration data of the big data cluster covers multiple aspects, mainly including hardware configuration, software configuration, network configuration and security configuration. Hardware configuration includes CPU model and core number, memory size, network card performance, power backup, cooling system, server redundancy, etc. Software configuration includes kernel parameters, big data framework, task parallelism, data partitioning, etc. Network configuration includes node IP address allocation, subnet division, port number of important services, etc. Security configuration includes user permission allocation, access control list, data storage encryption, etc.

[0064] When the data processing background receives the configuration data transmitted by the distributed collector, it mainly performs two tasks: one is the decoding and archiving of configuration items, and the other is intelligent rule verification. In the decoding and archiving part, the system will identify and process the configuration data according to the preset parsing mechanism, and then store it efficiently to ensure the availability and integrity of the data. For the rule verification part, the system will conduct a detailed review based on the set alarm rules and corresponding parameters such as blacklists. Once an abnormality or mismatch is captured, the system will not only immediately activate the alarm mechanism and notify relevant personnel in real time, but also automatically trigger the repair mechanism to quickly repair the erroneous server, thereby ensuring the stability of the system operation.

[0065] Users can manage some business configurations on the visualization platform, such as managing blacklists, managing whether to start automatic repair, and managing policy rules. The data processing background performs corresponding operations based on the business configuration in the visualization platform. In addition, the visualization platform can also provide a clear and intuitive way for users to easily view configuration data.

[0066] In this application, the distributed collector is responsible for the regular collection of configuration data in the big data cluster. The data processing background stores the collected data and implements intelligent early warning and automatic repair functions through customized rule verification. The visualization platform is responsible for the display of business data and supports business configuration functions. This visualizes complex and changeable configuration files, analyzes and processes messy configurations, predicts and discovers errors in advance, and avoids failures caused by inconsistent configurations. Compared with the inefficiency caused by manual management of complex and massive configuration data, this application improves management efficiency through system automation management.

[0067] As an optional implementation, intelligent warning of configuration data through alarm rules includes: determining the target configuration data to be analyzed in the configuration data; comparing the target configuration data with the corresponding historical configuration data or standard configuration data according to the attributes of the target configuration data to determine whether there is changed data; if there is changed data, intelligent warning is issued.

[0068] Configuration data has its own specific attributes. The system will retrieve and locate the corresponding historical configuration data or standard configuration data based on the specific attributes of the target configuration data (such as version number, configuration parameters, timestamp, etc.). Historical configuration data refers to the configuration status record of the same object at a previous point in time, while standard configuration data may be a set of predetermined, idealized configuration specifications. By comparing the current target configuration data with historical configuration data or standard configuration data, the system can identify the differences between the two. This comparison may involve matching configuration parameters or evaluation based on specific rules.

[0069] The system compares the currently collected configuration data with the most recent historical configuration data, which can be achieved by calculating hash values ​​or using a difference detection algorithm. If a difference is found, the system will generate a change report indicating which configuration items have changed. For example, if memory changes from 8GB to 16GB, the system will record this change. Or, the ip_address of the network part changes from 192.168.1.100 to 192.168.1.101; the disk_size of the storage part increases from 500GB to 1TB.

[0070] The system compares the target configuration data with the standard configuration data, which helps to confirm whether the current configuration meets the organization's standards or industry regulations. For example, a set of standard configuration data is defined for the Hadoop cluster in advance, including: Hadoop Yarn resource configuration parameters, such as yarn.nodemanager.resource.memory-mb is set to 8192 (8GB); the number of HDFS replicas, such as dfs.replication is set to 3; Spark configuration parameters, such as spark.executor.memory is set to 4g (4GB). The system uses automated scripts or configuration management tools to compare the target configuration data with the standard configuration data. If it is found that yarn.nodemanager.resource.memory-mb of a node is changed to 16384 (16GB), it is inconsistent with the standard configuration; if dfs.replication of HDFS is changed to 1 on some nodes, this also deviates from the standard configuration; if spark.executor.memory of Spark is not changed on a specific node, it is consistent with the standard configuration.

[0071] After the comparison process is completed, the system will evaluate whether there is any changed data, including but not limited to newly added configuration items, deleted configuration items, configuration parameter modifications, etc. If the system detects that there is changed data, and these changes meet the warning conditions (such as change type, change scope, change time, etc.), the intelligent warning mechanism will be triggered, thereby displaying warning information on the visualization platform or executing other preset response actions. Intelligent warning can facilitate the system to repair faults in a timely manner, and can also notify relevant personnel to pay attention to configuration changes, evaluate the impact of changes, and take necessary measures to ensure the stability and security of the system.

[0072] After detecting the change of configuration items, intelligent early warning can be carried out to avoid larger failures of the system, but the failure is inevitable at this time. Preferably, the system can also adopt machine learning or artificial intelligence strategies to make abnormal warnings more accurately and realize the prediction of potential problems, including the following: using all configuration data recorded in the data processing background and the early warning information corresponding to the change data as samples, the initial early warning model is trained; if the change data and early warning results output by the initial early warning model are consistent with the change data and early warning information in the sample, it is determined that the initial early warning model training is completed and the target early warning model is obtained; the latest configuration data is input into the target early warning model to obtain the change data output by the target early warning model and the early warning information corresponding to the change data.

[0073] Specifically, the data processing background records the historical versions of all configuration data, and records the change data in each version and the warning information corresponding to the change data. The system can use the warning information corresponding to the multiple versions of configuration data and change data stored in the data processing background as samples to train the initial warning model. The training phase includes two steps: forward propagation and back propagation. During the forward propagation process, the input data is passed through the network layer layer by layer to finally obtain the prediction result. During the back propagation process, the error between the prediction result and the actual result is calculated, and the network parameters are adjusted according to the error. This process will be repeated until the model converges or the preset number of training times is reached.

[0074] During the verification and testing phase, the system inputs the test configuration data into the trained initial warning model. If the change data and warning results output by the initial warning model are consistent with the change data and warning information corresponding to the test configuration data, the initial warning model training is determined to be complete, and the warning data is obtained. Subsequently, the latest configuration data can be input into the target warning model to obtain the change data output by the target warning model and the warning information corresponding to the change data.

[0075] This application uses machine learning to predict potential problems based on multiple versions of historical configuration data and change data, and provide early warning before failures occur.

[0076] In order to better support agile development and operation and maintenance, machine learning needs to be tightly integrated into the DevOps tool chain for early warning. The DevOps tool chain refers to DevOps tools such as Jenkins, Docker, Kubernetes, etc. DevOps is a software development method that emphasizes communication, collaboration, and integration between development and operation teams to deliver high-quality software and services faster and more frequently. The DevOps tool chain is a key tool collection to achieve this goal, which can assist the system to automate the process of building, testing, deploying, and monitoring applications.

[0077] Among them, Jenkins is an open source automation server that can help developers complete continuous integration and continuous delivery tasks. Docker is an open source application container engine that allows developers to package their applications and dependent packages into a portable container and then publish them to any platform that supports Docker. Kubernetes is an open source container orchestration system for automating the deployment, expansion, and management of containerized applications.

[0078] The system tightly integrates the DevOps tool chain and the target early warning model to ensure that the output of early warning information reduces manual intervention and improves efficiency and accuracy. In addition, after the target early warning model is trained, the application is packaged using Docker and then automatically deployed and expanded through Kubernetes, which helps to achieve rapid iteration and deployment in agile development and maintain high reliability and stability of the system.

[0079] In the management of big data clusters, configuration data change monitoring and timely repair are key steps to ensure system stability and performance. When the system detects configuration changes and issues an early warning, there are two basic repair mechanisms: manual repair and system automatic repair. The following is a detailed description of these two situations.

[0080] Manual repair relies on the metadata and change log system in the data processing background. This system records in detail the content, time, operator, and related context information of each configuration change. This data is stored in an accessible database or log file for subsequent query and analysis. When the system issues an early warning indicating that the configuration data has changed, the administrator can use the visualization platform to track the historical change records of specific configuration items. These change records provide a clear and intuitive timeline showing all changes of the configuration item from creation to the current state. By analyzing the historical change records, administrators can quickly locate the exact changes that caused the problem, including identifying who made the change, when the change was made, and the specific content of the change. This detailed audit trail improves the efficiency of troubleshooting and helps avoid future errors. Once the cause of the problem is determined, the administrator can restore the configuration item to its previous correct state based on the information in the historical record, or modify the configuration item according to best practices to optimize system performance.

[0081] System automatic repair includes the following: obtaining pre-set business configuration items from the visualization platform; when it is detected that the automatic repair item in the business configuration item is turned on, if there is a change in data after comparing the target configuration data with the corresponding standard configuration data, the automatic repair mechanism is triggered; based on the automatic repair mechanism, the target configuration data is replaced with the standard configuration data.

[0082] Specifically, system automatic repair is implemented based on preset business rules and policies, which are defined in the visualization platform and integrated with the data processing background through API interfaces. The visualization platform allows administrators to define and manage various business configuration items, such as blacklists, exception items, etc. These configuration items can be set as automatic repair items, which means that when they change, the system can automatically trigger the repair process. When the data processing background detects that a configuration item with the automatic repair function turned on has changed, it triggers the automatic repair mechanism, which replaces the target configuration data with the standard configuration data because the standard configuration data is considered to be the correct reference. Through a series of automated scripts or tools, the modified configuration data is rolled back to the previously defined standard state or updated according to the latest standards. This replacement process ensures the consistency and correctness of the system configuration. In addition, the automatic repair mechanism reduces the risk of system failure or performance degradation caused by configuration errors, improves the overall stability of the system, and ensures business continuity and high performance.

[0083] Whether it is manual repair or automatic system repair, the key is to be able to respond quickly to configuration changes to ensure the stability and security of the system. Manual repair relies on the experience of the administrator and the audit tracking capabilities of the system, while automatic repair relies on advanced automation tools and intelligent business rules. The combination of the two provides strong support for the configuration management of big data clusters, ensuring that the system can maintain efficient and reliable operation in a dynamically changing environment.

[0084] As an optional implementation method, collecting configuration data of multiple big data clusters through a distributed collector includes: periodically collecting the full amount of configuration data in the multiple big data clusters through the distributed collector; and when the distributed collector detects that the hash value of the configuration file of the target big data cluster changes, collecting the configuration data of the changed configuration file.

[0085] In the management of big data clusters, the real-time and accuracy of configuration data are crucial to ensuring the stability and performance of the system. As a key component, the distributed collector is designed to periodically collect the full configuration data from multiple big data clusters, and also has the ability to detect configuration changes in real time.

[0086] The distributed collector integrates a task scheduling system that can trigger collection tasks at set time intervals (such as every morning). In each collection cycle, the collector will access all nodes in the big data cluster, read and collect the current status of all configuration files, which ensures that the system has the latest configuration data snapshot, providing a basis for subsequent analysis and management. The configuration data collected from each node is aggregated to a central data processing background for unified storage and analysis, thereby building a global view to facilitate monitoring and management of the configuration status of the entire cluster.

[0087] In addition to scheduled full-volume collection, distributed collectors can also monitor changes in configuration files in real time. The collector will monitor the configuration files in real time and track changes in the file's metadata, such as file name and modification time. To improve efficiency, the collector can calculate the hash value of the configuration file and compare the current hash value with the previous hash value. Once a change in the hash value is found, it indicates that the configuration file has been modified. When a change in the hash value of the configuration file is detected, the collector will immediately determine that a configuration change has occurred and trigger a new collection cycle. It will automatically collect the configuration data of the changed configuration file to ensure that the data in the system is up to date. The collector can also configure a notification mechanism. Once an important configuration change is detected, it can immediately issue an early warning to the administrator so that timely measures can be taken.

[0088] The dual-mode mode of the distributed collector, which combines scheduled full data collection with real-time change detection, not only ensures the comprehensiveness and accuracy of configuration data, but also improves the system's response speed to configuration changes. This makes the configuration management of the big data cluster more efficient and reliable, and ensures the stable operation of the system.

[0089] As an optional implementation method, collecting configuration data of multiple big data clusters through a distributed collector includes: calling the API interface of the data processing background through the distributed collector to obtain a collection template, wherein each collection template corresponds to the data format of a type of configuration file; collecting the configuration data in the corresponding configuration file through the collection template, and converting the format of the configuration data into a unified format that can be parsed by the data processing background.

[0090] The big data cluster includes multiple types of configuration files, and the data format of each type of configuration file is different. This application calls the API interface of the data processing background to obtain the collection template, collect the configuration data in the corresponding format, and converts the multiple configuration file formats (such as XML, JSON, YAML, etc.) from different nodes in the cluster into a standard format with the same functional tags through a highly flexible and scalable parser. This application supports the conversion of files in more than 10 formats and has an easily expandable architecture to support more file formats in the future. This not only simplifies configuration management and improves operational efficiency, but also enhances the compatibility and flexibility of the system, making the system more robust and easy to maintain.

[0091] For example, the current system supports multiple configuration file formats or types, including single value (each line is K=V), DNS file format, XML, and file formats based on the number of lines. Once these configuration files are parsed by the collector, the resulting format will be presented in a regular format, including cluster name, service name, IP address, file path, key-value pairs, collection time and other information, making configuration management more unified.

[0092] The present application embodiment also provides a configuration data management system for a big data cluster, which includes three main components: a distributed collector, a data processing background, and a visualization platform. The following is a detailed description of each component:

[0093] 1. Distributed Collector

[0094] In order to run efficiently in a large number of nodes, the collector adopts a distributed architecture, running a collector instance on each node, which can independently collect data and send the data to the data processing background. The main responsibility of the distributed collector is to perform scheduled collection and real-time monitoring of the big data cluster, and then use MQ as the transport layer to achieve asynchronous, reliable and efficient data transmission.

[0095] 2. Data processing background

[0096] The data processing backend is responsible for analyzing and storing data collected by distributed collectors, supporting version management of configuration items, and recording the history of each change to facilitate rollback and auditing.

[0097] The data processing background captures anomalies or mismatches through corresponding parameters such as alarm rules and blacklists, immediately activates the alarm mechanism, and triggers the automatic repair mechanism to quickly repair the erroneous server, thereby ensuring the stability of system operation.

[0098] The data processing background uses machine learning or other artificial intelligence technologies to analyze historical data to predict and identify potential problems, and generate intelligent warnings to help administrators respond in a timely manner.

[0099] 3. Visualization Platform

[0100] The visualization platform displays complex configuration data in the form of graphics, tables or documents, so that non-professionals can easily understand the data. For example, it uses charts, dashboards, heat maps, etc. to display the data. It designs an easy-to-use business configuration interface so that users can easily modify configurations, set rules and monitor the system.

[0101] The visualization platform can also display historical configuration change information. Users can analyze historical configuration change information to track and retrieve historical change records of any configuration item, accurately locate specific change details, and improve the efficiency of system troubleshooting.

[0102] The visualization platform supports business configuration, and users can configure rules on the visualization platform according to actual needs.

[0103] Figure 3 This is the interaction flow chart between the distributed collector and the data background system. The distributed collector is deployed on each node of each cluster. It obtains the collection template by periodically calling the API of the data processing background. After obtaining the collection template, the collector starts to compare it with the configuration file on the server where it is located to detect any possible changes. This change may be based on the file name, modification time and MD5 hash value. Once a change is detected, the collector will read the configuration file and convert it into a format that can be parsed by the background, and then deliver this data to the message queue for processing and distribution. In addition, the distributed collector also has the ability to detect configuration changes in real time. Once a configuration is found to have changed, it can immediately trigger the collector to start collecting new configuration information.

[0104] Figure 4 This is the processing flow chart of the data background system. When the data processing background receives the configuration data transmitted by the distributed collector, it mainly performs two tasks: one is the decoding and archiving of the configuration items, and the other is the intelligent rule verification. In the decoding and archiving part, the system will identify and process the original data according to the preset parsing mechanism, and then store it in the database to ensure the availability and integrity of the data. For the rule verification part, the system will conduct a detailed review based on the corresponding parameters such as the alarm rules, whitelists and blacklists that have been set or read from the database. Once an abnormality or mismatch is captured, the system will not only immediately activate the alarm mechanism and notify the relevant personnel in real time, but also automatically trigger the repair mechanism to quickly repair the erroneous server, thereby ensuring the stability of the system operation.

[0105] Figure 5This is a functional diagram of the main components of the system in this application. It can be seen that the distributed collector can perform health checks, real-time monitoring, data collection, format processing and batch transmission. The data processing background can perform configuration management, rule verification, business storage, record change data, intelligently describe faults, intelligent early warning, and automatic repair. The functions of the visualization platform include managing configuration items, managing strategy planning, managing blacklists, change retrieval, and managing exception items.

[0106] This application automatically collects and parses configuration files in various formats, and automatically detects configuration errors and issues timely alarms by tracking historical versions. The system improves the efficiency of configuration management, enhances system security, and can conveniently display configuration history, which is very helpful for troubleshooting. The method in this application is conducive to effectively managing huge server resources, thereby optimizing operation and maintenance and improving business efficiency.

[0107] The method in this application is applicable to enterprises with large IT infrastructure that manage a large number of servers, including but not limited to large Internet companies, cloud service providers, large software companies, telecommunication operators, and financial institutions, etc., and can ensure the smooth operation of the business.

[0108] Based on the same technical concept, the embodiment of the present application also provides a configuration data management device for a big data cluster, such as Figure 6 As shown, the device comprises:

[0109] The collection module 601 is used to collect the configuration data of multiple big data clusters through a distributed collector;

[0110] The early warning and repair module 602 is used to store the configuration data through the data processing background, and to perform intelligent early warning and automatic repair on the configuration data through alarm rules;

[0111] The display module 603 is used to display the business configuration and configuration data on the visualization platform.

[0112] Optionally, the early warning and repair module 602 is used to:

[0113] Determine the target configuration data to be analyzed currently in the configuration data;

[0114] According to the attributes of the target configuration data, the target configuration data is compared with the corresponding historical configuration data or standard configuration data to determine whether there is changed data;

[0115] If there is any changed data, intelligent warning will be issued.

[0116] Optionally, the early warning and repair module 602 is used to:

[0117] Obtain pre-set business configuration items from the visualization platform;

[0118] When it is detected that the automatic repair item in the business configuration item is turned on, if there is changed data after comparing the target configuration data with the corresponding standard configuration data, the automatic repair mechanism is triggered;

[0119] The target configuration data is replaced with the standard configuration data based on an automatic repair mechanism.

[0120] Optionally, the device is also used for:

[0121] The data processing background records the historical versions of all configuration data and the changed data in each version.

[0122] Optionally, the device is also used for:

[0123] The initial warning model is trained using the warning information corresponding to all configuration data and change data recorded in the data processing background as samples;

[0124] If the change data and warning results output by the initial warning model are consistent with the change data and warning information in the sample, the initial warning model training is determined to be completed and the target warning model is obtained;

[0125] The latest configuration data is input into the target early warning model to obtain the change data output by the target early warning model and the early warning information corresponding to the change data.

[0126] Optionally, the acquisition module 601 is used to:

[0127] The configuration data of multiple big data clusters is collected through distributed collectors, including:

[0128] Use distributed collectors to periodically collect full configuration data from multiple big data clusters; and

[0129] When a hash value of a configuration file of a target big data cluster is detected to be changed through a distributed collector, configuration data of the changed configuration file is collected.

[0130] Optionally, the acquisition module 601 is used to:

[0131] The distributed collector calls the API interface of the data processing background to obtain the collection template, where each collection template corresponds to the data format of a type of configuration file;

[0132] The configuration data in the corresponding configuration file is collected through the collection template, and the format of the configuration data is converted into a unified format that can be parsed by the data processing background.

[0133] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 7As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0134] Memory 703, used for storing computer programs;

[0135] The processor 701 is used to implement the above steps when executing the program stored in the memory 703.

[0136] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] The communication interface is used for communication between the above electronic device and other devices.

[0138] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0139] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented, including the following contents.

[0141] Collect configuration data of multiple big data clusters through distributed collectors;

[0142] The configuration data is stored in the data processing background, and the configuration data is intelligently warned and automatically repaired through alarm rules;

[0143] Display business configuration and configuration data on the visualization platform.

[0144] Optionally, intelligent early warning of configuration data through alarm rules includes:

[0145] Determine the target configuration data to be analyzed currently in the configuration data;

[0146] According to the attributes of the target configuration data, the target configuration data is compared with the corresponding historical configuration data or standard configuration data to determine whether there is changed data;

[0147] If there is any changed data, intelligent warning will be issued.

[0148] Optionally, automatic repair of configuration data through alarm rules includes:

[0149] Obtain pre-set business configuration items from the visualization platform;

[0150] When it is detected that the automatic repair item in the business configuration item is turned on, if there is changed data after comparing the target configuration data with the corresponding standard configuration data, the automatic repair mechanism is triggered;

[0151] The target configuration data is replaced with the standard configuration data based on an automatic repair mechanism.

[0152] Optionally, the method further comprises:

[0153] The data processing background records the historical versions of all configuration data and the changed data in each version.

[0154] Optionally, the method further comprises:

[0155] The initial warning model is trained using the warning information corresponding to all configuration data and change data recorded in the data processing background as samples;

[0156] If the change data and warning results output by the initial warning model are consistent with the change data and warning information in the sample, the initial warning model training is determined to be completed and the target warning model is obtained;

[0157] The latest configuration data is input into the target early warning model to obtain the change data output by the target early warning model and the early warning information corresponding to the change data.

[0158] Optionally, collecting configuration data of multiple big data clusters through a distributed collector includes:

[0159] Use distributed collectors to periodically collect full configuration data from multiple big data clusters; and

[0160] When a hash value of a configuration file of a target big data cluster is detected to be changed through a distributed collector, configuration data of the changed configuration file is collected.

[0161] Optionally, collecting configuration data of multiple big data clusters through a distributed collector includes:

[0162] The distributed collector calls the API interface of the data processing background to obtain the collection template, where each collection template corresponds to the data format of a type of configuration file;

[0163] The configuration data in the corresponding configuration file is collected through the collection template, and the format of the configuration data is converted into a unified format that can be parsed by the data processing background.

[0164] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the methods in the above embodiments.

[0165] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0166] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0167] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A configuration data management method for a big data cluster, characterized in that: The method comprises: Collect configuration data of multiple big data clusters through distributed collectors; The configuration data is stored in the data processing background, and intelligent early warning and automatic repair of the configuration data are performed through alarm rules; Business configuration and the configuration data are displayed on the visualization platform.

2. The method according to claim 1, characterized in that Intelligent early warning of the configuration data through alarm rules includes: Determining target configuration data currently to be analyzed in the configuration data; According to the attributes of the target configuration data, the target configuration data is compared with the corresponding historical configuration data or standard configuration data to determine whether there is changed data; If there is any changed data, intelligent warning will be issued.

3. The method according to claim 2, characterized in that Automatically repairing the configuration data by using the alarm rule includes: Acquire preset business configuration items from the visualization platform; When it is detected that the automatic repair item in the business configuration item is turned on, if there is changed data after comparing the target configuration data with the corresponding standard configuration data, the automatic repair mechanism is triggered; The target configuration data is replaced with the standard configuration data based on the automatic repair mechanism.

4. The method according to claim 1, characterized in that: The method further comprises: The data processing background records the historical versions of all configuration data and the changed data in each version.

5. The method according to claim 4, characterized in that The method further comprises: Using the warning information corresponding to all configuration data and change data recorded in the data processing background as samples, the initial warning model is trained; If the change data and the warning result output by the initial warning model are consistent with the change data and the warning information in the sample, it is determined that the training of the initial warning model is completed and the target warning model is obtained; The latest configuration data is input into the target early warning model to obtain the change data output by the target early warning model and the early warning information corresponding to the change data.

6. The method according to claim 1, characterized in that The configuration data of multiple big data clusters is collected through distributed collectors, including: Collecting all configuration data in the multiple big data clusters regularly through a distributed collector; and When a hash value of a configuration file of a target big data cluster is detected to be changed through a distributed collector, configuration data of the changed configuration file is collected.

7. The method according to claim 1, characterized in that The configuration data of multiple big data clusters is collected through distributed collectors, including: The distributed collector calls the API interface of the data processing background to obtain a collection template, wherein each collection template corresponds to a data format of a type of configuration file; The configuration data in the corresponding configuration file is collected through the collection template, and the format of the configuration data is converted into a unified format that can be parsed by the data processing background.

8. A configuration data management system for a big data cluster, characterized in that: The system comprises: Distributed collector, used to collect configuration data of multiple big data clusters; A data processing backend, connected to the distributed collector, for storing the configuration data and performing intelligent early warning and automatic repair of the configuration data through alarm rules; A visualization platform is connected to the data processing backend and is used for business configuration and display of the configuration data.

9. A configuration data management device for a big data cluster, characterized in that: The device comprises: The collection module is used to collect the configuration data of multiple big data clusters through distributed collectors; An early warning and repair module is used to store the configuration data through a data processing background, and to perform intelligent early warning and automatic repair on the configuration data through alarm rules; The display module is used to display the business configuration and the configuration data on the visualization platform.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images