Configuration data management method and device for big data cluster, equipment and medium

Abstract

The invention relates to the technical field of financial or medical management, in particular to a configuration data management method and device for a big data cluster, equipment and a medium, and the method comprises the steps: obtaining multi-dimensional configuration data of each node in the big data cluster, processing the multi-dimensional configuration data based on a preset exception screening rule base, screening out an exception configuration data set, and storing the exception configuration data set in a database; the method comprises the following steps: acquiring abnormal configuration data in an abnormal configuration data set, performing priority ranking on various abnormal data in the abnormal configuration data set to obtain a target abnormal configuration data set, and inputting the target abnormal configuration data set into an abnormal prediction model based on a graph neural network and a deep learning network to obtain an abnormal prediction result, the exception prediction result comprises exception propagation path prediction and associated exception index prediction, and then according to the exception prediction result and the type of the target exception configuration data set, the target exception configuration data set is managed more efficiently by using a preset alarm repair rule, so that the efficiency of configuration data management of the big data cluster is improved.

Description

Technical Field

[0001] This invention relates to the fields of financial or medical management technology, and in particular to a configuration data management method, apparatus, equipment and medium for a big data cluster. Background Art

[0002] With the development of the internet and big data, and technological advancements, server clusters such as Kubernetes have emerged. Kubernetes is an open-source container orchestration engine used to automatically deploy containerized applications. This clustering technology has further facilitated the handling of large volumes of data in industries such as finance and healthcare. In large-scale cluster applications, where applications run on hundreds or even thousands of servers, the maintenance and management of configuration data becomes particularly crucial.

[0003] In existing configuration data management methods, configuration data is often manually set in the configuration files of each application server in the application server cluster, without a unified management platform. When there are dependencies between servers, if the configuration data of one server needs to be modified, the configuration of that server must be modified first, and then the configurations of other related servers must be modified to ensure that the configuration data on all servers is consistent. This process is also done manually. Since the types and numbers of application servers within a cluster system are diverse, relying too much on manual management of these massive configuration files will lead to inefficiency. Therefore, how to improve the efficiency of configuration data management in big data clusters is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] Therefore, in order to address the aforementioned technical problems, this invention provides a method, apparatus, device, and medium for managing configuration data of a big data cluster, thereby improving the efficiency of configuration data management for big data clusters.

[0005] A first aspect of this application provides a configuration data management method for a big data cluster, the configuration data management method for the big data cluster comprising: Obtain multi-dimensional configuration data for each node in a big data cluster; The multidimensional configuration data is processed based on a preset anomaly filtering rule base to filter out anomaly configuration datasets, and the anomaly data in the anomaly configuration datasets are prioritized to obtain the target anomaly configuration dataset. The target anomaly configuration dataset is input into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. Based on the anomaly prediction results and the type of the target anomaly configuration dataset, the target anomaly configuration dataset is managed using preset alarm repair rules.

[0006] A second aspect of this application provides a configuration data management device for a big data cluster, the configuration data management device for the big data cluster comprising: The acquisition module is used to acquire multi-dimensional configuration data for each node in the big data cluster. The filtering module is used to process the multidimensional configuration data based on a preset abnormal filtering rule base, filter out the abnormal configuration dataset, and sort the abnormal data in the abnormal configuration dataset by priority to obtain the target abnormal configuration dataset. The module is used to input the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. The management module is used to manage the target anomaly configuration dataset according to the anomaly prediction results and the type of the target anomaly configuration dataset, using preset alarm repair rules.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the configuration data management method for a big data cluster as described in the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the configuration data management method for a big data cluster as described in the first aspect.

[0009] In summary, this invention provides a method, apparatus, device, and medium for configuration data management in a big data cluster. By acquiring multi-dimensional configuration data from each node in the big data cluster, processing the multi-dimensional configuration data based on a preset anomaly filtering rule base, filtering out anomaly configuration datasets, and prioritizing the anomaly data in the anomaly configuration datasets to obtain a target anomaly configuration dataset, this target anomaly configuration dataset is input into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results. These anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. Based on the anomaly prediction results and the type of the target anomaly configuration dataset, preset alarm repair rules are used to manage the target anomaly configuration dataset. Therefore, this application improves the efficiency of configuration data management in big data clusters by inputting the filtered and sorted target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, and then combining this with the type of the target anomaly configuration dataset to manage it more efficiently using preset alarm repair rules. Attached Figure Description

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment for a configuration data management method for a big data cluster provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a configuration data management method for a big data cluster according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a configuration data management device for a big data cluster provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," or "in response to determination." Similarly, the phrase "if determined" or "if matched to [described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once matched to [described condition or event]," or "in response to matched to [described condition or event]."

[0016] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0019] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0020] An embodiment of the present invention provides a configuration data management method for a big data cluster, which can be applied to, for example... Figure 1In this application environment, the terminal device communicates with the server. The terminal device includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0021] A big data cluster is a system composed of multiple computers or servers used to store and process large-scale datasets. These computers are typically connected via a network and run specialized software to coordinate and allocate configuration data processing tasks. A single computer or server that makes up a big data cluster can be called a node. In core transaction systems of financial banks or picture archiving and communication systems (PACS) of hospitals, the use of big data clusters such as Kubernetes is a common practice to support various configuration data processing and analysis needs.

[0022] See Figure 2 This is a flowchart illustrating a configuration data management method for a big data cluster according to an embodiment of the present invention, as shown below. Figure 2 As shown, the configuration data management method for this big data cluster can be implemented through the following steps.

[0023] S201: Obtain multi-dimensional configuration data for each node in the big data cluster.

[0024] In step S201, a lightweight data acquisition agent is deployed on each cluster node, and a standardized data acquisition process is designed, including the data acquisition frequency (periodic or real-time acquisition), the acquisition triggering mechanism (manual or automatic triggering), and the data transmission method (via API interface, message queue, etc.). This allows for the acquisition of multi-dimensional configuration data from each node in the big data cluster. For example, a financial trading system acquires CPU usage and transaction latency every 10 seconds; a medical PACS acquires image processing time and storage node health status every 5 minutes. The acquired multi-dimensional configuration data is then transmitted to the data processing backend via MQ. Furthermore, the lightweight acquisition agent enables parallel acquisition of multi-dimensional configuration data, improving data acquisition efficiency.

[0025] Furthermore, multi-dimensional configuration data includes hardware configuration, software configuration, network configuration, and security configuration. The configuration data of a big data cluster covers multiple aspects, primarily including hardware configuration, software configuration, network configuration, and security configuration. Hardware configuration includes CPU model and core count, memory size, network card performance, power backup, cooling system, and server redundancy. Software configuration includes kernel parameters, big data framework, task parallelism, and data partitioning. Network configuration includes node IP address allocation, subnetting, and port numbers for important services. Security configuration includes user permission allocation, access control lists, and data storage encryption. This application does not impose any limitations on these aspects. For example, a large bank's data cluster is used to process massive amounts of customer transaction data, risk assessment data, and market analysis data. By acquiring the multi-dimensional configuration data of each node, the bank can achieve goals such as transaction performance optimization. Similarly, a regional medical big data center's cluster is responsible for integrating and analyzing medical data from multiple hospitals, providing support for regional medical collaboration, disease prediction, and precision medicine. By acquiring the multi-dimensional configuration data of each node, patient data security can be ensured.

[0026] In this embodiment, by quickly and accurately acquiring multi-dimensional configuration data of each node in the big data cluster, it is possible to comprehensively and meticulously grasp the hardware, software, network, and other configurations and operating status of each node in the big data cluster, thereby reducing manual intervention and laying the foundation for improving the accuracy of configuration data management of the big data cluster in the future.

[0027] S102: Process the multidimensional configuration data based on the preset anomaly filtering rule base, filter out the abnormal configuration dataset, and sort the abnormal data in the abnormal configuration dataset by priority to obtain the target abnormal configuration dataset.

[0028] In step S102, each configuration data in the multidimensional configuration data is preprocessed sequentially through user-defined configuration or built-in rules of the data system to determine the data unit and periodic change characteristics of each configuration data in the preprocessed multidimensional configuration data. Then, the data characteristics of each configuration data are determined based on the data unit and periodic change characteristics. Based on the data characteristics, an initial judgment threshold, a first upper limit threshold, and a first lower limit threshold for abnormal data are defined. An early warning level is set based on the first upper limit threshold, the first lower limit threshold, and a preset threshold range. An abnormal screening rule base is formulated based on the early warning level, the judgment threshold, the first upper limit threshold, and the first lower limit threshold, combined with a preset abnormal detection algorithm. The preset abnormal detection algorithm includes: linear regression algorithm, autoregressive integral moving average algorithm, random forest algorithm, and deep neural network algorithm. Then, the multidimensional configuration data is compared with the anomaly filtering rule base in real time to identify anomalies that meet the rules. Combined with supplementary information such as historical data, related node configuration, and business load, the authenticity of the anomalies is verified. The same type of anomalies that occur repeatedly on the same node are merged to avoid duplicate alarms. The anomaly records that pass the filtering are stored in the anomaly configuration dataset. After obtaining the anomaly configuration dataset, the anomalies in the anomaly configuration dataset are prioritized according to the anomaly length of each anomaly data in the anomaly configuration dataset to obtain the target anomaly configuration dataset.

[0029] In one embodiment of the invention, a preset anomaly filtering rule base includes mandatory rules, advisory rules, and dynamic adaptive rules. Based on this preset anomaly filtering rule base, multidimensional configuration data is processed to filter out anomaly configuration datasets, including: Based on the first anomaly identifier of the mandatory rule, filter out the first abnormal configuration data that matches the first anomaly identifier from the multidimensional configuration data; Based on the second anomaly identifier of the suggested rule, filter out second abnormal configuration data that matches the second anomaly identifier from the multidimensional configuration data; Based on the third anomaly identifier of the dynamic adaptive rule, filter out the third anomaly configuration data that matches the third anomaly identifier from the multidimensional configuration data; The first abnormal configuration data, the second abnormal configuration data, and the third abnormal configuration data are integrated to obtain an abnormal configuration dataset.

[0030] Specifically, mandatory rules are "hard conditions" that must be strictly followed, such as data format validation (numeric fields cannot be text) and threshold limits (transaction amounts must be ≤500,000 yuan). Using the first anomaly identifier of the mandatory rules (such as unique identifiers like "incorrect field type" or "exceeding the limit"), the multidimensional configuration data is matched field by field to filter out the first abnormal configuration data that meets the mandatory rule's anomaly conditions (such as incorrect format or exceeding the hard threshold). Suggestive rules are "flexible reference" conditions, set based on experience or probability models, such as "high-frequency access IP addresses are recommended for verification" and "transactions outside working hours are recommended for manual review." Using the second anomaly identifier of the suggestive rules (such as "high-frequency access" or "transactions outside working hours"), the second abnormal configuration data that meets the suggestive rules is filtered from the multidimensional configuration data. Dynamic adaptive rules can automatically adjust thresholds based on real-time data or historical patterns, such as updating "user behavior anomaly models" and "device failure prediction thresholds" in real time through machine learning. By using a third anomaly identifier in the dynamic adaptive rules (such as "behavior pattern deviation > latest threshold" or "device parameter fluctuation exceeding dynamic range"), the matching degree between multidimensional configuration data and dynamic rules is calculated in real time. Third-order abnormal configuration data that conforms to the dynamic adaptive rules is then filtered from the multidimensional configuration data. Subsequently, the first, second, and third-order abnormal configuration data are associated by node ID or configuration item and merged into an abnormal configuration dataset. Through these steps, abnormal configuration data is filtered from different rule dimensions, enabling a comprehensive examination of multidimensional configuration data from multiple perspectives. This ensures that no potential anomalies are overlooked, making the filtering process highly targeted and improving the accuracy of abnormal data filtering.

[0031] In one embodiment of the invention, the abnormal data in the abnormal configuration dataset are prioritized to obtain a target abnormal configuration dataset, including: Obtain the duration of each abnormal data item in the abnormal configuration dataset for each node within a specified time period; The degree of abnormality of each abnormal data item is determined based on the duration of each abnormality. Based on the degree of abnormality, the scope of impact of the abnormality, and the importance level of the nodes, the abnormal data in the abnormal configuration dataset are prioritized to obtain the target abnormal configuration dataset.

[0032] Specifically, each abnormal data entry in the abnormal configuration dataset is split by "node" (such as system module, device ID, business unit) and "timestamp". For example, if an abnormal data entry corresponds to "payment module - node A", the occurrence time is "2025-06-20 10:00:00", and the end time is "2025-06-20 10:15:00", then the abnormal duration is 15 minutes. For a specified time period (such as "last 7 days" or "this month"), the duration of each abnormal data entry is statistically analyzed by node group. If the same node triggers the same type of abnormality multiple times within a time period (such as node A frequently experiencing "memory overflow"), the duration of consecutive abnormalities needs to be merged (e.g., multiple abnormalities with an interval of <5 minutes are considered as one continuous abnormality). Furthermore, after obtaining the abnormal duration of each abnormal data entry in the abnormal configuration dataset for each node within a specified time period, an "abnormal duration - abnormality severity" mapping rule is established, converting the abnormal duration into an "abnormality severity index" of 0-100. For example, a basic threshold is set (e.g., ≤30 minutes for "low", 30-120 minutes for "medium", >120 minutes for "high"), and standardized using the formula (Abnormality Level = Abnormal Duration / Maximum Historical Duration × 100). The duration is weighted based on the abnormality type (mandatory / suggestive / dynamic). For example, the duration weight of mandatory abnormalities is multiplied by 1.5 (e.g., a 15-minute mandatory abnormality is equivalent to a 22.5-minute ordinary abnormality), and the weight of suggestive abnormalities is multiplied by 0.8, increasing the severity value of high-risk abnormalities. The abnormality level is directly calculated by weighting the abnormality duration and type. Node importance is prioritized so that abnormalities affecting core business nodes have higher priority than those affecting peripheral nodes, and the impact scope is prioritized so that abnormalities affecting 1000+ users have higher priority than those affecting only internal processes. Based on the severity of the anomaly, the scope of its impact, and the weight of each node's importance level, priority scores are calculated for each anomaly. The anomaly data is then sorted in descending order of priority score. Thresholds can be set as needed (e.g., retaining only the top 20% of high-priority anomalies), or a target anomaly configuration dataset can be generated by grouping by level (high / medium / low priority). By determining the anomaly severity based on its duration, the system can accurately identify the anomaly data that has the greatest impact on the system and requires the highest priority. This allows operations and maintenance personnel to focus on resolving the most urgent and important anomalies, improving the efficiency and quality of problem-solving and ensuring the stable operation of the system.

[0033] Taking the abnormal management of the hospital's intensive care unit system as an example, from 08:00 to 09:00 on June 21, 2025, abnormality X: Patient A's ventilator showed "abnormal oxygen concentration" for 8 minutes; abnormality Y: Patient B's monitor showed "heart rate curve fluctuation" warning for 30 minutes. Abnormality X: low degree of abnormality, abnormality Y: high degree of abnormality. If the abnormality level (60%) + equipment life support level (30%) + severity of the affected patient's condition (10%), then the calculated priority score of abnormality X is greater than the priority score of abnormality Y, and abnormality X is ranked before abnormality Y.

[0034] In this embodiment, multidimensional configuration data is processed based on a preset anomaly filtering rule base to filter out anomaly configuration datasets, which helps to improve the real-time performance and accuracy of anomaly configuration data identification. The anomaly data in the anomaly configuration dataset is prioritized to obtain the target anomaly configuration dataset, so that more accurate configuration data management of big data clusters can be achieved in the future, thereby improving the efficiency of configuration data management of big data clusters.

[0035] S203: Input the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction.

[0036] In step S203, anomaly propagation path prediction refers to analyzing the interrelationships and dependencies between components (nodes) in the system to predict which paths an anomaly might propagate to other nodes after an anomaly occurs at a certain node, as well as the order and scope of propagation. Corresponding anomaly indicator prediction refers to predicting the timing, extent, and trend of potential anomalies in other related indicators based on the current anomaly. For example, a significant fluctuation in the price of a financial stock might lead to a decline in the net asset value of a related fund, allowing for timely measures such as portfolio adjustments and risk hedging. By inputting the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks, the graph structure features output by the graph neural network are fused with the time series features output by the deep learning network to form fused anomaly correlation features. Then, based on these fused anomaly correlation features, the anomaly prediction model is used to predict the anomaly propagation path and related anomaly indicators.

[0037] In one embodiment of the invention, the target anomaly configuration dataset is input into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, including: The target abnormal configuration dataset is transformed to obtain graph structure configuration data; The graph structure data is feature-extracted by the graph neural network in the anomaly prediction model to obtain graph structure features; The deep learning network in the anomaly prediction model is used to extract features from the target anomaly configuration dataset to obtain time series features. The graph structure features and the non-time series features are fused based on an attention mechanism to obtain fused abnormal correlation features; The fused anomaly correlation features are input into the anomaly prediction model for prediction, and the anomaly prediction result is obtained.

[0038] Specifically, by defining each component (such as server, service, database, etc.) in the target anomaly configuration dataset as a node in a graph, and defining the dependencies, data flows, or communication links between components as edges, each node and edge is assigned corresponding attributes. For example, node attributes may include component type, configuration parameters, anomaly type, etc.; edge attributes may include dependency strength, data transmission direction, etc. Then, a graph database or graph processing tool (such as NetworkX, Neo4j) is used to connect the nodes and edges to construct a complete graph structure configuration data. The graph structure configuration data is input into a graph neural network, and through operations such as graph convolution and graph attention, the embedded representations of nodes and edges are extracted. These embedded representations can capture the local and global structural information in the graph. Graph structure features, such as node centrality, community structure, and the possibility of anomaly propagation paths, are extracted from the embedded representations. The target anomaly configuration dataset is input into a deep learning network (such as LSTM, Transformer), and time series features, such as trends, periodicity, and abrupt change points, are extracted through the network's hidden layers. These features reflect the changing patterns of anomalies over time. An attention mechanism is used to calculate the weights between graph structure features and time series features, determining the importance of different features to the final prediction result. Based on the attention weights, the graph structure features and time series features are weighted and fused to obtain fused anomaly correlation features. This fusion can comprehensively consider the system's topology and dynamic behavior, improving the expressive power of the features. The fused anomaly correlation features are then input into the anomaly prediction model, outputting anomaly prediction results, including anomaly propagation path prediction and correlation anomaly index prediction. The anomaly prediction model can be a classification model or a regression model; this application does not impose any limitations on this. By combining graph structure features and time series features, the model can simultaneously utilize the system's topology and dynamic behavior information, thereby achieving more accurate and comprehensive predictions of anomaly configuration data, enhancing adaptability to complex scenarios, and reducing the risk of a single model through the complementarity of multiple network models, thus enhancing the robustness of the prediction model.

[0039] In this embodiment, by inputting the target abnormal configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks, the obtained anomaly prediction results are made more accurate and comprehensive, enhancing the anomaly early warning capability and thus providing a more precise quantitative basis for configuration data management.

[0040] S204: Based on the anomaly prediction results and the type of the target anomaly configuration dataset, manage the target anomaly configuration dataset using preset alarm repair rules.

[0041] In step S204, different types of abnormal configuration datasets (such as network configuration anomalies, server configuration anomalies, software configuration anomalies, etc.) are classified, and different alarm thresholds and repair priorities are set for each type. Based on the anomaly severity index in the anomaly prediction results, the alarm and repair rules are further refined. It should be noted that the preset alarm and repair rules can be set according to actual conditions, and this application does not impose any limitations on this. A real-time monitoring mechanism is established to continuously match the target abnormal configuration dataset with the preset alarm and repair rules. When anomalies in the dataset meet the alarm conditions in the rules, the corresponding alarm is immediately triggered. For example, when an abnormal increase in server CPU utilization is detected and reaches the preset red alarm threshold, the system automatically issues a red alarm notification and sends alarm notifications to maintenance personnel through multiple channels, such as SMS, email, and instant messaging tools, to ensure that maintenance personnel receive the alarm information in a timely manner. Then, based on the alarm level and anomaly type, the corresponding repair strategy is selected from the preset alarm and repair rule library. For example, for network configuration anomalies at the red alarm level, the repair strategy of immediately restarting the network device or switching to a backup network is selected. After the remediation measures are implemented, the status of the target abnormal configuration dataset is continuously monitored to evaluate the remediation effect. If the remediation effect is not ideal, the cause of the anomaly needs to be re-analyzed, the remediation strategy adjusted, and the remediation operation executed again.

[0042] In one embodiment of the invention, the target abnormal configuration dataset is managed using preset alarm repair rules based on the anomaly prediction result and the type of the target abnormal configuration dataset, including: Based on the anomaly prediction results, the target anomaly configuration dataset is compared with the corresponding standard configuration data to determine whether there is any data change. If there is changed data, determine the current warning level of each abnormal data in the abnormal configuration dataset; Determine whether the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level; If the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level, then a corresponding target repair plan is determined according to the type of the target abnormal configuration dataset. Based on the target repair plan, the target abnormal configuration dataset is repaired to obtain the repaired target abnormal configuration dataset.

[0043] Specifically, the results output by the anomaly prediction model are compared item by item with the standard configuration data of the target system (such as normal transaction parameters of a financial system or standard operating parameters of medical equipment) to determine whether there is any changed data. This changed data includes, but is not limited to, newly added configuration items, deleted configuration items, and modifications to configuration parameters. If changed data exists, a current warning level (e.g., low, medium, high, emergency) is assigned to each anomaly based on the severity, scope of impact, and characteristics of the business scenario. The current warning level of each anomaly is then compared with a preset warning level. It should be noted that the preset warning level is set based on the business's risk tolerance. For example, the preset warning level for anomalies involving fund security in a financial system (such as large-value transaction anomalies) is "medium warning," and exceeding this level triggers an emergency handling process. When the current warning level of any anomaly in the anomaly configuration dataset exceeds the preset warning level, a corresponding repair solution (e.g., parameter reset, service restart, module replacement) is matched according to the type of the target anomaly configuration dataset (e.g., network configuration of a financial system or software parameters of medical equipment), and the repair operation is executed to obtain the repaired target anomaly configuration dataset. By following the steps above, a large amount of abnormal configuration data can be quickly identified and processed, avoiding delays caused by manual investigation, greatly improving the efficiency of anomaly handling, and thus ensuring that configuration data is managed efficiently.

[0044] In one embodiment of the invention, determining a corresponding target remediation scheme based on the type of the target anomaly configuration dataset includes: Based on the type of the target abnormal configuration dataset, the target abnormal configuration dataset is processed using a three-dimensional localization method to determine the cause of the abnormality; Based on the cause of the anomaly, an initial repair plan is generated using a pre-set contingency plan database retrieval and dynamic adaptation adjustment method. The initial repair plan was rehearsed in a sandbox environment, and it was determined whether the evaluation success rate of the initial repair plan met the preset evaluation success rate. If the success rate of the initial repair plan meets the preset success rate, then the initial repair plan will be used as the target repair plan. If the success rate of the initial repair plan does not meet the preset success rate, the initial repair plan needs to be regenerated.

[0045] Specifically, the three-dimensional localization method includes spatial, temporal, and relational dimensions. The spatial dimension divides the system architecture into different spatial layers based on the type of the target abnormal configuration dataset. The temporal dimension records the time information of the anomaly, including the start time, duration, and frequency. The relational dimension analyzes the relationship between the target abnormal configuration dataset and other relevant configuration datasets. Based on the type of the target abnormal configuration dataset, the abnormal information is mapped to the three dimensions of the three-dimensional localization method. By integrating the information from these three dimensions and conducting data analysis, the cause of the anomaly is determined. Then, based on the cause, a pre-set contingency plan database (containing...) is retrieved. The system stores historical effective repair solutions and matches modified solutions for similar cases. If the matching results are not unique or not fully compatible, an initial repair solution is generated through dynamic adaptation and adjustment methods (such as parameter fine-tuning and solution combination). The initial repair solution is simulated and executed in a sandbox environment isolated from the production environment. The success rate of the solution is evaluated through indicator monitoring (such as repair time, resource consumption, and anomaly reproduction rate), and it is determined whether the evaluation success rate of the initial repair solution meets the preset evaluation success rate. If the evaluation success rate meets the standard, the initial solution is directly used as the target repair solution. If it does not meet the standard, the system returns to the contingency plan library to re-search or adjust the adaptation logic and regenerate the initial repair solution until a solution that meets the requirements is generated. It should be noted that the preset evaluation success rate can be set according to the actual situation, and this application does not impose any restrictions on it. By analyzing the target anomaly configuration dataset from three dimensions—space, time, and correlation—through the above steps, we can comprehensively and deeply uncover the causes of anomalies, avoiding the limitations of single-dimensional analysis, improving the accuracy of anomaly cause localization, and conducting pre-rehearsal and evaluation of the initial remediation plan in the sandbox environment. This allows us to identify problems and shortcomings in the plan in advance, make timely adjustments and optimizations, ensure that the target remediation plan can be effectively implemented in the production environment, and improve the remediation success rate.

[0046] In one embodiment of the invention, after managing the target abnormal configuration dataset using preset alarm repair rules, the process includes: Analyze the data change information of the repaired target abnormal configuration dataset. If the data change information meets the policy adjustment conditions, adjust the configuration data management policy under the current business scenario to obtain the adjusted target configuration data management policy. The system uses a preset feedback mechanism to provide feedback on the repaired target abnormal configuration dataset and to provide feedback on the adjustment of the target configuration data management strategy.

[0047] Specifically, data change information can include new business requirements and regulatory changes. By analyzing the data change information of the repaired target abnormal configuration dataset, the data management strategy can be automatically adjusted. For example, based on new requirements, it may be necessary to adjust the data storage format and field requirements. A comprehensive analysis of the current configuration data management strategy is conducted. If problems or deficiencies exist in the current strategy, adjustments are made based on the data change information and strategy analysis results. The adjusted target configuration data management strategy is then validated through simulation testing or small-scale pilot programs to evaluate the effectiveness of the strategy's implementation. This allows for further optimization and improvement of the strategy until it meets business requirements and system objectives, resulting in the adjusted target configuration data management strategy. A preset feedback mechanism provides feedback on the repaired target abnormal configuration dataset, and the adjusted target data management strategy is also adjusted through this mechanism to ensure continuous optimization of data quality throughout the process. Through these steps, by analyzing the repaired data change information and adjusting the configuration data management strategy in a timely manner, the system can better adapt to actual operating conditions, promptly identify and resolve potential problems, improve the overall system performance and response speed, and thus ensure the flexibility and automation of configuration data management.

[0048] In this embodiment, by using preset alarm repair rules to efficiently manage the target abnormal configuration dataset based on the anomaly prediction results and the type of the target abnormal configuration dataset, the workload of manual intervention is reduced, the efficiency and accuracy of configuration data management of the big data cluster are improved, and business development is promoted.

[0049] In summary, this invention provides a method, apparatus, device, and medium for configuration data management in a big data cluster. By acquiring multi-dimensional configuration data from each node in the big data cluster, processing the multi-dimensional configuration data based on a preset anomaly filtering rule base, filtering out anomaly configuration datasets, and prioritizing the anomaly data in the anomaly configuration datasets to obtain a target anomaly configuration dataset, this target anomaly configuration dataset is input into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results. These anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. Based on the anomaly prediction results and the type of the target anomaly configuration dataset, preset alarm repair rules are used to manage the target anomaly configuration dataset. Therefore, this application improves the efficiency of configuration data management in big data clusters by inputting the filtered and sorted target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, and then combining this with the type of the target anomaly configuration dataset to manage it more efficiently using preset alarm repair rules.

[0050] Please see Figure 3 , Figure 3 This is a schematic diagram of the configuration data management device for a big data cluster provided in an embodiment of the present invention. This configuration data management device for a big data cluster corresponds one-to-one with the configuration data management method for the big data cluster in the above embodiments. Please refer to [link / reference] for details. Figure 2 as well as Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 3 The configuration data management device 30 for the big data cluster includes: an acquisition module 31, a filtering module 32, an acquisition module 33, and a management module 34.

[0051] Module 31 is used to acquire multi-dimensional configuration data for each node in the big data cluster; The filtering module 32 is used to process the multidimensional configuration data based on a preset abnormal filtering rule base, filter out the abnormal configuration dataset, and sort the abnormal data in the abnormal configuration dataset by priority to obtain the target abnormal configuration dataset. The module 33 is used to input the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. The management module 34 is used to manage the target anomaly configuration dataset according to the anomaly prediction results and the type of the target anomaly configuration dataset, using preset alarm repair rules.

[0052] Optionally, the filtering module 32 described above is specifically used for: Based on the first anomaly identifier of the mandatory rule, filter out the first abnormal configuration data that matches the first anomaly identifier from the multidimensional configuration data; Based on the second anomaly identifier of the suggested rule, filter out second abnormal configuration data that matches the second anomaly identifier from the multidimensional configuration data; Based on the third anomaly identifier of the dynamic adaptive rule, filter out the third anomaly configuration data that matches the third anomaly identifier from the multidimensional configuration data; The first abnormal configuration data, the second abnormal configuration data, and the third abnormal configuration data are integrated to obtain an abnormal configuration dataset.

[0053] Optionally, the filtering module 32 described above is also used for: Obtain the duration of each abnormal data item in the abnormal configuration dataset for each node within a specified time period; The degree of abnormality of each abnormal data item is determined based on the duration of each abnormality. Based on the degree of abnormality, the scope of impact of the abnormality, and the importance level of the nodes, the abnormal data in the abnormal configuration dataset are prioritized to obtain the target abnormal configuration dataset.

[0054] Optionally, the module 33 obtained above is specifically used for: The target abnormal configuration dataset is transformed to obtain graph structure configuration data; The graph structure data is feature-extracted by the graph neural network in the anomaly prediction model to obtain graph structure features; The deep learning network in the anomaly prediction model is used to extract features from the target anomaly configuration dataset to obtain time series features. The graph structure features and the non-time series features are fused based on an attention mechanism to obtain fused abnormal correlation features; The fused anomaly correlation features are input into the anomaly prediction model for prediction, and the anomaly prediction result is obtained.

[0055] Optionally, the aforementioned management module 34 is specifically used for: Based on the anomaly prediction results, the target anomaly configuration dataset is compared with the corresponding standard configuration data to determine whether there is any data change. If there is changed data, determine the current warning level of each abnormal data in the abnormal configuration dataset; Determine whether the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level; If the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level, then a corresponding target repair plan is determined according to the type of the target abnormal configuration dataset. Based on the target repair plan, the target abnormal configuration dataset is repaired to obtain the repaired target abnormal configuration dataset.

[0056] Optionally, the management module 34 described above is also used for: Based on the type of the target abnormal configuration dataset, the target abnormal configuration dataset is processed using a three-dimensional localization method to determine the cause of the abnormality; Based on the cause of the anomaly, an initial repair plan is generated using a pre-set contingency plan database retrieval and dynamic adaptation adjustment method. The initial repair plan was rehearsed in a sandbox environment, and it was determined whether the evaluation success rate of the initial repair plan met the preset evaluation success rate. If the success rate of the initial repair plan meets the preset success rate, then the initial repair plan will be used as the target repair plan. If the success rate of the initial repair plan does not meet the preset success rate, the initial repair plan needs to be regenerated.

[0057] Optionally, the generation module 34 described above is specifically used for: Analyze the data change information of the repaired target abnormal configuration dataset. If the data change information meets the policy adjustment conditions, adjust the configuration data management policy under the current business scenario to obtain the adjusted target configuration data management policy. The system uses a preset feedback mechanism to provide feedback on the repaired target abnormal configuration dataset and to provide feedback on the adjustment of the target configuration data management strategy.

[0058] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0059] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 The diagram shows only one of the following: a memory and a computer program stored in the memory and capable of running on at least one processor. When the processor executes the computer program, it implements the steps in the above embodiments of the configuration data management method for any big data cluster.

[0060] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input systems.

[0061] In one embodiment, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor in a computer device, enables the computer device to perform the steps of any embodiment of the configuration data management method for a big data cluster disclosed in this invention, which will not be repeated here. The computer-readable storage medium may be non-volatile or volatile.

[0062] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0063] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, cooperative applications, bootloader, data, and other programs, such as program code of computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0064] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0065] Those familiar with the technical field will understand that, for ease of description and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0066] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for managing configuration data in a big data cluster, characterized in that, include: Obtain multi-dimensional configuration data for each node in a big data cluster; The multidimensional configuration data is processed based on a preset anomaly filtering rule base to filter out anomaly configuration datasets, and the anomaly data in the anomaly configuration datasets are prioritized to obtain the target anomaly configuration dataset. The target anomaly configuration dataset is input into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. Based on the anomaly prediction results and the type of the target anomaly configuration dataset, the target anomaly configuration dataset is managed using preset alarm repair rules.

2. The configuration data management method for a big data cluster as described in claim 1, characterized in that, The preset anomaly filtering rule base includes mandatory rules, suggested rules, and dynamic adaptive rules. The process of processing the multidimensional configuration data based on the preset anomaly filtering rule base to filter out the anomaly configuration dataset includes: Based on the first anomaly identifier of the mandatory rule, filter out the first abnormal configuration data that matches the first anomaly identifier from the multidimensional configuration data; Based on the second anomaly identifier of the suggested rule, filter out second abnormal configuration data that matches the second anomaly identifier from the multidimensional configuration data; Based on the third anomaly identifier of the dynamic adaptive rule, filter out the third anomaly configuration data that matches the third anomaly identifier from the multidimensional configuration data; The first abnormal configuration data, the second abnormal configuration data, and the third abnormal configuration data are integrated to obtain an abnormal configuration dataset.

3. The configuration data management method for a big data cluster as described in claim 1, characterized in that, The step of prioritizing and sorting the abnormal data in the abnormal configuration dataset to obtain the target abnormal configuration dataset includes: Obtain the duration of each abnormal data item in the abnormal configuration dataset for each node within a specified time period; The degree of abnormality of each abnormal data item is determined based on the duration of each abnormality. Based on the degree of abnormality, the scope of impact of the abnormality, and the importance level of the nodes, the abnormal data in the abnormal configuration dataset are prioritized to obtain the target abnormal configuration dataset.

4. The configuration data management method for a big data cluster as described in claim 1, characterized in that, The step of inputting the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results includes: The target abnormal configuration dataset is transformed to obtain graph structure configuration data; The graph structure data is feature-extracted by the graph neural network in the anomaly prediction model to obtain graph structure features; The deep learning network in the anomaly prediction model is used to extract features from the target anomaly configuration dataset to obtain time series features. The graph structure features and the non-time series features are fused based on an attention mechanism to obtain fused abnormal correlation features; The fused anomaly correlation features are input into the anomaly prediction model for prediction, and the anomaly prediction result is obtained.

5. The configuration data management method for a big data cluster as described in claim 1, characterized in that, The step of managing the target abnormal configuration dataset using preset alarm repair rules based on the anomaly prediction results and the type of the target abnormal configuration dataset includes: Based on the anomaly prediction results, the target anomaly configuration dataset is compared with the corresponding standard configuration data to determine whether there is any data change. If there is changed data, determine the current warning level of each abnormal data in the abnormal configuration dataset; Determine whether the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level; If the current warning level of each abnormal data in the abnormal configuration dataset is greater than the preset warning level, then a corresponding target repair plan is determined according to the type of the target abnormal configuration dataset. Based on the target repair plan, the target abnormal configuration dataset is repaired to obtain the repaired target abnormal configuration dataset.

6. The configuration data management method for a big data cluster as described in claim 5, characterized in that, The step of determining the corresponding target remediation plan based on the type of the target anomaly configuration dataset includes: Based on the type of the target abnormal configuration dataset, the target abnormal configuration dataset is processed using a three-dimensional localization method to determine the cause of the abnormality; Based on the cause of the anomaly, an initial repair plan is generated using a pre-set contingency plan database retrieval and dynamic adaptation adjustment method. The initial repair plan was rehearsed in a sandbox environment, and it was determined whether the evaluation success rate of the initial repair plan met the preset evaluation success rate. If the success rate of the initial repair plan meets the preset success rate, then the initial repair plan will be used as the target repair plan. If the success rate of the initial repair plan does not meet the preset success rate, the initial repair plan needs to be regenerated.

7. The configuration data management method for a big data cluster as described in claim 1, characterized in that, After managing the target abnormal configuration dataset using preset alarm repair rules, the process includes: Analyze the data change information of the repaired target abnormal configuration dataset. If the data change information meets the policy adjustment conditions, adjust the configuration data management policy under the current business scenario to obtain the adjusted target configuration data management policy. The system uses a preset feedback mechanism to provide feedback on the repaired target abnormal configuration dataset and to provide feedback on the adjustment of the target configuration data management strategy.

8. A configuration data management device for a big data cluster, characterized in that, include: The acquisition module is used to acquire multi-dimensional configuration data for each node in the big data cluster. The filtering module is used to process the multidimensional configuration data based on a preset abnormal filtering rule base, filter out the abnormal configuration dataset, and sort the abnormal data in the abnormal configuration dataset by priority to obtain the target abnormal configuration dataset. The module is used to input the target anomaly configuration dataset into an anomaly prediction model based on graph neural networks and deep learning networks to obtain anomaly prediction results, wherein the anomaly prediction results include anomaly propagation path prediction and associated anomaly indicator prediction. The management module is used to manage the target anomaly configuration dataset according to the anomaly prediction results and the type of the target anomaly configuration dataset, using preset alarm repair rules.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the configuration data management method for the big data cluster as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the configuration data management method for the big data cluster as described in any one of claims 1 to 7.

Images