IT informatization double-model operation and maintenance management method based on multilayer data lake

Abstract

The invention discloses an IT informatization double-model operation and maintenance management method based on a multi-layer data lake, and the method comprises the following steps: collecting IT operation and maintenance related data from a plurality of data sources in real time, and carrying out the correlation processing, and forming a multi-layer operation and maintenance data network; the multi-layer operation and maintenance data network is embedded into a data lake through a layering technology to form a multi-layer operation and maintenance data lake, and rapid access and calling of large-scale operation and maintenance data are carried out through a multi-layer interface; performing big data analysis on the called data, completing accurate filtering of the data, extraction of key features and deep combination, and obtaining a plurality of key performance evaluation state quantity sets; inputting the state quantity set into an operation and maintenance identification model to identify potential factors which may cause system problems and performance bottlenecks; automatically adjusting IT resource configuration through an operation and maintenance adjustment model based on the potential factors so as to optimize the system performance and improve the resource utilization rate; and dynamically displaying the identification result and the adjustment result. The operation and maintenance efficiency and the system stability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent operation and maintenance management technology, and in particular to an IT informatization dual-model operation and maintenance management method based on a multi-layer data lake. Background Art

[0002] With the rapid development of information technology and the advancement of enterprise digital transformation, the complexity of IT systems continues to increase. Traditional IT operation and maintenance management methods can no longer meet the current needs of rapid growth in multi-source and multi-level data. The main challenges faced in IT operation and maintenance management include: diversified data sources, huge data volumes and high real-time requirements, difficulty in identifying system performance bottlenecks, and low efficiency in resource allocation optimization. The existence of these problems makes it difficult to improve the efficiency and reliability of IT operation and maintenance management, seriously affecting the stability and operating efficiency of enterprise information systems.

[0003] Existing IT operation and maintenance management methods mainly focus on analysis and adjustment based on rules or single models. Although they can identify and solve some system failures and performance bottlenecks, they still have the following problems: (1) Difficulty integrating multi-source heterogeneous data: The sources of IT operation and maintenance data are complex, including both structured data (such as performance indicators in the database) and unstructured data (such as log files and operation and maintenance documents). Existing methods are not capable of integrating multi-source heterogeneous data and cannot efficiently use the full amount of data for analysis.

[0004] (2) Low accuracy in identifying system performance problems: Traditional operation and maintenance management tools mainly rely on empirical rules or simple models to analyze and identify performance problems. They lack in-depth exploration of potential factors of system problems, resulting in insufficient accuracy and timeliness in problem diagnosis.

[0005] (3) Unintelligent resource allocation optimization: Most existing operation and maintenance management methods rely on static resource allocation strategies and lack the ability to dynamically perceive and adjust the system's operating status, making it difficult to maximize resource utilization.

[0006] (4) Lack of scalability and real-time support: With the expansion of business scale and the surge in data volume, the processing capacity of traditional IT operation and maintenance management tools is difficult to expand and cannot meet the needs of real-time operation and maintenance management. Summary of the invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention provides an IT information technology dual-model operation and maintenance management method based on a multi-layer data lake, which is used to solve the technical problems existing in the existing IT operation and maintenance management, such as the difficulty in integrating multi-source heterogeneous data, low accuracy in identifying system problems, insufficient efficiency in optimizing resource allocation, and limited real-time support capabilities, thereby forming an efficient, accurate, and intelligent IT information technology operation and maintenance management method, which significantly improves the operation and maintenance efficiency and system stability.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows: An IT information-based dual-model operation and maintenance management method based on a multi-layer data lake includes the following steps: Collect IT operation and maintenance related data from multiple data sources in real time, and form a multi-layer operation and maintenance data network after correlation processing; The multi-layer operation and maintenance data network is embedded into the data lake using layered technology to form a multi-layer operation and maintenance data lake, and the multi-layer interfaces of the multi-layer operation and maintenance data lake are used to quickly access and retrieve large-scale operation and maintenance data; Perform big data analysis on the retrieved large-scale operation and maintenance data, complete accurate data filtering, extract key features and deep integration, and obtain several key performance evaluation state quantity sets; Inputting the plurality of key performance evaluation state quantity sets into the operation and maintenance identification model to identify potential factors that may cause system problems and performance bottlenecks; Automatically adjust IT resource configuration based on the potential factors through the operation and maintenance adjustment model to optimize system performance and improve resource utilization; Dynamically displaying identification results and adjustment results by associating the operation and maintenance identification model with the operation and maintenance adjustment model to support operation and maintenance decision-making; Among them, when collecting IT operation and maintenance related data from multiple data sources in real time, the Kafka operation and maintenance cluster receives the data, divides the data receiving groups and configures nodes to complete HDFS storage.

[0009] As a preferred embodiment of the present invention, when receiving data through the Kafka operation and maintenance cluster, dividing the data receiving group and configuring the node to complete the HDFS storage, it includes: The collected IT operation and maintenance related data is transmitted to the Kafka operation and maintenance cluster through Flume; Multiple instances in the Kafka operation and maintenance cluster correspond to different data sources respectively, and are used to receive IT operation and maintenance related data from each data source, and multiple instances are divided into multiple data receiving groups according to a preset number; Equip each data receiving group with a corresponding node and aggregate the collected data into multiple nodes; Each node is equipped with a corresponding Producer, and the collected data is sent back to the Kafka operation and maintenance cluster through the Producer for HDFS storage.

[0010] As a preferred embodiment of the present invention, when forming a multi-layer operation and maintenance data network, it includes: The time series data and text data in the Kafka operation and maintenance cluster are stored in the data storage method of [time_stamp][key1][value1]…[keyN][valueN]; The number of key-value pairs [key][value] at each time point corresponding to [time_stamp] is expandable, so that time series data and text data can be stored in table structures at different levels; Combined query conditions are assigned to the time series data and text data in the table structure, and aggregation calculations are performed to form the multi-layer operation and maintenance data network.

[0011] As a preferred implementation mode of the present invention, when forming a multi-layer operation and maintenance data lake, it includes: Encode the data in the table structures at different levels into numerical data sequences with the same structural form according to the preset semantic logic, upload them to the cloud server for dimensionless processing and standardization, and upload them to the cloud database through the gateway; The data sequence uploaded by the gateway is parsed and continuously monitored in real time through the cloud server; When the data sequence flow is detected to be interrupted, the data sequence is extracted through the cloud server and compared with the preset data format; When the comparison is passed, the corresponding database interface is connected according to the data type, and an identification number is assigned and converted into JSON format to obtain the actual data value, which is then converted into a triple through the mapping rules; The triples are matched with concepts and entities in the data lake database connected to the corresponding database interface to obtain a data lake middle platform, and a multi-layer API interface is provided for the data lake middle platform.

[0012] As a preferred implementation mode of the present invention, when performing quick access and retrieval, it includes: Determine the search area and target conditions according to the access and retrieval conditions, and obtain the smallest top-level data block that can completely contain the search area; Determine whether the top-level data block meets the target condition, if so, insert the data block index number into the linked list, if not, continue to search for branches of the top-level data block until all data blocks that meet the target condition are found; The indexes of all data blocks that meet the target conditions are inserted into a linked list in order from left to right and from top to bottom, and large-scale operation and maintenance data are quickly accessed and retrieved through the linked list.

[0013] As a preferred embodiment of the present invention, when completing accurate filtering of data, extraction of key features and deep integration, it includes: Filter the data using filtering rules to form a key matrix model for identifying the system's operating status; Using the Apriori algorithm to obtain frequent multinomial sets, and mining the association rules between data types and fault types in the key matrix model, to establish the several key performance evaluation state quantity sets; Among them, when mining association rules, it includes: A cluster analysis is performed on the relationship between different data types and fault types in the key matrix model, the shape coefficient and silhouette coefficient of the associated data curve are mined, the collection amount of the relevant data type is extracted, and a key performance evaluation state quantity set for different fault types is obtained.

[0014] As a preferred embodiment of the present invention, when identifying potential factors, it includes: The operation and maintenance identification model provides: association rule antecedent and the association rule , association rule antecedent Represents the attribute characteristics of IT information operation and maintenance data, and the latter term of the association rule Indicates whether there are system problems or performance bottlenecks. Indicates a system problem. Indicates a performance bottleneck; Based on the antecedent and consequent items of association rules and , get the support of performance bottleneck ; Based on the antecedent and consequent items of association rules and , get the support level of system problems ; Will The attribute features greater than the threshold are taken as the first latent factor; Will Attribute features greater than the threshold are used as the second latent factors.

[0015] As a preferred embodiment of the present invention, when automatically adjusting IT resource configuration, it includes: The operation and maintenance adjustment model determines the first control coefficient and the second control coefficient according to the first potential factor and the second potential factor, and controls the corresponding type of IT resource flow through the first control coefficient and the second control coefficient, as shown in Formula 3 and Formula 4: (3); In the formula, For the planned period The first latent factor corresponds to The first control quantity of IT resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The first control coefficient of IT resources; (4); In the formula, For the planned period The second latent factor corresponds to The second control quantity of IT-like resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The second control coefficient of IT-like resources.

[0016] As a preferred embodiment of the present invention, when automatically adjusting IT resource configuration, it also includes: The first control amount and the second control amount are approached to the maximum extent, and the first IT resource optimization configuration result and the second IT resource optimization configuration result are obtained, as shown in Formula 5 and Formula 6: (5); In the formula, For the planned period The first latent factor corresponds to The actual amount of IT resources controlled, is the number of IT resource types corresponding to the first potential factor, is the total number of time periods in the first plan, Optimize the configuration results for the first IT resource; (6); In the formula, For the planned period The second latent factor corresponds to The actual amount of IT resources controlled, is the number of IT resource types corresponding to the second potential factor, is the total number of time periods in the second plan, Optimize configuration results for secondary IT resources.

[0017] As a preferred embodiment of the present invention, when performing dynamic display, it includes: A graph is defined, and the vertices of the identified first potential factor, the second potential factor, the first IT resource optimization configuration result, and the second IT resource optimization configuration result are determined by the host IP, the collection directory, and the file name template, respectively, and the attribute information of the vertex is set; The vertices are used as source ends, and the corresponding transmission target ends are matched, and the transmission target ends are used as vertices of the next layer of the dynamic display graph one by one, and their attribute information is set, and they are connected with the previous layer; The next layer of source information is matched, and the matched source information is used as the vertex of the next layer until no source information is matched.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes an IT information dual-model operation and maintenance management method based on a multi-layer data lake. By building a multi-layer data lake architecture and combining an operation and maintenance identification model with an operation and maintenance adjustment model, the invention realizes efficient integration of multi-source heterogeneous data in complex IT systems, accurate identification of potential problems, and dynamic optimization configuration of resources. The invention has the following beneficial effects: (1) Efficient integration and management of multi-source heterogeneous data The present invention adopts a layered integration of a multi-layer operation and maintenance data lake architecture and a data network to uniformly store and manage structured and unstructured data from multiple sources. The multi-layer API interface enables rapid access and retrieval of large-scale data, significantly improving the processing efficiency and expansion capabilities of IT operation and maintenance data.

[0019] (2) Accurate identification of system problems and performance bottlenecks The present invention conducts big data analysis on large-scale operation and maintenance data through an operation and maintenance identification model, utilizes association rules and clustering algorithms to mine key factors of potential system problems and performance bottlenecks, and combines the generation of key performance evaluation state quantity sets to accurately identify the root causes that may cause system problems and improve the accuracy and timeliness of problem diagnosis.

[0020] (3) Intelligent resource optimization configuration The operation and maintenance adjustment model dynamically determines the control coefficient of IT resources based on the identified potential factors, and realizes the intelligent scheduling and configuration of IT resources through optimization algorithms. The optimization results effectively improve resource utilization, reduce the risk of system performance bottlenecks, and reduce manual intervention in the operation and maintenance process.

[0021] (4) Real-time operation and maintenance monitoring and dynamic decision support By combining the operation and maintenance identification model and the operation and maintenance adjustment model, the present invention can realize the dynamic display of operation and maintenance identification results and adjustment results, and adopts the method of visual dynamic display graphics to enable operation and maintenance personnel to intuitively understand the system operation status and adjustment effects, providing efficient support for real-time operation and maintenance decision-making.

[0022] (5) Powerful scalability and real-time performance The present invention combines a multi-layer data lake architecture with a cloud server to achieve dimensionless processing and standardized storage of data, as well as continuous monitoring and dynamic analysis of real-time data streams. When data anomalies are detected, the problem can be quickly located and corresponding operations can be performed. It has good real-time and scalability, meeting the dynamic operation and maintenance requirements of large-scale IT systems.

[0023] (6) Comprehensive support for intelligent operation and maintenance of complex IT systems Based on the multi-layer data lake architecture, the present invention makes full use of big data analysis and intelligent algorithms to form a full-process operation and maintenance management solution from data collection and analysis to resource optimization. It is suitable for complex and diverse IT operation and maintenance scenarios, such as cloud computing platforms, distributed storage systems, large enterprise data centers, etc., and effectively improves the stability and operation and maintenance efficiency of the system. The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a step diagram of the IT informatization dual-model operation and maintenance management method based on a multi-layer data lake provided by the present invention. DETAILED DESCRIPTION

[0025] The IT informatization dual-model operation and maintenance management method based on the multi-layer data lake provided by the present invention is as follows: Figure 1 As shown, the following steps are included: Step S1: collect IT operation and maintenance related data from multiple data sources in real time, and form a multi-layer operation and maintenance data network after correlation processing; Step S2: embed the multi-layer operation and maintenance data network into the data lake using layered technology to form a multi-layer operation and maintenance data lake, and use the multi-layer interfaces of the multi-layer operation and maintenance data lake to quickly access and retrieve large-scale operation and maintenance data; Step S3: Perform big data analysis on the retrieved large-scale operation and maintenance data, complete accurate data filtering, key feature extraction and deep integration, and obtain several key performance evaluation state quantity sets; Step S4: inputting a number of key performance evaluation state quantity sets into the operation and maintenance identification model to identify potential factors that may cause system problems and performance bottlenecks; Step S5: Automatically adjust IT resource configuration through the operation and maintenance adjustment model based on the potential factors to optimize system performance and improve resource utilization; Step S6: Dynamically display the identification results and adjustment results by associating the operation and maintenance identification model and the operation and maintenance adjustment model to support operation and maintenance decision-making.

[0026] In the above step S1, when collecting IT operation and maintenance related data from multiple data sources in real time, it includes: The collected IT operation and maintenance related data is transmitted to the Kafka operation and maintenance cluster through Flume; Multiple instances in the Kafka operation and maintenance cluster correspond to different data sources respectively, which are used to receive IT operation and maintenance related data from each data source, and multiple instances are divided into multiple data receiving groups according to the preset number; Equip each data receiving group with a corresponding node and aggregate the collected data into multiple nodes; Each node is equipped with a corresponding Producer, and the collected data is sent back to the Kafka operation and maintenance cluster through the Producer for HDFS storage; Among them, the configuration of Flume is realized through SpoolDirectorySource, and the type of Flume is defined as spooldir, and the data input path for receiving IT operation and maintenance related data is specified; The Kafka operation and maintenance cluster manages cluster configuration through Zookeeper and performs rebalance when the leader is elected or the data receiving group changes.

[0027] Furthermore, when performing HDFS storage, it includes: When the data collected by the Producer enters the Kafka operation and maintenance cluster, the data is segmented through the HDFS Client, and the NameNode is used as the master node. By interacting with the NameNode, the namespace and data block mapping information of the HDFS are managed, the replication strategy is configured, requests are processed, and file location information is generated; Use DataNode as a slave node to store data, read and write data, and report storage information to NameNode; The Secondary NameNode assists the NameNode in running, regularly merges fsimage and fsedits and pushes them to the NameNode, and assists the NameNode in recovery.

[0028] In the above step S1, when forming a multi-layer operation and maintenance data network, it includes: The data storage method of [time_stamp][key1][value1][key2][value2][key3][value3]…[keyN][valueN] is used for time series data and text data in the Kafka operation and maintenance cluster. The number of key-value pairs [key][value] at each time point corresponding to [time_stamp] is expandable, so that time series data and text data can be stored in table structures at different levels; Assign combined query conditions to the time series data and text data in the table structure, and perform aggregate calculations to form a multi-layer operation and maintenance data network; Among them, the multi-layer operation and maintenance data network adopts a consistent or inconsistent mode in the form of data writing, and realizes data display in different dimensions and forms through the coordination of consistent and inconsistent modes; Aggregate calculations include: SELECT key2, sum(value1) / sum(value2) FROM metric WHERE key1 = "A" AND timestamp >= B AND timestamp < C GROUP BY key2.

[0029] Specifically, [time_stamp] is a timestamp used to identify the recording time of the data. Each pair of [key][value] represents an attribute and a value. The number of key-value pairs is dynamic and can be expanded to [keyN][valueN] according to actual needs.

[0030] In the above step S2, when forming a multi-layer operation and maintenance data lake, it includes: Encode the data in the table structures at different levels into numerical data sequences with the same structure according to the preset semantic logic, and upload them to the cloud server; Use cloud servers to perform dimensionless processing and standardization on numerical data sequences; After the dimensionless and standardized data sequences are collected and uploaded to the cloud database through the gateway, the data sequences uploaded by the gateway are parsed by the cloud server, and the data sequence flow information is continuously monitored in real time; When the data sequence flow is detected to be interrupted, the data sequence is extracted through the cloud server and compared with the preset data format; When the data format comparison is passed, the corresponding database interface is connected according to the data type, and an identification number is assigned and converted into JSON format to obtain the actual data value, which is then converted into a triple through the mapping rules; The concepts and entities in the data lake database connected to the triples are matched with the corresponding database interfaces to obtain the data lake middle platform, and a multi-layer API interface is provided for the data lake middle platform for subsequent big data analysis and dynamic display.

[0031] Furthermore, when performing dimensionless processing, it includes: For data that needs to be reduced in dimension using the PCA algorithm or data that needs to use distance to measure similarity, methods including zero-mean normalization are used to make data of different dimensions dimensionless; When standardizing, include: For data that does not involve distance measurement, covariance calculation, or data that does not conform to a normal distribution, methods including maximum and minimum standardization are used to limit the matrix values ​​to a preset range.

[0032] In the above step S2, when quickly accessing and retrieving large-scale operation and maintenance data, it includes: Determine the search area and target conditions according to the access and retrieval conditions, and obtain the smallest top-level data block that can completely contain the search area; Determine whether the top-level data block meets the target condition. If so, insert the data block index number into the linked list. If not, continue searching for the branch of the top-level data block until all data blocks that meet the target condition are found. The indexes of all data blocks that meet the target conditions are inserted into the linked list from left to right and from top to bottom, and the linked list is used to quickly access and retrieve large-scale operation and maintenance data.

[0033] In the above step S3, when completing the accurate filtering of data, extraction of key features and deep integration, it includes: Filter the data using filtering rules to form a key matrix model for identifying the system's operating status; The Apriori algorithm is used to obtain frequent multinomial sets, and the association rules between data types and fault types in the key matrix model are mined to establish several key performance evaluation state quantity sets; When filtering data using filtering rules, it includes: set up It is a collection of operation and maintenance data, providing a series of unique identifiers A transaction database consisting of transactions, each transaction Corresponding set A subset of operation and maintenance data on ,Right now , then the filtering rule is expressed as The logical implication of , ; Get the transaction database contains The percentage of transactions in the transaction database is filtered out, and the operation and maintenance data subsets whose percentage is less than a first preset value are obtained to obtain a pre-filtered operation and maintenance data set; Get the transaction database contains Transactions and inclusions , filtering out a subset of operation and maintenance data whose percentage is less than a second preset value, and forming a key matrix model; When mining association rules, include: Perform cluster analysis on the relationship between different data types and fault types in the key matrix model, mine the shape coefficient and contour coefficient of the associated data curve, extract the collection volume of related data types, and obtain the key performance evaluation state quantity set for different fault types; In the key matrix model, the horizontal axis is the data type and the vertical axis is the collection amount.

[0034] In the above step S4, when identifying potential factors, it includes: The operation and maintenance identification model provides: association rule antecedent and the association rule , association rule antecedent Represents the attribute characteristics of IT information operation and maintenance data, and the latter term of the association rule Indicates whether there are system problems or performance bottlenecks. Indicates a system problem. Indicates a performance bottleneck; Based on the antecedent of association rules and the association rule , get the support of performance bottleneck , as shown in Formula 1: (1); In the formula, For the The characteristic factor The support degree of the category attribute features for a certain type of performance bottleneck, For the The characteristic factor Category attribute features, For the The characteristic factor The total number of times the category attribute characteristics caused the operation to encounter performance bottlenecks, is the total number of runs; Based on the antecedent of association rules and the association rule , get the support level of system problems , as shown in Formula 2: (2); In the formula, For the The characteristic factor The support degree of the category attribute features for a certain type of system problem, For the The characteristic factor The total number of times the category attribute feature causes system problems to occur during operation; Support for performance bottlenecks The attribute features greater than the threshold are taken as the first latent factor; Support for system problems that will occur Attribute features greater than the threshold are used as the second latent factor.

[0035] Specifically, the attribute characteristics of IT information operation and maintenance data include: system performance data, log data, event and alarm data, configuration information data, user behavior data, data flow and network traffic data, monitoring and analysis data, automated operation and maintenance data, anomaly and problem diagnosis data, operation and maintenance management data, and security and compliance related data.

[0036] In the above step S5, when automatically adjusting IT resource configuration, it includes: The operation and maintenance adjustment model determines the first control coefficient and the second control coefficient according to the first potential factor and the second potential factor, and controls the corresponding type of IT resource flow through the first control coefficient and the second control coefficient, as shown in Formula 3 and Formula 4: (3); In the formula, For the planned period The first latent factor corresponds to The first control quantity of IT resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The first control coefficient of IT resources; (4); In the formula, For the planned period The second latent factor corresponds to The second control quantity of IT-like resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The second control coefficient of IT-like resources; The first control amount and the second control amount are approached to the maximum extent, and the first IT resource optimization configuration result and the second IT resource optimization configuration result are obtained, as shown in Formula 5 and Formula 6: (5); In the formula, For the planned period The first latent factor corresponds to The actual amount of IT resources controlled. is the number of IT resource types corresponding to the first potential factor, is the total number of time periods in the first plan, Optimize the configuration results for the first IT resource; (6); In the formula, For the planned period The second latent factor corresponds to The actual amount of IT resources controlled. is the number of IT resource types corresponding to the second potential factor, is the total number of time periods in the second plan, Optimize configuration results for secondary IT resources.

[0037] Specifically, various types of IT resource configuration include: computing resource configuration, storage resource configuration, network resource configuration, software resource configuration, security resource configuration, scheduling and automation resource configuration, energy resource configuration, user resource configuration, and high availability and disaster recovery resource configuration.

[0038] Furthermore, computing resource configuration includes: server resources, container resources, and cloud computing resources.

[0039] Storage resource configuration includes: storage capacity, distributed storage, backup storage, and database storage.

[0040] Network resource configuration includes: bandwidth, IP address, load balancing, firewall rules, and VPN configuration.

[0041] Software resource configuration includes: operating system, middleware, database, and application deployment.

[0042] Security resource configuration includes: identity authentication, permission management, security policy, logging and auditing.

[0043] Scheduling and automated resource configuration include: task scheduling, resource allocation strategy, and automated operation and maintenance.

[0044] Energy resource allocation includes: power supply allocation and heat dissipation resource allocation.

[0045] User resource configuration includes: account allocation, quota limit, user policy, and terminal device management.

[0046] High availability and disaster recovery resource configuration includes: redundant resources, backup strategies, and disaster recovery systems.

[0047] In the above step S6, when the dynamic display of the recognition result and the adjustment result is performed, it includes: A graph is defined, and the vertices of the identified first potential factor, the second potential factor, the first IT resource optimization configuration result, and the second IT resource optimization configuration result are determined by the host IP, the collection directory, and the file name template, respectively, and the attribute information of the vertex is set; Take the vertex as the source end and match the corresponding transmission target end, use the transmission target ends one by one as the vertices of the next layer of the dynamic display graph, set their attribute information, and connect them with the previous layer; Match the next layer of source information, and use the matched source information as the vertex of the next layer, until no source information can be matched; The node generated by the first latent factor is associated with the node generated by the first IT resource optimization configuration result, and the node generated by the second latent factor is associated with the node generated by the second IT resource optimization configuration result. The attribute information includes: font, color, and shape of the vertex.

[0048] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A dual-model operation and maintenance management method for IT informatization based on a multi-layer data lake, characterized in that: The following steps are involved: Collect IT operation and maintenance related data from multiple data sources in real time, and form a multi-layer operation and maintenance data network after correlation processing; The multi-layer operation and maintenance data network is embedded into the data lake using layered technology to form a multi-layer operation and maintenance data lake, and the multi-layer interfaces of the multi-layer operation and maintenance data lake are used to quickly access and retrieve large-scale operation and maintenance data; Perform big data analysis on the retrieved large-scale operation and maintenance data, complete accurate data filtering, extract key features and deep integration, and obtain several key performance evaluation state quantity sets; Inputting the plurality of key performance evaluation state quantity sets into the operation and maintenance identification model to identify potential factors that may cause system problems and performance bottlenecks; Automatically adjust IT resource configuration based on the potential factors through the operation and maintenance adjustment model to optimize system performance and improve resource utilization; Dynamically displaying identification results and adjustment results by associating the operation and maintenance identification model with the operation and maintenance adjustment model to support operation and maintenance decision-making; Among them, when collecting IT operation and maintenance related data from multiple data sources in real time, the Kafka operation and maintenance cluster receives the data, divides the data receiving groups and configures nodes to complete HDFS storage.

2. The IT informatization dual-model operation and maintenance management method based on a multi-layer data lake according to claim 1 is characterized in that: When receiving data through the Kafka operation and maintenance cluster, dividing the data receiving groups and configuring nodes to complete HDFS storage, it includes: The collected IT operation and maintenance related data is transmitted to the Kafka operation and maintenance cluster through Flume; Multiple instances in the Kafka operation and maintenance cluster correspond to different data sources respectively, and are used to receive IT operation and maintenance related data from each data source, and multiple instances are divided into multiple data receiving groups according to a preset number; Equip each data receiving group with a corresponding node and aggregate the collected data into multiple nodes; Each node is equipped with a corresponding Producer, and the collected data is sent back to the Kafka operation and maintenance cluster through the Producer for HDFS storage.

3. The IT information dual-model operation and maintenance management method based on a multi-layer data lake according to claim 2 is characterized in that: When forming a multi-layer operation and maintenance data network, it includes: The time series data and text data in the Kafka operation and maintenance cluster are stored in the data storage method of [time_stamp][key1][value1]…[keyN][valueN]; The number of key-value pairs [key][value] at each time point corresponding to [time_stamp] is expandable, so that time series data and text data can be stored in table structures at different levels; Combined query conditions are assigned to the time series data and text data in the table structure, and aggregation calculations are performed to form the multi-layer operation and maintenance data network.

4. The IT information dual-model operation and maintenance management method based on a multi-layer data lake according to claim 3 is characterized in that: When forming a multi-layer operation and maintenance data lake, it includes: Encode the data in the table structures at different levels into numerical data sequences with the same structural form according to the preset semantic logic, upload them to the cloud server for dimensionless processing and standardization, and upload them to the cloud database through the gateway; The data sequence uploaded by the gateway is parsed and continuously monitored in real time through the cloud server; When the data sequence flow is detected to be interrupted, the data sequence is extracted through the cloud server and compared with the preset data format; When the comparison is passed, the corresponding database interface is connected according to the data type, and an identification number is assigned and converted into JSON format to obtain the actual data value, which is then converted into a triple through the mapping rules; The triples are matched with concepts and entities in the data lake database connected to the corresponding database interface to obtain a data lake middle platform, and a multi-layer API interface is provided for the data lake middle platform.

5. The IT informatization dual-model operation and maintenance management method based on a multi-layer data lake according to any one of claims 1 to 4, characterized in that: When performing quick access and retrieval, it includes: Determine the search area and target conditions according to the access and retrieval conditions, and obtain the smallest top-level data block that can completely contain the search area; Determine whether the top-level data block meets the target condition, if so, insert the data block index number into the linked list, if not, continue to search for branches of the top-level data block until all data blocks that meet the target condition are found; The indexes of all data blocks that meet the target conditions are inserted into a linked list in order from left to right and from top to bottom, and large-scale operation and maintenance data are quickly accessed and retrieved through the linked list.

6. The IT informatization dual-model operation and maintenance management method based on a multi-layer data lake according to any one of claims 1 to 4, characterized in that: When completing accurate data filtering, key feature extraction and deep integration, it includes: Filter the data using filtering rules to form a key matrix model for identifying the system's operating status; Using the Apriori algorithm to obtain frequent multinomial sets, and mining the association rules between data types and fault types in the key matrix model, to establish the several key performance evaluation state quantity sets; Among them, when mining association rules, it includes: A cluster analysis is performed on the relationship between different data types and fault types in the key matrix model, the shape coefficient and silhouette coefficient of the associated data curve are mined, the collection amount of related data types is extracted, and a key performance evaluation state quantity set for different fault types is obtained.

7. The IT informatization dual-model operation and maintenance management method based on a multi-layer data lake according to any one of claims 1 to 4, characterized in that: When identifying potential factors, include: The operation and maintenance identification model provides: association rule antecedent and the association rule , association rule antecedent Represents the attribute characteristics of IT information operation and maintenance data, and the latter term of the association rule Indicates whether there are system problems or performance bottlenecks. Indicates a system problem. Indicates a performance bottleneck; Based on the antecedent and consequent items of association rules and , get the support of performance bottleneck ; Based on the antecedent and consequent items of association rules and , get the support level of system problems ; Will The attribute features greater than the threshold are taken as the first latent factor; Will Attribute features greater than the threshold are used as the second latent factors.

8. The IT information dual-model operation and maintenance management method based on a multi-layer data lake according to claim 7 is characterized in that: When automatically adjusting IT resource configuration, include: The operation and maintenance adjustment model determines the first control coefficient and the second control coefficient according to the first potential factor and the second potential factor, and controls the corresponding type of IT resource flow through the first control coefficient and the second control coefficient, as shown in Formula 3 and Formula 4: （3）； In the formula, For the planned period The first latent factor corresponds to The first control quantity of IT resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The first control coefficient of IT resources; （4）； In the formula, For the planned period The second latent factor corresponds to The second control quantity of IT-like resources; For the planned period No. strip IT resource flow, for The number of IT resource flows, For the planned period No. strip The second control coefficient of IT-like resources.

9. The IT informationization dual-model operation and maintenance management method based on a multi-layer data lake according to claim 8 is characterized in that: When automatically adjusting IT resource configuration, it also includes: The first control amount and the second control amount are approached to the maximum extent, and the first IT resource optimization configuration result and the second IT resource optimization configuration result are obtained, as shown in Formula 5 and Formula 6: （5）； In the formula, For the planned period The first latent factor corresponds to The actual amount of IT resources controlled, is the number of IT resource types corresponding to the first potential factor, is the total number of time periods in the first plan, Optimize the configuration results for the first IT resource; （6）； In the formula, For the planned period The second latent factor corresponds to The actual amount of IT resources controlled, is the number of IT resource types corresponding to the second potential factor, is the total number of time periods in the second plan, Optimize configuration results for secondary IT resources.

10. The IT informatization dual-model operation and maintenance management method based on a multi-layer data lake according to any one of claims 1 to 4, characterized in that: When performing dynamic display, it includes: A graph is defined, and the vertices of the identified first potential factor, the second potential factor, the first IT resource optimization configuration result, and the second IT resource optimization configuration result are determined by the host IP, the collection directory, and the file name template, respectively, and the attribute information of the vertex is set; The vertices are used as source ends, and the corresponding transmission target ends are matched, and the transmission target ends are used as vertices of the next layer of the dynamic display graph one by one, and their attribute information is set, and they are connected with the previous layer; The next layer of source information is matched, and the matched source information is used as the vertex of the next layer until no source information is matched.

Images