Multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system and method

Abstract

The invention discloses an intelligent fusion and standardization processing system and method for multi-source heterogeneous operation and maintenance data, and belongs to the technical field of operation and maintenance data. The system comprises a data acquisition module, a data preprocessing module, a data fusion module and an information extraction module; the data acquisition module is used for collecting original multi-source heterogeneous operation and maintenance data from different data sources, and the data sources comprise facilities, operating systems, basic components, middleware, databases and / or information systems; the data preprocessing module is used for cleaning, denoising and standardizing the acquired data; the data fusion module is used for selecting a low-level data fusion model, a middle-level data fusion model or a high-level data fusion model according to the fusion scene and fusing the preprocessed data; and the information extraction module is used for carrying out feature extraction on the fused data. According to the method, deep fusion and standardized processing of the multi-source heterogeneous operation and maintenance data are realized, and a unified and high-quality data basis is provided for data analysis and decision making.

Description

Technical Field

[0001] The present invention belongs to the field of operation and maintenance data technology, and specifically relates to a system and method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data. Background Art

[0002] In today's digital age, enterprise operations management faces enormous challenges brought by multi-source heterogeneous operations data. Traditional operations data processing methods have many obvious shortcomings.

[0003] Traditional data processing methods often struggle to cope with data from a wide range of sources and in a variety of formats. Operational data generated by different systems, devices, and applications may have different structures, semantics, and encoding methods, making data integration extremely difficult.

[0004] When it comes to data fusion, existing technologies often lack effective mechanisms to accurately link and integrate data from disparate sources. This leads to overlooking the interdependencies between data, making it difficult to form a comprehensive and accurate view of operational data. Traditional methods for data standardization often rely on predefined rules and templates, which lack flexibility and adaptability. New data types or format changes often require extensive manual intervention and reconfiguration, which is inefficient and prone to errors.

[0005] Furthermore, traditional data processing methods often fail to meet performance requirements when handling massive amounts of operational data. Data reading, conversion, and storage are time-consuming, hindering the timeliness of operational decisions. Existing methods also suffer from shortcomings in ensuring data quality. Due to the lack of effective data cleansing and validation mechanisms, data may contain duplicate, erroneous, or missing values, severely impacting its accuracy and usability. Furthermore, traditional data processing technologies are severely limited in their ability to process heterogeneous data from multiple sources, making it impossible to fully tap into the potential value within this data. Summary of the Invention

[0006] In response to the deficiencies in the prior art, the present invention provides a system and method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data. Based on big model technology, it realizes the deep fusion and standardized processing of multi-source heterogeneous operation and maintenance data, providing a unified, high-quality data foundation for data analysis and decision-making.

[0007] The present invention provides the following technical solutions:

[0008] In the first aspect, a multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system is provided, including: a data acquisition module, a data preprocessing module, a data fusion module, and an information extraction module;

[0009] The data acquisition module collects original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases and / or information systems;

[0010] The data preprocessing module cleans, removes noise and standardizes the acquired data;

[0011] The data fusion module selects a low-level data fusion model, a mid-level data fusion model or a high-level data fusion model according to the fusion scenario to fuse the pre-processed data;

[0012] The information extraction module extracts features from the fused data to analyze and make decisions on the fused data.

[0013] Optionally, the collection of original multi-source heterogeneous operation and maintenance data from different data sources includes: collecting operation and maintenance data of server facilities, network facilities, and storage facilities through SNMP protocol and IPMI protocol, collecting operation and maintenance data of operating systems and middleware through system commands and tools, including CPU usage, memory usage, disk I / O, process performance, and system service performance information, and collecting logs of information systems through log collection tools, including user behavior data and business-related data.

[0014] Optionally, the data acquisition module also performs label processing and storage on the collected original multi-source heterogeneous operation and maintenance data according to different data characteristics; wherein label processing is to mark the data according to the data source; storage specifically includes: storing semi-structured or unstructured data in a non-relational database, and storing structured data in a relational database.

[0015] Optionally, the data preprocessing module cleans, denoises, and standardizes the acquired data, specifically:

[0016] Clean and correct the acquired data: Based on the upstream and downstream relationships of the data link, if the acquired data outliers are downstream, check whether the upstream data is normal. If the upstream data is normal, determine that the downstream outliers are caused by data recording errors and correct the outliers. The correction method adopts the mean, median or mode method;

[0017] De-noise the acquired data: adopt the moving average method to calculate the average value of the data within the set window size as the smoothing value of the current point;

[0018] Standardize the acquired data: Z-score standardization is used for numerical data, and normalized vector length is used for non-numerical data.

[0019] Optionally, the low-level data fusion model, the middle-level data fusion model or the high-level data fusion model is selected according to the fusion scenario, specifically:

[0020] If the fusion scenario involves a large amount of data and is aimed at merging data with the same label, a low-level data fusion model is selected to merge homologous data;

[0021] If the fusion scenario is to compare and analyze the features of different data sources and identify the relationships and patterns between the data, then the mid-level data fusion model is selected to merge multi-source data;

[0022] If the fusion scenario is to combine domain knowledge and contextual information to perform semantic understanding and reasoning on data, a high-level data fusion model is selected to merge the data.

[0023] Optionally, the low-level data fusion model, the middle-level data fusion model or the high-level data fusion model fuses data using a weighted average method or a neural network fusion method;

[0024] The weighted average method is used to perform weighted averaging on data from multiple data sources. Specifically, each data is multiplied by the corresponding weight coefficient and then added together to obtain the fusion result. The weight coefficient is set according to the upstream and downstream of the data and the importance of the data.

[0025] The neural network fusion method is used to fuse data from multiple data sources. Specifically, the original multi-source heterogeneous operation and maintenance data are collected from different data sources as input, and the mapping relationship between the input data and the fusion results is learned by training the neural network.

[0026] Optionally, the information extraction module extracts features from the fused data to analyze and make decisions on the fused data, including:

[0027] Fault prediction is performed on the fused data; specifically: historical fault data and real-time operation data are integrated, operation logs, performance data, and event records from various systems are collected, and a fault prediction model is constructed through a deep learning model to predict faults.

[0028] Secondly, a method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data is provided, comprising the following steps:

[0029] Step S1: Collect original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases, and information systems. Label the collected data according to different data characteristics and store them in relational databases and non-relational databases according to different data types.

[0030] Step S2: cleaning, denoising and standardizing the acquired data to form preprocessed data;

[0031] Step S3: According to different data fusion scenarios, the pre-processed data is fused using a low-level data fusion model, a mid-level data fusion model, or a high-level data fusion model;

[0032] Step S4: extract features from the fused data to analyze and make decisions on the fused data.

[0033] In a third aspect, a computer device is provided, comprising a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data described in the second aspect are implemented.

[0034] In a fourth aspect, a computer-readable storage medium is provided for storing a computer program; when the computer program is executed by a processor, the steps of the method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data described in the second aspect are implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Based on large-scale model technology, it achieves intelligent fusion and standardized processing of multi-source heterogeneous operations and maintenance data, providing a unified, high-quality data foundation for data analysis and decision-making. This significantly improves the efficiency and accuracy of data processing, enabling rapid processing of large-scale data and complex data types, enhancing the timeliness and scientific nature of operations and maintenance decisions, and effectively safeguarding data quality and reliability. Real-time monitoring and processing of problematic data ensures data availability and credibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural block diagram of the multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0039] It should be noted that the term "comprise" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0040] Example 1

[0041] like Figure 1 As shown, a multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system is provided, including: a data acquisition module, a data preprocessing module, a data fusion module, and an information extraction module; the data acquisition module collects original multi-source heterogeneous operation and maintenance data from different data sources, and the data sources include facilities, operating systems, basic components, middleware, databases and / or information systems; the data preprocessing module cleans, denoises and standardizes the acquired data; the data fusion module selects a low-level data fusion model, a mid-level data fusion model or a high-level data fusion model according to the fusion scenario, and fuses the preprocessed data; the information extraction module extracts features from the fused data to analyze and make decisions on the fused data.

[0042] The data acquisition module collects raw, multi-source, heterogeneous operations and maintenance data from various data sources. It collects data on server facilities, network facilities, and storage facilities using the SNMP and IPMI protocols. It also collects data on operating systems and middleware using system commands and tools, including but not limited to CPU usage, memory usage, disk I / O, process performance, and system service performance. It also collects information system logs, including user behavior data and business-related data, using log collection tools. Data is labeled based on its source. For example, all data from Server A is labeled "ServerA," and operations and maintenance data from a database system is labeled "DB_Ops_[database name]." Labels are assigned based on the data type, such as CPU usage data labeled "Performance_CPU" and system log data labeled "Log_System." Semi-structured or unstructured data is stored in non-relational databases, such as MongoDB and Elasticsearch, while structured data is stored in relational databases, such as MySQL.

[0043] The data preprocessing module cleans and corrects the acquired data. Based on the upstream and downstream relationships of the data link, if the data outlier is downstream, the module checks whether the upstream data is normal. If the upstream data is normal, the downstream abnormal data is determined to be caused by data recording errors, and the outlier is corrected using the mean, median, or mode. For example, if a sensor failure causes a device's temperature reading to be too high, the abnormal temperature value can be corrected to a reasonable range by averaging the temperature readings within half an hour. The acquired data is denoised using a moving average method, calculating the average value of the data within a certain window size as the smoothed value at the current point. For example, for a data series containing daily server CPU usage, a moving average with a window size of 3 days is used. The smoothed CPU usage on the third day is the average of the CPU usage of the previous 3 days, which can effectively remove short-term fluctuation noise in the data.

[0044] The present invention selects a low-level data fusion model, a mid-level data fusion model, or a high-level data fusion model according to the fusion scenario to fuse the pre-processed data. The low-level data fusion model focuses on the integration of raw data, usually involving the direct merging of homologous data, and is suitable for scenarios with large amounts of data and the same label data processing. For example, in performance-based data fusion scenarios, CPU data from different facilities can be integrated through low-level fusion technology to monitor the CPU health of all facilities within the data center in real time. The mid-level data fusion model focuses on the extraction and analysis of data features. By comparing and analyzing the features of different data sources, the relationship and pattern between data can be identified. For example, in operating system data and facility data, by performing mid-level fusion on CPU and memory alarm data, it is possible to effectively identify which middleware is deployed in the operating system and whether the facility failure affects the middleware. It can also identify which information systems the middleware is used in and whether the facility failure affects the information system. The high-level data fusion model focuses on the semantic understanding and reasoning of data involved in high-level data fusion, usually requiring the combination of domain knowledge and contextual information, and is suitable for complex decision support systems. For example, by integrating historical business system failure case data with the real-time performance, operation records, and other data of the current inventory system, it is possible to predict whether the business system will fail and the type of failure, and take maintenance measures in advance.

[0045] The low-level data fusion model, the mid-level data fusion model or the high-level data fusion model of the present application fuses data using a weighted average method or a neural network fusion method;

[0046] Among them, the weighted average method is used to perform weighted averaging on data from multiple data sources, specifically: each data is multiplied by the corresponding weight coefficient and then added to obtain the fusion result; the weight coefficient is set according to the upstream and downstream of the data and the importance of the data; the weighted average method is usually used for facility data and operating system data; the weight setting is optional. If the router is downstream of the load balancer, the CPU usage of the device downstream of the load balancer is: load balancer CPU usage × 70% + router CPU usage × 30%.

[0047] The neural network fusion method is used to fuse data from multiple data sources, specifically: original multi-source heterogeneous operation and maintenance data are collected from different data sources as input, and the mapping relationship between the input data and the fusion result is learned by training the neural network. Utilizing the self-learning, self-organizing and adaptive capabilities of the neural network as well as the powerful nonlinear mapping capabilities, original multi-source heterogeneous operation and maintenance data are collected from different data sources as input, and the mapping relationship between the input data and the fusion result is learned by training the neural network, thereby achieving data fusion. For example, in facility data, CPU and memory data are associated and combined by means of labels, in database and middleware data, performance data are associated and combined by means of labels, and in information system data and facility data, cross-system logs are connected and integrated upstream and downstream through message header data. The large model technology of the present invention adopts a multi-layer neural network structure and combines it with natural language processing technology to improve the processing capability of multi-source heterogeneous data.

[0048] The request format for obtaining data in this application is in XML format or JSON format, and supports jsonp cross-domain access; the information extraction module extracts features from the fused data to analyze and make decisions on the fused data. The information extraction method uses the N-gram analysis method to divide all operation and maintenance document fragments related to user input questions into several word combinations.

[0049] The fused data is analyzed and decided upon, and is primarily used in intelligent operation and maintenance (O&M) scenarios, such as fault prediction and intelligent diagnosis. Specifically, this involves fusing historical fault data with real-time operation data, collecting operation logs, performance data, and event records from various systems, and constructing fault prediction models through deep learning models (such as long short-term memory networks (LSTMs) and knowledge graphs) to predict faults.

[0050] The present invention brings about significant beneficial effects in many aspects, realizes the deep integration and standardized processing of multi-source heterogeneous operation and maintenance data, provides a unified, high-quality data foundation for data analysis and decision-making, greatly improves the efficiency and accuracy of data processing, can quickly respond to large-scale data and complex data types, and improves the timeliness and scientificity of operation and maintenance decisions.

[0051] Example 2

[0052] A method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data, comprising the following steps:

[0053] Step S1: Collect original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases, and information systems. Label the collected data according to different data characteristics and store them in relational databases and non-relational databases according to different data types.

[0054] Step S2: cleaning, denoising and standardizing the acquired data to form preprocessed data;

[0055] Step S3: According to different data fusion scenarios, the pre-processed data is fused using a low-level data fusion model, a mid-level data fusion model, or a high-level data fusion model;

[0056] Step S4: extract features from the fused data to analyze and make decisions on the fused data.

[0057] Through deep knowledge mining, different data fusion models are selected for different data fusion scenarios. For example, for simple status detection, a low-level data fusion model is used to simply merge data. For global system performance analysis, a mid-level data fusion model is selected to extract performance characteristics from different layers, such as the operating system, middleware, and database. CPU and memory performance metrics are extracted from the operating system, request processing time and queue length are obtained from the middleware, and query execution time and cache hit rate are obtained from the database. Feature fusion is then performed using methods such as principal component analysis (PCA) to form a comprehensive performance feature vector. This vector can be used to evaluate and analyze the performance of the entire system. For fault prediction and intelligent diagnosis, a high-level data fusion model is selected. This model combines data from multiple systems for fusion decision-making, utilizing historical fault data and real-time operation data. Operational logs, performance data, and event records from various systems are collected. Deep learning models such as long-short-term memory (LSTM) networks and knowledge graphs are used to build a fault prediction model and predict faults.

[0058] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0059] Example 3

[0060] The present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the above-mentioned method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data are implemented.

[0061] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0062] Example 4

[0063] The present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the above-mentioned method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data are implemented.

[0064] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. The methods, devices, and storage media disclosed in the embodiments are described briefly because they correspond to the systems disclosed in the embodiments. For relevant details, refer to the method description.

[0066] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention or certain portions of the embodiments.

[0067] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system, characterized by: include: Data acquisition module, data preprocessing module, data fusion module, information extraction module; The data acquisition module collects original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases and / or information systems; The data preprocessing module cleans, removes noise and standardizes the acquired data; The data fusion module selects a low-level data fusion model, a mid-level data fusion model or a high-level data fusion model according to the fusion scenario to fuse the pre-processed data; The information extraction module extracts features from the fused data to analyze and make decisions on the fused data.

2. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized by: The collection of original multi-source heterogeneous operation and maintenance data from different data sources includes: collecting operation and maintenance data of server facilities, network facilities, and storage facilities through SNMP protocol and IPMI protocol, collecting operation and maintenance data of operating system and middleware through system commands and tools, including CPU usage, memory usage, disk I / O, process performance, system service performance information, and collecting logs of information system through log collection tools, including user behavior data and business-related data.

3. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized by: The data acquisition module also performs label processing and storage on the collected original multi-source heterogeneous operation and maintenance data according to different data characteristics; wherein label processing is to mark the data according to the data source; storage specifically includes: storing semi-structured or unstructured data in a non-relational database, and storing structured data in a relational database.

4. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized in that: The data preprocessing module cleans, removes noise and standardizes the acquired data, specifically: Clean and correct the acquired data: Based on the upstream and downstream relationships of the data link, if the acquired data outliers are downstream, check whether the upstream data is normal. If the upstream data is normal, determine that the downstream outliers are caused by data recording errors and correct the outliers. The correction method adopts the mean, median or mode method; De-noise the acquired data: adopt the moving average method to calculate the average value of the data within the set window size as the smoothing value of the current point; Standardize the acquired data: Z-score standardization is used for numerical data, and normalized vector length is used for non-numerical data.

5. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized in that: The selection of a low-level data fusion model, a mid-level data fusion model, or a high-level data fusion model according to the fusion scenario is specifically as follows: If the fusion scenario involves a large amount of data and is aimed at merging data with the same label, a low-level data fusion model is selected to merge homologous data; If the fusion scenario is to compare and analyze the features of different data sources and identify the relationships and patterns between the data, then the mid-level data fusion model is selected to merge multi-source data; If the fusion scenario is to combine domain knowledge and contextual information to perform semantic understanding and reasoning on data, a high-level data fusion model is selected to merge the data.

6. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized in that: The low-level data fusion model, the middle-level data fusion model or the high-level data fusion model fuses data using a weighted average method or a neural network fusion method; The weighted average method is used to perform weighted averaging on data from multiple data sources. Specifically, each data is multiplied by the corresponding weight coefficient and then added together to obtain the fusion result. The weight coefficient is set according to the upstream and downstream of the data and the importance of the data. The neural network fusion method is used to fuse data from multiple data sources. Specifically, the original multi-source heterogeneous operation and maintenance data are collected from different data sources as input, and the mapping relationship between the input data and the fusion results is learned by training the neural network.

7. The multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing system according to claim 1 is characterized in that: The information extraction module extracts features from the fused data to analyze and make decisions on the fused data, including: Fault prediction is performed on the fused data; specifically: historical fault data and real-time operation data are integrated, operation logs, performance data, and event records from various systems are collected, and a fault prediction model is constructed through a deep learning model to predict faults.

8. A method for intelligent fusion and standardization of multi-source heterogeneous operation and maintenance data, characterized by: The following steps are involved: Step S1: Collect original multi-source heterogeneous operation and maintenance data from different data sources, including facilities, operating systems, basic components, middleware, databases, and information systems. Label the collected data according to different data characteristics and store them in relational databases and non-relational databases according to different data types. Step S2: cleaning, denoising and standardizing the acquired data to form preprocessed data; Step S3: According to different data fusion scenarios, the pre-processed data is fused using a low-level data fusion model, a mid-level data fusion model, or a high-level data fusion model; Step S4: extract features from the fused data to analyze and make decisions on the fused data.

9. A computer device, characterized in that: It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the multi-source heterogeneous operation and maintenance data intelligent fusion and standardization processing method described in claim 8.

10. A computer-readable storage medium, characterized in that Used to store computer programs; when the computer programs are executed by the processor, the steps of the method for intelligent fusion and standardized processing of multi-source heterogeneous operation and maintenance data as described in claim 8 are implemented.

Images