Data acquisition method based on big data of intelligent operation and maintenance platform

Abstract

The invention discloses a data acquisition method based on big data of an intelligent operation and maintenance platform, and relates to the technical field of computers. The method comprises the steps of 1, confirming and accessing a data source; step 2, data acquisition configuration; step 3, data transmission and storage; 4, preprocessing the data; step 5, performance optimization; step 6, data security and privacy protection; step 7, supervision and early warning; according to the method, a unified data collection method is provided, various different types of data sources including log files, system monitoring data, application performance indexes and user behavior data can be processed, the complexity that various tools and methods need to be used in a traditional scheme is avoided, and the data collection efficiency is improved. The data processing flow of an enterprise is simplified, and the overall data processing efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a data collection method based on big data of an intelligent operation and maintenance platform. Background Art

[0002] With the continuous advancement of informatization and digitalization, enterprises and organizations have accumulated a large amount of data from different data sources in their daily operations. These data are of great significance to the operation and maintenance management and decision-making of enterprises. However, the existing data collection methods have many shortcomings when dealing with multiple different types of data sources. Traditional data collection methods can usually only handle a single type of data source and cannot adapt to multiple different types of data sources. This requires enterprises to use a variety of different tools and methods when conducting data analysis and operation and maintenance management, resulting in complex and inefficient data processing processes. Due to the lack of a unified data access and processing mechanism, data between different data sources is difficult to integrate and correlate, affecting the value of data utilization; and data collection methods are also deficient in data security and privacy protection. During data transmission and storage, sensitive information is vulnerable to unauthorized access and leakage risks. In addition, there is a lack of effective supervision and early warning mechanisms. When the system is abnormal or fails, it cannot be discovered and measures cannot be taken in time, which may lead to business interruption or loss. Therefore, we propose a data collection method based on big data of intelligent operation and maintenance platform Summary of the invention

[0003] The purpose of the present invention is to provide a data collection method based on the big data of an intelligent operation and maintenance platform. By providing a unified data collection method, it is capable of processing a variety of different types of data sources, including log files, system monitoring data, application performance indicators and user behavior data. This method avoids the complexity of the need to use a variety of tools and methods in traditional solutions, and solves the problem that traditional data collection methods can usually only process a single type of data source and cannot adapt to a variety of different types of data sources. This requires enterprises to use a variety of different tools and methods when performing data analysis and operation and maintenance management, resulting in a complex and inefficient data processing process. Due to the lack of a unified data access and processing mechanism, data between different data sources is difficult to integrate and correlate, affecting the value of data utilization.

[0004] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0005] The present invention is a data collection method based on big data of intelligent operation and maintenance platform, comprising the following steps:

[0006] Step 1: Data source confirmation and access;

[0007] Step 2: Data collection configuration;

[0008] Step 3: Data transmission and storage;

[0009] Step 4: Data preprocessing;

[0010] Step 5: Performance optimization;

[0011] Step 6: Data security and privacy protection;

[0012] Step 7: Supervision and early warning;

[0013] Step 8: Data visualization and report generation.

[0014] The step 1 includes the following sub-steps: Data source confirmation: Data sources can be divided into log files, system monitoring data, application performance indicators and user behavior data; various log files generated by the operation and maintenance system are one of the important data sources. These logs record the system's operating status, error information and user operations, among which log files include system logs, application logs and security logs; system monitoring data includes CPU and memory usage, network traffic and broadband usage, and disk input and output; application performance indicators include response time, error rate and transaction throughput; user behavior data includes click streams, page dwell time and user interaction data;

[0015] Data source access: Use HTTP protocol for communication, define clear resource paths and operation methods, enable different systems to exchange data through a unified interface, use Kafka's messaging mechanism to send data generated by the data source to the queue in the form of messages, and consumers read data from the queue for processing. Use ETL tools or custom scripts to clean and convert data, convert data from different data sources into a unified format, ensure data consistency and availability, and use Great Expectations to assess data quality and check whether the data meets the expected standards and rules.

[0016] Furthermore, the step 2 includes the following steps: configuration file setting: by setting the source location and the acquisition method, determine which configuration format to use, create a configuration file according to the selected format, and define the required configuration items; output target location: for the configuration file stored in the local file system, the application needs to use appropriate file input and output operations to read the file content, and parse the content to extract the value of the configuration item. Through the API or library function provided by the operating system, the application can access the environment variables defined in the current environment and obtain the corresponding configuration value;

[0017] Collection frequency and cycle: You can choose between real-time collection and periodic collection; real-time collection is for continuous monitoring and timely data collection; periodic collection is for batch collection of data at set time intervals. You can choose based on the required collection frequency.

[0018] Furthermore, the step three includes the following steps: Data transmission: including a connection-oriented protocol: before data transmission, the communicating parties must first establish a connection. This process is usually called a "three-way handshake", that is, the client sends a connection request, the server confirms the request, and the client confirms again. After the connection is established, data transmission can begin. After the data transmission is completed, "four handshakes" are required to disconnect the connection. This connection method ensures the reliability and order of data transmission;

[0019] Data storage: By designing the storage architecture, it is suitable for large-scale, high-concurrency data storage, such as the document-type database MongoDB and the key-value database Redis, which are used for long-term storage and analysis of historical data, and support query and report generation of large amounts of data.

[0020] Furthermore, the step four includes the following steps: data preprocessing: removing invalid data, such as null values, outliers and duplicate records, correcting data format and type errors, converting data from different sources into a unified format, applying data standardization and normalization techniques to eliminate data bias, and identifying and processing data points that do not meet expectations.

[0021] Furthermore, step five includes the following steps: performance optimization includes parallel processing, data compression and caching mechanism, wherein parallel processing is to process multiple data streams simultaneously to improve efficiency, data compression is to reduce data storage space and speed up transmission speed, and the caching mechanism is to temporarily store hot data to speed up access speed.

[0022] Furthermore, the step six includes the following steps: Data encryption: By adopting the strong encryption protocol TLS, an encryption layer is provided for the data packet. When the data is transmitted from the source system to the target system, the use of TLS encryption can ensure that even if the data passes through an unsecured network, it cannot be read by an unauthorized third party, preventing the data from being intercepted or tampered with during the transmission process. For sensitive data stored on the server, database or any storage medium, AES is used for encryption, which means that even if the physical medium is stolen or lost, people without the key cannot interpret the data content, and the implementation of disk encryption and file-level encryption can provide an additional security layer to ensure the integrity and confidentiality of the data;

[0023] Access control: RBAC is a widely used access management strategy that allows system administrators to assign different permissions based on user roles and only allow each user to obtain the minimum permissions necessary to complete their work, limiting potential internal threats and reducing the risk of data leakage;

[0024] Audit logs: All attempts to access sensitive data are recorded, including successful access and failed attempts. The logs should record in detail the identity of the visitor, the time of access, the actions performed, and the results of the actions. Audit logs should be reviewed regularly to detect abnormal patterns or potential security threats in a timely manner. At the same time, multiple login failures or unusual data access patterns should be recorded, which helps monitor normal activities and provides important investigation clues when security incidents occur.

[0025] Further, the step seven includes the following steps, real-time monitoring: continuously monitoring the data collection process from the source system to the target system, using a real-time dashboard to display key performance indicators, collection speed, success rate and error rate; tracking each link of data processing, including data cleaning, conversion and loading processes, monitoring processing delays, throughput and resource utilization; monitoring the status of data in the storage system, including storage space usage, data access speed and storage system health;

[0026] Alarm rules: Determine which performance indicators are important to business operations, such as data collection delay, processing error rate, or storage availability. These indicators will serve as the basis for triggering alarms. Set reasonable thresholds for each key performance indicator. When the indicator exceeds or falls below these thresholds, the system will automatically generate an alarm. Based on historical data and business needs, the alarm thresholds can be dynamically adjusted to adapt to changes in system performance and growth in business needs.

[0027] Notification method: Ensure that alarm information can be conveyed to relevant personnel in a timely manner through multiple channels, such as email, SMS, instant messaging or WeChat, and telephone. Different notification strategies are adopted according to the severity of the alarm, so as to ensure that relevant personnel can receive notifications quickly even in emergency situations.

[0028] Furthermore, the step eight includes the following steps: data visualization: customizing visualization components and layouts according to user needs, providing real-time data update and historical data comparison functions, helping users to quickly understand data change trends and abnormal situations, integrating data from different data sources into a unified visualization platform, and realizing centralized display and analysis of data, which helps to improve data accessibility and utilization;

[0029] Report generation: Automatically generate performance reports and data analysis reports based on preset templates and schedules. These reports include key performance indicators, system health status, and user behavior trend information. Reports are sent to relevant personnel regularly via email or other means to ensure that they are kept informed of the system's operating status and business performance.

[0030] The present invention has the following beneficial effects:

[0031] 1. The present invention provides a unified data collection method that can process a variety of different types of data sources, including log files, system monitoring data, application performance indicators and user behavior data. This method avoids the complexity of using multiple tools and methods in traditional solutions, simplifies the enterprise's data processing process, and significantly improves the overall data processing efficiency. In addition, data cleaning and conversion are performed through ETL tools or custom scripts, and data from different data sources are converted into a unified format to ensure data consistency and availability. This not only enhances the accuracy and reliability of data analysis, but also improves the data integration and correlation analysis capabilities, thereby better supporting the enterprise's operation and maintenance management and decision-making.

[0032] 2. The present invention adopts multi-level security measures in the process of data transmission and storage to improve the security and privacy protection level of data. First, the data is encrypted and transmitted through the TLS protocol to prevent the data from being intercepted or tampered with during the transmission process. Then, for sensitive data stored on the server, database or any storage medium, AES is used for encryption. Even if the physical medium is stolen or lost, people without the key cannot decipher the content. At the same time, access control strategies are implemented to limit the risk of potential internal threats and data leakage. All attempts to access sensitive data will be recorded, including successful access and failed attempts, so as to timely detect abnormal patterns or potential security threats. These measures together build a powerful data security system to effectively prevent unauthorized access and leakage of sensitive information.

[0033] 3. The present invention provides a real-time monitoring function that can continuously track the entire data collection process from the source system to the target system, and use a real-time dashboard to display key performance indicators. This real-time monitoring helps to quickly identify system anomalies or failures and take corresponding preventive measures. At the same time, reasonable alarm thresholds are set. When the indicators exceed or fall below these thresholds, the system will automatically generate alarm information and promptly convey it to relevant personnel through multiple channels such as e-mail, SMS, and instant messaging. This multi-channel notification method ensures that even in an emergency, relevant personnel can quickly receive notifications and take action. Regular review of audit logs can help discover long-term trends or potential problems, providing a basis for optimizing operation and maintenance management. This comprehensive supervision and early warning mechanism improves the efficiency and response speed of operation and maintenance management.

[0034] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 The present invention is a flowchart of a data collection method based on big data of an intelligent operation and maintenance platform. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 As shown, the present invention is a data collection method based on big data of intelligent operation and maintenance platform, comprising the following steps:

[0039] Step 1: Data source confirmation and access; Step 2: Data collection configuration; Step 3: Data transmission and storage; Step 4: Data preprocessing; Step 5: Performance optimization; Step 6: Data security and privacy protection; Step 7: Supervision and early warning; Step 8: Data visualization and report generation.

[0040] Step 1 includes the following sub-steps: Data source confirmation: Data sources can be divided into log files, system monitoring data, application performance indicators and user behavior data; various log files generated by the operation and maintenance system are one of the important data sources. These logs record the system's operating status, error information and user operations, among which log files include system logs, application logs and security logs; system monitoring data includes CPU and memory usage, network traffic and broadband usage, and disk input and output; application performance indicators include response time, error rate and transaction throughput; user behavior data includes click stream, page dwell time and user interaction data; Data source access: Use HTTP protocol for communication, define clear resource paths and operation methods, so that different systems can exchange data through a unified interface, use Kafka's messaging mechanism to send data generated by the data source to the queue in the form of messages, and consumers read data from the queue for processing; use ETL tools or custom scripts for data cleaning and conversion, convert data from different data sources into a unified format to ensure data consistency and availability, and use Great Expectations to assess data quality and check whether the data meets the expected standards and rules.

[0041] Step 2 includes the following steps: Configuration file setting: By setting the source location and collection method, determine which configuration format to use, create a configuration file according to the selected format, and define the required configuration items; Output target location: For configuration files stored in the local file system, the application needs to use appropriate file input and output operations to read the file content and parse the content to extract the value of the configuration item. Through the API or library function provided by the operating system, the application can access the environment variables defined in the current environment and obtain the corresponding configuration values; Collection frequency and period: You can choose real-time collection and periodic collection; real-time collection is for continuous monitoring and timely data collection; periodic collection is for batch collection at set time intervals. Select according to the required collection frequency.

[0042] Step three includes the following steps: Data transmission: including connection-oriented protocols: before data transmission, the communicating parties must first establish a connection. This process is usually called a "three-way handshake", that is, the client sends a connection request, the server confirms the request, and the client confirms again. After the connection is established, data transmission can begin. After the data transmission is completed, "four waves" are required to disconnect the connection. This connection method ensures the reliability and sequentiality of data transmission; Data storage: through the design of storage architecture, it is suitable for large-scale, high-concurrency data storage, such as document-type database MongoDB and key-value database Red is, which are used for long-term storage and analysis of historical data, and support large-scale data query and report generation.

[0043] Step 4 includes the following steps: Data preprocessing: remove invalid data such as null values, outliers and duplicate records, correct format and type errors in data, convert data from different sources into a unified format, apply data standardization and normalization techniques to eliminate data bias, and identify and process data points that do not meet expectations.

[0044] Step five includes the following steps: Performance optimization includes parallel processing, data compression and caching mechanism, wherein parallel processing is to process multiple data streams simultaneously to improve efficiency, data compression is to reduce data storage space and speed up transmission speed, and the caching mechanism is to temporarily store hot data to speed up access speed.

[0045] Step six includes the following steps: Data encryption: By adopting the strong encryption protocol TLS, an encryption layer is provided for the data packet. When the data is transmitted from the source system to the target system, the use of TLS encryption can ensure that even if the data passes through an insecure network, it cannot be read by unauthorized third parties, preventing the data from being intercepted or tampered with during transmission. For sensitive data stored on servers, databases or any storage media, AES is used for encryption, which means that even if the physical media is stolen or lost, people without the key cannot interpret the data content. In addition, the implementation of disk encryption and file-level encryption can provide an additional security layer to ensure the integrity and confidentiality of the data; Access control: RBAC is a widely used The access management strategy used allows system administrators to assign different permissions based on the user's role and only allows each user to obtain the minimum permissions necessary to complete their work, limiting potential internal threats and reducing the risk of data leakage; Audit log: All attempts to access sensitive data are recorded, including successful access and failed attempts. The log should record in detail the identity of the visitor, access time, operations performed, and results of the operations. Audit logs are reviewed regularly to promptly detect abnormal patterns or potential security threats. At the same time, multiple login failures or unusual data access patterns are recorded, which helps monitor normal activities and provides important investigation clues when security incidents occur.

[0046] Step seven includes the following steps: Real-time monitoring: Continuously monitor the data collection process from the source system to the target system, and use the real-time dashboard to display key performance indicators, collection speed, success rate and error rate; Track each link of data processing, including data cleaning, conversion and loading processes, and monitor processing delays, throughput and resource utilization; Monitor the status of data in the storage system, including storage space usage, data access speed and storage system health; Alarm rules: Determine which performance indicators are important to business operations, and use data collection delays, processing error rates or storage availability as the basis for triggering alarms; Set reasonable thresholds for each key performance indicator. When the indicator exceeds or falls below these thresholds, the system will automatically generate an alarm. According to historical data and business needs, the alarm threshold is dynamically adjusted to adapt to changes in system performance and growth in business needs; Notification method: Ensure that alarm information can be promptly conveyed to relevant personnel through multiple channels through e-mail, SMS, instant messaging or WeChat, and telephone. Different notification strategies are adopted according to the severity of the alarm, so as to ensure that relevant personnel can receive notifications quickly even in an emergency.

[0047] Step eight includes the following steps: Data visualization: Customize visualization components and layouts according to user needs, provide real-time data updates and historical data comparison functions, help users quickly understand data trends and anomalies, integrate data from different data sources into a unified visualization platform, and realize centralized display and analysis of data, which helps to improve data accessibility and utilization; Report generation: Automatically generate performance reports and data analysis reports based on preset templates and schedules. These reports include key performance indicators, system health status, user behavior trend information, and regularly send reports to relevant personnel via email or other means to ensure that they are aware of the system's operating status and business performance in a timely manner.

[0048] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0049] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data collection method based on big data of intelligent operation and maintenance platform, characterized by: The steps include: Step 1: Confirm and access the data source; Step 2: Data collection configuration; Step 3: Data transmission and storage; Step 4: Data preprocessing; Step 5: Performance optimization; Step 6: Data security and privacy protection; Step 7: Supervision and early warning; Step 8: Data visualization and report generation. The step 1 includes the following sub-steps: Data source confirmation: the data source can be divided into log files, system monitoring data, application performance indicators and user behavior data; The various log files generated by the operation and maintenance system are one of the important data sources. These logs record the system's operating status, error information, and user operations. The log files include system logs, application logs, and security logs. System monitoring data includes CPU and memory usage, network traffic and broadband usage, and disk input and output. Application performance indicators include response time, error rate, and transaction throughput. User behavior data includes click streams, page dwell time, and user interaction data. Data source access: Use HTTP protocol for communication, define clear resource paths and operation methods, enable different systems to exchange data through a unified interface, use Kafka's messaging mechanism to send data generated by the data source to the queue in the form of messages, and consumers read data from the queue for processing. Use ETL tools or custom scripts for data cleansing and conversion, convert data from different data sources into a unified format, ensure data consistency and availability, and use Great Expectations to assess data quality and check whether the data meets the expected standards and rules.

2. According to claim 1, a data collection method based on big data of intelligent operation and maintenance platform is characterized in that: The step 2 includes the following steps: configuration file setting: by setting the source location and acquisition method, determine which configuration format to use, create a configuration file according to the selected format, and define the required configuration items; output target location: for the configuration file stored in the local file system, the application needs to use appropriate file input and output operations to read the file content and parse the content to extract the value of the configuration item. Through the API or library function provided by the operating system, the application can access the environment variables defined in the current environment and obtain the corresponding configuration value; Collection frequency and cycle: You can choose between real-time collection and periodic collection; real-time collection is for continuous monitoring and timely data collection; periodic collection is for batch collection of data at set time intervals. You can choose based on the required collection frequency.

3. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: In step three The following steps are included: Data transmission: including connection-oriented protocols: Before data transmission, both parties must first establish a connection. This process is usually called a "three-way handshake", that is, the client sends a connection request, the server confirms the request, and the client confirms it again. After the connection is established, data transmission can begin. After the data transmission is completed, "four handshakes" are required to disconnect the connection; Data storage: By designing the storage architecture, it is suitable for large-scale, high-concurrency data storage, such as the document database MongoDB and the key-value database Redis, which are used for long-term storage and analysis of historical data, and support query and report generation of large amounts of data.

4. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: The step four includes the following steps: data preprocessing: removing invalid data, such as null values, outliers and duplicate records, correcting data format and type errors, converting data from different sources into a unified format, applying data standardization and normalization techniques to eliminate data bias, and identifying and processing data points that do not meet expectations.

5. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: The step five includes the following steps: performance optimization includes parallel processing, data compression and cache mechanism, wherein parallel processing is to process multiple data streams simultaneously to improve efficiency, data compression is to reduce data storage space and speed up transmission speed, and cache mechanism is to temporarily store hot data to speed up access speed.

6. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: In step six The following steps are included: Data encryption: By adopting the strong encryption protocol TLS, an encryption layer is provided for the data packet. When the data is transmitted from the source system to the target system, the use of TLS encryption can ensure that even if the data passes through an insecure network, it cannot be read by unauthorized third parties, preventing the data from being intercepted or tampered with during transmission. For sensitive data stored on the server, database or any storage medium, AES is used for encryption, which means that even if the physical medium is stolen or lost, people without the key cannot interpret the data content. In addition, the implementation of disk encryption and file-level encryption can provide an additional layer of security; Access control: RBAC is a widely used access management strategy that allows system administrators to assign different permissions based on user roles and only allow each user to obtain the minimum permissions necessary to complete their work, limiting potential internal threats and reducing the risk of data leakage; Audit logs: All attempts to access sensitive data are recorded, including successful access and failed attempts. The logs should record in detail the identity of the visitor, the time of access, the actions performed, and the results of the actions. Audit logs should be reviewed regularly to detect abnormal patterns or potential security threats in a timely manner, and multiple login failures or unusual data access patterns should be recorded.

7. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: The step seven includes the following steps, real-time monitoring: continuously monitoring the data collection process from the source system to the target system, using a real-time dashboard to display key performance indicators, collection speed, success rate and error rate; Track all aspects of data processing, including data cleaning, conversion and loading processes, and monitor processing latency, throughput and resource utilization; Monitor the status of data in the storage system, including storage space usage, data access speed, and storage system health status; Alarm rules: Determine which performance indicators are important to business operations, such as data collection delay, processing error rate, or storage availability. These indicators will serve as the basis for triggering alarms. Set reasonable thresholds for each key performance indicator. When the indicator exceeds or falls below these thresholds, the system will automatically generate an alarm. Based on historical data and business needs, the alarm thresholds can be dynamically adjusted to adapt to changes in system performance and growth in business needs. Notification method: Ensure that alarm information can be conveyed to relevant personnel in a timely manner through multiple channels, such as email, SMS, instant messaging or WeChat, and telephone. Different notification strategies are adopted according to the severity of the alarm, so as to ensure that relevant personnel can receive notifications quickly even in emergency situations.

8. The data collection method based on big data of intelligent operation and maintenance platform according to claim 1 is characterized in that: In step eight The following steps are included: Data visualization: Customize visualization components and layouts according to user needs, provide real-time data update and historical data comparison functions, help users quickly understand data trends and abnormal situations, integrate data from different data sources into a unified visualization platform, and realize centralized display and analysis of data; Report generation: Automatically generate performance reports and data analysis reports based on preset templates and schedules. These reports include key performance indicators, system health status, user behavior trend information, and send reports to relevant personnel regularly via email or other means.

Images