Computer data acquisition, processing and analysis system

Abstract

The invention discloses a computer data acquisition, processing and analysis system. The system comprises a data acquisition module which is responsible for acquiring data; the data processing module is responsible for preprocessing the collected data; the resource allocation module is used for dynamically allocating computing resources according to the system load condition; the intelligent analysis module is responsible for carrying out deep analysis and mining on the processed data and extracting valuable information and insight; the real-time monitoring and early warning module is responsible for monitoring various performance indexes of the system in real time and displaying a monitoring result through a visual tool; in the invention, the resource allocation module can dynamically allocate the computing resources according to the load condition of the system and ensure that enough processing capacity can be provided when the load is high, and the intelligent resource allocation mechanism is beneficial to optimizing the system performance, reducing the resource waste and improving the overall stability and reliability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, processing and analysis, and in particular to a computer data acquisition, processing and analysis system. Background Art

[0002] With the rise of digitalization, more and more businesses and organizations are beginning to value the value of data. As a key tool for digital transformation, data acquisition, processing, and analysis systems are gradually penetrating various industries. The rapid development of big data and artificial intelligence technologies has opened up a wider range of applications for data acquisition, processing, and analysis systems. By leveraging big data and artificial intelligence, these systems can rapidly process and analyze massive amounts of data, uncovering more valuable information and insights.

[0003] A search revealed an invention patent with Chinese patent number CN114826770A, which discloses a big data management platform for intelligent analysis of computer networks. The platform comprises a data acquisition and processing module, a computer network status acquisition module, a computer network diagnosis module, an intelligent analysis module, a computer network repair module, and a big data management module. The data acquisition and processing module is used to collect and process data input into the big data management platform and transmit the collected and processed data to the intelligent analysis module. The computer network status acquisition module is used to collect load information, network traffic information, and network status information during computer network operation and transmit the collected information to the computer network diagnosis module. The computer network diagnosis module is used to receive the collected information transmitted by the computer network status acquisition module, diagnose the computer network based on the received information, and transmit the diagnosis results to the intelligent analysis module. Compared with the existing technology, this invention patent with Chinese patent number CN114826770A marks the data whose actual propagation path does not match the expected propagation path, and determines the propagation path of the marked data transmitted through multiple paths based on the amount of data increase. This makes it easier to determine the specific fault point when a failure occurs in the big data management platform or computer network in the future. By managing the data or network before and after the fault point, the normal operation of the big data management platform can be guaranteed, further improving the data management effect of the platform.

[0004] However, in the above-mentioned use process, especially in a rapidly changing network environment, real-time processing and analysis of data is crucial. It may not be able to respond to data changes in time, resulting in network problems that cannot be discovered and solved in time. Therefore, a computer data acquisition, processing and analysis system is proposed. Summary of the Invention

[0005] The purpose of this invention is to solve the shortcomings of the existing technology, such as uneven resource distribution, difficulty in obtaining adequate support for some excellent projects due to limited resources, incomplete or in-time feedback mechanism of the system, and difficulty for students to understand their own shortcomings and make improvements in a timely manner. A computer data acquisition, processing and analysis system is proposed.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A computer data acquisition, processing and analysis system, comprising:

[0008] Data collection module: Responsible for collecting data from various data sources, including relational databases, non-relational databases, real-time data streams (such as message queues such as Kafka and RabbitMQ), API interfaces, etc. The data collection module needs to support access to multiple data formats and be able to realize real-time or batch data collection;

[0009] Data processing module: responsible for pre-processing the collected data, such as cleaning, deduplication, and conversion, to ensure the accuracy and consistency of the data;

[0010] Resource allocation module: dynamically allocates computing resources according to system load to ensure sufficient processing power under high load;

[0011] Intelligent analysis module: responsible for in-depth analysis and mining of processed data to extract valuable information and insights;

[0012] Real-time monitoring and early warning module: responsible for real-time monitoring of various performance indicators of the system and displaying the monitoring results through visualization tools (such as Grafana, Kibana, etc.);

[0013] Data storage and management module: responsible for persistent storage and efficient access to data, using distributed storage systems (such as HDFS, HBase, etc.) to store large amounts of data, and building data warehouses to integrate and manage data for subsequent data analysis and mining;

[0014] User interface and interaction module: responsible for providing an intuitive user interface, allowing users to view data analysis results, monitor system status, and configure system parameters through a graphical interface;

[0015] The data acquisition module transmits the collected data to the data processing module, and the data processing module transmits the processed data to the intelligent analysis module and the data storage and management module. The resource allocation module collects the performance indicators of each node and automatically adjusts the system configuration, such as starting or stopping the computing node, migrating tasks to more suitable nodes, etc. The intelligent analysis module transmits the analysis results to the user interface and interaction module for display. The intelligent analysis module cooperates with the real-time monitoring and early warning module to trigger early warning notifications based on the analysis results. The real-time monitoring and early warning module transmits the monitoring results and early warning notifications to the user interface and interaction module for display. The data storage and management module provides a data access interface for the intelligent analysis module, the real-time monitoring and early warning module, and the user interface and interaction module.

[0016] The above technical solution further includes:

[0017] Furthermore, the data acquisition module includes a data source interface unit, a data format parsing unit, a data cache and queue unit, and a data verification unit. The data source interface unit is responsible for establishing connections with various data sources, including database connection pool management, message queue subscription, API request sending, etc. The data source interface unit is the starting point of data acquisition and ensures stable communication with the data source. The data format parsing unit performs format parsing on the received data and converts it into a unified data format within the system. The data format parsing unit supports parsing of multiple data formats to ensure data readability and convenience of subsequent processing. During the data acquisition process, in order to relieve system pressure and improve processing efficiency, a data cache and queue mechanism is introduced. The data cache and queue unit is responsible for temporarily storing the parsed data in the cache or placing it in the message queue for subsequent processing. The data verification unit performs preliminary verification on the collected data to ensure the accuracy and consistency of the data. The data verification unit is an important link in data quality control. The data source interface unit transmits the collected raw data to the data format parsing unit for format parsing. The parsed data is transmitted to the data cache and queue unit for temporary storage or queued for processing. The data in the cache or queue is taken out in sequence and enters the data verification unit for preliminary verification.

[0018] Furthermore, the data processing module includes a data receiving unit, a preprocessing unit, a quality control unit and a distribution control unit. The data receiving unit is responsible for receiving the collected data transmitted from the data acquisition module. The preprocessing unit contains execution components for functions such as data cleaning, deduplication, and conversion, and uses data cleaning algorithms, deduplication technology and conversion tools for processing. The quality control unit performs quality checks on the preprocessed data to ensure the accuracy and consistency of the data, and includes mechanisms such as data verification and integrity checks. The distribution control unit distributes the processed data to the intelligent analysis module and the data storage and management module according to configuration or business requirements.

[0019] Furthermore, the resource allocation module includes a monitoring unit, an evaluation unit, an analysis unit, a decision unit, an execution unit, a feedback unit, and an optimization unit. The monitoring unit is responsible for collecting performance indicators of each node in the system in real time, such as CPU usage, memory usage, disk I / O, network bandwidth, etc. The evaluation unit evaluates the current system load and resource utilization efficiency based on the monitoring data and determines whether resource allocation is needed. The analysis unit predicts resource requirements for a period of time in the future based on business requirements, the complexity of data processing tasks, and current resource usage. The decision unit formulates resource allocation strategies based on resource requirements and the current resource status of the system, such as adding or reducing computing nodes and adjusting load distribution between nodes. The execution unit automatically adjusts the system configuration based on the decision results, such as starting or stopping computing nodes and migrating tasks to more suitable nodes. The feedback unit collects system performance data after resource allocation and evaluates the allocation effect. The optimization unit continuously optimizes the resource allocation strategy based on the feedback results to improve the overall stability and efficiency of the system. All units in the resource allocation module work together to form a closed-loop dynamic resource allocation.

[0020] Furthermore, the evaluation unit evaluates the current system load and resource utilization efficiency to determine whether resource allocation is required. Specifically, the following steps are performed:

[0021] Data preprocessing: Clean, organize, and standardize the performance indicators of each node in the collection system to eliminate noise, fill missing values, and ensure data consistency and comparability;

[0022] Key indicator extraction: Extract key indicators that are crucial for evaluating system load and resource utilization efficiency from preprocessed data. These key indicators include CPU usage, memory usage, disk I / O response time, network latency, etc.

[0023] Load assessment: Use machine learning algorithms to assess the current system load level, including whether the system is overloaded, normal, or idle.

[0024] Furthermore, in load assessment, LSTM is used to evaluate the current system load level. The specific steps are as follows:

[0025] Data preparation:

[0026] Obtain time series data from system monitoring tools, including CPU usage, memory usage, disk I / O, network bandwidth, etc., divide the time series data into fixed-length time windows (for example, data every minute or every hour is considered a sample), and extract features from each time window. The features include the original data value, statistics (such as mean and standard deviation), and possible lagged values ​​(i.e., data at past time points). The feature dataset is divided into a training set, a validation set, and a test set, with a ratio of 70%, 15%, and 15% respectively.

[0027] Model construction:

[0028] Building a model using LSTM:

[0029] Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0030] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means h t-1 and x t Splice into a vector;

[0031] Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: a sigmoid layer that determines which information will be updated; and a tanh layer that creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state.

[0032] Sigmoid layer: i t =σ(W i ·[h t-1 ,x t ]+b i );

[0033] tanh layer:

[0034] Cell status update:

[0035] Output gate: The output gate determines which part of the information based on the cell state is used for output, based on the current input txt and the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the combined action of sigmoid function and tanh function, the output h at the current moment is determined t ;

[0036] Sigmoid layer: o t =σ(W o ·[h t-1 ,x t ]+b o );

[0037] Output hidden state: h t =o t *tanh(C t );

[0038] Training process:

[0039] Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN;

[0040] Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label;

[0041] Backpropagation: Calculate the gradient of loss with respect to model parameters through the time backpropagation algorithm;

[0042] Parameter update: Use Adam to update model parameters according to the gradient;

[0043] Testing and Reasoning:

[0044] Use the trained LSTM model to predict the test set or real-time data to obtain the system load forecast value at a future time point. Based on the prediction results and the preset threshold, determine whether the system is currently or will be in an overloaded, normal, or idle state.

[0045] Furthermore, the intelligent analysis module includes a data analysis and mining unit, a result interpretation and insight extraction unit, and an interface and interaction unit. The data analysis and mining unit performs analysis and mining operations, the result interpretation and insight extraction unit interprets and explains the analysis results, and extracts information and insights that are valuable to the business. The interface and interaction unit is responsible for interacting with external systems or users, receiving input data, and displaying analysis results. The interface and interaction unit needs to provide an API interface so that other systems or applications can call the functions of the intelligent analysis module.

[0046] Furthermore, the real-time monitoring and early warning module includes a monitoring data acquisition unit, a data processing and analysis unit, a visualization display unit and an early warning notification unit. The monitoring data acquisition unit is responsible for collecting monitoring data from various components and services of the system to ensure the accuracy and real-time nature of the data. The monitoring data acquisition unit is integrated with the monitoring tool to realize automatic data collection and transmission. The data processing and analysis unit processes and analyzes the collected monitoring data to extract key performance indicators and abnormal information. The data processing and analysis unit determines whether the system performance is abnormal based on preset thresholds and rules. The visualization display unit uses visualization tools to display the processed data in the form of charts, dashboards, etc. to the operation and maintenance personnel. The visualization display unit supports custom configuration and interactive operations to improve monitoring efficiency and user experience. When system performance abnormalities are detected or data exceeds the threshold, the early warning notification unit is responsible for generating early warning notifications. The early warning notification unit supports multiple notification methods to ensure that the early warning information can be conveyed to the operation and maintenance personnel in a timely and accurate manner.

[0047] Furthermore, the data storage and management module includes a data storage unit, a data integration unit, a data access optimization unit and a data warehouse management unit. The data storage unit is responsible for data storage, including data writing, reading, updating and deleting operations. The data storage unit is tightly integrated with the distributed storage system to achieve distributed storage and access of data. The data integration unit is responsible for extracting data from multiple data sources and performing cleaning, conversion and integration to build a unified data warehouse, involving data ETL (Extract, Transform, Load) process to ensure data accuracy and consistency. The data access optimization unit is responsible for optimizing the data storage structure and query algorithm to improve data access efficiency. The data access optimization unit includes the creation and maintenance of data indexes, optimization of query statements, design of cache strategies, etc. The data warehouse management unit is responsible for the creation, maintenance and management of the data warehouse, including the architecture design of the data warehouse, the design of data models, the monitoring of data quality, etc. The data warehouse management unit provides an access interface to the data warehouse to support users in performing data query and analysis operations. The data integration unit extracts data from multiple data sources and writes the cleaned and converted data into the data storage unit for persistent storage. The data access optimization unit designs and optimizes query algorithms and data indexes based on the data structure and storage characteristics in the data storage unit. The data integration unit passes the integrated data to the data warehouse management unit.

[0048] Furthermore, the user interface and interaction module includes a front-end display unit, a back-end service unit and a configuration management unit. The front-end display unit is responsible for rendering and displaying the user interface, interacting with the back-end service unit to obtain data and send requests. The front-end display unit uses AJAX, Fetch and other technologies to realize asynchronous data interaction to improve user experience. The back-end service unit processes the request sent by the front-end display unit, calls the intelligent analysis module and the real-time monitoring and early warning module to obtain data or perform operations, and the back-end service unit returns the processing results to the front-end display unit for display. The back-end service unit is responsible for the implementation and management of the RESTful API interface. The configuration management unit manages system configuration parameters and provides the functions of reading, modifying and saving configuration information. The configuration management unit interacts with the back-end service unit to process configuration requests.

[0049] The present invention has the following beneficial effects:

[0050] 1. In the present invention, the resource allocation module can dynamically allocate computing resources according to the system load, ensuring that sufficient processing power can be provided under high load. This intelligent resource allocation mechanism helps to optimize system performance, reduce resource waste, and improve the overall stability and reliability of the system.

[0051] 2. In the present invention, the real-time monitoring and early warning module can monitor the various performance indicators of the system in real time and issue early warning notifications in time when the system has abnormalities or potential risks. This timely monitoring and early warning mechanism helps enterprises respond and handle problems quickly and reduce losses caused by system failures or performance degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a system block diagram of a computer data acquisition, processing and analysis system proposed by the present invention;

[0053] Figure 2 This is a system block diagram of the resource allocation module in the present invention;

[0054] Figure 3 This is a flow chart of the evaluation unit in the present invention evaluating the current system load and resource utilization efficiency. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] See also Figure 1 As shown, the present invention is a computer data acquisition, processing and analysis system, comprising:

[0057] Data collection module: Responsible for collecting data from various data sources, including relational databases, non-relational databases, real-time data streams (such as message queues such as Kafka and RabbitMQ), API interfaces, etc. The data collection module needs to support access to multiple data formats and be able to realize real-time or batch data collection;

[0058] Data processing module: responsible for pre-processing the collected data, such as cleaning, deduplication, and conversion, to ensure the accuracy and consistency of the data;

[0059] Resource allocation module: dynamically allocates computing resources according to system load to ensure sufficient processing power under high load;

[0060] Intelligent analysis module: responsible for in-depth analysis and mining of processed data to extract valuable information and insights;

[0061] Real-time monitoring and early warning module: responsible for real-time monitoring of various performance indicators of the system and displaying the monitoring results through visualization tools (such as Grafana, Kibana, etc.);

[0062] Data storage and management module: responsible for persistent storage and efficient access to data, using distributed storage systems (such as HDFS, HBase, etc.) to store large amounts of data, and building data warehouses to integrate and manage data for subsequent data analysis and mining;

[0063] User interface and interaction module: responsible for providing an intuitive user interface, allowing users to view data analysis results, monitor system status, and configure system parameters through a graphical interface;

[0064] The data acquisition module transmits the collected data to the data processing module, the data processing module transmits the processed data to the intelligent analysis module and the data storage and management module, the resource allocation module collects the performance indicators of each node and automatically adjusts the system configuration, such as starting or stopping the computing node, migrating tasks to more suitable nodes, etc. The intelligent analysis module transmits the analysis results to the user interface and interaction module for display. The intelligent analysis module collaborates with the real-time monitoring and early warning module to trigger early warning notifications based on the analysis results. The real-time monitoring and early warning module transmits the monitoring results and early warning notifications to the user interface and interaction module for display. The data storage and management module provides a data access interface for the intelligent analysis module, real-time monitoring and early warning module, and user interface and interaction module.

[0065] The working principle of a computer data acquisition, processing and analysis system proposed by the present invention is that the system first collects data from multiple data sources through a data acquisition module, and then pre-processes it through a data processing module to ensure data quality. The data acquisition module configures corresponding connection parameters and authentication information according to the data source type (database, message queue, API, etc.) to ensure that it can correctly connect to the data source. The data acquisition module defines the logic of data acquisition according to business needs, including the collected tables or files, fields, data formats, etc., and whether incremental acquisition or full acquisition is required. The data acquisition module collects data from the data source according to predefined rules. For real-time data streams, streaming processing technology is used for real-time acquisition. For batch data, timed tasks or event triggering are used for acquisition;

[0066] The resource allocation module dynamically adjusts resource allocation according to the system load. The resource allocation module monitors various system performance indicators in real time, such as CPU usage, memory usage, network bandwidth, etc. The resource allocation module analyzes the system load based on the monitored performance indicators and determines whether resource allocation needs to be adjusted. Based on the analysis results, the module automatically adjusts the system configuration, such as adding or reducing computing nodes and adjusting task allocation, to cope with different load requirements.

[0067] The intelligent analysis module conducts in-depth analysis of data and extracts valuable information. The real-time monitoring and early warning module monitors system performance and issues early warnings when anomalies are detected. The data storage and management module is responsible for persistent storage and efficient access to data. The user interface and interaction module provides an intuitive user interface to facilitate users to view and configure the system.

[0068] In one embodiment, for the above-mentioned data acquisition module, the data acquisition module includes a data source interface unit, a data format parsing unit, a data cache and queue unit, and a data verification unit. The data source interface unit is responsible for establishing connections with various data sources, including database connection pool management, message queue subscription, API request sending, etc. The data source interface unit is the starting point of data acquisition and ensures stable communication with the data source. The data format parsing unit performs format parsing on the received data and converts it into a unified data format within the system. The data format parsing unit supports parsing of multiple data formats to ensure data readability and convenience of subsequent processing. During the data acquisition process, in order to relieve system pressure and improve processing efficiency, a data cache and queue mechanism is introduced. The data cache and queue unit is responsible for temporarily storing the parsed data in the cache or placing it in the message queue for subsequent processing. The data verification unit performs preliminary verification on the collected data to ensure data accuracy and consistency. The data verification unit is an important link in data quality control. The data source interface unit transmits the collected raw data to the data format parsing unit for format parsing. The parsed data is transmitted to the data cache and queue unit for temporary storage or queueing for processing. The data in the cache or queue is taken out in sequence and enters the data verification unit for preliminary verification.

[0069] In one embodiment, for the above-mentioned data processing module, the data processing module includes a data receiving unit, a preprocessing unit, a quality control unit and a distribution control unit. The data receiving unit is responsible for receiving the collected data transmitted from the data acquisition module. The preprocessing unit contains execution components for functions such as data cleaning, deduplication, and conversion, and uses data cleaning algorithms, deduplication technology and conversion tools for processing. The quality control unit performs quality checks on the preprocessed data to ensure the accuracy and consistency of the data, and includes mechanisms such as data verification and integrity checks. The distribution control unit distributes the processed data to the intelligent analysis module and the data storage and management module according to configuration or business requirements.

[0070] In one embodiment, the resource allocation module includes a monitoring unit, an evaluation unit, an analysis unit, a decision unit, an execution unit, a feedback unit, and an optimization unit. The monitoring unit is responsible for collecting performance indicators of each node in the system in real time, such as CPU usage, memory usage, disk I / O, network bandwidth, etc. The evaluation unit evaluates the current system load and resource utilization efficiency based on the monitoring data and determines whether resource allocation is needed. The analysis unit predicts resource demand in the future based on business requirements, the complexity of data processing tasks, and current resource usage. The decision unit formulates resource allocation strategies based on resource requirements and the current resource status of the system, such as adding or reducing computing nodes and adjusting load distribution between nodes. The execution unit automatically adjusts the system configuration based on the decision results, such as starting or stopping computing nodes and migrating tasks to more suitable nodes. The feedback unit collects system performance data after resource allocation and evaluates the allocation effect. The optimization unit continuously optimizes the resource allocation strategy based on the feedback results to improve the overall stability and efficiency of the system. All units of the resource allocation module work together to form a closed-loop dynamic resource allocation.

[0071] In one embodiment, for the above-mentioned evaluation unit, the evaluation unit evaluates the current system load and resource utilization efficiency and determines whether resource allocation is required. Specifically, the steps are as follows:

[0072] Data preprocessing: Clean, organize, and standardize the performance indicators of each node in the collection system to eliminate noise, fill missing values, and ensure data consistency and comparability;

[0073] Key indicator extraction: Extract key indicators that are crucial for evaluating system load and resource utilization efficiency from preprocessed data. Key indicators include CPU usage, memory usage, disk I / O response time, network latency, etc.

[0074] Load assessment: Use machine learning algorithms to assess the current system load level, including whether the system is overloaded, normal, or idle.

[0075] In one embodiment, for the above load evaluation, LSTM is used to evaluate the load level of the current system. Specifically, the following steps are performed:

[0076] Data preparation:

[0077] Obtain time series data from system monitoring tools, including CPU usage, memory usage, disk I / O, network bandwidth, etc., divide the time series data into fixed-length time windows (for example, data every minute or every hour is considered a sample), and extract features from each time window. Features include raw data values, statistics (such as mean and standard deviation), and possible lagged values ​​(i.e., data from past time points). Split the feature dataset into training, validation, and test sets, with a ratio of 70%, 15%, and 15%;

[0078] Model construction:

[0079] Building a model using LSTM:

[0080] Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0081] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means h t-1 and x t Splice into a vector;

[0082] Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: a sigmoid layer that determines which information will be updated; and a tanh layer that creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state.

[0083] Sigmoid layer: i t =σ(W i ·[h t-1 ,x t ]+b i );

[0084] tanh layer:

[0085] Cell status update:

[0086] Output gate: The output gate determines which part of the information based on the cell state is used for output, according to the current input tx t , the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the combined action of sigmoid function and tanh function, the output h at the current moment is determinedt ;

[0087] Sigmoid layer: o t =σ(W o ·[h t-1 ,x t ]+b o );

[0088] Output hidden state: h t =o t *tanh(C t );

[0089] Training process:

[0090] Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN;

[0091] Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label;

[0092] Backpropagation: Calculate the gradient of loss with respect to model parameters through the time backpropagation algorithm;

[0093] Parameter update: Use Adam to update model parameters according to the gradient;

[0094] Testing and Reasoning:

[0095] Use the trained LSTM model to predict the test set or real-time data to obtain the system load forecast value at a future time point. Based on the prediction results and the preset threshold, determine whether the system is currently or will be in an overloaded, normal, or idle state.

[0096] In one embodiment, for the above-mentioned intelligent analysis module, the intelligent analysis module includes a data analysis and mining unit, a result interpretation and insight extraction unit, and an interface and interaction unit. The data analysis and mining unit performs analysis and mining operations, the result interpretation and insight extraction unit interprets and interprets the analysis results, and extracts information and insights that are valuable to the business. The interface and interaction unit is responsible for interacting with external systems or users, receiving input data, and displaying analysis results. The interface and interaction unit needs to provide an API interface so that other systems or applications can call the functions of the intelligent analysis module.

[0097] In one embodiment, for the above-mentioned real-time monitoring and early warning module, the real-time monitoring and early warning module includes a monitoring data acquisition unit, a data processing and analysis unit, a visualization display unit and an early warning notification unit. The monitoring data acquisition unit is responsible for collecting monitoring data from various components and services of the system to ensure the accuracy and real-time nature of the data. The monitoring data acquisition unit is integrated with the monitoring tool to realize automatic data collection and transmission. The data processing and analysis unit processes and analyzes the collected monitoring data to extract key performance indicators and abnormal information. The data processing and analysis unit determines whether the system performance is abnormal based on preset thresholds and rules. The visualization display unit uses visualization tools to display the processed data in the form of charts, dashboards, etc. to the operation and maintenance personnel. The visualization display unit supports custom configuration and interactive operations to improve monitoring efficiency and user experience. When system performance abnormalities are detected or data exceeds the threshold, the early warning notification unit is responsible for generating early warning notifications. The early warning notification unit supports multiple notification methods to ensure that the early warning information can be conveyed to the operation and maintenance personnel in a timely and accurate manner.

[0098] In one embodiment, for the above-mentioned data storage and management module, the data storage and management module includes a data storage unit, a data integration unit, a data access optimization unit and a data warehouse management unit. The data storage unit is responsible for data storage, including operations such as writing, reading, updating and deleting data. The data storage unit is closely integrated with the distributed storage system to realize distributed storage and access of data. The data integration unit is responsible for extracting data from multiple data sources and performing cleaning, conversion and integration to build a unified data warehouse, involving data ETL (Extract, Transform, Load) process to ensure the accuracy and consistency of data. The data access optimization unit is responsible for optimizing the data storage structure and query algorithm to To improve data access efficiency, the data access optimization unit includes the creation and maintenance of data indexes, optimization of query statements, design of cache strategies, etc. The data warehouse management unit is responsible for the creation, maintenance and management of the data warehouse, including the architecture design of the data warehouse, the design of the data model, the monitoring of data quality, etc. The data warehouse management unit provides an access interface to the data warehouse to support users in data query and analysis operations. The data integration unit extracts data from multiple data sources and writes the cleaned and converted data to the data storage unit for persistent storage. The data access optimization unit designs and optimizes query algorithms and data indexes based on the data structure and storage characteristics in the data storage unit. The data integration unit passes the integrated data to the data warehouse management unit.

[0099] In one embodiment, for the above-mentioned user interface and interaction module, the user interface and interaction module includes a front-end display unit, a back-end service unit and a configuration management unit. The front-end display unit is responsible for rendering and displaying the user interface, interacting with the back-end service unit to obtain data and send requests. The front-end display unit uses AJAX, Fetch and other technologies to realize asynchronous data interaction to improve user experience. The back-end service unit processes the request sent by the front-end display unit, calls the intelligent analysis module and the real-time monitoring and early warning module to obtain data or perform operations, and the back-end service unit returns the processing results to the front-end display unit for display. The back-end service unit is responsible for the implementation and management of the RESTful API interface. The configuration management unit manages system configuration parameters and provides the functions of reading, modifying and saving configuration information. The configuration management unit interacts with the back-end service unit to process configuration requests.

[0100] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A computer data acquisition, processing and analysis system, characterized in that: include: Data acquisition module: responsible for collecting data; Data processing module: responsible for pre-processing the collected data; Resource allocation module: dynamically allocates computing resources according to system load; Intelligent analysis module: responsible for in-depth analysis and mining of processed data to extract valuable information and insights; Real-time monitoring and early warning module: responsible for real-time monitoring of various performance indicators of the system and displaying the monitoring results through visualization tools; Data storage and management module: responsible for persistent storage and efficient access of data, and building a data warehouse to integrate and manage data; User interface and interaction module: responsible for providing the user interface, allowing users to view data analysis results, monitor system status, and configure system parameters; The data acquisition module transmits the collected data to the data processing module, the data processing module transmits the processed data to the intelligent analysis module and the data storage and management module, the resource allocation module collects the performance indicators of each node and automatically adjusts the system configuration, the intelligent analysis module transmits the analysis results to the user interface and interaction module for display, the intelligent analysis module cooperates with the real-time monitoring and early warning module to trigger early warning notifications based on the analysis results, the real-time monitoring and early warning module transmits the monitoring results and early warning notifications to the user interface and interaction module for display, and the data storage and management module provides a data access interface for the intelligent analysis module, the real-time monitoring and early warning module, and the user interface and interaction module.

2. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The data acquisition module includes a data source interface unit, a data format parsing unit, a data cache and queue unit, and a data verification unit. The data source interface unit is responsible for establishing connections with various data sources. The data format parsing unit performs format parsing on the received data and converts it into a unified data format within the system. During the data acquisition process, a data cache and queue mechanism is introduced. The data cache and queue unit is responsible for temporarily storing the parsed data in the cache or placing it in the message queue for subsequent processing. The data verification unit performs preliminary verification on the collected data. The data source interface unit transmits the collected original data to the data format parsing unit for format parsing. The parsed data is transmitted to the data cache and queue unit for temporary storage or queueing for processing. The data in the cache or queue is taken out in sequence and enters the data verification unit for preliminary verification.

3. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The data processing module includes a data receiving unit, a preprocessing unit, a quality control unit and a distribution control unit. The data receiving unit is responsible for receiving the collected data transmitted from the data acquisition module. The preprocessing unit uses data cleaning algorithms, deduplication technology and conversion tools for processing. The quality control unit performs quality inspection on the preprocessed data. The distribution control unit distributes the processed data to the intelligent analysis module and the data storage and management module according to configuration or business requirements.

4. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The resource allocation module includes a monitoring unit, an evaluation unit, an analysis unit, a decision unit, an execution unit, a feedback unit and an optimization unit. The monitoring unit is responsible for collecting the performance indicators of each node in the system in real time. The evaluation unit evaluates the current system load and resource utilization efficiency based on the monitoring data, and determines whether resource allocation is needed. The analysis unit predicts resource requirements in the future based on business requirements, the complexity of data processing tasks and current resource usage. The decision unit formulates a resource allocation strategy based on resource requirements and the current resource status of the system. The execution unit automatically adjusts the system configuration according to the decision results. The feedback unit collects system performance data after resource allocation and evaluates the allocation effect. The optimization unit continuously optimizes the resource allocation strategy according to the feedback results. All units of the resource allocation module work together to form a closed-loop dynamic resource allocation.

5. A computer data acquisition, processing and analysis system according to claim 4, characterized in that: The evaluation unit evaluates the current system load and resource utilization efficiency to determine whether resource allocation is required. The specific steps are: Data preprocessing: Clean, organize and standardize the performance indicators of each node in the collection system; Key indicator extraction: Extract key indicators that are crucial for evaluating system load and resource utilization efficiency from preprocessed data; Load assessment: Use machine learning algorithms to assess the current system load level, including whether the system is overloaded, normal, or idle.

6. A computer data acquisition, processing and analysis system according to claim 5, characterized in that: In load assessment, LSTM is used to evaluate the current system load level. The specific steps are as follows: Data preparation: Obtain time series data from system monitoring tools, divide the time series data into fixed-length time windows, extract features from each time window, and divide the feature dataset into training, validation, and test sets with a ratio of 70%, 15%, and 15%; Model construction: Building a model using LSTM: Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f ) Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means h t-1 and x t Splice into a vector; Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: a sigmoid layer that determines which information will be updated; and a tanh layer that creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state. Sigmoid layer: i t = σ(W i · [h t-1 , x t + b i ); tanh layer: Cell status update: Output gate: The output gate determines which part of the information based on the cell state is used for output, according to the current input tx t , the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the combined action of sigmoid function and tanh function, the output h at the current moment is determined t ; Sigmoid layer: o t = σ(W o · [h t-1 , x t + b o ); Output hidden state: h t =o t *tanh(C t ); Training process: Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN; Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label; Backpropagation: Calculate the gradient of loss with respect to model parameters through the time backpropagation algorithm; Parameter update: Use Adam to update model parameters according to the gradient; Testing and Reasoning: Use the trained LSTM model to predict the test set or real-time data to obtain the system load forecast value at a future time point. Based on the prediction results and the preset threshold, determine whether the system is currently or will be in an overloaded, normal, or idle state.

7. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The intelligent analysis module includes a data analysis and mining unit, a result interpretation and insight extraction unit, and an interface and interaction unit. The data analysis and mining unit performs analysis and mining operations, the result interpretation and insight extraction unit interprets and explains the analysis results, and extracts information and insights that are valuable to the business. The interface and interaction unit is responsible for interacting with external systems or users, receiving input data, and displaying analysis results.

8. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The real-time monitoring and early warning module includes a monitoring data acquisition unit, a data processing and analysis unit, a visualization display unit and an early warning notification unit. The monitoring data acquisition unit is responsible for collecting monitoring data from various components and services of the system. The data processing and analysis unit processes and analyzes the collected monitoring data and extracts key performance indicators and abnormal information. The data processing and analysis unit determines whether the system performance is abnormal based on preset thresholds and rules. The visualization display unit uses visualization tools to display the processed data to operation and maintenance personnel. When system performance abnormality is detected or data exceeds the threshold, the early warning notification unit is responsible for generating an early warning notification.

9. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The data storage and management module includes a data storage unit, a data integration unit, a data access optimization unit and a data warehouse management unit. The data storage unit is responsible for data storage. The data integration unit is responsible for extracting data from multiple data sources, and performing cleaning, conversion and integration to build a unified data warehouse. The data access optimization unit is responsible for optimizing the data storage structure and query algorithm. The data warehouse management unit is responsible for creating, maintaining and managing the data warehouse. The data warehouse management unit provides an access interface to the data warehouse to support users in performing data query and analysis operations. The data integration unit extracts data from multiple data sources and writes the cleaned and converted data into the data storage unit for persistent storage. The data access optimization unit designs and optimizes query algorithms and data indexes based on the data structure and storage characteristics in the data storage unit. The data integration unit passes the integrated data to the data warehouse management unit.

10. A computer data acquisition, processing and analysis system according to claim 1, characterized in that: The user interface and interaction module includes a front-end display unit, a back-end service unit and a configuration management unit. The front-end display unit is responsible for rendering and displaying the user interface, interacting with the back-end service unit to obtain data and send requests. The back-end service unit processes the requests sent by the front-end display unit, calls the intelligent analysis module and the real-time monitoring and early warning module to obtain data or perform operations, and the back-end service unit returns the processing results to the front-end display unit for display. The back-end service unit is responsible for the implementation and management of the RESTful API interface. The configuration management unit manages system configuration parameters and provides the functions of reading, modifying and saving configuration information. The configuration management unit interacts with the back-end service unit to process configuration requests.

Images