Computer environment data acquisition system and method

Abstract

The invention provides a computer environment data acquisition system and method, and relates to the technical field of computer data processing. The computer environment data acquisition system comprises a data acquisition layer, a data processing layer, a data storage layer and a data analysis layer, the data acquisition layer is used for receiving target area environment data transmitted by a data acquisition agent and sending the acquired target area environment data to the data processing layer, and the data processing layer and the data analysis layer are connected with the data storage layer. The method is used for cleaning target area environment data. By converting the data types into the same type, the problem that the data types are not matched can be effectively solved, the data cleaning quality is ensured, the accuracy and reliability of data analysis are improved, missing value processing is carried out on the data, the data quality can be improved, the decision-making ability can be enhanced, time and cost are saved, and the data cleaning efficiency is improved. Therefore, a more reliable basis is provided for data analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a computer environment data acquisition system and method. Background Art

[0002] The task of the computer environment data acquisition system is to collect various environmental parameters and send them to the computer. The computer will perform corresponding calculations and processing as needed to obtain the required data, and display or print the data as required. This system is usually used to monitor and manage the performance, resource usage and security events of the computer system.

[0003] The invention with the existing application number CN201811248348.7 relates to an environmental data collection system and method based on information processing technology. By cleaning the current environmental data, the fault tolerance of the collected data is improved, thereby greatly improving the stability of the system, and the data receiving server, the base database server and the background supervision server are set separately. Since the base database server only stores the environmental data of the target agricultural field, when the client reads the data, the application server only obtains the corresponding environmental data from the base database server, so that the data between enterprises are not exposed to each other, which greatly protects the data privacy of the enterprise and enhances the privacy protection performance of the system.

[0004] However, the above-mentioned environmental data collection system still has the following problems when in use: it may encounter various problems during the data cleaning process, including data type mismatch, improper handling of missing values ​​and data duplication. These problems may affect the quality and accuracy of the data, and thus affect the results of data analysis.

[0005] To this end, we have developed a new computer environment data collection system and method. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a computer environment data collection system and method to solve the problem.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a computer environment data acquisition system, which is built into a target computer, and the computer environment data acquisition system includes a data processing module, a data acquisition agent and a user interface layer. The data processing module includes a data acquisition layer, a data processing layer, a data storage layer and a data analysis layer. The data acquisition layer is used to receive the target area environment data transmitted by the data acquisition agent, and send the collected target area environment data to the data processing layer. The data processing layer and the data analysis layer are used to clean the target area environment data, and perform data type conversion, missing value processing and data duplicate identification deduplication during the data cleaning process. The processed target area environment data is parsed by the data analysis layer and sent to the data storage layer after parsing.

[0008] Preferably, the data storage layer is used to send the target environment data to a cloud server for storage and backup, and to send the target environment data to a target computer via wired transmission. The user interface layer is configured on the client to respond to a data reading request from the client. The target environment data stored and backed up can be read from the cloud server, or the target computer can be connected via a wired connection from the client and the target environment data can be read and fed back to the client. After receiving the target environment data, the client displays the target environment data on a front-end page.

[0009] Preferably, the data acquisition agent type is an HTTP / HTTPS agent, and the data acquisition agent is configured with a temperature sensor, a humidity sensor, a carbon dioxide concentration sensor, a wind speed sensor, an air pressure sensor, a light sensor, a soil moisture sensor, a noise sensor, and an air quality sensor to collect corresponding detection parameters in the environment in real time.

[0010] Preferably, the conversion method of the data type conversion category is explicit type conversion. In the data type conversion category, the data type needs to be clarified in advance, and type hints are used, and data verification is performed at the end.

[0011] Preferably, the missing value processing includes deleting missing values ​​and filling missing values. The missing value deletion method reduces the missing values ​​in the data set by deleting rows or columns containing missing values. The missing value filling method replaces the missing values ​​with specific values, means, medians or modes, etc. The missing value processing also includes interpolation methods and model prediction methods. The interpolation method estimates missing values ​​based on existing data, and the model prediction method uses a machine learning model to predict missing values.

[0012] Preferably, the data duplication identification and deduplication includes identifying duplicate data and deduplication, the identifying duplicate data includes using identifiers and data comparison, the deduplication method includes two steps: fuzzy deduplication and precise deduplication, and the tools used in the data duplication identification and deduplication include Pandas library and OpenRefine.

[0013] The present invention provides a computer environment data collection system and method. It has the following beneficial effects: A computer environment data acquisition system and method. Data type mismatch may lead to inaccurate data analysis results and affect decision-making. By converting data types to the same type, the problem of data type mismatch can be effectively handled to ensure the quality of data cleaning, thereby improving the accuracy and reliability of data analysis.

[0014] In this computer environment data acquisition system and method, processing missing values ​​is a key step, because missing values ​​may affect the accuracy of data analysis and model training. By processing missing values ​​in data, not only can data quality be improved, but decision-making capabilities can also be enhanced, time and costs can be saved, thereby providing a more reliable basis for data analysis.

[0015] This computer environment data collection system and method, avoiding data duplication is a key step to ensure data quality and analysis accuracy. By deduplicating data through duplicate identification, data quality can be improved, storage costs can be reduced, data processing efficiency can be improved, error rates can be reduced, data reliability can be improved, user experience can be enhanced, and data privacy can be protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural block diagram of the environmental data acquisition system of the present invention; Figure 2 A structural block diagram of each sensor configured for the data acquisition agent of the present invention; Figure 3 The figure is a flow chart of the environmental data collection method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0018] like Figure 1-2As shown, an embodiment of the present invention provides a computer environment data acquisition system, which is built in a target computer. The computer environment data acquisition system includes a data processing module, a data acquisition agent and a user interface layer. The data processing module includes a data acquisition layer, a data processing layer, a data storage layer and a data analysis layer. The data acquisition layer is used to receive the target area environment data transmitted by the data acquisition agent, and send the collected target area environment data to the data processing layer. The data processing layer and the data analysis layer are used to clean the target area environment data, and perform data type conversion, missing value processing and data duplicate identification deduplication during the data cleaning process. The processed target area environment data is parsed by the data analysis layer and sent to the data storage layer after parsing. The data storage layer is used to send the target environment data to the cloud server for storage and backup, and send the target environment data to the target computer through wired transmission. The user interface layer is configured on the client to respond to the client's data reading request, and can read the stored and backed up target environment data from the cloud server, or can be connected to the target computer through the client wired connection, and read the target environment data and feed it back to the client. After receiving the target environment data, the client displays the target environment data on the front-end page.

[0019] Among them, the data collection agent type is HTTP / HTTPS agent, which is suitable for most crawler data collection tasks, can process HTTP and HTTPS protocol requests, and is easy to integrate. The data collection agent is configured with temperature sensors, humidity sensors, carbon dioxide concentration sensors, wind speed sensors, air pressure sensors, light sensors, soil moisture sensors, noise sensors, and air quality sensors to collect the corresponding detection parameters in the environment in real time.

[0020] Among them, the temperature sensor is used to measure the ambient temperature, which can collect ambient temperature data in real time to ensure the appropriate temperature of crops or experimental environments.

[0021] Among them, humidity sensors are used to monitor the humidity level in the environment. They are very useful for greenhouse management, meteorological monitoring and other scenarios. By monitoring humidity in real time, it can ensure that crops or the environment are in the optimal humidity state.

[0022] Among them, the carbon dioxide concentration sensor is used to measure the carbon dioxide concentration in the air, which is crucial for applications such as greenhouse management and air quality monitoring. It can help control the carbon dioxide level in the greenhouse and promote plant photosynthesis.

[0023] Among them, the wind speed sensor is used to measure wind speed. It can monitor wind speed changes in real time and provide data support for weather forecasting and crop management.

[0024] Among them, the air pressure sensor is used to measure atmospheric pressure. It can monitor air pressure changes in real time and help predict weather changes.

[0025] Among them, the light sensor is used to measure light intensity. It can monitor light intensity in real time to ensure that crops get enough light.

[0026] Among them, the soil moisture sensor is used to measure the moisture of the soil. It can monitor the soil moisture in real time to ensure that crops get the right amount of water.

[0027] Among them, the noise sensor is used to measure the ambient noise level. It can monitor the noise level in real time and help control noise pollution.

[0028] Among them, the air quality sensor is used to measure the concentration of pollutants in the air, such as PM2.5, PM10, sulfur dioxide, nitrogen oxides, etc. It can monitor air quality in real time and help assess the degree of environmental pollution.

[0029] Among them, in the data type conversion category, the data type needs to be clarified in advance. Before conducting data analysis, the analysis goals and data types should be clarified to ensure that the data types are consistent with expectations. Type hints should be used to use variable names in the code to reflect their corresponding data types, which helps to track and maintain the code. Finally, data validation should be performed. Using data validation tools can help us discover and correct errors in the data, including errors in data types.

[0030] Missing value processing includes deleting missing values ​​and filling missing values. Missing value deletion is a simple and direct method to reduce missing values ​​in a data set by deleting rows or columns containing missing values. In Pandas, this can be achieved using the dropna() method.

[0031] In the specific implementation, delete the rows containing missing values: df.dropna(axis=0); Drop columns containing missing values: df.dropna(axis=1).

[0032] Filling missing values ​​is another commonly used method to deal with missing values. It replaces missing values ​​with specific values, mean, median or mode to maintain the integrity of the data.

[0033] In the specific implementation, a fixed value is used for filling: df.fillna(value=0); Fill with mean: df.fillna(df.mean()); Fill with median: df.fillna(df.median()); Fill with mode: df.fillna(df.mode()); Forward filling (ffill): df.fillna(method='ffill'); Backward filling (bfill): df.fillna(method='bfill').

[0034] Among them, missing value processing also includes interpolation and model prediction. Interpolation is a more complex filling method that estimates missing values ​​based on existing data.

[0035] In the specific implementation, Pandas provides the interpolate() method, which supports multiple interpolation methods, such as linear interpolation, polynomial interpolation, etc.

[0036] Among them, linear interpolation: df.interpolate(method='linear'); Polynomial interpolation: df.interpolate(method='polynomial', order=2).

[0037] The model prediction method uses a machine learning model to predict missing values. This method has high computational complexity but high prediction accuracy.

[0038] In specific implementation, which method to choose depends on the nature of the data, the pattern of missing values, and the analysis objectives. Therefore, in practical applications, it is necessary to combine multiple methods to achieve the best results.

[0039] Among them, data duplication identification and deduplication include identifying duplicate data and deduplication. Identifying duplicate data includes using identifiers and data comparison, wherein the identifier is used to identify duplicate records by comparing the unique identifier of the data (such as ID), wherein data comparison identifies duplicate values ​​in the data through methods such as data comparison and similarity calculation. The deduplication method includes two steps: fuzzy deduplication and precise deduplication. Among them, fuzzy deduplication, for text data, can use cosine similarity, Jaccard similarity and other methods to calculate the similarity between texts to determine whether they are duplicates. Among them, precise deduplication deduplicates data based on preset rules, such as the same field value.

[0040] In addition, it is important to note that when deleting duplicate data, you need to decide whether to keep the first occurrence of the record, the last occurrence of the record, or simply delete all duplicate records. In some cases, duplicate data may have business value, and you need to decide how to handle duplicate data based on specific business needs.

[0041] Among them, the tools used in data duplicate identification and deduplication include Pandas library and OpenRefine. In Python, Pandas library provides the drop_duplicates() method, which can easily delete duplicate rows in the data set, while OpenRefine is an open source data cleaning and conversion tool, which is particularly suitable for data organization and preprocessing. It supports running in the browser, which is convenient for editing data, deleting duplicates, and other operations.

[0042] like Figure 3 As shown, an embodiment of the present invention provides a computer environment data collection system collection method, The specific steps include: S1. Determine the collection target Performance indicators: such as CPU usage, memory usage, disk I / O, network traffic, etc. Environmental parameters: such as temperature, humidity, power status, etc.

[0043] S2. Select the acquisition tool Monitoring tools provided by the operating system: such as Linux's top, vmstat, iostat and other commands, Windows's task manager, etc. Third-party monitoring software: such as Nagios, Zabbix, Prometheus, etc. These tools provide a wealth of monitoring options and alarm functions; Custom scripts: Write scripts according to specific needs and use programming languages ​​such as Python and Perl to collect data.

[0044] S3. Design of data collection plan Sampling frequency: Determines how often data is collected, such as every second, every minute, or every hour; Data storage: Choose a suitable data storage method, such as database, file system, etc. Data transmission: If data collection points are distributed in different locations, the security and efficiency of data transmission need to be considered.

[0045] S4. Implementation of data collection Install and configure the acquisition tool: Install and configure the tool according to the selected tool; Write collection scripts: If necessary, write scripts to automate the data collection process; Deploy the collection system: Deploy the collection system to the target environment.

[0046] S5. Data processing and analysis Data cleaning: remove noise and outliers to ensure data accuracy; Data analysis: Use statistical methods and machine learning algorithms to analyze data and extract useful information; Visualization: Display analysis results in the form of charts, dashboards, etc. to facilitate understanding and decision-making.

[0047] S6. Monitoring and Maintenance Real-time monitoring: Continuously monitor the system's performance and environmental parameters to detect abnormal situations in a timely manner; Alarm mechanism: Set alarm thresholds, and automatically trigger alarms when data exceeds preset values; System maintenance: Regularly update and maintain collection tools and systems to ensure their stable operation; S7. Security and Compliance Processing Data encryption: ensure the security of data transmission and storage; Access control: limit access rights to collected data to prevent unauthorized access; Compliance check: Ensure that collection activities comply with relevant laws, regulations and industry standards.

[0048] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A computer environment data collection system, characterized in that: The computer environment data acquisition system is built into the target computer, and the computer environment data acquisition system includes a data processing module, a data acquisition agent and a user interface layer; The data processing module includes a data acquisition layer, a data processing layer, a data storage layer and a data analysis layer; The data collection layer is used to receive the target area environment data transmitted by the data collection agent, and send the collected target area environment data to the data processing layer; The data processing layer and the data analysis layer are used to clean the target area environmental data, and perform data type conversion, missing value processing, and data duplicate identification removal during the data cleaning process. The processed target area environmental data is parsed by the data analysis layer and sent to the data storage layer after parsing; The data storage layer is used to send the target environment data to a cloud server for storage and backup, and to send the target environment data to a target computer via wired transmission; The user interface layer is configured on the client and is used to respond to the client's data reading request. It can read the target environment data stored and backed up from the cloud server, or it can be connected to the target computer via the client wired connection and read the target environment data and feed it back to the client. After the client receives the target environment data, it displays the target environment data on the front-end page.

2. A computer environment data acquisition system according to claim 1, characterized in that: The data collection proxy type is HTTP / HTTPS proxy.

3. A computer environment data acquisition system according to claim 1, characterized in that: The data acquisition agent is configured with a temperature sensor, a humidity sensor, a carbon dioxide concentration sensor, a wind speed sensor, an air pressure sensor, a light sensor, a soil moisture sensor, a noise sensor, and an air quality sensor to collect corresponding detection parameters in the environment in real time.

4. A computer environment data acquisition system according to claim 1, characterized in that: The conversion method of the same type as the data type conversion is explicit type conversion.

5. A computer environment data acquisition system according to claim 1, characterized in that: In the data type conversion category, it is necessary to clarify the data type in advance, use type hints, and perform data verification at the end.

6. A computer environment data acquisition system according to claim 1, characterized in that: The missing value processing includes deleting missing values ​​and filling missing values. The missing value deletion method reduces the missing values ​​in the data set by deleting rows or columns containing missing values, and the missing value filling method replaces the missing values ​​with specific values, means, medians or modes.

7. A computer environment data acquisition system according to claim 1, characterized in that: The missing value processing also includes interpolation method and model prediction method. The interpolation method estimates the missing values ​​based on existing data, and the model prediction method predicts the missing values ​​using a machine learning model.

8. A computer environment data acquisition system according to claim 1, characterized in that: The data duplication identification and deduplication includes identifying duplicate data and deduplication, the identifying duplicate data includes using identifiers and data comparison, and the deduplication method includes two steps: fuzzy deduplication and precise deduplication.

9. A computer environment data collection system according to claim 1, characterized in that: The tools used in the data duplication identification and deduplication include Pandas library and OpenRefine.

10. A computer environment data collection system collection method according to any one of claims 1 to 9, characterized in that: The specific steps include: S1. Determine the acquisition objectives, including performance indicators and environmental parameters; S2. Select the collection tools, including the monitoring tools provided by the operating system, third-party monitoring software, and custom scripts; S3. Design data acquisition scheme, including sampling frequency, data storage and data transmission; S4. Implement data collection, including installing and configuring collection tools, writing collection scripts, and deploying collection systems; S5. Data processing and analysis, including data cleaning, data analysis and visualization; S6. Monitoring and maintenance, including real-time monitoring, alarm mechanisms and system maintenance; S7. Security and compliance processing, including data encryption, access control, and compliance checks.

Images