Data acquisition system and acquisition method based on big data

Abstract

The invention provides a data acquisition system and method based on big data, and relates to the field of data acquisition systems. According to the big data-based data acquisition system and acquisition method, the data acquisition system comprises four functional modules, namely a data management module, a data acquisition module, a system monitoring module and a hardware management module; the data acquisition module performs data acquisition through four acquisition modes of database acquisition, system log acquisition, file text acquisition and sensor data acquisition, and the database acquisition module comprises three acquisition modes of relational database acquisition, NoSQL database acquisition and web crawler acquisition. According to the method, large-scale data collection can be realized, a large amount of data can be collected in a short time regardless of various business systems and server logs in an enterprise or external social media, sensor networks and the like, and the requirement of big data analysis is met.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition systems, and in particular to a data acquisition system and acquisition method based on big data. Background Art

[0002] With the development of network digitization, the amount of data in various fields has exploded, and traditional data acquisition systems are facing huge challenges. On the one hand, data sources are becoming increasingly diversified, and traditional databases or network data are exponentially growing. At the same time, enterprises and organizations have higher and higher requirements for the real-time and accuracy of data. In a rapidly changing market environment, timely acquisition of accurate data is crucial to making wise decisions. The emergence of big data technology has provided new ideas for solving these problems. Data acquisition systems based on big data can handle large-scale and diversified data, realize efficient data integration and real-time acquisition, so as to quickly process massive data and provide accurate and reliable data support. Therefore, the present invention proposes a data acquisition system based on big data to provide enterprises and institutions with a more comprehensive data acquisition solution. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In view of the deficiencies in the prior art, the present invention provides a data collection system and a collection method based on big data. The big data collection system uses sensors and establishes a database collection tool on a network server to perform two collection methods: hardware data collection and network data collection. In addition, the collection module establishes multiple collection methods such as system log collection and file text collection to deal with server internal data and network data. Large-scale data collection can be achieved, whether it is from various business systems and server logs within the enterprise, or external social media, sensor networks, etc. A large amount of data can be collected in a short time to meet the needs of big data analysis.

[0005] (II) Technical solution

[0006] To achieve the above purpose, the present invention is implemented through the following technical solutions: a data acquisition system and acquisition method based on big data, the data acquisition system includes four functional modules: data management module, data acquisition module, system monitoring module and hardware management module, the data acquisition module performs data acquisition through four acquisition modes: database acquisition, system log acquisition, file text acquisition and sensor data acquisition, the database acquisition module includes three acquisition modes: relational database acquisition, NoSQL database acquisition and web crawler acquisition, the system log acquisition module includes local log system data acquisition and centralized log system data acquisition, the file text acquisition module includes text file data acquisition and binary file data acquisition, the sensor data acquisition includes industrial sensor equipment acquisition and Internet of Things equipment acquisition, the data management module includes four functional modules: data interface management function, data transmission management function, data analysis management function and data storage management function, the system monitoring module includes performance monitoring module, data monitoring module, alarm device and security management module, the hardware management module includes database transmission module and network server module;

[0007] Through the above technical solution, the big data-based data collection system has three data collection methods: text library data collection, network data collection, and hardware data collection. It can support data collection from multiple data sources at the same time and can cope with high-concurrency data inflow scenarios.

[0008] Preferably, the relational database of the data acquisition module directly connects to the relational database using database connection tools JDBC and ODBC, extracts required data from the database by writing SQL query statements, and can be screened and aggregated according to specific condition fields. The NoSQL database acquisition uses corresponding database drivers and APIs for data acquisition for non-relational databases, and according to the characteristics of NoSQL databases, MongoDB uses document-based query language, and HBase performs data retrieval through table scanning and filters. The web crawler collection function automatically accesses web pages and extracts useful information therein by writing web crawler programs. The local log system collects a large number of system logs generated by servers and applications, records system operation status, and user behavior information. The log acquisition tool Logstash is used to collect data regularly, and the log data is parsed, filtered, and converted to extract information.

[0009] Through the above technical solutions, database collection supports database collection and network collection, and web crawlers are used to deal with network data collection. Network data can be collected in real time and in large quantities. Web crawlers automatically access web pages and extract useful information by writing web crawler programs, traverse websites on the Internet, and capture web page content, pictures, links and other data, thereby greatly enhancing the ability to collect big data.

[0010] Preferably, the text file data acquisition of the text file acquisition module recognizes the data of files in CSV, XML, and JSON formats, and the data acquisition system can read these files regularly, import the data into a database or analyze them, the binary file data acquisition uses a binary file parsing tool to read and process the data in these binary files, the industrial sensor device module of the sensor data acquisition module connects to some sensors in industrial production by wired or wireless means, and collects sensor data in real time, and the Internet of Things device module connects to intelligent electronic devices by Bluetooth or Wi-Fi connection and transmits the collected data to the acquisition system;

[0011] Through the above technical solutions, various sensors are widely used in industrial production to monitor the operating status of equipment and production process parameters. These sensors are connected by wired or wireless means to collect sensor data in real time, and analyze and process them, which helps to improve industrial production efficiency and quality. Through the analysis of IoT device data, intelligent applications can be realized, such as automated control of smart homes and smart health monitoring.

[0012] Preferably, the data interface management module includes a data source identification function and a data source classification and marking function. The data source identification function connects various types of data sources, such as an enterprise internal database, an external API interface, a web crawler target website, and a sensor device. The data source classification and marking function classifies and marks the received data so that each data can be accurately located. The data transmission management function includes a transmission protocol unification function and a breakpoint protection and resuming function. The breakpoint protection and resuming function can continue the transmission from the breakpoint after the connection is restored if a network interruption occurs during the data transmission process, thereby avoiding data loss.

[0013] Through the above technical solution, API interface calls can usually obtain more standardized and accurate data.

[0014] Preferably, the data analysis management module of the data management module includes a data verification and analysis function, a data sorting function and a data model building function. The data sorting function summarizes and sorts the collected data. The data verification and analysis function verifies the collected data to ensure the integrity and accuracy of the data, checks whether the data format is correct, whether the data range is reasonable, whether the data is missing, and cleans and repairs the data with problems to remove invalid data, duplicate data and erroneous data. The data model building function generates a model for the data. The model prediction module includes a function of generating the results of data analysis and discrimination and the results of priority judgment. The data storage management module includes a distributed storage function, a data backup verification function and a data recovery function.

[0015] Through the above technical solution, the backup data is verified regularly to ensure the integrity and availability of the backup data. The backup data can be restored to the test environment for data verification and functional testing.

[0016] Preferably, the performance monitoring module of the system monitoring module includes a real-time monitoring function, a hardware performance evaluation function and a performance testing function. The real-time monitoring function monitors the hardware performance indicators of the data acquisition system, such as CPU usage, memory usage, disk space and network bandwidth, in real time, monitors the hardware performance of the data acquisition system, controls the power of the data acquisition system, discovers system performance bottlenecks and potential problems, and takes corresponding measures to optimize and adjust. The hardware performance evaluation function tests the data acquisition, data transmission, data storage and data analysis functions to verify whether the system can accurately collect and process data. The performance testing function evaluates the performance of the data acquisition system under different loads. The data monitoring module includes a quantity monitoring function, a collection information recording function and a data tracing function.

[0017] Through the above technical solutions, performance testing can evaluate the performance of the data acquisition system under different loads, test the system's response time, throughput and other indicators under high-concurrency data acquisition, large data volume storage and processing, identify performance bottlenecks and optimize them. Through performance testing, problems that may arise in the system under high load can be discovered, and corresponding measures can be taken to optimize and improve them.

[0018] Preferably, the alarm device of the system monitoring module includes an alarm information sending function and an automatic blocking function. The alarm information sending function sets the system alarm threshold. When the system performance index or data collection index exceeds the threshold, an alarm notification is automatically issued so that the administrator can take measures in time. The automatic blocking function automatically stops the system data collection work when the system exceeds the index threshold and blocks the system permissions. The security management module of the system monitoring module includes a permission management function, an access record function and a data encryption function. The permission management function manages the permissions of system access personnel to prevent unauthorized access and data leakage, and ensures the security and confidentiality of data. The data encryption function encrypts the collected data according to the permission level to ensure the confidentiality, integrity and availability of the data, encrypts and stores sensitive data, and sets access rights. The data backup verification function formulates a data backup plan, regularly backs up the collected data to other storage media or remote servers, and regularly verifies the backup data to ensure the integrity and availability of the backup data. The data recovery function formulates a data recovery process and plan, clarifies the recovery steps and methods, and ensures that the data is recovered in the shortest time;

[0019] Through the above technical solution, the real-time monitoring module monitors the performance indicators of the data acquisition system, such as CPU utilization, memory utilization, disk space, network bandwidth, etc. Through performance monitoring, system performance bottlenecks and potential problems can be discovered in a timely manner, and corresponding measures can be taken to optimize and adjust them. The distributed data storage function establishes multiple off-site data storage centers for real-time multi-storage center storage, ensuring multiple data storage to ensure that data will not be lost and the normal operation of the entire data acquisition system.

[0020] Preferably, the database transmission module of the hardware control management module is provided with a data analysis module, and the data analysis module analyzes the data transmitted by the hardware device and transmits it to the data acquisition system, and the network service module collects network data through the acquisition server.

[0021] Working principle: The collection method steps of the data collection system based on big data are as follows:

[0022] S1: Clarify the objectives of data collection and determine the specific objectives of data collection, including the type of data to be collected and the time span;

[0023] S2: Determine the data source and data collection method, and select internal data source collection or external data source collection;

[0024] S3: Select appropriate collection methods according to the type and characteristics of data sources. For network big data, use web crawlers to collect massive amounts of data. For database data, use database connection tools to collect data. For device data, use sensors and IoT devices to collect data, and determine the data collection plan. For data, determine the collection time and frequency.

[0025] S4: Select appropriate hardware and software equipment according to the data collection plan, build the data collection environment, configure the parameters of the collection environment, and determine the database connection string, web crawler crawling rules, and data collection card sampling frequency required in the data collection process;

[0026] S5: Establish a data collection and storage warehouse, select the appropriate storage method for the collected data according to the characteristics and needs of the data, and establish a relational database, NoSQL database, and distributed file system. The data warehouse integrates and stores data from different data sources to provide a unified data view for data analysis;

[0027] S6: Establish a data storage security management mechanism, encrypt and store the collected important data, set access permissions, improve security protection, add version control, backup and recovery to the collected data, and ensure the safe storage of data.

[0028] (III) Beneficial effects

[0029] The present invention provides a data collection system and collection method based on big data, which has the following beneficial effects:

[0030] 1. The present invention provides a data collection system and collection method based on big data. The data collection system establishes a big data collection system based on data sensors and network servers. The big data collection system uses sensors and database collection tools established on network servers to perform two collection methods: hardware data collection and network data collection. In addition, the collection module establishes multiple collection methods such as system log collection and file text collection to deal with server internal data and network data. Large-scale data collection can be achieved, whether it is from various business systems within the enterprise, server logs, or external social media, sensor networks, etc. A large amount of data can be collected in a short time to meet the needs of big data analysis.

[0031] 2. The present invention provides a data acquisition system and method based on big data. By adding a data analysis management module to the data acquisition system, the collected data can be sorted in real time, and various types of data sources can be integrated, including structured data (such as table data in a database), semi-structured data (such as data in XML and JSON formats) and unstructured data (such as text, images, audio, video, etc.), and it has the function of data verification and analysis, which can verify and clean the data, remove invalid data, duplicate data and erroneous data, and ensure the integrity and validity of the data. At the same time, the big data acquisition system also has a distributed storage center, which performs distributed multiple backups of the collected data, and is also equipped with data recovery and data encryption functions to ensure the security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall system structure diagram of the data acquisition system based on big data of the present invention;

[0033] Figure 2 It is a system diagram of a data acquisition module of a data acquisition system based on big data of the present invention;

[0034] Figure 3 A system diagram of a data management module of a data acquisition system based on big data of the present invention;

[0035] Figure 4 A system diagram of a system monitoring module of a data acquisition system based on big data of the present invention;

[0036] Figure 5 It is a data collection flow chart of the data collection system based on big data of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of this application, it should be noted that the terms used here are only for describing specific implementations, and are not intended to limit the exemplary implementations according to the present application. For ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be regarded as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, not as a limitation. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0038] Embodiment 1:

[0039] like Figure 1-5As shown, an embodiment of the present invention provides a data acquisition system and acquisition method based on big data. The data acquisition system includes four functional modules: a data management module, a data acquisition module, a system monitoring module and a hardware management module. The data acquisition module performs data acquisition through four acquisition modes: database acquisition, system log acquisition, file text acquisition and sensor data acquisition. The database acquisition module includes three acquisition modes: relational database acquisition, NoSQL database acquisition and web crawler acquisition. The system log acquisition module includes local log system data acquisition and centralized log system data acquisition. The file text acquisition module includes text file data acquisition and binary file data acquisition. The sensor data acquisition includes industrial sensor equipment acquisition and Internet of Things equipment acquisition. The data management module includes four functional modules: data interface management function, data transmission management function, data analysis management function and data storage management function. The system monitoring module includes a performance monitoring module, a data monitoring module, an alarm device and a security management module. The hardware management module includes a database transmission module. And network server module, the data type in the big data system refers to the type and format of the data, and it should deal with various types of data types and formats. Database big data collection usually refers to collecting a large amount of data from different data sources into a centralized database for analysis and application. These data sources can include various data sources such as sensors, websites, social networks, mobile devices, etc. The purpose of data collection is to collect enough data for in-depth analysis and mining. There are usually many data sources for collecting data, including sensors, databases, websites, social networks, etc. It is necessary to select the collection method and tools according to the characteristics of the data source. The collection methods of big data generally include batch collection, real-time collection, incremental collection and automatic collection. Batch collection refers to the collection of a large amount of data from a website or system, and the analysis and processing of the collected data. Real-time collection refers to the real-time collection of data for real-time processing and analysis. Incremental collection refers to the regular incremental collection of existing data to obtain the latest data. Automatic collection uses automated programs to realize data collection, reduce manual intervention, and improve collection efficiency.

[0040] The relational database of the data collection module uses the database connection tools JDBC and ODBC to directly connect to the relational database. By writing SQL query statements, the required data is extracted from the database, and it can be filtered and aggregated according to specific condition fields. NoSQL database collection uses the corresponding database driver and API for data collection for non-relational databases. According to the characteristics of NoSQL databases, MongoDB uses a document-based query language, and HBase retrieves data through table scanning and filters. The web crawler collection function automatically accesses web pages and extracts useful information by writing web crawler programs. The local log system collects a large number of system logs generated by servers and applications, records the system operation status, and user behavior information. The log collection tool Logstash is used to collect regularly, and the log data is parsed, filtered, and converted to extract information. Database collection supports database collection and network collection, and a web crawler is used to deal with network data collection. The network data can be collected in real time and in large quantities. The web crawler automatically accesses web pages and extracts useful information by writing web crawler programs, traverses websites on the Internet, and captures web page content, pictures, links and other data, thereby greatly enhancing the ability to collect big data. The text file data collection module The data is collected by identifying files in CSV, XML, and JSON formats. The data acquisition system can read these files regularly and import the data into the database or analyze them. Binary file data acquisition uses binary file parsing tools to read and process the data in these binary files. The industrial sensor device module of the sensor data acquisition module connects to some sensors in industrial production through wired or wireless methods to collect sensor data in real time. The Internet of Things device module connects to smart electronic devices through Bluetooth and Wi-Fi and transmits the collected data to the acquisition system. In industrial production, various sensors are widely used to monitor equipment operation status and production process parameters. Connecting these sensors through wired or wireless methods, collecting sensor data in real time, and analyzing and processing them will help improve industrial production efficiency and quality. Through the analysis of Internet of Things device data, intelligent applications can be realized, such as smart home automation control and smart health monitoring.

[0041] The data interface management module includes the functions of identifying data sources and classifying and marking data sources. The data source identification function connects various types of data sources, such as the enterprise's internal database, external API interface, web crawler target website, and sensor equipment. The data source classification and marking function classifies and marks the received data so that each data can be accurately located. The data transmission management function includes the transmission protocol unification function and the breakpoint protection and resumption function. The breakpoint protection and resumption function can continue the transmission from the breakpoint after the connection is restored if a network interruption occurs during the data transmission process to avoid data loss. API interface calls can usually obtain more standardized and accurate data. The data analysis management module of the data management module includes data verification and analysis functions, data sorting functions, and data model establishment functions. The sorting function summarizes and organizes the collected data. The data verification and analysis function verifies the collected data to ensure the integrity and accuracy of the data, checks whether the data format is correct, whether the data range is reasonable, whether there is any missing data, and cleans and repairs the data with problems to remove invalid data, duplicate data and erroneous data. The data model building function generates models for the data. The model prediction module includes the function of generating the results of data analysis and judgment as well as the results of priority judgment. The data storage management module includes distributed storage function, data backup verification function and data recovery function. The backup data is verified regularly to ensure the integrity and availability of the backup data. The backup data can be restored to the test environment for data verification and functional testing.

[0042] The performance monitoring module of the system monitoring module includes real-time monitoring function, hardware performance evaluation function and performance testing function. The real-time monitoring function monitors the hardware performance indicators of the data acquisition system, such as CPU usage, memory usage, disk space and network bandwidth, in real time, monitors the hardware performance of the data acquisition system, controls the power of the data acquisition system, discovers system performance bottlenecks and potential problems, and takes corresponding measures to optimize and adjust. The hardware performance evaluation function tests the data acquisition, data transmission, data storage and data analysis functions to verify whether the system can accurately collect and process data. The performance testing function evaluates the performance of the data acquisition system under different loads. The data monitoring module includes quantity monitoring function, collection information recording function and data traceability tracking function. Performance testing can evaluate the performance of the data acquisition system under different loads, test the response time, throughput and other indicators of the system under high concurrent data acquisition, large data storage and processing, discover performance bottlenecks and optimize them. Through performance testing, problems that may occur in the system under high load can be discovered, and corresponding measures can be taken to optimize and improve them.

[0043] The alarm device of the system monitoring module includes an alarm information sending function and an automatic blocking function. The alarm information sending function sets the system alarm threshold. When the system performance index or data collection index exceeds the threshold, an alarm notification is automatically issued so that the administrator can take timely measures. The automatic blocking function automatically stops the system data collection work when the system exceeds the index threshold and blocks the system permissions. The security management module of the system monitoring module includes a permission management function, an access record function and a data encryption function. The permission management function manages the permissions of system access personnel to prevent unauthorized access and data leakage, and ensures the security and confidentiality of data. The data encryption function encrypts the collected data according to the permission level to ensure the confidentiality, integrity and availability of the data, encrypts and stores sensitive data, and sets access permissions. The data backup verification function formulates a data backup plan and regularly backs up the collected data to other storage media The backup data is stored on a local or remote server, and the backup data is verified regularly to ensure the integrity and availability of the backup data. The data recovery function formulates a data recovery process and plan, clarifies the recovery steps and methods, and ensures that the data is recovered in the shortest time. The real-time monitoring module monitors the performance indicators of the data acquisition system, such as CPU utilization, memory utilization, disk space, network bandwidth, etc. Through performance monitoring, system performance bottlenecks and potential problems can be discovered in a timely manner, and corresponding measures can be taken to optimize and adjust them. The distributed data storage function establishes multiple remote data storage centers for real-time multi-storage center storage to ensure that multiple data storage will not ensure data loss and the normal operation of the entire data acquisition system. The database transmission module of the hardware control management module is equipped with a data analysis module, which analyzes the data transmitted by the hardware device and transmits it to the data acquisition system. The network service module collects network data through the acquisition server.

[0044] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data collection system based on big data, characterized by: The data acquisition system includes four functional modules: a data management module, a data acquisition module, a system monitoring module and a hardware management module. The data acquisition module performs data acquisition through four acquisition modes: database acquisition, system log acquisition, file text acquisition and sensor data acquisition. The database acquisition module includes three acquisition modes: relational database acquisition, NoSQL database acquisition and web crawler acquisition. The system log acquisition module includes local log system data acquisition and centralized log system data acquisition. The file text acquisition module includes text file data acquisition and binary file data acquisition. The sensor data acquisition includes industrial sensor equipment acquisition and Internet of Things equipment acquisition. The data management module includes four functional modules: data interface management function, data transmission management function, data analysis management function and data storage management function. The system monitoring module includes a performance monitoring module, a data monitoring module, an alarm device and a security management module. The hardware management module includes a database transmission module and a network server module.

2. The data collection system and method based on big data according to claim 1, characterized in that: The relational database of the data acquisition module uses the database connection tools JDBC and ODBC to directly connect to the relational database, and extracts the required data from the database by writing SQL query statements, which can be filtered and aggregated according to specific condition fields. The NoSQL database acquisition uses the corresponding database driver and API to collect data from non-relational databases, and according to the characteristics of NoSQL databases, MongoDB uses a document-based query language, and HBase retrieves data through table scanning and filters. The web crawler collection function automatically accesses web pages and extracts useful information therein by writing web crawler programs. The local log system collects a large number of system logs generated by servers and applications, records system operation status, and user behavior information. The log acquisition tool Logstash is used to collect regularly, and the log data is parsed, filtered, and converted to extract information.

3. The data acquisition system based on big data according to claim 1, characterized in that: The text file data acquisition of the text file acquisition module recognizes data in files in CSV, XML, and JSON formats. The data acquisition system can read these files regularly and import the data into a database or perform analysis. The binary file data acquisition uses a binary file parsing tool to read and process the data in these binary files. The industrial sensor device module of the sensor data acquisition module connects to some sensors in industrial production through wired or wireless means to collect sensor data in real time. The Internet of Things device module connects to smart electronic devices through Bluetooth and Wi-Fi connections and transmits the collected data to the acquisition system.

4. The data acquisition system based on big data according to claim 1, characterized in that: The data interface management module includes a data source identification function and a data source classification and marking function. The data source identification function connects various types of data sources, such as internal enterprise databases, external API interfaces, web crawler target websites, and sensor devices. The data source classification and marking function classifies and marks the received data so that each data can be accurately located. The data transmission management function includes a transmission protocol unification function and a breakpoint protection and resumption function. If a network interruption occurs during data transmission, the breakpoint protection and resumption function can continue transmission from the breakpoint after the connection is restored to avoid data loss.

5. The data acquisition system based on big data according to claim 1, characterized in that: The data analysis management module of the data management module includes data verification and analysis functions, data sorting functions and data model building functions. The data sorting function summarizes and organizes the collected data. The data verification and analysis function verifies the collected data to ensure the integrity and accuracy of the data, checks whether the data format is correct, whether the data range is reasonable, whether the data is missing, and cleans and repairs the data with problems to remove invalid data, duplicate data and erroneous data. The data model building function generates a model for the data. The model prediction module includes the function of generating the results of data analysis and processing and the results of priority judgment. The data storage management module includes a distributed storage function, a data backup verification function and a data recovery function.

6. The data acquisition system based on big data according to claim 1, characterized in that: The performance monitoring module of the system monitoring module includes real-time monitoring function, hardware performance evaluation function and performance testing function. The real-time monitoring function monitors the hardware performance indicators of the data acquisition system, such as CPU utilization, memory utilization, disk space and network bandwidth, in real time, monitors the hardware performance of the data acquisition system, controls the power of the data acquisition system, discovers system performance bottlenecks and potential problems, and takes corresponding measures to optimize and adjust. The hardware performance evaluation function tests the data acquisition, data transmission, data storage and data analysis functions to verify whether the system can accurately collect and process data. The performance testing function evaluates the performance of the data acquisition system under different loads. The data monitoring module includes quantity monitoring function, collection information recording function and data traceability tracking function.

7. The data acquisition system based on big data according to claim 1, characterized in that: The alarm device of the system monitoring module includes an alarm information sending function and an automatic blocking function. The alarm information sending function sets the system alarm threshold. When the system performance index or data collection index exceeds the threshold, an alarm notification is automatically issued so that the administrator can take measures in time. The automatic blocking function automatically stops the system data collection work when the system exceeds the index threshold and blocks the system permissions. The security management module of the system monitoring module includes a permission management function, an access record function and a data encryption function. The permission management function manages the permissions of system access personnel to prevent unauthorized access and data leakage, and ensures the security and confidentiality of data. The data encryption function encrypts the collected data according to the permission level to ensure the confidentiality, integrity and availability of the data, encrypts and stores sensitive data, and sets access permissions. The data backup verification function formulates a data backup plan, regularly backs up the collected data to other storage media or remote servers, and regularly verifies the backup data to ensure the integrity and availability of the backup data. The data recovery function formulates a data recovery process and plan, clarifies the recovery steps and methods, and ensures that the data is recovered in the shortest time.

8. The data acquisition system based on big data according to claim 1, characterized in that: The database transmission module of the hardware control management module is provided with a data analysis module, and the data analysis module analyzes the data transmitted by the hardware device and transmits it to the data acquisition system. The network service module collects network data through the acquisition server.

9. The data acquisition system based on big data according to claim 1, characterized in that: The collection method steps of the data collection system based on big data are as follows: S1: Clarify the objectives of data collection and determine the specific objectives of data collection, including the type of data to be collected and the time span; S2: Determine the data source and data collection method, and select internal data source collection or external data source collection; S3: Select appropriate collection methods according to the type and characteristics of data sources. For network big data, use web crawlers to collect massive amounts of data. For database data, use database connection tools to collect data. For device data, use sensors and IoT devices to collect data, and determine the data collection plan. For data, determine the collection time and frequency. S4: Select appropriate hardware and software equipment according to the data collection plan, build the data collection environment, configure the parameters of the collection environment, and determine the database connection string, web crawler crawling rules, and data collection card sampling frequency required in the data collection process; S5: Establish a data collection and storage warehouse, select the appropriate storage method for the collected data according to the characteristics and needs of the data, and establish a relational database, NoSQL database, and distributed file system. The data warehouse integrates and stores data from different data sources to provide a unified data view for data analysis; S6: Establish a data storage security management mechanism, encrypt and store the collected important data, set access permissions, improve security protection, add version control, backup and recovery to the collected data, and ensure the safe storage of data.

Images