Multi-platform data acquisition method

Abstract

The invention relates to the technical field of data acquisition, in particular to a multi-platform data acquisition method, which comprises the following steps of: detecting hardware and software environments of acquisition equipment; when it is detected that the acquisition devices run in different operating systems and network conditions, data acquisition scripts related to the acquisition devices are generated according to detection results; based on the data acquisition script, starting a protocol plug-in corresponding to each acquisition device, and establishing a communication link with each acquisition device; after the communication link is established, a data acquisition task is started, and the acquired data is subjected to standardization processing and converted into a unified format; the method comprises the following steps: performing primary processing, including filtering and aggregation, on converted data on edge equipment, and transmitting the processed data to a central cloud system for unification through an encryption transmission protocol, thereby being capable of adapting to the universality and compatibility of a data acquisition mode in a multi-platform and multi-equipment environment; and the accuracy and efficiency of data acquisition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a multi-platform data acquisition method. Background Art

[0002] In the context of the rapid development of informatization and intelligence, data has become the core element to promote the progress of various industries. As the first step in data processing, the accuracy and efficiency of data collection directly affect the results of subsequent data analysis, mining and application. With the popularization of the Internet of Things (IoT) and the widespread application of various devices, data collection technology is facing unprecedented challenges.

[0003] Existing data acquisition systems are mostly designed for specific platforms and devices. In this case, the compatibility and versatility between systems are relatively poor. In a multi-platform and multi-device environment, data acquisition involves different protocols, formats, and transmission methods, which requires complex adaptation and customized development of data acquisition systems. This not only increases the difficulty and cost of development, but also greatly prolongs the development cycle. In addition, during the acquisition process, the synchronization and consistency of data are difficult to guarantee, which further affects the accuracy of the data.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a multi-platform data collection method, which aims to solve the technical problems that the data collection methods in multi-platform and multi-device environments lack universality and compatibility, resulting in low collection efficiency, insufficient accuracy and long development cycle, and difficulty in meeting the needs of different devices and scenarios.

[0006] To achieve the above object, the present invention provides a multi-platform data collection method, the method comprising:

[0007] Detect the hardware and software environment of each acquisition device;

[0008] When it is detected that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result;

[0009] Based on the data collection script, the protocol plug-in corresponding to each of the collection devices is started, and a communication link is established with each of the collection devices;

[0010] After the communication link is established, the data collection task is started, and the collected data is standardized and converted into a unified format;

[0011] The converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol.

[0012] Optionally, starting a protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices includes:

[0013] Parse the data acquisition script, and load the required protocol plug-in from a predefined location according to the parsing result, wherein the predefined location is a local file system or a remote warehouse, and the protocol plug-in is a file stored in a dynamic library or a Python module;

[0014] A loaded protocol plug-in is configured for each of the acquisition devices according to the parameters in the data acquisition script, and a communication link is established with each of the acquisition devices based on the loaded protocol plug-in. The parameters in the data acquisition script include the IP address, port number, authentication information and data format of the target device.

[0015] Optionally, configuring the loaded protocol plug-in for each acquisition device according to the parameters in the data acquisition script includes:

[0016] Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information;

[0017] Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class;

[0018] Each device information in the script is traversed, and the corresponding plug-in module and class are found from the mapping relationship according to the protocol type of the device, and the protocol plug-in loaded for each acquisition device configuration is configured using a dynamic loading mechanism.

[0019] Optionally, starting the data collection task after establishing the communication link, and standardizing the collected data to convert it into a unified format, includes:

[0020] According to the data collection tasks defined in the script, configure each device, including data source, collection frequency and data format, and call the method in the plug-in class to start the data collection task;

[0021] Perform preliminary processing on the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Data errors found during the preprocessing process are corrected or discarded;

[0022] Map the preprocessed data to a standard format and convert the mapped data to a standard format.

[0023] Optionally, mapping the preprocessed data to a standard format and converting the mapped data to the standard format includes:

[0024] Clarify standard data formats, including timestamps, device IDs, data types, and data values;

[0025] The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into a target data structure.

[0026] Optionally, the converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol, including:

[0027] Performing preliminary processing on the converted data on the edge device, including filtering and aggregation, wherein the filtering includes filtering and selecting useful data according to preset data types and device types, data value ranges and time ranges, and the aggregation includes aggregating data within a certain time window and aggregating different data types;

[0028] The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system via the HTTPS encrypted transmission protocol for unified processing.

[0029] Optionally, the detecting of the hardware and software environment of each acquisition device includes:

[0030] The hardware and software environment of each acquisition device is detected through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information and network device information; the software environment includes operating system version, application information, network configuration and power management settings.

[0031] Optionally, when it is detected that each of the acquisition devices is respectively running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result includes:

[0032] Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device;

[0033] When the detection code based on the operating system and network condition of the acquisition device detects that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result.

[0034] Optionally, the edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is directly processed on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

[0035] Optionally, after the complex data with large processing volume is sent to the central cloud system through the transmission mechanism, the method further includes:

[0036] Determine the priority of complex and large-volume data, where the priority is a mark that distinguishes the importance and processing order of different tasks or data in the processing system;

[0037] The complex data with large processing volume are classified and stored in different queues based on the priority level, and different threads or processes are started to process the corresponding data queues according to the priority level.

[0038] The present invention provides a multi-platform data collection method, the method comprising: detecting the hardware and software environment of each collection device; when detecting that each of the collection devices is respectively running under different operating systems and network conditions, generating data collection scripts related to each of the collection devices according to the detection results; starting a protocol plug-in corresponding to each of the collection devices based on the data collection script, and establishing a communication link with each of the collection devices; starting a data collection task after the communication link is established, standardizing the collected data and converting it into a unified format; performing preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmitting the processed data to a central cloud system for unification through an encrypted transmission protocol, thereby being able to adapt to the universality and compatibility of data collection methods in a multi-platform and multi-device environment, and improving the accuracy and efficiency of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of the first embodiment of the multi-platform data collection method of the present invention;

[0040] Figure 2 It is a flowchart of the second embodiment of the multi-platform data collection method of the present invention;

[0041] Figure 3 It is a flowchart of the third embodiment of the multi-platform data collection method of the present invention.

[0042] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] Reference Figure 1 , Figure 1 The flowchart of the first embodiment of the multi-platform data collection method of the present invention is shown in FIG.

[0044] In a first embodiment, the multi-platform data collection method comprises the following steps:

[0045] S10: Detect the hardware and software environment of each acquisition device;

[0046] S20: When it is detected that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result;

[0047] S30: starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script, and establishing a communication link with each of the acquisition devices;

[0048] S40: after the communication link is established, the data collection task is started, and the collected data is standardized and converted into a unified format;

[0049] S50: Perform preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmit the processed data to the central cloud system through an encrypted transmission protocol for unified processing.

[0050] It should be noted that, first of all, the system automatically detects the hardware and software environment of each data collection device. This is similar to a person first understanding the basic information of each device, such as whether the operating system is Windows, Linux or other, whether the device is connected to Wi-Fi or Ethernet, etc. According to the different hardware and software environments detected, the system automatically generates corresponding data collection scripts. It is like tailoring "operation guides" for different devices to ensure that each device can correctly collect data. Using the generated data collection script, the system will start the communication protocol plug-in for each device. This is equivalent to finding a suitable "language translator" for each device, and then establishing a communication link to ensure that the device can smoothly communicate with the system. After establishing communication, the system starts the actual data collection task and standardizes the collected data and converts it into a unified format. Just like the currencies of different countries are converted into the same currency, which is convenient for subsequent processing and analysis. On the edge device, the standardized data is preliminarily processed (such as filtering and aggregation). This is similar to the preliminary processing of raw materials, removing redundant parts and integrating useful information, and then securely transmitting it to the central cloud system using an encrypted transmission protocol (similar to an encrypted envelope). Ultimately, the preliminarily processed data is transmitted to the central cloud system for more in-depth unified processing and analysis. The cloud system will perform more complex data analysis and applications, ultimately providing users with valuable information and decision support.

[0051] For example, in a smart city management system, there are various types of sensor devices, such as traffic signal sensors, pollution monitoring sensors, and smart street lights. Each device may use a different operating system (such as embedded systems, Linux, etc.) and network conditions (such as WIFI, cellular networks, etc.). Through the above method, the system will automatically identify the operating environment of each device, generate a specific collection script, start data collection after establishing a communication link, and standardize all data and transmit it to the edge device for preliminary processing. Finally, the processed data is securely transmitted to the cloud for real-time monitoring and decision-making by city managers.

[0052] It should be understood that this embodiment can improve the collection efficiency: by automatically generating and adapting scripts, the manual configuration time is reduced, and the data collection efficiency is significantly improved. Ensure data accuracy: Standardized processing and preliminary cleaning ensure data consistency and accuracy. Shorten the development cycle: Provide a general processing framework, reduce development time, and quickly respond to the needs of different devices. Enhance data security: Through encrypted transmission protocols, ensure the security of data during transmission. Adapt to multi-platform and multi-device needs: The flexibility of this method enables it to work stably under various operating systems and network conditions, meeting complex and diverse scenario requirements.

[0053] Reference Figure 2 , Figure 2 The flowchart of the second embodiment of the multi-platform data collection method of the present invention is as follows. In the second embodiment, the protocol plug-in corresponding to each of the collection devices is started based on the data collection script, and a communication link is established with each of the collection devices, including:

[0054] S301: Parse the data acquisition script, and load the required protocol plug-in from a predefined location according to the parsing result, where the predefined location is a local file system or a remote warehouse, and the protocol plug-in is a file stored in a dynamic library or a Python module;

[0055] S302: configuring a loaded protocol plug-in for each of the acquisition devices according to the parameters in the data acquisition script, and establishing a communication link with each of the acquisition devices based on the loaded protocol plug-in, wherein the parameters in the data acquisition script include the IP address, port number, authentication information and data format of the target device.

[0056] It should be noted that the data acquisition script is like a detailed set of instructions. The system first parses the script to understand the content and instructions. This process is similar to reading an operating manual to understand what tools and steps are needed. Based on the parsing results, the system determines which protocol plug-ins need to be loaded. These plug-ins can be stored in the local file system (such as files on the computer hard disk) or in a remote repository (such as an online resource library). These plug-ins can be dynamic library files (like DLL files) or files in Python modules (Python packages). The system configures the loaded protocol plug-ins for each acquisition device based on the parameters in the data acquisition script. These parameters include the IP address, port number, authentication information (such as username and password), and data format of the target device. This is equivalent to entering the necessary information according to the instructions in the manual and configuring the tool so that it can connect and work correctly with the target device. Using correctly configured plug-ins, the system establishes a communication link with each acquisition device to ensure that data can be transmitted smoothly, just like dialing the other party's phone number and starting a clear call.

[0057] For example, suppose there is a smart agricultural monitoring system with soil moisture sensors and temperature sensors. These sensors run different operating systems and are connected through different networks (one via Wi-Fi and the other via Ethernet cable). Here is how the method works: the system parses the script and finds that the soil moisture sensor requires a specific communication protocol plug-in, and the temperature sensor requires another plug-in.

[0058] According to the script instructions, the soil moisture sensor plug-in is loaded from the local file system, and the temperature sensor plug-in is loaded from a remote repository (for example, an online resource server). The script parameters include the soil moisture sensor's IP address, port number, authentication information (such as username and password), and data format (such as JSON). Similarly, the specific parameters of the temperature sensor are also listed in the script. The system configures the respective plug-ins based on these parameters. Through the configured plug-ins, the system establishes communication links with the soil moisture sensor and the temperature sensor and starts collecting the corresponding data.

[0059] It should be understood that the beneficial effects of this embodiment are as follows: Flexibility and compatibility: The system can flexibly load the required plug-ins locally or remotely, adapt to various devices and network environments, and ensure compatibility. Dynamic configuration: By parsing scripts and parameter configurations, the system can dynamically adapt to the needs of different devices without manual intervention, thereby improving the degree of automation. Shortened deployment time: The automated loading and configuration process reduces manual operation time and speeds up the deployment process. Reliability and accuracy: According to the detailed parameters in the script, the communication link of each device is ensured to be accurate and correct, thereby improving the reliability and accuracy of data collection. Easy to maintain and expand: Using scripts to manage and configure plug-ins, the system is easier to maintain and expand, and can adapt to new equipment and protocol requirements.

[0060] Further, in this embodiment, configuring the loaded protocol plug-in for each of the acquisition devices according to the parameters in the data acquisition script includes:

[0061] Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information;

[0062] Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class;

[0063] Each device information in the script is traversed, and the corresponding plug-in module and class are found from the mapping relationship according to the protocol type of the device, and the protocol plug-in loaded for each acquisition device configuration is configured using a dynamic loading mechanism.

[0064] It should be noted that the script read by the system contains information about multiple devices, including the unique ID of the device, the type of protocol to be used, and its configuration information. Parse this information into an easy-to-process data format, such as a dictionary or list. Establish a mapping relationship between each protocol and the corresponding protocol plug-in, and use a dictionary (or similar data structure) to store the relationship between the protocol name and the corresponding plug-in module and class. This dictionary is like a guide, telling the system which plug-in module and class to use for each protocol. The system traverses each device information in the script, finds the corresponding plug-in module and class from the mapping relationship according to the protocol type of the device, and loads and configures it through the dynamic loading mechanism. The dynamic loading mechanism means loading the corresponding module as needed at runtime, rather than loading all possible modules when the program starts.

[0065] It should be understood that the beneficial effects of this embodiment are as follows: Automated processing: Such a method enables the system to automatically parse and process various device configuration information, reducing manual configuration errors. Flexibility: Through the dynamic loading mechanism, the system can load specific protocol plug-ins as needed at runtime, improving flexibility and resource utilization efficiency. Scalability: When adding new device types or protocols, only the scripts and mapping relationships need to be updated without major changes to the core logic of the system. Easy management: Using dictionaries to manage the relationship between protocols and plug-ins makes maintenance and expansion more intuitive and simple. Precise configuration: Accurate configuration is performed according to the unique parameters of each device to ensure the accuracy and reliability of the communication process.

[0066] Reference Figure 3 , Figure 3 The flowchart of the third embodiment of the multi-platform data collection method of the present invention is as follows. In the third embodiment, the data collection task is started after the communication link is established, and the collected data is standardized and converted into a unified format, including:

[0067] S401: According to the data collection task defined in the script, each device is configured, including the data source, collection frequency and data format, and the method in the plug-in class is called to start the data collection task;

[0068] S402: Preliminary processing of the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Data errors found during the preprocessing process are corrected or discarded;

[0069] S403: Map the preprocessed data to a standard format, and convert the mapped data into the standard format.

[0070] It should be noted that each device is configured according to the data collection task defined in the script. The configuration content includes the data source (IP address and port of the device, etc.), the collection frequency (such as once a minute), and the data format (such as JSON or XML). Then, the method in the plug-in class is called to start the data collection task. The collected raw data may contain a lot of unnecessary information or noise data, so it needs to be processed initially. This includes parsing strings, extracting required fields (such as temperature values, humidity values, etc.), removing unnecessary noise data, and verifying the integrity and correctness of the data. If data errors are found, they are corrected or discarded. The preprocessed data needs to be mapped to a standard format, which usually means converting the data into a unified format to facilitate subsequent storage, processing, and analysis. For example, standardize the data of different sensors into a unified JSON format.

[0071] It should be understood that the beneficial effects of this embodiment are as follows: Efficient data management: Through automated scripts and configurations, data collection tasks can be efficiently managed and scheduled, reducing manual operations and configuration errors. Improved data quality: The preliminary processing steps can effectively remove noise data, correct erroneous data, improve data quality, and ensure data accuracy and reliability. Unified data format: By mapping the data to a unified standard format, data consistency is ensured, thereby facilitating subsequent storage, processing, and analysis. Real-time data processing: The dynamic configuration and loading mechanism enables the system to process various data sources and collection tasks in real time, improving the system's immediate response capabilities and processing efficiency. Scalability: The standardized and modular design makes the system easy to expand and maintain, and new device types and collection tasks can be easily added.

[0072] Furthermore, in this embodiment, mapping the preprocessed data to a standard format and converting the mapped data to a standard format include:

[0073] Clarify standard data formats, including timestamps, device IDs, data types, and data values;

[0074] The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into a target data structure.

[0075] It should be noted that the standard data format is to ensure that data from different devices can be represented and processed in a unified way. The standard format includes the following main fields: Timestamp: records the specific time of data collection, usually expressed in ISO 8601 format, such as "2024-06-06T06:00:00Z". Device ID (device_id): a unique identifier for each device, used to distinguish data from different sources. Data type (data_type): specifies the category of data, such as temperature, humidity, pressure, etc. Data value (data_value): a specific measurement value, such as the temperature value of 20.0 degrees and the humidity value of 45% at a certain moment. The raw data collected from the device may contain redundant information or noise, and needs to be preprocessed first. Preprocessing includes parsing the data, extracting the required fields (such as timestamps, temperature values), removing unnecessary information, and ensuring the integrity and correctness of the data. Use the preset mapping function to convert the preprocessed data into the standard format. The mapping function is responsible for receiving the processed raw data and generating structured data in a standard format.

[0076] Furthermore, in this embodiment, the converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing through an encrypted transmission protocol, including:

[0077] Performing preliminary processing on the converted data on the edge device, including filtering and aggregation, wherein the filtering includes filtering and selecting useful data according to preset data types and device types, data value ranges and time ranges, and the aggregation includes aggregating data within a certain time window and aggregating different data types;

[0078] The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system via the HTTPS encrypted transmission protocol for unified processing.

[0079] It should be noted that according to the data type and device type, only specific types of data are retained, such as only focusing on temperature data or data from a specific model of sensor. According to the data value range, a threshold is set to retain only data that meets a specific range, such as only retaining data with temperature values ​​between 20℃ and 30℃. According to the time range, a time window is set to process only data within a specific time period. Aggregation within a time window refers to aggregating multiple data points into one data point within a certain time interval (such as every minute or every hour), such as calculating the average, maximum, minimum, etc. Aggregation of different data types means that if a device generates multiple types of data (such as temperature and humidity), these data can be merged into one data packet. Symmetric encryption refers to the use of the same key for encryption and decryption, such as AES (Advanced Encryption Standard). Asymmetric encryption refers to the use of a pair of keys (public and private keys), one for encryption and the other for decryption, such as the RSA algorithm. Use HTTPS (Secure Hypertext Transfer Protocol) to ensure the security of data during transmission. HTTPS provides encryption through the TLS (Transport Layer Security) protocol to prevent data from being eavesdropped or tampered with during transmission. Data decryption refers to the decryption of data using a key or private key shared with the edge device. Unified processing refers to further processing and analysis of the decrypted data, which may include long-term storage, in-depth analysis, report generation, or triggering of specific events. The beneficial effect is that only necessary and valuable data is transmitted through filtering and aggregation, which greatly reduces the amount of data transmitted and improves network efficiency and transmission speed. After preliminary processing, the edge device can respond quickly to local events, such as triggering alarms or control operations based on real-time monitoring data without having to wait for cloud processing. The pre-processed data has been optimized and compressed, the amount of data transmitted to the cloud is reduced, and the resource consumption of cloud storage and processing is reduced. By performing encryption processing on the edge device, the security and privacy of data during transmission can be effectively guaranteed, and the risk of data leakage can be reduced.

[0080] Furthermore, in this embodiment, the step of detecting the hardware and software environment of each acquisition device includes:

[0081] The hardware and software environment of each acquisition device is detected through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information and network device information; the software environment includes operating system version, application information, network configuration and power management settings.

[0082] It should be noted that in order to effectively manage and maintain the acquisition devices distributed in various places, it is necessary to detect and monitor the hardware and software environment information of these devices through system calls and API interfaces. Detect the hardware environment and device model: by querying the underlying hardware information of the system or a specific API interface. On Linux systems, you can use the cat / proc / device-tree / model or lshw command. On Windows systems, you can use the wmic csproduct get name command. Processor type and frequency: On Linux, you can view the / proc / cpuinfo file or use the lscpu command. On Windows, you can use the wmic cpu get name,MaxClockSpeed ​​command. Memory information: On Linux, you can use free-m or cat / proc / meminfo commands. On Windows, you can use systeminfo|find "TotalPhysical Memory" or wmic memorychip get capacity. Storage information: On Linux, you can use the df-h command to view storage usage. On Windows, you can use the wmic logicaldisk get size,freespace,caption command. Network device information: On Linux, you can use the ifconfig or ip a command. On Windows, you can use the ipconfig / all command. Check the software environment and operating system version: On Linux, you can view the / etc / os-release file or use the uname-a command. On Windows, you can use the systeminfo|find "OS" command. Application information: On Linux, you can use package manager commands such as dpkg-l (Debian) or rpm-qa (RedHat). On Windows, you can use the wmic product get name,version command. Network configuration: On Linux, you can view network configuration files (such as / etc / network / interfaces or / etc / netplan / *.yaml) or use the nmcli command. On Windows, you can use netsh interface ipv4 show config to view network configuration. Power management settings: On Linux, you can use the upower command or view power management related files (such as / sys / class / power_supply / ). On Windows, you can use the powercfg / query command to view the power configuration.Use API interfaces to obtain information. For some advanced devices or embedded systems, manufacturers may provide API interfaces. By calling these interfaces, you can more conveniently obtain device information. For example, common interfaces include: SNMP (Simple Network Management Protocol): It can be used to obtain information about network devices and query through snmpget or snmpwalk tools. RESTful API: Some modern devices may provide RESTful API interfaces to obtain device information in JSON format through HTTP requests. Vendor-specific SDKs: Software development kits (SDKs) provided by manufacturers can realize device management and information acquisition.

[0083] Further, in this embodiment, when it is detected that each of the acquisition devices is respectively running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result, including:

[0084] Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device;

[0085] When the detection code based on the operating system and network condition of the acquisition device detects that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result.

[0086] It should be noted that in order to detect the operating system and network conditions and generate the corresponding data collection script, it is necessary to use Python's platform module and psutil module to obtain system information. Then, according to different operating systems and network conditions, corresponding data collection scripts are generated.

[0087] Furthermore, in this embodiment, the edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is directly processed on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

[0088] It should be noted that in a distributed system, classifying and processing the data of edge devices is the key to ensuring system efficiency and response speed. By processing data with high real-time requirements directly on the edge devices and sending data with complex processing volume to the central cloud system for processing, the performance and reliability of the overall system can be improved. The following is a specific implementation plan, including data classification, processing and transmission mechanisms. Data classification system, data with high real-time requirements: usually contains sensor data that requires a fast response, such as temperature, pressure, speed, etc. These data require low-latency processing so that immediate action or response can be taken. Real-time data processing is particularly important in the fields of industrial automation, smart transportation, smart home, etc. Complex processing volume data: contains large amounts of data that require complex calculations, analysis or historical data comparison, such as video stream analysis, machine learning reasoning, deep analysis, etc. These data do not require immediate processing and can be batch processed in the central cloud system. Implementation plan, 1. Data classification logic, classify the data collected on the edge device to determine which data needs to be processed locally and which needs to be sent to the cloud: Define classification rules: Real-time data: Set specific data types or data from specific sensors as data with high real-time requirements. Complex data: Set complex data as data that requires in-depth analysis or big data processing. Data identification and storage: Real-time data: Tag identification, and directly sent to the local processing queue. Complex data: Tag identification, cached locally for batch sending to the cloud. 2. Real-time data processing, for data with high real-time requirements, directly process on the edge device: Local processing: Data filtering, transformation and processing: such as temperature over-limit alarm, environmental control, etc. Store the results in the local database or cache for subsequent use by the local system. Event-driven mechanism: Configure the event-driven mechanism to trigger responses to specific conditions in real time, such as turning on the cooling fan. 3. Complex data transmission and processing, for complex data with large processing volume, send it to the central cloud system for processing through the transmission mechanism: Data caching and batch transmission: Cache mechanism: Temporarily store complex data in the local cache or database. Scheduled or triggered batch data transmission: Use scheduled tasks or threshold trigger mechanisms to batch transmit cached data to the cloud. Transmission mechanism: Use HTTPS protocol encrypted by SSL / TLS to ensure the security of data transmission. Use reliable message queue systems (such as RabbitMQ, Kafka) to ensure the reliability and efficiency of data transmission. Cloud processing: After the data is received in the central cloud system, it is decrypted and stored. Big data processing tools (such as Hadoop, Spark) and machine learning platforms are used to analyze and process complex data. Reports, prediction results or commands are generated and returned to the edge device as needed. By effectively combining the advantages of edge computing and cloud computing, the performance, response speed and processing power of the system can be significantly improved.Edge devices improve the efficiency of local response by processing high-real-time data in real time; at the same time, by transmitting complex data in batches to the cloud for in-depth analysis, they make full use of the powerful computing resources of the cloud and achieve efficient and comprehensive data processing. This solution is of great significance to the Internet of Things system, big data processing and smart device management.

[0089] Furthermore, in this embodiment, after the complex data with large processing volume is sent to the central cloud system through the transmission mechanism, the method further includes:

[0090] Determine the priority of complex and large-volume data, where the priority is a mark that distinguishes the importance and processing order of different tasks or data in the processing system;

[0091] The complex data with large processing volume are classified and stored in different queues based on the priority level, and different threads or processes are started to process the corresponding data queues according to the priority level.

[0092] It should be noted that to achieve priority management of complex data processing, complex data with large processing volume can be classified and stored in different queues based on priority, and different threads or processes can be started to process the corresponding data queues according to priority. By combining the queue and threading modules of Python, it is possible to store complex data in different priority queues according to priority, and process each queue separately through independent threads. This method can effectively improve the efficiency and response speed of the system in processing complex data.

[0093] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0094] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as names.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal user device (which can be a mobile phone, a computer, a server, an air conditioner, or a network user device, etc.) to execute the methods described in each embodiment of the present invention.

[0096] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-platform data collection method, characterized in that: The method comprises: Detect the hardware and software environment of each acquisition device; When it is detected that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result; Based on the data collection script, the protocol plug-in corresponding to each of the collection devices is started, and a communication link is established with each of the collection devices; After the communication link is established, the data collection task is started, and the collected data is standardized and converted into a unified format; The converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol.

2. The multi-platform data collection method according to claim 1, characterized in that: The method of starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices includes: Parse the data acquisition script, and load the required protocol plug-in from a predefined location according to the parsing result, wherein the predefined location is a local file system or a remote warehouse, and the protocol plug-in is a file stored in a dynamic library or a Python module; A loaded protocol plug-in is configured for each of the acquisition devices according to the parameters in the data acquisition script, and a communication link is established with each of the acquisition devices based on the loaded protocol plug-in. The parameters in the data acquisition script include the IP address, port number, authentication information and data format of the target device.

3. The multi-platform data collection method according to claim 2, characterized in that: The step of configuring the loaded protocol plug-in for each acquisition device according to the parameters in the data acquisition script includes: Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information; Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class; Each device information in the script is traversed, and the corresponding plug-in module and class are found from the mapping relationship according to the protocol type of the device, and the protocol plug-in loaded for each acquisition device configuration is configured using a dynamic loading mechanism.

4. The multi-platform data collection method according to claim 1, characterized in that: After the communication link is established, the data collection task is started, and the collected data is standardized and converted into a unified format, including: According to the data collection tasks defined in the script, configure each device, including data source, collection frequency and data format, and call the method in the plug-in class to start the data collection task; Perform preliminary processing on the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Data errors found during the preprocessing process are corrected or discarded; Map the preprocessed data to a standard format and convert the mapped data to a standard format.

5. The multi-platform data collection method according to claim 4, characterized in that: Mapping the preprocessed data to a standard format and converting the mapped data to a standard format include: Clarify standard data formats, including timestamps, device IDs, data types, and data values; The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into a target data structure.

6. The multi-platform data collection method according to claim 1, characterized in that: The converted data is initially processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing through an encrypted transmission protocol, including: Performing preliminary processing on the converted data on the edge device, including filtering and aggregation, wherein the filtering includes filtering and selecting useful data according to preset data types and device types, data value ranges and time ranges, and the aggregation includes aggregating data within a certain time window and aggregating different data types; The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system via the HTTPS encrypted transmission protocol for unified processing.

7. The multi-platform data collection method according to claim 1, characterized in that: The detection of the hardware and software environment of each acquisition device includes: The hardware and software environment of each acquisition device is detected through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information and network device information; the software environment includes operating system version, application information, network configuration and power management settings.

8. The multi-platform data collection method according to claim 1, characterized in that: When it is detected that each of the acquisition devices is respectively running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result, including: Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device; When the detection code based on the operating system and network condition of the acquisition device detects that each of the acquisition devices is running under different operating systems and network conditions, a data acquisition script related to each of the acquisition devices is generated according to the detection result.

9. The multi-platform data collection method according to claim 1, characterized in that: The edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is directly processed on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

10. The multi-platform data collection method according to claim 9, characterized in that: After the complex data with large processing volume is sent to the central cloud system through the transmission mechanism, it also includes: Determine the priority of complex and large-volume data, where the priority is a mark that distinguishes the importance and processing order of different tasks or data in the processing system; The complex data with large processing volume are classified and stored in different queues based on the priority level, and different threads or processes are started to process the corresponding data queues according to the priority level.

Images