Multi-platform data collection method

Abstract

The present invention relates to the field of data acquisition technology, and in particular to a multi-platform data acquisition method, the method comprising: detecting the hardware and software environment of each acquisition device; when detecting that each of the acquisition devices is respectively running on different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result; starting a protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script, and establishing a communication link with each of the acquisition devices; starting a data acquisition task after the communication link is established, standardizing the collected data and converting it into a unified format; performing preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmitting the processed data to a central cloud system for unification through an encrypted transmission protocol, thereby being able to adapt to the versatility and compatibility of data acquisition methods in a multi-platform and multi-device environment, and improving the accuracy and efficiency of data acquisition.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a multi-platform data acquisition method. Background Art

[0002] Amid the rapid development of informatization and intelligent technology, data has become a core driver of progress across various industries. Data collection, the first step in data processing, directly impacts subsequent data analysis, mining, and application effectiveness through its accuracy and efficiency. With the increasing popularity of the Internet of Things (IoT) and the widespread use of diverse devices, data collection technology is facing unprecedented challenges.

[0003] Existing data acquisition systems are often designed for specific platforms and devices, resulting in relatively poor interoperability and interoperability across systems. In a multi-platform, multi-device environment, data acquisition involves varying protocols, formats, and transmission methods, requiring complex adaptation and custom development of data acquisition systems. This not only increases development difficulty and costs, but also significantly prolongs the development cycle. Furthermore, data synchronization and consistency are difficult to ensure during the acquisition process, further impacting data accuracy.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a multi-platform data collection method, which aims to solve the technical problems that the data collection methods in multi-platform and multi-device environments lack universality and compatibility, resulting in low collection efficiency, insufficient accuracy and long development cycle, and difficulty in meeting the needs of different devices and scenarios.

[0006] To achieve the above object, the present invention provides a multi-platform data acquisition method, the method comprising:

[0007] Detect the hardware and software environment of each acquisition device;

[0008] When detecting that each of the acquisition devices is running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result;

[0009] Starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices;

[0010] After establishing the communication link, the data collection task is started, and the collected data is standardized and converted into a unified format;

[0011] The converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol.

[0012] Optionally, starting a protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices includes:

[0013] Parse the data acquisition script and load the required protocol plug-in from a predefined location based on the parsing result. The predefined location is a local file system or a remote repository. The protocol plug-in is a file stored in a dynamic library or a Python module.

[0014] A loaded protocol plug-in is configured for each of the acquisition devices according to the parameters in the data acquisition script, and a communication link is established with each of the acquisition devices based on the loaded protocol plug-in. The parameters in the data acquisition script include the IP address, port number, authentication information and data format of the target device.

[0015] Optionally, configuring a loaded protocol plug-in for each acquisition device according to parameters in the data acquisition script includes:

[0016] Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information;

[0017] Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class;

[0018] Traverse each device information in the script, find the corresponding plug-in module and class from the mapping relationship according to the protocol type of the device, and use the dynamic loading mechanism to configure the loaded protocol plug-in for each of the acquisition devices.

[0019] Optionally, after establishing the communication link, starting the data collection task and standardizing the collected data to convert it into a unified format may include:

[0020] According to the data collection task defined in the script, each device is configured, including the data source, collection frequency, and data format, and the method in the plug-in class is called to start the data collection task;

[0021] Perform preliminary processing on the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Any data errors found during the preprocessing process will be corrected or discarded.

[0022] Map the preprocessed data to a standard format and convert the mapped data to a standard format.

[0023] Optionally, mapping the preprocessed data to a standard format, converting the mapped data to the standard format, includes:

[0024] Clarify standard data formats, including timestamps, device IDs, data types, and data values;

[0025] The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into the target data structure.

[0026] Optionally, performing preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmitting the processed data to the central cloud system for unified processing via an encrypted transmission protocol, including:

[0027] Perform preliminary processing on the converted data on the edge device, including filtering and aggregation. The filtering includes filtering and selecting useful data based on preset data types and device types, data value ranges, and time ranges. The aggregation includes aggregating data within a certain time window and aggregating data of different data types.

[0028] The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system for unified processing via the HTTPS encrypted transmission protocol.

[0029] Optionally, detecting the hardware and software environment of each acquisition device includes:

[0030] The hardware and software environment of each acquisition device is detected through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information and network device information; the software environment includes operating system version, application information, network configuration and power management settings.

[0031] Optionally, when detecting that each of the acquisition devices is respectively running on a different operating system and network condition, generating a data acquisition script related to each of the acquisition devices according to the detection result includes:

[0032] Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device;

[0033] When the detection code based on the operating system and network condition of the acquisition device detects that each acquisition device is running under different operating systems and network conditions, a data acquisition script related to each acquisition device is generated according to the detection result.

[0034] Optionally, the edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is processed directly on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

[0035] Optionally, after the complex data with large processing volume is sent to the central cloud system via a transmission mechanism, the method further includes:

[0036] Determine the priority of complex and large-volume data. Priority refers to an identifier that distinguishes the importance and processing order of different tasks or data in the processing system;

[0037] The data with large processing volume and complex structure are classified and stored in different queues based on the priority level, and different threads or processes are started to process the corresponding data queues according to the priority level.

[0038] The present invention provides a multi-platform data collection method, which includes: detecting the hardware and software environment of each collection device; when detecting that each of the collection devices is running under different operating systems and network conditions, generating a data collection script related to each of the collection devices according to the detection results; starting the protocol plug-in corresponding to each of the collection devices based on the data collection script, and establishing a communication link with each of the collection devices; starting the data collection task after the communication link is established, standardizing the collected data and converting it into a unified format; performing preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmitting the processed data to the central cloud system for unification through an encrypted transmission protocol, thereby being able to adapt to the versatility and compatibility of data collection methods in multi-platform and multi-device environments, and improving the accuracy and efficiency of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a first embodiment of the multi-platform data acquisition method of the present invention;

[0040] Figure 2 This is a flow chart of a second embodiment of the multi-platform data acquisition method of the present invention;

[0041] Figure 3 2 is a flow chart of the third embodiment of the multi-platform data acquisition method of the present invention.

[0042] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0043] Reference Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the multi-platform data collection method of the present invention, which provides the first embodiment of the multi-platform data collection method of the present invention.

[0044] In a first embodiment, the multi-platform data collection method includes the following steps:

[0045] S10: Detect the hardware and software environment of each acquisition device;

[0046] S20: When it is detected that each of the acquisition devices is running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result;

[0047] S30: Starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script, and establishing a communication link with each of the acquisition devices;

[0048] S40: After the communication link is established, the data collection task is started, and the collected data is standardized and converted into a unified format;

[0049] S50: Perform preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmit the processed data to the central cloud system for unified processing via an encrypted transmission protocol.

[0050] It's important to note that the system first automatically detects the hardware and software environment of each data collection device. This is similar to a person first understanding the basic information of each device, such as whether the operating system is Windows, Linux, or another, and whether the device is connected to Wi-Fi or Ethernet. Based on the detected hardware and software environments, the system automatically generates corresponding data collection scripts. This is like a customized "operation guide" for each device, ensuring that each device can correctly collect data. Using the generated data collection scripts, the system activates the communication protocol plug-in appropriate for each device. This is equivalent to finding the appropriate "language translator" for each device, establishing a communication link and ensuring smooth communication between the device and the system. Once communication is established, the system begins the actual data collection task and standardizes the collected data, converting it into a unified format. This is similar to converting currencies from different countries into a single currency, facilitating subsequent processing and analysis. Initial processing (such as filtering and aggregation) is performed on the standardized data on the edge device. This is similar to the initial processing of raw materials, removing unnecessary components and integrating useful information. The data is then securely transmitted to the central cloud system using an encrypted transmission protocol (similar to an encrypted envelope). Ultimately, the initially processed data is transmitted to a central cloud system for further unified processing and analysis. The cloud system then performs more complex data analysis and applications, ultimately providing users with valuable information and decision support.

[0051] For example, in a smart city management system, there are various types of sensor devices, such as traffic signal sensors, pollution monitoring sensors, and smart streetlights. Each device may use a different operating system (e.g., embedded systems, Linux, etc.) and network conditions (e.g., Wi-Fi, cellular networks, etc.). Using the above method, the system automatically identifies the operating environment of each device, generates a specific collection script, establishes a communication link, begins data collection, and standardizes all data before transmitting it to the edge device for preliminary processing. Finally, the processed data is securely transmitted to the cloud for real-time monitoring and decision-making by city managers.

[0052] It should be understood that this embodiment can improve collection efficiency: by automatically generating and adapting scripts, it reduces manual configuration time and significantly improves data collection efficiency. Ensure data accuracy: Standardized processing and preliminary cleaning ensure data consistency and accuracy. Shorten the development cycle: Provide a general processing framework to reduce development time and quickly respond to the needs of different devices. Enhance data security: Ensure the security of data during transmission through encrypted transmission protocols. Adapt to multi-platform and multi-device needs: The flexibility of this method enables it to work stably under various operating systems and network conditions, meeting the needs of complex and diverse scenarios.

[0053] Reference Figure 2 , Figure 2 This is a flow chart of a second embodiment of the multi-platform data acquisition method of the present invention. In the second embodiment, starting the protocol plug-in corresponding to each acquisition device based on the data acquisition script and establishing a communication link with each acquisition device includes:

[0054] S301: Parse the data acquisition script and load a required protocol plug-in from a predefined location based on the parsing result. The predefined location is a local file system or a remote repository, and the protocol plug-in is a file stored in a dynamic library or a Python module.

[0055] S302: configuring a loaded protocol plug-in for each of the acquisition devices according to the parameters in the data acquisition script, and establishing a communication link with each of the acquisition devices based on the loaded protocol plug-in, wherein the parameters in the data acquisition script include the IP address, port number, authentication information and data format of the target device.

[0056] It's important to note that a data acquisition script is like a detailed set of instructions. The system first parses the script to understand its contents and instructions. This process is similar to reading an operating manual to understand the required tools and steps. Based on the parsing results, the system determines which protocol plug-ins need to be loaded. These plug-ins can be stored in the local file system (such as files on a computer's hard drive) or in a remote repository (such as an online repository). These plug-ins can be dynamic library files (like DLL files) or files within Python modules (Python packages). The system configures the loaded protocol plug-ins for each acquisition device based on the parameters in the data acquisition script. These parameters include the target device's IP address, port number, authentication information (such as username and password), and data format. This is equivalent to entering the necessary information according to the instructions in the manual and configuring the tool so that it can correctly connect and work with the target device. Using correctly configured plug-ins, the system establishes a communication link with each acquisition device, ensuring smooth data transmission, just like dialing the other party's phone number and starting a clear conversation.

[0057] For example, consider a smart agricultural monitoring system with a soil moisture sensor and a temperature sensor. These sensors run different operating systems and are connected via different networks (one via Wi-Fi, the other via Ethernet cable). Here's how this approach works: The system parses the script and discovers that the soil moisture sensor requires a specific communication protocol plugin, while the temperature sensor requires another plugin.

[0058] Following the script's instructions, the soil moisture sensor plugin is loaded from the local file system, while the temperature sensor plugin is loaded from a remote repository (e.g., an online resource server). The script parameters include the soil moisture sensor's IP address, port number, authentication information (such as username and password), and data format (e.g., JSON). Similarly, the temperature sensor's specific parameters are listed in the script. The system configures the respective plugins based on these parameters. Using the configured plugins, the system establishes communication links with the soil moisture and temperature sensors and begins collecting the corresponding data.

[0059] It should be understood that the beneficial effects of this embodiment are as follows: Flexibility and compatibility: The system can flexibly load the required plug-ins locally or remotely, adapt to various devices and network environments, and ensure compatibility. Dynamic configuration: By parsing scripts and parameter configurations, the system can dynamically adapt to the needs of different devices without manual intervention, thereby improving the degree of automation. Shortened deployment time: The automated loading and configuration process reduces manual operation time and speeds up the deployment process. Reliability and accuracy: According to the detailed parameters in the script, the communication link of each device is ensured to be accurate and correct, improving the reliability and accuracy of data collection. Easy to maintain and expand: Using scripts to manage and configure plug-ins, the system is easier to maintain and expand, and can adapt to new equipment and protocol requirements.

[0060] Furthermore, in this embodiment, configuring the loaded protocol plug-in for each acquisition device according to the parameters in the data acquisition script includes:

[0061] Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information;

[0062] Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class;

[0063] Traverse each device information in the script, find the corresponding plug-in module and class from the mapping relationship according to the protocol type of the device, and use the dynamic loading mechanism to configure the loaded protocol plug-in for each of the acquisition devices.

[0064] It should be noted that the script read by the system contains information about multiple devices, including the unique ID of the device, the type of protocol to be used, and its configuration information. Parse this information into an easy-to-process data format, such as a dictionary or list. Establish a mapping relationship between each protocol and the corresponding protocol plug-in, and use a dictionary (or similar data structure) to store the relationship between the protocol name and the corresponding plug-in module and class. This dictionary is like a guide, telling the system which plug-in module and class to use for each protocol. The system traverses each device information in the script, finds the corresponding plug-in module and class from the mapping relationship according to the protocol type of the device, and loads and configures it through the dynamic loading mechanism. The dynamic loading mechanism means loading the corresponding module as needed at runtime, rather than loading all possible modules when the program starts.

[0065] It should be understood that the beneficial effects of this embodiment are as follows: Automated processing: Such a method enables the system to automatically parse and process various device configuration information, reducing manual configuration errors. Flexibility: Through the dynamic loading mechanism, the system can load specific protocol plug-ins as needed at runtime, improving flexibility and resource utilization efficiency. Scalability: When adding new device types or protocols, only the scripts and mapping relationships need to be updated without major changes to the core logic of the system. Easy management: Using a dictionary to manage the relationship between protocols and plug-ins makes maintenance and expansion more intuitive and simple. Precise configuration: Precise configuration is performed according to the unique parameters of each device to ensure the accuracy and reliability of the communication process.

[0066] Reference Figure 3 , Figure 3 This is a flow chart of a third embodiment of the multi-platform data collection method of the present invention. In the third embodiment, after establishing a communication link, the data collection task is started, and the collected data is standardized and converted into a unified format, including:

[0067] S401: According to the data collection task defined in the script, each device is configured, including the data source, collection frequency, and data format, and the method in the plug-in class is called to start the data collection task;

[0068] S402: Perform preliminary processing on the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Data errors found during the preprocessing process are corrected or discarded.

[0069] S403: Mapping the pre-processed data to a standard format, converting the mapped data to a standard format.

[0070] It should be noted that each device is configured according to the data collection task defined in the script. The configuration content includes the data source (such as the device's IP address and port), the collection frequency (such as once a minute), and the data format (such as JSON or XML). Then, the method in the plug-in class is called to start the data collection task. The collected raw data may contain a lot of unnecessary information or noise data, so it needs to be preliminarily processed. This includes parsing the string, extracting the required fields (such as temperature values, humidity values, etc.), removing unnecessary noise data, and verifying the integrity and correctness of the data. If data errors are found, they are corrected or discarded. The preprocessed data needs to be mapped to a standard format, which usually means converting the data into a unified format to facilitate subsequent storage, processing, and analysis. For example, the data from different sensors can be standardized into a unified JSON format.

[0071] It should be understood that the beneficial effects of this embodiment are as follows: Efficient data management: Through automated scripts and configurations, data collection tasks can be efficiently managed and scheduled, reducing manual operations and configuration errors. Data quality improvement: The preliminary processing steps can effectively remove noise data, correct erroneous data, improve data quality, and ensure the accuracy and reliability of the data. Unified data format: By mapping the data to a unified standard format, the consistency of the data is ensured, which facilitates subsequent storage, processing and analysis. Real-time data processing: The dynamic configuration and loading mechanism enables the system to process various data sources and collection tasks in real time, improving the system's immediate response capabilities and processing efficiency. Scalability: The standardized and modular design makes the system easy to expand and maintain, and new device types and collection tasks can be easily added.

[0072] Furthermore, in this embodiment, mapping the preprocessed data to a standard format and converting the mapped data to the standard format include:

[0073] Clarify standard data formats, including timestamps, device IDs, data types, and data values;

[0074] The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into the target data structure.

[0075] It should be noted that the standard data format ensures that data from different devices can be represented and processed in a unified manner. The standard format includes the following key fields: Timestamp: Records the specific time of data collection, typically expressed in ISO 8601 format, such as "2024-06-06T06:00:00Z". Device ID (device_id): A unique identifier for each device, used to distinguish data from different sources. Data type (data_type): Specifies the data type, such as temperature, humidity, or pressure. Data value (data_value): The specific measured value, such as the temperature at a certain moment being 20.0 degrees Celsius or the humidity being 45%. Raw data collected from devices may contain redundant information or noise and requires preprocessing. Preprocessing involves parsing the data, extracting required fields (such as timestamps and temperature values), removing unnecessary information, and ensuring data integrity and accuracy. Preprocessed data is converted to a standard format using a pre-defined mapping function. The mapping function receives the processed raw data and generates structured data according to the standard format.

[0076] Furthermore, in this embodiment, the converted data is preliminarily processed on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol, including:

[0077] Perform preliminary processing on the converted data on the edge device, including filtering and aggregation. The filtering includes filtering and selecting useful data based on preset data types and device types, data value ranges, and time ranges. The aggregation includes aggregating data within a certain time window and aggregating data of different data types.

[0078] The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system for unified processing via the HTTPS encrypted transmission protocol.

[0079] It should be noted that data type and device type refer to retaining only specific types of data, such as focusing on temperature data or data from a specific sensor model. Data value range refers to setting a threshold to retain only data that falls within a specific range, such as retaining only temperature data between 20°C and 30°C. Time range refers to setting a time window to process only data within a specific time period. Aggregation within a time window refers to aggregating multiple data points into a single data point within a certain time interval (such as every minute or every hour), for example, calculating the average, maximum, or minimum value. Aggregation of different data types means that if a device generates multiple types of data (such as temperature and humidity), these data can be combined into a single data packet. Symmetric encryption uses the same key for encryption and decryption, such as AES (Advanced Encryption Standard). Asymmetric encryption uses a pair of keys (public and private), one for encryption and the other for decryption, such as the RSA algorithm. Use HTTPS (Hypertext Transfer Protocol Secure) to ensure data security during transmission. HTTPS provides encryption through the TLS (Transport Layer Security) protocol, preventing data from being eavesdropped or tampered with during transmission. Data decryption refers to the decryption of data using a key or private key shared with the edge device. Unified processing refers to further processing and analysis of the decrypted data, which may include long-term storage, in-depth analysis, report generation, or triggering of specific events. The beneficial effect is that through filtering and aggregation, only necessary and valuable data is transmitted, which greatly reduces the amount of data transmitted and improves network efficiency and transmission speed. After preliminary processing, the edge device can quickly respond to local events, such as triggering alarms or control operations based on real-time monitoring data, without having to wait for cloud processing. The pre-processed data has been optimized and compressed, and the amount of data transmitted to the cloud is reduced, reducing the resource consumption of cloud storage and processing. By performing encryption processing on the edge device, the security and privacy of data during transmission can be effectively guaranteed, and the risk of data leakage can be reduced.

[0080] Furthermore, in this embodiment, the detecting of the hardware and software environment of each acquisition device includes:

[0081] The hardware and software environment of each acquisition device is detected through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information and network device information; the software environment includes operating system version, application information, network configuration and power management settings.

[0082] It should be noted that in order to effectively manage and maintain distributed acquisition devices, it is necessary to detect and monitor the hardware and software environment information of these devices through system calls and API interfaces. Detect the hardware environment and device model: by querying the system's underlying hardware information or specific API interfaces. On Linux systems, you can use the cat / proc / device-tree / model or lshw command. On Windows systems, you can use the wmic csproduct get name command. Processor type and frequency: On Linux, you can view the / proc / cpuinfo file or use the lscpu command. On Windows, you can use the wmic cpu get name,MaxClockSpeed ​​command. Memory information: On Linux, you can use the free -m or cat / proc / meminfo command. On Windows, you can use systeminfo | find "TotalPhysical Memory" or wmic memorychip get capacity. Storage information: On Linux, you can use the df -h command to view storage usage. On Windows, you can use the wmic logicaldisk get size, freespace, caption command. Network device information: On Linux, you can use the ifconfig or ip a command. On Windows, you can use the ipconfig / all command. Check the software environment and operating system version: On Linux, you can view the / etc / os-release file or use the uname-a command. On Windows, you can use the systeminfo|find "OS" command. Application information: On Linux, you can use package manager commands such as dpkg-l (Debian) or rpm-qa (RedHat). On Windows, you can use the wmic product get name,version command. Network configuration: On Linux, you can view the network configuration file (such as / etc / network / interfaces or / etc / netplan / *.yaml) or use the nmcli command. On Windows, you can use netsh interface ipv4 show config to view the network configuration. Power management settings: On Linux, you can use the upower command or view power management related files (such as / sys / class / power_supply / ). On Windows, you can use the powercfg / query command to view the power configuration.Use APIs to obtain information. For some advanced devices or embedded systems, manufacturers may provide APIs that make it easier to obtain device information. For example, common interfaces include: SNMP (Simple Network Management Protocol): This can be used to obtain information about network devices, querying them using tools like snmpget or snmpwalk. RESTful APIs: Some modern devices may provide RESTful APIs that retrieve device information in JSON format via HTTP requests. Vendor-specific SDKs: Software development kits (SDKs) provided by manufacturers that enable device management and information acquisition.

[0083] Furthermore, in this embodiment, when it is detected that each of the acquisition devices is running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result includes:

[0084] Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device;

[0085] When the detection code based on the operating system and network condition of the acquisition device detects that each acquisition device is running under different operating systems and network conditions, a data acquisition script related to each acquisition device is generated according to the detection result.

[0086] It should be noted that in order to detect operating system and network conditions and generate corresponding data collection scripts, it is necessary to use Python's platform module and psutil module to obtain system information. Then, according to different operating systems and network conditions, corresponding data collection scripts are generated.

[0087] Furthermore, in this embodiment, the edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is directly processed on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

[0088] It's important to note that in distributed systems, classifying and processing data from edge devices is key to ensuring system efficiency and responsiveness. By processing data with high real-time requirements directly on edge devices and sending data with complex processing requirements to a central cloud system for processing, overall system performance and reliability can be improved. The following is a specific implementation plan, including data classification, processing, and transmission mechanisms. Data Classification System: Data with High Real-Time Requirements: This typically includes sensor data that requires a fast response, such as temperature, pressure, and speed. This data requires low-latency processing to enable immediate action or response. Real-time data processing is particularly important in areas such as industrial automation, smart transportation, and smart homes. Data with High Complex Processing Requirements: This includes large data volumes that require complex computation, analysis, or historical data comparison, such as video stream analysis, machine learning inference, and deep analytics. This data doesn't require immediate processing and can be batch processed in a central cloud system. Implementation Plan: 1. Data Classification Logic: Classify the data collected by edge devices to determine which data needs to be processed locally and which needs to be sent to the cloud: Define classification rules: Real-time data: Specify specific data types or data from specific sensors as data with high real-time requirements. Complex Data: Data requiring in-depth analysis or big data processing is designated as complex data. Data Identification and Storage: Real-time data is tagged and directly sent to a local processing queue. Complex data is tagged and cached locally for batch transmission to the cloud. 2. Real-time Data Processing: For data with high real-time requirements, processing is performed directly on the edge device. Local processing involves data filtering, transformation, and processing, such as temperature over-limit alarms and environmental control. Results are stored in a local database or cache for subsequent use by local systems. Event-driven mechanisms: Configure event-driven mechanisms to trigger responses to specific conditions in real time, such as turning on a cooling fan. 3. Complex Data Transmission and Processing: For complex data with large processing volumes, data is sent to a central cloud system via a transmission mechanism for processing. Data Caching and Batch Transfer: Caching mechanisms temporarily store complex data in a local cache or database. Scheduled or Triggered Batch Data Transfer: Utilize scheduled tasks or threshold-triggered mechanisms to batch transmit cached data to the cloud. Transmission: Utilize the HTTPS protocol with SSL / TLS encryption to ensure secure data transmission. Use a reliable message queue system (such as RabbitMQ or Kafka) to ensure reliable and efficient data transmission. Cloud Processing: After receiving data, the central cloud system decrypts and stores it. Big data processing tools (such as Hadoop and Spark) and machine learning platforms are used to analyze and process complex data. Reports, predictions, or commands are generated and returned to edge devices as needed. By effectively combining the advantages of edge computing and cloud computing, system performance, responsiveness, and processing capabilities can be significantly improved.Edge devices improve local response efficiency by processing highly critical data in real time. Furthermore, by transferring complex data in batches to the cloud for in-depth analysis, they fully utilize the cloud's powerful computing resources and achieve efficient and comprehensive data processing. This solution is of great significance for IoT systems, big data processing, and smart device management.

[0089] Furthermore, in this embodiment, after the complex data with large processing volume is sent to the central cloud system via the transmission mechanism, the process further includes:

[0090] Determine the priority of complex and large-volume data. Priority refers to an identifier that distinguishes the importance and processing order of different tasks or data in the processing system;

[0091] The complex data with large processing volume are classified and stored in different queues based on the priority level, and different threads or processes are started to process the corresponding data queues according to the priority level.

[0092] It should be noted that to implement priority management for complex data processing, data with large processing volumes can be classified and stored in different queues based on priority, and different threads or processes can be started to process the corresponding data queues according to priority. By combining Python's queue and threading modules, complex data can be stored in different priority queues based on priority, and each queue can be processed separately by independent threads. This method can effectively improve the efficiency and responsiveness of the system in processing complex data.

[0093] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0094] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that enumerates several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order; these terms should be interpreted as designations.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for enabling an end-user device (which can be a mobile phone, a computer, a server, an air conditioner, or a network user device, etc.) to execute the methods described in each embodiment of the present invention.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-platform data collection method, characterized in that: The method comprises: Detect the hardware and software environment of each acquisition device; When detecting that each of the acquisition devices is running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result; Starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices; After establishing the communication link, the data collection task is started, and the collected data is standardized and converted into a unified format; Perform preliminary processing on the converted data on the edge device, including filtering and aggregation, and transmit the processed data to the central cloud system for unified processing via an encrypted transmission protocol; The step of starting the protocol plug-in corresponding to each of the acquisition devices based on the data acquisition script and establishing a communication link with each of the acquisition devices includes: Parsing the data acquisition script and loading the required protocol plug-in from a predefined location based on the parsing result. The predefined location is a local file system or a remote repository. The protocol plug-in is a file stored in a dynamic library or a Python module. Configuring a loaded protocol plug-in for each of the acquisition devices according to parameters in the data acquisition script, and establishing a communication link with each of the acquisition devices based on the loaded protocol plug-in, wherein the parameters in the data acquisition script include the IP address, port number, authentication information, and data format of the target device; The protocol plug-in configured and loaded for each acquisition device according to the parameters in the data acquisition script includes: Parsing the read script into a data format suitable for processing, wherein the data format includes information of multiple devices, each of which has its own unique ID, protocol type, and corresponding configuration information; Establish a mapping relationship between each protocol and its corresponding protocol plug-in, and use a dictionary to store the relationship between the protocol name and the corresponding plug-in module and class; Traverse each device information in the script, find the corresponding plug-in module and class from the mapping relationship according to the protocol type of the device, and use the dynamic loading mechanism to configure the loaded protocol plug-in for each of the acquisition devices; The detection of the hardware and software environment of each acquisition device includes: Detect the hardware and software environment of each acquisition device through system calls or API interfaces. The hardware environment includes device model, processor type and frequency, memory information, storage information, and network device information. The software environment includes operating system version, application information, network configuration, and power management settings. When it is detected that each of the acquisition devices is running under different operating systems and network conditions, generating a data acquisition script related to each of the acquisition devices according to the detection result includes: Use the platform module and psutil module in the Python standard library to write the operating system and network condition detection code of the acquisition device; When the detection code based on the operating system and network condition of the acquisition device detects that each acquisition device is running under different operating systems and network conditions, a data acquisition script related to each acquisition device is generated according to the detection result.

2. The multi-platform data collection method according to claim 1, wherein: After establishing the communication link, the data collection task is started, and the collected data is standardized and converted into a unified format, including: According to the data collection task defined in the script, each device is configured, including the data source, collection frequency, and data format, and the method in the plug-in class is called to start the data collection task; Perform preliminary processing on the collected raw data, including parsing character strings, extracting required fields, removing unnecessary noise data, and verifying the integrity and correctness of the data. Any data errors found during the preprocessing process will be corrected or discarded. Map the preprocessed data to a standard format and convert the mapped data to a standard format.

3. The multi-platform data collection method according to claim 2, wherein: Mapping the preprocessed data to a standard format and converting the mapped data to the standard format include: Clarify standard data formats, including timestamps, device IDs, data types, and data values; The preprocessed data is mapped to a standard format through a preset mapping function, and the mapped data is converted into the target data structure.

4. The multi-platform data collection method according to claim 1, wherein: The converted data is processed initially on the edge device, including filtering and aggregation, and the processed data is transmitted to the central cloud system for unified processing via an encrypted transmission protocol, including: Perform preliminary processing on the converted data on the edge device, including filtering and aggregation. The filtering includes filtering and selecting useful data based on preset data types and device types, data value ranges, and time ranges. The aggregation includes aggregating data within a certain time window and aggregating data of different data types. The aggregated data is encrypted using symmetric or asymmetric encryption, and the processed data is transmitted to the central cloud system via the HTTPS encrypted transmission protocol for unified processing.

5. The multi-platform data collection method according to claim 1, wherein: The edge device is also used to classify the converted data, including data with high real-time requirements and data with large complex processing volume. The data with high real-time requirements is directly processed on the edge device, and the data with large complex processing volume is sent to the central cloud system through a transmission mechanism.

6. The multi-platform data collection method according to claim 5, characterized in that: After the complex and heavy-duty data is sent to the central cloud system via a transmission mechanism, the following steps are also included: Determine the priority of complex and large-volume data. Priority refers to an identifier that distinguishes the importance and processing order of different tasks or data in the processing system; The data with complex processing volume and large processing capacity are classified and stored in different queues based on the priority levels, and different threads or processes are started to process the corresponding data queues according to the priority levels.

Images