Acquisition method, device and system based on IOT (Internet of Things) data and medium

Abstract

The invention belongs to the technical field of Internet of Things, and discloses an IOT data acquisition method, device and system and a medium, and the method comprises the steps that a data acquisition module acquires multi-source heterogeneous data from hardware equipment, and transmits the acquired multi-source heterogeneous data to a data format conversion module; a data format conversion module converts the multi-source heterogeneous data into a uniform data format which can be identified by an Internet of Things platform; according to results of data acquisition and data format conversion, the abnormal data processing module is responsible for processing abnormal data occurring in the process of data acquisition and data format conversion; and the data reliable transmission module is responsible for transmitting the converted data from the Internet of Things platform to the user side. According to the invention, efficient, accurate and reliable transmission of hardware equipment data to the Internet of Things platform is realized, and the problems of incompatible data formats, low data acquisition efficiency, insufficient abnormal data processing capability, unreliable data transmission and the like are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to the connection function between terminal equipment and a data acquisition system. Background Art

[0002] With the rapid development of IoT technology, more and more hardware devices are connected to the IoT platform. However, since the data formats of hardware devices are diverse and may contain a large amount of multi-source heterogeneous data, directly connecting them to the IoT platform often leads to problems such as incompatible data formats, low data collection efficiency, insufficient abnormal data processing capabilities, and unreliable data transmission. Therefore, it is crucial to develop a solution that can improve the quality of data collection. Summary of the invention

[0003] To solve this problem, the present invention provides a method and system for collecting IOT data, which can realize data format conversion, data collection, abnormal data processing and reliable data transmission, aiming to achieve efficient, accurate and reliable transmission of hardware device data to the Internet of Things platform.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for collecting IOT data based on the Internet of Things includes the following steps:

[0006] Step 1: The data acquisition module collects multi-source heterogeneous data from the hardware device and transmits the collected multi-source heterogeneous data to the data format conversion module;

[0007] Step 2: The data format conversion module converts the multi-source heterogeneous data into a unified data format that can be recognized by the Internet of Things platform. The data format conversion module first identifies the data type, format and communication protocol of the hardware device, and then converts the multi-source heterogeneous data into a standard data format according to the preset conversion rules. Data parsing and encoding technology are used in the conversion process to ensure the accuracy and efficiency of data conversion.

[0008] Step 3: The abnormal data processing module processes the abnormal data that occurs during data collection and data format conversion. The abnormal data processing module first defines the threshold and judgment rules of the abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data. For abnormal data, the abnormal data processing module triggers an alarm process, including sending an alarm notification to the user, controlling device behavior or adjusting business processes; the abnormal data processing module also records detailed information of the abnormal data for subsequent analysis and processing;

[0009] Step 4: The data reliable transmission module transmits the converted data from the IoT platform to the user end. The data reliable transmission module adopts data encryption, data verification, and retransmission mechanisms to ensure the reliability of data transmission. In the process of data transmission, if an error or loss occurs, the data reliable transmission module automatically starts the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately.

[0010] Furthermore, step one specifically includes:

[0011] Data source connection: The data acquisition module connects to various hardware device data sources, including sensors, gateway devices, and APIs;

[0012] Data extraction: The data acquisition module extracts the required data from connected data sources, including sensor readings, gateway acquisition, and API calls;

[0013] Data cleaning: After extracting data, the data collection module performs data cleaning to remove duplicate data, delete invalid data, and fill in missing values ​​to improve data quality;

[0014] Data storage and transmission: The cleaned data is stored and transmitted to the data format conversion module for further processing.

[0015] Furthermore, the data acquisition module supports multiple data acquisition methods, including software direct communication acquisition, acquisition through a gateway, and acquisition through a digital controller. According to the specific conditions of the hardware device, the data acquisition module automatically selects an appropriate acquisition method to ensure the integrity and real-time nature of data acquisition;

[0016] The details of software direct communication acquisition, acquisition through gateway and acquisition through digital controller are as follows:

[0017] Software direct communication acquisition: The data acquisition module communicates directly with the hardware device through a specific software protocol or interface to obtain data for data exchange with the programmable logic controller PLC or other intelligent devices;

[0018] Collect data through a gateway: In some cases, the data collection module cannot communicate directly with the hardware device. In this case, the gateway is used as an intermediary to collect data. The gateway can support various communication protocols and interfaces to achieve communication with a variety of external devices.

[0019] Collection through digital controllers: The digital controllers are connected to various sensors and devices, and the data acquisition module communicates with the digital controllers to indirectly obtain data from the connected sensors and devices. This method is suitable for large-scale control systems and complex industrial environments.

[0020] Furthermore, in the process of data collection and reception, first determine the data type, which includes numeric, text, structured and unstructured data; secondly, select the communication protocol: select HTTP, MQTT, CoAP or Modbus communication protocol for data transmission according to the application scenario; finally, write a data receiving program: use the corresponding network library or API to listen to and receive data, and for complex data transmission networks, implement intelligent routing algorithms to select the optimal communication path according to network conditions and device status.

[0021] Furthermore, step 2 specifically includes:

[0022] Determine the data format: determine the data format according to the received data header information or metadata, the data format includes JSON and XML;

[0023] Select parsing technology: select a corresponding parsing technology according to the data format, the parsing technology includes JSON parsing, XML parsing or text parsing;

[0024] Parsing data: using the selected parsing technique to convert the raw data into a program's internal data structure, which may include a dictionary, list, or custom object;

[0025] Data processing and encoding: clean, convert, and aggregate the parsed data; select encoding technology based on needs; convert the processed data into the selected encoding format; for large-scale data streams, use streaming processing technology for real-time processing to reduce memory usage and latency; for sensitive data, add encryption algorithms during the encoding process to ensure data security.

[0026] Furthermore, the specific steps of step three are as follows:

[0027] 1) Define the threshold and judgment rules for abnormal data

[0028] Collect business requirements: clarify the business's expectations and requirements for data;

[0029] Analyze data characteristics: study the distribution, change patterns and trends of historical data;

[0030] Set thresholds: Set reasonable thresholds based on business requirements and data characteristics;

[0031] Defining decision rules: In addition to simple threshold comparison, more complex decision rules are defined, including statistical anomaly detection algorithms, data change rate analysis, and logical relationships between data;

[0032] 2) Abnormal data detection

[0033] Real-time detection: During the data collection process, each data point is checked in real time to see if it exceeds the threshold or does not meet the judgment rules;

[0034] Batch detection: For historical data or batch imported data, batch detection is performed to identify abnormal data; when performing anomaly detection in real-time data streams, efficient streaming algorithms are used to reduce resource consumption;

[0035] 3) Abnormal data processing

[0036] Marking anomalies: Marking detected abnormal data as "anomalies" for subsequent processing and analysis;

[0037] Data correction: Correct abnormal data. The correction methods used include interpolation and average value methods.

[0038] Data elimination: eliminate abnormal data that cannot be corrected or still does not meet the requirements after correction;

[0039] 4) Alarm process

[0040] Alarm notification: Alarm notifications are sent to users via email, SMS, or APP push notifications to inform them of abnormal data. The notifications include detailed information about the abnormal data, possible impacts, and recommended countermeasures. The detailed information includes timestamps, data values, and abnormality types.

[0041] Emergency processing: For serious abnormalities, emergency processing procedures can be triggered, including emergency shutdown and emergency switching of backup systems.

[0042] Set smart notification strategies: intelligently adjust the frequency and content of notifications based on user behavior and preferences;

[0043] 5) Equipment control or business process adjustment

[0044] Equipment control: Automatically control the operating status of the equipment according to the type and degree of abnormal data, including shutting down, restarting or adjusting operating parameters;

[0045] Business process adjustment: Adjust the business process for abnormal data related to the business process;

[0046] 6) Abnormal data recording and analysis

[0047] Detailed record: record detailed information of abnormal data;

[0048] Data analysis: Conduct in-depth analysis of abnormal data to identify abnormal causes, trends and patterns, provide a basis for optimizing the system and preventing future abnormalities, and use data visualization tools to display the analysis results of abnormal data to improve readability and understanding;

[0049] Intelligent report generation: Automatically generate customized abnormal data reports based on user needs and preferences, and provide management or related teams with statistical and analytical results of abnormal data;

[0050] 7) User interface and interaction

[0051] Abnormal data display: Display the detailed information of abnormal data through the user interface, including the list, chart and map of abnormal data;

[0052] User operations: Allow users to view, confirm, process and analyze abnormal data, provide users with interactive data analysis tools, and enable users to customize analysis logic and visual display;

[0053] Feedback mechanism: Provide a user feedback mechanism to collect users' opinions and suggestions on the abnormal data processing module so as to continuously optimize and improve it.

[0054] Furthermore, the specific steps of step 4 are as follows:

[0055] 1) Data encryption

[0056] Select encryption algorithm: Select a strong encryption algorithm, such as AES or RSA, based on the importance of the data and the security requirements of the transmission;

[0057] Generate keys: Use key management tools or algorithms to generate keys to ensure the complexity and randomness of the keys.

[0058] Encrypted data: Encrypt the data to be transmitted using the selected encryption algorithm and the generated key;

[0059] Key management: realize the generation, distribution, update and destruction management functions of keys to ensure the security of key storage and use;

[0060] 2) Data verification

[0061] Select the verification algorithm: According to the reliability and integrity requirements of data transmission, select a suitable verification algorithm, including CRC, MD5 or SHA-256;

[0062] Calculate the checksum: Before sending the data, use the selected checksum algorithm to calculate the checksum of the data and append the checksum to the data.

[0063] Receiving verification: After receiving the data, the receiver uses the same verification algorithm to recalculate the verification value and compare it with the received verification value;

[0064] Determine data integrity: If the calculated checksum is consistent with the received checksum, the data is complete; otherwise, an error occurred during data transmission;

[0065] 3) Retransmission mechanism

[0066] Error detection: The receiver uses data verification to detect whether errors or losses occur during data transmission;

[0067] Triggered retransmission: When an error or loss is detected, the receiver triggers a retransmission request based on the retransmission strategy;

[0068] Select retransmission strategy: select timeout retransmission, selective retransmission or fast retransmission strategy according to actual situation;

[0069] Retransmission control:

[0070] The sender adjusts the retransmission strategy based on factors such as network conditions and error types, including increasing the number of retransmissions and reducing the size of the data packet;

[0071] 4) Congestion Control

[0072] Congestion detection: Determine whether the network is congested by monitoring performance indicators in the network, such as packet loss rate and delay;

[0073] Adjust the transmission rate: When congestion is detected, gradually reduce the data transmission rate to avoid further aggravating the congestion;

[0074] Adopt congestion avoidance algorithm: Use slow start, congestion avoidance and other algorithms to gradually increase the transmission rate during data transmission to achieve network stability and efficiency;

[0075] Monitoring and alarm: Real-time monitoring of data transmission status and performance indicators, and triggering alarm notifications when abnormalities occur, so that problems can be handled in a timely manner.

[0076] A data collection device based on the Internet of Things (IOT), comprising:

[0077] A data acquisition module is used to collect multi-source heterogeneous data from hardware devices and transmit the collected multi-source heterogeneous data to a data format conversion module;

[0078] A data format conversion module is used to convert the multi-source heterogeneous data into a unified data format that can be recognized by the Internet of Things platform. The data format conversion module first identifies the data type, format and communication protocol of the hardware device, and then converts the multi-source heterogeneous data into a standard data format according to preset conversion rules. Data parsing and encoding technology are used in the conversion process to ensure the accuracy and efficiency of data conversion;

[0079] The abnormal data processing module is used to process abnormal data that occurs during data collection and data format conversion. The abnormal data processing module first defines the threshold and judgment rules of abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data. For abnormal data, the abnormal data processing module triggers an alarm process, including sending an alarm notification to the user, controlling device behavior or adjusting business processes; the abnormal data processing module also records detailed information of the abnormal data for subsequent analysis and processing;

[0080] The data reliable transmission module is used to transmit the converted data from the Internet of Things platform to the user end. The data reliable transmission module adopts data encryption, data verification, and retransmission mechanisms to ensure the reliability of data transmission. In the process of data transmission, if an error or loss occurs, the data reliable transmission module automatically starts the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately.

[0081] A data collection system based on the Internet of Things (IoT), comprising: a computer-readable storage medium and a processor;

[0082] The computer-readable storage medium is used to store executable instructions;

[0083] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method for collecting data based on the Internet of Things (IOT).

[0084] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for collecting data based on the Internet of Things (IoT).

[0085] The present invention has the following beneficial effects:

[0086] The present invention is responsible for collecting data from hardware devices through a data acquisition module and transmitting it to a data format conversion module; the data format conversion module is responsible for converting multi-source heterogeneous data generated by hardware devices into a unified data format that can be recognized by the Internet of Things platform; the abnormal data processing module is responsible for processing abnormal data that occurs during data acquisition and data format conversion. The module first defines the threshold and judgment rules of abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data; the data reliable transmission module is responsible for transmitting the converted data from the Internet of Things platform to the user end through long-term data calibration to ensure that the data can be transmitted to the user end completely and accurately; it solves the problems of incompatible data formats, low data acquisition efficiency, insufficient abnormal data processing capabilities, and unreliable data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1The present invention is a flowchart of a method for collecting IOT data based on an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The embodiment of the present application solves the problems of incompatible data formats, low data collection efficiency, insufficient abnormal data processing capability, and unreliable data transmission by providing a collection method based on Internet of Things (IoT) data.

[0089] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with specific implementation methods. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0090] like Figure 1 As shown, the present invention provides a method for collecting IOT data based on the Internet of Things, including:

[0091] Step 101, the data acquisition module is responsible for collecting data from the hardware device and transmitting it to the data format conversion module. This module supports multiple data acquisition methods, including software direct communication acquisition, acquisition through a gateway, and acquisition through a digital controller. According to the specific situation of the hardware device, the data acquisition module can automatically select the appropriate acquisition method to ensure the integrity and real-time nature of data acquisition. Specifically, the implementation steps of step 101 are as follows:

[0092] Data source connection: The data acquisition module first connects to various hardware device data sources, including sensors, gateways, APIs, etc.

[0093] Data Extraction: The module extracts the required data from connected data sources, which can include sensor readings, gateway devices, etc.

[0094] Data cleaning: After extracting data, the module will clean the data to remove duplicate data, delete invalid data, fill in missing values, etc. to improve data quality;

[0095] Data storage and transmission: The cleaned data will be stored and transmitted to the data format conversion module for further processing.

[0096] Software direct communication acquisition: The data acquisition module communicates directly with the hardware device through a specific software protocol or interface to obtain data. This method is usually used for data exchange with programmable logic controllers (PLCs) or other intelligent devices.

[0097] Data acquisition through a gateway: In some cases, the data acquisition module may not be able to communicate directly with the hardware device. In this case, a gateway (such as a PLC gateway) can be used as an intermediary for data acquisition. The gateway can support various communication protocols and interfaces to achieve communication with a variety of external devices.

[0098] Data acquisition through digital controllers: Digital controllers (such as PLCs) can be connected to various sensors and devices, and data acquisition modules can communicate with digital controllers to indirectly obtain data from connected sensors and devices. This method is suitable for large-scale control systems and complex industrial environments.

[0099] Step 102, the data format conversion module is responsible for converting the multi-source heterogeneous data generated by the hardware device into a unified data format that can be recognized by the IoT platform. The module first identifies the data type, format and communication protocol of the hardware device, and then converts the original data into a standard data format according to the preset conversion rules. During the conversion process, efficient data parsing and encoding technology is used to ensure the accuracy and efficiency of data conversion.

[0100] Specifically, according to the data collection results, the data format is converted into the system requirements. The specific steps of data format conversion are as follows:

[0101] Data parsing technologies include: JSON parsing, using the JSON library to parse JSON format data into program internal data structures, such as dictionaries or lists. XML parsing, using the XML parsing library to convert XML data into program internal data structures. Text parsing, using regular expressions and other technologies to parse text data and extract the required information.

[0102] Data encoding technology, JSON encoding: convert the data structure inside the program (such as dictionaries, lists) into a string in JSON format for easy transmission and storage. XML encoding: convert the data structure inside the program into a string in XML format. Encoding conversion: when processing text data, you may encounter different text encoding formats (such as UTF-8, GBK, etc.). Use encoding conversion technology to ensure the correct display and processing of data.

[0103] Finally, select the corresponding communication protocol, for example: HTTP (Hypertext Transfer Protocol): an application layer protocol commonly used for communication between web browsers and servers. MQTT (Message Queuing Telemetry Transport): a lightweight publish / subscribe message transmission protocol widely used in the field of Internet of Things. CoAP (Constrained Application Protocol): a lightweight protocol designed specifically for the Internet of Things, suitable for resource-constrained devices. Modbus (Modbus protocol): an industrial communication protocol, mainly used for communication between industrial automation equipment. Modbus is a master-slave communication protocol that defines the data communication method, the request and response process, and the data structure.

[0104] Step 103: According to the data collection and data format conversion methods of steps 101 and 102, data has been successfully collected into the system. However, abnormal data may appear in the process of data collection and data format conversion. The specific steps for processing abnormal data are as follows:

[0105] First, define the threshold and judgment rules for abnormal data. During the data collection process, check in real time whether each data point exceeds the threshold or does not meet the judgment rules. For historical data or batch imported data, perform batch detection to identify abnormal data. Mark the detected abnormal data as "abnormal" for subsequent processing and analysis. For abnormal data that cannot be corrected or still does not meet the requirements after correction, you can choose to remove it. Send alarm notifications to users through emails, text messages, APP push, etc. to inform them of relevant information about abnormal data.

[0106] For serious abnormalities, it may be necessary to trigger emergency processing procedures, such as emergency shutdown, emergency switching of backup systems, etc.

[0107] Business process adjustment: For abnormal data related to business processes, it may be necessary to adjust the business process, such as suspending order processing, switching production lines, etc. Generate abnormal data reports regularly and provide management or related teams with statistical and analytical results of abnormal data. Allow users to view, confirm, process and analyze abnormal data, such as re-evaluating abnormalities, adjusting thresholds, exporting data, etc.

[0108] Step 104, the data reliable transmission module is responsible for transmitting the converted data from the IoT platform to the user end. This module uses a variety of technical means to ensure the reliability of data transmission, including data encryption, data verification, retransmission mechanism, etc. During the data transmission process, if an error or loss occurs, the data reliable transmission module will automatically start the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately;

[0109] The present invention provides a data collection method based on the Internet of Things (IoT), which effectively improves the accuracy and reliability of data collection, thereby promoting the optimized implementation of data collection and transmission in the IoT system. This method overcomes the problems of incompatible data formats, low data collection efficiency, insufficient abnormal data processing capabilities, and unreliable data transmission in traditional methods, making it have a wider range of applicable scenarios and stronger data collection performance. In addition, the method is easy to operate and implement, and simplifies the data collection process. This highly flexible and adaptable feature makes the method have great potential and broad prospects in future application fields such as IoT data collection.

[0110] Another aspect of the present invention provides a data collection device based on the Internet of Things (IoT), comprising:

[0111] A data acquisition module is used to collect multi-source heterogeneous data from hardware devices and transmit the collected multi-source heterogeneous data to a data format conversion module;

[0112] A data format conversion module is used to convert the multi-source heterogeneous data into a unified data format that can be recognized by the Internet of Things platform. The data format conversion module first identifies the data type, format and communication protocol of the hardware device, and then converts the multi-source heterogeneous data into a standard data format according to preset conversion rules. Data parsing and encoding technology are used in the conversion process to ensure the accuracy and efficiency of data conversion;

[0113] The abnormal data processing module is used to process abnormal data that occurs during data collection and data format conversion. The abnormal data processing module first defines the threshold and judgment rules of abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data. For abnormal data, the abnormal data processing module triggers an alarm process, including sending an alarm notification to the user, controlling device behavior or adjusting business processes; the abnormal data processing module also records detailed information of the abnormal data for subsequent analysis and processing;

[0114] The data reliable transmission module is used to transmit the converted data from the Internet of Things platform to the user end. The data reliable transmission module adopts data encryption, data verification, and retransmission mechanisms to ensure the reliability of data transmission. In the process of data transmission, if an error or loss occurs, the data reliable transmission module automatically starts the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately.

[0115] Another aspect of the present invention provides a data collection system based on the Internet of Things (IoT), including: a computer-readable storage medium and a processor;

[0116] The computer-readable storage medium is used to store executable instructions;

[0117] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the method for collecting Internet of Things (IOT) data described in the first aspect.

[0118] Another aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the Internet of Things (IoT) data collection and presentation method described in the first aspect.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of embodiments of software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for collecting data based on the Internet of Things (IoT), characterized in that: The steps include: Step 1: The data acquisition module collects multi-source heterogeneous data from the hardware device and transmits the collected multi-source heterogeneous data to the data format conversion module; Step 2: The data format conversion module converts the multi-source heterogeneous data into a unified data format that can be recognized by the Internet of Things platform. The data format conversion module first identifies the data type, format and communication protocol of the hardware device, and then converts the multi-source heterogeneous data into a standard data format according to the preset conversion rules. Data parsing and encoding technology are used in the conversion process to ensure the accuracy and efficiency of data conversion. Step 3: The abnormal data processing module processes the abnormal data that occurs during data collection and data format conversion. The abnormal data processing module first defines the threshold and judgment rules of the abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data. For abnormal data, the abnormal data processing module triggers an alarm process, including sending an alarm notification to the user, controlling device behavior or adjusting business processes; the abnormal data processing module also records detailed information of the abnormal data for subsequent analysis and processing; Step 4: The data reliable transmission module transmits the converted data from the IoT platform to the user end. The data reliable transmission module adopts data encryption, data verification, and retransmission mechanisms to ensure the reliability of data transmission. In the process of data transmission, if an error or loss occurs, the data reliable transmission module automatically starts the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately.

2. The method for collecting data based on the Internet of Things (IOT) according to claim 1, characterized in that: Step 1 specifically includes: Data source connection: The data acquisition module connects to various hardware device data sources, including sensors, gateway devices, and APIs; Data extraction: The data acquisition module extracts the required data from connected data sources, including sensor readings, gateway acquisition, and API calls; Data cleaning: After extracting data, the data collection module performs data cleaning to remove duplicate data, delete invalid data, and fill in missing values ​​to improve data quality; Data storage and transmission: The cleaned data is stored and transmitted to the data format conversion module for further processing.

3. The method for collecting data based on the Internet of Things (IOT) according to claim 1, characterized in that: The data acquisition module supports multiple data acquisition methods, including software direct communication acquisition, acquisition through a gateway, and acquisition through a digital controller. According to the specific conditions of the hardware equipment, the data acquisition module automatically selects the appropriate acquisition method to ensure the integrity and real-time nature of data acquisition; The details of software direct communication acquisition, acquisition through gateway and acquisition through digital controller are as follows: Software direct communication acquisition: The data acquisition module communicates directly with the hardware device through a specific software protocol or interface to obtain data for data exchange with the programmable logic controller PLC or other intelligent devices; Collect data through a gateway: In some cases, the data collection module cannot communicate directly with the hardware device. In this case, the gateway is used as an intermediary to collect data. The gateway can support various communication protocols and interfaces to achieve communication with a variety of external devices. Collection through digital controllers: The digital controllers are connected to various sensors and devices, and the data acquisition module communicates with the digital controllers to indirectly obtain data from the connected sensors and devices. This method is suitable for large-scale control systems and complex industrial environments.

4. The method for collecting data based on the Internet of Things (IOT) according to claim 1 is characterized in that: In the process of data collection and reception, first determine the data type, which includes numeric, text, structured and unstructured data; secondly, select the communication protocol: select HTTP, MQTT, CoAP or Modbus communication protocol for data transmission according to the application scenario; finally, write the data receiving program: use the corresponding network library or API to listen and receive data, and implement intelligent routing algorithms for complex data transmission networks to select the optimal communication path according to network conditions and device status.

5. The method for collecting data based on the Internet of Things (IOT) according to claim 1 is characterized in that: Step 2 specifically includes: Determine the data format: determine the data format according to the received data header information or metadata, the data format includes JSON and XML; Select parsing technology: select a corresponding parsing technology according to the data format, the parsing technology includes JSON parsing, XML parsing or text parsing; Parsing data: using the selected parsing technique to convert the raw data into a program's internal data structure, such as a dictionary, list, or custom object; Data processing and encoding: clean, convert, and aggregate the parsed data; select encoding technology based on needs; convert the processed data into the selected encoding format; for large-scale data streams, use streaming processing technology for real-time processing to reduce memory usage and latency; for sensitive data, add encryption algorithms during the encoding process to ensure data security.

6. The method for collecting data based on the Internet of Things (IOT) according to claim 1, characterized in that: Step 3 The specific steps are as follows: 1) Define the threshold and judgment rules for abnormal data Collect business requirements: clarify the business's expectations and requirements for data; Analyze data characteristics: study the distribution, change patterns and trends of historical data; Set thresholds: Set reasonable thresholds based on business requirements and data characteristics; Defining decision rules: In addition to simple threshold comparison, more complex decision rules are defined, including statistical anomaly detection algorithms, data change rate analysis, and logical relationships between data; 2) Abnormal data detection Real-time detection: During the data collection process, each data point is checked in real time to see if it exceeds the threshold or does not meet the judgment rules; Batch detection: For historical data or batch imported data, batch detection is performed to identify abnormal data; when performing anomaly detection in real-time data streams, efficient streaming algorithms are used to reduce resource consumption; 3) Abnormal data processing Mark anomalies: Mark the detected abnormal data as "anomalies" for subsequent processing and analysis; Data correction: Correct abnormal data. The correction methods used include interpolation and average value methods. Data elimination: eliminate abnormal data that cannot be corrected or still does not meet the requirements after correction; 4) Alarm process Alarm notification: Alarm notifications are sent to users via email, SMS, or APP push notifications to inform them of abnormal data. The notifications include detailed information about the abnormal data, possible impacts, and recommended countermeasures. The detailed information includes timestamps, data values, and abnormality types. Emergency processing: For serious abnormalities, emergency processing procedures can be triggered, including emergency shutdown and emergency switching of backup systems. Set smart notification strategies: intelligently adjust the frequency and content of notifications based on user behavior and preferences; 5) Equipment control or business process adjustment Equipment control: Automatically control the operating status of the equipment according to the type and degree of abnormal data, including shutting down, restarting or adjusting operating parameters; Business process adjustment: Adjust the business process for abnormal data related to the business process; 6) Abnormal data recording and analysis Detailed record: record detailed information of abnormal data; Data analysis: Conduct in-depth analysis of abnormal data to identify abnormal causes, trends and patterns, provide a basis for optimizing the system and preventing future abnormalities, and use data visualization tools to display the analysis results of abnormal data to improve readability and understanding; Intelligent report generation: Automatically generate customized abnormal data reports based on user needs and preferences, and provide management or related teams with statistical and analytical results of abnormal data; 7) User interface and interaction Abnormal data display: Display the detailed information of abnormal data through the user interface, including the list, chart and map of abnormal data; User operations: Allow users to view, confirm, process and analyze abnormal data, provide users with interactive data analysis tools, and enable users to customize analysis logic and visual display; Feedback mechanism: Provide a user feedback mechanism to collect users' opinions and suggestions on the abnormal data processing module so as to continuously optimize and improve it.

7. The method for collecting IOT data according to claim 1 is characterized in that: Step 4 The specific steps are as follows: 1) Data encryption Select encryption algorithm: Select a strong encryption algorithm, such as AES or RSA, based on the importance of the data and the security requirements of the transmission; Generate keys: Use key management tools or algorithms to generate keys and ensure the complexity and randomness of the keys. Encrypted data: Encrypt the data to be transmitted using the selected encryption algorithm and the generated key; Key management: realize the generation, distribution, update and destruction management functions of keys to ensure the security of key storage and use; 2) Data verification Select the verification algorithm: According to the reliability and integrity requirements of data transmission, select a suitable verification algorithm, including CRC, MD5 or SHA-256; Calculate the checksum: Before sending the data, use the selected checksum algorithm to calculate the checksum of the data and append the checksum to the data. Receiving verification: After receiving the data, the receiver uses the same verification algorithm to recalculate the verification value and compares it with the received verification value; Determine data integrity: If the calculated checksum is consistent with the received checksum, the data is complete; Otherwise, data errors occur during transmission; 3) Retransmission mechanism Error detection: The receiver uses data verification to detect whether errors or losses occur during data transmission; Trigger retransmission: When an error or loss is detected, the receiver triggers a retransmission request based on the retransmission strategy; Select retransmission strategy: select timeout retransmission, selective retransmission or fast retransmission strategy according to actual situation; Retransmission control: The sender adjusts the retransmission strategy based on factors such as network conditions and error types, including increasing the number of retransmissions and reducing the size of the data packet; 4) Congestion Control Congestion detection: Determine whether the network is congested by monitoring performance indicators in the network, such as packet loss rate and delay; Adjust the transmission rate: When congestion is detected, gradually reduce the data transmission rate to avoid further aggravating the congestion; Adopt congestion avoidance algorithm: Use slow start, congestion avoidance and other algorithms to gradually increase the transmission rate during data transmission to achieve network stability and efficiency; Monitoring and alarm: Real-time monitoring of data transmission status and performance indicators, and triggering alarm notifications when abnormalities occur, so that problems can be handled in a timely manner.

8. A data collection device based on the Internet of Things (IOT), characterized in that: include: A data acquisition module is used to collect multi-source heterogeneous data from hardware devices and transmit the collected multi-source heterogeneous data to a data format conversion module; A data format conversion module is used to convert the multi-source heterogeneous data into a unified data format that can be recognized by the Internet of Things platform. The data format conversion module first identifies the data type, format and communication protocol of the hardware device, and then converts the multi-source heterogeneous data into a standard data format according to preset conversion rules. Data parsing and encoding technology are used in the conversion process to ensure the accuracy and efficiency of data conversion; The abnormal data processing module is used to process abnormal data that occurs during data collection and data format conversion. The abnormal data processing module first defines the threshold and judgment rules of abnormal data. When the collected data exceeds the threshold or does not meet the judgment rules, it is judged as abnormal data. For abnormal data, the abnormal data processing module triggers an alarm process, including sending an alarm notification to the user, controlling device behavior or adjusting business processes; the abnormal data processing module also records detailed information of the abnormal data for subsequent analysis and processing; The data reliable transmission module is used to transmit the converted data from the Internet of Things platform to the user end. The data reliable transmission module adopts data encryption, data verification, and retransmission mechanisms to ensure the reliability of data transmission. In the process of data transmission, if an error or loss occurs, the data reliable transmission module automatically starts the retransmission mechanism to ensure that the data can be transmitted to the user end completely and accurately.

9. A data collection system based on the Internet of Things (IOT), characterized in that: include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method for collecting Internet of Things (IOT) data according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for collecting data based on the Internet of Things (IOT) described in any one of claims 1 to 7 is implemented.

Images