Intelligent processing method, device and equipment for Internet of Things data and medium

Abstract

The invention relates to the technical field of Internet of Things science and technology, and discloses an intelligent processing method, device and equipment for Internet of Things data and a medium, and the method comprises the steps: obtaining initial data which are transmitted and collected by different types of equipment to a platform unified interface of an Internet of Things platform; sorting and classifying the initial data, and distributing the classified initial data to different first message queues according to classification rules of classification; after the initial data passes through the calculation engine, calculating the initial data to obtain different efficient data, and storing the different efficient data into different second message queues; and sending the efficient data conforming to the preset alarm calculation rule to a preset large model to obtain an intelligent processing instruction, and pushing the intelligent processing instruction to a task arrangement service for arrangement so as to be read and executed by a corresponding external device.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things technology, and in particular to an intelligent processing method, device, equipment and medium for Internet of Things data. Background Art

[0002] Based on cutting-edge technologies such as the Internet of Things, cloud computing, artificial intelligence, and big data analysis, we have built a powerful integrated platform. This platform integrates core capabilities such as device access, device management, message subscription, message forwarding, and data services (including storage, analysis, filtering, parsing, and integration, etc.). It can not only support the connection and data collection of a large number of devices downward, realize the cloud transmission of the initial data of the devices, but also provide cloud APIs upward, enabling the server to call these APIs through the cloud SDK and send instructions to the device side, thereby realizing remote control. However, a major challenge faced by the current technology is that due to the differences of various devices, the collected metrics are different, which increases the difficulty of computing and processing and the complexity of instruction issuance. Summary of the Invention

[0003] Embodiments of the present invention provide an intelligent processing method, device, computer equipment and medium for Internet of Things data to solve the technical problem that due to the differences of various devices, the collected metrics are different, increasing the difficulty of computing and processing and the complexity of instruction issuance.

[0004] In a first aspect, an intelligent processing method for Internet of Things data is provided, including:

[0005] Obtain the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform;

[0006] Sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification rules;

[0007] After the initial data passes through the computing engine, calculate the initial data to obtain different efficient data and store them in different second message queues;

[0008] Send the efficient data that meets the preset alarm calculation rules to a preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration for corresponding external devices to read and execute.

[0009] In a second aspect, an intelligent processing device for Internet of Things data is provided, including:

[0010] An obtaining module, configured to obtain the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform;

[0011] A classification module, configured to sort and classify initial data, and distribute the classified initial data to different first message queues according to the classification rules of different categories.

[0012] A calculation module, configured to calculate the initial data after the initial data passes through the calculation engine, so as to obtain different efficient data and store them in different second message queues.

[0013] A sending module, configured to send the efficient data that meets the preset alarm calculation rules to a preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration, so as to be read and executed by corresponding external devices.

[0014] Thirdly, a computer device is provided. The device includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus.

[0015] The memory is used to store a computer program.

[0016] The processor is configured to implement the steps of the above-mentioned intelligent processing method for Internet of Things data when executing the program stored on the memory.

[0017] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is characterized in that when the computer program is executed by a processor, the steps of the above-mentioned intelligent processing method for Internet of Things data are implemented.

[0018] In the solutions implemented by the above intelligent processing method, device, computer equipment and storage medium for Internet of Things data, initial data collected by different types of devices can be sent to the unified platform interface of the Internet of Things platform to unify the data format, so as to efficiently collect initial data from different devices, perform unified processing and analysis, and provide support for subsequent applications; sort and classify the initial data, and send the classified initial data to different first message queues according to the classification rules of the categories to store, process in parallel and efficiently schedule the data in a classified manner, ensuring that the system has the capabilities of high throughput and low latency when processing large-scale data; when the initial data passes through the computing engine, calculate the initial data to obtain different efficient data and store them in different second message queues to improve the efficiency, scalability and reliability of data processing; send the efficient data that conforms to the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration, for corresponding external devices to read and execute. Through intelligent processing, classified storage and distributed computing, the system can efficiently process data from multiple devices, ensure quick response and generate high-quality target data. Combining the analysis and intelligent processing of the large model, the system can automatically analyze the device status, predict potential problems and optimize the system performance, reducing manual intervention. Then, through the task orchestration module, the system can automatically execute tasks (such as device adjustment, data processing, etc.) based on the target data, improving the accuracy and efficiency of decision-making execution. Finally, through the unified interface, database rules and message queues, the system can flexibly handle multiple devices and data types, ensuring the efficiency of data management and computing processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is an application environment schematic diagram of the intelligent processing method for Internet of Things data in an embodiment of the present invention;

[0021] Figure 2 is a flowchart of the intelligent processing method for Internet of Things data in an embodiment of the present invention;

[0022] Figure 3 is Figure 2 a flowchart of a specific implementation manner of step S10 in

[0023] Figure 4 is Figure 2Schematic flowchart of a specific implementation manner of step S30 in

[0024] Figure 5 is a schematic structural diagram of an intelligent processing device for Internet of Things data in an embodiment of the present invention;

[0025] Figure 6 is a schematic structural diagram of a computer device in an embodiment of the present invention;

[0026] Figure 7 is another schematic structural diagram of a computer device in an embodiment of the present invention. Specific implementation manner

[0027] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] The intelligent processing method for Internet of Things data provided by the embodiments of the present invention can be applied to an application environment such as Figure 1 For the intelligent processing and management of Internet of Things data, where the client obtains the initial data collected by different types of devices and sends it to the platform unified interface of the Internet of Things platform; sorts and classifies the initial data, and distributes the classified initial data to different first message queues according to the classification category rules; when the initial data passes through the computing engine, the initial data is calculated to obtain different efficient data and stored in different second message queues; the efficient data that meets the preset alarm calculation rules is sent to the preset large model to obtain intelligent processing instructions, and pushed to the task orchestration service for orchestration, for corresponding external devices to read and execute. Through intelligent processing, classified storage, and distributed computing, the system can efficiently process data from multiple devices, ensure quick response and generate high-quality target data. Combining the analysis and intelligent processing of the large model, the system can automatically analyze the device status, predict potential problems, and optimize the system performance, reducing manual intervention. Then, through the task orchestration module, the system can automatically execute tasks (such as device adjustment, data processing, etc.) based on the target data, improving the accuracy and efficiency of decision execution. Finally, through the unified interface, sub-library rules, and message queues, the system can flexibly handle multiple devices and data types, ensuring the efficiency of data management and computing processing. Among them, the client can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0029] Please refer to Figure 2 as shown Figure 2 This is a schematic flowchart of an intelligent processing method for Internet of Things data provided by an embodiment of the present invention, including the following steps:

[0030] S10: Obtain the initial data collected by different types of devices and send it to the platform unified interface of the Internet of Things platform;

[0031] The intelligent processing method for Internet of Things data provided by the present invention can be applied to the unified intelligent management of data in various application scenarios. The unified intelligent management of data is usually realized through a platform that can receive the data collected by devices in real time. The present invention has a unified physical model management and rule engine management mechanism. The open API interface helps the data fusion and interconnection between services, supports the customization of instruction parameters and the scenario linkage mechanism, combines the usage requirements of each business scenario, realizes the intelligent scenario fusion management solution, supports the adaptation of hundreds of IoT protocols, and is equipped with a self-developed Internet of Things gateway to support the unified parsing and collection of multi-source data of devices and the rapid access of devices.

[0032] Specifically, obtaining different types of devices (such as mobile devices, PCs, IoT devices, etc.) and sending the collected initial data through the platform unified interface usually involves the following key steps: device-side data collection and format unification to ensure the unification of data formats of different devices, device identification and data tagging, and then data cleaning and preprocessing; transmission protocol and interface design, selecting a suitable data transmission protocol (HTTP, MQTT, WebSocket, etc.) according to the device characteristics and setting the API interface; platform data reception and storage, receiving data through the unified interface of the platform and using a suitable storage and processing solution for subsequent analysis. This step realizes the efficient collection of initial data from different devices, conducts unified processing and analysis, and provides support for subsequent applications.

[0033] First, different types of devices usually need to collect different kinds of initial data. Therefore, it is necessary to clarify the types of initial data to be collected on each device. For example, for common device types and data types, device types include: mobile devices (such as smartphones, tablets), PC devices (such as desktop computers, laptops), IoT devices (such as sensors, smart homes, industrial devices). Among them, mobile devices (such as smartphones, tablets) mainly collect data types such as device information (model, operating system version), user behavior (clicks, browsing), location data, sensor data (accelerometer, GPS), network status (Wi-Fi, 4G / 5G), etc.; PC devices (such as desktop computers, laptops) mainly collect data types such as device configuration (operating system, hardware information), user behavior, web access data, software usage, network status, etc.; IoT devices (such as sensors, smart homes, industrial devices) mainly collect data types: sensor data (temperature, humidity, pressure, etc.), device status (on / off, fault alarm), geographical location, operation logs, etc.; other devices (such as embedded devices, POS machines, etc.) mainly collect data types such as device operating status, operation data, transmission data, etc.

[0034] The initial data on different devices may have different formats (such as JSON, XML, Protobuf, etc.). Therefore, it is necessary to define a unified data format for the initial data on different devices. Common data formats are: JSON, Protocol Buffers (Protobuf), Avro. Among them, JSON is easy to parse and suitable for multiple devices and languages; Protocol Buffers (Protobuf) is an efficient binary data format and is suitable for devices with limited bandwidth and storage; Avro is another binary serialization format and is suitable for large-scale data processing. Specifically, select one of the above data formats according to the requirements and set the initial data on different devices to the selected data format.

[0035] In order to better identify the data source, it is necessary to add information such as device identifier, data label, and timestamp to the collected initial data. After identification, it can help the IoT platform identify the data source and the time attribute of the data, facilitating subsequent processing and analysis. Among them, the device identifier (device_id) is the unique identifier for each device; set data labels, such as data_type, sensor_type, etc.; identify the type of data, such as temperature sensor, location sensor, etc.; set the timestamp of the data, and the timestamp is used to ensure the timeliness of the data.

[0036] The initial data collected by the device may contain redundant, duplicate, or inconsistent parts, so data cleaning and preprocessing are required. Data cleaning can be performed on the device side or on the platform side. The data cleaning steps include: outlier filtering, null value filling, and data standardization. Outlier filtering is used to remove obviously incorrect data; null value filling is used to fill in missing data; data standardization is used to unify the numerical range and format.

[0037] The initial data collection on the device side is usually completed by embedded software or applications. The following several protocols are used for data transmission: HTTP / HTTPS, MQTT, WebSocket, CoAP. For example, different devices upload data through HTTP / HTTPS. The platform side needs to design a unified interface to receive the initial data from different devices. Specifically, open API interfaces are set up, and the API interfaces are authenticated and authorized through methods such as OAuth and API keys to prevent unauthorized device or malicious data access. If a device has a large amount of initial data to transmit, it supports batch data processing, supports batch data sending to the interface, supports unified parsing and collection of multi-source data of the device, and fast access of the device, reducing communication costs.

[0038] In one embodiment, when there is a connection failure between an external device and the unified platform interface of the Internet of Things platform, if it is detected that the connection is restored, the receiving device receives the initial data set collected during the fault time period, and the initial data set includes the data generated by the device during the fault time period.

[0039] The connection failure between the external device and the unified platform interface of the Internet of Things platform specifically means that problems such as unstable network and excessive data volume may be encountered during the initial data upload process of the device. The Internet of Things platform can take the following measures to ensure the smooth transmission of the initial data: retry mechanism, and the failed transmission requests on the device side can be automatically retried; data caching, the device side can temporarily cache the initial data. If it is detected that the connection is restored, the receiving device receives the initial data set collected during the fault time period, and the initial data set includes the data generated by the device during the fault time period; distributed message queue, using message queues (such as Kafka, RabbitMQ) as data caches to ensure that the initial data is not lost and can be delivered to the Internet of Things platform at the appropriate time.

[0040] After the initial data of the device is uploaded, the IoT platform stores the initial data and performs subsequent processing. Specifically, the initial data of the device is stored through a relational database (such as MySQL) or a NoSQL database (such as MongoDB). In another embodiment, in order to adapt to the storage of large-scale unstructured data, HDFS or a cloud data lake (such as AWS S3) is often used. In another embodiment, a streaming computing framework (such as Apache Flink, Apache Kafka Streams) is used for real-time data processing and analysis.

[0041] The uploaded initial data of the device can be further analyzed and visually displayed: real-time monitoring, real-time data analysis, generating alarms and reports. It is also possible to perform deep learning and pattern recognition on the initial data of the device through a data analysis model to discover potential trends. The initial data of the device is visually displayed through a dashboard (such as Grafana, Tableau) to provide suitable real-time decisions for decision-makers.

[0042] Among them, as Figure 4 shown, in this embodiment, S10 includes:

[0043] S11: Configure the first mapping relationship between the initial data of the device and the platform standard data of the IoT platform, and add the second mapping relationship between non-standard metrics and standard metrics;

[0044] S12: Obtain the native data collected by different types of devices;

[0045] S13: According to the first mapping relationship and the second mapping relationship, perform corresponding processing on the collected native data sent to the unified platform interface of the IoT platform to obtain the initial data that meets the platform standards.

[0046] Regarding S11-13, specifically, the initial data of a device refers to the raw data collected by different types of sensors, modules, or systems when the device is connected to the Internet of Things (IoT) platform. The initial data usually has a certain format and unit, and varies depending on the type and manufacturer of the device. For example, the initial data provided by a temperature sensor may include the temperature value, collection timestamp, device ID, etc. The platform standard data is the data in a unified format defined by the platform, and usually normalizes and standardizes the data of the connected devices. Among them, the data of different types of devices are unified according to the format and unit of the platform standard, so that the data collected by different types of devices can be stored, processed, and analyzed uniformly. The configuration of the first mapping relationship is to establish a connection between the initial data of the device (such as the raw data provided by the device) and the standard data format of the platform (such as the unified data model of the platform). For example: the data field temperature_value of the device is mapped to the platform standard field temperature; the unit provided by the device may be °C, while the platform standard may require it to be °C, so the unit needs to be converted.

[0047] Steps for configuring the first mapping relationship: First, the IoT platform needs to define a standard data model (such as JSON Schema or database model), including fields, data types, units, descriptions, etc.; Second, the device collects raw data through sensors or other means. Immediately afterwards, according to the format of the device data and the platform standard data model, establish the mapping relationship between the fields. For example, the humidity_raw data collected by the device sensor should be converted into the humidity standard field of the IoT platform. Finally, if the units of the device and the platform are different, unit conversion rules (such as the conversion between Celsius and Fahrenheit, the conversion between kilograms and tons, etc.) also need to be configured.

[0048] In adding the second mapping relationship between non-standard indicators and standard indicators, non-standard indicators refer to data that does not conform to the platform standard, which may be generated due to device customization or manufacturer self-definition. Non-standard indicators do not necessarily conform to the unified standard of the platform, but still need to be connected and processed. For example, some devices may provide custom device status, working hours, etc. information, which may not be the standard indicators predefined by the platform. Standard indicators are the indicators defined and standardized by the platform, usually cross-device and data items that can be processed uniformly.

[0049] Method for adding the second mapping relationship between non-standard metrics and standard metrics: First, identify the non-standard data provided by the device. The non-standard data may be fields unique to the device manufacturer. Second, define mapping rules for the non-standard data to set the mapping with the platform standard metrics. For example, the device_uptime (device running duration) of the device can be mapped to the operating_duration standard field of the platform; if the temperature data provided by the device does not conform to the standard data model of the Internet of Things platform (such as the temp_measure field), a mapping relationship temp_measure→temperature can be created for it. Immediately afterwards, the non-standard data can be converted (such as unit conversion, field standardization, etc.) and then be made consistent with the standard data. Finally, integrate the second mapping relationship into the data model of the Internet of Things platform to ensure that the Internet of Things platform can correctly receive and process the non-standard data.

[0050] Before device data collection and transmission, it is first necessary to find the mapping relationship between the device data and the platform standard data. This is usually carried out in the platform background configuration, and the mapping rules can be queried through the database or read from the configuration file. For example, there may be a table or mapping configuration file in the platform database that lists the corresponding relationships between the device data fields and the platform standard fields. According to the first mapping relationship and the second mapping relationship, convert the raw data collected by different types of devices into the standard format required by the platform. This includes at least: converting the field names, handling unit conversions, and if the device data is non-standard data, necessary cleaning and conversion operations also need to be performed.

[0051] S20: Sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification category rules;

[0052] Sort and classify the initial data and distribute it to different first message queues according to the classification rules, aiming to efficiently manage, store, and process the data later. Through this step, classified storage, parallel processing, and efficient scheduling of the data can be achieved, ensuring that the system has high throughput and low latency capabilities when processing large-scale data.

[0053] Among them, the purpose of sorting and classifying the initial data is to preprocess the original data for subsequent storage, analysis, and consumption. The initial data usually comes from different devices or systems, and its format and structure may vary greatly, so unified processing and classification are required. First, clean and preprocess the initial data. Common steps include: removing invalid data (including removing null values, duplicates, error data, etc.); data standardization, formatting the data uniformly (such as unifying data units, date formats, etc.); formatting processing, converting the initial data into a unified data format (such as JSON, XML, etc.) according to requirements for subsequent unified processing; outlier processing, correcting or discarding possible outliers during the acquisition process.

[0054] Secondly, through predefined rules, allocate the sorted initial data to different categories. Classification methods include at least: device type classification, data generated by different devices (such as temperature sensors, humidity sensors, pressure sensors, etc.) can be classified according to device categories; data type classification, classifying the data according to its content, such as environmental data (temperature, humidity, etc.), status data (device operating status, alarm information, etc.), log data, etc.; importance / priority classification of data, classifying the data into high-priority and low-priority based on the real-time nature or business importance of the data; time dimension classification, classifying the data into real-time data and historical data, etc. based on the data collection time. Manage classified data, label and manage metadata for the data to ensure that each piece of data contains sufficient context information for subsequent classification and storage. For example: meta-information such as device ID, data source, data type, timestamp, etc.; data classification labels, such as temperature_data, humidity_data, device_status, to help identify data categories.

[0055] When the data is sorted and classified, the next step is to store this data in a suitable message queue. Message queues (such as Kafka, RabbitMQ, ActiveMQ, etc.) are an asynchronous message passing mechanism that can effectively relieve the pressure of high-concurrency data streams and achieve data decoupling, asynchronous processing, and efficient transmission.

[0056] Different types of message queues have different characteristics, and the appropriate message queue can be selected according to the data transmission requirements and system architecture to store classified data. The types of message queues at least include: Apache Kafka, RabbitMQ, ActiveMQ, Pulsar. Apache Kafka is suitable for high-throughput and large-scale data transmission, and can ensure high availability and reliability of data; RabbitMQ is suitable for scenarios that require high concurrency and low latency, supports multiple message protocols, and is easy to use and integrate; ActiveMQ is suitable for relatively lightweight message passing and is suitable for medium and small-scale data streams; Pulsar is suitable for large-scale, cross-region distributed message passing systems.

[0057] Create different message queues according to the classification of data. For example: an environmental data queue, such as environment_data_queue, stores data related to the environment (such as temperature, humidity, air pressure, etc.). A device status data queue, such as device_status_queue, stores the operating status, alarm information, fault logs, etc. of the device. A real-time data queue, such as real_time_data_queue, stores real-time events or high-frequency data. A historical data queue, such as historical_data_queue, stores historical data accumulated over a long time.

[0058] After the message queue is created, according to the classification of data, use different routing strategies to send the data to the corresponding message queue. The routing of data is usually based on the type, source or other characteristics of the data. For example: if the data belongs to "temperature data", it is sent to environment_data_queue. If the data is the fault status of the device, it is sent to device_status_queue. Real-time data (such as dynamic change data of device sensors) is sent to real_time_data_queue. This routing logic can be implemented through the logical judgment of the application program, the routing function of the message middleware, or by using mechanisms such as topics and subscriptions.

[0059] The messages stored in the message queue usually need to be processed in a certain structured way. The data formats at least include: JSON format: simple and easy to parse, suitable for cross-platform data transmission. Protobuf format: efficient and compact, suitable for large-scale data transmission. Avro format: suitable for data serialization and data transmission.

[0060] To ensure the reliability of data transmission, the following exception handling and confirmation mechanisms need to be set up: persistence, confirmation mechanism, and retry mechanism. Persistence is used to ensure the persistence of messages in the message queue to prevent data loss. The confirmation mechanism is for the consumer to confirm that the message has been correctly consumed to ensure data is not lost. If the message cannot be successfully processed, the retry mechanism can be triggered. The retry mechanism is used to set up a retry mechanism when the message queue transmission fails to ensure that the data can ultimately be processed.

[0061] The message queue also supports asynchronous consumption. Consumers can process different types of messages in parallel to avoid synchronous blocking. For example: The environmental data consumer consumes the data in the environment_data_queue for analysis such as temperature and humidity. The device status consumer consumes the device operation status data in the device_status_queue for status monitoring and alarm processing. The real-time data consumer consumes the real-time event data in the real_time_data_queue for real-time response.

[0062] S30: After the initial data passes through the computing engine, the initial data is calculated to obtain different efficient data and stored in different second message queues;

[0063] Based on different computing engines combined with different message queues to calculate and classify the initial data, aiming to utilize the advantages of distributed computing for data processing and analysis, thereby improving the efficiency, scalability, and reliability of data processing. This step involves selecting appropriate computing engines and message queues to meet the requirements of data flow, calculation, and analysis.

[0064] The initial data has been classified according to business requirements and processing rules and stored in different first message queues. The classified data can be divided into multiple categories, and the classified data is assigned to different first message queues. After the data is stored, stream computing based on the first message queue is performed, and the data in the first message queue is analyzed through real-time or batch computing by the computing engine.

[0065] The purpose of selecting the computing engine is to efficiently process, analyze, and calculate the data obtained from the first message queue. The computing engine at least includes a stream processing engine, a batch processing engine, and a hybrid computing engine. The following are the computing engines and their applicable scenarios:

[0066] Apache Flink is a stream processing engine designed specifically for high-throughput and low-latency data processing. It can support event-driven stream computing, has powerful time windowing and state management capabilities, and supports multiple message queues such as Kafka and RabbitMQ as data sources and data targets. Application scenarios include real-time data processing, real-time analysis, complex event processing (CEP), real-time monitoring and alerting systems, etc.

[0067] Apache Spark is a large-scale data processing engine that supports batch processing and stream processing (through Spark Streaming). It can perform highly optimized batch computing, support large-scale machine learning, graph computing, and SQL queries, and support multiple data sources, including message queues, databases, file systems, etc. Application scenarios include large-scale data analysis, machine learning tasks, big data ETL (extraction, transformation, loading), etc.

[0068] Apache Storm is a distributed real-time computing system designed specifically for low-latency, stream processing tasks. It can support ultra-low-latency real-time computing, has strong fault tolerance, and is suitable for processing high-frequency, high-throughput event streams. Application scenarios include real-time monitoring, online data processing, and event stream analysis, etc.

[0069] Apache Kafka Streams is a stream processing library for Kafka, used to directly process data streams in Kafka. It can closely integrate stream processing and message queues and seamlessly connect with Kafka data storage, and support complex stream processing operations such as filtering, aggregation, windowing, etc. Application scenarios include lightweight stream data processing, mainly used in conjunction with Kafka data streams.

[0070] Different computing engines can be used in combination with the first message queue to achieve stream computing and real-time data processing. The following are the specific steps for achieving efficient data processing based on the combination of different first message queues and computing engines.

[0071] Combination One: Use the combination of Apache Kafka and Apache Flink:

[0072] Apache Kafka is a popular distributed message queue that can handle large amounts of data streams, support high throughput, and horizontal scalability. The computing engine is Apache Flink, which can directly consume data from Apache Kafka for real-time stream computing. Workflow: Devices send classified data (such as temperature, humidity, etc.) to Apache Kafka in real time. Apache Flink consumes messages from Apache Kafka for real-time computing (such as aggregation, filtering, anomaly detection, etc.). The computing results of Apache Flink can be output to another Kafka queue, database, or external system. For example, when monitoring temperature data, Flink can calculate the average temperature of each device in real time, detect whether it exceeds a preset threshold, and generate alarm information.

[0073] Specifically, step S30 includes:

[0074] Use the first message queue as a data buffer. When the classified initial data in the first message queue of the data buffer accumulates to a preset storage value, trigger the execution instruction of the computing task;

[0075] Batch-compute and process the classified initial data to obtain efficient data and store it in different second message queues;

[0076] Store the computing results in a database or output them to the second message queue for further processing by subsequent services.

[0077] Specifically, to balance the data stream and computing tasks and avoid system overload, the first message queue acts as a data buffer, storing the classified data collected from different devices. Among them, as a buffer, the device side pushes the initial classified data into the first message queue. The data is transmitted to the subsequent processing flow through the first message queue. Since the data volume may be extremely large or the arrival frequency is extremely high, the first message queue continuously receives data and stores it in the queue until a certain condition is met (for example, the data volume in the queue reaches a preset threshold). When the data in the queue accumulates to a certain storage volume (such as the number of messages, total size, or time window of the data, etc.), the system triggers a computing task according to the configured rules. For example, when the data volume in the queue reaches 100MB or 10,000 messages, the execution of the computing task is triggered. By using the first message queue as a buffer, the load between the upstream data acquisition system and the downstream computing system can be balanced, avoiding the overload of the computing system caused by data fluctuations. The first message queue decouples data acquisition and computing tasks, ensuring the flexibility and scalability of the system. The system can set the triggering mechanism to be timed (for example, triggering a calculation once an hour) or based on the data volume (for example, triggering when a certain amount of data accumulates). Or, a consumer program periodically monitors the first message queue, checks the message volume in the current queue, and if the threshold condition is met, sends a signal to notify the downstream computing engine to start the computing task.

[0078] To efficiently process the accumulated classified data and perform various analyses and calculations, when the first message queue accumulates enough data volume, the computing engine will start a batch computing task according to the triggering instruction. At the beginning of the computing task, the computing engine (such as Apache Flink, Spark, etc.) consumes the initial classified data from the first message queue. Among them, the computing engine can obtain data by using appropriate consumption methods (such as by batch, by stream, by time window, etc.). When the data volume accumulates to a certain extent, the computing engine performs batch calculations on the data. Batch calculations allow the computing engine to optimize on a set of data, thereby improving computing efficiency and performance. The computing task can include aggregating data, summing, calculating averages, analyzing maximum and minimum values, time series prediction, inference of machine learning models, etc. After the calculation, the data is screened to eliminate abnormal or invalid data. Detect complex patterns or events in the data, such as device anomalies, trend changes, etc.

[0079] To improve computational efficiency, data can be split into multiple small batches for parallel processing. For example, Spark divides data into multiple partitions and performs parallel computations on different nodes in the cluster, thus accelerating task execution. For streaming data, the computing engine usually uses time windows to segment the data for calculation, ensuring the timeliness and accuracy of the calculation. After the calculation, the system will obtain efficient data results, such as: aggregated statistical information (such as the average device utilization rate), the future state of the device predicted based on time series, and anomaly detection results (such as device fault warnings).

[0080] Store the results in a database or push them to a second message queue for further use by subsequent systems. After the computing task is completed, the obtained results need to be stored or further processed. The calculation results can also be stored in a relational database (such as MySQL, PostgreSQL) or a non-relational database (such as MongoDB, Cassandra) for subsequent querying, analysis, and business processing. If it is data that needs to be persisted, such as the historical state of devices, statistical data, etc., it is usually stored in a database. If the data needs to be subjected to more complex analysis, the results can be stored in a data warehouse (such as Amazon Redshift, Google BigQuery) for OLAP queries by business analysts. Suppose that after the calculation, the system finds that the utilization rate of some devices is too high and there may be a failure. The system pushes the calculation results to a second message queue, such as device_maintenance_queue, so that subsequent services can trigger maintenance operations based on the calculation results, facilitating further processing by other subsequent services or systems. For example, the calculation results can be used for real-time monitoring, pushing the calculated device status to the monitoring system for real-time display and alarm; the calculation results can be used for pushing notifications. If the calculation results indicate that a certain device needs maintenance or repair, the warning information can be pushed to another queue for relevant personnel to handle.

[0081] Function of the message queue: By pushing the calculation results to the second message queue again, the computing engine can be decoupled from the subsequent processing modules, making the system flexibly extensible. After the data is output to the second message queue, subsequent consumers can process the data asynchronously. For example, the calculation results can be used to generate reports, update the user interface, trigger business logic, etc.

[0082] Combination Two: Use the combination of RabbitMQ and Apache Spark:

[0083] The message queue selects RabbitMQ. RabbitMQ supports reliable message delivery and is suitable for low-latency asynchronous tasks. The computing engine is Apache Spark. Apache Spark uses batch processing and stream processing methods and can process high-throughput data obtained from RabbitMQ. Workflow: The device sends the classified data (such as device status, logs, etc.) to the second message queue through RabbitMQ; Apache Spark uses Spark Streaming to process the data stream obtained from RabbitMQ for batch analysis or streaming computing; the analysis results can be output to the database, file system, dashboard, or notification system. For example, Apache Spark can consume device status data from RabbitMQ and make predictions based on historical data and machine learning models to evaluate the health status of the device.

[0084] Combination Three: Use the combination of Apache Kafka and Kafka Streams:

[0085] The first message queue selects Apache Kafka. Kafka Streams is a stream processing extension of Apache Kafka and is directly integrated with Apache Kafka, simplifying the development of stream computing. The computing engine is Kafka Streams. Kafka Streams allows you to directly perform stream processing operations in Kafka, such as aggregation, grouping, windowing, etc. Workflow: The device pushes data (such as environmental monitoring data, device status) to Apache Kafka through Apache Kafka; Kafka Streams consumes data from Apache Kafka for real-time computing. For example, aggregating temperature data, detecting abnormal changes in environmental indicators, etc.; the calculation results are directly written back to Apache Kafka, or sent to the downstream system, or stored in the second message queue. For example, Kafka Streams can process the temperature data stream in real time and store the processed aggregation results in Kafka for subsequent analysis.

[0086] Among them, as Figure 4 shown, specifically regarding step S30, it includes:

[0087] S31, Use the first message queue as the data stream input source, and control the computing engine to read the corresponding initial data in real time from different first message queues;

[0088] S32, Control the computing engine to process the read initial data according to the preset calculation rules to obtain the corresponding efficient data and store it in different second message queues.

[0089] Regarding steps S34 - S35, specifically, the first message queue serves as the input source of the data stream, and the computing engine consumes the classified initial data in it in real time for data processing. The specific process is as follows: The message queue serves as the input source of the data stream, and the computing engine consumes the data in real time and processes the data in real time. By transmitting data in real time through the message queue, the computing engine can quickly process the data after it arrives, ensuring real - time performance. The message queue can efficiently process large - scale data streams, while the computing engine ensures high - throughput data processing, ensuring that real - time data streams flow efficiently from various data sources into the computing engine to support the real - time consumption of large - scale data.

[0090] First, the initial data collected by the device is first transmitted through the first message queue (such as Kafka, RabbitMQ, ActiveMQ, etc.). The data flows through the first message queue in order to ensure that the subsequent system can process it efficiently. Before the data enters the first message queue, it has already undergone classification processing. The classified data may include various types of data such as environmental data, device status, and user behavior. Each type of data may be stored in a different queue for subsequent targeted processing.

[0091] Secondly, the computing engine (such as Apache Flink, Apache Spark Streaming, Apache Storm, etc.) acts as a consumer to obtain data from the message queue. The computing engine consumes the data stream in the first message queue in real time and processes it. The computing engine usually consumes the data in the first message queue in a streaming manner. Whenever new data arrives at the first consumption queue, the computing engine will process it immediately. This real - time data stream consumption can achieve low - latency data processing. Among them, in the scenario of Apache Kafka, the computing engine consumes the data in the first message queue in real time through the Kafka consumer API (such as Kafka Streams). In Apache Flink, the data will be consumed and subjected to streaming computing to process each message in real time.

[0092] The computing engine processes the data stream in real time or in batches to generate efficient data results. The specific steps include data cleaning, aggregation, filtering, statistical analysis, etc.: The purpose of data cleaning is to handle the noise and inconsistencies in the data to ensure that clean and accurate data can be used for subsequent analysis and calculations. Data cleaning operations include: deduplication, deleting duplicate data records; filling missing values, filling or interpolating for lost or missing data; standardization, unifying the data into a standard format (such as date format, numerical range, etc.). Through data cleaning, aggregation, filtering, statistical analysis, etc., the computing engine can obtain efficient and structured data results. Data aggregation is to summarize the data to a higher level according to certain rules for analysis or simplified processing. Aggregation operations include: summation, calculating the sum according to dimensions such as devices, time periods, or regions; average value, calculating the average value of a certain time period or device group; maximum / minimum value, obtaining the maximum or minimum value of certain metrics. For example, summing and averaging the sensor data of multiple devices to obtain the total energy consumption or total temperature of the devices. Data filtering is used to remove data that does not meet the conditions and retain data useful for subsequent analysis. Filtering operations include: threshold-based filtering, such as removing data with temperature values exceeding a certain preset range; time-range-based filtering, filtering out data not within the valid time period; outlier filtering, using statistical methods to identify and remove outliers or abnormal values. Statistical analysis is to perform statistical analysis on the processed data to obtain more valuable information. Statistical analysis includes: grouped statistics, grouping the data according to dimensions such as device type, region, time, etc., and performing aggregation statistics (such as counting, average, sum, etc.); trend analysis, analyzing the trend of certain metrics over time (such as temperature, device usage rate, etc.); association analysis, analyzing the correlation between data, such as the relationship between device status and environmental conditions.

[0093] The computing engine analyzes during the process of the data stream and generates statistical information, aggregated data, or warning signals in real time. For example, in the device data stream, by calculating the average value and standard deviation of the real-time temperature, the device status can be monitored in real time; for real-time data streams, the computing engine usually uses the sliding window technique to calculate the data for each time period to ensure the timeliness of data analysis.

[0094] After calculation and processing, the calculation results need to be stored in different second message queues or further transmitted to downstream systems. This ensures the storage, display, or triggering of subsequent operations. The processing results are stored in a database or different second message queues, or transmitted to downstream systems through the second message queue to provide support for subsequent operations and decisions.

[0095] Store the calculation results, including database storage, file storage, and data warehouses. Database storage: The calculation results can be written into relational databases (such as MySQL, PostgreSQL) or NoSQL databases (such as MongoDB, Cassandra). Databases are used for persistent data storage and facilitate subsequent querying and analysis. Relational databases: Suitable for storing structured data and support SQL queries, such as device usage, sensor historical data, etc.; NoSQL databases: Suitable for storing large-scale, flexible unstructured data, such as data in JSON format. File storage: If the calculation results need to be stored in a batch processing manner, the calculation results can be stored in the file system, usually text files or binary files (such as CSV, Parquet format), for subsequent processing or analysis. Data warehouse: If the data is provided for business analysis or decision support systems, the results can be stored in a data warehouse (such as Amazon Redshift, Google BigQuery) for more complex analysis and reporting.

[0096] The calculation results can also be passed to downstream systems through the second message queue for further processing. For example: Monitoring system: If the calculation results show that the temperature of a certain device is abnormal, the results can be pushed to the monitoring system through the second message queue to trigger an alarm. Data push: Push the calculation results to other microservices for the next step of business logic processing. Notification system: If the calculation results need to remind users (such as device maintenance or fault warning), notification messages can be sent through the second message queue.

[0097] The calculation results can be pushed to the real-time notification system through the second message queue to automatically send alarms, push messages, etc., to notify relevant personnel for processing. For example, push the warning results of the device to the real-time processing system through the Kafka message queue to automatically trigger a maintenance request.

[0098] The calculation results can be provided to the data visualization platform through the message queue or database to display the processing results in real time for managers or business personnel to view. The data stream is passed to the front-end dashboard through the message queue to display the device status, environmental changes, statistical data, etc. in real time.

[0099] In this embodiment, the preset calculation rules include standard calculation rules and alarm calculation rules. The high-efficiency data includes first high-efficiency data and second high-efficiency data. The step of the control calculation engine processing the read initial data according to the preset calculation rules to obtain the corresponding high-efficiency data and storing them in different second message queues further includes:

[0100] The control calculation engine processes the read initial data according to the preset standard calculation rules to obtain the first high-efficiency data;

[0101] The control computing engine processes the read initial data according to a preset warning calculation rule to obtain second high-efficiency data;

[0102] Store the obtained first high-efficiency data and second high-efficiency data into different second message queues.

[0103] The preset standard calculation rule is the standard value of the stored device data stream, which is used to determine whether the initial data collected by the device meets the standard; the preset warning calculation rule is the critical point value when the preset data reaches the warning value, which is used to determine whether the initial data collected by the device triggers an alarm. Among them, the control computing engine is respectively controlled to process the read initial data according to the preset standard calculation rule to obtain first high-efficiency data, and the control computing engine processes the read initial data according to the preset warning calculation rule to obtain second high-efficiency data, and stores the obtained first high-efficiency data and second high-efficiency data into different second message queues. The aim is to separate the initial data that meets the preset standard calculation rule from the initial data that meets the preset warning calculation rule, which is convenient for data classification, data statistics and data analysis, and effectively filters out valid data and abnormal data, compares the problem of poor system operation effect caused by data redundancy, and further realizes efficient data results.

[0104] The computing engine performs real-time analysis on the data. For example, it calculates the operating status, average temperature, pressure, etc. of the device. According to the historical data of the device, machine learning model or threshold rule, it detects abnormal changes in the data. Analyze the change trend of the data over time and discover potential failure risks.

[0105] If the computing engine detects that some parameters of the device exceed the set threshold, or detects abnormal behavior (such as too high temperature, too large vibration, etc.), the system will generate a device alarm. The alarm can be: a real-time alarm or a warning. Real-time alarm: Notify the user or relevant system that the device needs maintenance or inspection. Warning: Notify in advance the risk that the device may malfunction.

[0106] Combination three: Use the combination of Apache Kafka and Apache Storm:

[0107] The message queue is Apache Kafka. As a high-throughput data transmission platform, Kafka is used to process a large amount of streaming data. The computing engine is Apache Storm, which is used for low-latency real-time processing of streaming data and is suitable for event-driven processing tasks. Workflow: The data of the device (such as real-time sensor data) is published through Kafka, and Storm consumes the data from Kafka and analyzes the data stream in real time. For example, the health status of the device, alarm data stream, etc. are calculated in real time, and the processed data can be stored back in Kafka or directly sent to the consumer. For example, Storm can analyze the sensor data stream obtained from Kafka in real time for real-time alarm and monitoring.

[0108] By combining different computing engines and message queues, the data processing efficiency can be improved through the following optimization methods: combining batch data processing and stream processing, distributed computing, real-time monitoring and alerting. Combining batch data processing and stream processing: A computing engine (such as Apache Spark) can process both batch data and stream data simultaneously. Batch processing can be used to reduce data processing latency, while stream processing can respond to data in real time. Distributed computing: Distributed computing frameworks such as Apache Flink and Apache Spark can process data in parallel on multiple nodes, reducing computing bottlenecks and improving processing efficiency. Real-time monitoring and alerting: Combining with a stream computing engine, the data stream is monitored in real time, and anomalies (such as device failures, high-temperature warnings, etc.) are quickly detected, and the alerting mechanism is triggered. By combining different computing engines (such as Apache Flink, Apache Spark, Apache Storm, Kafka Streams, etc.) and message queues (such as Kafka, RabbitMQ, etc.), efficient streaming computing of the classified initial data can be achieved. The above computing engines can process real-time data streams, batch data, and high-concurrency scenarios, and output the calculation results to the downstream system. Through streaming computing, distributed processing, and low-latency data processing methods, the system can provide real-time data analysis and decision support under the requirements of high throughput and low latency.

[0109] Of course, when there is a connection failure in the platform unified interface between the external device and the Internet of Things platform, if the restoration of the connection is detected, the initial data set collected by the receiving device during the fault time period is obtained. The initial data set includes the data generated by the device during the fault time period. When an alarm occurs, the classified initial data is sent into the computing engine and the device alarm is calculated; if there is a problem with the device network or for special reasons the data cannot be sent to the large model, the fault data collected during the fault time period is compressed into a file. When the sending channel is restored, the fault data is sent to the large model, and the fault data is processed by hadoop / hive / spark to generate an alarm.

[0110] Under normal circumstances, the data generated by the device will be classified and processed, and then enter the computing engine for real-time processing and analysis. The computing engine generates device alarms based on the classified data. During the data collection stage, the data collected by the device will be classified according to predefined rules, such as classification by device type, sensor type, collection frequency, etc. The classified data is sent to the computing engine through a message queue or directly. The computing engine (such as Apache Flink, Spark Streaming) will process these data streams in real time and generate alarms according to predefined rules (such as thresholds, anomaly detection, trend analysis, etc.).

[0111] When the device encounters network problems or fails to successfully transmit data to the large model due to other reasons (such as device failures, data loss, etc.), measures need to be taken to avoid data loss and supplement this data during subsequent recovery. The device may lose connection with the cloud or the computing platform, resulting in the inability to send the collected data to the computing engine or the large model. Device failures or blocked data transmission channels may also cause data to fail to reach the large model for processing in a timely manner. During the period of data transmission failure, the device will locally save the data during the failure. The data during the failure includes all device status data collected during the device network interruption or other problems. To reduce storage space, the device usually compresses these data into files (for example: compressed CSV files, JSON files, or binary files). The device stores these data in batches and compresses them. The compression process not only reduces storage space but also ensures the integrity of the data during storage and the efficiency during transmission. These compressed files can include information such as the device's temperature, pressure, humidity, operating status, sensor readings, etc.

[0112] When the device's network problem is resolved or the data transmission channel is restored, the device will attempt to transmit the previously unsent failure data to the computing platform (large model) for processing and generate alarms during this process. After the network is restored, the device will actively send the locally saved compressed failure data files to the computing engine or the large model. The data upload process can be: batch upload, uploading the failure data within a certain period of time at once; batch-by-batch upload, splitting the failure data into multiple small batches for upload to reduce the network pressure during upload. The transmission method can be through API interfaces, message queues (such as Kafka), or directly uploading to cloud storage (such as AWS S3, Azure Blob Storage), etc. for data transmission. After receiving the data, the large model will decompress and parse the compressed file. After parsing, the data will be classified and processed according to the device, time, sensor type, etc.

[0113] For data that cannot be sent, the computing engine will compare it with normal data and process it uniformly. This includes: data alignment, synchronizing fault data with real-time data so that there are no gaps in the monitoring data of the equipment; data correction, if there is data missing or abnormal, the data may need to be corrected or filled to ensure the accuracy of subsequent calculations and analysis.

[0114] After processing the fault data, the computing engine will generate new alarms based on data changes or preset rules. For example, the device may have been abnormal during a network outage, but because the data was not uploaded in time, the computing engine could not trigger an alarm in time. After uploading the fault data, the system will recalculate these historical data and generate relevant alarms. Alarm triggering rules, if the temperature, humidity or pressure data of the device exceeds the threshold, the system will generate an alarm; if the device fails or operates abnormally, the system will generate a fault alarm according to the rules of the large model. After the alarm is generated, the system will notify relevant personnel (such as operators, engineers, etc.) in an appropriate manner, such as through SMS, email, application push, dashboard warnings, etc.

[0115] In this embodiment, the method also includes: writing all efficient data in the second message queue into different time series databases according to the library partitioning rules for display by the display module; if an indicator control instruction initiated by the user through the display module is detected, the indicator control instruction is pushed to the task scheduling service for scheduling, so that the corresponding external device can read and execute it.

[0116] First, define the sub-library rules according to the characteristics of the data (such as device type, region, time period, etc.). For example, different types of device data can be stored in different databases, or the data can be sub-library according to time (such as by day or by hour). According to the preset sub-library rules, the efficient data in the second message queue is written to the corresponding time series database. Time series databases (such as InfluxDB, TimescaleDB, OpenTSDB) are good at storing and processing time series data and are suitable for high-frequency data collection and query. Device data storage is allocated to the corresponding database or data table according to the sub-library rules based on different types of device data. For example, temperature sensor data is stored in temperature_db, device status data is stored in status_db, and so on.

[0117] Time series data usually comes with timestamps to ensure that the data is stored in chronological order for subsequent querying and analysis. The data stored in the time series database after sharding is further transmitted to the large model for advanced analysis, prediction, and decision support. The sending methods can be: calling the large model interface through the API, using a message queue (such as Kafka) to stream the data to the large model's processing system, and exporting the data from the time series database periodically through batch processing and sending it to the large model. After receiving the data, the large model performs data fusion, analysis, prediction, etc., and finally generates device health status, warning information, or optimization suggestions, and returns them to the downstream system for processing or display. To ensure that the data is effectively stored in different time series databases according to the classification rules and is transmitted to the large model in a suitable way for further processing and analysis to support more efficient decision-making and management.

[0118] S40: Send the high-efficiency data that meets the preset alarm rules to the preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration for the corresponding external devices to read and execute.

[0119] The IoT platform issues control instructions to the devices according to requirements. These instructions are usually related to the operating status and parameters of the devices, such as temperature regulation, device start / stop, acquisition frequency, etc. These instructions are usually sent to the devices through the platform's API or SDK to ensure that the devices can collect, adjust, or transmit data according to requirements. The platform performs intelligent processing based on the issued device instructions and the data feedback from the devices. This includes: fusing the device data with the data from other systems or platforms to obtain a more comprehensive view. Then using machine learning, prediction models, etc. to analyze the data to automatically identify patterns, trends, anomalies, etc. Finally, according to the analysis results, adjust the working status or acquisition strategy of the devices to optimize the system performance.

[0120] Among them, the task orchestration module is responsible for generating task instructions according to the predetermined business requirements, device types, and operating strategies. The task instructions usually define the operations that the devices need to perform, such as the frequency of data acquisition, the monitored metrics, the control commands that need to be executed, etc. The task orchestration module generates corresponding task instructions according to the current business logic and device status. For example, if the device needs to collect temperature data once a minute, the task orchestration module will generate a task instruction to set the frequency of the device to collect temperature. The task orchestration module issues the generated task instructions to each device through the unified platform interface of the IoT platform or the second message queue. The task instructions include the specific details of the device operations, for example: the device needs to collect specific target data (such as temperature, humidity, pressure, etc.) at a frequency of once a minute. The issuance of the task instructions is usually through network communication protocols such as HTTP, MQTT, or other IoT protocols. The platform will ensure that the task instructions are reliably delivered to each device.

[0121] After the device receives the task instruction, it starts to execute the task at the frequency set by the instruction (such as per minute) and periodically collects the required target data. The target data may include the device status, sensor data, environmental conditions, etc. The device will actively collect data at the specified time interval according to the collection period set by the instruction (for example, once per minute). After each data collection, the device will upload the data to the cloud or local server through the platform interface for subsequent processing and analysis.

[0122] During the task execution process, the device will regularly feedback the execution status to the platform, such as whether the task is successfully executed and whether a failure occurs. The feedback execution status helps the platform with task management and fault detection. According to the situation feedback by the device, the IoT platform may adjust the task scheduling strategy, such as adjusting the collection frequency, modifying the task content, or rescheduling the task when the device fails.

[0123] Specifically, step S40 includes: controlling a preset large model to analyze the high-efficiency data that meets the preset alarm calculation rules based on the fault data processed by historical indicators and the currently set repair rules, so as to generate corresponding fault handling measures;

[0124] Generate corresponding intelligent processing instructions according to the generated fault handling measures and push them to the task scheduling service for scheduling, so that external devices can read the task scheduling instructions corresponding to the device at a preset fixed frequency.

[0125] After the device fails, the intelligent large model analyzes the high-efficiency data that meets the preset alarm calculation rules based on the fault data processed by historical indicators or the currently set repair rules, generates the fault handling measures for the device according to the analysis of the high-efficiency data that meets the preset alarm calculation rules, and thus automatically issues a repair instruction to achieve the function of automatic repair. For example: when the platform obtains that a gas stove in a certain family is in the on state and also detects that there is no one at home or the people are resting, it calculates a gas-on alarm according to our calculation rules. The intelligent platform of the platform automatically issues an operation to turn off the gas stove or close the gas valve according to the experience and rules of historical alarm repair.

[0126] For enterprises with standard repair processes, it supports automatically generating itsm work orders or automatically sending them to work groups and emails, etc. after the device fails, so as to notify specific users so that they can accurately and quickly discover and solve problems.

[0127] When receiving intelligent processing instructions, the intelligent processing result obtains target data. After intelligent processing, the platform can obtain the expected target data, which usually includes: optimized device status, such as energy-saving mode, performance improvement, fault warning, etc.; analysis results, such as device health status, predicted maintenance time, trend analysis, etc. The target data can be transmitted to the decision-making system, business applications or visualization platform to support subsequent decision-making or operations.

[0128] By issuing device control instructions through the platform and combining the real-time data of the device with intelligent processing algorithms, the platform can perform intelligent analysis on high-efficiency data and finally obtain target data for optimizing device management, warning and decision-making.

[0129] Among them, "intelligent processing" generally refers to the use of advanced data analysis technologies (such as machine learning, artificial intelligence algorithms, etc.) to deeply analyze and process data. The process of obtaining target data based on the results of intelligent processing is a data-driven closed-loop system, from device data collection, intelligent analysis, decision optimization, to finally generating target data for improving equipment management, operation and maintenance efficiency, production optimization, etc. The target data not only helps in the real-time monitoring of devices and systems, but also improves the overall system intelligence level through prediction, early warning, and automatic optimization. The processing process at least includes: data cleaning and preprocessing, data fusion, feature extraction, model training and prediction. Data cleaning and preprocessing specifically involves removing outliers, incorrect data, or incomplete data. For example, if a temperature sensor fails, it may result in some invalid temperature data, and the cleaning process will remove this data. Handling missing data, such as using interpolation methods to fill in missing temperature values. Data fusion specifically means that data from multiple sensors or devices will be fused to obtain a unified and more accurate view. For example, after fusing temperature and humidity data, device status data, and energy consumption data, the overall health status of device operation can be analyzed. Upstream and downstream data fusion: Sometimes, the platform needs to fuse data from different devices or systems to generate more comprehensive target data. For example, combining device data with environmental data to form a more accurate workload analysis. Feature extraction specifically means that in intelligent algorithms, feature extraction is a very crucial step, which transforms the original data into discriminative features for the model to learn. For example, extracting statistical features such as frequency, amplitude, average value, and standard deviation from the original sensor data for device fault detection or early warning. For time series data (such as device operation data, sensor data, etc.), time-related features such as trend changes and periodic fluctuations often need to be extracted. Model training and prediction specifically means that based on the historical data of the device, the platform can use methods such as supervised learning (such as regression analysis, classification algorithms), unsupervised learning (such as clustering algorithms), or deep learning (such as neural networks) for model training. For example, using historical device sensor data to train a model to predict possible device failures (such as overheating, abnormal device vibration). Using machine learning models to identify the operating state of the device and determine whether the device is in a normal, to-be-repaired, to-be-maintained, or faulty state. Based on past data and current status, predicting future device requirements or maintenance needs (such as predicting the remaining life of the device or the load change trend). In addition to machine learning models, the platform can also adopt technologies such as rule engines and expert systems to dynamically adjust the operating state of the device and optimize intelligent algorithms through set rules. Optimizing intelligent algorithms, for example, based on the current state of the device and external environmental conditions, adjusting the operating parameters of the device (such as temperature setting, energy consumption optimization, etc.).

[0130] The generated target data will be output to different business systems, management platforms, user terminals, or visualization dashboards for relevant personnel to use. The target data can be transmitted to other systems or platforms through API interfaces, message queues, data streams, etc. The target data is usually displayed on a visualization panel for decision-makers to view. For example, device monitoring panels, production monitoring dashboards, etc. Some platforms will automatically execute decisions based on the target data, such as automatically adjusting device settings, triggering maintenance notifications, etc.

[0131] It can be seen that in the above solution, for the unified intelligent management of Internet of Things data, first, different types of devices are used to send the collected initial data to the platform unified interface of the Internet of Things platform; the initial data is sorted and classified, and the classified initial data is distributed to different first message queues according to the classification category rules; after the initial data passes through the computing engine, the initial data is calculated to obtain different efficient data and stored in different second message queues; the efficient data that meets the preset alarm calculation rules is sent to the preset large model to obtain intelligent processing instructions and pushed to the task orchestration service for orchestration for corresponding external devices to read and execute. Through intelligent processing, classified storage, and distributed computing, the system can efficiently process data from multiple devices, ensure quick response and generate high-quality target data. Combining the analysis of the large model and the issuance of intelligent instructions, the system can automatically analyze the device status, predict potential problems, and optimize system performance, reducing manual intervention. Then, through the task orchestration module, the system can automatically execute tasks (such as device adjustment, data processing, etc.) based on the target data, improving the accuracy and efficiency of decision execution. Finally, through the unified interface, sub-library rules, and message queues, the system can flexibly handle multiple devices and data types, ensuring the efficiency of data management and computing processing.

[0132] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0133] In one embodiment, an intelligent processing device for Internet of Things data is provided. The intelligent processing device for Internet of Things data corresponds one-to-one with the intelligent processing method for Internet of Things data in the above embodiment. As Figure 5 shown, the intelligent processing device for Internet of Things data includes an acquisition module 101, a classification module 102, a calculation module 103, and a sending module 104. The detailed description of each functional module is as follows:

[0134] The acquisition module 101 is used to acquire the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform;

[0135] The classification module 102 is used to sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification category rules;

[0136] The calculation module 103 is used to calculate the initial data after the initial data passes through the calculation engine, so as to obtain different efficient data and store them in different second message queues;

[0137] The sending module 104 is used to send the efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration, so that the corresponding external devices can read and execute.

[0138] In one embodiment, the calculation module 103 is specifically used for:

[0139] Taking the first message queue as the data stream input source, controlling the calculation engine to read the corresponding initial data from different first message queues in real time;

[0140] Controlling the calculation engine to process the read initial data according to the preset calculation rules to obtain the corresponding efficient data and store them in different second message queues.

[0141] In one embodiment, the calculation module 103 is specifically further used for:

[0142] Controlling the calculation engine to process the read initial data according to the preset standard calculation rules to obtain the first efficient data;

[0143] Controlling the calculation engine to process the read initial data according to the preset alarm calculation rules to obtain the second efficient data;

[0144] Storing the obtained first efficient data and second efficient data in different second message queues.

[0145] In one embodiment, the acquisition module 101 is specifically further used for:

[0146] Configuring the first mapping relationship between the initial data of the device and the platform standard data of the Internet of Things platform, and adding the second mapping relationship between the non-standard indicators and the standard indicators;

[0147] Obtaining the native data collected by different types of devices;

[0148] According to the first mapping relationship and the second mapping relationship, performing corresponding processing on the collected native data sent to the platform unified interface of the Internet of Things platform to obtain the initial data that meets the platform standards.

[0149] In one embodiment, the sending module 104 is specifically used for:

[0150] Control the preset large model to analyze the efficient data that meets the preset alarm calculation rules based on the fault data processed according to historical metrics and the currently set repair rules, so as to generate corresponding fault handling means;

[0151] Generate corresponding intelligent processing instructions according to the generated fault handling means, and push them to the task orchestration service for orchestration, so that external devices can read the task orchestration instructions corresponding to the device at a preset fixed frequency.

[0152] In one embodiment, the device further includes:

[0153] The sub-database module is used to write the efficient data in all the second message queues into different time series databases according to the sub-database rules for the display module to display; if it detects an index control instruction initiated by the user through the display module, it will push the index control instruction to the task orchestration service for orchestration, so that the corresponding external device can read and execute it.

[0154] In one embodiment, the device further includes:

[0155] The fault module is used to, when there is a connection fault between an external device and the platform unified interface of the Internet of Things platform, if it detects the restoration of the connection, receive the initial data set collected by the device during the fault time period, and the initial data set includes the data generated by the device during the fault time period.

[0156] The present invention provides an intelligent processing device for Internet of Things data. First, it obtains the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform; sorts and classifies the initial data, and distributes the classified initial data to different first message queues according to the classification category rules; after the initial data passes through the calculation engine, it calculates the initial data to obtain different efficient data and stores them in different second message queues; sends the efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and pushes them to the task orchestration service for orchestration, so that the corresponding external devices can read and execute them. Through intelligent processing, classified storage, and distributed computing, the system can efficiently process data from multiple devices, ensure quick response and generate high-quality target data. Combining the analysis of the large model and the issuance of intelligent instructions, the system can automatically analyze the device status, predict potential problems, and optimize the system performance, reducing manual intervention. Then, through the task orchestration module, the system can automatically execute tasks (such as device adjustment, data processing, etc.) based on the target data, improving the accuracy and efficiency of decision-making execution. Finally, through the unified interface, sub-database rules, and message queues, the system can flexibly handle multiple devices and data types, ensuring the efficiency of data management and calculation processing.

[0157] For the specific limitations of the intelligent processing device for Internet of Things data, reference can be made to the limitations of the intelligent processing method for Internet of Things data in the foregoing text, which will not be elaborated herein. Each module in the above-mentioned intelligent processing device for Internet of Things data can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor in the computer device in hardware form or independent of the processor, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0158] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities and execute the programs stored on the memory. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory, and the memory is used to store computer programs. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an intelligent processing method for Internet of Things data.

[0159] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as Figure 7 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of an intelligent processing method for Internet of Things data

[0160] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0161] Obtain the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform;

[0162] Sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification rules of the categories.

[0163] After the initial data passes through the computing engine, calculate the initial data to obtain different efficient data and store them in different second message queues.

[0164] Send the efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration, for the corresponding external devices to read and execute.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0166] Obtain the initial data collected by different types of devices and sent to the platform unified interface of the Internet of Things platform.

[0167] Sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification rules of the categories.

[0168] After the initial data passes through the computing engine, calculate the initial data to obtain different efficient data and store them in different second message queues.

[0169] Send the efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and push them to the task orchestration service for orchestration, for the corresponding external devices to read and execute.

[0170] It should be noted that for the functions or steps that the above computer-readable storage medium or computer device can achieve, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.

[0171] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0173] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An intelligent processing method for Internet of Things data, characterized in that: The method is applied to an Internet of Things platform, and the method comprises: Obtain the initial data collected by different types of devices and send it to the unified interface of the IoT platform; Arrange and classify the initial data, and distribute the classified initial data to different first message queues according to the classification rules; After the initial data passes through the calculation engine, the initial data is calculated to obtain different efficient data and stored in different second message queues; Efficient data that meets the preset alarm calculation rules is sent to the preset large model to obtain intelligent processing instructions, and pushed to the task scheduling service for scheduling so that the corresponding external devices can read and execute.

2. The intelligent processing method for Internet of Things data according to claim 1, characterized in that: After the initial data passes through the calculation engine, the steps of calculating the initial data to obtain different efficient data and storing them in different second message queues include: Using the first message queue as a data stream input source, controlling the computing engine to read corresponding initial data from different first message queues in real time; The control calculation engine processes the read initial data according to a preset calculation rule to obtain corresponding efficient data and store the data in different second message queues.

3. The intelligent processing method of Internet of Things data according to claim 2, characterized in that: The preset calculation rules include standard calculation rules and alarm calculation rules, the efficient data include first efficient data and second efficient data, and the control calculation engine processes the read initial data according to the preset calculation rules to obtain corresponding efficient data and store them in different second message queues, and the step also includes: Controlling the calculation engine to process the read initial data according to a preset standard calculation rule to obtain first efficient data; The control calculation engine processes the read initial data according to a preset alarm calculation rule to obtain second efficient data; The obtained first high-efficiency data and second high-efficiency data are stored in different second message queues.

4. The intelligent processing method for Internet of Things data according to claim 3, characterized in that: The method further comprises: According to the database partitioning rules, all the efficient data in the second message queue are written into different time series databases for display by the display module; If an indicator control instruction initiated by the user through the display module is detected, the indicator control instruction is pushed to the task scheduling service for scheduling, so as to be read and executed by the corresponding external device.

5. The intelligent processing method for Internet of Things data according to claim 1, characterized in that: The method further comprises: When there is a connection failure between an external device and the platform unified interface of the Internet of Things platform, if the connection is restored, an initial data set collected by the receiving device during the failure time period is received, and the initial data set includes data generated by the device during the failure time period.

6. The intelligent processing method of Internet of Things data according to claim 1, characterized in that: The step of obtaining different types of devices to send collected initial data to the platform unified interface of the Internet of Things platform includes: A first mapping relationship between the initial data of the configuration device and the platform standard data of the Internet of Things platform, and a second mapping relationship between non-standard indicators and standard indicators are added; Get the raw data collected by different types of devices; According to the first mapping relationship and the second mapping relationship, the native data collected and sent by the platform unified interface of the Internet of Things platform is processed accordingly to obtain initial data that meets the platform standard.

7. The intelligent processing method for Internet of Things data according to claim 1, characterized in that: The step of sending the efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and pushing it to the task scheduling service for scheduling, so as to be read and executed by the corresponding external device, also includes: The control preset big model analyzes the efficient data that meets the preset alarm calculation rules according to the fault data processed by historical indicators and the currently set repair rules to generate corresponding fault handling measures; Corresponding intelligent processing instructions are generated according to the generated fault handling means, and pushed to the task scheduling service for scheduling, so that the external device can read the task scheduling instructions corresponding to the device through a preset fixed frequency.

8. An intelligent processing device for Internet of Things data, characterized in that: The acquisition module is used to acquire the initial data collected by different types of devices and sent to the unified interface of the IoT platform; A classification module is used to sort and classify the initial data, and distribute the classified initial data to different first message queues according to the classification rules; A calculation module, used for calculating the initial data after the initial data passes through the calculation engine to obtain different efficient data and store them in different second message queues; The sending module is used to send efficient data that meets the preset alarm calculation rules to the preset large model to obtain intelligent processing instructions, and push them to the task scheduling service for scheduling so that the corresponding external devices can read and execute them.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the intelligent processing method of Internet of Things data as described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent processing method for Internet of Things data according to any one of claims 1 to 7 are implemented.

Images