A method and system for processing equipment data on an industrial internet platform

By introducing a data processing service between the access machine and the server, and utilizing data transformation scripts from Kafka and RabbitMQ systems, the problem of inconsistent device data models was solved, achieving standardized data processing and system stability, and reducing server load and the risk of data loss.

CN115357397BActive Publication Date: 2026-11-17HANYUN TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211055937.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-31
Publication Date
2026-11-17
Estimated Expiration
2042-08-31

AI Technical Summary

Technical Problem

Different devices report different data models, resulting in inconsistent data. Existing technologies make it difficult to achieve standardized processing, and this increases the load on the server side, affecting system stability and increasing the risk of data loss.

Method used

A data processing service is introduced between the access machine and the server. Through the Kafka and RabbitMQ systems, data transformation scripts are used to convert the raw data into standard data, thereby decoupling data processing from uploading.

Benefits of technology

It achieves standardized processing of data models from different devices, reduces the workload of third-party applications, avoids increased server load and data loss, and improves system stability and scalability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115357397B_ABST
    Figure CN115357397B_ABST
Patent Text Reader

Abstract

This invention relates to a device data processing method and system for an industrial internet platform. The method involves establishing a data processing service that runs parallel to the access device's link and can convert raw data into standardized data. This service acquires the raw data uploaded by the access device and performs data processing, ensuring that the access device's link only handles data transmission. By establishing this data processing service, the access device is not responsible for business processing; it only needs to receive and upload data, thus decoupling data uploading and data processing. This allows for the processing of device data—which may have different data models but express the same meaning—into standardized data for analysis by third-party applications, without interrupting the connection between the device and the platform, without data loss, without changing the hardware development logic, and without requiring additional data processing by third-party applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of industrial internet platforms, specifically to a method and system for processing equipment data on an industrial internet platform. Background Technology

[0002] In recent years, with the widespread and in-depth application of information and communication technologies, the integration of manufacturing and the Internet has been continuously deepened, laying a solid foundation for the further development of the industrial Internet and giving rise to numerous industrial Internet platforms.

[0003] In industrial settings, devices operate on the MQTT protocol, acting as MQTT clients and using an industrial internet platform as the MQTT server to enable device access and data reporting. When a device acts as an MQTT client, embedded development is required to hardcode the interaction logic between the device and the platform into the hardware program, which is then soldered onto the device. Therefore, when a device uploads data to the industrial internet platform, it uses a fixed data model, including field names, field types, data length, and units.

[0004] Enterprises often use equipment from different manufacturers, and even within the same manufacturer, equipment models and versions can vary, leading to different data models used for data reporting, despite potentially conveying the same meaning. For example, equipment A might report electricity consumption as {"50001":2000}, while equipment B might report it as {"electric":2.0}. Both represent an electricity consumption of 2 kWh, but the field names, data types, and units differ. Therefore, it's necessary to integrate data representing the same meaning into standardized data sets to facilitate data retrieval by third-party applications.

[0005] To standardize data, it is unrealistic to require different companies to purchase equipment from the same manufacturer, model, and version. Modifying the hardware programs on the equipment to achieve standardization would require a lot of manpower and financial resources. Requiring third-party applications to parse the data and standardize it would involve a lot of repetitive work, and the consistency of the data could not be guaranteed due to different parsing methods. This approach has a low fault tolerance rate and a large workload.

[0006] Some have suggested that data reception, processing, and reporting should be completed on the server side. However, this approach increases the server's load, affects its stability, and can lead to disconnection from devices and data loss. Data loss is also likely when updating data processing functions.

[0007] The above problems are worth solving. Summary of the Invention

[0008] To address the issue of inconsistent data due to different data models but the same reported meaning during data collection, and to avoid increasing server load, affecting system stability, and posing a risk of data loss, this invention provides a simple, flexible, and highly scalable method and system for processing equipment data on an industrial internet platform with minimal impact on users and the platform, thereby achieving standardized data processing.

[0009] The technical solution of this invention is as follows: A method for processing device data on an industrial internet platform involves establishing a data processing service that runs parallel to the link where the access device is located and can convert raw data into standard data. This data processing service acquires the raw data uploaded by the access device and performs data processing, ensuring that the link where the access device is located only performs data transmission. The method includes the following steps: Step 1: The access machine, acting as the producer of the Kafka system, sends raw data to the first topic of the Kafka system; Step 2: Establish a data processing service as the first consumer in the Kafka system to consume raw data from the first topic of the Kafka system; Step 3: The data processing service queries the pre-uploaded data conversion script; Step 4: The data processing service executes the data transformation script corresponding to the original data, transforms the original data into standard data, and sends it to the second topic of the Kafka system for consumption by other consumers.

[0010] According to the present invention based on the above scheme, the access machine is an MQTT server, and different devices send their respective raw data to the MQTT server through the MQTT protocol.

[0011] According to the present invention based on the above-described scheme, step 3 includes the following steps: S301, Pre-upload the data conversion script; S302, Data service query corresponding data conversion script.

[0012] Furthermore, the pre-uploaded data conversion script specifically includes: The data transformation script is sent as a message to one of the message queues in the RabbitMQ system using the console. The data processing service subscribes to the message queue and retrieves the script message. The data processing service parses the script message and updates the script message to the local cache device using a combination of productKey and streamName as the key.

[0013] According to the present invention based on the above scheme, the console also includes a tool for deleting script messages in the RabbitMQ system message queue.

[0014] Furthermore, the data service queries the corresponding data transformation script, specifically as follows: Query script messages in the local cache. Determine if a script message exists that corresponds to the original data. The matching rules include: the key name carried by the original data and the script message are consistent. If it exists, proceed to step 4 and execute the script message; If it does not exist, the raw data is sent directly to the second topic of the Kafka system.

[0015] According to the present invention based on the above scheme, the data conversion script is a JS script, and the method name of the JS script is dataStream_productKey_streamName(byte), where dataStream indicates that the script operates on the data stream, productKey indicates the unique identifier of the product to which the device belongs, streamName indicates the identifier of the data stream model, and byte indicates the binary stream of the original data of the device.

[0016] Furthermore, the method of the data conversion script includes modifying the structure of the original data, modifying the data type, and modifying the value. The return value of the data conversion script is the standard data of the device.

[0017] According to the present invention based on the above scheme, after step 4, it further includes: step 5, a third-party application client acts as the second consumer client of the Kafka system and consumes data from the second topic of the Kafka system.

[0018] This invention also provides a device data processing system for an industrial internet platform, used for running the device data processing method of the industrial internet platform described above, including a Kafka system and a RabbitMQ system, and further comprising: Several devices are used to generate and transmit raw data; The access machine is used to receive raw data from the device and, as the producer of the Kafka system, produce the raw data to the Kafka system. The data processing service, as the first consumer of the Kafka system, is used to consume data from the first topic of the Kafka system and send data to the second topic of the Kafka system; the data processing service also serves as a consumer of the RabbitMQ system, used to retrieve and parse messages from the RabbitMQ system; A local cache is used to store script messages parsed by the data processing service; The console, as the producer end of the RabbitMQ system, is used to send script messages; The third-party application, acting as the second consumer in the Kafka system, is used to consume data from the second topic of the Kafka system.

[0019] According to the above-described solution, the beneficial effects of this invention are as follows: This invention not only enables the direct generation of standardized data, but also solves the problem of different data models caused by different devices. For example, it reduces the workload of third-party applications and avoids data inconsistency caused by the differences in how different applications process data. Furthermore, by establishing a data processing service that runs parallel to the link where the access machine is located and can convert raw data into standard data, the data processing service consumes data and performs logical processing. The server, i.e., the access machine, is not responsible for business processing; it only needs to receive data from the device and then upload it, thus achieving decoupling between data uploading and data processing. Therefore, even if the logic for processing data changes or other functions are expanded, it will only affect the data processing service and will not affect the access machine, causing the device connection to be lost or data to be lost. When updating the data processing service, there will only be a brief consumption delay, and no data will be lost. Attached Figure Description

[0020] Figure 1 This is a data processing flowchart of the present invention. Detailed Implementation

[0021] To better understand the purpose, technical solution, and technical effects of this invention, the invention will be further explained and described below in conjunction with the accompanying drawings and embodiments. It should also be stated that the embodiments described below are only for explaining this invention and are not intended to limit this invention.

[0022] It should be noted that when a component is described as "fixed to" or "set on" another component, it can be directly on the other component or there may be an intervening component. When a component is described as "connected to" another component, it can be directly connected to the other component or there may be an intervening component.

[0023] The terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or specifying the number of technical features. “Several” means two or more, unless otherwise expressly and specifically defined.

[0024] To enable those skilled in the art to more easily understand the technical solutions of the embodiments of the present invention, the technical terms involved in the present invention and the preliminary technical processes will be explained first: Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data from consumers on a website, including: Producer – The producer of messages and data, which can publish messages to a Kafka topic; Consumer – The consumer / consumer of messages and data, who can subscribe to topics and process their published content; Topic – a physical grouping. A topic can be divided into multiple partitions, and each partition is an ordered queue. Each message in a partition is assigned an ordered ID (offset). A message is the basic unit of communication; each producer can publish messages to a topic.

[0025] RabbitMQ is a message-oriented middleware that implements the Advanced Message Queuing Protocol (AMQP). Each producer or consumer must establish a connection with RabbitMQ before sending or consuming messages. This connection is a persistent TCP connection.

[0026] Third-party applications refer to other microservices / application platforms that need to obtain data information from this system.

[0027] like Figure 1 As shown, this invention provides a device data processing method for an industrial internet platform. Specifically, it establishes a data processing service that runs parallel to the link where the access device is located and can convert raw data into standard data. This data processing service acquires the raw data uploaded by the access device and performs data processing, so that the link where the access device is located only performs data transmission. The method includes the following steps: Step 1: The access machine, acting as the producer of the Kafka system, sends raw data to the first topic of the Kafka system (i.e., the origin-data-topic). The access machine is an MQTT server, and different devices send their respective raw data to the MQTT server via the MQTT protocol.

[0028] Step 2: Establish a data processing service as the first consumer of the Kafka system to consume raw data from the first topic of the Kafka system.

[0029] Step 3: The data processing service queries the pre-uploaded data transformation script; specifically, it includes the following steps: Step 301: Upload the data conversion script in advance: The console allows you to send data transformation scripts as messages to one of the message queues in the RabbitMQ system. The data processing service subscribes to this message queue, and once the message arrives in RabbitMQ, it is immediately pushed to the data processing service, which then retrieves the script message. The console also includes a function to delete script messages from the RabbitMQ system message queue. In addition, to prevent malicious attacks using scripts, user-uploaded scripts require administrator approval, or only certain roles are allowed to upload scripts.

[0030] The data processing service parses the script message and updates the script message to the local cache device using a combination of productKey and streamName as the key.

[0031] Step S302: Query the corresponding data transformation script using the data service. Query script messages in the local cache. Since the device also carries productKey and streamName when uploading raw data, it can be determined whether there is a correspondence between the raw data and the script message. The correspondence rules include: the key name carried by the raw data and the script message is consistent. If a script message corresponding to the original data exists, proceed to step 4: the data processing service executes the data transformation script corresponding to the original data, converting the original data into standard data and sending it to the second topic of the Kafka system (i.e., the standard data topic, normal-data-topic), for consumption by other consumers; specifically: The corresponding script message is executed, and the return value is sent to the second topic of the Kafka system, completing the transformation of raw data into standard data. The data transformation script is a JavaScript script, and the method name of the JavaScript script is dataStream_productKey_streamName(byte), where dataStream indicates that the script operates on the data stream, productKey indicates the unique identifier of the product to which the device belongs, streamName indicates the identifier of the data stream model, and byte indicates the binary stream of the device's raw data. The code snippet of the JavaScript script is as follows: function dataStream_ajnROzoQ_demo(byte) { var array = []; for (var i = 0; i <byte.length; i++) { array.push(byte[i]); } var JsonStr = String.fromCharCode.apply(String, array); JsonStr = JsonStr.replace( / ' / g, '"'); var tmp = JSON.parse(JsonStr); for (var val in tmp) { if (val == "a") { tmp["aa"] = tmp[val]; } if (val == "b") { tmp["c"] = tmp[val]; delete tmp[val]; } } return JSON.stringify(tmp); } The original data {"a":100,"b":200} will be converted into the standard data {"a":100,"aa":100,"c":200}. The method of the above data conversion script includes modifying the structure of the original data, modifying the data type, and modifying the value. The return value of the data conversion script is the standard data of the device.

[0032] If no script message corresponding to the original data exists, the original data is directly sent to the second topic of the Kafka system. There are two scenarios: First, the data model of the original data is a standard data model, and other consumers successfully consume it. Second, the data model of the original data is not a standard data model. After other consumers consume the data, they receive an error message, and the server learns that the data transformation script has a missing type. The server can then add a new data transformation script to the database (local cache) using the console.

[0033] Step 5: The third-party application acts as the second consumer in the Kafka system, consuming data from the second topic of the Kafka system.

[0034] In summary, this invention establishes a data processing service that runs parallel to the link where the access device is located and can convert raw data into standard data. The data processing service consumes data and performs logical processing, while the server, i.e., the access device, is not responsible for business processing. It only needs to receive data from the device and then upload it, thus ultimately decoupling data uploading and data processing.

[0035] This invention also provides a device data processing system for an industrial internet platform, used for running the device data processing method of the industrial internet platform described above, including a Kafka system and a RabbitMQ system, and further comprising: Several devices are used to generate and transmit raw data; The access machine is used to receive raw data from the device and, as the producer of the Kafka system, produce the raw data to the Kafka system. The data processing service, as the first consumer of the Kafka system, is used to consume data from the first topic of the Kafka system and send data to the second topic of the Kafka system; the data processing service also serves as a consumer of the RabbitMQ system, used to retrieve and parse messages from the RabbitMQ system; A local cache is used to store script messages parsed by the data processing service; The console, as the producer end of the RabbitMQ system, is used to send script messages; The third-party application, acting as the second consumer in the Kafka system, is used to consume data from the second topic of the Kafka system.

[0036] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0037] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for processing equipment data on an industrial internet platform, characterized in that, Establish a data processing service that runs parallel to the link where the access device is located and can convert raw data into standard data. This data processing service acquires the raw data uploaded by the access device and performs data processing, so that the link where the access device is located only performs data transmission. The process includes the following steps: Step 1: The access machine, acting as the producer of the Kafka system, sends raw data to the first topic of the Kafka system. The raw data message carries a productKey to identify the product to which the device belongs and a streamName to identify the data stream model. Step 2: Establish a data processing service as the first consumer in the Kafka system to consume raw data from the first topic of the Kafka system; Step 3: The data processing service queries the pre-uploaded data transformation script. Specifically, this includes: the data transformation script being pushed to the RabbitMQ system's message queue via the console; the data processing service subscribing to the message queue and receiving the script message actively pushed by RabbitMQ; updating the local cache device with the parsed script message using a combination of productKey and streamName as the key; and the data processing service extracting productKey and streamName from the original data and combining them into a key, thereby querying the local cache device to determine if a data transformation script corresponding to the original data exists. Step 4: If a data transformation script corresponding to the original data exists in the local cache device, the data processing service executes the data transformation script to convert the original data into standard data and sends the standard data to the second topic of the Kafka system for consumption by other consumers. Step 5: If the local cache does not contain a data transformation script corresponding to the original data, the original data is directly sent to the second topic of the Kafka system. There are two scenarios: First, the data model of the original data is a standard data model, and other consumers successfully consume it. Second, the data model of the original data is not a standard data model, and other consumers receive an error message after consumption. The server then learns that the data transformation script has a missing type and adds a new data transformation script to the local cache using the console.

2. The equipment data processing method of the industrial internet platform according to claim 1, characterized in that, The access machine is an MQTT server, and different devices send their raw data to the MQTT server via the MQTT protocol.

3. The equipment data processing method of the industrial internet platform according to claim 1, characterized in that, The console is also used to delete script messages in the RabbitMQ system message queue.

4. The equipment data processing method of the industrial internet platform according to claim 1, characterized in that, The data conversion script method includes modifying the structure of the original data, modifying the data type, and modifying the value. The return value of the data conversion script is the standard data of the device.

5. A device data processing system for an industrial internet platform, used to run the device data processing method for the industrial internet platform as described in any one of claims 1 to 4, characterized in that, Including Kafka and RabbitMQ systems, and also: Several devices are used to generate and transmit raw data; The access machine is used to receive raw data from the device and, as the producer of the Kafka system, produce the raw data to the Kafka system. The data processing service, as the first consumer of the Kafka system, is used to consume data from the first topic of the Kafka system and send data to the second topic of the Kafka system; the data processing service also serves as a consumer of the RabbitMQ system, used to retrieve and parse messages from the RabbitMQ system; A local cache is used to store script messages parsed by the data processing service; The console, as the producer end of the RabbitMQ system, is used to send script messages; The third-party application, acting as the second consumer in the Kafka system, is used to consume data from the second topic of the Kafka system.

Citation Information

Patent Citations

  • Data processing method and device, computer equipment and readable storage medium

    CN114356996A