Method for analyzing reported data of equipment based on Protocol Buffer

Abstract

The invention belongs to the technical field of computer data processing, and discloses a method for analyzing data reported by equipment based on Protocol Buffer, which comprises the following steps of: S1, configuring a protobuf file according to service requirements; s2, the sending end generates a message data stream according to the data collected by the equipment; s3, the sending end sends the message data stream to a message server; s4, the receiving end converts the protobuf file into a class function which can be called by the Java code; and S5, the receiving end monitors the message server in real time, and constructs the data stream into a corresponding object through the Java class function of the protobuf after receiving the data stream issued by the message server. When the data collected by the equipment is changed, source codes do not need to be modified, only the protobuf file needs to be redefined, and a new class function is generated. When a large batch of data is processed, the data represented by protobuf can be more efficient, the compressed data is smaller, and the transmission efficiency is improved. Moreover, due to the fact that the data structure is defined in advance, data analysis is more efficient.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data processing, and in particular relates to a method for analyzing device reported data based on Protocol Buffer. Background Art

[0002] At present, with the rapid development of information technology, there is an increasing need for large-scale, efficient and flexible data processing.

[0003] In conventional applications, hard coding is often required to parse data streams. This approach takes longer to process large amounts of data, and often requires code modifications when the data structure changes later. Summary of the Invention

[0004] In order to solve the problems in the related art, the present application provides a method for parsing device reported data based on Protocol Buffer to solve the defects mentioned in the background technology.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention discloses a method for parsing device reported data based on Protocol Buffer, comprising the following steps:

[0007] S1. Configure a protobuf file according to business requirements. The protobuf file contains at least one complete object information, and then convert the protobuf file into a class function callable by C code.

[0008] S2. The sending end generates a message data stream from the data collected by the device through the class function of the protobuf;

[0009] S3. The sending end sends the message data stream to the message server;

[0010] S4. The receiving end converts the configured protobuf file into a class function callable by Java code through the Maven plugin protobuf;

[0011] S5. The receiving end monitors the message server in real time, and after receiving the data stream sent by the message server, constructs the data stream into a corresponding object through the Java class function of the protobuf.

[0012] In a further embodiment, the complete object information includes type, attribute name, sort and output class name.

[0013] In a further embodiment, the protobuf file is converted into a class function callable by C code using a C language code generation tool officially provided by Protocol Buffer.

[0014] In a further embodiment, the sending end sends the message data stream to the message server through a network interface, and the network interface includes but is not limited to an Ethernet interface and a Wi-Fi interface.

[0015] In a further embodiment, the receiving end obtains the data stream sent by the message server in real time by monitoring a specific port of the message server.

[0016] In a further embodiment, when the data collected by the device changes, the source code is not modified, only the protobuf file is redefined, and new class functions callable by C code and class functions callable by Java code are generated.

[0017] In a further embodiment, the message server is a distributed message server, including but not limited to Kafka and RabbitMQ.

[0018] In a further embodiment, when processing large amounts of data, the data represented by Protocol Buffer has a smaller data volume and higher transmission efficiency than data in XML or JSON format.

[0019] In a further embodiment, the receiving end monitors the message server through multi-threading technology to improve the efficiency of data reception and processing.

[0020] In a further embodiment, the sending end encrypts the message data stream before sending it to the message server, and the receiving end decrypts the data stream after receiving it.

[0021] By adopting the above technical solution, compared with the existing technology, the beneficial effects of the technology of this patent are:

[0022] When the device's collected data changes, the present invention eliminates the need to modify the source code; instead, it simply redefines the protobuf file and generates new class functions. When processing large amounts of data, data represented using protobuf is more efficient and can be compressed to a smaller size, improving transmission efficiency. Furthermore, because the data structure is predefined, data parsing is more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0026] Example

[0027] like Figure 1 As shown, a method for parsing device reported data based on Protocol Buffer includes the following steps:

[0028] S1. Configure a protobuf file based on business requirements. The protobuf file contains at least one complete object, including type, attribute name, order, and output class name. The protobuf file is then converted into a class function callable by C code. This conversion is done using the C language code generation tool provided by Protocol Buffer.

[0029] S2. The sender uses protobuf methods to generate a message data stream from the data collected by the device. Before sending the message data stream to the message server, the sender encrypts it. The sender then calls the generated C code callable method to populate the collected data into a DeviceData object and serialize it into a message data stream.

[0030] When the data collected by the device changes, the source code is not modified. Only the protobuf file is redefined, and new class functions callable by C code and Java code are generated.

[0031] When processing large amounts of data, data represented by Protocol Buffer has a smaller data volume and higher transmission efficiency than data in XML or JSON format.

[0032] S3. The sender sends the message data stream to the message server. The sender sends the message data stream to the message server via a network interface. Network interfaces include, but are not limited to, Ethernet interfaces and Wi-Fi interfaces. The message server is a distributed message server, including, but not limited to, Kafka and RabbitMQ.

[0033] S4. The receiving end converts the configured protobuf file into a class function callable by Java code through the Maven plugin protobuf.

[0034] The receiving end decrypts the data stream after receiving it.

[0035] S5. The receiving end monitors the message server in real time. By monitoring a specific port on the message server, the receiving end obtains the data stream sent by the message server in real time. After receiving the data stream from the message server, the receiving end constructs the data stream into a corresponding object using the Java class function of protobuf.

[0036] The receiving end uses multi-threading technology to monitor the message server to improve the efficiency of data reception and processing.

[0037] Specific implementation:

[0038] Ten thousand devices collect data every 10 seconds. Without considering the data transmission time, JSON and Protobuf are compared only by generating data streams, parsing data streams and data length. It is found that JSON takes 126ms to generate data streams with a data length of 106, and 93ms to parse the data stream; while Protobuf takes 49ms to generate data streams with a data length of 34, and 23ms to parse the data stream.

[0039] Performance optimization comparison experiment

[0040] Test environment:

[0041] 1) Number of devices: 100;

[0042] 2) Single data size: 1KB (protobuf) vs 5KB (JSON);

[0043] 3) Transmission bandwidth: 10Mbps;

[0044] Test results table:

[0045]

[0046] Conclusion The PROTOBUF format is significantly superior to JSON in terms of transmission efficiency and parsing performance, and is particularly suitable for high-density data reporting scenarios of IoT devices.

[0047] Those skilled in the art will readily appreciate other embodiments of the present invention upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the appended claims.

[0048] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for parsing device reported data based on Protocol Buffer, characterized in that: The following steps are involved: S1. Configure a protobuf file according to business requirements. The protobuf file contains at least one complete object information. Then, convert the protobuf file into a class function callable by C code. S2. The sending end generates a message data stream from the data collected by the device through the class function of the protobuf; S3. The sending end sends the message data stream to the message server; S4. The receiving end converts the configured protobuf file into a class function callable by Java code through the Maven protobuf plugin; S5. The receiving end monitors the message server in real time, and after receiving the data stream sent by the message server, constructs the data stream into a corresponding object through the Java class function of the protobuf.

2. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The complete object information includes type, attribute name, sort and output class name.

3. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The conversion of the protobuf file into a class function callable by C code uses the C language code generation tool officially provided by Protocol Buffer.

4. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The sending end sends the message data stream to the message server through a network interface, and the network interface includes but is not limited to an Ethernet interface and a Wi-Fi interface.

5. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The receiving end obtains the data stream sent by the message server in real time by monitoring the specific port of the message server.

6. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: When the data collected by the device changes, the source code is not modified. Only the protobuf file is redefined, and new class functions callable by C code and Java code are generated.

7. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The message server is a distributed message server, including but not limited to Kafka and RabbitMQ.

8. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: When processing large amounts of data, the data represented by Protocol Buffer has a smaller data volume and higher transmission efficiency than data in XML or JSON format.

9. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The receiving end monitors the message server through multi-threading technology to improve the efficiency of data reception and processing.

10. The method for analyzing device data reporting based on Protocol Buffer according to claim 1, characterized in that: The sending end encrypts the message data stream before sending it to the message server, and the receiving end decrypts the data stream after receiving it.

Images