Half packet sticking processing method based on Internet of Things equipment communication message and netty

Abstract

A half-packet sticky packet processing method based on Internet of Things equipment communication messages and netty comprises the following steps: constructing a transmission control protocol server or client by using a Netty framework to support message packet receiving and sending; inserting a user-defined analyzer into a Netty channel pipeline, wherein the user-defined analyzer is responsible for analyzing the received transmission control protocol data stream; the method comprises the following steps: firstly, searching an initial sequence from a transmission control protocol data stream by a user-defined analyzer, then extracting a message length field, and then reading a corresponding length and a 5-byte message packet structure length according to the length to serve as a complete message packet; and S4, half packet sticking processing: reading the complete message packet, handing over the complete message packet to a next pipeline in the netty framework for processing, moving the read subscript to the last bit of the complete message packet, and waiting for the next message packet to continue to be analyzed. According to the invention, the accuracy and efficiency of unpacking are improved, various types of messages can be quickly and accurately processed, and data errors and loss are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a processing method for half-packet sticking based on Internet of Things device communication messages and Netty. Background Art

[0002] As the Internet of Things (IoT) is developing rapidly, efficient and accurate communication between devices is crucial. In IoT device communication, a large amount of data is transmitted through the Transmission Control Protocol (TCP), and the Netty framework, as a high-performance network application framework, is widely used to build network communication systems. However, in the actual data transmission process, half-packet sticking of messages often occurs, which brings great challenges to the accurate parsing and processing of data.

[0003] At present, some common solutions to the half-packet sticking problem have many shortcomings. For example, the simple fixed-length unpacking strategy cannot adapt to changes in message length, which can easily lead to data loss or incorrect parsing. In addition, the processing efficiency and resource utilization are poor, and cannot meet the real-time and low power consumption requirements of IoT devices. Summary of the invention

[0004] The object of the present invention is to provide a processing method for half-packet sticking based on Internet of Things device communication messages and Netty, which can effectively solve the technical problems mentioned in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for processing half-packet sticky packets based on Internet of Things device communication messages and Netty, comprising the following steps: S1. Use the Netty framework to build a transmission control protocol server or client to support packet reception and transmission; S2. Specified protocol: The packet structure is defined as the start character, software version, data unit length, data unit and check code, and a custom structure is added in the middle of the check code; S3. Insert a custom parser into the channel pipeline of Netty. The custom parser is responsible for parsing the received transmission control protocol data stream. The custom parser first searches for the start sequence from the transmission control protocol data stream, then extracts the message length field, and then reads the corresponding length plus the 5-byte message packet structure length as a complete message packet according to the length. The steps for a custom parser to read a complete message packet include: 1) Read the length of the currently received packet and determine whether the length is greater than the starting character length. If it is less than the starting character length, the execution ends and the parsing continues after the subsequent data packets arrive. If it is greater than or equal to the starting character length, the execution continues. 2) Locate the starting sequence position by linearly searching the packet data and return it. If found, return the specific position index. If not found, return a custom fixed identifier. 3) Determine whether the starting sequence position is a custom fixed identifier. If so, terminate the execution and wait for the subsequent message packet to arrive before continuing the parsing; otherwise, adjust the transmission control protocol data stream reading position to return the specific position index and continue the next parsing; 4) Determine whether the total length of the message packet is greater than the minimum byte length of the message packet specification of 5 bytes. If it is not reached, the execution is terminated and the parsing is continued after the subsequent message packet arrives; if it is reached, the parsing is continued to the next step; 5) Find the data unit length. After reading the total length of the current message packet, first determine whether it is greater than the maximum specified parsing length. Otherwise, continue to execute. If it is, clear the current byte cache and end the execution. Then determine whether the message packet has received the parsed read length. If it has not reached the parsed length, end the execution and wait for the subsequent message packet to arrive before continuing the parsing. If it has reached the parsing length, continue to the next parsing. 6) Analyze the total length byte stream of the message packet and pass it to the subsequent processing code flow; Search the byte cache twice in sequence, traverse all data, and find whether it contains the starting sequence. The first traversal obtains a single cached data in the byte cache. The second traversal compares whether the starting sequence is contained in the length of the starting sequence from the first traversal to the length of the starting sequence. If so, the starting sequence position is returned. If not, continue the first backward traversal. S4, half-package sticking treatment: S41. If the transmission control protocol data stream has ended before reading the specified length of data, or the start sequence is not obtained, the custom parser enters the waiting state, and the Netty input and output event loop will continue to monitor the transmission control protocol connection and wait for the arrival of subsequent data; The custom parser maintains a buffer internally to temporarily store fragments that fail to form a complete packet. When new data arrives, the custom parser will try to splice it with the data in the buffer to form a complete packet. The Netty framework provides an encapsulated byte cache buffer. The byte cache maintains two different indexes: one for reading and one for writing. When reading data from the byte cache, the read index will start incrementing on the index of the read data; similarly, when writing to the byte cache, the write index will also be incremented. When the data in the byte cache has not been read out, new data is received at this time, and the data will continue to append data after the original byte cache, and the write index will be incremented; S42, if a complete packet is not received within a certain time or when the amount of data in the buffer reaches a certain threshold, the data segment is considered invalid and discarded, and the buffer is cleared to prepare for receiving a new packet; When the length of the read packet is greater than the data volume threshold, first check whether the segment of the data packet contains the starting sequence. If it does, set the read index to the starting sequence position and continue to execute downward; if it does not contain it, discard the entire segment of the data packet; S5. Read the complete message packet and hand it over to the next pipeline in the netty framework for processing. Move the read index to the last bit of the complete message packet and wait for the next message packet to continue parsing.

[0006] Preferably, in step S1, the step of constructing a transmission control protocol server or client includes: Step 1: Import the netty project dependency in the dependency configuration file; Step 2: Add the netty server communication port number in the startup configuration file; Step 3: Add the netty server communication configuration class, set the built-in communication transmission mode to non-blocking input and output mode in the Netty server communication configuration class, set the channel option to sequential processing, only process one client connection at the same time, bind the communication port number, add a message processor to process the message, Step 4: Set the channel to blocking waiting, and the message processor will be called when the input and output operations are completed.

[0007] Preferably, in the message packet structure definition of step S2, the message packet is a hexadecimal byte stream data packet, the starting character in the message structure is defined as 1 byte, represented by ASCII code characters, and is an identifier of the current message, the software version is 1 byte, and the valid value range is 1 to 255; the data unit length is 2 bytes, and the valid value range is 0 to 65531, which stores the length of the actual transmitted data; the data unit stores the actual transmitted data, and the length can be defined; the check code adopts the exclusive OR check method, and the check range starts from the first byte of the command unit, and is exclusive ORed with the next byte until the byte before the check code. The check code occupies 1 byte, and the minimum length of the data packet specification is 5 bytes.

[0008] Preferably, in step S3, a custom parser for message decoding is added to the message processor, the custom parser inherits the byte message decoder abstract base class provided by netty, and implements the custom decoding function by rewriting the decoding method. Every time a message packet information is sent by the transmission control protocol, all decoding methods that inherit the message decoder abstract base class are called to decode the message; The steps for a custom parser to parse the received TCP data stream include: The Netty framework provides a byte message decoder class. After inheriting the byte message decoder class, the custom parser decodes and parses the packet data transmitted by the transmission control protocol by overriding the decoding. When the decoding method is called, a byte buffer containing the incoming data and a collection for adding decoded messages will be passed in; the call to the decoding method will be repeated until it is determined that no new packet data is added to the collection, or there are no more bytes to read in the byte buffer; if the collection is not empty, its contents will be passed to the next channel inbound processor in the channel pipeline.

[0009] Preferably, after reading the complete message packet in step S5, error processing and recovery are performed. The custom parser can identify and process error message packets caused by network anomalies and data corruption through the error detection and recovery mechanism, including: The Netty framework provides an inbound message processing class. It creates a message packet receiving processing class that inherits from the abstract base class and processes the decoded message. The detailed processing flow is written in the channel reading method. The abstract base class provides an exception capture method. When an error occurs in program processing, this method will be called. In this method, the parsed data will be re-encoded according to the decoding rules and sent to the byte cache for further processing. Each message maintains a unique ID. When the processing fails a certain number of times, the message is discarded, and a parsing failure data is generated and stored in the database, and a prompt message is sent to the operator.

[0010] Compared with the prior art, the present invention has the following beneficial effects: The present invention improves the accuracy and efficiency of unpacking, can process various types of messages quickly and accurately, and reduce data errors and losses; it has good adaptability, can dynamically adjust the unpacking strategy according to different network environments and message characteristics, and adapt to the diverse communication needs of Internet of Things devices; it optimizes resource utilization, reduces the system's memory consumption and CPU occupancy, and improves the overall performance and stability of the system; it enhances the scalability and maintainability of the system, and facilitates subsequent functional upgrades and optimizations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a work flow chart of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] like Figure 1 As shown, a method for processing half-packet sticky packets based on IoT device communication messages and Netty is characterized in that it includes the following steps: S1. Use Netty framework (Java open source framework): Use Netty framework to build a transmission control protocol server or client to support packet reception and transmission; The steps to build a TCP server or client include: Step 1: Import the netty project dependency in the dependency configuration file (pom.xml); Step 2: Add the netty server communication port number in the startup configuration file (application.yml); Step 3: Add the netty server communication configuration class, set the built-in communication transmission mode to non-blocking input and output mode (NioEventLoopGroup) in the Netty server communication configuration class, set the channel option to sequential processing, only process one client connection at the same time, bind the communication port number, and add a message processor (childHandler) to process messages. Step 4: Set the channel to blocking waiting. When the input and output operations are completed, the message processor will be called; S2. Specified protocol: The packet structure is defined as the start character, software version, data unit length, data unit and check code, and a custom structure is added in the middle of the check code; The message packet is a hexadecimal byte stream data packet. The starting character in the message structure is defined as 1 byte, represented by ASCII code characters, which is an identifier of the current message, such as: 2323; the software version is 1 byte, with a valid value range of 1 to 255; the data unit length is 2 bytes, with a valid value range of 0 to 65531, which stores the length of the actual transmitted data; the data unit can store the actual transmitted data, and the length can be defined; the check code adopts the exclusive or check (BCC) method, and the check range starts from the first byte of the command unit, and is exclusive ored with the next byte until the byte before the check code. The check code occupies 1 byte, and the minimum length of the data packet specification is 5 bytes; S3. Custom parser: Insert a custom parser into Netty's ChannelPipeline. The custom parser is responsible for parsing the received TCP data stream. The custom parser first tries to find the start sequence from the TCP data stream, then extracts the message length field, and then reads the corresponding length plus the 5-byte packet structure length as a complete packet based on the length. A custom parser for message decoding is added to the message processor. This custom parser inherits the ByteToMessageDecoder abstract base class provided by Netty and implements the custom decoding function by rewriting the decoding method. Every time a packet of information is sent by the transmission control protocol, all decoding methods that inherit the message decoder abstract base class will be called to decode the message. S31. The steps of the custom decoder to parse and receive the transmission control protocol data stream include: The Netty framework provides a byte message decoder class. After inheriting the byte message decoder class, the custom parser decodes and parses the packet data transmitted by the transmission control protocol by rewriting the decoding. When the decode (decode()) method is called, a byte buffer containing the incoming data and a collection for adding decoded messages (i.e., a data set for storing messages) will be passed in; the call to the decoding method will be repeated until it is determined that no new packet data is added to the collection, or there are no more bytes to read in the byte buffer; if the collection is not empty, its contents will be passed to the next channel inbound handler (ChannelInboundHandler) in the channel pipeline (ChannelPipeline); S32, the steps of customizing the parser to read the complete message packet include: 1) Read the length of the currently received packet and determine whether the length is greater than the starting character length. If it is less than the starting character length, the execution ends and the parsing continues after the subsequent data packets arrive. If it is greater than or equal to the starting character length, the execution continues. 2) Locate the starting sequence position by linearly searching the packet data and return it. If found, return the specific position index. If not found, return a custom fixed identifier. 3) Determine whether the starting sequence position is a custom fixed identifier. If so, terminate the execution and wait for the subsequent message packet to arrive before continuing the parsing; otherwise, adjust the transmission control protocol data stream reading position to return the specific position index and continue the next parsing; 4) Determine whether the total length of the message packet is greater than the minimum byte length of the message packet specification of 5 bytes. If it is not reached, the execution is terminated and the parsing is continued after the subsequent message packet arrives; if it is reached, the parsing is continued to the next step; 5) Find the data unit length. After reading the total length of the current message packet, first determine whether it is greater than the maximum specified parsing length. Otherwise, continue to execute. If it is, clear the current byte cache and end the execution. Then determine whether the message packet has received the parsed read length. If it has not reached the parsed length, end the execution and wait for the subsequent message packet to arrive before continuing the parsing. If it has reached the parsing length, continue to the next parsing. 6) Analyze the total length byte stream of the message packet and pass it to the subsequent processing code flow; Search the byte cache twice in sequence, traverse all data, and find whether it contains the starting sequence. The first traversal obtains a single cached data in the byte cache. The second traversal compares whether the starting sequence is contained in the length of the starting sequence from the first traversal to the length of the starting sequence. If so, the starting sequence position is returned. If not, continue the first backward traversal. S4, half-package sticking treatment: S41. If the TCP data stream ends (i.e., half a packet) before reading data of the specified length, or the start sequence is not obtained, the custom parser enters the waiting state, and the Netty input and output (IO) event loop continues to monitor the TCP connection and wait for the arrival of subsequent data; The custom parser maintains a buffer internally to temporarily store fragments that fail to form a complete packet. When new data arrives, the custom parser will try to splice it with the data in the buffer to form a complete packet. The Netty framework provides an encapsulated byte cache buffer. The byte cache maintains two different indexes: one for reading and one for writing. When reading data from the byte cache, the read index (readerIndex) will start incrementing at the index of the read data; similarly, when writing to the byte cache, the write index (writerIndex) will also be incremented. When the data in the byte cache has not been read out, if new data is received at this time, the data will continue to append data after the original byte cache, and the write index (writerIndex) will be incremented; S42, if a complete packet is not received within a certain period of time (such as a timeout setting) or when the amount of data in the buffer reaches a certain threshold, the data segment is considered invalid and discarded, and the buffer is cleared to prepare for receiving a new packet; When the length of the packet read is greater than the data volume threshold of 2048, first check whether the packet contains the start sequence. If it does, set the readerIndex to the start sequence position and continue to execute. If it does not, discard the packet. S6, read the complete message packet, hand it over to the next pipeline in the netty framework for processing, move the read index to the last bit of the complete message packet, and wait for the next message packet to continue parsing; S7. Error handling and recovery: The custom parser must also have an error detection and recovery mechanism that can identify and handle error packets caused by network anomalies, data corruption, etc., to ensure the stability and reliability of the system.

[0014] The Netty framework provides an inbound message processing class (abstract base class SimpleChannelInboundHandler), creates a message packet receiving message processing class, inherits from the abstract base class, processes the decoded message, and the detailed processing flow is written in the channel read (channelRead0()) method. The abstract base class provides an exception capture (exceptionCaught()) method. When an error occurs in program processing, this method will be called. In this method, the parsed data will be re-encoded according to the decoding rules and sent to the byte cache for further processing. Each message maintains a unique ID. When the processing fails 3 times, the message is discarded, and a parsing failure data is generated and stored in the database, and a prompt message is sent to the operator.

[0015] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for processing half-packet sticky packets based on IoT device communication messages and Netty, characterized in that: The following steps are involved: S1. Use the Netty framework to build a transmission control protocol server or client to support packet reception and transmission; S2. Specified protocol: The packet structure is defined as the start character, software version, data unit length, data unit and check code, and a custom structure is added in the middle of the check code; S3. Insert a custom parser into the channel pipeline of Netty. The custom parser is responsible for parsing the received transmission control protocol data stream. The custom parser first searches for the start sequence from the transmission control protocol data stream, then extracts the message length field, and then reads the corresponding length plus the 5-byte message packet structure length as a complete message packet according to the length. The steps for a custom parser to read a complete message packet include: 1) Read the length of the currently received packet and determine whether the length is greater than the starting character length. If it is less than the starting character length, the execution ends and the parsing continues after the subsequent data packets arrive. If it is greater than or equal to the starting character length, the execution continues. 2) Locate the starting sequence position by linearly searching the packet data and return it. If found, return the specific position index. If not found, return a custom fixed identifier. 3) Determine whether the starting sequence position is a custom fixed identifier. If so, terminate the execution and wait for the subsequent message packet to arrive before continuing the parsing; otherwise, adjust the transmission control protocol data stream reading position to return the specific position index and continue the next parsing; 4) Determine whether the total length of the message packet is greater than the minimum byte length of the message packet specification of 5 bytes. If it is not reached, the execution is terminated and the parsing is continued after the subsequent message packets arrive; if it is reached, the parsing is continued to the next step; 5) Find the data unit length. After reading the total length of the current message packet, first determine whether it is greater than the maximum specified parsing length. Otherwise, continue to execute. If it is, clear the current byte cache and end the execution. Then determine whether the message packet has received the parsed read length. If it has not reached the parsed length, end the execution and wait for the subsequent message packet to arrive before continuing the parsing. If it has reached the parsing length, continue to the next parsing. 6) Analyze the total length byte stream of the message packet and pass it to the subsequent processing code flow; Search the byte cache twice in sequence, traverse all data, and find whether it contains the starting sequence. The first traversal obtains a single cached data in the byte cache. The second traversal compares whether the starting sequence is contained in the length of the starting sequence from the first traversal to the length of the starting sequence. If so, the starting sequence position is returned. If not, continue the first backward traversal. S4, half-package sticking treatment: S41. If the transmission control protocol data stream has ended before reading the specified length of data, or the start sequence is not obtained, the custom parser enters the waiting state, and the Netty input and output event loop will continue to monitor the transmission control protocol connection and wait for the arrival of subsequent data; The custom parser maintains a buffer internally to temporarily store fragments that fail to form a complete packet. When new data arrives, the custom parser will try to splice it with the data in the buffer to form a complete packet. The Netty framework provides an encapsulated byte cache buffer. The byte cache maintains two different indexes: one for reading and one for writing. When reading data from the byte cache, the read index will start incrementing on the index of the read data; similarly, when writing to the byte cache, the write index will also be incremented. When the data in the byte cache has not been read out, new data is received at this time, and the data will continue to append data after the original byte cache, and the write index will be incremented; S42, if a complete packet is not received within a certain time or when the amount of data in the buffer reaches a certain threshold, the data segment is considered invalid and discarded, and the buffer is cleared to prepare for receiving a new packet; When the length of the read packet is greater than the data volume threshold, first check whether the segment of the data packet contains the starting sequence. If it does, set the read index to the starting sequence position and continue to execute downward; if it does not contain it, discard the entire segment of the data packet; S5. Read the complete message packet and hand it over to the next pipeline in the netty framework for processing. Move the read index to the last bit of the complete message packet and wait for the next message packet to continue parsing.

2. According to claim 1, the method for processing half-packet sticky packets based on the communication message of the Internet of Things device and Netty is characterized in that: In step S1, the steps of constructing a transmission control protocol server or client include: Step 1: Import the netty project dependency in the dependency configuration file; Step 2: Add the netty server communication port number in the startup configuration file; Step 3: Add the netty server communication configuration class, set the built-in communication transmission mode to non-blocking input and output mode in the Netty server communication configuration class, set the channel option to sequential processing, only process one client connection at the same time, bind the communication port number, add a message processor to process the message, Step 4: Set the channel to blocking waiting, and the message processor will be called when the input and output operations are completed.

3. The method for processing half-packet sticky packets based on Internet of Things device communication messages and Netty according to claim 1 is characterized in that: In the message packet structure definition of step S2, the message packet is a hexadecimal byte stream data packet. The starting character in the message structure is defined as 1 byte, represented by ASCII code characters, which is an identifier of the current message. The software version is 1 byte, and the valid value range is 1~255; the data unit length is 2 bytes, and the valid value range is 0~65531, which stores the length of the actual transmitted data; the data unit stores the actual transmitted data, and the length can be defined; the check code adopts the XOR check method, and the check range starts from the first byte of the command unit, and is XORed with the next byte until the byte before the check code. The check code occupies 1 byte, and the minimum length of the data packet specification is 5 bytes.

4. The method for processing half-packet sticky packets based on Internet of Things device communication messages and Netty according to claim 1 is characterized in that: In step S3, a custom parser for message decoding is added to the message processor. The custom parser inherits the byte message decoder abstract base class provided by netty and implements the custom decoding function by rewriting the decoding method. Every time a packet of information is sent by the transmission control protocol, all decoding methods that inherit the message decoder abstract base class are called to decode the message. The steps for a custom parser to parse the received TCP data stream include: The Netty framework provides a byte message decoder class. After inheriting the byte message decoder class, the custom parser decodes and parses the packet data transmitted by the transmission control protocol by rewriting the decoding. When the decoding method is called, a byte buffer containing the incoming data and a collection for adding decoded messages will be passed in; the call to the decoding method will be repeated until it is determined that no new packet data is added to the collection, or there are no more bytes to read in the byte buffer; If the collection is not empty, its contents are passed to the next channel inbound handler in the channel pipeline.

5. The method for processing half-packet sticky packets based on Internet of Things device communication messages and Netty according to claim 1 is characterized in that: After reading the complete message packet in step S5, error processing and recovery are performed. The custom parser can identify and process error message packets caused by network anomalies and data corruption through error detection and recovery mechanisms, including: The Netty framework provides an inbound message processing class. It creates a message packet receiving processing class that inherits from the abstract base class and processes the decoded message. The detailed processing flow is written in the channel reading method. The abstract base class provides an exception capture method. When an error occurs in program processing, this method will be called. In this method, the parsed data will be re-encoded according to the decoding rules and sent to the byte cache for further processing. Each message maintains a unique ID. When the processing fails a certain number of times, the message is discarded, and a parsing failure data is generated and stored in the database, and a prompt message is sent to the operator.

Images