Continuous high-speed packet receiving and unpacking method and communication device for splicing broken packets using a single buffer

Abstract

The present invention discloses a continuous extremely fast packet receiving and unpacking method and communication equipment for splicing broken packets with a single buffer, and provides a general unpacking and broken packet splicing method applicable to "any protocol message with a packet start identifier field in the packet header". The present invention uses only a single thread and a single receiving buffer, does not open up any auxiliary space, and reduces the time overhead of the packet starting point positioning process by searching for the starting identifier of each packet in the area to be unpacked through the KMP string matching algorithm, reduces the scope of the pre-unpacking cross-boundary check of whether the remaining part is long enough for a packet by dividing the possible broken packet area, quickly identifies the broken packet by the cross-boundary check in this possible broken packet area, and quickly splices the broken packet by transporting the broken packet with a length of N to the buffer first address buf for storage and using buf+N as the cache starting point when receiving the next round of packets. The present invention meets the communication industry's requirements for low time and space complexity, versatility, etc. for unpacking, and overcomes the communication unpacking technical difficulties caused by sticky packets, broken packets, etc.

Description

Technical Field

[0001] The present invention discloses a method for continuously and extremely quickly receiving and unpacking packets by splicing broken packets with a single buffer and a communication device. The method is applicable to the Internet field, the big data field and the network communication system research and development field that continuously receive and unpack packets through network sockets, and is also applicable to all wired or wireless communication technical fields that need to receive and parse Ethernet messages, and is also applicable to the measurement and control field that needs to continuously collect and parse data messages of a device under test and comprehensively analyze various performance indicators of the device under test, and is also applicable to the embedded industrial control field and the electronic information field that need to interact with a host computer and receive and parse command packets sent by the host computer in any application layer interaction protocol format at any time, and the streaming media field that needs to continuously receive and decode audio and video frames or video frames in real time. The present invention specifically relates to a method for continuously and extremely quickly receiving and unpacking packets by splicing broken packets with a single buffer and a communication device. Background Art

[0002] Packet receiving and unpacking is one of the most common requirements in all communication fields. A packet is a protocol packet or a message. It is essentially the basic transmission unit of the protocol, namely PDU, which is composed of payload data plus a protocol header and a packet trailer. The process of adding a packet header and a packet trailer to form a basic transmission unit is called packet assembly or packet formation. After the receiver receives a data stream of a specified length, it restores the boundaries of each packet from the buffer and selects to disassemble and use each field including the payload data according to the needs. This process is called unpacking. Most protocol headers contain a start identifier field that serves as a start identifier. The headers of some commonly used protocols also have a field indicating the payload data length or the total packet length. The end of the packet is usually a checksum field, which is used to help the receiver check whether the checksum result is consistent with the transmitter after receiving the packet to ensure the correct reception of the packet, and avoid erroneous packet reception or unpacking due to transmission errors such as byte loss during transmission or the inaccuracy of the unpacking algorithm itself, such as CRC cyclic redundancy check, sum check, parity check, etc.; the receiver cannot predict the number, length, and boundaries of each protocol packet to be transmitted in advance, so it generally specifies the amount of received data to quickly receive and cache each protocol packet transmitted in the form of a stream. The phenomenon that multiple packets are received and cached at the same time and stuck together is called sticky packets; in addition, because the amount of received data specified by the communication equipment may not be exactly equal to the amount of data transmitted in sequence The sum of the lengths of one or more packets in each protocol packet makes it very likely that the end of the cache is a packet that has not been completely received and has a missing format, that is, a broken packet. The missing part of the packet that has not been received will be stored at the starting point of the cache address specified by the communication device when it receives a sufficiently long data stream next time; in order to avoid the transmission of a single packet that is too long, most communication protocols specify the maximum transmission unit MTU of the protocol packet. When the total packet length exceeds the MTU due to the excessive length of the payload data during the packet assembly process, it will be divided into multiple packets smaller than the MTU according to the specific protocol rules, and the payload will be divided into multiple parts as the payload fields of each packet. This process is called packetization or fragmentation. Taking Ethernet frames as an example, when the Ethernet frame length exceeds the corresponding MTU, that is, 1500 bytes, the IP data packet will be fragmented according to the rules at the network layer, and then the fragments will be transmitted in sequence through the network cable. Some protocols also specify the minimum transmission unit, that is, the minimum packet length limit, to avoid wasting channel resources by using protocol packets when the payload data is much smaller than the header or tail length. Therefore, even without packet splitting, the length of each protocol packet transmitted can be any length between the minimum packet length limit and the maximum transmission unit, that is, the length of each packet is different, which increases the difficulty for the receiver to locate the boundaries of each packet, that is, the starting point and the end point during the unpacking process. The above-mentioned phenomena of packet sticking, packet splitting, packet breakage and different packet lengths are the root causes of the technical difficulties in unpacking. Among them, the broken packet is the most difficult part for the receiver to deal with in the process of continuous first packet unpacking, and it is the most troublesome unpacking problem in the field of communication technology.Because the program code for continuous unpacking is often a loop, each round of the loop is a round of unpacking process, and if the entire receiving buffer is checked for out-of-bounds access to avoid out-of-bounds access during unpacking, it means that in the process of moving the traversal pointer to match the packet start identifier, it is necessary to check byte by byte to determine whether the remaining unpacked part starting from this point is long enough for a packet, and this is only one round of the loop body. This large number of redundant judgments greatly increases the time complexity of the continuous unpacking program. This results in that there are very few general unpacking methods that meet low time complexity and low space complexity and are applicable to any layer of the OSI seven-layer network model framework structure. The scope of unpacking is extremely wide, which can be TCP / IP datagrams sent by network cards in Ethernet or forwarded by routers, command packets or handshake signals received from host computers through network cables, serial ports, spi, IIC or any other data buses, or any type of data packets continuously collected and captured for monitoring the object under test, or any audio and video frames in the field of streaming media that need to be efficiently and quickly received and decoded, or private protocol data packets received directly from clients through stream sockets. Some protocols even allow the start identifier field of the packet header to be repeated with the payload part, which leads to some unpacking methods mistaking the payload as the packet starting point during the matching and positioning packet starting point during unpacking and unpacking incorrectly. In addition, the communication transmission process itself has a certain bit error rate. If the unpacking method also increases the error rate of data reception, this is intolerable in the field of high-precision communication technology. Therefore, the existing field of communication technology is constantly seeking unpacking methods with low time complexity, space complexity, no increase in data reception error rate and multi-protocol versatility. Summary of the invention

[0003] The present invention overcomes the above-mentioned technical difficulties of existing communication unpacking, discloses a continuous and extremely fast packet receiving and unpacking method and communication equipment for splicing broken packets using a single buffer, and provides a universal continuous unpacking and broken packet identification and splicing method applicable to "any protocol message with a packet start identifier field in the packet header". The present invention uses only a single-threaded, single storage space continuous receiving buffer, does not open any auxiliary space, searches for the start identifier of each packet in the to-be-unpacked area through the KMP string matching algorithm to reduce the time complexity of the packet starting point positioning process, divides the possible packet break area, reduces the scope of the pre-unpacking cross-boundary check on whether the remaining part is long enough for one packet, and quickly identifies the broken packet by performing the cross-boundary check in the possible packet break area, and quickly splices the broken packet by transporting the broken packet with a length of N to the buffer first address buf for storage and designating buf+N as the cache starting point in the next round of packet collection. The algorithm of the present invention meets the strict requirements of the communication industry for low space complexity, low time complexity, no increase in receiving error rate, etc. of the unpacking program, and at the same time ensures that the continuous and extremely fast packet receiving and unpacking method for splicing broken packets with a single buffer is suitable for communication equipment with various protocol types and equipped with various types of processors.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A continuous and extremely fast packet receiving and unpacking method for splicing broken packets with a single buffer, which is applied to a communication device for continuous packet receiving and unpacking. Its algorithm steps include:

[0006] 1) Create a receiving buffer with continuous storage space, whose starting address is buf and size is size; the variable N is used to save the amount of unpacked data in the unpacking area of the receiving buffer. The unpacking area refers to the area where data is cached waiting to be unpacked. Before the first packet reception, the amount of unpacked data N is equal to 0; create a pointer variable char* p to traverse the unpacking area to find and locate the starting addresses of each packet cached;

[0007] 2) After the communication device receives protocol packets in a stream form, cache the actual amount of data recv received this time with buf + N as the starting address, and update the amount of unpacked data (N = N + recv);

[0008] 3) Divide the unpacking area [buf, buf + N - 1] according to the size relationship between the maximum single - packet length M specified by the protocol and the amount of unpacked data N. If N < M, the entire unpacking area is divided into a possible broken - packet area. If N ≥ M, the unpacking area is divided into a complete - packet area [buf, buf + N - M] and a possible broken - packet area [buf + N - M + 1, buf + N - 1];

[0009] 4) Use the KMP string - matching algorithm to move the pointer p in the unpacking area to sequentially match and search for the packet - starting identifier field specified by the protocol from left to right. Whenever the starting identifier of a packet is found, use this as the starting point of the packet and unpack to the right according to the format and length of each field of the protocol. After unpacking a packet, first make the pointer p equal to the end address of this packet plus 1, and then use this as a new search starting point to continue to locate and disassemble the next packet to the right. If the pointer p has entered the possible broken - packet area during the process of shifting to the right from the new search starting point to match the starting identifier of the next packet, it is necessary to first perform an out - of - bounds check on the remaining unpacked part [p, buf + N - 1] according to the protocol format to confirm that the remaining unpacked part is long enough for one packet to continue unpacking. If the remaining unpacked part is less than one packet long, the remaining unpacked part caches an incompletely received broken packet and cannot continue unpacking. At this time, call the function memcpy(buf, p, N) to move this broken packet to the starting point of the buffer for storage, and then update the amount of unpacked data N to the length of the broken packet (buf + N - p), and end this round of unpacking. If no broken packet is found until the new search starting point pointed to by p has left the unpacking area, clear the amount of unpacked data N and end this round of unpacking;

[0010] 5) Go back to step 2) and continue the next round of packet receiving and unpacking. At this time, if the amount of data to be unpacked N is not 0, it is the length of the broken packets left over and moved in the previous round of unpacking process. This round uses buf+N as the starting point of the receiving cache to achieve rapid splicing of broken packets and new data.

[0011] In some embodiments, a method for continuous high-speed packet reception and unpacking using a single buffer for splicing broken packets is used, and the unpacking described in the method steps is communication unpacking, that is, after the communication device receives each protocol packet transmitted in the form of a stream, it restores the boundaries of each packet from the cached data stream according to the protocol format used by the transmission party, including the starting and ending positions of each packet in the buffer, and performs one or more processes including at least "taking the packet start identifier field as the starting position of the packet and disassembling the header fields to the right, disassembling and saving the load data, and checking whether the packet is correctly received when the packet tail contains a check field" according to actual needs; the broken packet refers to the fact that the amount of received data specified by the communication device is not exactly equal to "the sum of the lengths of one or more packets" in the protocol packets transmitted in sequence, resulting in a protocol packet cached in the end part of the specified receiving buffer that is missing the format due to incomplete reception, and the missing part of the packet that is not received is cached in the starting part area starting from the first address of the receiving buffer specified this time when the communication device receives a sufficiently long data stream next time.

[0012] In some embodiments, the method for continuous and extremely fast packet reception and unpacking using a single buffer to splice broken packets is a general communication unpacking method applicable to messages of any layer of the protocol in the OSI seven-layer network model framework structure, rather than a specific unpacking method applicable only to a single specific protocol. Any protocol message whose packet header contains a field that serves as a packet start identifier can use the algorithm to continuously receive and unpack packets and identify and process broken packets, which at least include "snmp or tftp or HTTP protocol messages at the application layer", "tcp or udp messages at the transport layer", "IP datagrams at the network layer", "PPP protocol packets at the data link layer" and other public or private protocol messages whose packet headers contain a field used as a start identifier.

[0013] In some embodiments, the method for continuous high-speed packet reception and unpacking using a single buffer to splice broken packets, wherein the method only uses a single storage space continuous receiving buffer during the process of continuous packet reception, unpacking and splicing broken packets, and does not open up any auxiliary space to achieve extremely low space complexity.

[0014] In some embodiments, the method for continuous high-speed packet reception and unpacking using a single buffer to splice broken packets completes any round of packet reception, unpacking and broken packet splicing by only traversing the receiving buffer from left to right once to achieve lower time complexity. This is also to ensure that for individual protocol messages that "allow header fields to be repeated with payload parts", in the process of matching and searching for packet start identifiers, the "payload field that is repeated with the header field is not first encountered" and the payload part is not mistakenly taken as the packet start position, resulting in erroneous unpacking. In order to ensure that the process of continuous packet reception, unpacking and broken packet splicing can be completed using a single thread, the scope of application of the single buffer broken packet splicing method is not limited to communication devices equipped with multi-core processors, but can also be used in communication devices with only single-core processors and single-thread processing capabilities.

[0015] Preferably, the continuous extremely fast packet receiving and unpacking method for splicing broken packets with a single buffer regards the data stream of the receiving buffer as the main string and the packet start identifier specified by the protocol as the destination string, and moves the pointer p from left to right in the receiving buffer by using the KMP string matching algorithm to match and search for the position of the packet start identifier field, and when the main string and the packet start identifier substring starting from the pointer p fail to match character by character, the pointer p continues to move right to search, and the NextVal array generated by the KMP algorithm is used to remove unnecessary backtracking and repeated comparison processes, so as to reduce the time complexity of the unpacking method for locating the starting position of each packet, but the continuous extremely fast packet receiving and unpacking method also supports the use of a brute force matching algorithm or other means instead of the KMP algorithm to match and search for the position of each packet start identifier;

[0016] Preferably, the continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer uses the memcpy function provided by the C standard library to achieve the purpose of moving the broken packets to the first address of the buffer as the starting point for storage, so as to ensure efficient data transfer to the continuous storage area, but the continuous high-speed packet receiving and unpacking method also supports calling other functions or using other alternative methods to move broken packets;

[0017] The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer, wherein the process of performing an out-of-bounds check on the remaining unpacked portion [p, buf+N-1] according to the protocol format is only performed in a possible broken packet area where "the length must be less than the maximum single packet length M specified by the protocol", so as to avoid repeatedly checking whether the unpacked portion is long enough for a packet in all the areas to be unpacked, thereby increasing a large amount of redundant judgment time; and for the pointer p, if any field is missing or wrong after successfully matching and locating the packet start identifier "in the packet complete area", the number of transmitted error packets is counted and added by 1, and the "pointer p is equal to the start identifier address of this error packet plus 1", so as to directly skip this error packet and continue to match and search for the next packet; the situation in which the packet start identifier is matched in the "possible broken packet area" and the out-of-bounds check on the remaining unpacked portion before unpacking is found to be a broken packet at least includes:

[0018] The protocol specifies a minimum single packet length, and the remaining unpacked portion is smaller than this length, or;

[0019] The protocol stipulates that the length of the packet header field is fixed to H bytes, and the remaining unpacked portion is less than H bytes, or;

[0020] The packet header of the protocol contains a field indicating the length of the payload data, and the length of the remaining unpacked portion is less than the sum of the packet header length and the payload data length, or;

[0021] The length of the remaining unpacked portion is smaller than "the sum of the lengths of any fields of known length that can be determined by the protocol format".

[0022] Preferably, for the continuous high-speed packet receiving and unpacking method using a single buffer to splice broken packets, if the protocol stipulates that the length of the packet header field is fixed at H bytes and the length of the possible broken packet area is ≥ H, the [buf+N-H+1, buf+N-1] part of the possible broken packet area can be subdivided into the packet header possible missing area, and the rest is the packet header complete area; thereby, in the out-of-bounds check before matching and searching for the packet start identifier in the packet header complete area, it is no longer calculated whether the remaining unpacked part is less than the packet header length H, so as to further speed up the speed of searching for the packet start identifier in the possible broken packet area.

[0023] The present invention also provides a communication device, comprising at least:

[0024] A general-purpose computer or embedded computer that continuously receives and parses application layer protocol packets through a network stream socket using a continuous extremely fast packet receiving and unpacking method that uses a single buffer to splice broken packets;

[0025] A network card, an end system card, and a network chip that receive and parse Ethernet messages using the single buffer and broken packet splicing continuous high-speed packet receiving and unpacking method;

[0026] A router that receives TCP / IP data packets and parses the source IP address and destination IP address of each packet using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method, and manages and forwards messages according to the IP address;

[0027] A switch that receives Ethernet frames and parses the source MAC address and destination MAC address of each frame using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method, and manages and forwards data frames according to the MAC address;

[0028] A lower computer that uses the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method to parse the private interactive protocol format command packet sent by the industrial control upper computer;

[0029] A measurement and control device that uses the single buffer splicing broken packet continuous high-speed packet collection and unpacking method to continuously collect data messages from the measured object, parse the messages, and comprehensively analyze various data indicators to analyze various functional performance indicators of the measured object;

[0030] An audio frame decoder that continuously receives and decodes audio frames using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method;

[0031] A video frame decoder that continuously receives and decodes video frames using the single buffer zone splicing broken packets and continuous high-speed packet receiving and unpacking method.

[0032] The present invention has at least the following beneficial effects:

[0033] (1) The method for continuously receiving and unpacking packets at a very high speed by splicing broken packets with a single buffer only uses a single storage space continuous receiving buffer, does not allocate any auxiliary space for unpacking and processing broken packets, and has extremely low space complexity; (2) The method for continuously receiving and unpacking packets at a very high speed by splicing broken packets with a single buffer uses a single thread for continuous packet receiving and unpacking, and the program for the method for continuously receiving and unpacking packets at a very high speed by splicing broken packets with a single buffer can run on any type of communication device processor, including single-core or multi-core processors; (3) The method for continuously receiving and unpacking packets at a very high speed by splicing broken packets with a single buffer is universal and can be used for continuous packet receiving and unpacking of any protocol message that gives a start identifier in the packet header, and can be applicable to any layer protocol in the OSI seven-layer network model framework structure. (4) The continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer reduces the time complexity of the packet starting point positioning process by using the KMP string matching algorithm to search for the starting identifier of each packet in the area to be unpacked; (5) The continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer divides the possible broken packet area in advance according to the relationship between the maximum length of a single packet specified in the protocol and the amount of data to be unpacked, thereby reducing the scope of the pre-unpacking cross-border check to see whether the remaining part is long enough for a packet; (6) The continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer quickly identifies the receiving buffer through cross-border checks in the possible broken packet area that is less than the maximum single packet length. (7) A method for continuously receiving and unpacking broken packets by splicing broken packets with a single buffer at a very high speed provides a fast and clever method for splicing broken packets, and sequentially implements continuous packet reception and unpacking; (8) A method for continuously receiving and unpacking broken packets by splicing broken packets with a single buffer only traverses the receiving buffer from left to right once in each round of locating the packet start identifier and unpacking process, thereby avoiding the situation in which the payload is mistakenly taken as the starting point in the process of locating the packet start point for individual protocols that allow the packet header start identifier and the payload to be repeated. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0035] Figure 1 A schematic diagram of a method for continuously receiving and unpacking packets at a very high speed by splicing broken packets with a single buffer provided by the present invention;

[0036] Figure 2 This is an example diagram of the format of a private protocol packet from a host computer that needs to be parsed by a lower computer in an embodiment of the present invention;

[0037] Figure 3 This is a code example of a method for continuously receiving and unpacking packets at an extremely high speed by splicing broken packets using a single buffer provided by the present invention. Detailed implementation manners

[0038] The following further elaborates on the embodiments of the present invention with reference to the accompanying drawings, so that those skilled in the art can implement it according to the description in the specification.

[0039] As Figure 1 shown, in the simple case where the protocol format stipulates that the packet header only contains the start identifier field, the main steps of the continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer are illustrated, including:

[0040] 1) Create a receiving buffer with continuous storage space, whose starting address is buf; create a variable N, which is used to save the amount of unpacked data in the unpacking area of the receiving buffer. Before the first packet reception, the amount of unpacked data N is equal to 0; create a pointer variable char* p, which is used to traverse the unpacking area to find and locate the packet start identifier fields of each cached packet;

[0041] 2) After the communication device receives the protocol packet in a stream form, it caches the actually received data volume recv with buf + N as the starting address, and updates the amount of unpacked data (N = N + recv);

[0042] 3) Divide the unpacking area [buf, buf + N - 1] according to the size relationship between the maximum single-packet length M stipulated by the protocol and the amount of unpacked data N. If N < M, the entire unpacking area is classified as a possible broken-packet area; otherwise, the unpacking area is divided into a complete-packet area [buf, buf + N - M] and a possible broken-packet area [buf + N - M + 1, buf + N - 1];

[0043] 4) Move the pointer p in the area to be unpacked by using the KMP string matching algorithm to match the packet start identifier field specified in the search protocol from left to right. When the start identifier of the current packet is found, the found start identifier is used as the packet start point, and the packets are unpacked to the right in sequence according to the format and length of each field specified in the protocol. When the unpacking of the current packet is completed, the pointer p is equal to the end address of this packet plus 1, and then the pointer is used as the new search starting point to continue to locate the next packet of the current packet to the right. If the pointer p has entered the possible packet break area during the process of moving right from the new search starting point to match the start identifier of the next packet, the remaining packets must be matched according to the protocol format. The unpacked part [p, buf+N-1] is checked for out-of-bounds to confirm that the remaining unpacked part is long enough for one packet to continue unpacking. If the remaining unpacked part is less than one packet, the remaining unpacked part caches the broken packets that have not been completely received and cannot continue unpacking. At this time, the function memcpy(buf, p, N) is called to move the broken packets to the starting point of the buffer for storage, and the amount of data to be unpacked N is updated to be equal to the length of the broken packet (buf+Np), and then the current round of unpacking ends. If no broken packet is found until the new search starting point pointed to by p has left the area to be unpacked, the amount of data to be unpacked N is cleared and the current round of unpacking ends;

[0044] 5) Go back to step 2) and continue the next round of packet receiving and unpacking. At this time, if the amount of data to be unpacked N is not 0, it is the length of the broken packets left over and moved during the previous round of unpacking. This round uses buf+N as the starting point of the receiving cache, which just realizes the rapid splicing of broken packets and new data.

[0045] Obviously, the continuous high-speed packet receiving and unpacking method for splicing broken packets with a single buffer of the present invention uses a single continuous buffer for storage, does not need to open and use additional auxiliary space, and completes the starting position positioning of all cached packets by only a single traversal of the to-be-unpacked area by using the KMP pattern matching algorithm. After each packet is located, it is immediately unpacked to the right, and after unpacking, the next packet is continuously positioned to the right, and the traversal pointer does not roll back; the KMP algorithm is a well-known string efficient matching algorithm, which calls the main string the destination string and the substring the pattern string, and the process of searching for the matching and locating the position of the pattern string in the destination string is called pattern matching; different from the brute force matching algorithm, the KMP algorithm will generate the NextVal array in advance according to the content of the pattern string, so as to eliminate unnecessary repeated comparison and fallback search processes when the pattern matching fails, and the time complexity of the KMP algorithm is "O (main string length + substring length)", which is much lower than the time complexity of the brute force matching algorithm "O (main string length * substring length)". In addition, at the end of each round of unpacking, the position where the traversal pointer stops is used to identify whether there are broken packets in the area to be unpacked, as well as the first address and length of the broken packets. Then, by moving the broken packets to the starting point of the receiving buffer, the next time the receiving buffer is close, efficient broken packet splicing and continuous receiving and unpacking are achieved. This is one of the biggest features that distinguish the present invention from other packet receiving and unpacking methods. Broken packets do not necessarily exist in every round of caching. If they exist, the area where they are located must be at the end of this cache. Broken packets are packets that have not been completely received, and their length is naturally smaller than the maximum length of a complete single packet. Therefore, the range of the possible single packet area divided is smaller than the maximum length of a single packet specified by the protocol. The present invention only traverses the spatially continuous receiving buffer from left to right once and moves the broken packet only once to complete a full round of unpacking. Its time complexity and space complexity are much lower than traditional unpacking methods that require multiple data copies and auxiliary space to handle broken packets. Figure 1 Only the disassembled load fields are shown in the figure, because load data is often the most needed field;

[0046] It should be noted that Figure 1 It is only a schematic diagram of the core method of the present invention, and does not represent all the steps and preferred solutions of the present invention. For example, based on this method, parallel unpacking can be achieved by dividing the area to be unpacked into blocks, and the receiving unpacking program runs on a single-core CPU, or when the load pressure is not continuously extremely high, directly using the method described in the present invention to design the unpacking program is the optimal solution. The reason is that, firstly, a single-core processor running multiple threads can not only fail to execute in parallel, but also increase the thread switching time and waste the processor scheduling time. Secondly, directly using the example code of the present invention can obtain much lower time and space complexity than traditional unpacking methods in most cases, while also saving processor scheduling resources and operating system resources for storing environment variables as much as possible. It is a universal continuous and extremely fast packet receiving and unpacking method with a wide range of applications.

[0047] Figure 2 The present invention shows an example of a private protocol data packet format that can be used for communication and interaction between a host computer and a slave computer. The characteristics of the protocol format are very representative, and it retains many common features of mainstream communication protocols. The protocol packet format contains a header, a payload, and a tail, wherein the header contains a protection packet start identifier field and a packet length field, and the tail is a CRC check word. The host computer often does not directly transmit pure data to the slave computer but uses an interactive protocol. The general reason is that the slave computer, as the receiver, cannot confirm whether the data sent by the host computer represents instructions, pure data, handshake signals or other meanings. For example, if the slave computer receives 4 bytes 0x12345678, if the unified protocol format is not agreed in advance, it is impossible to confirm whether these 4 bytes are values ​​or ASCII codes corresponding to each character of the string, or represent the checksum of the data packet. Even if the two communicating parties agree in advance to only transmit pure data, the receiver cannot judge whether all the data sent by the sender has been received.

[0048] Figure 3 Demonstrated for Figure 2The protocol format shown is an example of a continuous packet receiving and unpacking program designed based on the method of the present invention. After the lower computer continuously receives and caches each protocol packet transmitted from the upper computer in a stream form, the unpacked load is saved in a file for sequential integration; receive is an example function for the convenience of explanation, which is used to represent the specific function called when the communication device receives or collects in a blocking manner, and does not exist in the C standard library. The first parameter of this function is the cache starting point, and the second parameter is the amount of data to be received. The function returns the actual amount of data received. After the thread runs the receive function, it will enter a blocked state and will not be awakened until the data is received. That is, since the actual amount of data received is not necessarily exactly equal to the sum of the lengths of one or more packets transmitted in sequence, there may be a broken packet at the end of the cache. If a broken packet is found after each round of unpacking, it will be moved to the first address of the buffer buf as the starting point for storage. In the next round of packet receiving, new data is cached with buf+N as the starting point to achieve broken packet splicing, so that the next round of packet receiving and unpacking process can be started smoothly. Since the packet receiving and unpacking method uses the improved KMP algorithm, the pointer p traversed in the entire area to be unpacked will never go back, and the time complexity is extremely low.GetNext generates the NextVal array required by the KMP algorithm in advance based on the known pattern string, i.e., the substring. It will help reduce a large number of unnecessary comparisons. The NextVal array generation method of KMP is one of the most common and efficient string matching algorithms that have been made public. It should be noted that in the process of using the KMP algorithm to match and locate the packet header, some improvements have been made to the KMP algorithm. For example, the current position counting variable i of the main string is replaced by directly using the pointer p to move and determine whether it is less than the end point, and the original KMP algorithm's condition of i<main string length is replaced by p≤pObjEnd. In the packet completion area, directly set pCompleRegionEnd = buf + N - M;, which is actually from the end point of the packet completion area, that is, it ensures that the traversal pointer p is at [buf, buf, buf+NM]. In this complete packet area, then with p as the starting point, there is at least one complete packet on its right, thus eliminating the problem of repeatedly calculating the packet end point in the entire complete packet area to check whether it exceeds the area to be unpacked, and the possible broken packet area must be smaller than the maximum length M of a single packet. In this example, the maximum single packet length M=1518 bytes specified by the protocol is insignificant for the entire area to be unpacked, so the time and scope for out-of-bounds check in the possible broken packet area are extremely small and negligible, greatly reducing the time complexity of traditional unpacking methods; the present invention only uses a single continuous buffer, without using additional auxiliary space, and the space complexity is also much lower than that of traditional unpacking methods; it is worth noting that the present application can complete the identification and processing of broken packets with only 3 lines of code. If the traversal pointer p still stays in the area to be unpacked after the current round of unpacking ends, then it is obvious that the remaining unpacked part starting from p is a broken packet, which makes it impossible to continue unpacking to the right, so p stays in the area to be unpacked, then directly Let N = the length of the broken packet, and move the broken packet to the starting point of the buffer. The next time buf+N is used to continue receiving the buffer, the broken packet splicing is completed. This is an extremely efficient and clever way to handle broken packets. If p>pObjEnd, it means that p has left the area to be unpacked and completed the parsing of all data. No broken packets are found and the amount of data to be unpacked is updated to N=0. Here, pObjEnd is the end point of the area to be unpacked, buf+N-1. It should be noted that when using the improved KMP algorithm, it does not matter whether a certain byte is a value of 0, because it is compared byte by byte, that is, whether the packet header is a string or not, whether the area to be unpacked is a string or not, because it can still locate and match the starting point of each packet by whether the value of each byte of the substring is exactly the same as a part of the main string. Therefore, the so-called KMP string matching object string pObjStr does not refer to a string of characters ending with 0 in a narrow sense, but data of any format and length. In addition,. Figure 3The code examples given are not the best codes in the program of the embodiment of the present invention, nor are they the optimal solutions in various situations. For example, when the load pressure of continuous reception is extremely high and the program is running on a multi-core processor, it is entirely possible to expand this method to implement parallel unpacking of multiple threads in different regions; in addition, the present application also optimizes the execution efficiency of the program of the receiving and unpacking method based on the principle of fewer corresponding statements after assembly. The receive function can be replaced with specific functions such as recvfrom or read, and the specific implementations of CRC_Check and GetNext can be filled in. It can be used through the pipe communication method or network socket communication method of the Linux system. Figure 3 The routine shown here simulates the process of continuous packet receiving, unpacking, and splicing of broken packets.

[0049] It should be noted that the purpose of the present invention is not to provide a specific unpacking method for a single specific protocol, but to provide a general method with extremely low time and space complexity, which does not occupy too much system processor resources as much as possible and ensures efficient and continuous packet receiving and unpacking. Figure 3 In the example program, you can actually replace the line of code "pCompleRegionEnd = buf+NM;" with "pCompleRegionEnd = buf+N-M+H-1;". Even if the possible packet break area is not the smallest area, it will not cause unpacking errors. It is just a redundant out-of-bounds check in a small part of the area.

[0050] Although the embodiments of the present invention are disclosed above, they are not limited to the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the embodiments of the present invention. For those familiar with the art, other modifications can be easily realized. Therefore, without departing from the general concept defined by the claims and equivalent scopes, the embodiments of the present invention are not limited to the specific details and the legends shown and described here.

Claims

1. A method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer, which is applied to a communication device for continuously receiving and unpacking packets, and is characterized in that: Including: 1) Create a receiving buffer with continuous storage space, whose starting address is buf; Create a variable N, where the variable N is used to represent the amount of unpacked data in the unpacking area stored in the receiving buffer. Before the first packet reception, the amount of unpacked data N is equal to 0; create a pointer variable char* p, where the pointer variable char* p is used to traverse the unpacking area to find and locate the packet start identifier field of each cached packet; 2) After the communication device receives protocol packets in a stream form, cache the currently actually received data volume recv with buf+N as the starting address, and update the amount of unpacked data (N = N + recv); 3) Divide the unpacking area [buf, buf+N-1] according to the size relationship between the maximum single-packet length M specified by the protocol and the amount of unpacked data N. If N<M, the entire unpacking area is divided into a possible fragmented packet area. If N≥M, the unpacking area is divided into a complete packet area [buf, buf+N-M] and a possible fragmented packet area [buf+N-M+1, buf+N-1]; 4) Use the KMP string matching algorithm to move the pointer p in the unpacking area to sequentially match and search for the packet start identifier field specified by the protocol from left to right. When the start identifier of the current packet is found, use the found start identifier as the packet start point, and unpack to the right sequentially according to the format and length of each field specified by the protocol. When the current packet unpacking is completed, make the pointer p equal to the end address of this packet plus 1, and then use this as a new search start point to continue to the right to locate and disassemble the next packet of the current packet. If the pointer p has entered the possible fragmented packet area during the process of moving to the right from the new search start point to match the start identifier of the next packet, first perform an out-of-bounds check on the remaining unpacked part [p, buf+N-1] according to the protocol format to confirm that the remaining unpacked part is long enough for one packet to continue unpacking. If the remaining unpacked part is less than one packet length, the remaining unpacked part caches a fragmented packet that has not been completely received and cannot continue unpacking. At this time, call the function memcpy(buf, p, N) to move the fragmented packet to the start of the buffer for storage, and update the amount of unpacked data N to the length of the fragmented packet (buf+N-p), and then end this round of unpacking. If no fragmented packet is found until the new search start point pointed to by p has left the unpacking area, clear the amount of unpacked data N and end this round of unpacking; 5) Return to step 2) to continue the next round of packet reception and unpacking. At this time, if the amount of unpacked data N is non-zero, it is the length of the fragmented packet left over and moved during the previous round of unpacking. This round uses buf+N as the starting point of the receiving cache to achieve fast splicing of the fragmented packet and new data; Among them, in the process of completing any round of packet reception, unpacking, and fragmented packet splicing, the method only traverses the receiving buffer once from left to right to achieve a lower time complexity, and at the same time ensures that for other protocol packets that allow the header field and the payload part to be repeated, during the process of matching and searching for the packet start identifier, the payload field that repeats with the header field will not be encountered first, resulting in incorrect unpacking by misidentifying the payload part as the packet start position.

2. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: The process of the next round of packet receiving and unpacking specifically includes: after the communication device receives each protocol packet transmitted in the form of a stream, it restores the boundaries of each packet from the cached data stream according to the protocol format used by the transmission party, and the boundaries of each packet include the starting point and end point of each packet in the buffer, and performs one or more processes according to needs, including at least disassembling the packet header fields to the right with the packet start identifier field as the starting point of the packet, disassembling and saving the load data field, and checking whether the packet is correctly received when the packet tail contains a check field; the broken packet is a protocol packet cached in the end part of the designated receiving buffer due to incomplete reception caused by the amount of received data specified by the communication device not being exactly equal to the sum of the lengths of one or more packets in the sequentially transmitted protocol packets, and the format is missing due to incomplete reception. The missing part of the packet that is not received is cached in the starting part area starting from the first address of the receiving buffer specified this time when the communication device receives a sufficiently long data stream in the next round of packet receiving.

3. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: The continuous and extremely fast packet receiving and unpacking method is a universal communication unpacking method applicable to messages of any layer protocol in the OSI seven-layer network model framework structure. Any protocol message whose packet header contains a field that serves as a packet start identifier can use the algorithm to continuously receive and unpack packets and identify and process broken packets. Among them, the protocol messages applicable to the continuous and extremely fast packet receiving and unpacking method include at least SNMP, TFTP or HTTP protocol messages of the application layer, TCP or UDP messages of the transport layer, IP datagrams of the network layer, PPP protocol packets of the data link layer, and other public or private protocol messages whose packet headers contain a field used as a start identifier.

4. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: In the process of continuous packet receiving and unpacking and broken packet splicing, only a single storage space continuous receiving buffer is used, and no auxiliary space is required to achieve extremely low space complexity.

5. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: The algorithm uses the data stream received in the cache as the main string and the packet start identifier specified in the protocol as the target string. The KMP string matching algorithm is used to move the pointer p from left to right in the receiving buffer to match and locate the position of the packet start identifier field. When the main string and the packet start identifier substring starting from the pointer p fail to match character by character, the pointer p continues to move right to search. The NextVal array generated by the KMP algorithm is used to eliminate unnecessary backtracking and repeated comparison processes to reduce the time complexity of the method in locating the starting point of each packet. The method supports the use of a brute force matching algorithm instead of the KMP algorithm to match and locate the position of each packet start identifier.

6. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: The continuous high-speed packet receiving and unpacking method uses the memcpy function provided by the C standard library to achieve the purpose of moving broken packets to the first address of the buffer for storage as the starting point, so as to ensure efficient data transfer in the continuous storage area, wherein the continuous high-speed packet receiving and unpacking method supports calling the strncpy function or the sprintf function instead of the memcpy function to move broken packets.

7. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: The process of performing an out-of-bounds check on the remaining unpacked part [p, buf+N-1] according to the protocol format is only performed in a possible packet-break area whose length is less than the maximum single packet length M specified by the protocol, so as to avoid repeatedly checking whether the unpacked part is long enough for a packet in all the areas to be unpacked, thereby increasing a large amount of redundant judgment time; for the pointer p, if any field is missing or wrong during the unpacking process after successfully matching and locating the packet start identifier in the packet complete area, the number of transmitted error packets is counted plus 1 and the pointer p is equal to the start identifier address of the error packet plus 1, so as to directly skip the error packet and continue to match and search for the next packet; wherein, the situation in which the packet start identifier is matched in the possible packet-break area and the remaining unpacked part is checked for an out-of-bounds error before unpacking and the part is found to be a broken packet includes at least: The protocol specifies a minimum single packet length, and the remaining unpacked portion is smaller than this length, or; The protocol stipulates that the length of the packet header field is fixed to H bytes, and the remaining unpacked portion is less than H bytes, or; The packet header of the protocol contains a field indicating the length of the payload data, and the length of the remaining unpacked portion is less than the sum of the packet header length and the payload data length, or; The length of the remaining unpacked portion is less than the sum of the lengths of any fields of known length that can be determined by the protocol format.

8. The method for continuously receiving and unpacking packets at extremely high speed by splicing broken packets with a single buffer as claimed in claim 1, characterized in that: If the protocol stipulates that the length of the packet header field is fixed at H bytes and the length of the possible packet broken area is ≥ H bytes, then the [buf+N-H+1, buf+N-1] part of the possible packet broken area is the packet header possible missing area, and the rest is the packet header complete area. Therefore, in the out-of-bounds check before matching and searching for the packet start identifier in the packet header complete area, it is no longer necessary to calculate whether the remaining unpacked part is less than the packet header length H, so as to further speed up the search for the packet start identifier in the possible packet broken area.

9. A communication device, characterized in that: At least: A general-purpose computer or embedded computer that continuously receives and parses application layer protocol packets through a network stream socket using a continuous extremely fast packet receiving and unpacking method that uses a single buffer to splice broken packets; A network card, an end system card or / and a network chip receiving and parsing Ethernet messages using the single buffer zone splicing broken packets continuous high-speed packet receiving and unpacking method; A router that receives TCP / IP data packets and parses the source IP address and destination IP address of each packet using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method, and manages and forwards messages according to the IP address; A switch that receives Ethernet frames and parses the source MAC address and destination MAC address of each frame using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method, and manages and forwards data frames according to the MAC address; A lower computer that uses the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method to parse the private interactive protocol format command packet sent by the industrial control upper computer; A measurement and control device that uses the single buffer splicing broken packet continuous high-speed packet collection and unpacking method to continuously collect data messages from the measured object, parse the messages, and comprehensively analyze various performance indicators of the measured object based on various data indicators; An audio frame decoder that continuously receives and decodes audio frames using the single buffer splicing broken packets continuous high-speed packet receiving and unpacking method; A video frame decoder that continuously receives and decodes video frames using the single buffer zone splicing broken packets and continuous high-speed packet receiving and unpacking method.

Images