Queue-based data stream effective data packet real-time extraction method

Abstract

The invention discloses a queue-based data stream effective data packet real-time extraction method, which comprises the following steps of: (1) periodically accessing a data stream source, reading data at one time and storing the data into a data queue; (2) checking the queue length, and if the queue length is greater than the length of one data packet, executing the step (3); otherwise, returning to the step (1); (3) searching a data packet header from the head of the queue, discarding invalid bytes until the packet header is found, and if the length of the queue is greater than the length of one data packet at the moment, executing the step (4); otherwise, returning to the step (1); (4) reading and verifying the data packet, and if the verification is passed, executing the step (5); if verification fails, clearing queue data with the length of one data packet, and returning to the step (2); (5) extracting effective data, emptying the queue, and returning to the step (1); and if the data integrity is required at the same time, only clearing the queue data with the length of one data packet, and returning to the step (2). According to the method provided by the invention, through a queue mechanism and an effective data packet processing flow, the data packet can be reliably and efficiently extracted and processed in real time, and meanwhile, the integrity of data and the stability of a system can also be ensured.

Description

Technical Field

[0001] The invention relates to a method for real-time extraction of valid data packets of data stream based on a queue, belonging to the technical field of data transmission. Background Art

[0002] Data stream transmission refers to the process of transmitting and processing data between different systems or devices in a continuous and sequential manner. With the continuous development of information technology, data stream transmission has become an important technical means in modern communication systems. A typical form of data stream transmission is transmission through a serial communication interface. As a specific implementation of data stream transmission, serial communication is widely used in various data communication occasions due to its advantages such as simple interface and easy use. In practical applications, the data sender usually encapsulates the data into data packets and transmits them through a serial communication interface in the form of a data stream. The receiving end needs to accurately extract the complete data packet from the received data stream. Therefore, how to extract valid data packets from these data streams in real time and effectively is a key issue, especially in distributed asynchronous data acquisition and control systems. Due to the problems of asynchronous transmission rate, data packet loss, signal interference, etc., the real-time and accuracy of data are crucial.

[0003] The patent with publication number CN105022712A discloses a data packet extraction method based on data stream transmission, which performs different steps to extract data packets by judging the unpacking progress value. However, due to the need to frequently access the receiving queue and check the synchronization word of the data packet, processing delays may occur. It is not efficient enough for high-speed data streams and cannot meet the requirements of high real-time performance and high reliability. The patent with publication number CN107766265B discloses a serial port data extraction method that supports fixed-length packets, variable-length packets, and mixed packets. According to the configured packet format, a data packet is extracted from the buffer. However, since all the data is placed in the buffer and only the data of one data packet is read each time, a large amount of system resources will be occupied. At the same time, at high data transmission rates, real-time requirements may not be met.

[0004] In order to solve the above problems, it is necessary to propose an efficient and reliable data stream effective data packet extraction method that can ensure the real-time and accuracy of data, so that the system can ensure the real-time and accuracy of data in a complex communication environment and improve the performance and stability of the entire system. Summary of the invention

[0005] The present invention proposes a method for extracting valid data packets from a data stream in real time based on a queue. Through a queue mechanism and an effective data packet processing flow, data packets can be extracted and processed reliably, efficiently and in real time, while also ensuring data integrity and system stability.

[0006] The technical solution of the present invention:

[0007] (1) Periodically access the data stream source, read all the data at once and store them in a data queue in sequence;

[0008] (2) Check the length of the data queue. If it is greater than or equal to the length of a data packet, execute step (3); if the length is insufficient, return to step (1);

[0009] (3) Search for the packet header from the head of the queue, discard invalid bytes until the packet header is found. If the queue length is greater than or equal to the length of a packet, execute step (4); otherwise, return to step (1);

[0010] (4) Read and verify the data packet. If the verification passes, execute step (5); if the verification fails, clear the queue data of the length of a data packet and issue an alarm to indicate that the system communication is abnormal, and return to step (2);

[0011] (5) Extract valid data and clear the queue, indicating that the system communication is normal, and return to step (1); if data integrity is required at the same time, only clear the queue data of the length of one data packet, and return to step (2) to continue parsing and extracting data.

[0012] In step (1), the data stream source is periodically accessed, all data is read at one time and stored in a data queue in sequence, specifically including: setting a fixed time interval to access the data stream source, each time the access is performed, all current data is read from the buffer of the data stream source and all the read data is stored in sequence in a data queue (first-in-first-out queue) in sequence. It is used to temporarily store and manage the received data bytes in chronological order.

[0013] In step (2), the length of the data queue is checked. If it is greater than or equal to the length of a data packet, step (3) is executed; if the length is insufficient, step (1) is returned, which specifically includes:

[0014] Check the total length of the data in the current data queue. If the length of the data in the data queue is greater than or equal to the length of a valid data packet, proceed to step (3) and start searching for a valid data packet; if the data length is insufficient, temporarily stop further operations and return to step (1) to wait for the next periodic data reading.

[0015] In step (3), the packet header is searched from the head of the queue, and invalid bytes are discarded until the packet header is found. If the queue length is greater than or equal to the length of a packet, step (4) is executed; otherwise, step (1) is returned, which specifically includes:

[0016] Start checking the data one by one from the head byte of the queue (i.e. the earliest received byte). The data packet header is usually one or more specific byte sequences used to identify the beginning of the data packet. According to the protocol definition, these byte sequences may be fixed values ​​or specific formats. If the byte sequence currently checked is not the data packet header, discard one or more invalid bytes of the current header from the queue and continue searching backwards. Once the data packet header is found, check whether the data length in the queue is sufficient for a complete data packet. If the length is sufficient, go to step (4), otherwise return to step (1).

[0017] In step (4), the data packet is read and verified. If the verification passes, step (5) is executed; if the verification fails, the queue data of the length of a data packet is cleared and an alarm is issued to indicate that the system communication is abnormal, and the process returns to step (2), which specifically includes:

[0018] After finding the packet header, read the following data, which includes the actual data after the packet header and the check data. According to the format definition of the packet, extract the necessary fields and perform packet check. If the check passes, it means the packet is valid and go to step (5); if the check fails, it means the packet may be damaged or the data is incorrect, discard the data of one packet length from the queue, and continue to search for the next packet header from step (2), while recording the total number of check failures (each failure is incremented by 1). If the total number reaches the set value, an alarm is issued to indicate that the system is in an abnormal communication state.

[0019] In step (5), valid data is extracted and the queue is cleared, indicating that the system communication is normal, and the process returns to step (1); if data integrity is required at the same time, only the queue data of the length of one data packet is cleared, and the process returns to step (2) to continue parsing and extracting data, which specifically includes:

[0020] Extract the valid data portion from the data packet that has passed the verification. Remove the entire processed data packet from the queue to free up space for receiving and processing more data. Clear the total number of verification failures to zero to indicate that the system is in a normal communication state. After processing the current data packet, do not continue to parse subsequent data, clear the queue, and return to step (1) to restart periodic data reading. If data integrity is required, after processing the current data packet, return to step (2), continue to parse the remaining data in the queue during the current cycle, and extract more valid data packets. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for real-time extraction of valid data packets from a data stream based on a queue according to the present invention;

[0022] Figure 2 A LabVIEW program block diagram for implementing the method of the present invention;

[0023] Figure 3 The format diagram of the data packet is defined;

[0024] Figure 4 Partial program diagram for sending data to the lower computer;

[0025] Figure 5 This is a diagram showing the effect of using the LabVIEW program to receive data. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0027] like Figure 1 As shown, the present invention provides a method for real-time extraction of valid data packets from a data stream based on a queue, comprising the following steps:

[0028] (1) Periodically access the data stream source, read all the data at once and store them in a data queue in sequence;

[0029] like Figure 2 As shown in the figure, the data stream source is accessed through VISA, a queue is created to store data, and the main loop is set to 10ms to access the data stream source regularly. Each time it is accessed, all current data is read from the buffer of the data stream source and all the read data is stored in the data queue in sequence. It is used to temporarily store and manage the received data bytes in chronological order.

[0030] (2) Check the length of the data queue. If it is greater than or equal to the length of a data packet, execute step (3); if the length is insufficient, return to step (1);

[0031] Define the data packet format as Figure 3 As shown in the figure, there are 9 bytes in total. The packet header is 55AA, followed by 6 bytes of actual valid data, and the packet tail is the checksum. Figure 2 As shown, the total length of the data in the current data queue is checked. If the length of the data in the data queue is greater than or equal to the length of a valid data packet, then go to step (3) and start searching for a valid data packet; if the data length is insufficient, then temporarily stop further operations, end the loop, return to step (1), and wait for the next periodic data reading.

[0032] (3) Search for the packet header from the head of the queue, discard invalid bytes until the packet header is found. If the queue length is greater than or equal to the length of a packet, execute step (4); otherwise, return to step (1);

[0033] like Figure 2As shown, the data is checked one by one starting from the head byte of the queue (i.e., the earliest received byte). If the first byte of the current queue head is not 55 or the second byte is not AA, the current head byte is discarded from the queue and the search continues backward. If the first byte of the current queue head is 55 and the second byte is AA, check whether the length of the data in the queue is sufficient for a complete data packet. If the length is sufficient, go to step (4), otherwise end the loop and return to step (1).

[0034] (4) Read and verify the data packet. If the verification passes, execute step (5); if the verification fails, clear the queue data of the length of a data packet and issue an alarm to indicate that the system communication is abnormal, and return to step (2);

[0035] like Figure 2 As shown, after finding the packet header, read the following data, which includes the actual data and check data after the packet header. The actual data has a total of 6 bytes, and the check bit of the last byte is the last 8 bits of the sum of the 6 bits of actual data. According to the format definition of the data packet, perform data packet verification. If the verification passes, it means that the data packet is valid, and go to step (5); if the verification fails, it means that the data packet may be damaged or the data is incorrect, and discard the data of a packet length from the queue, and continue to search for the next data packet header from step (2), and record the total number of verification failures (each failure number is incremented by 1). If the total number reaches 10 times, an alarm is issued to indicate that the system is in an abnormal communication state.

[0036] (5) Extract valid data and clear the queue, indicating that the system communication is normal, and return to step (1); if data integrity is required at the same time, only clear the queue data of the length of one data packet, and return to step (2) to continue parsing and extracting data.

[0037] like Figure 2 As shown, after the verification is passed, the valid data part is extracted from the data packet that has passed the verification and displayed in the waveform diagram. The entire processed data packet is removed from the queue to free up space for receiving and processing more data. The total number of verification failures is cleared to indicate that the system is in a normal communication state. After processing the current data packet, the subsequent data is no longer parsed, the queue is cleared, the loop ends, and the process returns to step (1) to restart the periodic data reading. If the integrity of the data is required, the communication requirement in the program is changed from "real-time" to "data integrity". After processing the current data packet, the process returns to step (2) to continue parsing the remaining data in the queue during the current cycle to extract more valid data packets.

[0038] Part of the program for sending data from the lower computer is as follows Figure 4As shown in the figure, the initial 6 actual data sent are 0x00, 0x14, 0x28, 0x3C, 0x50, and 0x64. The lower computer sends a data packet every 10ms, and the 6 actual data will be incremented by 4 after each transmission. Figure 5 As shown, the upper computer program written using the method proposed in the present invention can extract the data sent by the lower computer in real time and accurately.

[0039] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for extracting valid data packets from a data stream in real time based on a queue, characterized in that The following steps are involved: (1) Periodically access the data stream source, read all the data at once and store them in a data queue in sequence; (2) Check the length of the data queue. If it is greater than or equal to the length of a data packet, execute step (3); if the length is insufficient, return to step (1); (3) Search for the packet header from the head of the queue, discard invalid bytes until the packet header is found. If the queue length is greater than or equal to the length of a packet, execute step (4); otherwise, return to step (1); (4) Read and verify the data packet. If the verification passes, execute step (5); if the verification fails, clear the queue data of the length of a data packet and issue an alarm to indicate that the system communication is abnormal, and return to step (2); (5) Extract valid data and clear the queue, indicating that the system communication is normal, and return to step (1); if data integrity is required at the same time, only clear the queue data of the length of one data packet, and return to step (2) to continue parsing and extracting data.

2. The method for extracting valid data packets from a data stream in real time based on a queue as claimed in claim 1, characterized in that In the step (1), the data stream source is periodically accessed, all data is read at one time and stored in a data queue in sequence, which specifically includes: Set a fixed time interval to access the data stream source. Each time you access it, read all the current data from the data stream source's buffer and store all the read data in a data queue (first-in, first-out queue) in order. It is used to temporarily store and manage the received data bytes in chronological order.

3. The method for extracting valid data packets from a data stream in real time based on a queue as claimed in claim 1, characterized in that In the step (2), the length of the data queue is checked. If it is greater than or equal to the length of a data packet, step (3) is executed; if the length is insufficient, step (1) is returned, which specifically includes: Check the total length of the data in the current data queue. If the length of the data in the data queue is greater than or equal to the length of a valid data packet, proceed to step (3) and start searching for a valid data packet; if the data length is insufficient, temporarily stop further operations and return to step (1) to wait for the next periodic data reading.

4. The method for extracting valid data packets from a data stream in real time based on a queue as claimed in claim 1, characterized in that In the step (3), the packet header is searched from the head of the queue, and invalid bytes are discarded until the packet header is found. If the queue length is greater than or equal to the length of a packet, step (4) is executed; otherwise, step (1) is returned, which specifically includes: Start checking the data one by one from the head byte of the queue (i.e. the earliest received byte). The data packet header is usually one or more specific byte sequences used to identify the beginning of the data packet. According to the protocol definition, these byte sequences may be fixed values ​​or specific formats. If the byte sequence currently checked is not the data packet header, discard one or more invalid bytes of the current header from the queue and continue searching backwards. Once the data packet header is found, check whether the data length in the queue is sufficient for a complete data packet. If the length is sufficient, go to step (4), otherwise return to step (1).

5. The method for extracting valid data packets from a data stream in real time based on a queue as claimed in claim 1, characterized in that In the step (4), the data packet is read and verified. If the verification passes, step (5) is executed; if the verification fails, the queue data of the length of a data packet is cleared and an alarm is issued to indicate that the system communication is abnormal, and the process returns to step (2), which specifically includes: After finding the packet header, read the following data, which includes the actual data after the packet header and the check data. According to the format definition of the packet, extract the necessary fields and perform packet check. If the check passes, it means the packet is valid and go to step (5); if the check fails, it means the packet may be damaged or the data is incorrect, discard the data of one packet length from the queue, and continue to search for the next packet header from step (2), while recording the total number of check failures (each failure is incremented by 1). If the total number reaches the set value, an alarm is issued to indicate that the system is in an abnormal communication state.

6. The method for extracting valid data packets from a data stream in real time based on a queue as claimed in claim 1, characterized in that In the step (5), valid data is extracted and the queue is cleared, indicating that the system communication is normal, and the process returns to step (1); if data integrity is required at the same time, only the queue data of the length of one data packet is cleared, and the process returns to step (2) to continue parsing and extracting data, which specifically includes: Extract the valid data portion from the packet that has passed the verification. Remove the entire processed packet from the queue to free up space for receiving and processing more data. Clear the total number of verification failures to zero to indicate that the system is in a normal communication state. After processing the current packet, do not continue to parse subsequent data, clear the queue, and return to step (1) to restart periodic data reading. If the integrity of the data is required to be guaranteed, after processing the current data packet, return to step (2) and continue to parse the remaining data in the queue in the current cycle to extract more valid data packets.

Images