A real-time acquisition and intelligent processing method and system for communication data

Abstract

The application discloses a real-time collection and intelligent processing method of communication data, which comprises the following steps: immediately storing the data received by a bottom interrupt service program into a bottom temporary data buffer area in a first-in first-out order; setting a double circular buffer area in an application layer, and adopting a read-write pointer separation mechanism to avoid resource competition; reading the bottom data by the application layer through a periodic task, and storing the data into a first circular buffer area after frame boundary detection and integrity check; carrying out packet sticking segmentation, packet recombination and secondary check on the data in the first circular buffer area, and storing the effective data into a second circular buffer area for use by an upper layer. The system corresponds to a bottom interrupt module, a bottom temporary data buffer area and an application layer module comprising a double circular buffer area and a data processing unit. The application eliminates the processing delay of a traditional method, avoids resource competition during multi-task access, significantly improves the real-time performance and reliability of data reception, has good universality and expansibility, and is easy to realize in an embedded system.

Description

Technical Field

[0001] This invention relates to the field of data communication processing technology, and in particular to a data stream processing method and its supporting system applicable to scenarios where communication links are susceptible to interference in environments such as industrial control, vehicle networks, and the Internet of Things. It is especially suitable for communication methods with signal fluctuation risks, such as CAN bus, serial communication, and Ethernet interface. Background Technology

[0002] In industrial automation, automotive electronic systems, and smart terminal devices, communication buses such as Controller Area Network (CAN) and Universal Asynchronous Receiver / Transmitter (UART) are widely used due to their stability and real-time performance. However, in actual operating conditions, factors such as electromagnetic interference, poor contact, and transient switching of equipment often lead to a decline in communication quality, which in turn causes three common anomalies during data reception: packet merging (multiple packets are sent together), packet breakage (incomplete data frames), and channel noise interference (especially at the beginning and end of communication).

[0003] Traditional data reception and processing methods typically employ an "interrupt + queue" mechanism: the underlying hardware triggers an interrupt when it detects data arrival, the interrupt service routine writes the data to a temporary queue, and the operating system notifies the upper-layer application to read the data when the queue accumulates to a certain quantity or during system scheduling. This model has revealed the following shortcomings in practice: First, the queuing of data and task scheduling notifications can introduce uncontrollable latency, which can easily lead to data loss for data streams that require strict real-time processing. Second, concurrent read and write operations on the same queue by interrupt service routines and applications can easily cause resource contention and disrupt data consistency. Finally, this mechanism lacks the ability to actively identify and verify the structure of data frames. When faced with data concatenation, frame breaks, or random noise, it often discards data directly or parses it incorrectly, seriously affecting the overall stability and reliability of the system.

[0004] Therefore, there is an urgent need to design a method that can process data streams in real time and accurately under unstable communication environments in order to overcome the shortcomings of the traditional methods mentioned above. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time acquisition and intelligent processing of communication data, which solves the problems of packet fragmentation, packet breakage, and noise interference caused by unstable communication links, as well as the problems of poor real-time performance, severe resource contention, and easy data loss in traditional processing methods.

[0006] To address the aforementioned technical problems, this invention provides a method for real-time acquisition and intelligent processing of communication data, comprising the following steps: ① When the underlying interrupt service routine receives communication data, it immediately reads the data and stores it in the underlying temporary data buffer. The underlying temporary data buffer adopts a first-in-first-out queue structure. The interrupt service routine does not perform any waiting or notification operations to the upper layer during the writing process. ② Set up a first circular buffer and a second circular buffer at the application layer. The first circular buffer and the second circular buffer each adopt a read-write pointer separation mechanism. ③ The application layer reads data from the underlying temporary data buffer through periodic tasks, performs frame boundary detection and integrity verification on the read data, and stores the complete data frames that have been verified correctly into the first circular buffer. ④ The application layer reads data from the first circular buffer, performs protocol parsing and reassembly, determines whether there are packet merging, packet splitting or packet breakage, and segments, splices and performs secondary verification on the data frames, and stores the correctly processed valid data frames into the second circular buffer.

[0007] In step ③, frame boundary detection and integrity verification of the data are performed, which is divided into the following steps: (3.1) Obtain the total number of data elements in the underlying temporary data cache; (3.2) When the total number of current data elements is not less than the number of bytes required for a frame of data, read a byte from the head of the buffer area and determine whether the byte matches the preset frame header feature; if it does not match, remove the byte as noise from the buffer area and repeat step (3.2). (3.3) If the frame header matches, continue to read a complete frame of data from the buffer, calculate the sum of the read data, and compare it with the checksum carried at the end of the frame; (3.4) If the sum is consistent with the checksum, it is determined to be a valid data frame. The entire frame data is moved out of the underlying temporary data buffer and stored in the first circular buffer. If they are inconsistent, only the frame header bytes are moved out of the buffer and the process returns to step (3.2) to continue searching for the next frame header.

[0008] The preset frame header feature is two consecutive bytes 0xAA, 0xAA.

[0009] In step ④, after the application layer reads data from the first circular buffer, it parses the data length field according to the predefined frame format, uses the sliding window mechanism to segment the packets, caches the incomplete packets and splices them together after subsequent data arrives, and performs secondary verification on the reassembled complete data frame.

[0010] 5. A real-time acquisition and intelligent processing system for communication data, applied to the real-time acquisition and intelligent processing method for communication data as described in any one of claims 1 to 4, the system comprising: Low-level interrupt module: When communication data is received, it immediately reads the data and stores it in the low-level buffer without performing any waiting or notification operations to the upper layer; The underlying buffer is a first-in-first-out queue used to temporarily store the raw data stream; Application layer module: includes a first circular buffer, a second circular buffer, and a data processing unit.

[0011] The data processing unit includes a frame detection subunit and a data reconstruction subunit.

[0012] The frame detection subunit is used to periodically read data from the underlying buffer, perform frame boundary detection and integrity verification on the data, and store the complete data frames that have been verified correctly into the first circular buffer.

[0013] The data reassembly subunit is used to read data from the first circular buffer, perform packet fragmentation, packet reassembly, and secondary verification on the data, and store the correctly processed valid data frame into the second circular buffer.

[0014] Compared to existing technologies, this invention directly reads data into the buffer via low-level interrupts, eliminating the delays caused by queue waiting and message notifications in traditional methods. This significantly improves the real-time performance of data reception and reduces the risk of data loss. The application layer employs a dual-ring buffer design with separate read and write pointers, effectively avoiding resource contention during concurrent access by multiple tasks and ensuring data consistency and integrity. By using a sliding window mechanism to search the frame header byte by byte and perform verification and validation, it can accurately identify and filter noise interference, correctly segment contiguous data packets, and complete incomplete packets, significantly improving data reliability in unstable communication environments. The algorithm design is simple and efficient, easy to implement in embedded systems, and the frame format and buffer size can be flexibly configured to adapt to various communication protocols, exhibiting strong versatility and scalability.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0017] Figure 1 This is a schematic diagram of the overall data processing flow provided by at least one embodiment of the present invention; Figure 2 This is a detailed flowchart of frame boundary detection and integrity verification in at least one embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of this invention to facilitate a better understanding of this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this invention. The embodiments can be combined with and referenced by each other without contradiction.

[0019] Example 1 A method for real-time acquisition and intelligent processing of communication data includes the following steps: Step S1: When the underlying interrupt service routine receives data, it immediately reads the data and stores it in the underlying buffer without performing any waiting or notification operations; the underlying buffer is a first-in-first-out queue.

[0020] Step S2: Set up a first circular cache and a second circular cache at the application layer. The first circular cache and the second circular cache adopt a read-write pointer separation mechanism, that is, the read pointer and the write pointer operate independently to avoid resource contention when multiple tasks access concurrently.

[0021] Step S3: The application layer reads data from the underlying buffer through a periodic task, performs frame boundary detection and integrity verification on the read data, and stores the complete data frames that have been verified correctly into the first circular buffer.

[0022] Step S4: The application layer reads data from the first circular buffer, performs protocol parsing and reassembly, determines whether there are packet merging, packet splitting, or packet breakage, and segments, splices, and performs secondary verification on the data frames. The correctly processed data frames are then stored in the second circular buffer for use by the upper-layer application.

[0023] Furthermore, step S3, which involves frame boundary detection and integrity verification of the data, specifically includes the following sub-steps: Step S31: Obtain the total number of data elements in the underlying cache. Step S32: When the total number of data elements is not less than the number of bytes required for one frame of data, read one byte from the head of the buffer area and determine whether it matches the preset frame header features; if it does not match, remove the byte as noise from the buffer area and repeat this step. Step S33: If the frame header matches, continue reading a complete frame of data (including length, payload, and checksum fields) from the buffer, calculate its sum and compare it with the checksum at the end of the frame; Step S34: If the checksum is correct, it is determined to be a valid data frame. The entire frame data is moved out of the underlying buffer and stored in the first circular buffer, while the correct frame count is incremented. If the checksum is incorrect, only the frame header byte is moved out of the buffer (slide one byte), and then the process returns to step S32 to continue searching for a new frame header.

[0024] Furthermore, the preset frame header features can be flexibly configured according to the communication protocol. In this embodiment, two consecutive bytes of 0xAA are preferred. The checksum is preferably an additive checksum, but other check methods such as CRC can also be used.

[0025] Furthermore, in step S4, after the application layer reads data from the first circular buffer, it parses the data length field according to the frame format defined by the protocol, uses the sliding window mechanism to segment possible packet fragments, and caches incomplete fragments for later reassembly after subsequent data arrives, ensuring that the data finally stored in the second circular buffer is complete and ordered.

[0026] On the other hand, the present invention also provides a real-time acquisition and intelligent processing system for communication data, comprising: The underlying interrupt module is used to immediately read data and store it in the underlying buffer when data is received, without performing any waiting operations; The underlying cache is used to temporarily store the raw data stream; The application layer module includes a first circular buffer and a second circular buffer, as well as a data processing unit; Specifically, the data processing unit further includes a frame detection subunit and a data reconstruction subunit; Furthermore, the frame detection subunit is used to periodically read data from the underlying buffer, perform frame boundary detection and integrity verification, and store the correct data frames into the first circular buffer. Furthermore, the data reassembly subunit is used to read data from the first circular buffer, perform packet fragmentation, packet reassembly, and secondary verification, and finally store the valid data into the second circular buffer.

[0027] Example 2 This embodiment provides a data processing method for CAN bus communication, aiming to solve the problems of packet sticking, packet breakage, and noise when the bus is unstable. Figure 1 This embodiment demonstrates the overall data processing flow. Figure 2 The process of frame boundary detection and integrity verification is described in detail below: Step 1: Read data directly from the underlying interrupt. like Figure 1 As shown, when the CAN controller receives data and generates an interrupt, the interrupt service routine immediately reads the data bytes from the hardware register and writes them byte by byte into a low-level buffer. This low-level buffer is implemented using a first-in-first-out (FIFO) queue. The interrupt service routine is only responsible for the write operation and does not wait or notify the upper layer, thereby minimizing the interrupt processing time and avoiding data overflow and loss due to interrupt delay.

[0028] Step 2: Initialize the double-loop buffer at the application layer At the application layer, two circular buffers are pre-allocated: buffer A and buffer B. Both buffers maintain independent read and write pointers, allowing multiple tasks (such as periodic read tasks and upper-layer application tasks) to read and write simultaneously without conflicts. Buffer A is used to temporarily store valid data frames that have undergone frame boundary detection and integrity verification, while buffer B is used to store the final data after application layer protocol parsing and reassembly.

[0029] Step 3: The application layer periodically reads the underlying buffer and performs frame synchronization and integrity checks. The application layer calls a data reception and processing function (such as "exc_RdSerialpData") through a periodic task (e.g., executed every 10 milliseconds). This function implements frame boundary detection and integrity verification, and its detailed process is as follows: Figure 2 As shown. Specifically includes: a) Get the current total number of data elements in the underlying cache, "iQueueCount".

[0030] b) Determine if "iQueueCount" is greater than or equal to the number of bytes "iNedRdNum" required for one frame of data. If not, exit the current processing; if so, enter the loop processing.

[0031] c) Read one byte from the header of the underlying buffer and determine if it matches the preset frame header characteristics. In this embodiment, the preset frame header is two consecutive bytes "0xAA" followed by "0xAA" (this can be adjusted according to the actual protocol). If the first byte is not "0xAA", remove it from the queue as noise, update "iQueueCount", and continue the loop.

[0032] d) If the first byte is "0xAA", then read the next byte. If the second byte is also "0xAA", then the frame header is confirmed to be correct. At this point, continue reading the next "iNedRdNum-2" bytes (containing length, data, checksum, etc.) from the buffer into the temporary buffer.

[0033] e) Calculate the sum of the read data (excluding the checksum byte) and compare it with the checksum byte at the end of the frame. If they are equal, the data is considered a valid frame. The entire frame is removed from the underlying buffer and copied to buffer A, and the correct frame count is incremented by 1. If they are not equal, only the first byte of the frame header ("0xAA") is removed from the queue (equivalent to sliding one byte), and then the process returns to step c to continue searching for a new frame header.

[0034] f) Repeat the above process until the underlying buffer has less than one frame of data remaining, then exit the loop.

[0035] This step, through a sliding window and dual check (frame header + checksum) mechanism, can effectively filter out noise, correctly identify frame boundaries, and send the complete data frame into buffer A.

[0036] Step 4: The application layer processes the data in cache A. The application layer reads data frames from buffer A and parses the data content according to the format defined by the protocol. Due to potential communication instability, buffer A may contain multiple consecutive data packets (packet merging) or incomplete data packets (packet fragmentation). The application layer segments the data using the length field in the frame format. If the current data length exceeds one frame, it is segmented into multiple frames according to the length; if the data length is less than one frame, it is temporarily buffered and reassembled when the next data cycle arrives. The reassembled data frame undergoes secondary verification (such as other verification methods specified by the protocol) and is then stored in buffer B for use by upper-layer applications.

[0037] Through the above steps, this embodiment achieves reliable reception and processing of unstable communication data, effectively solving the problems of packet fragmentation, packet breakage, and noise.

[0038] In summary, the method of this invention includes: after receiving data, the underlying interrupt service routine immediately stores it in the underlying temporary data buffer, storing it in a first-in-first-out (FIFO) order; the application layer sets up a dual-loop buffer, employing a read / write pointer separation mechanism to avoid resource contention; the application layer reads the underlying data through periodic tasks, performs frame boundary detection and integrity verification (sliding window to find frame header features + checksum verification), and then stores it in the first loop buffer; the data in the first loop buffer undergoes packet fragmentation, packet reassembly, and secondary verification, and the valid data is stored in the second loop buffer for use by the upper layer. The corresponding system includes: a underlying interrupt module, a underlying temporary data buffer, and an application layer module containing a dual-loop buffer and data processing units (frame detection subunit and data reassembly subunit). Therefore, this invention eliminates the processing delay of traditional methods, avoids resource contention during multi-task access, significantly improves the real-time performance and reliability of data reception, has good versatility and scalability, and is easy to implement in embedded systems.

[0039] Those skilled in the art will understand that the above embodiments can be modified in form and detail in practical applications without departing from the spirit and scope of the invention.

Claims

1. A method for real-time acquisition and intelligent processing of communication data, characterized in that: Includes the following steps: ① When the underlying interrupt service routine receives communication data, it immediately reads the data and stores it in the underlying temporary data buffer. The underlying temporary data buffer adopts a first-in-first-out queue structure. The interrupt service routine does not perform any waiting or notification operations to the upper layer during the writing process. ② Set up a first circular buffer and a second circular buffer at the application layer. The first circular buffer and the second circular buffer each adopt a read-write pointer separation mechanism. ③ The application layer reads data from the underlying temporary data buffer through periodic tasks, performs frame boundary detection and integrity verification on the read data, and stores the complete data frames that have been verified correctly into the first circular buffer. ④ The application layer reads data from the first circular buffer, performs protocol parsing and reassembly, determines whether there are packet merging, packet splitting or packet breakage, and segments, splices and performs secondary verification on the data frames, and stores the correctly processed valid data frames into the second circular buffer.

2. The real-time acquisition and intelligent processing method and system for communication data as described in claim 1, characterized in that: In step ③, frame boundary detection and integrity verification of the data are performed, which is divided into the following steps: (3.1) Obtain the total number of data elements in the underlying temporary data cache; (3.2) When the total number of current data elements is not less than the number of bytes required for a frame of data, read a byte from the head of the buffer area and determine whether the byte matches the preset frame header feature; if it does not match, remove the byte as noise from the buffer area and repeat step (3.2). (3.3) If the frame header matches, continue to read a complete frame of data from the buffer, calculate the sum of the read data, and compare it with the checksum carried at the end of the frame; (3.4) If the sum of the data and the checksum are consistent, the data is determined to be a valid data frame. The entire data frame is moved out of the underlying temporary data buffer and stored in the first circular buffer. If they are inconsistent, only the frame header bytes are moved out of the buffer and the process returns to step (3.2) to continue searching for the next frame header.

3. The real-time acquisition and intelligent processing method and system for communication data as described in claim 2, characterized in that: The preset frame header feature is two consecutive bytes 0xAA, 0xAA.

4. The real-time acquisition and intelligent processing method and system for communication data as described in claim 1, characterized in that: In step ④, after the application layer reads data from the first circular buffer, it parses the data length field according to the predefined frame format, uses the sliding window mechanism to segment the packets, caches the incomplete packets and splices them together after subsequent data arrives, and performs secondary verification on the reassembled complete data frame.

5. A real-time acquisition and intelligent processing system for communication data, applied to the real-time acquisition and intelligent processing method for communication data as described in any one of claims 1 to 4, characterized in that: The system includes: Low-level interrupt module: When communication data is received, it immediately reads the data and stores it in the low-level buffer without performing any waiting or notification operations to the upper layer; The underlying buffer is a first-in-first-out queue used to temporarily store the raw data stream; Application layer module: includes a first circular buffer, a second circular buffer, and a data processing unit.

6. The real-time acquisition and intelligent processing system for communication data as described in claim 5, characterized in that: The data processing unit includes a frame detection subunit and a data reconstruction subunit.

7. The real-time acquisition and intelligent processing method and system for communication data as described in claim 6, characterized in that: The frame detection subunit is used to periodically read data from the underlying buffer, perform frame boundary detection and integrity verification on the data, and store the complete data frames that have been verified correctly into the first circular buffer.

8. The real-time acquisition and intelligent processing method and system for communication data as described in claim 6, characterized in that: The data reassembly subunit is used to read data from the first circular buffer, perform packet fragmentation, packet reassembly, and secondary verification on the data, and store the correctly processed valid data frame into the second circular buffer.

Images