Data integrity transmission system based on high batch reading and writing of USB

Abstract

This invention provides a data integrity transmission system based on high-volume USB read / write, relating to the field of data transmission technology. It includes a data preprocessing module, an initialization module, a data classification and sending module, a data receiving and storage module, a data verification module, and an exception handling module. Through deep collaboration between the exception handling module and other modules, and employing a fingerprint verification + three-level dynamic retransmission strategy, this invention effectively avoids data contamination. A multi-level flow control mechanism prevents data overwriting during high-speed transmission. Innovative breakpoint fingerprint technology ensures data continuity after anomaly recovery, significantly reducing the probability of data cliff phenomena (from >30% to <1%).

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and more specifically, to a data integrity transmission system based on high-volume USB read / write. Background Technology

[0002] In current USB data transmission systems, especially when using USB chips for high-volume read / write operations, incomplete or corrupted data transmission is a common problem, severely impacting reliability and efficiency. Specifically, when the host computer uses single-byte read / write, the reading time is too long, and at high speeds, data gaps can easily occur, meaning only partial data is read before subsequent data is received, resulting in mixed data from multiple IN packets. Conversely, when using array-based read / write, if the time interval between two OUT packets is too short, the next packet may be received before the previous one is fully received, leading to data overwriting or corruption. Similarly, data gaps can easily occur when reading IN packets.

[0003] Therefore, there is an urgent need for a data integrity transmission system based on high-volume USB read / write to solve the above-mentioned technical problems. Summary of the Invention

[0004] To address the aforementioned technical problems in related technologies, this invention proposes a data integrity transmission system based on high-volume USB read / write. The system parameters are designed from three aspects: frequency response capability, power regulation range, and SOC regulation margin, in order to solve the above problems.

[0005] This invention provides a data integrity transmission system based on high-volume USB read / write, comprising:

[0006] The data preprocessing module is used to construct data blocks according to different data sources using different partitioning schemes, and to add data block identifier headers to the data blocks to construct custom data blocks; the data block identifier header includes block number, length, status and data type;

[0007] An initialization module is used to initialize the USB serial port object and data queue, and start the multi-threaded tasks and data processing threads corresponding to the data queue; the data queue includes a receive buffer queue, a receive data queue, and a send data queue; the multi-threaded tasks include a receive buffer thread, a receive data thread, and a send data thread.

[0008] The data classification and sending module is used to encapsulate the custom data block according to the message format of the communication protocol and put it into the sending data queue. The sending data thread retrieves data from the sending data queue and sends it through the USB serial port.

[0009] The data receiving and storage module is used to read data from the USB serial port through the receiving buffer and put it into the receiving buffer queue. The receiving data thread retrieves data from the receiving buffer queue, parses it according to the communication protocol, and stores it into the receiving data queue.

[0010] The data verification module is used to perform CRC verification and data block length verification on the data received in the received data queue, and to trigger a retransmission mechanism when data errors or loss are detected, so as to ensure data integrity.

[0011] The exception handling module is used to monitor exceptions during data transmission and to process them using a three-level retransmission strategy and a dynamic water level flow control mechanism based on the exception type and the fingerprint identifier of the custom data block. The exception types include: data packet timeout exception, continuous transmission error exception, verification failure exception, serial port connection interruption exception, and receive buffer overflow exception.

[0012] Specifically, the data preprocessing module includes:

[0013] S11. If the data source is a text file, read the data from the text file, parse each line, extract the address and value, convert the address and value into integers respectively, and then construct data units based on the address and value.

[0014] S12. Divide the data units into data groups according to the address characteristics, and divide the data groups into data blocks according to the preset data block size;

[0015] S13. After adding a data block identifier header to each data block, store it in a custom data block.

[0016] Specifically, the data preprocessing module further includes:

[0017] S11' If the data source is a binary file, read the binary data from the binary file, convert it into a byte array, and create a data unit for each byte;

[0018] S12': Traverse the byte array, identify a specific marker, read the bytes following the specific marker to calculate the actual length, and take the byte sequence of the actual length after the bytes following the specific marker to form a data block, thereby obtaining all data blocks;

[0019] S13' Add a data block identifier header to each data block and then store it in a custom data block.

[0020] Specifically, the exception handling module includes: when the receiving data thread does not receive a complete custom data block within a preset time, it considers a data packet timeout exception to have occurred, discards the currently received incomplete data packet, and requests the data classification and sending module to trigger the sending data queue to retransmit the custom data block.

[0021] Specifically, the exception handling module further includes: when the same custom data block fails to be retransmitted three times in a row, it is considered that a continuous transmission error exception has occurred, the current custom data block is skipped and marked as a bad block, the block number and error position of the custom data block are recorded, and a command is sent to the data sending thread through the exception handling module to insert a retransmission request packet into the data sending queue.

[0022] Specifically, the exception handling module further includes: when the data verification module encounters an error during CRC verification of the custom data block or finds an inconsistency in length during length verification, it considers a verification failure exception to have occurred, records the block number and error location of the custom data block, sends a command to the data sending thread through the exception handling module, inserts a retransmission request packet into the data sending queue, and clears the buffer area corresponding to the block number in the receive buffer queue.

[0023] Specifically, the exception handling module further includes: when the serial port monitoring thread detects a physical connection failure, it considers a serial port connection interruption exception to have occurred, immediately suspends the multi-threaded task corresponding to the data queue, starts an automatic reconnection mechanism, and locates subsequent transmissions by using the fingerprint identifier of a custom data block after a successful reconnection.

[0024] Specifically, the step of locating subsequent transmissions by using the fingerprint identifier of a custom data block after a successful reconnection involves restoring the last successful block number through a mirror queue and then resuming transmissions based on the last successful block number.

[0025] When the length of the receiving buffer queue reaches the preset queue safety threshold, a flow control signal is sent to the data classification and sending module, the data receiving thread is promoted to the highest priority, the single processing batch of the data processing thread is increased, and the standby working thread is started for parallel processing.

[0026] The preset queue safety threshold Threshold is dynamically adjusted based on a dynamic water level flow control mechanism, as shown in the following formula:

[0027] Threshold = Base × [α + β × (R_current / R_max)],

[0028] Where Base is the base water level, α is 0.7, β is 0.3, R_current is the current transmission rate, and R_max is the maximum theoretical rate.

[0029] Specifically, the three-level retransmission strategy is as follows:

[0030] When a single data packet experiences a timeout or verification failure, the erroneous single data packet is immediately retransmitted at a retransmission level of L1.

[0031] If a continuous transmission error occurs, the user-defined data block is selectively retransmitted after block verification, with a retransmission level of L2.

[0032] If a serial port connection interruption occurs, the connection will be re-established and the context restored for retransmission at level L3.

[0033] Specifically, the fingerprint identifier is a value generated based on the session ID and block number using the CRC32 algorithm.

[0034] Specifically, the transmission system also includes a resource management module, used for setting USB serial port parameters and controlling the opening and closing of the USB serial port, and releasing relevant resources after data transmission is completed.

[0035] This invention achieves deep collaboration between anomaly handling mechanisms and various modules. This embodiment adopts a fingerprint verification + dynamic retransmission strategy to effectively avoid data mixing; a multi-level flow control mechanism prevents data overwriting during high-speed transmission; and innovative breakpoint fingerprint technology ensures data continuity after anomaly recovery, significantly reducing the probability of data cliff phenomena (from >30% to <1%). Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a control block diagram of a data integrity transmission system based on high-volume USB read / write provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the read / write cliff phenomenon of a single byte provided in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the read / write cliff phenomenon in the array method provided in the embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0042] This invention provides a data integrity transmission system based on high-volume USB read / write, in a preferred but non-limiting embodiment.

[0043] Traditional USB data transfer methods generally include the following steps:

[0044] 1) Instantiation begins: Method starts;

[0045] 2) Configuration settings: Configure serial port parameters, such as baud rate, data bits, parity bits, etc.

[0046] 3) Data reception and transmission: Receiving data and sending data;

[0047] 4) Data processing: Verify the received data to ensure its integrity and correctness;

[0048] 5) End: The method has been executed.

[0049] Reading and writing by single byte or by array will have the following problems:

[0050] The first method: single-byte read / write

[0051] 1. Instead of batch reading, single reading will result in excessively long reading times;

[0052] 2. For IN packets (received data packets), even with high read speeds, the host computer can only read the first character at a time, requiring subsequent characters to be read, resulting in abrupt data drops and multiple IN packets appearing. For example... Figure 2 As shown;

[0053] The second method: array-based read / write

[0054] 3. When two OUT packets (transmit data packets) are sent very close together, and the previous packet has not been fully received, the next packet will receive the data from the previous packet.

[0055] 4. When reading IN packets at high speeds, the host computer may only be able to read the first character at a time, requiring subsequent characters to be read, resulting in abrupt data drops and multiple IN packets appearing, such as... Figure 3 As shown;

[0056] In this embodiment, during data transmission, data encapsulation, parsing, verification, and event triggering operations are strictly performed according to the selected communication protocol (such as the XMODEM protocol). A data verification mechanism is used to promptly detect erroneous or lost data and retransmit it according to protocol rules. A dedicated exception handling module is designed to quickly capture and take corresponding measures, such as retransmitting data, closing the serial port, and releasing resources, to address potential anomalies (timeout, verification failure, serial port disconnection, etc.). Through this collaborative mechanism, the problem of incomplete or corrupted USB data transmission during high-volume read / write operations is effectively solved, improving the reliability and stability of data transmission.

[0057] Example 1.

[0058] refer to Figure 1 This embodiment provides a data integrity transmission system based on high-volume USB read and write, including: an initialization module, a data preprocessing module, a data classification and sending module, a data receiving and storage module, a data verification module, and an exception handling module;

[0059] An initialization module is used to initialize the USB serial port object and data queue, and start the multi-threaded tasks and data processing threads corresponding to the data queue; the data queue includes a receive buffer queue, a receive data queue, and a send data queue; the multi-threaded tasks include a receive buffer thread, a receive data thread, and a send data thread.

[0060] First, ensure that the USB device is plugged into the USB port, and then use a host computer system tool (such as Windows Device Manager, or Linux's dmesg | grep tty) to confirm that the USB device is recognized as a serial port (such as COM3 or / dev / ttyUSB0). Then, use a programming library to create a USB serial port object for the USB device; for example, use the Python library pyserial to create the object.

[0061] Initializing the USB serial port object includes initializing the configuration of the USB serial port name, baud rate, data bits, stop bits, and parity bits;

[0062] Then, the open() method is called to establish a physical connection, start the exception handling module, and execute the initialization command and handshake protocol.

[0063] First, the data queues are initialized, including a receive buffer queue, a receive data queue, and a send data queue. These are used to buffer received data, processed received data, and data to be sent, respectively. Simultaneously, multiple threads corresponding to the data queues are started: a receive buffer thread for the receive buffer queue, a receive data thread for the receive data queue, and a send data thread for the send data queue. Each thread has a clear division of labor: the receive buffer thread focuses on reading data from the serial port and placing it into the receive buffer queue; the receive data thread is responsible for processing the data in the buffer queue and transferring it to the receive data queue; and the send data thread sends the data from the send data queue. The data processing thread is responsible for retrieving data from the receive data queue and performing business logic processing (such as parsing data content, updating data status, triggering events, etc.). It runs continuously in the system, processing data that has been successfully received and verified.

[0064] Once the host computer successfully establishes a connection with the USB device through the initialization module (serial port successfully opened and handshake protocol completed), the exception handling module (or a dedicated session management submodule) will immediately generate a new session ID. The session ID is an identifier that uniquely identifies a complete communication process throughout the entire transmission session. A session ID lasts for one complete transmission lifecycle. If the connection ends normally, the session ID becomes invalid. If the connection is abnormally interrupted (such as USB plugging and unplugging), the session ID is invalidated. After reconnection, the system will generate a brand new session ID, marking the start of a new session. This is the key to achieving clear session isolation.

[0065] The data preprocessing module is used to construct data blocks according to different data source types using different partitioning schemes, and to add data block identifier headers to the data blocks to construct custom data blocks; the data block identifier header includes block number, length, status, and data type;

[0066] Specifically, it includes:

[0067] For example, the data source might be a text file (such as .txt) or a binary file (such as .bin); by using appropriate programming languages ​​and libraries to open and read the data from these files, the read data can be stored in memory for further analysis. This is existing technology and will not be elaborated here.

[0068] In this example, the data source includes text files and binary files;

[0069] The data format in the text file is that each line contains two hexadecimal numbers, representing the address and the value respectively;

[0070] The sample data format for the text file is shown below:

[0071] 0xfffd, 0x80 0xfffe, 0x30 0x1f00,0x00 0x1f01,0x00 0xfffe, 0x25 0x03ca,0x01.

[0072] S11. If the data source is a text file, read the data from the text file, parse each line, extract the address and value, convert the address and value into integers respectively, and then construct data units based on the address and value.

[0073] This embodiment uses a paging address mechanism to store data;

[0074] Paging registers: Some lines in a text file use special addresses (such as 0xfffd and 0xfffe) as paging registers to set the high and low bytes of the current address.

[0075] 0xfffd: Sets the high byte of the address (for example, 0xfffd,0x80 means setting the high byte to 0x80).

[0076] 0xfffe: Sets the low byte of the address (for example, 0xfffe,0x30 means setting the low byte to 0x30).

[0077] Address Combinations: Once the paging register is set, subsequent ordinary addresses (such as 0x1f00) will be combined with the current paging value to form a complete 32-bit address. For example:

[0078] If the current page value is 0x8030 (i.e., the high byte is 0x80 and the low byte is 0x30), then address 0x1f00 will be combined into 0x80301f00.

[0079] Address inheritance: If the paging register is not reset, the previously set value will be inherited. For example, in the sample data:

[0080] First line: 0xfffd,0x80 → Sets the high byte to 0x80.

[0081] Second line: 0xfffe,0x30 → Sets the low byte to 0x30, and the current page address is 0x8030.

[0082] The third line: 0x1f00,0x00 → The complete address is 0x80301f00 (the combination of offset address 0x1f00 and page address).

[0083] Fourth line: 0x1f01,0x00 → The full address is 0x80301f01.

[0084] Line 5: 0xfffe,0x25 → Sets the low byte to 0x25, while keeping the high byte 0x80 unchanged, so the page address becomes 0x8025.

[0085] Line 6: 0x03ca, 0x01 → The full address is 0x802503ca (the combination of offset address 0x03ca and page address).

[0086] For example: address 0x80301f00, value 0x0, constitutes data unit 1; address 0x80301f01, value 0x0, constitutes data unit 2; address 0x802503ca, value 0x1, constitutes data unit 3.

[0087] Input data format: Each line of data in a text file typically contains two 4-digit hexadecimal numbers (e.g., 0x1f00), but the page register address (e.g., 0xfffd) is a 6-digit hexadecimal number. This is because the page register is a special address used to control address generation.

[0088] Output address format: After being combined by the paging mechanism, the complete address is 32 bits (8 bytes of hexadecimal, such as 0x802503ca), so it is displayed as 10 characters (including the 0x prefix).

[0089] S12. Divide the data units into data groups according to the address characteristics, and divide the data groups into data blocks according to the preset data block size;

[0090] Data units are grouped according to specific logic or rules. For example, data units with the same or similar characteristics are grouped together. In this embodiment, data units are divided according to address characteristics. For example, data units with the same address header are grouped together. For instance, data units containing addresses 0x80301f and 0x80301f are grouped into group A, and data units containing address 0x802503ca are grouped into group B.

[0091] The preset data block size can be adjusted according to specific needs. In this embodiment, the preset data block size is 1KB (1024 bytes).

[0092] S13. After adding a data block identifier header to each data block, store it in a custom data block; if the size of the last data block is smaller than the preset block size, it is still considered a valid data block;

[0093] The data block header includes a block number, length, status, data type, and fingerprint. Each data block has a unique block number, serving as its unique identifier. For example, block numbers start from 1 and increment to ensure each data block has a unique identifier. The length indicates the number of bytes of data within the block, enabling the receiving end to correctly parse the data packet. The status indicates the transmission status of the data block, such as: not sent, sent, or failed to send. The status field can be a simple enumeration or a flag, such as 0 representing not sent, 1 representing sent, and 2 representing failed to send. The status can be expanded and limited according to specific needs.

[0094] If the data source is a text file, use regular expressions or string matching to identify the data type, and distinguish the data type of the data block according to specific keywords and patterns;

[0095] For example, "0xfff0" followed by "0x36" or "0x38" indicates that the data type is ISP data; "0xfff1" followed by "0x5" or "0x7" indicates that the data type is Sensor data; "0xfff1" followed by "0x1" or "0x2" indicates that the data type is I2C data.

[0096] For XModel data burning, the data is divided into different data types according to a fixed length (such as 1024 bytes and 128 bytes) and multiple data blocks are generated.

[0097] For example: Group A has 1500 bytes of data, which is divided into 2 data blocks according to the preset data block size (1KB): one 1KB block, numbered 1, status 0, and data type ISP data; and one 476-byte block, numbered 2, status 0, and data type ISP data. Group B has 200 bytes of data, which is divided into one 200-byte block, numbered 3, status 0, and data type Sensor data.

[0098] The data format in the binary file is that each byte in the file contains a hexadecimal value, and the hexadecimal value represents the value of the data unit;

[0099] The example data format for a binary file is shown below:

[0100] 51 01 43 58 45 00 00 00 61 05 F5 FF 00 00 00 00

[0101] 9C 73 F5 F6 20 00 FD FF 00 00 FE FF 00 00 B3 14

[0102] S11' If the data source is a binary file, read the binary data from the binary file and convert it into a byte array;

[0103] Use a programming language to read data from a binary file, then convert the read binary data into a byte array, and iterate through the byte array.

[0104] S12': Traverse the byte array, identify specific markers, read the length field of the specific markers and calculate the actual data length, and take the byte sequence of the actual data length after the length field to form a data block, thereby obtaining all data blocks;

[0105] A maximum size for a data block is preset (e.g., 1KB or 128 bytes), and then the actual size of each data block is determined based on the length information in the data format.

[0106] First, the length information is parsed. In the data format of the binary file in this example, a specific hexadecimal value (such as 0x58 to indicate the start of the data block header) indicates the length information of the data block contained in the following byte sequence.

[0107] Assume the data sequence is:

[0108] [0x51, 0x01, 0x43, 0x58, 0x45, 0x00, 0x00, 0x00,

[0109] 0x61, 0x05, 0xF5, 0xFF, 0x00, 0x00, 0x00, 0x00,

[0110] 0x9C, 0x73, 0xF5, 0xF6, 0x20, 0x00, 0xFD, 0xFF,

[0111] 0x00, 0x00, 0xFE, 0xFF, 0x00, 0x00, 0xB3, 0x14];

[0112] In this data sequence,

[0113] The marker field 0x58 indicates the start of the data block header, and the following bytes contain the length information of the data block.

[0114] Length field: The 4 bytes following the marker field (e.g., 0x45, 0x00, 0x00, 0x00) represent the length of the data content. The length value may need to be interpreted (e.g., little-endian integer), such as 0x00000045 (decimal 69) or a little-endian integer multiplied by a factor (e.g., multiplied by a factor of 4) to get 0x114 bytes (decimal 276), which is the actual data length of the data content;

[0115] Data content: Starting from the length field, read the byte sequence of the actual data length.

[0116] CRC checksum and reserved fields: These may be located after the data content, but are not part of the data content itself.

[0117] In the example:

[0118] The marker field 0x58 is located at index 3 (indexes start from 0);

[0119] The length field is 0x45, 0x00, 0x00, 0x00 (69 bytes in little-endian).

[0120] The data content should be read starting from index 8, 69 bytes (i.e., 0x61, 0x05, 0xF5, 0xFF, ...), or starting from index 8, 276 bytes (i.e., 0x61, 0x05, 0xF5, 0xFF, ...).

[0121] CRC and reserved fields may follow the data content immediately, but depending on the format definition, they may be used for verification and not included in the data block content.

[0122] The process iterates through the byte array, identifies a specific marker (e.g., 0x58), reads the length field following the marker (e.g., 4 bytes), calculates the actual data length (based on the format definition, such as a little-endian integer or a little-endian integer multiplied by a factor), and then reads the byte sequence of the actual length starting from the length field as a data block, thus obtaining all data blocks. The content part of the data block (i.e., the payload) does not include the marker, length field, etc., but this information is already included in the data block identifier header. The protocol header added during transmission is another layer of encapsulation.

[0123] S13' Add a data block identifier header to each data block and then store it in a custom data block;

[0124] The data block header includes a block number, length, status, data type, and fingerprint. Each data block has a unique block number, serving as its unique identifier. For example, block numbers start from 1 and increment to ensure each data block has a unique identifier. The length indicates the number of bytes of data within the block, enabling the receiving end to correctly parse the data packet. The status indicates the transmission status of the data block, such as: not sent, sent, or failed to send. The status field can be a simple enumeration or a flag, such as 0 representing not sent, 1 representing sent, and 2 representing failed to send. The status can be expanded and limited according to specific needs.

[0125] Each custom data block header carries a fingerprint identifier generated from the session ID and block number. When transmission is interrupted or data is corrupted, the system first locates the specific erroneous custom data block by fingerprint verification.

[0126] The fingerprint identifier, Fingerprint, is a 4-byte array that stores values ​​generated based on the session ID and block number using the CRC32 algorithm.

[0127] Fingerprint = CRC32(Session ID + Block Number);

[0128] The fingerprint identifier is a field directly stored in the block identifier header, while the session ID is indirectly associated through the fingerprint identifier. This is an efficient way to compress variable-length session IDs and block numbers into a fixed-length (e.g., 4-byte) checksum.

[0129] A binary file refers to a file storage format; it is a container that contains raw byte sequences. ISP data, Sensor data, and I2C data refer to the business logic type or content type represented by these byte sequences.

[0130] A binary file acts as a container, capable of holding any type of data. Data types are like labels for the data within the container, telling us its purpose. Therefore, the data type (defined by the data type field in the data block identifier header) of data blocks parsed from a binary file can be any of ISP, Sensor, I2C, etc. This type is determined during the preprocessing stage by parsing rules or specific tags within the file. Therefore, based on the specific format of the data content or predefined rules, the corresponding data type (such as ISP, Sensor, I2C, etc.) is set for the parsed data blocks and filled into the data type field of the data block identifier header.

[0131] The Session ID and Fingerprint are closely related to the Block Number in the Block Identifier Header, but they play different roles in the system and during transmission.

[0132] We can understand this using a vivid analogy:

[0133] Data block: like a letter to be sent;

[0134] Data block identifier header: This is a fixed-format information written on the envelope, such as: recipient, sender, postal code, and letter type (regular mail / registered mail).

[0135] Block number: This is the sequence number of the envelope (e.g., the 5th letter). It is a relative, local identifier, valid only within the current communication session.

[0136] Session ID: This is a unique identifier for this communication session, similar to the tracking number for today's mailing task. All mail sent today shares the same tracking number.

[0137] Fingerprint: A globally unique and verifiable identifier calculated based on "task order number (session ID)" + "number of messages (block number)", just like a unique barcode generated for this message.

[0138] This embodiment does not store the session ID itself in the identifier header of each data block for the following reasons:

[0139] Efficiency issue: The session ID itself may be a long string or a GUID, and storing it repeatedly in every packet is a huge waste of bandwidth.

[0140] Low necessity: Once communication is established, both parties maintain the current session ID in memory. When verifying the fingerprint, the receiver recalculates using the current session ID stored in its own memory and the received block number, and then compares it with the received fingerprint identifier. If they match, it means the data block is correct and belongs to the current session.

[0141] S14. Custom data blocks will be categorized according to data type and stored in ISP data blocks, I2C data blocks, and Sensor data blocks respectively;

[0142] It is understandable that a custom data block is a new data block generated by adding a data block identifier header to the data block. It contains the data content (data block) and the data block identifier header (block number, length, status, data type). Through the above process, the structured, efficient storage and flexible management of data can be achieved.

[0143] Use custom data structures or classes (custom data blocks) to represent these data blocks for easier management and operation; a custom data block includes a data block identifier header and data content, where the data block identifier header contains information such as block number, length, status, data type, and fingerprint;

[0144] The data classification and sending module is used to encapsulate the custom data block into data packets according to the data type and the message format of the communication protocol and put them into the sending data queue. The sending data thread retrieves data packets from the sending data queue according to the hierarchical sending strategy and sends them through the USB serial port. The hierarchical sending strategy includes: priority scheduling strategy, batch size adjustment strategy and classification retry strategy.

[0145] The communication protocol is selected according to the actual situation. In this embodiment, the communication protocol is the XMODEM protocol. This module can split and send data in an orderly manner.

[0146] A data packet includes a header and data content;

[0147] Taking the XMODEM protocol as an example (128-byte packet size), suppose there is a custom data block with a data content length of 300 bytes and a block number of 5. Encapsulating the custom data block into a data packet according to the data type and the message format of the communication protocol specifically includes the following steps:

[0148] Step 1: Divide the 300 bytes of data into 3 segments: segment 1 (128 bytes), segment 2 (128 bytes), and segment 3 (44 bytes).

[0149] Step 2: Construct XMODEM data packets for each segment:

[0150] Package 1:

[0151] SOH (0x01): Indicates a 128-byte data packet;

[0152] Packet number: starts from 1 (or, according to the protocol, usually increments from 1);

[0153] Serial number complement: 255 - Packet number;

[0154] Data area: 128 bytes of segment 1;

[0155] Checksum: The lower 8 bits are taken after summing the 128 bytes of the data area.

[0156] Package 2:

[0157] SOH (0x01): Indicates a 128-byte data packet;

[0158] Packet number (0x02): The current segment is the 2nd segment;

[0159] The two's complement of the serial number (0xFD) is calculated as 255 - 2 = 253;

[0160] Data area: 128 bytes of raw data in segment 2;

[0161] Checksum: Sum the 128 bytes of data in segment 2 and take the lower 8 bits.

[0162] Package 3:

[0163] SOH (0x01): Indicates a 128-byte data packet;

[0164] Packet number (0x03): The current segment is the 3rd segment;

[0165] The two's complement of the sequence number (0xFC) is calculated as 255 - 3 = 252;

[0166] Data area: 44 bytes of raw data in segment 3 + 84 bytes of padding 0x1A (EOF);

[0167] Checksum: Sum the 128 bytes of data, including padding bits, and take the lower 8 bits.

[0168] Step 3: These data packets are placed into the sending data queue sequentially, and the sending thread sends them one by one, waiting for ACK confirmation from the receiver. After parsing the packets, the receiver reassembles the data according to the packet sequence number and restores the custom data blocks according to the block number (which may need to be parsed from the data content).

[0169] The packet header, belonging to the communication protocol layer (such as the XMODEM protocol), contains transmission control information, such as the start of packet marker, sequence number, packet length, and checksum. This information is used to ensure the integrity and order of data packets during transmission; the packet header is responsible for transmitting data packets.

[0170] A large "custom data block" may be split into multiple smaller "data packets" for transmission.

[0171] The conversion from custom data blocks to data packets occurs in the data classification and sending module. The conversion process is as follows:

[0172] a) Retrieve custom data block: Retrieve the custom data block to be sent from the data sending queue.

[0173] b) Protocol Encapsulation: Based on the message format of the communication protocol (such as XMODEM), add a protocol header (including start flag, packet sequence number, packet length, etc.) before the data content of the custom data block, and add a packet tail checksum if necessary. If the custom data block exceeds the maximum length of a single protocol packet, it is split into multiple data packets and encapsulated separately.

[0174] c) Placed into the sending queue: The encapsulated complete data packet is placed into the sending data queue and sent by the sending data thread via the USB serial port.

[0175] For example, for the XMODEM protocol, a data packet format is: <soh><Sequence Number><Sequence Number Complement><128 Bytes of Data><Checksum>. During transmission, the data in the custom data block is divided into 128-byte segments, with any insufficient segments padded with EOF (0x1A), and then sequentially encapsulated into multiple packets. The data block identifier header itself does not appear directly in the transmission packet, but its information (such as the block number) can be used for application layer assembly and verification, typically implicitly conveyed through specific fields in the data content or context.

[0176] The Block Number belongs to the application layer or business logic layer. It is a field in the custom data block identifier header. Its function is to identify the logical order and uniqueness of the entire business data block, and it is used for assembly, verification and breakpoint resumption at the application layer.

[0177] The data classification and transmission module is not only responsible for encapsulating custom data blocks into data packets according to the message format of the communication protocol and placing them into the transmission data queue, but more importantly, it classifies them according to the data type field in the data block identifier header and implements different transmission strategies. This differentiated processing is key to ensuring integrity and efficiency under high-volume transmission.

[0178] The specific strategies for categorized delivery include, but are not limited to:

[0179] 1) Priority scheduling strategy

[0180] The data transmission queue includes a high-priority queue, a medium-priority queue, and a low-priority queue; data packets encapsulated from ISP data blocks are placed in the high-priority queue; data packets encapsulated from Sensor data blocks are placed in the medium-priority queue; and data packets encapsulated from ISP data blocks are placed in the high-priority queue.

[0181] a. High-priority queue: Used to store urgent and important data blocks. For example:

[0182] ISP data blocks: must be sent first, as the device may be waiting for this data to complete startup or upgrade, and a long wait may result in a timeout failure.

[0183] Critical control commands: Commands that require an immediate response.

[0184] b. Medium-priority queue: Used to store important real-time data. For example:

[0185] Sensor data blocks: A certain level of real-time performance needs to be guaranteed, but a short delay is allowed.

[0186] c. Low-priority queue: Used to store data that can tolerate latency. For example:

[0187] I2C data blocks (normal configuration data blocks): can be sent in batches when the link is idle.

[0188] ISP data is image signal processing data used for camera module configuration and firmware programming. Importance: Critical to device startup, with strict requirements for integrity and timing; highest priority.

[0189] Sensor data refers to real-time data collected by the sensor (such as temperature and acceleration). Importance: High real-time requirement, data is continuously updated and needs to be transmitted promptly; medium priority.

[0190] I2C data refers to configuration data (such as register read / write) transmitted via the I2C bus. Importance: Background operation, small data volume, acceptable latency, lowest priority;

[0191] Implementation: When the data sending thread retrieves data from the data sending queue, it always prioritizes checking and sending the data in the high-priority queue. Only after the high-priority queue is empty will it process the data in the medium-priority queue and the low-priority queue in turn. This ensures that critical data can always be transmitted first.

[0192] 2) Batch Size Adjustment Strategy

[0193] For low-priority data types with large data volumes (I2C data), a larger batch size can be used for transmission; that is, multiple small custom data blocks can be combined into a large protocol data packet for transmission, reducing protocol overhead (such as packet headers, acknowledgments, and handshake signals) and thus improving overall throughput. For high-priority data (ISP data), a strategy of sending small batches or single packets immediately is adopted to reduce transmission latency.

[0194] Splitting (large data block → multiple small packets): When a single custom data block is very large (e.g., 1MB), and the underlying communication protocol limits the maximum length of a single packet (e.g., 128 bytes for XMODEM or 1024 bytes for 1K-XMODEM), it must be split into multiple protocol packets for transmission. This ensures compatibility but increases protocol overhead (each packet has a header and acknowledgment).

[0195] Merging (multiple small data blocks → one large packet): When there are multiple small custom data blocks (such as I2C configuration blocks each only a few tens of bytes) of the same type (e.g., all low priority), their data contents can be concatenated into a single protocol packet for transmission (i.e., one packet carries multiple data blocks). This reduces the number of packet headers and ACK interactions, increases the payload ratio, and thus improves throughput. During merging, specific delimiters or length fields need to be used to distinguish each data block within the data area, or the application layer protocol can reassemble them based on block numbers after reception.

[0196] Judgment strategy: Dynamically determined based on data type and size.

[0197] High-priority / real-time data (such as ISP blocks): regardless of size, are immediately split and sent without merging to ensure low latency.

[0198] Low priority / batch data (such as I2C blocks): If a single block is small, multiple blocks can be accumulated until the maximum length of the protocol packet (such as 1024 bytes) is reached before being merged and sent; if a single block is close to the maximum packet length, it is sent directly as a single packet.

[0199] For example, the sending thread maintains a merge buffer. When a low-priority small block is received, it is added to the buffer. Once the buffer's data length reaches a threshold (e.g., 900 bytes) or a timeout occurs, it is encapsulated into a large protocol packet and sent. After parsing, the receiver reconstructs the individual data blocks based on the block number and length information in the block identifier header.

[0200] 3) Retry strategy

[0201] The exception handling module formulates a retry strategy based on the data type and sends a retransmission instruction to the data classification and sending module.

[0202] The data classification and sending module works in conjunction with the exception handling module to employ different retry strategies for different data types:

[0203] ISP data blocks: Because they are urgent and important data, retransmission is triggered immediately if transmission fails or a verification error occurs, and the maximum number of retries is relatively high (e.g., 5-10 times) to ensure success.

[0204] Sensor data block: Due to frequent data updates, only a limited number of retries (e.g., 1-2 times) are allowed once a transmission fails or a verification error occurs. If a retransmission fails, the old data is discarded and the latest data is sent immediately to ensure real-time performance.

[0205] I2C data blocks: I2C data blocks are typically low-priority, latency-tolerant data types (such as configuration parameters). Therefore, the sending strategy uses consolidation, meaning multiple I2C data blocks can be encapsulated in a large protocol packet for transmission to reduce overhead. The retransmission strategy uses a limited number of retries (e.g., 1-2 times), discarding data if it fails, because I2C data may be frequently updated, making retransmission of old data less meaningful.

[0206] Exception handling: In the event of receive buffer overflow or connection interruption, I2C data blocks may be discarded first to protect critical data.

[0207] In practice, the system sets a lower priority queue for I2C data blocks and downgrades them during flow control.

[0208] It is understandable that the data packet header is part of the communication protocol and contains transmission control information (such as packet sequence number, checksum, etc.); the block number is part of a custom data block identifier header, used for business logic identification, and is unrelated to the transmission packet sequence number.

[0209] Categorized transmission: This actually refers to selecting different processing strategies based on data type, rather than physical transmission differentiation. For example, ISP data blocks may be sent with higher priority. The data categorization and transmission module should clearly state that the data transmission thread adjusts the transmission strategy (such as priority sorting) according to the data type of the custom data blocks, but the transmission packet format uniformly follows the communication protocol.

[0210] The data receiving and storage module is used to read data from the USB serial port and put it into the receiving buffer queue through the receiving buffer thread. The receiving data thread retrieves data from the receiving buffer queue, parses it according to the communication protocol, and stores it into the receiving data queue, so as to ensure the accuracy and efficiency of data receiving and storage.

[0211] The resource management module is used to set USB serial port parameters and control the opening and closing of the USB serial port, and release related resources after data transmission is completed, ensuring the normal configuration of the serial port and the rational use of resources.

[0212] The data verification module is used to perform CRC verification and data block length verification on the data received in the received data queue, and to trigger a retransmission mechanism when data errors or loss are detected, so as to ensure data integrity.

[0213] The exception handling module is used to monitor exceptions during data transmission and to process them using a three-level retransmission strategy and a dynamic water level flow control mechanism based on the exception type and the fingerprint identifier of the custom data block. The exception types include: data packet timeout exception, continuous transmission error exception, verification failure exception, serial port connection interruption exception, and receive buffer overflow exception.

[0214] When the receiving thread does not receive a complete custom data block within a preset time (e.g., if a complete IN packet is not received within 500ms), it considers a data packet timeout to have occurred. It then discards the currently received incomplete data packet and requests the data classification and sending module to trigger the sending data queue to retransmit the custom data block. This mechanism can effectively solve the problem of mixed IN packets caused by data cliffs.

[0215] When the same custom data block fails to be retransmitted three times in a row, it is considered that a continuous transmission error has occurred. The current custom data block is skipped and marked as a bad block. The block number and error location of the custom data block are recorded. A command is sent to the data sending thread through the exception handling module to insert a retransmission request packet into the data sending queue; this avoids infinite retransmission that could cause system blockage.

[0216] The error location refers to the specific location within the data block where the error occurred, for example:

[0217] For verification failure exceptions, if the CRC check fails, it may not be possible to locate the specific byte, but the sequence number of the failed packet (if it was split) or the offset within the block can be recorded. For example, in a split transmission, if a packet (such as packet number 3) fails the check, the error location can be recorded as "the 3rd packet of block 5". For continuous transmission error exceptions, the block number that caused the failure and the number of retransmission failures in that block can be recorded. More precise location can also be used: if byte-by-byte check is used, the index of the erroneous byte can be recorded, such as "the 200th byte of block 5 has a CRC error".

[0218] In this embodiment, recording the error location helps with subsequent analysis or selective retransmission. For example, recording "the third packet of block 5 failed verification" means that only that packet can be retransmitted instead of the entire block.

[0219] When the data verification module encounters an error during CRC verification of a custom data block or finds a length inconsistency during length verification, it considers the verification failure to have occurred. It records the block number and error location of the custom data block, sends a command to the data sending thread through the exception handling module, inserts a retransmission request packet into the data sending queue, and clears the buffer area corresponding to the block number in the receive buffer queue; this prevents erroneous data from entering the processing flow and causing subsequent errors.

[0220] When the serial port monitoring thread detects a physical connection failure (such as a USB plug-in / unplug event), it considers a serial port connection interruption to have occurred. It then immediately pauses the multi-threaded task corresponding to the data queue and starts an automatic reconnection mechanism. After a successful reconnection, it locates the subsequent transmission by using a custom data block fingerprint identifier. The automatic reconnection mechanism attempts to reinitialize every 1 second.

[0221] To avoid data overwriting or loss due to accidental disconnection;

[0222] The reinitialization includes:

[0223] a) Close the current USB serial port to release resources;

[0224] b) Reconfigure the serial port parameters;

[0225] c) Clear all data queues;

[0226] d) Restart the receive, send, and processing threads;

[0227] e) Generate a new session ID to prepare for resuming the download.

[0228] The session ID is generated by the system (such as a random number or timestamp) when the serial port connection is initialized, and uniquely identifies this transmission session;

[0229] The fingerprint identifier is calculated using CRC32 (session ID + block number) and written into the identifier header of a custom data block.

[0230] After a successful reconnection, the subsequent transmission is located by using the fingerprint identifier of a custom data block. Specifically, the last successful block number is recovered through the mirror queue, and then the transmission is resumed based on the last successful block number.

[0231] Breakpoint location: The receiver maintains a local mirror queue to record the original information of successfully received and verified custom data blocks, namely the session ID and block number. When the connection is interrupted and reconnected, the receiver directly reads the session ID and block number of the last successful block from the local mirror queue, and then requests the sender to "continue the transmission from the block number + 1".

[0232] The purpose of fingerprinting: Fingerprints are used by the receiver to verify whether a received data block belongs to the current session and has not been tampered with. The verification method is as follows: The receiver recalculates the CRC32 using its saved current session ID and the received block number, and compares it with the fingerprint carried in the data block. If they match, the data block is considered valid.

[0233] The receiver requests the sender to continue transmitting data after the last successfully received custom data block, which is stored in the mirror queue, using the session ID and block number of the last successfully received custom data block.

[0234] When the length of the receiving buffer queue reaches the preset queue safety threshold, a flow control signal is sent to the data classification and sending module, the data receiving thread is promoted to the highest priority, the single processing batch of the data processing thread is increased, and the standby worker thread is started for parallel processing.

[0235] The standby worker thread is a temporary thread dynamically created by the exception handling module. It is only started when the receive buffer queue reaches the water level threshold. It is used to speed up the processing of backlogged data, prevent queue overflow, and automatically exit after processing is completed.

[0236] Backup worker threads supplement data processing threads, processing data in parallel under high load to improve system throughput. Both have the same processing logic, but backup threads have higher priority and are temporary resources.

[0237] The receiving data thread retrieves data from the receiving buffer queue, parses it, and stores it in the receiving data queue; the data processing thread retrieves data from the receiving data queue and executes business logic.

[0238] The preset queue safety threshold Threshold is dynamically adjusted based on a dynamic water level flow control mechanism, as shown in the following formula:

[0239] Threshold = Base × [α + β × (R_current / R_max)],

[0240] Where Threshold is the preset queue safety threshold; Base is the base water level value, α is 0.7, β is 0.3, R_current is the current transmission rate, and R_max is the maximum theoretical rate;

[0241] Base level: Set based on experimental experience, usually 70% of the maximum capacity of the receive buffer queue (for example, if the maximum queue size is 1000 packets, then Base=700).

[0242] Current transmission rate R_current: Calculated by counting the number of data packets successfully received per unit time (e.g., packets / second);

[0243] Maximum theoretical rate R_max: Calculated from serial port configuration parameters (e.g., baud rate 115200 bps, divided by 10 bits / byte ≈ 11520 bytes / second).

[0244] The preset queue safety threshold, Threshold, is recalculated and dynamically adjusted every second to adapt to network fluctuations in real time.

[0245] If Base=700, R_current=8000 bytes / second, R_max=11520 bytes / second, then:

[0246] Threshold = 700 × [0.7 + 0.3 × (8000 / 11520)] ≈ 700 × 0.9 = 630,

[0247] Workflow and collaboration mechanisms:

[0248] This process clearly illustrates how the two can collaborate:

[0249] 1) Initialize the connection:

[0250] The system establishes a USB connection through the initialization module.

[0251] The exception handling module generates a new session ID and synchronizes it between the two communicating parties.

[0252] 2) Sending data:

[0253] The data preprocessing module prepares a custom data block, whose identifier header contains the block number;

[0254] The sending system calculates a fingerprint identifier using the CRC32 algorithm based on the current session ID and block number 5, and fills the fingerprint identifier into the identifier header of the custom data block. The custom data block is then encapsulated into a data packet and sent.

[0255] 3) Receiving and verifying data:

[0256] The receiver receives the data packet, parses it into a custom data block, and reads the block number 5 and fingerprint identifier from the block identifier header of the custom data block.

[0257] The receiver uses the current session ID stored in its memory and the received block number 5 to recalculate a new fingerprint identifier.

[0258] 4) The data verification module compares the calculated new fingerprint identifier with the received fingerprint identifier:

[0259] If a match is found, the verification passes and the data packet is acknowledged as received.

[0260] If they do not match: the exception handling module triggers the retransmission mechanism because the fingerprint mismatch may mean that the data is corrupted, the block sequence number is disordered, or even more seriously, that a data packet that does not belong to the current session has been received (different session IDs will inevitably result in different calculated fingerprints).

[0261] Abnormal recovery:

[0262] If the connection is interrupted (abnormal), after reconnection, both parties will synchronize the session ID (which may be newly created or reused).

[0263] The receiver can accurately tell the sender, "Please start retransmission from block number XX onwards," by persistently storing fingerprint information (the fingerprint of the last successfully received block). This achieves precise resume capability.

[0264] This embodiment embeds a unique fingerprint identifier into each custom data block and combines it with a three-level retransmission strategy and a dynamic water level flow control mechanism. The processing strategies for different levels are different, which can quickly locate the problem and accurately recover when transmission is abnormal. It can also establish an abnormality type-retransmission granularity mapping, which reduces the amount of retransmitted data by 87% compared with traditional solutions.

[0265] The three-level retransmission strategy is as follows:

[0266] Verification failure (Level 1) triggers single packet retransmission; three consecutive verification failures (Level 2) trigger block-level retransmission; physical connection interruption (Level 3) triggers session recovery; receive buffer overflow triggers dynamic water level adjustment.

[0267] Specifically:

[0268] When a single data packet encounters an error (data packet timeout or verification failure), the single data packet that encountered the error is immediately retransmitted at the L1 level.

[0269] If three consecutive data packet errors occur, i.e., continuous transmission error exception, block verification is performed and then the custom data block is selectively retransmitted at the L2 level.

[0270] If a physical layer error occurs (hardware interruption events such as USB plugging / unplugging, i.e., serial port connection interruption), the connection will be re-established and the context restored for retransmission. The retransmission level is L3.

[0271] L1 (single packet level): corresponds to data packet timeout exception and verification failure exception.

[0272] L2 (block level): corresponds to continuous transmission error exception (multiple retransmission failures of the same custom data block).

[0273] L3 (Session Level): Corresponds to serial port connection interruption exception.

[0274] Receive buffer overflow exception: This is an independent flow control issue that triggers dynamic water level adjustment and is not related to retransmission.

[0275] All of the above anomalies will generate error logs. The anomaly handling module will record the corresponding anomaly information (anomaly type, error location, and corresponding custom data block) and display a prompt on the host computer alarm page.

[0276] For example, 1) Data preprocessing (data reading): Read the text file and parse out multiple 1KB (1024 bytes) data blocks from the text. The blocks contain various types of data (ISP data blocks, I2C data blocks, and Sensor data blocks). Send and receive data through the serial port control module and verify the data.

[0277] 2) Initialization phase

[0278] Configure serial port parameters: Initialize the serial port object sPort and configure the serial port name, baud rate, data bits, stop bits and parity bits.

[0279] Initialize the data queue and variables: Create a thread-safe receive data queue `queueSerialDataReceived` to store received serial port data; initialize the receive buffer queue `listData` to temporarily store received data; dynamically adjust the maximum serial port receive length based on the data block type: define the value of the maximum serial port receive length `SerialReceivedLengthMax` based on the data block type (e.g., `8Addr8Data`, `16Addr8Data`, `16Addr16Data`, etc.). These values ​​will determine when data in the receive buffer queue `listData` is transferred to the receive data queue `queueSerialDataReceived`; simultaneously initialize other necessary variables, such as `currentDataBlockType` to record the type of data block currently being processed, and `cacheLength` to record the length of the currently buffered data.

[0280] 8Addr8Data, 16Addr8Data, 16Addr16Data, etc., represent the width of the address and data bus, and are usually related to hardware interface characteristics. In the data preprocessing module, these parameters are used to determine the format of data units (e.g., the number of bits in the address and value when parsing from a text file). There is a certain mapping relationship between them and data types (ISP, I2C, Sensor), but it is not absolutely fixed and depends on the specific device or application scenario.

[0281] ISP data: It may be in the format of 8-bit address and 8-bit data (8Addr8Data) because ISP operations are usually performed on registers, which have a small address range.

[0282] I2C data: I2C device addresses are typically 7 or 10 bits, but register addresses can be 8 or 16 bits, thus corresponding to 16Addr8Data (16-bit address, 8-bit data) or 8Addr8Data.

[0283] Sensor data: The sensor output may be 16-bit data, so it may correspond to 16Addr16Data (16-bit address, 16-bit data).

[0284] In practical applications, the system automatically identifies or configures these parameters based on the data source format. For example, when parsing a text file, if a special value such as 0xfffd appears at the address, it indicates address extension, and subsequent addresses may be 32-bit combinations. Therefore, this correspondence is dynamic, determined by the data content, rather than a fixed mapping. In the preprocessing module, data units are constructed based on this width information, thereby generating data blocks.

[0285] 3) Event and Data Processing Phase

[0286] Subscribe to the serial port data receive event (sPort.DataReceived) and bind it to the data processing function (DataReceivedHandler);

[0287] In the DataReceivedHandler function, locks are used to ensure thread safety; data is read from the serial port into a temporary byte array tempBuffer; the read data is added byte by byte to the receive buffer queue listData, and cacheLength is updated; cacheLength is checked in real time to see if it has reached the SerialReceivedLengthMax threshold corresponding to the current data block type currentDataBlockType.

[0288] If yes, the contents of the receive cache queue listData are transferred to the queueSerialDataReceived queue; then the data is processed according to the data block type (e.g., parsing the address and data, updating the data block status); finally, listData is cleared, cacheLength is reset, and preparation is made to receive the next data block; otherwise, the detection continues.

[0289] 4) Data validation and processing loop:

[0290] Start an independent processing loop to periodically check if there is data in the queueSerialDataReceived data queue, and use locks to ensure that the data in the queue is processed safely by threads;

[0291] If there is data in the queue, the data block is retrieved from the queue and a custom event EventSerialPortDataReceivedProcess is triggered, passing the parsed data as a parameter.

[0292] 5) Exception handling:

[0293] Capture exceptions during data reading and processing, and log error messages or take other error handling measures;

[0294] 6) Serial port shutdown procedure:

[0295] If the serial port is closed, the loop is broken, a lock is used to ensure that the queue is cleared when the serial port is closed, all resources are released, and the method execution ends.

[0296] This embodiment solves a key contradiction in high-volume USB transmission through the synergistic innovation of a dynamic water level flow control mechanism and a three-level retransmission strategy:

[0297] 1. By proposing a combination of fingerprint verification and dynamic water level, the limitations of traditional fixed threshold flow control are overcome;

[0298] 2. An exception type-retransmission granularity mapping model was established, which reduced the amount of retransmitted data by 87% compared with the traditional solution;

[0299] 3. Session mirroring queues enable microsecond-level breakpoint location, improving recovery efficiency by 5 times;

[0300] This embodiment provides a data cliff rate testing method. The test platform uses an XC1281 development board combined with a Windows host, supported by a custom stress test program and a USB protocol analyzer. The test process consists of three steps: first, 10^6 random data packets are sent to the system under test; then, three types of interference (500ms delay, data packet loss, and bit flip error) are randomly injected during transmission, with hardware-level signal jammers used for anomaly injection; finally, the proportion of complete received packets is counted. Key parameter settings include: a total of 10^6 data packets, and interference types covering delay, packet loss, and bit errors. Transmission reliability is evaluated by single-byte cliff rate, batch transmission cliff rate, retransmission response time, and disconnection recovery success rate. The test verification results are shown in Table 1.

[0301] The key data calculation formula is shown below:

[0302] Cliff rate = (1 - number of successfully received packets / number of sent packets) × 100%

[0303] Recovery success rate = (Number of successfully recovered sessions / Number of manually interrupted sessions) × 100%

[0304] Table 1 Test Verification Results

[0305]

[0306] This embodiment effectively avoids data mixing by deeply collaborating the anomaly handling module with other modules and employing a fingerprint verification + three-level dynamic retransmission strategy. It also prevents data overwriting during high-speed transmission through a multi-level flow control mechanism. The innovative breakpoint fingerprint technology ensures data continuity after anomaly recovery and significantly reduces the probability of data cliff phenomena (from >30% to <1%).

[0307] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / soh>

Claims

1. A data integrity transmission system based on high-volume USB read / write, characterized in that, Includes the following modules: The data preprocessing module is used to construct data blocks according to different data sources using different partitioning schemes, and to add data block identifier headers to the data blocks to construct custom data blocks; the data block identifier header includes block number, length, status and data type; An initialization module is used to initialize the USB serial port object and data queue, and start the multi-threaded tasks and data processing threads corresponding to the data queue; the data queue includes a receive buffer queue, a receive data queue, and a send data queue; the multi-threaded tasks include a receive buffer thread, a receive data thread, and a send data thread. The data classification and sending module is used to encapsulate the custom data block according to the message format of the communication protocol and put it into the sending data queue. The sending data thread retrieves data from the sending data queue and sends it through the USB serial port. The data receiving and storage module is used to read data from the USB serial port through the receiving buffer and put it into the receiving buffer queue. The receiving data thread retrieves data from the receiving buffer queue, parses it according to the communication protocol, and stores it into the receiving data queue. The data verification module is used to perform CRC verification and data block length verification on the data received in the received data queue, and to trigger a retransmission mechanism when data errors or loss are detected, so as to ensure data integrity. The exception handling module is used to monitor exceptions during data transmission and to process them using a three-level retransmission strategy and a dynamic water level flow control mechanism based on the exception type and the fingerprint identifier of the custom data block. The exception types include: data packet timeout exception, continuous transmission error exception, verification failure exception, serial port connection interruption exception, and receive buffer overflow exception.

2. The data integrity transmission system according to claim 1, characterized in that, The data preprocessing module specifically includes: S11. If the data source is a text file, read the data from the text file, parse each line, extract the address and value, convert the address and value into integers respectively, and then construct data units based on the address and value. S12. Divide the data units into data groups according to the address characteristics, and divide the data groups into data blocks according to the preset data block size; S13. After adding a data block identifier header to each data block, store it in a custom data block.

3. The data integrity transmission system according to claim 1, characterized in that, The data preprocessing module further includes: S11' If the data source is a binary file, read the binary data from the binary file, convert it into a byte array, and create a data unit for each byte; S12': Traverse the byte array, identify a specific marker, read the bytes following the specific marker to calculate the actual length, and take the byte sequence of the actual length after the bytes following the specific marker to form a data block, thereby obtaining all data blocks; S13' Add a data block identifier header to each data block and then store it in a custom data block.

4. The data integrity transmission system according to claim 1, characterized in that, The exception handling module includes: when the receiving data thread does not receive a complete custom data block within a preset time, it considers a data packet timeout exception to have occurred, discards the currently received incomplete data packet, and requests the data classification and sending module to trigger the sending data queue to retransmit the custom data block.

5. The data integrity transmission system according to claim 4, characterized in that, The exception handling module further includes: when the same custom data block fails to be retransmitted 3 times in a row, it is considered that a continuous transmission error exception has occurred, the current custom data block is skipped and marked as a bad block, the block number and error position of the custom data block are recorded, and a command is sent to the data sending thread through the exception handling module to insert a retransmission request packet into the data sending queue.

6. The data integrity transmission system according to claim 5, characterized in that, The exception handling module further includes: when the data verification module encounters an error during CRC verification of the custom data block or finds an inconsistency in length during length verification, it considers a verification failure exception to have occurred, records the block number and error location of the custom data block, sends a command to the data sending thread through the exception handling module, inserts a retransmission request packet into the data sending queue, and clears the buffer area corresponding to the block number in the receive buffer queue.

7. The data integrity transmission system according to claim 6, characterized in that, The exception handling module further includes: when the serial port monitoring thread detects a physical connection disconnection, it considers a serial port connection interruption exception to have occurred, immediately suspends the multi-threaded task corresponding to the data queue, starts an automatic reconnection mechanism, and locates subsequent transmissions by using a custom data block fingerprint identifier after successful reconnection; the fingerprint identifier is a value generated based on the session ID and block number using the CRC32 algorithm.

8. The data integrity transmission system according to claim 7, characterized in that, The step of locating and retransmitting data after a successful reconnection using a custom data block fingerprint identifier specifically involves: restoring the last successful block number through a mirror queue, and then retransmitting based on the last successful block number. When the length of the receiving buffer queue reaches the preset queue safety threshold, a flow control signal is sent to the data classification and sending module, the data receiving thread is promoted to the highest priority, the single processing batch of the data processing thread is increased, and the standby working thread is started for parallel processing. The preset queue safety threshold Threshold is dynamically adjusted based on a dynamic water level flow control mechanism, as shown in the following formula: Threshold = Base × [α + β × (R_current / R_max)], Where Base is the base water level, α is 0.7, β is 0.3, R_current is the current transmission rate, and R_max is the maximum theoretical rate.

9. The data integrity transmission system according to claim 1, characterized in that, The three-level retransmission strategy is as follows: When a single data packet experiences a timeout or verification failure, the erroneous single data packet is immediately retransmitted at a retransmission level of L1. If a continuous transmission error occurs, the user-defined data block is selectively retransmitted after block verification, with a retransmission level of L2. If a serial port connection interruption occurs, the connection will be re-established and the context restored for retransmission at level L3.

10. The data integrity transmission system according to claim 1, characterized in that, The transmission system also includes a resource management module, which is used to set USB serial port parameters and control the opening and closing of the USB serial port, and to release relevant resources after data transmission is completed.

Images