A data incremental reading method based on offset consistency control and a system thereof

Abstract

The application discloses a data incremental reading method and system based on offset consistency control, and the method comprises the following steps: generating a unique file identifier for each monitored file, and establishing a mapping relationship between the file identifier and the offset; obtaining the change situation of the current monitored file, comparing the file size of the current monitored file with the current stored offset, when the file size of the monitored file is greater than the current stored offset, reading the new data from the position indicated by the offset pointer, dividing the read new data into data blocks according to the preset data block size, encapsulating and sending to the receiving end, when the confirmation offset carried in the confirmation information returned by the receiving end is consistent with the end position of the current data block, updating the offset pointer to the end position of the current data block, updating the offset to the sum of the current offset and the length of the current data block, and storing the updated offset.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data incremental reading method and system based on offset consistency control. Background Technology

[0002] In industrial IoT systems, various sensing devices, such as radar, weather sensors, and environmental monitoring instruments, typically write collected data continuously to local storage in file formats, such as XML, TXT, and binary files. To achieve data aggregation and processing, the system often needs to deploy data acquisition middleware on an industrial control computer or edge server. This middleware detects file changes through a file monitoring mechanism and transmits the file content to the edge gateway or cloud platform.

[0003] Currently, data reading and transmission typically employ a full-file read approach. When a file change is detected, the middleware reads the entire file content and transmits it. This method meets basic needs in scenarios with small files or low change frequency. However, in real-world industrial environments, sensing devices often continuously write data to files in small increments, appending only a small amount of data to the end, and individual files are often quite large. In such scenarios, performing a full read every time would generate a large number of unnecessary disk I / O operations and network transmission overhead, severely impacting system efficiency and potentially shortening the lifespan of storage devices.

[0004] Another approach is incremental reading, where the middleware records the timestamp or file size of the last read and only reads the newly added portion each time. This method reduces data transfer volume to some extent. However, such solutions typically store read progress information in memory. If the middleware process restarts or the system loses power, the progress of data that has been read but not yet transferred will be lost, requiring the entire file to be retransmitted, thus failing to achieve true resume functionality. Furthermore, when a file is truncated, overwritten, or the log is rotated, the original progress information will become invalid. The system often cannot correctly identify such anomalies, easily leading to incorrect data reading, duplicate reading, or data omission.

[0005] Another approach is based on message acknowledgment, where the sender transmits data to the receiver, and the receiver returns an acknowledgment message. However, in existing solutions, the acknowledgment mechanism is typically only used to determine whether data has arrived and does not directly participate in controlling the sender's reading progress. This leads to a problem: the sender's file reading progress may differ from the receiver's actual processing progress. For example, the sender may have read and sent a data block from the file, but the receiver has not successfully acknowledged this data block (e.g., due to network packet loss or receiver verification failure). If the sender continues to advance the read offset, data loss will occur; conversely, if the sender rewinds the read offset, data duplication may result. Summary of the Invention

[0006] The purpose of this invention is to provide a data incremental reading method based on offset consistency control to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a data incremental reading method based on offset consistency control, the method comprising:

[0008] Generate a unique file identifier for each monitored file and establish a mapping relationship between the file identifier and the offset;

[0009] The system obtains the changes of the currently monitored file, compares the file size of the currently monitored file with the current storage offset, and when the file size of the monitored file is greater than the current storage offset, reads the new data starting from the position indicated by the offset pointer, divides the read new data into data blocks according to the preset data block size, encapsulates them, and sends them to the receiving end.

[0010] When the acknowledgment offset carried in the acknowledgment information returned by the receiving end is consistent with the end position of the current data block, the offset pointer is updated to the end position of the current data block, the offset is updated to the sum of the current offset and the current data block length, and the updated offset is stored.

[0011] Furthermore, if the size of the monitored file is less than the current storage offset, it is determined that the monitored file has been truncated or rewritten, and the offset is reset to the initial value of 0.

[0012] Furthermore, the receiving end verifies the data block and returns confirmation information after verification. If no confirmation information is received, or if a timeout occurs, or if a negative confirmation is received, the offset is not updated, and the current data block is retransmitted.

[0013] Furthermore, during the transmission process, if a network anomaly is detected, data transmission to the receiving end is paused, while incremental data reading and data block partitioning and encapsulation operations continue to be performed. Data blocks are written to the local temporary storage queue in sequence, while the offset remains unchanged. After the network is restored, data blocks are read from the local temporary storage queue first and sent in sequence. After receiving an acknowledgment message for each data block and completing the offset update, the next data block is sent, until the local temporary storage queue is cleared, and then the real-time incremental reading and sending process is resumed.

[0014] Furthermore, when the monitored file size is found to be equal to the current storage offset, it is determined that the monitored file has been fully read, the file status of the monitored file is marked as fully read, and file deletion, archiving or retention operations are triggered according to the preset strategy, while the offset mapping record corresponding to the monitored file is cleared.

[0015] Furthermore, the method for generating a unique file identifier for each monitored file is as follows: the unique file identifier is generated by hashing based on at least one of the following: the absolute path of the file, the file creation time, the file last modification time, and the file size.

[0016] Furthermore, the data block is encapsulated after being appended with one or more of the following: file identifier, block sequence number, offset, data length, data content, verification information, and generation time. The verification information is used by the receiving end to verify the data integrity and legality.

[0017] Furthermore, the confirmation offset is the end offset of the data block that the receiving end has successfully received and verified.

[0018] A data incremental reading system based on offset consistency control includes:

[0019] Monitoring equipment is used to continuously generate monitoring data and write it to data files;

[0020] A local file storage module is used to store the data files;

[0021] The data acquisition terminal is deployed on an industrial control computer or edge server and is used to execute the data incremental reading method based on offset consistency control as described in any one of claims 1 to 8.

[0022] The receiving end is used to receive data blocks and return acknowledgment messages or negative acknowledgment messages.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] 1. This invention maintains an independent offset for each monitored file, reading new data only when the current file size is greater than the offset pointer. This reduces disk I / O operations and network data transmission, significantly lowering system overhead. The offset update is strictly bound to the confirmation message returned by the receiving end. The offset pointer is only updated when the confirmation offset carried in the confirmation message is exactly the same as the end position of the current data block. This ensures that the file reading progress is always consistent with the actual confirmation progress of the receiving end, and that no data is lost during transmission, ensuring data integrity. The mapping relationship between file identifiers and offsets is stored locally. Even if the middleware process exits abnormally, the system loses power, or restarts, the offset information can still be read after restarting. Data that has not yet been confirmed can continue to be read and sent from the breakpoint without retransmitting the entire file, effectively supporting the ability to resume interrupted transmissions.

[0025] 2. When the network is abnormal, data transmission to the receiving end is paused, but incremental file reading and data block encapsulation operations continue to be performed, and data blocks are written to the local temporary storage queue in sequence while keeping the offset unchanged. This decouples data reading operations from network transmission: new data continuously written by the front-end sensing device can be read and cached normally by the middleware, and will not be blocked or discarded due to network interruption. After the network is restored, the system prioritizes sending data blocks from the temporary storage queue in sequence, and advances the offset pointer only once after each data block is confirmed, ensuring the connection between temporary data and real-time new data. This effectively solves the problems of read-write coupling, data backlog, and even loss in existing technologies when the network is abnormal, and will not be blocked or discarded due to network interruption, thus significantly improving the stability of the system in network fluctuation environments and ensuring no duplication or loss during data processing.

[0026] 3. During each file change detection, the current file size is compared with the stored offset pointer. If the current file size is smaller than the offset pointer, it is determined that the file has been truncated, overwritten, or rebuilt (e.g., in log rotation scenarios). The system automatically resets the offset to its initial value of 0 and restarts incremental reading from the beginning of the file. This mechanism enables the invention to correctly adapt to various abnormal scenarios where files are overwritten or truncated, avoiding data misreading or missed reading due to offset failure. Attached Figure Description

[0027] Figure 1 This is a flowchart of the incremental data reading method based on offset consistency control according to the present invention;

[0028] Figure 2 This is a timing diagram of offset update and data block confirmation in the data incremental reading method based on offset consistency control of the present invention;

[0029] Figure 3 This is a diagram showing the state transition between the temporary queue and the offset pointer in the incremental data reading method based on offset consistency control in this invention when a network anomaly occurs. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This embodiment provides a data incremental reading system based on offset consistency control, including a monitoring device, a file storage module, a data acquisition end, and a receiving end. In this embodiment, the monitoring device is deployed at the monitored site. The monitoring device can be a radar, weather sensor, environmental detector, etc., continuously generating monitoring data and writing it to a data file, which is the monitored file. The file storage module and the data acquisition end can be deployed on an on-site industrial control computer or edge server. The file storage module persistently stores the monitored file in the form of a local file system. The data acquisition end is also middleware, realizing data aggregation and processing, generating a unique file identifier for each monitored file, establishing a mapping relationship between the file identifier and the offset, obtaining the current changes of the monitored file, comparing the current file size of the monitored file with the currently stored offset, and when the file size of the monitored file changes, the data acquisition end receives the data. When the file size exceeds the current storage offset, new data is read starting from the position indicated by the offset pointer. The read new data is divided into data blocks according to the preset data block size, encapsulated, and sent to the receiving end. The receiving end is an edge gateway or cloud platform. After receiving the data block, the receiving end de-escapes and restores the data block, performs integrity verification, and determines the legality of the data. If the verification is successful, an acknowledgment message is returned, confirming the offset. This confirmed offset is the end offset of the successfully received and verified data block. If the verification fails, the data block sequence number is returned, and a retransmission is requested. When the acquisition end receives the acknowledgment message returned by the receiving end, and the confirmed offset is consistent with the end position of the current data block, the offset pointer is updated to the end position of the current data block, the offset is updated to the sum of the current offset and the current data block length, and the updated offset is stored.

[0032] This embodiment also provides a data incremental reading method based on offset consistency control, the method including the following steps:

[0033] Generate a unique file identifier for each monitored file and establish a mapping relationship between the file identifier and the offset;

[0034] The system obtains the changes of the currently monitored file, compares the file size of the currently monitored file with the current storage offset, and when the file size of the monitored file is greater than the current storage offset, reads the new data starting from the position indicated by the offset pointer, divides the read new data into data blocks according to the preset data block size, encapsulates them, and sends them to the receiving end.

[0035] When the acknowledgment offset carried in the acknowledgment information returned by the receiving end is consistent with the end position of the current data block, the offset pointer is updated to the end position of the current data block, the offset is updated to the sum of the current offset and the current data block length, and the updated offset is stored.

[0036] The following is a detailed explanation: The monitoring equipment continuously generates monitoring data and writes it to a data file to form a monitored file, which is then stored in the local file system, such as... Figure 1 As shown, then perform the following steps:

[0037] (I) Initialization Phase

[0038] Each monitored file is registered and a unique file identifier is generated. The unique file identifier FileID can be generated by hashing based on the file's absolute path, file creation time, file last modification time, and file size.

[0039] Initialize the offset, set offset = 0, and persist it.

[0040] Establish a mapping relationship between file identifiers and offsets: FileID → offset

[0041] (II) Document Change Detection Phase

[0042] Real-time or periodic detection of changes in monitored files, reading the current file size (currentSize) and querying the current storage offset (LastOffset);

[0043] (III) Rewinding Inspection Stage

[0044] The file size currentSize of the monitored file is compared with the current storage offset LastOffset. When the file size currentSize of the monitored file is less than the current storage offset LastOffset, i.e., currentSize < LastOffset, it is determined that the monitored file has been truncated or rewritten. The current storage offset LastOffset is reset to the initial value of 0 and the log is recorded.

[0045] (iv) Incremental data reading phase

[0046] When the file size currentSize of the monitored file is greater than the current storage offset LastOffset (i.e., currentSize > LastOffset), new data is read starting from the position indicated by the current storage offset pointer. The read new data is divided into data chunks according to the preset data block size. Each data chunk is encapsulated with a file identifier (FileID), sequence number, offset, data length, data content (payload), verification information (CRC / Hash), and generation time. The verification information (CRC / Hash) is used by the receiving end to verify the integrity and legality of the data.

[0047] (v) Data encapsulation and transmission stage

[0048] Press A to move the data block chunk. （） …B （） The protocol is encapsulated (A0 / B0 escaped) and sent sequentially to the receiving end via the Netty protocol, for example, to an edge gateway, cloud platform, etc.

[0049] (vi) Receiving and Verification Stage

[0050] The receiving end deescapes and restores the data block chunk, performs integrity verification (CRC / Hash), and determines the validity of the data. If the verification is successful, it sends an ACK and confirms the offset, which is the end offset of the successfully received and verified data block. If the verification fails, it sends a NAK, returns the data block sequence number, and requests retransmission.

[0051] (vii) Confirm the offset update mechanism of the driver

[0052] The sending end executes the following based on the feedback from the receiving end:

[0053] 1. If an ACK is received, that is, confirmation that the offset matches the end position of the current data block,

[0054] Update the offset pointer to the end position of the current data block, update the offset to the sum of the current offset and the length of the current data block, i.e.: LastOffset = LastOffset + length, and persist the updated offset LastOffset.

[0055] 2. If a NAK is received or a timeout occurs:

[0056] If the LastOffset is not updated, the current data block is retransmitted. The number of retransmissions is limited, and an alarm is issued if the limit is exceeded.

[0057] like Figure 2As shown, for example, the middleware (collector) reads block N, finds the current offset to be X and the read length to be L, and sends block N to the edge gateway (receiver). The edge gateway successfully receives and processes block N, confirms the offset to be X + L, and therefore returns ACK (confirming offset = X + L) to the collector, writes the offset = X + L to the persistent storage offsetMap, and after receiving the ACK, the collector confirms that block N has been successfully received, and the starting point for subsequent data reading is updated to X + L;

[0058] If the acquisition end does not receive an ACK after sending block N, or receives a NAK, or triggers a timeout, it will not update the local offset, which will remain X. Block N will be resent (sequence number = N, offset pointer offset = X). The edge gateway may return a NAK (sequence number = N) due to network packet loss, data verification errors, or its own processing failure. Alternatively, if the acquisition end continues to fail to receive an ACK and triggers a timeout, after receiving a NAK or after a timeout, the acquisition end will repeatedly retransmit block N until it receives an ACK. If the limit is exceeded, an alarm will be issued.

[0059] After receiving the ACK (acknowledgment offset = X+L) for block N, the acquisition end writes the offset = X+L to the persistent storage offsetMap, reads block N+1, and the length of block N+1 is L2. Data is read starting from offset = X+L. After the receiving end processes block N+1, it returns ACK (acknowledgment offset = X+L+L2). After receiving the ACK, the acquisition end writes the offset = X+L+L2 to the persistent storage offsetMap and prepares to read the next data block. This process is repeated.

[0060] (viii) Network anomaly handling mechanism

[0061] During the transmission process, if a network anomaly is detected, such as a network disconnection:

[0062] Pause sending data to the receiving end, continue to perform incremental data reading and data block partitioning and encapsulation operations, write data blocks sequentially into the local temporary storage queue, and keep the offset unchanged;

[0063] After the network is restored:

[0064] Data blocks are read from the local temporary queue first and sent in sequence. After receiving an acknowledgment message for each data block and updating the offset, the next data block is sent. This continues until the local temporary queue is empty, and then the real-time incremental reading and sending process is resumed.

[0065] like Figure 3As shown, for example, when the network is in a normal state, it enters S0 and performs normal reading and sending, that is, real-time reading and real-time sending. After receiving an ACK, the offset is updated and persistently stored. Its internal process includes file change detection, wraparound detection, reading data blocks from the offset pointer, sending data blocks, waiting for the receiver to return ACK / NAK, and when an ACK is returned, the offset is updated. When a NAK is returned or a timeout occurs, the offset is not updated, and the data block is retransmitted. See steps (II) to (VII) for details. When the network is disconnected, it enters S1. At this time, the network is in an abnormal state. Data transmission to the receiver is paused, and incremental data reading and data block partitioning and encapsulation operations are continuously performed. Data blocks are written to the local temporary storage queue in sequence, while keeping the offset unchanged. Its internal process includes file change detection, wraparound detection, reading data blocks from the offset pointer, and sending data blocks from the offset pointer. Read data blocks at ffset and write them sequentially to the local temporary queue. See steps (ii) to (iv) and step (viii) for details. When the network recovers, enter S2. At this time, the network is in the recovery transmission state. Read data blocks from the local temporary queue first and send them sequentially. After receiving an acknowledgment message for each data block and completing the offset update, continue to send the next data block until the local temporary queue is empty. Then resume the real-time incremental reading and sending process. The internal process includes sending the data blocks in the local temporary queue, waiting for the receiver to return ACK / NAK, updating the offset when ACK is returned, not updating the offset when NAK is returned or timeout occurs, and retransmitting the data block. When the local temporary queue is empty, return to step S0. See steps (viii) to (ix) and steps (ii) to (vii) for details.

[0066] (ix) Termination conditions and cleaning mechanism

[0067] When the monitored file size currentSize is equal to the current offset LastOffset (i.e., currentSize=LastOffset), it is determined that the monitored file has been fully read. The file status of the monitored file is marked as fully read, and file deletion, archiving, or retention operations are triggered according to the preset strategy. At the same time, the offset mapping record corresponding to the monitored file is cleared.

[0068] Implementation Case:

[0069] Taking a meteorological monitoring system in an industrial park as an example, a set of meteorological sensors is deployed on-site. The device generates and continuously writes an XML data file every 5 minutes, with each file ultimately reaching a size of approximately 50MB. The data file includes monitoring fields such as collection time, temperature, humidity, wind speed, wind direction, air pressure, and rainfall. Middleware is deployed on the on-site industrial control computer or edge server to monitor the data files generated by the device and reliably transmit incremental data to the edge gateway.

[0070] 1. System Deployment Requirements

[0071] In this embodiment, the system includes meteorological monitoring equipment, a local file system, a data acquisition terminal, an edge gateway, and local persistent storage.

[0072] Meteorological monitoring equipment continuously generates monitoring data and writes it to a data file, generating a monitored file in XML format;

[0073] The local file system is used to store the monitored files generated by the meteorological monitoring equipment;

[0074] The acquisition terminal is used to monitor changes in the monitored files, read incremental data, encapsulate data blocks, and send them.

[0075] The edge gateway is used to receive data blocks, verify data integrity, and return confirmation information;

[0076] The acquisition end is also used to update the offset based on the confirmation offset carried in the confirmation information returned by the receiving end;

[0077] Local persistent storage is used to save the offset and file status information of the monitored file.

[0078] The data file storage directory can be configured as: / data / weather /

[0079] The example file is named: weather_20260521_100000.xml

[0080] 2. File registration and offset initialization

[0081] Once the middleware starts, it first scans the monitored directory and generates a unique file identifier (FileID) for each file to be collected. The FileID can be generated based on the following information: FileID = Hash(absolute file path + file creation time + file last modification time + file size).

[0082] For example: FileID = Hash(" / data / weather / weather_20260521_100000.xml" +createTime + modifyTime + fileSize)

[0083] The middleware establishes the following mapping in the local persistent storage offsetMap: FileID → offset

[0084] On the first read, the offset is initialized to offset = 0 and written to the local persistent storage offsetMap.

[0085] 3. File change detection

[0086] The middleware monitors the target directory through a file monitoring mechanism. When it detects that an XML file has been created or modified, it obtains the current size of the file, currentSize.

[0087] For example, at a certain moment, the file size is: currentSize = 10240 bytes; offset = 0.

[0088] Because: currentSize > offset

[0089] This indicates that the file contains newly added data and an incremental read operation is required.

[0090] 4. File wrapping or overwriting detection

[0091] Before each read, the middleware compares the current file size (currentSize) with the saved offset (offset).

[0092] If currentSize < offset

[0093] This indicates that the file may have been truncated, overwritten, or regenerated. The system then performs the following operations:

[0094] ① Reset the offset to 0;

[0095] ② Update the offsets in offsetMap;

[0096] ③ Record file wraparound logs;

[0097] ④ Re-execute incremental reading from the beginning of the file.

[0098] For example:

[0099] offset = 40960 Byte

[0100] currentSize = 2048 Byte

[0101] The system determines that the file has been overwritten or cleared, and then executes:

[0102] offset = 0

[0103] 5. Incremental data block reading

[0104] In this embodiment, for example, the preset data block size chunkSize is 8KB, that is: chunkSize = 8192 Bytes

[0105] When the current file size is greater than the offset, the middleware starts reading new data from the offset position.

[0106] For example: offset = 0; currentSize = 10240 bytes

[0107] Then the first read:

[0108] Read start position = 0

[0109] Read length = min(10240 - 0, 8192) = 8192 bytes

[0110] After reading is complete, the first data block, Block-0, is generated.

[0111] Once the first data block is successfully confirmed, the offset is updated to 8192. If the file continues to grow to 20480 bytes, the next read will proceed as follows:

[0112] Read starting position = 8192

[0113] Read length = min(20480 - 8192, 8192) = 8192 bytes

[0114] Generate the second data block, Block-1.

[0115] 6. Data block encapsulation

[0116] Each data block is structured and encapsulated before being sent. A data block must include at least the following fields:

[0117] FileID F20260521100000 Unique identifier for files BlockNo 0 Data block sequence number StartOffset 0 Data block start offset Length 8192 Data block length Payload XML fragment content Actual data content Checksum CRC32 value Data integrity check value Timestamp 2026-05-21 10:00:05 Data block generation time

[0118] The example data block structure is as follows:

[0119] {

[0120] "fileId": "F20260521100000",

[0121] "blockNo": 0,

[0122] "startOffset": 0,

[0123] "length": 8192,

[0124] "checksum": "A91F23BC",

[0125] "payload": "......"

[0126] }

[0127] The checksum can be generated using CRC32, MD5, or SHA-256 algorithms.

[0128] 7. Data sending and receiving end verification

[0129] The middleware sends the encapsulated data blocks to the edge gateway via TCP, Netty, HTTP, or message queues.

[0130] After receiving the data block, the edge gateway performs the following processing:

[0131] ① Parse the data block header information;

[0132] ② Retrieve FileID, BlockNo, StartOffset, Length, and Checksum;

[0133] ③ Recalculate the checksum for the payload;

[0134] ④ Compare the calculation results with the checksum;

[0135] ⑤ Determine whether the data block is complete, duplicated, or in the correct order.

[0136] If the verification passes, the edge gateway returns an ACK confirmation message:

[0137] {

[0138] "fileId": "F20260521100000",

[0139] "blockNo": 0,

[0140] "ackOffset": 8192,

[0141] "status": "ACK"

[0142] }

[0143] If the verification fails, the edge gateway returns a NAK message:

[0144] {

[0145] "fileId": "F20260521100000",

[0146] "blockNo": 0,

[0147] "ackOffset": 0,

[0148] "status": "NAK",

[0149] "reason": "checksum_error"

[0150] }

[0151] 8. Confirm driver offset update

[0152] After receiving the ACK message from the edge gateway, the middleware does not update the offset directly, but first checks whether the acknowledgment offset in the ACK is consistent with the end position of the current data block.

[0153] For example:

[0154] Current data block StartOffset = 0

[0155] Length = 8192

[0156] Expected confirmation offset = 0 + 8192 = 8192

[0157] ACK returns ackOffset = 8192

[0158] Since the ACK return value is consistent with expectations, the middleware executes:

[0159] offset = 8192

[0160] And immediately persist it to offsetMap.

[0161] If a NAK is received, or an ACK is not received within the preset time, then execute:

[0162] offset remains unchanged

[0163] Resend the current data block

[0164] For example, if a timeout occurs after Block-0 is sent, it will still be maintained:

[0165] offset = 0

[0166] And reread or resend Block-0.

[0167] 9. Network Anomaly Handling

[0168] When the middleware detects a network connection interruption with the edge gateway, the system enters a network anomaly state.

[0169] In this state, the system executes the following strategy:

[0170] ① File monitoring continues;

[0171] ②Incremental data is still read according to the offset position;

[0172] ③ The read data block is not sent immediately;

[0173] ④ Data blocks are written to the local temporary storage queue in BlockNo order;

[0174] ⑤ The offset will not be advanced until an ACK is received.

[0175] For example:

[0176] The currently confirmed offset is:

[0177] offset = 8192

[0178] Even after the network connection is lost, file writing continues, and the system reads data sequentially:

[0179] Block-1: 8192 ~ 16384

[0180] Block-2: 16384 ~ 24576

[0181] Block-3: 24576 ~ 32768

[0182] The aforementioned data block is written to the temporary queue:

[0183] Queue = [Block-1, Block-2, Block-3]

[0184] At this point, the offset remains unchanged:

[0185] offset = 8192

[0186] 10. Compensation sent after network recovery

[0187] Once the network is restored, the middleware prioritizes reading the data blocks in the local temporary queue and sends them to the edge gateway in BlockNo order.

[0188] The sending order is as follows:

[0189] Block-1 → Block-2 → Block-3

[0190] For each data block sent, we must wait for the edge gateway to return an ACK confirmation.

[0191] For example:

[0192] After the edge gateway confirms Block-1, it returns the following:

[0193] ackOffset = 16384

[0194] Middleware execution:

[0195] offset = 16384

[0196] And persist it.

[0197] Then continue sending Block-2, confirm and update:

[0198] offset = 24576

[0199] The system resumes the real-time incremental reading process only after the temporary queue is cleared.

[0200] 11. File reading complete and cleanup

[0201] When the file no longer changes, and the following conditions are met:

[0202] currentSize = offset

[0203] The system then determines that the file has been successfully read.

[0204] For example:

[0205] currentSize = 52428800 Byte

[0206] offset = 52428800 Byte

[0207] This indicates that all data in the 50MB file has been confirmed by the receiving end.

[0208] The system can then perform the following processing:

[0209] ① Mark the file status as "fully read";

[0210] ② Retain or archive the original documents;

[0211] ③ Clear the temporary cache data corresponding to this file;

[0212] ④ Determine whether to clear the records in offsetMap based on the configuration.

[0213] This invention reduces disk I / O operations and network data transmission volume by maintaining an independent offset for each monitored file and reading new data only when the current file size is greater than the offset pointer, significantly reducing system overhead. Offset updates are strictly bound to the acknowledgment messages returned by the receiving end; the offset pointer is only updated when the acknowledgment offset carried in the acknowledgment message is exactly consistent with the end position of the current data block. This ensures that the file reading progress always matches the actual acknowledgment progress of the receiving end, preventing data loss during transmission and ensuring data integrity. The mapping relationship between file identifiers and offsets is stored locally. Even if the middleware process exits abnormally, the system loses power, or restarts, the offset information can still be read after restarting, allowing the reading and sending of unacknowledged data from the breakpoint position without retransmitting the entire file, effectively supporting breakpoint resumption capabilities. In the event of network anomalies, data transmission to the receiving end is paused, but incremental file reading and data block encapsulation operations continue, and data blocks are written sequentially to the local temporary queue while maintaining the offset unchanged. This system achieves decoupling between data reading operations and network transmission: New data continuously written by the front-end sensing device can be normally read and cached by the middleware, without being blocked or discarded due to network interruptions. After network recovery, the system prioritizes sending data blocks sequentially from the temporary queue, advancing the offset pointer only after each data block is confirmed. This ensures seamless integration between temporary data and real-time new data, effectively solving the problems of read-write coupling, data backlog, and even data loss in existing technologies during network anomalies. It prevents data from being blocked or discarded due to network interruptions, significantly improving system stability under fluctuating network conditions and ensuring no duplication or loss during data processing. During each file change detection, the current file size is compared with the stored offset pointer. If the current file size is smaller than the offset pointer, it indicates that the file has been truncated, overwritten, or rebuilt (e.g., in log rotation scenarios). The system automatically resets the offset to its initial value of 0 and restarts incremental reading from the beginning of the file. This mechanism enables the invention to correctly adapt to various abnormal scenarios where files are overwritten or truncated, avoiding data misreading or missed reading due to offset failure.

[0214] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for incremental data reading based on offset consistency control, characterized in that, The method includes: Generate a unique file identifier for each monitored file and establish a mapping relationship between the file identifier and the offset; The system obtains the changes of the currently monitored file, compares the file size of the currently monitored file with the current storage offset, and when the file size of the monitored file is greater than the current storage offset, reads the new data starting from the position indicated by the offset pointer, divides the read new data into data blocks according to the preset data block size, encapsulates them, and sends them to the receiving end. When the acknowledgment offset carried in the acknowledgment information returned by the receiving end is consistent with the end position of the current data block, the offset pointer is updated to the end position of the current data block, the offset is updated to the sum of the current offset and the current data block length, and the updated offset is stored.

2. The incremental data reading method based on offset consistency control according to claim 1, characterized in that: When the size of the monitored file is less than the current storage offset, it is determined that the monitored file has been truncated or rewritten, and the offset is reset to the initial value of 0.

3. The data incremental reading method based on offset consistency control according to claim 1, characterized in that: The receiving end verifies the data block and returns an acknowledgment after verification. If no acknowledgment is received, or if a timeout occurs, or if a negative acknowledgment is received, the offset is not updated, and the current data block is retransmitted.

4. The data incremental reading method based on offset consistency control according to claim 1, characterized in that: If a network anomaly is detected during transmission, data transmission to the receiving end is paused, while incremental data reading and data block partitioning and encapsulation operations continue to be performed. Data blocks are written to the local temporary storage queue in sequence, while the offset remains unchanged. After the network is restored, data blocks are read from the local temporary storage queue first and sent in sequence. After receiving an acknowledgment message for each data block and updating the offset, the next data block is sent, until the local temporary storage queue is cleared. Then, the real-time incremental reading and sending process is resumed.

5. The data incremental reading method based on offset consistency control according to claim 1, characterized in that: When the monitored file size is found to be equal to the current storage offset, it is determined that the monitored file has been fully read, the file status of the monitored file is marked as fully read, and file deletion, archiving or retention operations are triggered according to the preset strategy. At the same time, the offset mapping record corresponding to the monitored file is cleared.

6. The incremental data reading method based on offset consistency control according to claim 1, characterized in that, The method for generating a unique file identifier for each monitored file is as follows: the unique file identifier is generated by hashing based on at least one of the following: the absolute path of the file, the file creation time, the file last modification time, and the file size.

7. The data incremental reading method based on offset consistency control according to claim 1, characterized in that: The data block is encapsulated after being appended with one or more of the following: file identifier, block number, offset, data length, data content, verification information, and generation time. The verification information is used by the receiving end to verify the data integrity and legality.

8. The incremental data reading method based on offset consistency control according to claim 1, characterized in that: The confirmation offset is the end offset of the data block that the receiving end has successfully received and verified.

9. A data incremental reading system based on offset consistency control, characterized in that, include: Monitoring equipment is used to continuously generate monitoring data and write it to data files; A local file storage module is used to store the data files; The data acquisition terminal is deployed on an industrial control computer or edge server and is used to execute the data incremental reading method based on offset consistency control as described in any one of claims 1 to 8. The receiving end is used to receive data blocks and return acknowledgment messages or negative acknowledgment messages.

Images