Engineering edge data fragmentation reorganization and service breakpoint continuation method in weak network environment

By segmenting data by region and time interval in a weak network environment and adding spatiotemporal identifiers and check codes, the problem of poor data transmission stability was solved, and efficient data reassembly and breakpoint resumption were achieved, thus improving business continuity and reliability.

CN120602477BActive Publication Date: 2025-10-21ZHEJIANG COMM SERVICES +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511108062.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-08-08
Publication Date
2025-10-21
Estimated Expiration
2045-08-08

AI Technical Summary

Technical Problem

In weak network environments, data transmission at the engineering edge is unstable and prone to interruption, leading to data loss or duplicate transmission, which affects business continuity. Existing technologies are difficult to adapt to the characteristics of multi-area and long-term data collection at construction sites, and lack dynamic tracking of the integrity of data segments and business status.

Method used

By segmenting data according to construction area and time interval, adding spatiotemporal identifiers and check codes, segmented data packets are generated. The spatiotemporal identifiers are then used for temporary storage, verification, preprocessing, and reassembly to generate a segmented cache set with processing progress, enabling breakpoint resume and reassembly of data.

Benefits of technology

It improves data transmission efficiency and business continuity in weak network environments, meets the reliability requirements of engineering edge scenarios, and ensures data integrity and rapid recovery of business processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120602477B_ABST
    Figure CN120602477B_ABST
Patent Text Reader

Abstract

The application discloses a kind of engineering edge-end data fragment reorganization and service breakpoint continuation methods under weak network environment, method includes: through engineering edge-end, original service data collected in situ is received, and it is fragmented according to construction area division and time interval, generates the fragmented data packet with space-time mark and check code;The fragmented data packet received is checked, and the fragmented cache set with processing progress is generated;When all fragments of certain area-time segment in fragmented cache set are received, the pre-processing progress of corresponding service is read from the persistent file, and the remaining service processing is continued from the progress interruption point;Integrate the reorganization original data of each area-time segment, service processing result, breakpoint continuation record and fragmented transmission log, generate the service data archives containing complete data chain and processing track.Using the embodiment of the application, the data transmission efficiency and service continuity under weak network environment can be improved, and the reliability requirement of engineering edge-end scene is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular to a method for engineering edge data segmentation and reorganization and service breakpoint resumption in a weak network environment. Background Art

[0002] In the scenario of data collection and processing at the edge of the project, data transmission in a weak network environment is unstable and easily interrupted, resulting in data loss or repeated transmission, which seriously affects business continuity. Traditional fragmented transmission methods are mostly based on dividing data in a single time or space dimension. They are difficult to adapt to the characteristics of multi-area and long-term collection at the construction site, and lack dynamic tracking of fragment integrity and business status. Existing breakpoint resumption technology usually relies on global data verification, and when fragments arrive out of order or are partially lost, they need to be retransmitted, which is inefficient. In addition, business processing needs to be executed from the beginning after interruption, and cannot be quickly restored in combination with the preprocessing progress, resulting in a waste of resources. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for engineering edge data segmentation and reorganization and business breakpoint resumption in a weak network environment to solve the shortcomings of the existing technology, improve data transmission efficiency and business continuity in a weak network environment, and meet the reliability requirements of engineering edge scenarios.

[0004] An embodiment of the present application provides a method for engineering edge data fragmentation reorganization and service breakpoint resumption in a weak network environment, the method comprising:

[0005] The original business data collected on-site is received by the project edge, and is fragmented according to the construction area and time interval. A time-space identifier containing the area code, timestamp, fragment sequence number, and total number of fragments is added to each piece of data. At the same time, a check code is generated for the fragmented data, and a fragmented data packet with the time-space identifier and check code is generated.

[0006] Verify the received fragmented data packets and temporarily store them in the temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier and arrange them in order. Synchronously record the reception time and verification result of each fragment. Perform business preprocessing on the received complete region-time fragments. Write the preprocessing progress, intermediate results and fragment reception status to the persistent file of local storage to generate a fragment cache set with processing progress.

[0007] When all shards of a certain region-time segment in the shard cache are received, the original data of the segment is reassembled according to the region code and timestamp sequence of the spatiotemporal identifier. The pre-processing progress of the corresponding business is read from the persistent file, and the remaining business processing is continued from the point where the progress was interrupted. The business processing result is output with a resume completion mark.

[0008] Integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each region and time segment, package them according to the project batch number, and generate a business data archive containing a complete data chain and processing trajectory.

[0009] Optionally, the raw business data collected on-site is received by the project edge, and is fragmented according to the construction area and time interval. A time-space identifier including an area code, a timestamp, a fragment sequence number, and a total number of fragments is added to each fragment of data, and a check code is generated for the fragmented data. The fragmented data packet with the time-space identifier and the check code is generated, including:

[0010] Extract features from the original business data collected at the engineering site, identify the spatial and temporal features in the data, construct a spatiotemporal feature vector, and simultaneously calculate the initial hash value of the data to generate a spatiotemporal feature vector with the original hash;

[0011] Based on the spatial features in the spatiotemporal feature vector, the region is divided and the K-means clustering algorithm is used to automatically determine the optimal number of regions. Each region is assigned a unique code to generate a spatiotemporal feature vector with region codes.

[0012] Dynamically determine the sharding time interval based on the data generation cycle and processing capacity, perform sliding sharding on the spatiotemporal feature vector with region coding according to the time window, add a timestamp and shard sequence number to each piece of data, and calculate the total number of pieces at the same time to generate sharded data with spatiotemporal identifiers;

[0013] The fragmented data with time and space identifiers are checked in blocks, and a check code is generated by combining cyclic redundancy check and Bloom filter. The check code is embedded in the time and space identifier to generate a fragmented data packet with time and space identifier and check code.

[0014] Optionally, the received fragmented data packets are verified, and temporarily stored in a temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier and arranged in sequence, the reception time and verification result of each fragment are synchronously recorded, business preprocessing is performed on the received complete region-time fragment, the preprocessing progress, intermediate results and fragment reception status are written to a persistent file in local storage, and a fragment cache set with processing progress is generated, including:

[0015] Verify the checksum of the received fragmented data packets, quickly match them with the Bloom filter through parallel CRC calculation, mark the packets that pass the checksum, and generate fragmented data packets with checksum marks;

[0016] According to the region code in the time-space identifier, the fragmented data packets with checksum marks are routed to the temporary storage area of ​​the corresponding region, and a skip table data structure is used to quickly insert and sort the fragment sequence numbers to generate an ordered fragment set;

[0017] Record the reception time and verification result of each shard, construct a five-tuple log containing the region code, timestamp, shard sequence number, reception time, and verification result, and write it to the local disk in real time using memory-mapped file technology to generate a time series log stream.

[0018] Perform business preprocessing on the received complete region-time segments, define the preprocessing process using a state machine model, and generate an intermediate result set with processing status;

[0019] Integrate the preprocessing progress, intermediate result set and time series log stream, write them to the local storage state file through the incremental persistence algorithm, and generate a shard cache set with processing progress.

[0020] Optionally, when all the fragments of a certain region-time segment in the fragment cache are received, the original data of the segment is reassembled according to the region code and timestamp sequence of the spatiotemporal identifier, the preprocessing progress of the corresponding business is read from the persistent file, the remaining business processing is continued from the point where the progress was interrupted, and the business processing result with the resume completion mark is output, including:

[0021] When all shards of a certain region-time segment in the shard cache pass verification, the reassembly process is triggered. Based on the region code and timestamp order of the spatiotemporal identifier, a merge sort algorithm is used to merge the ordered shard sets to generate reassembled original data with region time stamps.

[0022] Read the preprocessing progress of the corresponding business from the persistence file, parse the processing status mark in the status file, locate the breakpoint position of the most recent successful processing, and generate a breakpoint context snapshot;

[0023] Load the breakpoint context snapshot, import the reorganized raw data with regional time stamps into the business processing environment, and continue executing the remaining business processing flow from the breakpoint. Use shadow execution technology to synchronously record intermediate calculation steps and state changes to generate a continued execution trajectory.

[0024] After the business processing is completed, the execution trace of the subsequent transmission is compared and verified with the original processing record, and the integrity of the processing process is ensured through the hash chain, and a verified business processing result is generated;

[0025] Embed a resume completion mark containing the breakpoint location, resume time, and verification hash in the business processing result. Use a digital signature to ensure that the mark cannot be tampered with, and output the business processing result with the resume completion mark.

[0026] Optionally, the reorganized original data, business processing results, breakpoint resume records, and fragment transmission logs of each region-time segment are integrated and packaged according to the project batch number to generate a business data archive containing a complete data chain and processing trajectory, including:

[0027] Collect the reconstructed original data, business processing results, breakpoint resume records, and fragment transmission logs for each region-time segment, build a four-tuple data structure, generate a unique content addressable identifier for each data item, and generate a data set with an identifier;

[0028] Group the data sets with identifiers by engineering batch number, use directed acyclic graph to model the dependency between data items, and generate a data relationship graph;

[0029] Add a timestamp proof to each data item, build a complete data processing trace based on the data relationship graph, and generate a processing trace chain with a timestamp proof;

[0030] The data relationship graph and processing trajectory chain are encapsulated into a Merkle tree structure, and a root hash is generated as the data archive fingerprint. At the same time, an index file containing all metadata is generated, ultimately forming a business data archive containing a complete data chain and processing trajectory.

[0031] Another embodiment of the present application provides a system for engineering edge data fragmentation and reassembly and service breakpoint resumption in a weak network environment, the system comprising:

[0032] The generation module is used to receive the original business data collected on-site through the project edge, segment it according to the construction area and time interval, add a time and space identifier containing the area code, timestamp, segment sequence number and total number of segments to each piece of data, and generate a check code for the segmented data, generating a segmented data packet with the time and space identifier and check code;

[0033] The processing module is used to verify the received fragmented data packets, temporarily store them in the temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier and arrange them in sequence, synchronously record the reception time and verification result of each fragment, perform business preprocessing on the received complete region-time fragments, write the preprocessing progress, intermediate results and fragment reception status to the persistent file of local storage, and generate a fragment cache set with processing progress;

[0034] The execution module is used to reorganize the original data of a certain region-time segment in the shard cache according to the region code and timestamp sequence of the spatiotemporal identifier after receiving all the shards of the region-time segment. It reads the pre-processing progress of the corresponding business from the persistent file, continues to execute the remaining business processing from the point where the progress was interrupted, and outputs the business processing result with the resuming completion mark.

[0035] The integration module is used to integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each region and time segment, package them according to the project batch number, and generate a business data archive containing a complete data chain and processing trajectory.

[0036] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when run.

[0037] Yet another embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0038] Compared with the prior art, the present invention provides a method for data fragmentation, reorganization and business breakpoint resumption at the engineering edge in a weak network environment. The method receives the original business data collected on-site at the engineering edge, fragments it according to the construction area division and time interval, and generates a fragmented data packet with time and space identification and check code; verifies the received fragmented data packet to generate a fragmented cache set with processing progress; when all fragments of a certain area-time segment in the fragmented cache set are received, the preprocessing progress of the corresponding business is read from the persistent file, and the remaining business processing is continued from the point where the progress is interrupted; integrates the reorganized original data, business processing results, breakpoint resumption records and fragmented transmission logs of each area-time segment to generate a business data archive containing a complete data chain and processing trajectory, thereby improving data transmission efficiency and business continuity in a weak network environment and meeting the reliability requirements of engineering edge scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A hardware structure block diagram of a computer terminal for a method of engineering edge data fragmentation and reassembly and service breakpoint resumption in a weak network environment provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of a method for data fragmentation and reassembly at the engineering edge and service breakpoint resumption in a weak network environment provided by an embodiment of the present invention;

[0041] Figure 3 A structural diagram of a system for engineering edge data segmentation and reorganization and service breakpoint resumption in a weak network environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0043] The embodiment of the present invention first provides a method for engineering edge data fragmentation reorganization and service breakpoint resumption in a weak network environment. The method can be applied to electronic devices such as computer terminals, specifically ordinary computers.

[0044] The following describes it in detail by taking running on a computer terminal as an example. Figure 1The hardware structure block diagram of a computer terminal for a method of engineering edge data fragmentation and reassembly and service breakpoint resumption in a weak network environment provided by an embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0045] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, enable the processor to perform any method for engineering edge data fragmentation and reorganization and service breakpoint resumption in a weak network environment.

[0046] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0047] The internal memory provides an environment for the operation of computer programs in non-volatile storage media. When the computer program is executed by the processor, the processor can execute any method of engineering edge data segmentation and reorganization and business breakpoint resumption in a weak network environment.

[0048] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0049] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0050] See also Figure 2 The embodiment of the present invention provides a method for data fragmentation and reassembly at the engineering edge and service breakpoint resumption in a weak network environment, which may include the following steps:

[0051] S201: Receive raw business data collected on-site through the project edge, segment it according to the construction area and time interval, add a spatiotemporal identifier containing the area code, timestamp, segment sequence number, and total number of segments to each segment, and generate a check code for the segmented data, thus generating a segmented data packet with the spatiotemporal identifier and check code.

[0052] Specifically, the original business data collected at the engineering site can be feature extracted to identify the spatial and temporal features in the data, construct a spatiotemporal feature vector, and simultaneously calculate the initial hash value of the data to generate a spatiotemporal feature vector with the original hash.

[0053] Edge equipment at the construction site collects raw business data in real time through a network of sensors (such as GNSS positioning modules, inertial measurement units, and laser rangefinders). For example, in bridge stress monitoring, a single data entry might include: 3D coordinates (X=114.25 meters, Y=38.02 meters, Z=12.75 meters), collection time (UTC time 2023-05-12 14:25:36.128), stress value (18.75 MPa), and device ID (SENSOR_007). Spatial feature extraction first parses the coordinate data, converting it to an absolute position in the project coordinate system (e.g., using WGS84 to UTM projection), and adding a terrain elevation correction (e.g., +1.2 meters). Temporal feature extraction focuses on generating a time series from the data: timestamps are decomposed into a date ordinal (DOY=132) and a millisecond-level day offset (MS=51936128 milliseconds), while also calculating the interval between the timestamp and the previous data entry (Δt=250 milliseconds). After normalization (for example, coordinates are scaled to kilometers by dividing by 1000, and timestamps are converted to milliseconds in Unix epochs), these features form a spatio-temporal feature vector (STFV). For example, its dimensions are: [X:0.11425, Y:0.03802, Z:0.01275, DOY:132, MS:51936.128, Δt:0.250]. Simultaneously, the SHA-256 hash algorithm is applied to the raw data binary stream (containing coordinates, time, and measurement values) to generate a 64-bit hexadecimal initial hash value (e.g., "9a3f8b..."), which serves as a baseline anchor for data integrity. The resulting structured object, consisting of the STFV vector and hash value, is called a hash-augmented STFV (HA-STFV).

[0054] The feature extraction process must adapt to multi-source, heterogeneous data. For example, in a tunnel boring scenario, shield machine sensor data includes parameters such as cutterhead torque (1200 kN·m), propulsion speed (35 mm / min), and earth pressure (2.3 bar). Spatial features must be associated with the real-time excavation distance of the cutterhead center (e.g., K12+345.6), which is converted into three-dimensional coordinates (X, Y, Z) using the shield guidance system. Temporal features, in addition to the acquisition time, must also include the time offset from the previous segment assembly (e.g., +15 minutes and 30 seconds). This offset reflects the continuity of the construction phase. For unstructured data (e.g., crack images captured by inspection drones), spatial features are extracted using GPS coordinates in the image's EXIF ​​format, and temporal features are derived from the file creation timestamp. A VGG16 convolutional neural network is used to extract deep image features (e.g., crack length proportion 0.15, width variance 0.08). These features are then integrated into the STFV vector after dimensionality reduction. Hashing is performed on the image binary stream to ensure that any pixel change results in a significant change in the hash value. Finally, HA-STFV forms a unified data representation format, laying the foundation for subsequent processing.

[0055] To ensure feature robustness, the system introduces a dual verification mechanism. First, spatial features must pass spatial topology verification: for example, in foundation pit monitoring, if the distance difference between adjacent measuring points (measured 3.2 meters vs. designed value 3.5 meters) exceeds a threshold (±0.5 meters), the coordinate correction algorithm (such as position adjustment based on the least squares method) is triggered. Temporal features implement a timing rationality check: if the Δt of two consecutive data from a sensor is 0 (possibly due to a clock failure), the system automatically replaces it with the average sampling interval of the same type of sensor (such as 200 milliseconds). The hash calculation adopts a block strategy - large files (such as BIM models) are divided into 1MB data blocks, the SHA-256 hash is calculated independently for each block, and then the block hash set is aggregated by Merkle tree to generate a global initial hash. This design not only ensures the efficiency of large file processing, but also supports subsequent fragmentation verification. The final output HA-STFV is encapsulated in JSON format, including a feature vector array, a hash value, a data source type tag (such as "vibration" and "image"), and a verification status code (such as STATUS_CODE=200, indicating verification passed).

[0056] Based on the spatial features in the spatiotemporal feature vector, the region is divided and the K-means clustering algorithm is used to automatically determine the optimal number of regions. Each region is assigned a unique code to generate a spatiotemporal feature vector with region codes.

[0057] The core of regional segmentation is spatial clustering. The system extracts a subset of three-dimensional coordinates from the HA-STFV (e.g., [X, Y, Z] in the previous example) and constructs an N×3 spatial coordinate matrix (N is the number of data to be processed). K-means clustering first determines the optimal number of clusters, K, using a combination of the elbow method and the silhouette coefficient. In practice, the search range for K is set (e.g., K_min=2, K_max=10), and 10 clustering iterations are performed for each K value to eliminate randomness in the initial centroids. The elbow method calculates the within-cluster sum of squares (WCSS) for different K values. When K increases from 3 to 4, the WCSS decreases sharply from 15% to 5%, forming an "elbow point," leading to the initial determination of K=3. The silhouette coefficient, which measures cluster compactness (ranging from -1 to 1), reaches an average of 0.62 for K=3 (better than 0.48 for K=2 and 0.55 for K=4), confirming that K=3 is optimal. Finally, the number of clusters K = 3 is automatically selected, and the initial position of the centroid is generated by the K-means++ optimization algorithm to avoid the local optimum caused by traditional random initialization.

[0058] The clustering process employs an iterative reassignment strategy. For example, in subway construction, the initial centroid locations might be set at the shield tunnel launch shaft (C1 = [X1, Y1, Z1]), the intermediate ventilation shaft (C2 = [X2, Y2, Z2]), and the receiving shaft (C3 = [X3, Y3, Z3]). In the first iteration, the Euclidean distances from each measuring point (e.g., segment displacement sensor) to the three centroids are calculated (e.g., Dist1 = 35.2 meters, Dist2 = 120.5 meters, Dist3 = 98.7 meters), and each point is assigned to the cluster with the closest centroid. After assignment, the centroid coordinates of each cluster are recalculated (e.g., the new coordinates of C1 are the mean of all points within the cluster). After five iterations, clustering converges when the centroid movement distance is less than a threshold (e.g., 0.1 meters). This results in three spatial clusters: Cluster 1 (ring 0-300 of the launch segment), Cluster 2 (ring 301-650 of the intermediate segment), and Cluster 3 (ring 651-950 of the receiving segment). Each cluster is assigned a unique area code (AC) using the "project ID_cluster ID" encoding format (e.g., PROJ002_AC01). This encoding information is injected into the original HA-STFV to form an area-coded spatiotemporal feature vector (Area-CodedSTFV, AC-STFV). For example, the following fields are added: {"area_code": "PROJ002_AC01", "centroid_dist":12.5} (the latter is the distance from the data point to the centroid of the cluster, used for subsequent anomaly detection).

[0059] For dynamic construction scenarios, the system implements incremental cluster updates. As new data continues to arrive (e.g., every 10 tunneling rings), only new data points are rapidly assigned to clusters: their distance to the existing K centroids is calculated, assigned to the nearest cluster, and the coordinates of that cluster's centroid are updated (no full re-clustering is required). Cluster evolution is also monitored. If the radius of a cluster (the distance from its farthest point to its centroid) continues to increase beyond a threshold (e.g., 50 meters), or if the distance between adjacent cluster centroids decreases to a critical value (e.g., 20 meters), a cluster split / merge mechanism is triggered. For example, if the distance between two tunnel working surfaces is less than 20 meters, the original AC01 and AC02 clusters are automatically merged into a new cluster, AC04, and the system reassigns region codes. All region boundary information is written to a spatial index file (e.g., an R-tree structure) to support fast geofence queries. Ultimately, the region codes carried by the AC-STFV become the key index for subsequent fragmentation and reorganization.

[0060] Dynamically determine the sharding time interval based on the data generation cycle and processing capacity, perform sliding sharding on the spatiotemporal feature vector with region coding according to the time window, add a timestamp and shard sequence number to each piece of data, and calculate the total number of pieces at the same time to generate sharded data with spatiotemporal identifiers;

[0061] The core parameter of time slicing is the dynamic time window (DTW). The window length T_win is determined by two factors:

[0062] Data generation period: Calculates the average arrival interval Δt_avg (e.g., 200 milliseconds) of the most recent M data packets (e.g., M=100). If Δt_avg ≤ 500 milliseconds, the data flow is defined as "high frequency." If 500 milliseconds < Δt_avg ≤ 2 seconds, it is defined as "medium frequency." Δt_avg > 2 seconds is defined as "low frequency."

[0063] Edge processing capacity: Use the performance monitoring module to obtain CPU utilization (e.g., 75%) and remaining memory (e.g., 512MB). The processing capacity index (CPI) is defined as (1 - CPU utilization) × remaining memory. A CPI > 300 indicates "high load," 150-300 indicates "medium load," and < 150 indicates "low load."

[0064] Adaptively set T_win according to the following table:

[0065] High frequency + high load: T_win = 1 second (fast sharding reduces real-time pressure); medium frequency + medium load: T_win = 5 seconds (balancing sharding efficiency and processing granularity); low frequency + low load: T_win = 30 seconds (large sharding reduces transmission overhead). For example, if Δt_avg = 300 milliseconds (high frequency) and CPI = 280 (medium load), T_win = 3 seconds.

[0066] Implement sliding time slicing based on T_win. Taking the region code PROJ002_AC01 as an example, the timestamp range of the cluster data is T_start = 14:25:00 to T_end = 14:30:00. Starting from T_start, split the data with a window length of T_win = 3 seconds and a sliding step size S_step = 1.5 seconds (the default is T_win / 2):

[0067] Window 1: 14:25:00.000 - 14:25:03.000;

[0068] Window 2: 14:25:01.500 - 14:25:04.500; ...

[0069] All AC-STFVs in each window form a data shard. Add metadata to the shard:

[0070] Timestamp: window start time (e.g., 14:25:01.500);

[0071] Shard sequence number: increasing in chronological order (Shard_ID=002);

[0072] Total number of slices: The estimated total number of slices for this region-time segment is N_total = ceil((T_end - T_start) / S_step).

[0073] For example, in a 10-minute period, when S_step = 1.5 seconds, N_total = 400. The resulting shard data with spatio-temporal tags (STT-Shard) contains the following structure:

[0074] Identification header: {region: "PROJ002_AC01", start_time: 1683901501500, shard_id: 002, total_shards: 400};

[0075] Data body: the collection of all AC-STFVs in the window.

[0076] To cope with data bursts (such as the instantaneous generation of thousands of data items during blast monitoring), the system introduces an adaptive window expansion mechanism. When the amount of data in a single window exceeds a threshold (e.g., 500 items), the following actions are automatically triggered:

[0077] Dynamically expand the current window length to T_win' = T_win × 2 (e.g., 3 seconds → 6 seconds);

[0078] Freeze subsequent window generation and temporarily store overflow data in the buffer queue;

[0079] After the current window is split, reset T_win to its initial value.

[0080] Each shard is also marked with a density flag (e.g., density_level=HIGH). High-density shards are prioritized for transmission over weak networks using compression algorithms (e.g., LZ77). The spatiotemporal identifiers of all shards are written to the global shard map table (Shard MapTable). This table uses a two-level index: the first level is partitioned by region-coded hash, and the second level is sorted by timestamp range (implemented as a B+ tree), enabling fast retrieval of missing shards. Finally, STT-Shards are written directly to the send buffer using zero-copy technology, reducing memory copy overhead.

[0081] The fragmented data with time and space identifiers are checked in blocks, and a check code is generated by combining cyclic redundancy check and Bloom filter. The check code is embedded in the time and space identifier to generate a fragmented data packet with time and space identifier and check code.

[0082] Sharding verification adopts a layered verification architecture:

[0083] First layer: Cyclic Redundancy Check (CRC)

[0084] The fragmented data body is divided into blocks of 1KB in size, and the CRC-32 checksum (32-bit polynomial 0x04C11DB7) is calculated independently for each block.

[0085] For example, if a segment contains 80KB of data, 80 CRC values ​​(CRC_001 to CRC_080) are generated.

[0086] All CRC values ​​are aggregated again to calculate the super-CRC as the overall check code of the data body.

[0087] Second layer: Bloom filter

[0088] Create a bit array with length L = 1024 bits (about 128 bytes) and use K = 3 hash functions (such as MurmurHash3, FNV1a, Jenkins).

[0089] Input the unique identifier of each data block (such as "region code_shard ID_block number") into the Bloom filter, corresponding to position 1.

[0090] Finally, the Bloom filter bitmap (BF_Bitmap) is generated.

[0091] The check code (CC) is composed of the Super-CRC and BF_Bitmap, with a total length of 160 bytes (4-byte CRC + 128-byte bitmap).

[0092] The verification code embedded in the time and space identification needs to resolve the conflict problem. System design identification extension protocol:

[0093] Add a "Check Header" field to the original time and space identification header:

[0094] Format identifier (2 bytes): fixed 0xCCBF indicates the use of CRC+BF combination;

[0095] Block size (2 bytes): For example, 1024 indicates 1KB blocks;

[0096] Bloom filter parameters (4 bytes): includes L value (10 bits), K value (6 bits), and reserved bits (20 bits).

[0097] The 160-byte checksum is appended to the end of the identification header. For example, the complete identification header becomes:

[0098] {region:"PROJ002_AC01",start_time:...,shard_id:...,total_shards:...,check_header:{format:0xCCBF,block_size:1024,bf_params:0xA3F00000},check_code:[Super-CRC, BF_Bitmap]}.

[0099] This structure is called the Enhanced Spatio-Temporal Tag (EST-Tag). In the Bloom filter parameters, L = 1024 corresponds to the upper 10 bits of bf_params, which have a value of 1023 (0x3FF), and K = 3 corresponds to the lower 6 bits, which have a value of 3 (0x03). Therefore, bf_params = 0x3FF << 20 | 0x03 << 14 = 0xA3F00000 (all reserved bits are 0).

[0100] The receiving end uses a two-level verification strategy:

[0101] Fast verification layer: Extracts the BF_Bitmap in the EST-Tag and checks whether all data block identifiers are in the bitmap (the false positive probability is about 0.01%). If it passes, it enters the transmission queue; if it fails, it triggers a retransmission request.

[0102] Deep Verification Layer: For fragments that pass the BF check, CRC-32 is calculated for each block and compared with the Super-CRC. If a block CRC error occurs (e.g., Block_032 CRC mismatch), the error block number is recorded. Only the error block (not the entire fragment) is requested to be retransmitted, and the offset is determined based on the block size in the EST-Tag.

[0103] The final data packet consists of three parts:

[0104] Packet header: EST-Tag (including checksum);

[0105] Package body: original shard data;

[0106] Packet tail: frame end mark (0xFFFF);

[0107] This structure takes into account both verification strength and transmission efficiency - BF provides second-level false detection screening, CRC ensures bit-level accurate verification, and the block mechanism supports local retransmission, significantly reducing bandwidth waste in weak network environments.

[0108] S202: Verify the received fragmented data packets and temporarily store them in a temporary storage area of ​​the corresponding region according to the fragment sequence number in the spatiotemporal identifier and arrange them in order. Synchronously record the reception time and verification result of each fragment. Perform business preprocessing on the received complete region-time fragments. Write the preprocessing progress, intermediate results, and fragment reception status to a persistent file in local storage to generate a fragment cache set with processing progress.

[0109] Specifically, the received fragmented data packets can be verified by checking the code, and the packets that pass the check can be marked by parallel CRC calculation and Bloom filter fast matching, and fragmented data packets with check marks can be generated;

[0110] When the engineering edge device receives a fragmented data packet transmitted from the field, it first initiates the checksum verification process. Two checksums are embedded in the spatiotemporal identifier of each data packet: a cyclic redundancy check (CRC) and a bitmap signature generated by a Bloom filter. The CRC detects bit-level errors in data transmission (such as bit flips caused by signal interference), while the Bloom filter quickly determines whether a data packet is a legitimate fragment of the current project batch (to prevent forged or misplaced data injection). The verification process utilizes a parallel computing architecture: multiple computing units simultaneously process different data packets. For CRC verification, each computing unit independently calculates the CRC32 value (a 32-bit cyclic redundancy check) of the received data fragment and compares it with the original CRC embedded in the data packet. If the two match, the result is marked as "CRC_PASS"; otherwise, it is marked as "CRC_FAIL". Bloom filter verification is performed by querying a pre-generated bitmap. This bitmap maps the unique identifiers of all legitimate fragments (such as the region code, timestamp, and fragment sequence number combination) when the data fragments are generated. The verification unit extracts key fields from the packet's spatiotemporal identifier, applies the same hash function, and then queries the bitmap. If all mapped bits are "1," the packet is marked "BF_PASS" (high probability of validity); if any mapped bit is "0," the packet is marked "BF_FAIL" (definitely invalid). Only if a packet receives both "CRC_PASS" and "BF_PASS" is it marked as passed and a green "VALID" label is added to the packet header; otherwise, a red "INVALID" label is added. This parallel verification mechanism completes in milliseconds, ensuring efficient filtering of corrupted or malicious packets in weak network environments.

[0111] The reliability of the verification process is enhanced through a multi-layered security mechanism. Parallel CRC calculations are accelerated using the SIMD (Single Instruction Multiple Data) instruction set. For example, on processors equipped with the ARM NEON or Intel AVX2 instruction set, CRC values ​​for 4-8 data blocks can be calculated simultaneously, significantly improving throughput. To address potential false positives in Bloom filters, the system sets a dynamic false positive rate (FPR) threshold, initially at 0.001. When the volume of a particular batch of data increases dramatically, the system automatically expands the bitmap size and adjusts the number of hash functions to keep the FPR within the preset threshold. All verification results (including CRC calculation results, Bloom query results, and final markings) are recorded in a temporary memory table and associated with the packet's spatiotemporal identifier. For packets marked "INVALID," the system triggers a retransmission request: the region code, timestamp, and fragment sequence number from the spatiotemporal identifier are extracted, encapsulated into a NACK (negative acknowledgment) message, and rapidly fed back to the data source via the UDP protocol. At the same time, to cope with network jitter, the system sets a verification timeout window (for example, 2000 milliseconds). Data packets that fail to complete verification within the timeout are automatically marked as "TIMEOUT_INVALID" to avoid blocking subsequent processes.

[0112] The final generated fragmented data packet with checksum contains three core pieces of information:

[0113] Original data payload: uncompressed fragment content; spatiotemporal identifier: region code (such as "A-102"), timestamp (such as "20231015143000"), fragment sequence number (such as "5 / 20"); checksum segment: includes CRC status (CRC_PASS / CRC_FAIL), Bloom status (BF_PASS / BF_FAIL), comprehensive verification result (VALID / INVALID), and timestamp (marker generation time).

[0114] This structure is encapsulated as a binary message and passed to the next processing module via a shared memory queue. Data packets marked "VALID" enter the cache; "INVALID" packets are moved to a quarantine area for manual review or automatic cleanup. This step ensures that only complete, legitimate, and untampered data fragments are subsequently reassembled, providing the first line of defense for data reliability in weak network environments.

[0115] According to the region code in the time-space identifier, the fragmented data packets with checksum marks are routed to the temporary storage area of ​​the corresponding region, and a skip table data structure is used to quickly insert and sort the fragment sequence numbers to generate an ordered fragment set;

[0116] Packets that pass verification are intelligently routed based on the area code in their spatiotemporal identifier (e.g., "A-102" represents Construction Unit 102 in Area A). The system maintains a regional routing table, recording the memory address of the temporary storage zone (TSZ) corresponding to each area code. The routing engine parses the area code field in the packet header, matches the routing table entry through a hash lookup, and directs the packet to the target TSZ. Each TSZ is an independent storage unit managed by a memory pool, physically isolating data from different zones to prevent data from being mixed up. For example, packets from area "A-102" enter only the TSZ_A102 pool, while packets from area "B-205" enter the TSZ_B205 pool. This design meets the requirements for regional independence at construction sites and facilitates parallel processing.

[0117] After a data packet enters the target TSZ, it must be sorted and stored by fragment number (e.g., "5" in "5 / 20"). Traditional linked list sorting is inefficient in scenarios with frequent insertions, so a skip list data structure is used. A skip list is a multi-layered, ordered linked list whose core approach is to accelerate searches by constructing an index layer through probability balancing. The specific implementation is as follows:

[0118] Basic layer: stores all shard nodes and links them in ascending order;

[0119] Index layer: Randomly select some nodes to build the upper-level index (for example, select one node from every two nodes to the L1 layer, and then select one node from every two L1 nodes to the L2 layer);

[0120] Insertion Logic: When a new data packet arrives, the search begins at the highest-level index (e.g., L2), descending layer by layer to locate the insertion position, ultimately inserting the new node at the base layer. The average time complexity is O(log n), far superior to the O(n) time complexity of a linked list. For example, when inserting shard number "7," the skip list might first locate "6" through the L2 layer, then locate "6" between "6" and "8" through the L1 layer, ultimately inserting "7" between the two at the base layer. Furthermore, the system dynamically adjusts index density: when the number of shards within a TSZ exceeds a threshold (e.g., 100), the number of index levels is automatically increased to maintain efficient search.

[0121] The resulting ordered shard collection has the following characteristics:

[0122] Physical order: The base layer nodes are arranged in a strictly ascending order according to the shard number;

[0123] Fast positioning: The index layer can locate any serial number shard in logarithmic time;

[0124] Fault-tolerance processing: When a sequence number conflict is detected (such as repeated insertion of sequence number "5"), the system retains the shard with the latest receiving timestamp and records the conflict event in the log;

[0125] Visible status: Each node associates the checksum, reception timestamp, and origin pointer of the data packet.

[0126] This structure provides an immediately available ordered data chain for subsequent reorganization. Even if the fragments arrive out of order, they can be quickly reassembled into a time-corrected set in memory.

[0127] Record the reception time and verification result of each shard, construct a five-tuple log containing the region code, timestamp, shard sequence number, reception time, and verification result, and write it to the local disk in real time using memory-mapped file technology to generate a time series log stream.

[0128] Key events in each shard processing process (such as reception completion, verification results, and storage location) must be persistently recorded. The system defines a five-tuple log format:

[0129] Zone ID: such as "A-102";

[0130] Timestamp (Data_TS): the original timestamp of the shard (such as "20231015143000");

[0131] Segment number (Seg_Num): such as "5" in "5 / 20";

[0132] Receiving time (Recv_Time): the time when the edge actually completes receiving the data (e.g., "20231015143025.423");

[0133] Check result (Check_Result): such as "VALID" or "INVALID".

[0134] Whenever a shard completes routing storage, the log generator packages these five fields into a log record.

[0135] Log writing uses memory-mapped file (MMF) technology. This technology maps disk files to the process virtual memory space, making log writing as efficient as operating memory:

[0136] Initialization: Create a log file of fixed size (e.g. 500MB) and map it to the virtual address space (e.g. starting address 0x70000000);

[0137] Writing process: Each five-tuple log is serialized into a binary block (e.g., 64 bytes) and written directly to the current offset in the mapping area.

[0138] Synchronization mechanism: Every time 50 logs are written or every 50 milliseconds, an msync() system call is triggered to flush dirty pages to disk;

[0139] Rolling archive: When the log file is 80% full, a new file is automatically created and the old file is marked as read-only.

[0140] This technology avoids the system call overhead of traditional file IO, with a write latency of less than 100 microseconds. In addition, when a crash occurs, the operating system automatically synchronizes unflushed data to ensure that logs are not lost.

[0141] Continuously written logs form a time-ordered log stream, which has the following management features:

[0142] Global ordering: Each log is timestamped (accurate to milliseconds) to ensure that all system events can be traced back.

[0143] Fast retrieval: Create a hash index of region code + shard sequence number to support locating specific shard logs in seconds;

[0144] Stream processing: The background process parses the log stream in real time and generates a heat map of the region-level shard receiving status (e.g., "A-102: 15 / 20 shards received").

[0145] Fault recovery: When the system restarts, replaying the log stream can rebuild the state of the shard cache set in memory.

[0146] This time series log stream is the core basis for resuming data transmission and is also the legal record for auditing data integrity.

[0147] Perform business preprocessing on the received complete region-time segments, use the state machine model to define the preprocessing process, and generate an intermediate result set with processing status; integrate the preprocessing progress, intermediate result set and time series log stream, write them to the locally stored state file through the incremental persistence algorithm, and generate a shard cache set with processing progress.

[0148] When all shards for a region-time segment (e.g., "A-102_20231015143000") arrive and pass verification (e.g., 20 / 20 shards received), the system automatically triggers business preprocessing. Preprocessing performs lightweight computations (e.g., data decoding, outlier filtering, and statistical feature extraction) on the raw shard data to prepare for subsequent main business processing. This process is strictly defined by a Finite State Machine (FSM):

[0149] State: includes pending, decoding, cleaning, feature extraction, and completed;

[0150] Transition condition: automatically jump after the above state is completed;

[0151] Action: Each state is bound to a processing function (such as decode_JT808() for parsing JT808 protocol data).

[0152] For example, the data preprocessing process of a certain engineering machinery:

[0153] To be processed → (fragments collected) → Decoding: parsing the original CAN bus frame;

[0154] Decoding → (Decoding successful) → Cleaning: Filter speed exceeds limit;

[0155] Cleaning → (Cleaning Completed) → Feature Extraction: Calculating 5-minute average fuel consumption;

[0156] Feature extraction in progress → (Calculation completed) → Completed.

[0157] The state machine ensures that the processing flow can be interrupted and rolled back.

[0158] The preprocessing process generates an intermediate result set with processing status:

[0159] Metadata: current state machine status (e.g., "Feature Extraction"), progress percentage (e.g., "70%");

[0160] Intermediate data: outputs from each stage (such as decoded JSON arrays, cleaned data tables, and feature vectors);

[0161] Checkpoint: A snapshot of key variables (such as the cumulative number of processed records and the timestamp of the last valid data).

[0162] The result set is temporarily stored in shared memory for direct reference by the main business processing module.

[0163] To achieve breakpoint resumable transmission, the system needs to persist the preprocessing progress. An incremental persistence algorithm is used:

[0164] Initial full write: When the state machine enters a new phase, the entire intermediate result set is serialized and written to the state file;

[0165] Incremental append: During processing, only the changed parts (such as the 100 newly added decoding records) are appended to the end of the file;

[0166] Index separation: Store index files independently and record the offset and checksum of full / incremental data;

[0167] Merge and compress: The background process periodically merges incremental records to generate a new full snapshot.

[0168] For example, during the feature extraction phase, feature calculation results are appended to a file every minute, and merged to generate a new snapshot every hour. Persistent data and time series log streams are associated through region-time segment IDs, forming a sharded cache set with processing progress. This cache set supports three key operations:

[0169] Progress query: Returns the status of any area-time segment in real time (e.g. "Cleaning, progress 45%");

[0170] Breakpoint recovery: If the system crashes, load the most recent snapshot from the state file to continue processing;

[0171] Cache elimination: Automatically archive data that has been pre-processed for more than 24 hours to release space.

[0172] At this point, the sharded data has completed the entire process of receiving, verifying, sorting, and preprocessing, providing a ready environment for resuming breakpoint transmission.

[0173] S203: When all the fragments of a certain region-time segment in the fragment cache are received, the original data of the segment is reassembled according to the region code and timestamp sequence of the spatiotemporal identifier, the pre-processing progress of the corresponding business is read from the persistent file, the remaining business processing is continued from the point where the progress was interrupted, and the business processing result is output with a resume completion mark;

[0174] Specifically, when all shards of a certain region-time segment in the shard cache are verified, the reorganization process is triggered. According to the region code and timestamp order of the spatiotemporal identifier, a merge sort algorithm is used to merge the ordered shard sets to generate reorganized original data with region time stamps.

[0175] When the monitoring service on the edge computing node detects that all fragments for a specific zone-time segment (for example, "Construction Zone B - May 1, 2024, 2:00 PM to 2:05 PM") have been received and verified (with the verification status marked as "Valid"), the system automatically triggers the data reassembly process. First, the reassembly engine extracts all fragmented data packets for this segment from the temporary storage area of ​​the fragment cache. These fragments are sorted in a skip list data structure based on the chunk sequence number (CSN) in the spatiotemporal identifier. The reassembly engine globally sorts the fragments based on two core dimensions of the spatiotemporal identifier: the zone code (ZC, such as "ZC-B-102") and the timestamp (TS, such as "20240501140000" to "20240501140500"). The zone code isolates data from different construction zones, while the timestamp ensures that data within the same zone is sorted chronologically. For example, three shards with adjacent timestamps in region B (TS1:14:00:00, CSN=1; TS2:14:02:00, CSN=1; TS1:14:00:00, CSN=2) will be sorted as [TS1-CSN1, TS1-CSN2, TS2-CSN1].

[0176] After sorting, the reorganization engine uses the Merge Sort algorithm to merge shard data. This algorithm is particularly well-suited for processing partially ordered shard sets: shards are first split into smallest units by time window (e.g., one group per time window), and then merged and sorted pairwise. For example, the 10 shards in region B (covering five consecutive time windows) are divided into five groups. Adjacent window groups (e.g., windows 1+2, 3+4) are first merged, and the resulting groups are then recursively merged. During the merge process, the algorithm compares the shards' timestamp-sequence composite key (TSCK) and dynamically adjusts the shard positions. The merge operation is performed in memory, using zero-copy technology to directly manipulate the buffer pointers of the shard packets, avoiding the performance loss associated with data duplication. The final output reassembled raw data (RRD) is a continuous, complete binary data stream with a zone-time marker (ZTM) embedded in its header in the format of "ZC-B-102|20240501140000-20240501140500", which is used to identify the temporal and spatial attributes of the data.

[0177] To ensure the reliability of the reorganization process, the system calculates the cyclic redundancy check code (CRC) of the reorganized data in real time during the merge process. For example, a CRC-32 value is calculated for every 1MB of data merged and cumulatively compared with the original shard checksum. If a data inconsistency is detected (such as a bit error within a shard caused by network transmission), the reorganization engine immediately retrieves the backup copy of the shard from the persistent log and repairs it. After the reorganization is complete, the RRD is written to the cache, triggering a retransmission notification in the business processing module. The entire process is protected by transactional memory, ensuring that even in the event of an unexpected system outage, the reorganized state can be rolled back to the most recent consistent point.

[0178] Read the preprocessing progress of the corresponding business from the persistence file, parse the processing status mark in the status file, locate the breakpoint position of the most recent successful processing, and generate a breakpoint context snapshot;

[0179] After the business retransmission process is initiated, the system accesses a locally stored persistent state file (PSF). This file stores the preprocessing progress (PP) for each region-time segment in the form of key-value pairs. The key is the region time stamp (such as "ZC-B-102|20240501140000"), and the value is a structured state record. The state record contains three key fields:

[0180] Processing State Flag (PSF): such as "UNPROCESSED", "PARTIAL", and "COMPLETE";

[0181] Breakpoint Position Identifier (BPI): records the offset of the last successfully processed data block (e.g., byte offset "offset=10240");

[0182] Intermediate Result Digest (IRD): A compact representation of the preprocessing output (such as the hash value of a feature vector).

[0183] The system loads the PSF into memory using memory-mapped file (MMF) technology, achieving millisecond-level access.

[0184] The parsing engine locates the corresponding status record based on the currently processed zone timestamp (e.g., "ZC-B-102|20240501140000"). If the PSF value is "PARTIAL," the BPI and IRD fields must be parsed. The breakpoint position (BreakpointPosition) is typically stored as a two-tuple:

[0185] Data Offset (DO): The byte position in the reconstructed raw data (RRD) (e.g., "offset=20480");

[0186] Processing Stage ID (PSID): Identifies a specific step in the preprocessing pipeline (e.g., "STAGE3_FEATURE_EXTRACTION").

[0187] For example, the parsing result might be "The breakpoint is at byte 20480 of the RRD, corresponding to the input buffer of the feature extraction stage." At the same time, the IRD fields are restored to intermediate result objects (such as the deserialized feature vector matrix).

[0188] Based on the resolved breakpoint location, the system generates a breakpoint context snapshot (BCS). The snapshot contains three key contexts:

[0189] Data context: Extracts the data window before and after the breakpoint from the RRD (e.g., 2KB of data before and after the breakpoint) to reinitialize the input buffer of the processing module.

[0190] Processing state context: Reconstructs the runtime state of the preprocessing module, including register values, stack frames, thread lock status, etc.

[0191] Intermediate result context: loads persisted intermediate results (such as deserialized feature matrices in IRDs).

[0192] Snapshot generation uses Copy-on-Write (COW) technology to ensure that the original data is not modified. Finally, the BCS is encapsulated as an independent memory object for the resume process to call.

[0193] Load the breakpoint context snapshot, import the reorganized raw data with regional time stamps into the business processing environment, and continue executing the remaining business processing flow from the breakpoint. Use shadow execution technology to synchronously record intermediate calculation steps and state changes to generate a continued execution trajectory.

[0194] After the business processing engine loads the breakpoint context snapshot (BCS), the execution environment is restored to the state at the breakpoint. First, the reconstructed raw data (RRD) is imported into the processing environment as needed:

[0195] Processed part: only create memory mapping pointer to avoid repeated loading;

[0196] Unprocessed portion: Starting from the breakpoint location, it is dynamically loaded into the input buffer in a streaming manner (for example, 4KB of data is loaded each time).

[0197] The processing environment initializes business parameters based on the zone time tag (ZTM). For example, if the ZTM contains the zone code "ZC-B-102," the system automatically loads the zone-specific processing template (such as the concrete strength analysis algorithm for zone B).

[0198] The resumed transfer starts precisely from the breakpoint location (defined by BPI). To ensure execution continuity, the system uses Shadow Execution Technology:

[0199] Main execution thread: actually executes business logic and continues running from the breakpoint;

[0200] Shadow thread: Simulates the main thread operations in parallel and records all intermediate state changes (such as variable modifications, function call stacks, and memory allocations).

[0201] For example, in the "Vibration Signal Spectrum Analysis" service, the main thread resumes the calculation from the breakpoint (e.g., the 512th sample point in the FFT calculation), while the shadow thread records the amplitude calculation results and spectrum matrix update status for each sample point in real time. All operations are recorded as Execution Trace Events (ETEs), which contain metadata such as the operation type, timestamp, and impact range.

[0202] The resume execution trace (RET) generated by the shadow thread is stored in a structured manner:

[0203] Operation Sequence: records state change events in chronological order;

[0204] State Snapshot Chain: A complete state backup of key nodes (e.g., a snapshot is generated every time 1% of data is processed);

[0205] Dependency Graph: Marks data dependencies between operations.

[0206] Trajectory data is written to non-volatile memory (NVM) in real time to ensure data is not lost during power outages. For example, in a structural health monitoring application, the trajectory records the results and dependencies of 3,200 newly added strain calculation nodes after resuming the data from the interruption point.

[0207] After the business processing is completed, the execution trace of the subsequent transmission is compared and verified with the original processing record, and the integrity of the processing process is ensured through the hash chain, and a verified business processing result is generated;

[0208] After the business processing is completed, the system starts the integrity verification process. First, the original processing record (OPR) is extracted from the persistent file. This record contains the execution trace summary before the interruption (such as the Merkle tree root hash of the operation sequence). At the same time, the execution trace (RET) of this transfer is read from the NVM. The verification is divided into two steps:

[0209] Timeline connection verification: confirm the logical connection between the start event of the resumed transmission trajectory and the interruption event of the original record (for example, the last operation during the interruption is "OP1023", and the first operation during the resumed transmission is "OP1024");

[0210] State continuity check: Compare the state snapshot at the time of interruption with the initial state of the resumed transfer to see if they are consistent (such as memory checksum comparison).

[0211] The core verification mechanism is Hash Chain technology:

[0212] Original processing stage: Each operation generates a hash value (such as SHA-256), which contains the current operation content and the hash value of the previous operation, forming a chain structure;

[0213] Continuation phase: Starting from the end hash of the original hash chain (such as "H_OP1023"), a continuation hash chain is generated for the new operation (such as "H_OP1024 = SHA256(OP1024 || H_OP1023)").

[0214] During verification, the system recalculates the cumulative hash value of the entire retransmission trajectory and compares it with the final stored Terminal Hash (TH). If they match, it proves that the processing process has not been tampered with or omitted.

[0215] After verification, the system generates a Business Processing Result (BPR). The result contains three types of data:

[0216] Analysis output: such as structural stress distribution diagrams and equipment fault diagnosis reports; quality indicators: such as data integrity rate of 99.98% and processing delay statistics; verification credentials: including hash chain terminal hash and verification timestamp.

[0217] The results are encapsulated into a self-describing data structure (e.g., JSON-LD format) and marked as "VERIFIED." Results that fail verification trigger an alert and are isolated.

[0218] Embed a resume completion mark containing the breakpoint location, resume time, and verification hash in the business processing result. Use a digital signature to ensure that the mark cannot be tampered with, and output the business processing result with the resume completion mark.

[0219] The system generates a Resume Completion Mark (RCM), which contains three key fields:

[0220] Breakpoint Position (BP): inherited from the BPI in the breakpoint context snapshot (e.g., "offset=20480|STAGE3"); Resume Timestamp (RT): the exact time when the resume start (e.g., "2024-05-01T14:07:23.456Z"); Verification Hash (VH): the terminal hash value of the hash chain (e.g., "SHA3-512:9a8f...").

[0221] Tags are stored using TLV encoding (Type-Length-Value) to ensure scalability.

[0222] To prevent tampering, digital signatures are used to protect the tag: the RCM is signed using the edge node's private key (e.g., using the ECDSA algorithm); a signature value (SV) is appended to the tag's tail; and the edge node's certificate chain is embedded in the business result, allowing the verifier to verify the certificate's legitimacy. For example, the signing process involves first calculating the RCM's SHA-256 digest, then encrypting the digest using the P-256 elliptic curve private key, and outputting a 64-byte signature.

[0223] The final output of the business processing result with the resumable transfer completion mark (Signed Result with RCM, SR-RCM) contains four layers:

[0224] Result data ontology: such as concrete strength prediction value; retransmission mark: RCM containing BP / RT / VH; digital signature: joint signature of result ontology + RCM; certificate reference: URL pointing to edge node certificate.

[0225] The results are sent to the cloud via a message queue (e.g., MQTT) and backed up locally. Any recipient can verify the integrity and authenticity of the results after resuming transmission on a weak network by verifying the signature and hash chain.

[0226] S204: Integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each region-time segment, package them according to the project batch number, and generate a business data archive containing a complete data chain and processing trajectory.

[0227] Specifically, the reconstructed original data, business processing results, breakpoint resume records and fragment transmission logs of each region-time segment can be collected to build a four-tuple data structure, generate a unique content addressable identifier for each data item, and generate a data set with an identifier;

[0228] Data collection and structured packaging:

[0229] The system first extracts four key data from distributed storage nodes:

[0230] Reorganize the original data: Use spatiotemporal identifiers (area code + timestamp) to locate the merged and sorted complete business data segments (such as the sensor reading set of Area A of a construction site from 2023-10-01 09:00 to 09:15).

[0231] Business processing results: Load the processing result file with the resumable transfer completion mark (such as the noise filtering and unit conversion results of the above data).

[0232] Resume download record: Read the breakpoint context snapshot in the persistent file (recording the resume start position, resume timestamp, and verification hash value).

[0233] Shard transfer log: retrieves five-tuple records (region code, timestamp, shard sequence number, receipt time, verification result) from the time series log stream.

[0234] The above four data are encapsulated into a four-tuple data structure at the "region-time segment" granularity. Its memory layout contains four fixed field pointers, which point to the physical storage address and metadata description (such as data size and generation time) of the above data respectively.

[0235] Content-addressable identifier generation:

[0236] Generate a unique content addressable identifier (CID) for each independent data item in the quadruple, using multi-layer hash chain technology:

[0237] Apply the SHA3-256 algorithm to the binary stream of a single data item (such as a shard transmission log) to calculate the initial hash value (output length 256 bits).

[0238] The hash value is concatenated with the data feature descriptor (such as data type and region code), and the BLAKE3 hash operation is performed again to generate the final CID (for example, bafkreihdwdcefgh4dq6vhuo).

[0239] CID is content-dependent: if the data is tampered with, the identifier becomes invalid, ensuring data integrity. For example, the CID of a business processing result is associated with all its intermediate state hashes. Changing any parameter will cause the CID to mismatch.

[0240] Construct a data set with an identifier:

[0241] Register all data items in the quadruple and their CIDs to the Distributed Hash Table (DHT):

[0242] Use CID as the key and data storage path as the value to write to the local node and cloud backup node.

[0243] At the same time, an index list file is generated to record the CID mapping relationship of the data items in the quadruple (such as reorganized data CID → business result CID).

[0244] Finally, a data set with an identifier is formed, which is physically stored as multiple four-tuple file packages. The CID index table is embedded in the header of each file package, supporting data positioning by CID in seconds.

[0245] Group the data sets with identifiers by engineering batch number, use directed acyclic graph to model the dependency between data items, and generate a data relationship graph;

[0246] Engineering batch grouping strategy:

[0247] The system defines the project batch number (Batch ID) according to the construction management rules. The format is "project number-construction stage-date" (such as PJ1024-PH3-20231001). Traverse all data sets with CID:

[0248] Extract the area code and timestamp from its spatiotemporal identification and match it with the preset construction schedule (e.g. area code A01 belongs to batch PJ1024-PH3-20231001 on 2023-10-01).

[0249] All quadruple data sets in the same batch are merged into the same virtual container, which is named after the batch ID (e.g., / batch_store / PJ1024-PH3-20231001 / ).

[0250] Directed Acyclic Graph Dependency Modeling:

[0251] Use a directed acyclic graph (DAG) to express the dependencies between data items within a batch:

[0252] Node definition: Each quadruple data item is a graph node, and its attributes include its CID, data type, and time and space range.

[0253] Edge definition: Dependencies are divided into three categories:

[0254] Time series dependency: concatenates data from consecutive time periods in timestamp order (e.g., preprocessing results from 09:00-09:15 → input from 09:15-09:30).

[0255] Logical dependency: Business processing results point to the reconstructed original data of their source (associated through CID).

[0256] Breakpoint association: The resume record points to its corresponding shard log (recording the shard reception status when the breakpoint occurred).

[0257] The graph database Neo4j is used to store the DAG model, and the edge attributes record the dependency type and confidence weight (for example, the temporal dependency weight is 1.0 and the logical dependency weight is 0.8).

[0258] Data relationship graph generation:

[0259] Resolve dependency conflicts using a topological sorting algorithm:

[0260] Calculate the in-degree of each node (the number of predecessor nodes that depend on it).

[0261] Iteratively remove nodes with zero in-degree and generate sequences to ensure that there are no cycles in the processing order (for example, data reorganization in region A must be completed before cross-region analysis can be performed).

[0262] The final output data relationship graph file contains a node list, an edge list, and a topology sequence in JSON-LD (Linked Data) format to support semantic queries.

[0263] Add a timestamp proof to each data item, build a complete data processing trace based on the data relationship graph, and generate a processing trace chain with a timestamp proof;

[0264] Trusted timestamp embedding:

[0265] Attach a timestamp proof to each individual data item in the data set:

[0266] Call the Trusted Timestamp Authority (TSA) and send the CID of the data item and the current UTC time to the TSA server.

[0267] The TSA uses its private key to sign the "CID+time" and returns a proof package containing the UTC time, an accuracy value of ±5 milliseconds, and a digital signature (the format complies with the RFC3161 standard).

[0268] The certificate package is stored in conjunction with the data item, for example, a business processing result file is embedded with the TSA signature of the moment it was generated.

[0269] Processing trajectory chain construction:

[0270] Topological sequence concatenation timestamp based on data relationship graph:

[0271] Starting from the starting node of the DAG (the node with in-degree zero), extract the UTC time in its timestamp proof as the starting point of the trajectory.

[0272] Traverse the nodes sequentially along the dependency edges, using the timestamp difference between adjacent nodes as the processing interval (e.g., node A timestamp 12:00:00 → node B 12:00:15, interval 15 seconds).

[0273] To dynamically insert track event markers:

[0274] Breakpoint event: When a node is associated with a resume record, a "breakpoint recovery" event and the resume time are inserted into the track chain; verification event: When a node relies on a shard log, a "shard verification passed" event and the verification time are inserted.

[0275] Finally, a processing trajectory chain is generated, whose data structure is an ordered timeline sequence. Each node contains a timestamp certificate, event type, and upstream and downstream CID references.

[0276] Track Integrity Enhancement:

[0277] Use Hash Chain to ensure that the track cannot be tampered with:

[0278] Calculate the joint hash value H0 of the starting node's CID and timestamp.

[0279] For subsequent nodes, the hash value H_i-1 of the previous node is concatenated with the CID and timestamp of the current node to calculate the new hash H_i.

[0280] The end node hash value is used as the trace root hash and is stored in a read-only storage area together with all timestamp proofs.

[0281] Any modification to the trace node will cause the root hash to change, achieving end-to-end verifiability.

[0282] The data relationship graph and processing trajectory chain are encapsulated into a Merkle tree structure, and a root hash is generated as the data archive fingerprint. At the same time, an index file containing all metadata is generated, ultimately forming a business data archive containing a complete data chain and processing trajectory.

[0283] Merkle tree construction and fingerprint generation:

[0284] Use the output of the previous step as a leaf node to build a Merkle tree:

[0285] Leaf nodes: Contains four types of data: data relationship graph file (JSON-LD representation of DAG); processing trajectory chain file (timestamp sequence and event record); data set index list with CID; summary file of timestamp proof set.

[0286] Hash calculation: Calculate the SHA-256 hash of each leaf node data (e.g., H_graph = Hash(DAG file)). The hashes of adjacent leaf nodes are concatenated and hashed again (e.g., H1-2 = Hash(H1+H2)). This is done recursively to generate the Merkle Root Hash (MRH) (length 256 bits).

[0287] MRH, as a data file fingerprint, is unique and collision-resistant. For example, the MRH of file PJ1024-PH3-20231001 is 7d4fe0...c3a8b9.

[0288] Metadata index file generation:

[0289] Create an archive index file (Archive Index), which includes: basic description: project batch number, coverage time and space range, and total data volume (such as area A01-A08, time 2023-10-01 08:00-18:00, total size 147GB).

[0290] Structural mapping: the entry CID of the data relationship graph (pointing to the Neo4j export file); processing the start / end timestamps and the root hash of the trajectory chain; batch CID list of the quadruple data set (classified by type).

[0291] Verification information: Merkle tree structure parameters (tree height, hash algorithm type); root hash MRH and generation timestamp; public key certificate summary of the trusted timestamp service (used to verify the timestamp signature).

[0292] Business data file packaging:

[0293] Finally packaged into a self-describing archive package:

[0294] Physical storage: uses the ZIP container format and contains the following directories: / data / : stores all original data files with CIDs (reorganized data, business results, etc.); / metadata / : stores data relationship graphs, processing trajectory chains, and index files; / proofs / : centrally stores all timestamp proof files.

[0295] Signature mechanism: Use the device private key at the engineering edge to digitally sign the archive package (ECDSA algorithm); attach the signature file (signature.sig) and device certificate (device.crt) to the package.

[0296] Integrity identification: Write the Merkle root hash MRH in the archive header as the unique fingerprint of the archive.

[0297] The archive also meets the following requirements: data chain integrity: the data source and processing path can be traced through CID and DAG; processing trajectory can be verified: the processing process can be reproduced based on the timestamp proof chain and hash chain; anti-tampering capability: Merkle root hash and digital signature provide dual protection.

[0298] It can be seen that the original business data collected on-site is received by the engineering edge, and is fragmented according to the construction area division and time interval to generate fragmented data packets with time and space identification and check codes; the received fragmented data packets are verified to generate a fragmented cache set with processing progress; when all fragments of a certain area-time segment in the fragmented cache set are received, the pre-processing progress of the corresponding business is read from the persistent file, and the remaining business processing is continued from the point where the progress is interrupted; the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each area-time segment are integrated to generate a business data archive containing a complete data chain and processing trajectory, thereby improving data transmission efficiency and business continuity in weak network environments and meeting the reliability requirements of engineering edge scenarios.

[0299] Another embodiment of the present invention provides a system for data fragmentation and reassembly at the edge of an engineering project and for service breakpoint resumption in a weak network environment. Figure 3 , the system may include:

[0300] The generation module 301 is used to receive the original business data collected on-site through the project edge, segment it according to the construction area and time interval, add a time and space identifier containing the area code, timestamp, segment sequence number and total number of segments to each segment, and generate a check code for the segment data, thereby generating a segment data packet with the time and space identifier and check code;

[0301] Processing module 302 is used to verify the received fragmented data packets, temporarily store them in the temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier, and arrange them in sequence. It also synchronously records the reception time and verification result of each fragment, performs business preprocessing on the received complete region-time fragment, writes the preprocessing progress, intermediate results, and fragment reception status to a persistent file in local storage, and generates a fragment cache set with processing progress.

[0302] Execution module 303 is used to, after receiving all the fragments of a certain region-time segment in the fragment cache, reconstruct the original data of the segment according to the region code and timestamp sequence of the spatiotemporal identifier, read the pre-processing progress of the corresponding business from the persistent file, continue to execute the remaining business processing from the point where the progress was interrupted, and output the business processing result with the resume completion mark;

[0303] The integration module 304 is used to integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each area-time segment, package them according to the project batch number, and generate a business data file containing a complete data chain and processing trajectory.

[0304] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0305] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0306] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0307] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the scope of implementation of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which do not exceed the spirit covered by the description and drawings, should be within the scope of protection of the present invention.

Claims

1. A method for data fragmentation and reassembly at the edge of an engineering project and for service breakpoint resumption in a weak network environment, characterized in that: The method comprises: The original business data collected on-site is received by the project edge, and is fragmented according to the construction area and time interval. A time-space identifier containing the area code, timestamp, fragment sequence number, and total number of fragments is added to each piece of data. At the same time, a check code is generated for the fragmented data, and a fragmented data packet with the time-space identifier and check code is generated. Verify the received fragmented data packets and temporarily store them in the temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier and arrange them in order. Synchronously record the reception time and verification result of each fragment. Perform business preprocessing on the received complete region-time fragments. Write the preprocessing progress, intermediate results and fragment reception status to the persistent file of local storage to generate a fragment cache set with processing progress. When all the shards of a certain region-time segment in the shard cache are received, the original data of the segment is reassembled according to the region code and timestamp sequence of the time-space identifier, the pre-processing progress of the corresponding business is read from the persistent file, the remaining business processing is continued from the point where the progress is interrupted, and the business processing results with the retransmission completion mark are output; wherein, when all the shards of a certain region-time segment in the shard cache are verified, the reassembly process is triggered, and the ordered shard set is merged according to the region code and timestamp sequence of the time-space identifier to generate the reassembled original data with the region time mark; the pre-processing progress of the corresponding business is read from the persistent file, the processing status mark in the status file is parsed, the breakpoint position of the most recent successful processing is located, and the breakpoint context snapshot is generated; Load the breakpoint context snapshot, import the reorganized original data with regional time stamps into the business processing environment, and continue executing the remaining business processing flow from the breakpoint. Use shadow execution technology to synchronously record intermediate calculation steps and state changes to generate a resume execution trajectory. After the business processing is completed, compare and verify the resume execution trajectory with the original processing record, ensure the integrity of the processing process through a hash chain, and generate a verified business processing result. Embed a resume completion mark containing the breakpoint location, resume time, and verification hash in the business processing result. Use a digital signature to ensure that the mark cannot be tampered with, and output the business processing result with the resume completion mark. Integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each region and time segment, package them according to the project batch number, and generate a business data archive containing a complete data chain and processing trajectory.

2. The method according to claim 1, characterized in that The raw business data collected on-site is received by the project edge, and is fragmented according to the construction area and time interval. A time-space identifier including the area code, timestamp, fragment sequence number and total number of fragments is added to each piece of data. A check code for the fragmented data is generated at the same time, and a fragmented data packet with the time-space identifier and check code is generated, including: Extract features from the original business data collected at the engineering site, identify the spatial and temporal features in the data, construct a spatiotemporal feature vector, and simultaneously calculate the initial hash value of the data to generate a spatiotemporal feature vector with the original hash; Based on the spatial features in the spatiotemporal feature vector, the region is divided and the K-means clustering algorithm is used to automatically determine the optimal number of regions. Each region is assigned a unique code to generate a spatiotemporal feature vector with region codes. Dynamically determine the sharding time interval based on the data generation cycle and processing capacity, perform sliding sharding on the spatiotemporal feature vector with region coding according to the time window, add a timestamp and shard sequence number to each piece of data, and calculate the total number of pieces at the same time to generate sharded data with spatiotemporal identifiers; The fragmented data with time and space identifiers are checked in blocks, and a check code is generated by combining cyclic redundancy check and Bloom filter. The check code is embedded in the time and space identifier to generate a fragmented data packet with time and space identifier and check code.

3. The method according to claim 2, characterized in that The received fragmented data packets are verified, and temporarily stored in the temporary storage area of ​​the corresponding area according to the fragment sequence number in the time-space identifier and arranged in sequence, the reception time and verification result of each fragment are synchronously recorded, and business preprocessing is performed on the received complete area-time fragments. The preprocessing progress, intermediate results and fragment reception status are written to the persistent file of the local storage, and a fragment cache set with processing progress is generated, including: Verify the checksum of the received fragmented data packets, quickly match them with the Bloom filter through parallel CRC calculation, mark the packets that pass the checksum, and generate fragmented data packets with checksum marks; According to the region code in the time-space identifier, the fragmented data packets with checksum marks are routed to the temporary storage area of ​​the corresponding region, and a skip table data structure is used to quickly insert and sort the fragment sequence numbers to generate an ordered fragment set; Record the reception time and verification result of each shard, construct a five-tuple log containing the region code, timestamp, shard sequence number, reception time, and verification result, and write it to the local disk in real time using memory-mapped file technology to generate a time series log stream. Perform business preprocessing on the received complete region-time segments, define the preprocessing process using a state machine model, and generate an intermediate result set with processing status; Integrate the preprocessing progress, intermediate result set and time series log stream, write them to the local storage state file through the incremental persistence algorithm, and generate a shard cache set with processing progress.

4. The method according to claim 3, characterized in that The integration of reorganized original data, business processing results, breakpoint resume records and fragment transmission logs for each region and time segment is packaged according to the project batch number to generate a business data archive containing a complete data chain and processing trajectory, including: Collect the reconstructed original data, business processing results, breakpoint resume records, and fragment transmission logs for each region-time segment, build a four-tuple data structure, generate a unique content addressable identifier for each data item, and generate a data set with an identifier; Group the data sets with identifiers by engineering batch number, use directed acyclic graph to model the dependency relationships between data items, and generate a data relationship graph; Add a timestamp proof to each data item, build a complete data processing trace based on the data relationship graph, and generate a processing trace chain with a timestamp proof; The data relationship graph and processing trajectory chain are encapsulated into a Merkle tree structure, and a root hash is generated as the data archive fingerprint. At the same time, an index file containing all metadata is generated, ultimately forming a business data archive containing a complete data chain and processing trajectory.

5. A system for data fragmentation and reassembly at the edge of an engineering project and for service breakpoint resumption in a weak network environment, characterized in that: The system comprises: The generation module is used to receive the original business data collected on-site through the project edge, segment it according to the construction area and time interval, add a time and space identifier containing the area code, timestamp, segment sequence number and total number of segments to each piece of data, and generate a check code for the segmented data, generating a segmented data packet with the time and space identifier and check code; The processing module is used to verify the received fragmented data packets, temporarily store them in the temporary storage area of ​​the corresponding region according to the fragment sequence number in the time-space identifier and arrange them in sequence, synchronously record the reception time and verification result of each fragment, perform business preprocessing on the received complete region-time fragments, write the preprocessing progress, intermediate results and fragment reception status to the persistent file of local storage, and generate a fragment cache set with processing progress; The execution module is used to reorganize the original data of a certain region-time segment in the shard cache according to the region code and timestamp sequence of the spatiotemporal identifier after receiving all the shards of the region-time segment. It reads the pre-processing progress of the corresponding business from the persistent file, continues to execute the remaining business processing from the point where the progress was interrupted, and outputs the business processing result with the resuming completion mark. The integration module is used to integrate the reorganized original data, business processing results, breakpoint resume records and fragment transmission logs of each region and time segment, package them according to the project batch number, and generate a business data archive containing a complete data chain and processing trajectory; The system further includes: when all shards of a certain region-time segment in the shard cache set pass verification, triggering a reorganization process, merging the ordered shard set using a merge sort algorithm based on the region code and timestamp order of the spatiotemporal identifier, and generating reorganized original data with region time stamps; reading the preprocessing progress of the corresponding business from a persistent file, parsing the processing status mark in the status file, locating the breakpoint position of the most recent successful processing, and generating a breakpoint context snapshot; Load the breakpoint context snapshot, import the reorganized original data with regional time stamps into the business processing environment, continue to execute the remaining business processing flow from the breakpoint position, use shadow execution technology to synchronously record intermediate calculation steps and state changes, and generate a resume execution trajectory; after the business processing is completed, compare and verify the resume execution trajectory with the original processing record, ensure the integrity of the processing process through the hash chain, and generate a verified business processing result; embed the resume completion mark containing the breakpoint position, resume time, and verification hash in the business processing result, ensure that the mark cannot be tampered with through a digital signature, and output the business processing result with the resume completion mark.

6. The system according to claim 5, characterized in that The generation module is specifically used to: Extract features from the original business data collected at the engineering site, identify the spatial and temporal features in the data, construct a spatiotemporal feature vector, and simultaneously calculate the initial hash value of the data to generate a spatiotemporal feature vector with the original hash; Based on the spatial features in the spatiotemporal feature vector, the region is divided and the K-means clustering algorithm is used to automatically determine the optimal number of regions. Each region is assigned a unique code to generate a spatiotemporal feature vector with region codes. Dynamically determine the sharding time interval based on the data generation cycle and processing capacity, perform sliding sharding on the spatiotemporal feature vector with region coding according to the time window, add a timestamp and shard sequence number to each piece of data, and calculate the total number of pieces at the same time to generate sharded data with spatiotemporal identifiers; The fragmented data with time and space identifiers are checked in blocks, and a check code is generated by combining cyclic redundancy check and Bloom filter. The check code is embedded in the time and space identifier to generate a fragmented data packet with time and space identifier and check code.

7. The system according to claim 6, characterized in that The processing module is specifically used to: Verify the checksum of the received fragmented data packets, quickly match them with the Bloom filter through parallel CRC calculation, mark the packets that pass the checksum, and generate fragmented data packets with checksum marks; According to the region code in the time-space identifier, the fragmented data packets with checksum marks are routed to the temporary storage area of ​​the corresponding region, and a skip table data structure is used to quickly insert and sort the fragment sequence numbers to generate an ordered fragment set; Record the reception time and verification result of each shard, construct a five-tuple log containing the region code, timestamp, shard sequence number, reception time, and verification result, and write it to the local disk in real time using memory-mapped file technology to generate a time series log stream. Perform business preprocessing on the received complete region-time segments, define the preprocessing process using a state machine model, and generate an intermediate result set with processing status; Integrate the preprocessing progress, intermediate result set and time series log stream, write them to the locally stored state file through the incremental persistence algorithm, and generate a shard cache set with processing progress.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 4 when run.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Server firmware remote upgrading method and system

    CN120151195A

  • New energy automobile calibration data processing cloud platform and method

    CN120378451A