A method, system, device and medium for storing smart traffic data based on the Internet of Things

By introducing DAG protocol, multi-level sharding and erasure coding mechanisms into the smart transportation system, the problems of high storage delay, large redundancy and difficult cross-node consistency in the smart transportation system are solved, efficient data storage and recovery are achieved, and the scalability and reliability of the system are improved.

CN120336431BActive Publication Date: 2025-08-26GANSU CHANGLONG HIGHWAY MAINTENANCE TECH RES INST CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510787955.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-06-13
Publication Date
2025-08-26
Estimated Expiration
2045-06-13

AI Technical Summary

Technical Problem

Existing centralized or simple distributed storage architectures face problems such as high concurrent write delay, serious data redundancy, difficulty in ensuring cross-node consistency, inaccurate timing alignment, and inefficient equipment failure recovery efficiency in smart transportation systems, and it is difficult to meet the requirements of real-time, reliability and scalability.

Method used

Using the distributed intelligent transportation data storage method based on the Internet of Things, data consistency verification, deduplication and failure recovery are achieved by introducing a directed acyclic graph (DAG), multi-level semantic sharding and variable-length content addressing deduplication algorithm, hardware-level nanosecond-level timestamp and table jump index, as well as local/global erasure coding and multi-replica self-healing mechanism.

Benefits of technology

Significantly reduce write acknowledge latency, reduce storage redundancy and network bandwidth consumption, ensure accurate timing alignment across nodes, improve failure recovery speed and data availability, and have excellent scalability, real-time and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120336431B_ABST
    Figure CN120336431B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system, device and medium for storing smart transportation data based on the Internet of Things, which relates to the field of electronic digital data processing technology, including: the edge node encapsulates the received data and inserts it into the local directed acyclic graph account book; divides the verified data into Level-0 fragments according to a fixed time window, and performs variable-length fragmentation based on the Rabin fingerprint algorithm; calculates hash values ​​for the variable-length fragments, and de-duplicates them in turn through Bloom filters and hash index tables; generates local erasure blocks and global erasure blocks for the de-duplicated fragments, and stores complete copies of key fragments on multiple storage nodes; periodically verifies the local erasure blocks and complete copies, and triggers self-healing reconstruction when fragment loss is detected. The present invention solves technical problems such as high latency in distributed write consistency, high storage redundancy, complex cross-node timing correction, slow data recovery and poor system scalability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing technology, and in particular to a method, system, device and medium for storing smart traffic data based on the Internet of Things. Background Art

[0002] With the large-scale deployment of IoT devices in traffic monitoring, vehicle-mounted terminals, and roadside sensing nodes, the images, videos, trajectories, and environmental monitoring data generated by intelligent transportation systems are growing exponentially. Existing centralized or simple distributed storage architectures face problems such as high concurrent write latency, severe data redundancy, difficulty in ensuring cross-node consistency, inaccurate timing alignment, and low efficiency in device fault recovery, making it difficult to meet real-time, reliability, and scalability requirements. Summary of the Invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the present invention provides a distributed intelligent transportation data storage method based on the Internet of Things to address the above problems. By introducing a directed acyclic graph (DAG) consistency verification protocol at the edge node, combining multi-level semantic sharding with a variable-length content-addressed deduplication algorithm, hardware-level nanosecond timestamps and skip table indexes, as well as local / global erasure codes and multi-copy self-healing mechanisms, it solves technical problems such as high latency in distributed write consistency, high storage redundancy, complex cross-node timing correction, slow data recovery, and poor system scalability.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a method for storing intelligent traffic data based on the Internet of Things, comprising:

[0006] The IoT terminal collects traffic data and sends it to the edge node. The edge node encapsulates the received data into a transaction node containing a transaction identifier, a list of parent transaction identifiers, a payload summary, and a hardware-level timestamp, and inserts it into the local directed acyclic graph ledger for consistency verification. A multi-level sharding strategy is implemented, and the verified data is divided into Level-0 shards according to a fixed-length window. Key frame shards, motion area shards, and static background shards are extracted from each Level-0 shard, and then variable-length sharding is performed based on the Rabin fingerprint algorithm. Hash values ​​are calculated for the variable-length shards, and deduplication is performed through Bloom filters and hash index tables in turn. Local erasure blocks and global erasure blocks are generated for the deduplicated shards, and complete copies of key shards are stored on multiple storage nodes. Local erasure blocks and complete copies are periodically verified, and self-healing reconstruction is triggered when shard loss is detected.

[0007] As a preferred embodiment of the IoT-based smart traffic data storage method described in the present invention, the edge node collects a hardware-level timestamp generated by a PTP module and calibrated by a GNSS module at the data packet outbound port, performs a SHA-256 operation on the original traffic data block to obtain a payload digest, retrieves the transaction identifiers of two confirmed transactions from the local DAG ledger, inputs the hardware-level timestamp, payload digest, and parent transaction identifier into the SHA-256 operation to generate a unique transaction identifier, and serializes the hardware-level timestamp field, payload digest field, parent transaction identifier list field, and transaction identifier field into a transaction node structure according to a predefined format, and inserts the transaction node structure into the local DAG ledger.

[0008] The hardware-level timestamp field is obtained by calling the PTP module or GNSS module at the network interface to sample and write it into the metadata area; the payload summary field is obtained by performing a SHA-256 hash operation on the received traffic data; the parent transaction identification list field is obtained by retrieving the transaction identification of the confirmed transaction from the local DAG ledger; the transaction identification field is obtained by performing a SHA-256 hash operation on the aforementioned hardware-level timestamp field, payload summary field, and parent transaction identification list field; then the hardware-level timestamp field, payload summary field, parent transaction identification list field, and transaction identification field are spliced ​​in a predefined order and reversibly encoded to form a serialized transaction node structure.

[0009] As a preferred solution of the smart transportation data storage method based on the Internet of Things described in the present invention, the multi-level sharding strategy includes three levels. The first level divides the Level-0 shards according to a fixed time window; the second level extracts key frame shards, motion area shards and static background shards within the Level-0 shards; the third level uses the Rabin fingerprint algorithm to perform variable length division on the second-level shards.

[0010] As a preferred solution of the smart traffic data storage method based on the Internet of Things described in the present invention, the key frame fragments are generated by a key frame extraction algorithm, the motion area fragments are generated by a frame difference detection algorithm, and the static background fragments are generated by an image boundary extraction algorithm.

[0011] As a preferred solution of the smart traffic data storage method based on the Internet of Things described in the present invention, the size of the third-level shards is dynamically adjusted according to data characteristics and system requirements, and a hash value is calculated for each shard.

[0012] As a preferred solution of the method for storing smart traffic data based on the Internet of Things described in the present invention, wherein: the deduplication by using the Bloom filter and the hash index table in sequence includes calculating the SHA-256 digest for each shard and using A hash function maps the digest to Bloom bit array is pre-filtered, where is the total number of hash functions, is the total number of Bloom bit arrays;

[0013] If the pre-filtering result is negative, the summary is added to the Bloom filter and written to the distributed hash index bucket of the corresponding node according to the consistent hashing algorithm; if the pre-filtering result is positive, the index bucket is located by the consistent hashing algorithm and the summary list in the bucket is queried. When the summary does not hit, the summary is written to the index bucket and inserted into the Bloom filter. When the summary hits, the current shard is marked as duplicate and storage is skipped.

[0014] As a preferred solution of the Internet of Things-based smart transportation data storage method described in the present invention, the generation of local erasure blocks and global erasure blocks includes using a local recoverable erasure code algorithm to generate a local check block for each original shard, and then using a cross-cluster erasure code algorithm to generate a global check block, and writing the original shard and the local check block into the same storage node group, writing the global check block into a remote storage node through a consistent hash map, and writing a complete copy of the original shard marked as critical onto two different physical storage nodes.

[0015] As a preferred solution of the Internet of Things-based smart transportation data storage system described in the present invention, it includes: a data acquisition module, an algorithm module and a verification and reconstruction module; the data acquisition module is used to collect and transmit data information; the algorithm module is used to carry the algorithm for system operation; the verification and reconstruction module is used to perform integrity verification, and when a shard loss is detected, it uses a local erasure block for reconstruction or uses a global erasure block or a complete copy to complete the reconstruction.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the steps of a smart traffic data storage method based on the Internet of Things.

[0017] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a smart traffic data storage method based on the Internet of Things are implemented.

[0018] Beneficial effects of the present invention: The overall solution of the present invention realizes lightweight local consistency verification to significantly reduce write confirmation latency, effectively introduces semantic-based multi-level sharding and content-addressed deduplication to greatly reduce storage redundancy and network bandwidth consumption, utilizes hardware-level nanosecond timestamps combined with skip table indexes to ensure precise cross-node timing alignment and efficient range retrieval, and improves fault recovery speed and data availability through local and global erasure codes combined with a multi-copy self-healing reconstruction mechanism, thereby ensuring real-time, reliability and security while having excellent scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a method for storing smart transportation data based on the Internet of Things is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0021] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0024] The present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0025] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] In this disclosure, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they may refer to fixed, removable, or integral connections. They may also refer to mechanical, electrical, or direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure.

[0027] Example 1, with reference to Figure 1 , which is the first embodiment of the present invention, provides a method for storing intelligent transportation data based on the Internet of Things, including:

[0028] S1: The IoT terminal collects traffic data and sends it to the edge node. The edge node encapsulates the received data into a transaction node containing a transaction ID, a list of parent transaction IDs, a payload summary, and a hardware-level timestamp, and inserts it into the local directed acyclic graph ledger for consistency verification.

[0029] The edge node collects the hardware-level timestamp generated by the PTP module and calibrated by the GNSS module at the data packet outlet, performs a SHA-256 operation on the original traffic data block to obtain a load digest, and retrieves the transaction identifiers of two confirmed transactions from the local DAG ledger. The hardware-level timestamp, load digest, and parent transaction identifier are input into the SHA-256 operation to generate a unique transaction identifier, and the hardware-level timestamp field, load digest field, parent transaction identifier list field, and transaction identifier field are serialized into a transaction node structure in a predefined format, and the transaction node structure is inserted into the local DAG ledger.

[0030] The hardware-level timestamp field is obtained by calling the PTP module or GNSS module at the network interface to sample and write it into the metadata area; the payload summary field is obtained by performing a SHA-256 hash operation on the received traffic data; the parent transaction identification list field is obtained by retrieving the transaction identification of the confirmed transaction from the local DAG ledger; the transaction identification field is obtained by performing a SHA-256 hash operation on the aforementioned hardware-level timestamp field, payload summary field, and parent transaction identification list field; then the hardware-level timestamp field, payload summary field, parent transaction identification list field, and transaction identification field are spliced ​​in a predefined order and reversibly encoded to form a serialized transaction node structure.

[0031] S2: Implement a multi-level fragmentation strategy to divide the verified data into Level-0 fragments according to fixed-length windows. Within each Level-0 fragment, keyframe fragments, motion area fragments, and static background fragments are extracted and then fragmented into variable-length fragments based on the Rabin fingerprint algorithm.

[0032] The multi-level fragmentation strategy includes three levels. The first level divides the Level-0 fragments into fixed-length windows; the second level extracts key frame fragments, motion area fragments and static background fragments within the Level-0 fragments; the third level uses the Rabin fingerprint algorithm to perform variable-length division on the second-level fragments.

[0033] The key frame segments are generated by a key frame extraction algorithm, the motion region segments are generated by a frame difference detection algorithm, and the static background segments are generated by an image boundary extraction algorithm.

[0034] The size of the third-level shards is dynamically adjusted according to data characteristics and system requirements, and a hash value is calculated for each shard.

[0035] Specifically, first divide the original data sequence into first-level shards according to fixed-length windows:

[0036]

[0037] in, is the original data sequence, For the The start time of a fixed-length window, is a fixed time window length, For each first-level shard, three algorithms are called to generate non-overlapping second-level shards:

[0038]

[0039] and satisfy

[0040]

[0041] in, For key frame extraction, frame difference detection, image boundary extraction function, Three algorithms are called to generate non-overlapping secondary shards. Is an empty set.

[0042] Complexity measurement and dynamic parameter determination for

[0043] in, is the measurement weight coefficient; is the secondary sharding complexity metric, is the fragment pixel entropy function, is the variance function of exercise intensity; is the secondary shard, i is the index; is the complexity metric of the i-th secondary shard; are the upper and lower limits of the sliding window length, is the target shard size, is the dynamic sliding window length, is the maximum complexity value, Modulus for dynamic boundary detection;

[0044] In each secondary shard, the length The sliding window calculates the fingerprint:

[0045]

[0046] Generate a shard boundary set based on the fingerprint and modulus remainder matching condition, and add the start and end offsets:

[0047]

[0048] in, The length of the secondary fragment.

[0049] Finally, sort the boundary set in ascending order as And split out all three levels of shards:

[0050]

[0051] in, Represents the original data sequence From the first Start position, offset Data unit after unit; For location The rolling fingerprint value at For the rolling hash base, is a modular polynomial, is the remainder trigger constant, is the shard boundary offset set, is the length of the secondary fragment, is the first offsets, is the first offsets, is the third-level variable-length fragment, n is the offset index in the sliding window, and m is the starting offset position of the current sliding window in the entire second-level fragment. It is the last index value in the shard boundary set. The boundary set contains offset positions, thus generating The third level of fragmentation is variable length.

[0052] S3: Calculate the hash value for the variable-length shards and deduplicate them by using the Bloom filter and hash index table in turn.

[0053] The deduplication process is performed by using Bloom filters and hash index tables, and then calculating the SHA-256 digest for each shard. A hash function maps the digest to Bloom bit array is pre-filtered, where is the total number of hash functions, is the total number of Bloom bit arrays;

[0054] If the pre-filtering result is negative, the summary is added to the Bloom filter and written to the distributed hash index bucket of the corresponding node according to the consistent hashing algorithm; if the pre-filtering result is positive, the index bucket is located by the consistent hashing algorithm and the summary list in the bucket is queried. When the summary does not hit, the summary is written to the index bucket and inserted into the Bloom filter. When the summary hits, the current shard is marked as duplicate and storage is skipped.

[0055] S4: Generate local erasure blocks and global erasure blocks for the deduplicated shards, and store complete copies of key shards on multiple storage nodes.

[0056] The generation of local erasure blocks and global erasure blocks includes generating a local check block for each original shard using a local recoverable erasure code algorithm, generating a global check block using a cross-cluster erasure code algorithm, writing the original shards and local check blocks into the same storage node group, writing the global check block into a remote storage node through a consistent hash map, and writing a complete copy of the original shard marked as critical onto two different physical storage nodes.

[0057] S5: Periodically checks local erasure blocks and complete replicas, and triggers self-recovery and reconstruction when shard loss is detected.

[0058] To ensure that the first check can be executed, the initial value of the lost set is set to an empty set. And the local threshold is initially the base value ,The system triggers the verification task at fixed intervals:

[0059]

[0060] At the moment The scheduling module reads the The original shard contents are compared with the stored reference hash to generate a consistency indication. The set of lost shard indexes is:

[0061]

[0062] Dynamically adjust the local recoverable threshold based on the last loss scale:

[0063]

[0064] Select the reconstruction method based on the current loss amount:

[0065]

[0066] In completion After rebuilding, regenerate the checksum and update the mapping:

[0067]

[0068] in, The initial value of the lost set is is the initial value of the local threshold, is the initial verification trigger moment, is the calibration period interval, For the The original shards at time The data content, For the The reference hash value when the shard was initially written, Hash is the SHA-256 hash function, For the The fragment in The consistency indicator in the secondary checksum is 1 for consistency and 0 for loss. For the The index set of the shard is lost during the second check. For the The index set of the shard is lost during the second check. is the recoverable threshold of the basic local erasure block, is the threshold adaptive coefficient, which is used to adjust the local threshold according to the last loss scale. For the The local erasure recovery threshold of times verification, is the number of available global erasure blocks, are the generator matrices representing local and global erasure codes respectively; is the submatrix formed by extracting the corresponding missing index rows from the generator matrix, are the column vectors of local and global erasure blocks corresponding to the lost indices, The reconstructed The column vector of the lost fragment data of the second check, For the A complete copy of the data for each shard, is the overall local erasure block and global erasure block matrix generated after reconstruction; D is the vector of original shard data; is the size of the lost set, that is, the number of lost shards, The sequence number of the verification task.

[0069] Example 2

[0070] A second embodiment of the present invention provides an Internet of Things-based intelligent transportation data storage system, specifically comprising:

[0071] The system includes a data acquisition module, an algorithm module and a verification and reconstruction module; the data acquisition module is used to collect and transmit data information; the algorithm module is used to carry the algorithm for system operation; the verification and reconstruction module is used to perform integrity verification, and when a shard loss is detected, it uses local erasure blocks for reconstruction or uses global erasure blocks or a complete copy to complete the reconstruction.

[0072] Example 3

[0073] The third embodiment of the present invention is different from the first two embodiments in that:

[0074] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0078] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0079] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for storing intelligent traffic data based on the Internet of Things, characterized by: include, The IoT terminal collects traffic data and sends it to the edge node. The edge node encapsulates the received data into a transaction node containing a transaction ID, a list of parent transaction IDs, a payload summary, and a hardware-level timestamp, and inserts it into the local directed acyclic graph ledger for consistency verification. Implement a multi-level sharding strategy, dividing the verified data into Level-0 shards based on fixed-length windows. Within each Level-0 shard, extract keyframe shards, motion region shards, and static background shards, and then perform variable-length sharding based on the Rabin fingerprint algorithm. Calculate the hash value for the variable-length shards and deduplicate them by using the Bloom filter and hash index table in turn; Generate local and global erasure blocks for the deduplicated shards, and store complete copies of key shards on multiple storage nodes; Periodically check local erasure blocks and complete replicas, and trigger self-healing reconstruction when shard loss is detected.

2. The method for storing intelligent traffic data based on the Internet of Things according to claim 1, wherein: The edge node collects the hardware-level timestamp generated by the PTP module and calibrated by the GNSS module at the data packet outbound port, performs a SHA-256 operation on the original traffic data block to obtain a payload digest, retrieves the transaction identifiers of two confirmed transactions from the local DAG ledger, inputs the hardware-level timestamp, payload digest, and parent transaction identifier into the SHA-256 operation to generate a unique transaction identifier, and serializes the hardware-level timestamp field, payload digest field, parent transaction identifier list field, and transaction identifier field into a transaction node structure in a predefined format, and inserts the transaction node structure into the local DAG ledger; The hardware-level timestamp field is obtained by calling the PTP module or the GNSS module at the network interface to sample and write into the metadata area; The payload summary field is obtained by performing a SHA-256 hash operation on the received traffic data; the parent transaction identifier list field is obtained by retrieving the transaction identifier of the confirmed transaction from the local DAG ledger; the transaction identifier field is obtained by performing a SHA-256 hash operation on the aforementioned hardware-level timestamp field, payload summary field, and parent transaction identifier list field; the hardware-level timestamp field, payload summary field, parent transaction identifier list field, and transaction identifier field are then concatenated in a predefined order and reversibly encoded to form a serialized transaction node structure.

3. The method for storing intelligent traffic data based on the Internet of Things according to claim 2, wherein: The multi-level sharding strategy includes three levels. The first level divides Level-0 shards into fixed-length windows; The second level extracts key frame segments, motion area segments and static background segments from Level-0 segments. The third level uses Rabin fingerprint algorithm to perform variable length segmentation on the second level segments.

4. The method for storing intelligent traffic data based on the Internet of Things according to claim 3, wherein: The key frame segments are generated by a key frame extraction algorithm, the motion region segments are generated by a frame difference detection algorithm, and the static background segments are generated by an image boundary extraction algorithm.

5. The method for storing intelligent transportation data based on the Internet of Things according to claim 4, characterized in that: The size of the third-level shards is dynamically adjusted according to data characteristics and system requirements, and a hash value is calculated for each shard.

6. The method for storing intelligent transportation data based on the Internet of Things according to claim 5, characterized in that: The deduplication process is performed by using Bloom filters and hash index tables, and then calculating the SHA-256 digest for each shard. A hash function maps the digest to Bloom bit array is pre-filtered, where is the total number of hash functions, is the total number of Bloom bit arrays; If the pre-filtering result is negative, the summary is added to the Bloom filter and written to the distributed hash index bucket of the corresponding node according to the consistent hashing algorithm; if the pre-filtering result is positive, the index bucket is located by the consistent hashing algorithm and the summary list in the bucket is queried. When the summary does not hit, the summary is written to the index bucket and inserted into the Bloom filter. When the summary hits, the current shard is marked as duplicate and storage is skipped.

7. The method for storing intelligent transportation data based on the Internet of Things according to claim 6, characterized in that: The generation of local erasure blocks and global erasure blocks includes generating a local check block for each original shard using a local recoverable erasure code algorithm, generating a global check block using a cross-cluster erasure code algorithm, writing the original shards and local check blocks into the same storage node group, writing the global check block into a remote storage node through a consistent hash map, and writing a complete copy of the original shard marked as critical onto two different physical storage nodes.

8. A system using the method for storing intelligent transportation data based on the Internet of Things according to any one of claims 1 to 7, characterized in that: Including data acquisition module, algorithm module and verification and reconstruction module; The data acquisition module is used to collect and transmit data information; The algorithm module is used to carry the algorithm running the system; The check and reconstruction module is used to perform integrity check and, when a shard loss is detected, to use a local erasure block for reconstruction or to use a global erasure block or a complete copy to complete the reconstruction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Erasure code-based storage optimization method for decentralized storage system

    CN119759640A

  • High-concurrency transaction data processing system and method combined with edge computing

    CN119988021A