Fault-tolerant processing method for data storage module of Internet of Things acquisition instrument

Abstract

The invention relates to the technical field of Internet of Things data storage, and discloses an Internet of Things acquisition instrument data storage module fault-tolerant processing method, which comprises the following steps: initializing a storage module, dividing a main storage area and a backup storage area, setting the capacity ratio of the main storage area to the backup storage area to be 7: 3, establishing a dynamic index table based on a hash table, each entry in the index table comprises a data block physical address, a timestamp, a check value and a validity identifier; and receiving real-time data of the Internet of Things acquisition instrument, packaging the real-time data into a 512-byte fixed data block according to a structure of equipment ID + acquisition timestamp + data value + CRC32 check bit, and writing the 512-byte fixed data block into a main storage area through a parallel write-in channel. According to the fault-tolerant processing method for the data storage module of the Internet of Things acquisition instrument, the data integrity guarantee capability is improved through multi-layer verification, and the effectiveness of fault-tolerant processing is further enhanced.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) data storage technology, specifically to a fault-tolerant processing method for IoT data acquisition module data storage. Background Technology

[0002] In existing technologies, various solutions have been developed for data fault tolerance processing in different scenarios. In the scenario of image data reception with insufficient bus bandwidth, asynchronous FIFO is used to realize clock domain conversion, combined with buffered FIFO to store data. When both are full, some data is discarded to avoid excessively low frame rate or frequent system resets. In the scenario of non-volatile internal memory log storage, the virtual address of the log storage area is determined by the initialization module and the memory is mounted. After power failure, the original log and legacy log are synchronized to disk backup. When remounted, they are synchronized back to memory to ensure the safety of log data. In the scenario of secondary radar point data processing, a "FIFO1+RAM+FIFO2" structure is adopted to filter external request signals. The data is buffered by FIFO1, temporarily stored in RAM, and output by FIFO2. Combined with protocol verification, the data is ensured to meet the format requirements for scheduling by lower-level modules. These solutions have achieved basic fault tolerance processing in their respective application scenarios through cache control, backup synchronization, and multi-level storage verification.

[0003] However, existing technologies employ fixed mechanisms to address faults: image data fault tolerance relies solely on discarding data, which suffers from insufficient processing bandwidth and fails to dynamically adjust redundancy levels based on data importance; log fault tolerance depends on a fixed disk backup synchronization cycle, making it impossible to flexibly adjust backup frequency according to storage load; and radar spot data fault tolerance uses a fixed FIFO and RAM storage structure, which cannot optimize storage resource allocation as data volume changes. As a result, existing solutions struggle to meet the fault tolerance requirements of complex scenarios in terms of dynamic adaptability, fault foresight, and recovery efficiency. Summary of the Invention

[0004] The purpose of this invention is to solve the problems mentioned above, and to propose a fault-tolerant processing method for the data storage module of an Internet of Things (IoT) data acquisition device.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A fault-tolerant processing method for data storage module of an IoT data acquisition device includes the following steps: S10: Initialize the storage module, divide the main storage area and the backup storage area, with a capacity ratio of 7:3 between the main storage area and the backup storage area, and establish a dynamic index table based on a hash table. Each entry in the index table contains the physical address of the data block, timestamp, check value and validity identifier. S20: Receives real-time data from the IoT data acquisition device, encapsulates it into a 512-byte fixed data block according to the structure of "Device ID + Acquisition Timestamp + Data Value + CRC32 Check Bit", writes it to the main storage area through the parallel write channel, and simultaneously updates the index information of the data block to the dynamic index table. S30: Real-time monitoring of the write status of the main storage area, detecting single / multi-bit errors through hardware ECC circuit: If it is a single-bit error, it is directly corrected by the ECC circuit and the error count is marked; if it is a multi-bit error, the fault tolerance module is triggered, the data block is marked as "invalid" and the error location is recorded, and the pre-stored Reed-Solomon (RS(16,8)) check block in the backup storage area is called, and the error is corrected using the formula... ,in To verify the block data, To generate a matrix, reconstruct the data blocks; S40: Execute a periodic synchronization mechanism, triggering incremental synchronization once per hour: By comparing the differences between the index tables of the primary storage area and the backup storage area, only newly added or modified data blocks are synchronized to the backup storage area. During the synchronization process, a "write first, delete later" strategy is adopted to ensure the data integrity of the backup storage area. S50: When the proportion of erroneous data blocks in the main storage area exceeds 5%, the storage area switching process is initiated: the data read / write path is switched to the backup storage area via a hardware switch, and the complete data sequence is reassembled based on the chain structure of the dynamic index table, with a switching response time of no more than 10ms. Based on the above technical solution, the present invention can also be improved as follows.

[0006] Furthermore, the hash table of the dynamic index table in S10 adopts a double-hash collision resolution strategy, with an initial capacity of 2048 entries and a load factor threshold set to 0.7. When the load factor exceeds the threshold, it automatically expands to twice the current capacity. Each entry in the index table also includes a version number field to record the number of times the data block is updated. When a data block is modified, the version number is automatically incremented by 1 to ensure accurate identification of the old and new states of the data block during synchronization. The hash table is stored independently of the main / backup storage area, using independent FRAM to avoid data loss in case of power failure. FRAM with a capacity of not less than 128KB and a read / write lifespan of not less than [missing information]. Second-rate.

[0007] Furthermore, the calculation of the CRC32 checksum in S20 employs iterative hardware acceleration, as detailed in the following formula: Set the initial check value The data to be verified is divided into 32-bit word lengths. ,in Each iteration calculates: ,in, This represents the number of bits to shift, with a value of 8. A 256-order CRC lookup table array (composed of polynomials) First, generate the checksum; after iteration, the final checksum is generated. The 32-bit CRC checksum is written to the main memory area. This calculation process is performed by a dedicated CRC hardware accelerator, and the time consumption is controlled within [timeframe missing]. Within a data block, the system automatically verifies the consistency between the accelerator output and the software calculation results after every 100 data blocks are written, avoiding verification errors caused by hardware failures.

[0008] Furthermore, the generation of the Reed-Solomon (RS(16,8)) check block in S30 adopts hardware encoding based on primitive polynomials, and the specific formula is as follows: Let there be 8 data blocks. Each block is 64 bytes, generating 8 check blocks. , of which The first check block byte Calculation formula: In the formula For the Gara Flower Domain The primitive element is composed of the primitive polynomial. generate, The exponent is mapped to integers from 0 to 255 using an exponent table. The check block and data block are then... Sequential storage ensures that if any four data blocks fail, the complete data can be reconstructed by solving a system of eight linear equations, with a reconstruction time of ≤3ms.

[0009] Furthermore, the generation process of the Reed-Solomon (RS(16,8)) check block in S30 is as follows: Eight data blocks are grouped together, and eight check blocks are generated using the RS encoder. The check blocks and data blocks are stored in consecutive sectors of the backup storage area in the order of "data block 1-8 + check block 1-8", generating a matrix. It is a 16×8 matrix, where the first 8 columns are the identity matrix and the last 8 columns are passed through the primitive polynomial. Generate data to ensure that any 8 blocks can be used to reconstruct the complete set of data.

[0010] Furthermore, the incremental synchronization process of S40 includes: a1. When synchronous triggering occurs, first calculate the MD5 hash value of the dynamic index table in the main storage area and compare it with the hash value of the index table in the backup storage area. b1. If the hash values ​​are different, the version number field of the index table is compared item by item to filter out the items in the main storage area with version numbers higher than those in the backup storage area, forming a difference list. c1. Read the data block in the main storage area according to the physical address in the difference list, and transfer it to the backup storage area through the DMA channel. During the transfer, a check is performed every 4KB of data transferred. d1. After synchronization is complete, update the index table and hash value of the backup storage area, and record the synchronization log to the non-volatile register.

[0011] Furthermore, the storage area switching process in S50 specifically includes: e1. Hardware level: The data bus connection is switched via an analog switch controlled by a CPLD, and the switching time is less than 1ms; f1. Software level: Immediately stop all read and write operations in the main storage area, copy the entries in the "valid" state of the dynamic index table to the backup index table, and at the same time map the physical address of each entry to the corresponding location in the backup storage area. g1. Data Reassembly: Based on the preceding / following block address field of the index table, starting from the first valid data block in the backup storage area, the data chain is reassembled in timestamp order, ignoring entries marked as "invalid" during the reassembly process; h1. After the switch is completed, a switch completion signal is output through the GPIO pin, and a status message containing the switch reason and switch time is sent to the main control module of the IoT data acquisition device.

[0012] Furthermore, in the linked structure of the dynamic index table, each data block's index entry contains "previous block address" and "next block address" fields, forming a doubly linked list: for data blocks collected consecutively in chronological order, the previous block address points to the adjacent preceding data block, and the next block address points to the adjacent next data block; for non-consecutive data blocks, the next block address is set to... This structure ensures that even if some index entries are lost when the main storage area is damaged, most of the continuous data can still be recovered by traversing the linked list in reverse or forward, thus improving the robustness of data recovery.

[0013] Furthermore, it includes an error warning mechanism: in step S30, an error counter is maintained for each data block. The counter is incremented by 1 after a single-bit error is corrected, and the counter is set to zero when a multi-bit error occurs. When the error counter of a data block reaches 16, it is marked as "pre-failure" even if no multi-bit error has occurred. During the next synchronization, the data block is moved to the reserved sector of the backup storage area in advance, and its physical address is updated in the index table to avoid data loss caused by the complete failure of the data block.

[0014] Furthermore, the parallel write channel in S20 adopts a priority scheduling mechanism: when the IoT data acquisition device transmits multiple types of data at the same time, the priority is distinguished by the high 4 bits of the device ID. Key data with a priority ≥ 10 is allocated an independent write channel, while ordinary data with a priority < 10 shares the remaining channel. The token bucket algorithm is used to control the bandwidth during writing. The token bucket capacity for key data is 1MB, and for ordinary data it is 512KB, ensuring that key data can still be written first when the bus is congested, reducing its error probability.

[0015] Compared with the prior art, the technical solution of this application has the following beneficial technical effects: This invention achieves rational allocation and dynamic management of storage resources by dividing the main storage area and backup storage area into a 7:3 capacity ratio, combined with a hash table-based dynamic index table. The dynamic index table records information such as the physical address and timestamp of data blocks and can be updated in real time as data is written. Furthermore, the handling of data errors does not employ a fixed discard or single backup strategy, but rather dynamically adjusts based on the error type: single-bit errors are directly corrected by ECC circuitry, while multi-bit errors are reconstructed using Reed-Solomon parity blocks in the backup storage area. Redundancy levels are also dynamically adjusted based on data importance, overcoming the deficiency of existing image data fault tolerance systems that do not adjust redundancy according to importance. Secondly, in terms of fault prediction, by real-time monitoring of the main storage area's write status and continuously detecting and recording errors using hardware ECC circuitry, the storage area switching process is proactively initiated when the cumulative percentage of erroneous data blocks exceeds 5%, rather than waiting for complete failure before processing. This enables early prediction and intervention of potential faults. Finally, regarding recovery efficiency, compared to the recovery delay caused by the fixed period of log synchronization and the scheduling complexity brought by multi-level storage verification of radar data, parallel write channels improve data writing efficiency. The chain structure based on dynamic index tables can complete storage area switching and reassemble the complete data sequence in a short time. Moreover, the incremental synchronization mechanism only synchronizes newly added or modified data blocks, and the "write-before-delete" strategy ensures backup integrity, significantly improving the speed and reliability of data recovery. In addition, data blocks are encapsulated with a fixed structure and combined with CRC32 check and RS check blocks. Multi-layer verification enhances data integrity protection capabilities and further strengthens the effectiveness of fault tolerance. Attached Figure Description

[0016] Figure 1 This is a flowchart of the core fault-tolerant processing of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The present invention provides a fault-tolerant processing method for a data storage module of an Internet of Things (IoT) data acquisition device, comprising the following steps: S10: Initialize the storage module, divide the main storage area (using NAND flash memory) and the backup storage area (using NOR flash memory), with a capacity ratio of 7:3 between the main storage area and the backup storage area, and establish a dynamic index table based on a hash table. Each entry in the index table contains the physical address of the data block, timestamp, check value and validity identifier. S20: Receives real-time data from the IoT data acquisition device, encapsulates it into a 512-byte fixed data block according to the structure of "Device ID (16 bits) + acquisition timestamp (32 bits) + data value (64 bits) + CRC32 check bit (32 bits)," writes it to the main storage area through the parallel write channel, and simultaneously updates the index information of the data block to the dynamic index table. S30: Real-time monitoring of the write status of the main storage area, detecting single / multi-bit errors through hardware ECC circuit: If it is a single-bit error, it is directly corrected by the ECC circuit and the error count is marked; if it is a multi-bit error, the fault tolerance module is triggered, the data block is marked as "invalid" and the error location is recorded, and the pre-stored Reed-Solomon (RS(16,8)) check block in the backup storage area is called, and the error is corrected using the formula... ,in To verify the block data, To generate a matrix, reconstruct the data blocks; S40: Execute a periodic synchronization mechanism, triggering incremental synchronization once per hour: By comparing the differences between the index tables of the primary storage area and the backup storage area, only newly added or modified data blocks are synchronized to the backup storage area. During the synchronization process, a "write first, delete later" strategy is adopted to ensure the data integrity of the backup storage area. S50: When the proportion of erroneous data blocks in the main storage area exceeds 5%, the storage area switching process is initiated: the data read and write path is switched to the backup storage area through a hardware switch, and the complete data sequence is reassembled based on the chain structure of the dynamic index table (each data block contains the address of the previous / next block). The switching response time does not exceed 10ms.

[0019] The hash table of the dynamic index table in S10 adopts a double hash collision resolution strategy, with an initial capacity of 2048 entries (2 11The load factor threshold is set to 0.7. When the load factor exceeds the threshold, the capacity is automatically expanded to twice the current capacity (the expanded capacities are 4096, 8192, etc.). Each entry in the index table also contains a version number field (8 bits) to record the number of times the data block has been updated. When a data block is modified, the version number is automatically incremented by 1 to ensure accurate identification of the old and new states of the data block during synchronization. The hash table is stored independently of the main / backup storage area, using independent FRAM (ferroelectric memory) to avoid data loss in case of power failure. FRAM with a capacity of at least 128KB (such as FM25V10) is selected, and its read / write lifespan is at least [missing information]. The initial capacity and load factor settings can balance storage resources and query efficiency, avoiding resource waste caused by excessively large initial capacity or frequent expansion caused by excessively small initial capacity; the automatic expansion mechanism ensures that the index table can maintain high performance when the data volume grows; the version number field solves the problem of identifying the old and new data blocks during synchronization after the data block is updated, avoiding synchronization errors; the independent FRAM storage of the index table completely solves the problem of index information loss when power is off, and the high read and write life meets the needs of long-term frequent updates, ensuring that the index table works stably for a long time.

[0020] In the dual-hash collision resolution strategy, the first hash function uses the DJB2 algorithm to calculate the initial address, and the second hash function uses the SDBM algorithm to calculate the probe step size. When a collision occurs, the second hash function determines the next probe position, effectively avoiding the clustering effect. The FRAM specifically uses the FM25V02 model, which operates in the voltage range of 2.7-3.6V and supports a fast read / write speed of 15ns. It can save data instantly upon power failure (less than 100ns), ensuring that the index table information is not lost. The version number field (8 bits) can record up to 255 updates. When a data block is corrected due to recalibration of the acquisition instrument, the version number is incremented synchronously. During incremental synchronization, the backup storage area can accurately determine whether an update is needed by comparing the version number, without recalculating the data block differences, further improving synchronization efficiency.

[0021] The calculation of the CRC32 checksum in S20 uses iterative hardware acceleration, and the specific formula is as follows: Set the initial check value The data to be verified is divided into 32-bit word lengths. ,in Each iteration calculates: ,in, This represents the number of bits to shift, with a value of 8. A 256-order CRC lookup table array (composed of polynomials) First, generate the checksum; after iteration, the final checksum is generated. (Bitwise inversion) is used as the 32-bit CRC checksum and written to the main memory area. This calculation process is performed by a dedicated CRC hardware accelerator, with the time controlled within 1μs per data block. Furthermore, the consistency between the accelerator output and the software calculation result is automatically checked after every 100 data blocks are written, avoiding checksum errors caused by hardware failures. Iterative hardware-accelerated calculation of the CRC32 checksum significantly improves the calculation speed of the checksum value. The data block processing time is much shorter than that of software computation, meeting the high-speed writing requirements of real-time data acquisition devices; the periodic comparison between hardware accelerator and software computation results effectively avoids verification errors caused by hardware failures, ensuring the accuracy of the check bits; the combination of 32-bit word length segmentation and specific iterative formula improves the anti-interference capability of the verification and can more accurately detect damage to data blocks during transmission or storage.

[0022] The hardware accelerator is implemented using a dedicated ASIC chip, integrating a 32-bit parallel processing unit that supports pipelined operation. When processing 128 32-bit words of data, the next data word can be computed before the previous one is completed, further reducing the overall processing time. The software computation uses a lookup table method implemented in C. In a consistency check every 100 data blocks, if three consecutive discrepancies occur, the system automatically switches to software computation mode and notifies the main control module via an interrupt signal to troubleshoot hardware faults. The initial checksum... The setting is to avoid misjudgment caused by the check value of all-zero data blocks being zero, and to ensure that even if the data block is all zero, its integrity can be accurately identified through the check bit.

[0023] The generation of the Reed-Solomon (RS(16,8)) parity block in S30 uses hardware encoding based on primitive polynomials, and the specific formula is as follows: Let there be 8 data blocks. Each block is 64 bytes, generating 8 check blocks. , of which The first check block byte Calculation formula: In the formula For the Gara Flower Domain The primitive element is composed of the primitive polynomial. generate, Mapped to integers from 0 to 255 through an exponent table (e.g.) ), check blocks and data blocks are arranged according to Sequential storage ensures that even if any four data blocks fail, the complete data can be reconstructed by solving a system of eight linear equations with a reconstruction time of ≤3ms (implemented by a hardware accelerator). Hardware encoding based on primitive polynomials generates RS(16,8) check blocks, improving the efficiency of check block generation and making it more suitable for real-time data processing scenarios compared to software encoding. The design of generating eight check blocks from eight data blocks allows the system to tolerate the failure of any four data blocks, significantly improving the fault tolerance capability of data block reconstruction. (Galohua Domain) The combination of computation and generation matrix ensures the accuracy of data block reconstruction; the reconstruction time of ≤3ms achieved by the hardware accelerator meets the real-time requirements of IoT data acquisition devices and avoids data processing interruption due to reconstruction delay.

[0024] Primitive polynomials The generated Galois domain In this context, each element corresponds to an 8-bit binary number. arrive The mapping relationship is implemented through a pre-stored index table, which is directly looked up during hardware encoding. The value of the generator matrix does not require real-time calculation, saving encoding time; It is a 16-row, 8-column binary matrix. The first 8 rows form the identity matrix, and the last 8 rows are derived from the coefficients of the primitive polynomial, ensuring that the check block can accurately reflect the redundancy information of the data block. In actual testing, when 4 consecutive data blocks fail, the hardware accelerator solves the system of 8 linear equations, and the average reconstruction time is 2.3ms. Even in a high-temperature (85℃) environment, the reconstruction time does not exceed 3ms, demonstrating excellent stability.

[0025] The generation process of Reed-Solomon (RS(16,8)) parity blocks in S30 is as follows: Eight data blocks are grouped together, and eight parity blocks are generated using the RS encoder. The parity blocks and data blocks are stored in consecutive sectors of the backup storage area in the order of "data block 1-8 + parity block 1-8", generating a matrix. It is a 16×8 matrix, where the first 8 columns are the identity matrix and the last 8 columns are passed through the primitive polynomial. The system generates and ensures that any 8 blocks (data blocks or check blocks) can reconstruct a complete set of data. When reconstructing a single data block, only the 8 check blocks of the corresponding group need to be called, without reading other data blocks, reducing the bus load during reconstruction. Incremental synchronization uses hash value comparison and version number filtering to synchronize only the different data blocks, which significantly reduces the amount of data transmission and bus load compared to full synchronization. DMA channel transmission improves data transmission efficiency and avoids latency caused by MCU intervention. CRC32 checksums every 4KB of data ensure data integrity during synchronization. Synchronization log recording facilitates later tracking of synchronization status and timely detection and resolution of synchronization anomalies. Backup index table and hash value updates ensure consistency of index information in the primary and backup storage areas.

[0026] The hash value calculation uses the SHA-256 algorithm. All entries in the index table are sorted by physical address before hashing, ensuring the hash value accurately reflects the overall state of the index table. The DMA channel employs a dual-channel design: the primary channel transmits data blocks, and the backup channel transmits verification information. When an error occurs in the primary channel, it automatically switches to the backup channel, maintaining a stable transmission rate of 10MB / s. For 4KB verification, the 4KB data is divided into 16 256-byte sub-blocks. The CRC32 value of each sub-block is calculated and compared with the primary verification value. If a sub-block fails verification, only that sub-block is retransmitted, reducing the amount of retransmitted data. The synchronization log is stored in a dedicated 1MB partition in the FRAM, recording the most recent 1000 synchronization events. The log uses a circular overwrite mechanism to ensure that the latest log is not overwritten by older logs.

[0027] The incremental synchronization process of S40 includes: a1. When synchronous triggering occurs, first calculate the MD5 hash value of the dynamic index table in the main storage area and compare it with the hash value of the index table in the backup storage area. b1. If the hash values ​​are different, the version number field of the index table is compared item by item to filter out the items in the main storage area with version numbers higher than those in the backup storage area, forming a difference list. c1. Read the data block in the main storage area according to the physical address in the difference list, and transfer it to the backup storage area through the DMA channel. During the transfer, a check is performed every 4KB of data transferred (comparing the CRC32 value). d1. After synchronization is complete, update the index table and hash value of the backup storage area, and record the synchronization log (including synchronization time, number of data blocks, and number of errors) to a non-volatile register. The synchronization log adopts a circular overwrite mechanism, retaining only the most recent 1000 records to ensure reasonable utilization of storage resources. Software-level read and write operations are stopped and the index table is copied to avoid data corruption during the switchover process. Pre-stored address mapping relationships ensure efficient physical address translation. Data reassembly is based on a chain structure and sorted by timestamp to ensure the integrity of the data sequence after the switchover. GPIO signals and status messages enable the system to provide timely feedback on the switchover status, facilitating subsequent processing by upper-layer modules.

[0028] The CPLD uses the XC95144 model, operates at a frequency of 50MHz, and can initiate switch switching within 300ns after receiving a switching command. The analog switch uses the ADG732 chip, with an on-resistance of less than [missing value]. This ensures that the data bus signal transmission is attenuated; the address mapping relationship is stored in FRAM in the form of a table, which includes the physical address of the main memory area, the corresponding address of the backup memory area, and the mapping validity identifier. The table supports dynamic updates, and the mapping relationship can be automatically updated when there are bad blocks in the backup memory area; during the data reconstruction process, if the preceding and following address of a certain data block is found to be incorrect, the system will supplement the missing data by timestamp interpolation and mark the missing position in the status message to facilitate later data repair; the GPIO output switching completion signal is connected to the INT pin of the main control module, and the high level lasts for 100ms to ensure that the main control module can reliably identify it.

[0029] The storage area switching process in S50 specifically includes: e1. Hardware level: The data bus connection is switched via an analog switch controlled by a CPLD, and the switching time is less than 1ms; f1. Software level: Immediately stop all read and write operations in the main storage area, copy the entries in the "valid" state of the dynamic index table to the backup index table, and at the same time map the physical address of each entry to the corresponding location in the backup storage area (the mapping relationship is pre-stored in FRAM). g1. Data Reassembly: Based on the preceding / following block address field of the index table, starting from the first valid data block in the backup storage area, the data chain is reassembled in timestamp order, ignoring entries marked as "invalid" during the reassembly process; h1. After the switch is completed, a switch completion signal (active high) is output through the GPIO pin, and a status message containing the switch reason and switch time is sent to the main control module of the IoT data acquisition device. The doubly linked list structure establishes a relationship between data blocks. Even if some index entries are lost, they can still be traced or traversed through the previous / next order addresses, which greatly improves the possibility of data recovery. The next order address of non-contiguous data blocks is specially marked to clarify the interval status of data acquisition and avoid timing disorder during recovery. This structure, combined with the dynamic index table, enhances the system's data loss resistance when the main memory area is damaged.

[0030] The maintenance of the doubly linked list is handled by a dedicated linked list management module. When a new data block is written, the post-order address of the previous block and the pre-order address of the current block are automatically updated to ensure the continuity of the linked list. The 1-second interval judgment is based on the data block's collection timestamp. If the difference between the timestamp of the current data block and the previous block is greater than 1000ms, it is determined to be a non-contiguous data block, and its post-order address is set to 0xFFFFFFFF. During reverse tracing, starting from the last valid data block, the search proceeds backward through the pre-order address until the first data block is found. If a block with a post-order address of 0xFFFFFFFF is encountered along the way, the breakpoint position is recorded, and the time interval is marked at the breakpoint after recovery. Forward traversal starts from the first valid data block and searches backward through the post-order address, stopping when 0xFFFFFFFF is encountered, ensuring that the recovered data is arranged in the collection order.

[0031] In the linked structure of the dynamic index table, each data block's index entry contains "previous block address (32 bits)" and "next block address (32 bits)" fields, forming a doubly linked list: for data blocks collected continuously in chronological order, the previous block address points to the adjacent previous data block, and the next block address points to the adjacent next data block; for non-contiguous data blocks (such as those with a collection interval exceeding 1 second), the next block address is set to 0xFFFFFFFF. This structure ensures that even if some index entries are lost when the main storage area is damaged, most of the continuous data can still be recovered by tracing back or forward through the linked list, improving the robustness of data recovery. The error warning mechanism monitors the data block status in real time through an error counter. When a single-bit error accumulates to a threshold, it is marked as "pre-failure" and migrated in advance, avoiding data loss caused by sudden data block failure; the reserved sector setting provides a dedicated storage area for pre-failure data blocks, ensuring the orderliness of the migration process; the updating of the index table's physical address ensures that the data block can be accessed correctly after migration, improving the system's ability to predict and handle potential errors.

[0032] The error counter uses an 8-bit unsigned integer, initially set to 0. It increments by 1 when a single-bit error in a data block is successfully corrected, and resets to zero if the data block is rebuilt or modified. Reserved sectors employ an independent erase / write strategy, with a write / write limit of 100,000 times, lower than other areas of the backup storage area, thus extending the lifespan of the reserved sectors. When migrating pre-expired data blocks, the data block is first copied to the reserved sector, its CRC32 value is calculated and compared with the original data block, and the index table address is updated only after confirmation of consistency. Finally, the original data block is marked as "invalid." The entire migration process is completed within 5ms and does not affect normal data writing. When the reserved sector utilization rate exceeds 80%, the system automatically converts the earliest migrated "pre-expired" data block to "valid" and migrates it to the main area of ​​the backup storage area, releasing reserved sector space.

[0033] It also includes an error warning mechanism: In step S30, an error counter (8 bits) is maintained for each data block. The counter is incremented by 1 after a single-bit error is corrected, and the counter is set to 0xFF (maximum value) when a multi-bit error occurs. When the error counter of a data block reaches 16, it is marked as "pre-failure" even if no multi-bit error has occurred. During the next synchronization, the data block is migrated to the reserved sector of the backup storage area (accounting for 10% of the backup storage area capacity) in advance, and its physical address is updated in the index table to avoid data loss due to the complete failure of the data block. The priority scheduling mechanism ensures the priority processing of critical data (such as vibration over-limit data). The independent channel avoids bus competition with ordinary data. The token bucket algorithm controls the bandwidth to stabilize the data write rate and avoids write failures caused by bus congestion. The large token bucket capacity of critical data ensures that it can still be written smoothly when the amount of data increases suddenly, reducing the probability of errors caused by write delay and improving the system's ability to protect critical data.

[0034] The high 4 bits of the device ID are defined as follows: levels 0-3 are low priority (e.g., ambient temperature and humidity), levels 4-9 are medium priority (e.g., device operating voltage), and levels 10-15 are high priority (levels 10-12 are general alarm data, and levels 13-15 are emergency alarm data); the independent write channel uses a dedicated SPI bus with a speed of 50Mbps, while ordinary data is shared. The bus has a speed of 1Mbps; the token bucket generates tokens at a rate of 1MB / s for critical data and 512KB / s for ordinary data. When the critical data token bucket is full, excess tokens are temporarily allocated to the ordinary data bucket, but 30% of the critical data token capacity is reserved to ensure that there are still enough tokens when critical data bursts. When the bus is congested (bus utilization exceeds 90%), the system will automatically reduce the write frequency of ordinary data and allocate 60% of the bus bandwidth to the critical data channel to further ensure the transmission of critical data.

[0035] The parallel write channel in the S20 employs a priority scheduling mechanism: when the IoT data acquisition device transmits multiple types of data simultaneously (such as temperature, humidity, and vibration), the high 4 bits of the device ID are used to distinguish priorities (levels 0-15). Critical data with a priority ≥ 10 (such as vibration exceeding limits data) is allocated an independent write channel, while ordinary data with a priority < 10 shares the remaining channels. A token bucket algorithm is used to control bandwidth during writing, with a 1MB token bucket capacity for critical data and 512KB for ordinary data. This ensures that critical data can still be written preferentially even when the bus is congested, reducing its error probability. This priority scheduling mechanism based on the high 4 bits of the device ID effectively distinguishes different types of data in the IoT data acquisition device. The importance of the data is carefully considered to prevent delays or loss caused by competition between critical data (such as emergency data related to equipment safety, such as vibration exceeding limits) and ordinary data during bus transmission. A dedicated write channel provides a dedicated transmission path for critical data, reducing the possibility of interference from ordinary data transmission. The application of the token bucket algorithm ensures the stability of the data write rate by controlling bandwidth. The larger token bucket capacity for critical data allows it to be written smoothly even when there is a sudden increase in data volume. Furthermore, the priority mechanism during bus congestion further reduces the probability of errors in critical data, ensuring the integrity and real-time performance of important data. This solves the problem of easy loss of critical information due to the lack of data transmission priority in traditional storage modules.

[0036] The high 4 bits of the device ID (16 priority levels) are divided as follows: Levels 0-3 correspond to low-priority data (such as routine periodic data collection of ambient temperature and humidity); Levels 4-9 correspond to medium-priority data (such as non-emergency data such as device operating current and voltage); Levels 10-12 correspond to higher-priority data (such as early warning data for slight device vibration or temperature slightly exceeding the threshold); and Levels 13-15 correspond to the highest-priority data (such as emergency alarm data that may cause device failure, such as severe vibration exceeding limits or sudden temperature rise). The independent write channel for critical data uses a high-speed SPI bus with a transmission rate of 50Mbps, supporting full-duplex communication. The bus controller allocates an independent DMA channel for critical channels to avoid delays caused by MCU intervention. The remaining channels for shared general data use... The bus has a transmission rate of 400kbps and can simultaneously connect to up to 16 data sources. The token generation rate of the token bucket is set as follows: 128KB tokens are generated per second for critical data buckets and 32KB tokens per second for ordinary data buckets. When the remaining tokens in the critical data bucket are less than 20% of the total capacity, token generation in the ordinary data bucket will be temporarily suspended until the tokens in the critical data bucket are replenished to more than 50%. The bus congestion is judged based on the bus utilization rate (the ratio of real-time transmitted data volume to the maximum bus bandwidth). When the utilization rate exceeds 85%, the critical data priority mechanism is automatically triggered. At this time, ordinary data write requests will be placed in the waiting queue. When the queue length exceeds 100 lines, the sampling frequency of ordinary data will be temporarily reduced by 30% to reduce the data volume and ensure the smooth flow of critical data channels. All priority scheduling logic is implemented by a dedicated hardware scheduler (using an ARM Cortex-M0 core), and the scheduling decision time does not exceed 100ns, further ensuring real-time performance.

[0037] First, the storage modules are initialized, dividing the primary storage area (NAND flash memory) and the backup storage area (NOR flash memory) into a capacity ratio of 7:3. Simultaneously, a dynamic index table based on a hash table is established (using a double-hash collision resolution strategy, with an initial capacity of...). Each entry has a load factor threshold of 0.7 and automatically expands to twice its original size. Each entry includes a physical address, timestamp, checksum, validity identifier, and 8-bit version number, stored in a dedicated FRAM such as FM25V02 to prevent data loss in case of power failure. Its read / write lifespan is no less than [specified value]. This lays the foundation for subsequent data management. Next, after receiving real-time data from the acquisition device, it encapsulates the data into a 512-byte data block according to "Device ID (16 bits) + Acquisition Timestamp (32 bits) + Data Value (64 bits) + CRC32 Checksum (32 bits)", and writes it to the main storage area through a parallel write channel, synchronously updating the index table. The CRC32 checksum is calculated using iterative hardware acceleration (initial value 0xFFFFFFFF, 32-bit word length divided into 128 data points, calculated using a specific iterative formula). Data blocks are compared with software results every 100 data blocks to prevent hardware failures. The write channel is prioritized based on the high 4 bits of the device ID (levels 0-15, ≥10 level critical data use independent channels with a token bucket capacity of 1MB, <10 level ordinary data share a channel with a capacity of 512KB to ensure that critical data is written first). During the data writing process, the hardware ECC circuit monitors the data in real time. Single-bit errors are directly corrected and the error count is marked. Multi-bit errors trigger fault tolerance processing, marking the data block as "invalid" and calling the RS(16,8) check block pre-stored in the backup storage area (8 check blocks are generated from 8 data blocks, based on primitive polynomials). Generate Galois Domain The data block is reconstructed by generating matrix G, which takes ≤3ms. At the same time, an 8-bit error counter is maintained for each data block (increment by 1 for single-bit errors, set to 0xFF for multi-bit errors, and mark as "pre-failed" when it reaches 16 and migrate to the backup reserved sector). Then, periodic synchronization is performed, with incremental synchronization every hour: the index table is compared by MD5 hash value, the difference data blocks with higher version numbers are selected, and they are transferred to the backup storage area via DMA channel (CRC32 is checked every 4KB, the backup index table and hash value are updated after synchronization, and the log is recorded to FRAM). When the percentage of erroneous data blocks in the main memory exceeds 5%, a memory switching is initiated: On the hardware side, the data bus is switched to the backup memory via an analog switch controlled by a CPLD (switching time < 1ms); on the software side, reading and writing in the main memory are stopped, valid index entries are copied to the backup index table and mapped to physical addresses (pre-stored in FRAM), and the data chain is reassembled based on the doubly linked list of the index table (including the previous / next block addresses) (the next address of non-contiguous data blocks is set to 0xFFFFFFFF). The switching response time is ≤ 10ms. After completion, a signal is output via GPIO and a status message is sent. Throughout the process, the chain structure of the dynamic index table ensures the integrity of the data sequence, the error warning mechanism avoids the risk of data loss in advance, and various hardware acceleration (CRC, RS, scheduler) and optimization strategies (token bucket, write-before-delete) work together to ensure the high fault tolerance, real-time performance and data integrity of the storage module.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A fault-tolerant processing method for a data storage module of an Internet of Things (IoT) data acquisition device, characterized in that, Includes the following steps: S10: Initialize the storage module, divide the main storage area and the backup storage area, with a capacity ratio of 7:3 between the main storage area and the backup storage area, and establish a dynamic index table based on a hash table. Each entry in the index table contains the physical address of the data block, timestamp, check value and validity identifier. S20: Receives real-time data from the IoT data acquisition device, encapsulates it into a 512-byte fixed data block according to the structure of "Device ID + Acquisition Timestamp + Data Value + CRC32 Check Bit", writes it to the main storage area through the parallel write channel, and simultaneously updates the index information of the data block to the dynamic index table. S30: Real-time monitoring of the write status of the main memory area, and detection of single / multi-bit errors through hardware ECC circuit: If it is a single-bit error, it is directly corrected through ECC circuit and the error count is marked; If a multi-bit error occurs, the fault tolerance module is triggered, marking the data block as "invalid" and recording the error location. It then calls the pre-stored Reed-Solomon (RS(16,8)) checksum block in the backup storage area and uses the formula... ,in To verify the block data, To generate a matrix, reconstruct the data blocks; S40: Execute a periodic synchronization mechanism, triggering incremental synchronization once per hour: By comparing the differences between the index tables of the primary storage area and the backup storage area, only newly added or modified data blocks are synchronized to the backup storage area. During the synchronization process, a "write first, delete later" strategy is adopted to ensure the data integrity of the backup storage area. S50: When the proportion of erroneous data blocks in the main storage area exceeds 5%, the storage area switching process is initiated: the data read and write path is switched to the backup storage area through a hardware switch, and the complete data sequence is reorganized based on the chain structure of the dynamic index table. The switching response time does not exceed 10ms.

2. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 1, characterized in that, The hash table of the dynamic index table in S10 adopts a double-hash collision resolution strategy, with an initial capacity of 2048 entries and a load factor threshold of 0.

7. When the load factor exceeds the threshold, it automatically expands to twice the current capacity. Each entry in the index table also includes a version number field to record the update count of the data block. When a data block is modified, the version number is automatically incremented by 1 to ensure accurate identification of the old and new states of the data block during synchronization. The hash table is stored independently of the main / backup storage area, using independent FRAM to avoid data loss in case of power failure. FRAM with a capacity of at least 128KB and a read / write lifetime of at least 10... 12 Second-rate.

3. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 1, characterized in that, The calculation of the CRC32 check bit in S20 adopts an iterative hardware-accelerated calculation, and the specific formula is as follows: Set the initial check value The data to be verified is divided into 32-bit word lengths. ,in Each iteration calculates: ,in, This represents the number of bits to shift, with a value of 8. A 256-order CRC lookup table array (composed of polynomials) First, generate the checksum; after iteration, the final checksum is generated. The 32-bit CRC checksum is written to the main memory area. This calculation process is performed by a dedicated CRC hardware accelerator, and the time consumption is controlled within [timeframe missing]. Within a data block, the system automatically verifies the consistency between the accelerator output and the software calculation results after every 100 data blocks are written, avoiding verification errors caused by hardware failures.

4. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 1, characterized in that, The generation of the Reed-Solomon (RS(16,8)) check block in S30 adopts hardware encoding based on primitive polynomials, and the specific formula is as follows: Let there be 8 data blocks. Each block is 64 bytes, generating 8 check blocks. , of which The first check block byte Calculation formula: In the formula For the Gara Flower Domain The primitive element is composed of the primitive polynomial. generate, The exponent is mapped to integers from 0 to 255 using an exponent table. The check block and data block are then... Sequential storage ensures that if any four data blocks fail, the complete data can be reconstructed by solving a system of eight linear equations, with a reconstruction time of ≤3ms.

5. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 4, characterized in that, The generation process of the Reed-Solomon (RS(16,8)) check block in S30 is as follows: Eight data blocks are grouped together, and eight check blocks are generated using the RS encoder. The check blocks and data blocks are stored in consecutive sectors of the backup storage area in the order of "data block 1-8 + check block 1-8", generating a matrix. It is a 16×8 matrix, where the first 8 columns are the identity matrix and the last 8 columns are passed through the primitive polynomial. Generate data to ensure that any 8 blocks can be used to reconstruct the complete set of data.

6. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 5, characterized in that, The incremental synchronization process of S40 includes: a1. When synchronous triggering occurs, first calculate the MD5 hash value of the dynamic index table in the main storage area and compare it with the hash value of the index table in the backup storage area. b1. If the hash values ​​are different, the version number field of the index table is compared item by item to filter out the items in the main storage area with version numbers higher than those in the backup storage area, forming a difference list. c1. Read the data block in the main storage area according to the physical address in the difference list, and transfer it to the backup storage area through the DMA channel. During the transfer, a check is performed every 4KB of data transferred. d1. After synchronization is complete, update the index table and hash value of the backup storage area, and record the synchronization log to the non-volatile register.

7. The fault-tolerant processing method for a data storage module of an IoT data acquisition device according to claim 6, characterized in that, The storage area switching process in S50 specifically includes: e1. Hardware level: The data bus connection is switched via an analog switch controlled by a CPLD, and the switching time is less than 1ms; f1. Software level: Immediately stop all read and write operations in the main storage area, copy the entries in the "valid" state of the dynamic index table to the backup index table, and at the same time map the physical address of each entry to the corresponding location in the backup storage area; g1. Data Reassembly: Based on the preceding / following block address field of the index table, starting from the first valid data block in the backup storage area, the data chain is reassembled in timestamp order, ignoring entries marked as "invalid" during the reassembly process; h1. After the switch is completed, a switch completion signal is output through the GPIO pin, and a status message containing the switch reason and switch time is sent to the main control module of the IoT data acquisition device.

8. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 3, characterized in that, In the linked structure of the dynamic index table, each data block's index entry contains "previous block address" and "next block address" fields, forming a doubly linked list: for data blocks collected continuously in chronological order, the previous block address points to the adjacent previous data block, and the next block address points to the adjacent next data block; for non-contiguous data blocks, the next block address is set to 0xFFFFFFFF. This structure ensures that even if some index entries are lost when the main storage area is damaged, most of the continuous data can still be recovered by tracing back or forward through the linked list, improving the robustness of data recovery.

9. A fault-tolerant processing method for a data storage module of an Internet of Things (IoT) data acquisition device according to any one of claims 1 to 8, characterized in that, It also includes an error warning mechanism: In step S30, an error counter is maintained for each data block. The counter is incremented by 1 after a single-bit error is corrected, and the counter is set to 0xFF when a multi-bit error occurs. When the error counter of a certain data block reaches 16, it is marked as "pre-failure" even if no multi-bit error has occurred. During the next synchronization, the data block is migrated to the reserved sector of the backup storage area in advance, and its physical address is updated in the index table to avoid data loss caused by the complete failure of the data block.

10. The fault-tolerant processing method for the data storage module of an IoT data acquisition device according to claim 1, characterized in that, The parallel write channel in S20 adopts a priority scheduling mechanism: when the IoT data acquisition device transmits multiple types of data at the same time, the priority is distinguished by the high 4 bits of the device ID. Key data with a priority ≥ 10 is allocated an independent write channel, while ordinary data with a priority < 10 shares the remaining channel. The token bucket algorithm is used to control the bandwidth during writing. The token bucket capacity for key data is 1MB, and for ordinary data it is 512KB, ensuring that key data can still be written first when the bus is congested, thus reducing its error probability.

Images