A data storage method and system for embedded devices

Abstract

This application provides a data storage method and system for embedded devices. The method mentions that when there is a data write request, the logical address to which the data needs to be written is obtained, the physical address corresponding to the logical address is found on the dynamic mapping table, and the data is directly written to the physical address. After the data is written to the physical address, a checksum is read. By verifying the checksum matching degree, it is determined whether the physical sector of the corresponding physical address is a bad block. If the physical sector is a bad block, a new physical sector is allocated from the main storage medium for writing the data. The data is written to the new sector to avoid write failure due to the existence of bad blocks and the determination that the main storage medium cannot be written. It completely abandons centralized metadata. Local bad blocks only affect the corresponding data and will not cause the entire storage volume to be read-only or inaccessible, making the reliability guarantee more deterministic.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method and system for embedded devices. Background Technology

[0002] In large-capacity data storage scenarios for embedded devices (such as video surveillance), existing technologies typically use traditional general-purpose file systems such as FAT32, exFAT, and EXT4, or lightweight file systems such as SPIFFS and LittleFS to implement data storage. Both types of file systems have significant drawbacks.

[0003] While traditional general-purpose file systems offer good compatibility, they have inherent drawbacks: frequent read and write operations can generate a large amount of fragmentation, reducing read and write efficiency and accelerating wear and tear on storage media; metadata (such as FAT tables and inodes) is crucial, and if damaged due to bad blocks on the media or unexpected power outages, the entire storage volume can become read-only or inaccessible, resulting in catastrophic data loss; they are unaware of the health status of the storage media and can only passively respond to errors, resulting in weak exception handling mechanisms; and to maintain metadata and log structures, they introduce write amplification, consuming storage space and I / O bandwidth.

[0004] While lightweight file systems are optimized for Flash characteristics, offering small size and strong power-loss resistance, they were originally designed for NOR Flash and are not optimal for continuous large file writes to high-capacity NAND Flash (such as eMMC and TF cards). Their log-structured write method still generates fragmentation, and their garbage collection mechanism is inefficient in "write-heavy, delete-light" video storage scenarios, requiring additional resources and causing write amplification. They still need to maintain their own metadata, which is intertwined with user data, posing a risk of data loss when facing severe media corruption. Furthermore, they lack deep awareness of the underlying physical media health status, making preventative maintenance impossible.

[0005] Existing improvement solutions still suffer from single points of failure due to reliance on centralized metadata. The storage system may fail due to local physical media damage, making it difficult to meet the high reliability requirements of embedded devices. Summary of the Invention

[0006] The technical problem this application aims to solve is that existing improvement solutions still suffer from single points of failure due to reliance on centralized metadata. The storage system may experience global failures due to local physical media damage, making it difficult to meet the high reliability requirements of embedded devices.

[0007] To solve the above-mentioned technical problems, or at least partially solve them, this application provides a data storage method and system for embedded devices.

[0008] In a first aspect, the present invention discloses a data storage method for an embedded device, comprising: Obtain the data and the logical address where the data is written. Based on the dynamic mapping table, obtain the physical address corresponding to the logical address where the data is written, and write the data to the physical address of the main storage medium. After the data is written to the physical address of the main storage medium, a checksum is read to determine whether the physical sector is a bad block. If the physical sector is a bad block, a new sector is allocated from the spare sector pool of the main storage medium for writing the data.

[0009] Preferably, after the data is written to the physical address of the main storage medium, a checksum reading process is performed. The checksum reading is used to determine whether the physical sector is a bad block. If the physical sector is a bad block, a new sector is allocated from the spare sector pool for writing data. Specifically, this includes the following steps: After the data is written to the physical address, a random check code is generated based on the CRC32 algorithm. After writing, the data is read immediately. Based on the read data, a real-time check code is generated based on the CRC32 algorithm. The random check code and the real-time check code are compared to see if they match. The checksum is then matched with the real-time checksum, and the data is successfully written to the physical address. A data write success message is output. An index item set is generated based on the written data and stored in the index area. The index item includes the data acquisition time, logical address, data length, and random checksum. If the read fails or the checksum does not match, it is determined that the data was not written to the physical address. The physical sector is identified as a bad block, the physical address where the write failed is marked as a bad block, a new physical address is allocated from the spare sector pool, a new mapping relationship is established between the logical address corresponding to the bad block and the new physical address, the dynamic mapping table is updated, and the data is written to the new physical address.

[0010] Preferably, the steps preceding the acquisition of data and the logical address for data writing include: Initialize the primary storage medium, logically partition the primary storage medium to obtain logical sectors, and establish a one-to-one correspondence between the logical addresses of the logical sectors and the physical addresses of the primary storage medium to obtain a logical-physical mapping table, thus obtaining a dynamic mapping table; the logical sector includes a data area, an index area, a system area, and a spare sector pool.

[0011] Preferably, the following steps are then included: Periodically obtain relevant parameters of the main storage medium, calculate a comprehensive health score based on these parameters, and determine the health of the main storage medium based on the comprehensive health score.

[0012] Preferably, the step of periodically acquiring relevant parameters of the main storage medium, calculating a comprehensive health score based on these parameters, and determining the health of the main storage medium based on the comprehensive health score specifically includes the following steps: A preset time period is used to query the SMART information of the storage medium at preset time intervals. The SMART information includes the count of reallocated sectors, the count of programming / erase failures, the count of CRC errors, and the media wear level. The overall health score is calculated based on the real-time SMART information and change trends of the storage medium and the thresholds defined by the manufacturer. The system presets a first threshold and a second threshold, compares them with the overall health score, and adjusts the early warning strategy based on the comparison results. When the overall health score is higher than the first threshold, the system continues to monitor the health of the main storage medium. When the overall health score is lower than the first threshold but higher than the second threshold, the system log will issue an alarm, prompting the user to pay attention to the health of the main storage media. When the overall value falls below the second threshold, the system proactively sends an emergency alarm event and sends a message to replace the main storage medium.

[0013] Preferably, the following steps are then included: Obtain a data retrieval request, which includes an index item; load the index table of the index area into spare memory. Construct a binary search algorithm model, perform search processing based on the index table of index items and index area to obtain the set of index items, obtain the logical address of the data from the set of index items, query the dynamic mapping table to obtain the physical address, and directly read the original data; While reading the original data, a random checksum is verified. If the verification is successful, the original data is output. If the verification fails, bad blocks are marked and data corruption is recorded. Information about lost original data is also output.

[0014] In a second aspect, the present invention discloses a data storage system for embedded devices, including an upper-layer application module, a raw disk storage management engine, a spare flash memory and a main storage medium, wherein the raw disk storage management engine interacts with the upper-layer application module, the spare flash memory and the main storage medium respectively; The upper-layer application module obtains the logical address of the data and writes it. The raw disk storage management engine obtains the physical address corresponding to the logical address according to the built-in dynamic mapping table, writes the data into the main storage medium, and generates an index item set. The index item set is stored in the main storage medium and the backup flash memory. The raw disk storage management engine detects bad blocks in the main storage medium, reallocates new physical sectors, updates the index table and bad block table, and stores the index table and bad block table in the backup flash memory.

[0015] Preferably, the upper-layer application module includes an encoder and a retrieval service unit. The encoder transmits signals with the main storage medium, and the retrieval service unit transmits signals with the raw disk storage management engine. The encoder is used to write data and obtain data write requests, and the retrieval service unit is used to obtain retrieval requests.

[0016] Preferably, the raw disk storage management engine includes an L2P mapping module, an index management module, a health and bad block management module, and a redundancy backup module; The L2P mapping module is used to generate, update, and query dynamic mapping tables; the index management module is used to generate, update, and query index tables; the health and bad block management module performs health score calculations; and the redundant backup module connects the backup flash memory and the main storage medium respectively, storing the index table and the bad block table.

[0017] Preferably, the main storage medium includes an index area, a system area, a data area, and a spare sector pool. The data area is used to store the original data, the system area is used to store the dynamic mapping table and the bad block table, the index area is used to store the index table, and the spare sector pool is used for bad block replacement.

[0018] The technical solution provided in this application has the following advantages compared with the prior art: This application provides a data storage method and system for embedded devices. The method involves obtaining the logical address to which the data needs to be written when a data write request is received, finding the corresponding physical address in a dynamic mapping table, and directly writing the data to the physical address. After the data is written to the physical address, a checksum is read. By verifying the checksum matching degree, it is determined whether the physical sector of the corresponding physical address is a bad block. If the physical sector is a bad block, a new physical sector is allocated from the main storage medium for writing the data. This avoids write failures caused by bad blocks and prevents the main storage medium from being deemed unwritable. It completely abandons centralized metadata; local bad blocks only affect the corresponding data and do not cause the entire storage volume to become read-only or inaccessible, thus providing more deterministic reliability assurance.

[0019] Furthermore, after the application data is encapsulated, it is directly written to the physical medium according to the sector address, realizing direct writing to the bare disk. This completely avoids all the extra I / O operations brought by the file system layer, such as fragmentation, logging, and metadata updates. It eliminates the extra overhead of file system fragmentation, logging, and garbage collection, significantly improves continuous write performance, has higher throughput, and effectively extends the lifespan of the storage medium.

[0020] Furthermore, based on SMART information, health assessment and multi-level early warning have enabled a shift from passively responding to faults to proactively warning and intelligently avoiding them, supporting predictive maintenance and greatly reducing the risk of data loss.

[0021] Furthermore, the self-developed index table and bad block table, as well as other key metadata, are simultaneously written to a backup storage medium through a redundant backup module while being written to the primary storage medium. When the primary medium is replaced, the system can prioritize loading the latest metadata from the backup medium, quickly rebuild the storage structure, and restore data retrieval capabilities. Even if the primary storage medium suffers a serious failure, the system can maximize the traceability and recoverability of the data, resulting in stronger business continuity. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a data storage method for an embedded device provided in this application; Figure 2 A schematic diagram of a data retrieval process for a data storage method for an embedded device provided in this application; Figure 3 A flowchart of a data storage method for an embedded device provided in this application; Figure 4 A flowchart of a data retrieval method for data storage in an embedded device is provided in this application; Figure 5 A flowchart of a health detection method for data storage in an embedded device is provided in this application; Figure 6 This application provides a block diagram of a data storage system for embedded devices. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Firstly, see Figures 1-5This invention discloses a data storage method for embedded devices, comprising: Step S1: Obtain the data and the logical address for writing the data. Based on the dynamic mapping table, obtain the physical address corresponding to the logical address for writing the data, and write the data to the physical address of the main storage medium. Step S2: After the data is written to the physical address of the main storage medium, a checksum reading process is performed. The checksum reading is used to determine whether the physical sector is a bad block. If the physical sector is a bad block, a new sector is allocated from the spare sector pool of the main storage medium for writing data.

[0027] Specifically, in step S1, the raw disk storage management engine receives a write request initiated by the upper-layer application, obtains the complete raw data packet (such as a 4MB video clip) and the specified target logical address (LBA) from the request, and optionally receives auxiliary identifiers such as data type and channel number. The L2P mapping module of the raw disk storage management engine accesses the system area of ​​the main storage medium and queries the dynamic mapping table. The dynamic mapping table is a lookup table of logical addresses (LBA) and physical addresses (PBA) in the main storage medium. The engine obtains the physical address corresponding to the target logical address. The raw disk storage management engine skips the file system driver and directly writes the raw data packet in binary form to the physical sector corresponding to the logical address in the data area of ​​the main storage medium. The writing process does not perform any additional operations such as metadata updates or log recording at the file system layer. The CRC32 random check code calculated based on the raw data packet before writing is temporarily stored in memory for subsequent verification.

[0028] Specifically, in step S2, after the data is written to the target physical address, the engine immediately reads all the written data in that physical sector and uses the same CRC32 algorithm (Cyclic Redundancy Check) as the random checksum to calculate a real-time checksum for the read data. The real-time checksum is compared bit by bit with the random checksum, and at the same time, it is determined whether there are any abnormalities such as read timeout or read failure during the data reading process. If a checksum mismatch or read abnormality occurs, the physical sector corresponding to the current physical address is directly determined to be a bad block. The health status monitoring and bad block management module writes the physical address, fault type (checksum failure / read failure), and marking time of the bad block into the bad block table of the main storage medium system area and backs it up synchronously to the backup flash memory. This physical address will not be allocated again in the future. The L2P mapping module allocates a new unused sector (i.e., a new physical address) from the spare sector pool (5% of the total capacity) of the main storage medium, updates the dynamic mapping table, and establishes a new mapping relationship between the original target logical address and the new physical address. The engine rewrites the original data packet to the physical sector corresponding to the new physical address. After writing, the real-time checksum calculation and random checksum comparison process is repeated to confirm successful writing. If the rewrite verification is successful, a write success message is returned to the upper-layer application; if it fails after 3 retries, a write failure message is returned, and the medium should be checked.

[0029] It's understandable that when a data write request is received, the logical address where the data needs to be written is obtained, the physical address corresponding to the logical address is found on the dynamic mapping table, and the data is directly written to the physical address. After the data is written to the physical address, a checksum needs to be read. By verifying the checksum matching degree, it is determined whether the physical sector of the corresponding physical address is a bad block. If the physical sector is a bad block, a new physical sector is allocated from the main storage medium for writing the data. The data is written to the new sector to avoid write failure due to the existence of bad blocks and to determine that the main storage medium cannot be written to. This completely abandons centralized metadata. Local bad blocks only affect the corresponding data and will not cause the entire storage volume to be read-only or inaccessible, making the reliability guarantee more deterministic.

[0030] Furthermore, after the application data is encapsulated, it is directly written to the physical medium according to the sector address, realizing direct writing to the bare disk. This completely avoids all the extra I / O operations brought by the file system layer, such as fragmentation, logging, and metadata updates. It eliminates the extra overhead of file system fragmentation, logging, and garbage collection, significantly improves continuous write performance, has higher throughput, and effectively extends the lifespan of the storage medium.

[0031] Step S2 specifically includes the following steps: Step S21: After the data is written to the physical address, a random check code is generated based on the CRC32 algorithm. After writing, the data is read immediately. Based on the read data, a real-time check code is generated based on the CRC32 algorithm. The random check code and the real-time check code are compared to see if they match. Step S22: The random check code is matched with the real-time check code, the data is successfully written to the physical address, and the data writing success message is output; an index item set is generated based on the written data, and the index item set is stored in the index area. The index item includes the data acquisition time, logical address, data length, and random check code. Step S23: If the read fails or the checksum does not match, it is determined that the data writing to the physical address failed, the physical sector is determined to be a bad block, the physical address where the write failed is marked as a bad block, a new physical address is allocated in the spare sector pool, a new mapping relationship is established between the logical address corresponding to the bad block and the new physical address, the dynamic mapping table is updated, and the data is written to the new physical address.

[0032] Specifically, before data is written to the physical address, a random checksum is calculated using the CRC32 algorithm based on the complete original data packet transmitted by the upper-layer application and temporarily stored in spare memory. After the data is successfully written to the target physical address (PBA), the engine immediately initiates a read operation to read all the data already written in that physical sector (consistent with the write length, such as 4MB), ensuring that the read range and write range are completely matched. For the read written data, the CRC32 algorithm, which is exactly the same as the random checksum, is executed to generate a real-time checksum. The real-time checksum is then compared bit by bit with the random checksum temporarily stored in memory. At the same time, the read process is monitored for any anomalies such as read timeout, incomplete data read, or hardware feedback of read failure. By verifying immediately after writing, it is confirmed that the data has been written to the physical sector completely and without tampering, avoiding incomplete writing due to momentary media failure. After confirming a perfect match of the checksum and no errors during reading, the engine returns a successful data write feedback to the upper-layer application, informing it that subsequent write requests can proceed. Based on the key information from this write operation, index entries are generated according to a preset index entry structure, including the data acquisition timestamp, target logical address (LBA), data length, and random checksum. Optional fields such as data type and channel number can be added. The generated index entries are stored in a memory cache (with a maximum cache size of 10 index entries). When the cache reaches its limit or more than 5 minutes have passed since the last persistence, the index entries are sequentially written to the index area of ​​the main storage medium in batches, while simultaneously being written to backup flash memory (such as NOR Flash) via a redundant backup module. The index entries record the core location information of the data (LBA, checksum, etc.), providing the index target for subsequent data retrieval based on timestamps, data types, and other conditions. The dual-write index entries, using both main and backup storage, completely avoid the risk of index loss and ensure data traceability. Furthermore, the index entries are cached before being written in batches, avoiding I / O performance fluctuations caused by frequent small-volume writes and adapting to continuous high-volume write scenarios. If a read failure or checksum mismatch occurs, the data write operation is immediately deemed a failure. Simultaneously, the physical sector corresponding to the target physical address (PBA) is marked as a bad block. The physical address of the bad block, the fault type (checksum failure / read failure), and the marking time are recorded in the bad block table of the main storage medium's system area. Simultaneously, the bad block table in the backup flash memory is updated to ensure no bad block information is lost. The L2P mapping module accesses the spare sector pool of the main storage medium (5% of the total capacity reserved space), allocates an unused, intact physical address (PBA_new), removes the original logical address (LBA) from the physical address of the bad block (PBA), establishes a new mapping between the logical address (LBA) and the new physical address (PBA_new), and updates the dynamic mapping table (L2P mapping table) of the main storage medium's system area. The engine rewrites the original data packet to the physical sector corresponding to the new physical address (PBA_new). After the write is complete, the above checksum logic can be reused to confirm a successful rewrite.Timely marking of bad blocks and prohibiting their subsequent allocation prevents data loss or corruption caused by the reuse of bad blocks. Data is rewritten using spare sectors to ensure that the original data of upper-layer applications is not lost, maintaining business data continuity. From bad block marking and new address allocation to data rewriting, the entire process is automated without manual intervention, making it suitable for unattended scenarios in embedded devices. Spare sector pools are allocated on demand, preventing the spread of bad blocks from causing overall media failure and significantly extending the lifespan of the main storage media.

[0033] The steps preceding step S1 include: Step S0: Initialize the main storage medium, logically partition the main storage medium to obtain logical sectors, map the logical addresses of the logical sectors to the physical addresses of the main storage medium one-to-one, establish a logical-physical mapping table, and obtain a dynamic mapping table; the logical sector includes a data area, an index area, a system area, and a spare sector pool.

[0034] Specifically, upon the first boot of the embedded device, the insertion of a new primary storage medium (such as eMMC or TF card), or when the user manually triggers the initialization command, the raw disk storage management engine automatically initiates the primary storage medium initialization process. The raw disk storage management engine performs a low-level formatting operation on the primary storage medium, without creating a traditional file system (such as FAT32 or EXT4). Instead, it divides the primary storage medium into several logical sectors of uniform size according to a fixed specification (4MB / sector, matching the continuous data storage requirements of video clips, etc.), generating a continuous logical address (LBA) sequence (such as LBA 0, LBA 1, ..., LBA 1). N-1, where N = total main storage capacity / 4MB), based on the formatted logical sectors, is divided into four independent logical partitions according to function: data area, index area, system area, and spare sector pool. The data area stores raw user data generated by upper-layer applications (such as video streams and sensor data). The index area stores a set of index entries (including key retrieval information such as data acquisition time, LBA, and checksum). The system area stores core system data such as the dynamic mapping table (L2P mapping table) and bad block table. The spare sector pool reserves intact logical sectors for subsequent bad block replacement. Following a linear correspondence rule, the logical address (LBA) of each logical sector is bound one-to-one with the physical address (PBA) of the main storage medium, i.e., LBA[n] = PBA[n] (n is an integer from 0 to N-1), forming the initial logical-physical mapping table, also known as the dynamic mapping table. The dynamic mapping table is stored in the system area of ​​the main storage medium and simultaneously backed up to backup flash memory (such as NOR Flash) to ensure the initial state of the mapping table is traceable.

[0035] Understandably, by performing low-level formatting and partitioning, the raw physical storage medium is transformed into a usable storage medium with clearly defined functional partitions and identifiable addresses. This adapts to the requirements of a raw disk write-to-grid architecture, without relying on the partition tables and metadata of traditional file systems. The formatting and partitioning process is extremely lightweight, with no additional overhead, making it suitable for embedded scenarios with continuous high-volume writes. Initial linear mapping enables logical addresses to directly locate physical addresses, resulting in extremely high address translation efficiency during writes and reads. At the same time, the dynamic mapping table supports flexible updates and can quickly respond to bad block replacement needs. User data, indexes, system configurations, and spare sectors are stored separately to avoid interference between different types of data.

[0036] Furthermore, the self-developed index table and bad block table, as well as other key metadata, are simultaneously written to a backup storage medium through a redundant backup module while being written to the primary storage medium. When the primary medium is replaced, the system can prioritize loading the latest metadata from the backup medium, quickly rebuild the storage structure, and restore data retrieval capabilities. Even if the primary storage medium suffers a serious failure, the system can maximize the traceability and recoverability of the data, resulting in stronger business continuity.

[0037] Step S2 is followed by the following steps: Step S3: Periodically obtain relevant parameters of the main storage medium, calculate the comprehensive health score based on the relevant parameters of the main storage medium, and judge the health of the main storage medium based on the comprehensive health score.

[0038] Specifically, the health status monitoring and bad block management module of the raw disk storage management engine initiates health checks automatically according to a preset cycle, or by receiving manual triggering commands from upper-layer applications. It communicates with the main storage medium through the SCSI ATA command set (or eMMC equivalent commands) to read its SMART information (Self-Monitoring, Analysis and Reporting Technology), including four types of parameters: reallocated sector count, programming / erase failure count, CRC error count, and media wear. Through a built-in algorithm, it calculates a health score. The health scores obtained based on each parameter are weighted and summed using a weighted scoring algorithm to calculate the individual scores with fixed weights, finally obtaining a comprehensive health score of 0-100. Based on the comprehensive health score, a preset threshold is matched, and corresponding processing actions are executed. Breaking away from the limitations of existing technologies that passively respond to faults, this technology regularly collects core health parameters, monitors the wear and tear and failure risk of the main storage media in real time, triggers multi-level alarms based on score thresholds, and reminds users to replace the media before it completely fails. This avoids data loss caused by media failure from the source, provides data basis for the maintenance of storage media in embedded devices, avoids excessive or delayed replacement, and optimizes maintenance costs and efficiency.

[0039] Step S3 specifically includes the following steps: Step S31: Set a preset time period and query the SMART information of the storage medium at preset time intervals. The SMART information includes the count of reallocated sectors, the count of programming / erase failures, the count of CRC errors, and the wear level of the medium. Step S32: Calculate the overall health score based on the real-time acquired SMART information and change trends of the storage medium and the threshold defined by the manufacturer; Step S33: Preset a first threshold and a second threshold, compare them with the overall health score, and adjust the early warning strategy based on the comparison results; Step S34: When the overall health score is higher than the first threshold, the system continues to monitor the health of the main storage medium; Step S35: When the overall health score is lower than the first threshold but higher than the second threshold, the system log will issue an alarm, prompting the user to pay attention to the health of the main storage medium; Step S36: When the overall value is below the second threshold, the system actively sends an emergency alarm event and sends information to replace the main storage medium.

[0040] Specifically, during system initialization, a fixed query time period is set, and periodic checks are triggered by a timer. Simultaneously, it supports manual initiation of real-time query commands by upper-layer applications. The health status monitoring module of the raw disk storage management engine establishes communication with the primary storage medium, reads its raw SMART information, and filters four key parameters from the SMART information: the count of reallocated sectors (number of damaged sectors replaced), the count of failed programming / erase operations (number of failed write / erase operations), the count of CRC errors (number of failed data transmission verifications), and the media wear level (overall media wear ratio, 0-100%). The collected parameter values ​​and timestamps are stored in the health log of the primary storage medium's system area and simultaneously synchronized to the backup flash memory for subsequent trend tracking. The real-time SMART parameters collected are extracted and combined with historical parameters stored in the system area to analyze the changing trends of each parameter. The preset parameter safety thresholds of the primary storage medium manufacturer are queried, and parameters exceeding the safety range are adjusted in weight. A multi-dimensional weighted scoring algorithm is used to calculate a comprehensive health score. The calculated comprehensive health score and its calculation basis are stored in the health log and simultaneously backed up to the backup flash memory. During system initialization, two health thresholds are preset: a first threshold (e.g., 60 points) is the warning threshold, and a second threshold (e.g., 40 points) is the emergency threshold. Users can adjust these thresholds based on media type and application scenario through the upper-level management platform. The first and second thresholds, along with their corresponding relationships, are stored in a configuration file in the main storage media's system area and synchronously backed up to backup flash memory to prevent threshold loss. When the overall health score exceeds the first threshold, the main storage media is considered healthy, and the health check result, including the score and parameter values, is recorded in the health log without requiring additional feedback. The system continues to periodically collect SMART information within the preset query time period to continuously monitor changes in media status.When the first threshold > overall health score > second threshold (e.g., 60 points > score > 40 points), the primary storage medium is determined to be in a warning state. A detailed alarm record is generated in the system log, including the health score, abnormal parameters, and trend. The upper-layer application backend displays prompts indicating the primary storage medium's health status requires attention. Users can view detailed alarm logs and parameter details through the device's local interface or management platform. The subsequent SMART information query cycle is shortened (e.g., from 6 hours to 3 hours), increasing the monitoring frequency of the media status. When the overall health score is ≤ the second threshold (e.g., ≤ 40 points), the primary storage medium is determined to be in a warning state. In an emergency, where there is a risk of short-term failure, an emergency alarm event is sent to the upper-level management platform (such as a monitoring center or IoT cloud platform) via the device's communication module (such as network or serial port). This alarm event includes the media ID, health score, key abnormal parameters, and collection timestamp. The device displays a prominent pop-up window indicating that the primary storage media is about to fail and should be replaced immediately. At the same time, a replacement notification is pushed to the user's bound terminal (such as a mobile APP) through the upper-level application. In addition, critical data is prioritized for backup, and recently written core user data, index tables, and bad block tables are resynchronized to the backup flash memory to reduce the risk of data loss during the media replacement process.

[0041] The health management module allows for periodic queries of the storage medium's raw health parameters (such as reallocation sector counts and wear indicators) using standard commands (e.g., ATA SMART). A built-in algorithm calculates a comprehensive health score based on these parameters and sets multi-level thresholds. When the health score falls below a threshold, the system proactively sends an alert to the upper-level management system, prompting the replacement of the medium before data loss occurs. This addresses the shortcomings of known technologies and closest solutions, such as lack of awareness of media health status and weak anomaly handling mechanisms. It achieves a shift from passively responding to faults to proactive early warning and intelligent avoidance, supporting predictive maintenance and significantly reducing the risk of data loss. Furthermore, based on SMART information, health assessment and multi-level alerts achieve this shift from passively responding to faults to proactive early warning and intelligent avoidance, supporting predictive maintenance and significantly reducing the risk of data loss.

[0042] Step S2 is followed by the following steps: Obtain a data retrieval request, which includes an index item; load the index table of the index area into spare memory. Construct a binary search algorithm model, perform search processing based on the index table of index items and index area to obtain the set of index items, obtain the logical address of the data from the set of index items, query the dynamic mapping table to obtain the physical address, and directly read the original data; While reading the original data, a random checksum is verified. If the verification is successful, the original data is output. If the verification fails, bad blocks are marked and data corruption is recorded. Information about lost original data is also output.

[0043] Specifically, the raw disk storage management engine receives data retrieval requests from upper-layer applications and parses the target index conditions contained in the request, such as core retrieval dimensions like data collection time range and data type. The engine reads the complete index table from the index area of ​​the main storage medium. The index table contains index entries for all data, collection timestamps, logical address LBAs, data lengths, random checksums, etc., and loads it into the device's spare memory to avoid performance losses caused by frequent readings of the main storage medium's index area. The index table loaded into memory is sorted by data collection timestamps to ensure the orderliness of subsequent binary search, while filtering out invalid index entries. Based on the ordered nature of the index table, a binary search model is constructed. Matching rules for the target conditions are defined, and the binary search is performed on the ordered index table in memory, using the index item conditions in the request as the matching basis. The index item in the middle position of the index table is retrieved, and its core fields (such as timestamps) are compared with the target conditions. The search range is narrowed based on the comparison results (if greater than the target, the left half is searched; if less, the right half is searched). This process is repeated until all matching index items are located. All successfully matched index items are aggregated into an index item set, containing key information such as the logical address (LBA), data length, and random checksum for each index item. The logical address and data length information for each index item are extracted one by one from the index item set. The engine accesses the dynamic mapping table in the system area of ​​the main storage medium, inputs the target logical address, and obtains the currently bound physical address. If the original LBA-corresponding sector is a bad block and has been remapped to a new physical address in the spare sector pool, the engine skips the file system layer and directly reads the original binary data corresponding to that physical address in the main storage medium's data area, without any additional metadata parsing overhead. While reading the raw data, the engine performs a CRC32 algorithm on the read binary data to generate a real-time checksum. This real-time checksum is then compared bit-by-bit with the random checksum stored in the corresponding index entry in the index entry set. If they match, the data is deemed complete, and the raw data is returned to the upper-layer application. If they do not match or the read fails, the data is deemed corrupted, the current PBA is marked as a bad block and written to the bad block table, a data corruption log is recorded, and a message indicating raw data loss is output to the upper-layer application. Based on the verification result, the engine returns complete raw data or a data loss message plus a fault log to the application, completing the retrieval loop.

[0044] It is understandable that the data retrieval mechanism transforms global metadata into decentralized, independently manageable entries and stores them separately from the data, thus completely eliminating single points of failure in metadata.

[0045] Secondly, see Figure 6This invention discloses a data storage system for embedded devices, including an upper-layer application module, a raw disk storage management engine, a backup flash memory, and a main storage medium. The raw disk storage management engine interacts with the upper-layer application module, the backup flash memory, and the main storage medium. The upper-layer application module obtains data and the logical address to be written. The raw disk storage management engine obtains the physical address corresponding to the logical address according to the built-in dynamic mapping table, writes the data into the main storage medium, and generates an index set. The index set is stored in the main storage medium and the backup flash memory. The raw disk storage management engine detects bad blocks in the main storage medium, reallocates new physical sectors, updates the index table and the bad block table, and stores the index table and the bad block table in the backup flash memory.

[0046] Specifically, the upper-layer application module includes an encoder and a retrieval service unit. The encoder transmits signals with the main storage medium, and the retrieval service unit transmits signals with the raw disk storage management engine. The encoder is used to write data and obtain data write requests, while the retrieval service unit is used to obtain retrieval requests. It can be understood that the encoder, as the core data generation unit of the embedded device (such as the video encoder of a surveillance camera or the image encoder of a dashcam), is responsible for converting physical signals (images, audio, etc.) into binary data conforming to storage standards and initiating data write requests. Through standardized data preprocessing and request encapsulation, it reduces the processing burden on the raw disk storage management engine while ensuring the standardization of data writing. Real-time interaction with the engine ensures the continuity of business data, adapting to the scenario requirements of continuous generation and immediate storage in embedded devices. The retrieval service unit receives retrieval requests from users or the upper-layer management platform, converts them into standardized requests recognizable by the engine, and receives retrieval results and feeds them back to the terminal.

[0047] The raw disk storage management engine includes an L2P mapping module, an index management module, a health and bad block management module, and a redundancy backup module. The L2P mapping module is used to generate, update, and query dynamic mapping tables. The index management module is used to generate, update, and query index tables. The health and bad block management module performs health score calculations. The redundancy backup module connects the backup flash memory and the main storage medium respectively, and stores the index table and the bad block table.

[0048] The L2P mapping module is responsible for managing the mapping between logical addresses (LBA) and physical addresses (PBA). It dynamically updates the mapping table to achieve transparent replacement of bad blocks, forming the foundation of the raw disk write-through architecture. During system initialization, an initial dynamic mapping table is created in the system area of ​​the main storage medium, using an array structure (index is LBA, value is PBA), initially in a linear mapping state (LBA[n] = PBA[n]). When receiving write / retrieval requests from upper-layer applications, it quickly queries the dynamic mapping table based on the logical address in the request to obtain the corresponding logical address. After receiving bad block notifications from the health and bad block management module, it allocates new physical addresses from the spare sector pool, updates the mapping relationship between the original logical address and the new physical address to the mapping table, and deletes the original mapping to ensure that subsequent accesses do not point to bad blocks. The index management module generates index information for the data, constructs an efficient retrieval index table, and enables rapid data location, providing core assurance for data traceability in raw disk storage scenarios. The health and bad block management module monitors the status of the main storage medium in real time, detects and marks bad blocks, and calculates health scores, making it the core module for proactive fault warning. The redundant backup module is responsible for synchronously backing up critical metadata such as the index table and bad block table to the backup flash memory, providing recovery support in the event of a failure of the main storage medium, and serving as a safety net for the system's high reliability.

[0049] Through redundant backup design, critical metadata such as index tables and bad block tables are simultaneously written to a separate, more reliable backup flash memory (such as SPI NOR Flash) via a redundant backup module while being written to the primary storage medium. When the primary medium is replaced, the system can prioritize loading the latest metadata from the backup medium, quickly rebuild the storage structure, and restore data retrieval capabilities. Even if the primary storage medium experiences a serious failure, the system can maximize data traceability and recoverability, resulting in stronger business continuity.

[0050] The main storage medium includes an index area, a system area, a data area, and a spare sector pool. The data area stores raw data, the system area stores the dynamic mapping table and bad block table, the index area stores the index table, and the spare sector pool is used for bad block replacement. The storage medium is the core data storage unit of the system, preferably using NAND Flash media such as eMMC or TF cards. Internally, it is functionally divided into four independent logical sectors: index area, system area, data area, and spare sector pool. Corresponding to these four logical sectors, the main storage medium is divided into four physical sectors, achieving functional isolation and controllable risk. Each sector is partitioned according to preset requirements, and the proportion of each area is adjustable.

[0051] The backup flash memory is a high-reliability guarantee unit for the system. It preferably uses NOR Flash media and exists independently of the main storage media. It only stores two types of core metadata: the index table and the bad block table. It does not store original user data, thus avoiding the occupation of a large amount of storage space. When the main storage media is physically damaged and cannot be identified, after replacing it with a new main storage, the raw disk storage management engine can read the latest index table and bad block table from the backup flash memory, quickly rebuild the dynamic mapping table and index structure, and quickly restore the system's storage capacity. When the main storage system area is damaged, the mapping table and bad block table can be read from the backup flash memory to directly overwrite and repair the system area data of the main storage, thus avoiding system paralysis.

[0052] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0053] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0054] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0055] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0058] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0059] The above description describes specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data storage method for an embedded device, characterized by, include: Initialize the main storage medium, logically partition the main storage medium to obtain logical sectors, map the logical addresses of the logical sectors to the physical addresses of the main storage medium one-to-one, establish a logical-physical mapping table, and obtain a dynamic mapping table; the logical sector includes a data area, an index area, a system area, and a spare sector pool; Obtain the data and the logical address where the data is written. Based on the dynamic mapping table of the main storage medium, obtain the physical address corresponding to the logical address where the data is written, and write the data to the physical address of the main storage medium. After data is written to the physical address of the main storage medium, a checksum reading process is performed. The checksum reading is used to determine whether the physical sector is a bad block. If the physical sector is a bad block, a new sector is allocated from the spare sector pool of the main storage medium for writing data. Specifically, the process includes the following steps: after the data is written to the physical address, a random checksum is generated based on the CRC32 algorithm. After writing, the data is read immediately. Based on the read data, a real-time checksum is generated based on the CRC32 algorithm. The random checksum and the real-time checksum are compared to see if they match. The checksum is then matched with the real-time checksum, and the data is successfully written to the physical address. A data write success message is output. An index item set is generated based on the written data and stored in the index area. The index item includes the data acquisition time, logical address, data length, and random checksum. If the read fails or the checksum does not match, it is determined that the data was not written to the physical address. The physical sector is identified as a bad block, the physical address where the write failed is marked as a bad block, a new physical address is allocated from the spare sector pool, a new mapping relationship is established between the logical address corresponding to the bad block and the new physical address, the dynamic mapping table is updated, and the data is written to the new physical address. Obtain a data retrieval request, which includes an index item; load the index table of the index area into spare memory. Construct a binary search algorithm model, perform search processing based on the index table of index items and index area to obtain the set of index items, obtain the logical address of the data from the set of index items, query the dynamic mapping table to obtain the physical address, and directly read the original data; While reading the original data, a random checksum is verified. If the verification is successful, the original data is output. If the verification fails, bad blocks are marked and data corruption is recorded. Information about lost original data is also output.

2. The method of claim 1, wherein, Includes the following steps: Periodically obtain relevant parameters of the main storage medium, calculate a comprehensive health score based on these parameters, and determine the health of the main storage medium based on the comprehensive health score.

3. The method of claim 2, wherein, The process of periodically acquiring relevant parameters of the main storage medium, calculating a comprehensive health score based on these parameters, and determining the health of the main storage medium based on the comprehensive health score includes the following steps: A preset time period is used to query the SMART information of the storage medium at preset time intervals. The SMART information includes the count of reallocated sectors, the count of programming / erase failures, the count of CRC errors, and the media wear level. The overall health score is calculated based on the real-time SMART information and change trends of the storage medium and the thresholds defined by the manufacturer. The system presets a first threshold and a second threshold, compares them with the overall health score, and adjusts the early warning strategy based on the comparison results. When the overall health score is higher than the first threshold, the system continues to monitor the health of the main storage medium. When the overall health score is lower than the first threshold but higher than the second threshold, the system log will issue an alarm, prompting the user to pay attention to the health of the main storage media. When the overall value falls below the second threshold, the system proactively sends an emergency alarm event and sends a message to replace the main storage medium.

4. A data storage system for an embedded device, applicable to the method described in any one of claims 1-3, characterized in that, It includes an upper-layer application module, a raw disk storage management engine, a backup flash memory, and a main storage medium. The raw disk storage management engine interacts with the upper-layer application module, the backup flash memory, and the main storage medium, respectively. The upper-layer application module obtains the logical address of the data and writes it. The raw disk storage management engine obtains the physical address corresponding to the logical address according to the built-in dynamic mapping table, writes the data into the main storage medium, and generates an index item set. The index item set is stored in the main storage medium and the backup flash memory. The raw disk storage management engine detects bad blocks in the main storage medium, reallocates new physical sectors, updates the index table and bad block table, and stores the index table and bad block table in the backup flash memory.

5. The system according to claim 4, characterized in that, The upper-layer application module includes an encoder and a retrieval service unit. The encoder, as the core data generation unit of the embedded device, is responsible for converting physical signals into binary data that conforms to storage standards and initiating data write requests. Through standardized data preprocessing and request encapsulation, it reduces the processing burden of the raw disk storage management engine while ensuring the standardization of data writing. Real-time interaction with the engine ensures the continuity of business data and adapts to the scenario requirements of continuous generation and instant storage of embedded devices. The retrieval service unit undertakes the retrieval requirements of users or upper-layer management platforms, converts them into standardized requests that the engine can recognize, and receives retrieval results and feeds them back to the terminal.

6. The system according to claim 4, characterized in that, The raw disk storage management engine includes an L2P mapping module, an index management module, a health and bad block management module, and a redundancy backup module; The L2P mapping module is used to generate, update, and query dynamic mapping tables; the index management module is used to generate, update, and query index tables; the health and bad block management module performs health score calculations; and the redundant backup module connects the backup flash memory and the main storage medium respectively, storing the index table and the bad block table.

7. The system according to claim 4, characterized in that, The main storage medium includes an index area, a system area, a data area, and a spare sector pool. The data area is used to store raw data, the system area is used to store dynamic mapping tables and bad block tables, the index area is used to store index tables, and the spare sector pool is used for bad block replacement.

8. The system according to claim 4, characterized in that, The backup flash memory is a high-reliability guarantee unit of the system. It exists independently of the main storage medium and only stores two types of core metadata: index table and bad block table. It does not store original user data. When the primary storage medium becomes unrecognizable due to physical damage, after replacing it with a new primary storage, the raw disk storage management engine can read the latest index table and bad block table from the spare flash memory, quickly rebuild the dynamic mapping table and index structure, and quickly restore the system storage capacity. When the primary storage system area is damaged, the mapping table and bad block table are read from the spare flash memory to directly overwrite and repair the system area data of the primary storage.

Images