Data storage method and system for embedded storage chip

Abstract

The invention relates to the technical field of embedded data storage, and discloses a data storage method and system for an embedded storage chip. The method comprises the following steps: acquiring a to-be-stored original data stream through a data acquisition interface and performing segmentation marking; performing hierarchical decomposition on the data segments by adopting a dynamic decomposition algorithm to generate a plurality of data components; calculating an information density index of each data component, and screening key data components and non-key data components according to the information density index; key data components are stored in a high-speed cache region, and non-key data components are stored in a conventional storage region; performing redundancy check on the key data components in the cache region and generating a check result; after correcting the key data component based on the verification result, recombining the key data component with the non-key data component to generate an optimized data stream; writing the optimized data stream into a physical storage medium and recording a storage position; and monitoring the storage load state, and dynamically adjusting the distribution proportion of the two storage areas.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of embedded data storage, in particular to an embedded storage chip data storage method and system. BACKGROUND

[0002] Embedded systems play a core role in modern electronic devices, from smart wearables to industrial controllers, their stable operation highly depends on reliable data storage capabilities. Embedded storage chips, as the physical carriers of data storage, their performance and reliability directly affect the efficiency of the entire system. However, embedded environments often face resource constraints, including limited storage capacity, lower computing power, and strict power consumption limits. Traditional storage methods often use uniform storage strategies, treating all data equally, without fully considering the characteristics and access requirements of data itself, leading to low storage efficiency, important data vulnerable to damage or loss.

[0003] Existing embedded storage technologies mostly use fixed storage architectures and simple error detection mechanisms when processing data. For example, some systems use cyclic redundancy check or parity check to ensure data integrity, but these methods have large computational overhead and are difficult to deal with sudden storage errors. In addition, as the amount of data processed by embedded devices grows, the types of data also become increasingly complex, including real-time sensor data, user configuration information, program code, etc., their importance, access frequency, and reliability requirements are different. The existing technology lacks fine differentiation of data value and differentiated storage management strategies, and cannot realize the optimal allocation of storage resources.

[0004] In terms of storage architecture, many embedded chips use a single storage area, failing to fully utilize the multi-level storage structure that may exist within the chip. Even if some chips support multi-level storage, their data allocation strategies are often static or semi-static, unable to dynamically adjust according to real-time storage load and data characteristics. This may result in cache areas being occupied by low-frequency access data, while critical data requiring fast read and write cannot be given priority for storage, reducing the performance of the overall storage system. At the same time, as the running time of embedded devices increases, the wear and tear of storage media and external interference may cause data errors, and traditional methods are difficult to detect and repair these errors in a timely manner, posing a threat to data security.

[0005] There is an urgent need in the field of embedded data storage for a storage management method that can intelligently differentiate data value, dynamically optimize storage resource allocation, and actively correct errors. SUMMARY

[0006] The purpose of the present application is to provide an embedded storage chip data storage method and system to solve the problems raised in the background art.

[0007] To achieve the above object, the application provides a data storage method for embedded memory chips, which comprises the following steps: An original data stream to be stored is acquired through a data acquisition interface of an intelligent chip, and the original data stream is subjected to segment marking processing; The marked data segments are subjected to hierarchical decomposition by using a dynamic decomposition algorithm, thereby generating a plurality of data components; An information density index of each data component is calculated, and key data components and non-key data components are screened according to the information density index; The key data components are input into a cache area of the intelligent chip, and the non-key data components are input into a regular storage area of the intelligent chip; A redundancy check operation is performed on the key data components in the cache area, thereby generating a check result; Error correction is performed on the key data components based on the check result, and the corrected key data components and the non-key data components are recombined, thereby generating an optimized data stream; The optimized data stream is written into a physical storage medium of the intelligent chip, and storage location information is recorded; The storage load state of the intelligent chip is monitored, and the allocation ratio of the cache area and the regular storage area is dynamically adjusted according to the storage load state; When a data reading request is detected, the corresponding optimized data stream is extracted from the physical storage medium according to the storage location information, and is restored to the original data stream according to a reverse recombination process.

[0008] Preferably, the hierarchical decomposition of the marked data segments by using the dynamic decomposition algorithm to generate a plurality of data components comprises the following steps: An initial decomposition threshold value and an incremental step length of the dynamic decomposition algorithm are set; An adaptive adjustment coefficient of the dynamic decomposition algorithm is calculated according to characteristic parameters of the original data stream; Real-time decomposition parameters of the dynamic decomposition algorithm are generated based on the initial decomposition threshold value, the incremental step length and the adaptive adjustment coefficient; The marked data segments are subjected to multi-level decomposition operation by using the real-time decomposition parameters, thereby generating a plurality of data components having hierarchical correlation.

[0009] Preferably, the calculation of the information density index of each data component comprises the following steps: Time domain features and frequency domain features of each data component are extracted; The time domain features and the frequency domain features are input into a pre-trained density evaluation model, thereby outputting time domain density values and frequency domain density values of each data component; The time domain density values and the frequency domain density values are subjected to weighted fusion, thereby generating the information density index of the data component.

[0010] Preferably, the redundancy check operation performed on the critical data components in the cache includes: Configure corresponding validation rules based on the data type of the key data components; The key data components are checked in blocks according to the verification rules, and a check code is generated for each data block. The verification code is compared with the pre-stored baseline verification code to generate the verification result.

[0011] Preferably, the error correction of key data components based on the verification results includes: Identify erroneous data blocks and their error types in the verification results; Select the corresponding correction strategy based on the error type, including direct replacement strategy or interpolation reconstruction strategy; The error data block is corrected by applying a correction strategy, and the corrected key data components are generated.

[0012] Preferably, the recombination of the corrected critical data components with the non-critical data components includes: Read the hierarchical relationship between critical data components and non-critical data components; Alignment operations are performed on the corrected critical data components and non-critical data components according to the hierarchical relationship. The aligned data components are merged in the order of their markings to generate an optimized data stream.

[0013] Preferably, the monitoring of the storage load status of the smart chip includes: Collect real-time occupancy and access frequency of the cache and regular storage areas; Calculate storage load balancing based on real-time occupancy and access frequency; When the storage load balancing exceeds a preset threshold, an allocation ratio adjustment operation is triggered.

[0014] Preferably, the step of dynamically adjusting the allocation ratio between the cache area and the regular storage area according to the storage load status includes: Establish a mapping table between storage load status and allocation ratio. Query the mapping table based on the current storage load status to obtain the target allocation ratio, and then redistribute the storage capacity of the cache area and the regular storage area according to the target allocation ratio.

[0015] Preferably, the step of restoring the original data stream according to the reverse reconstruction process includes: Extract key and non-key data components from the optimized data stream; Perform reverse verification on key data components to ensure data integrity; Based on the hierarchical relationship, key data components and non-key data components are merged into a labeled data segment; Remove the marker information from the data segments to restore the original data stream.

[0016] Preferably, the present invention also includes an embedded memory chip data storage system, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described embedded memory chip data storage method.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves precise differentiation and differentiated storage of data value through dynamic decomposition algorithms and information density assessment. Traditional methods treat all data equally, while this method can identify key information in the data stream and prioritize its storage in high-speed areas, ensuring high-speed access and high reliability of important data and improving the utilization efficiency of storage resources.

[0018] By employing redundancy verification and error correction mechanisms to provide focused protection for critical data, the reliability of the storage system is significantly enhanced. Critical data is verified and corrected before being written to the physical media, effectively reducing the risk of data errors caused by storage media instability or external interference, and providing additional protection for the integrity of the system's core data.

[0019] The strategy of dynamically adjusting the allocation ratio of storage regions enables the system to adapt to changing storage loads. By monitoring the load status in real time, the system can optimize the space configuration of cache and regular storage, ensuring high storage performance under different operating conditions and avoiding performance bottlenecks that may be caused by rigid resource configuration.

[0020] The design of the data reconstruction and reverse restoration process optimizes storage while ensuring data integrity and recoverability. This method does not simply separate data for storage; instead, it uses a reversible reconstruction mechanism to maintain the logical structure and semantic consistency of data during writing and reading, facilitating seamless use by upper-layer applications.

[0021] This method constructs a complete closed-loop storage management system encompassing data classification, prioritized storage, error checking, and dynamic optimization. The system can continuously self-optimize based on data characteristics and system status, forming an intelligent, adaptive, and highly reliable data storage solution that helps improve the overall performance and data security of embedded systems. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the working principle of the embedded memory chip data storage method described in this invention. Figure 2 A flowchart for the hierarchical decomposition of labeled data segments; Figure 3 A flowchart for redundancy verification of key data components; Figure 4 Analysis chart for data storage error correction. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figure 1 This invention provides a data storage method and system for embedded memory chips. The method includes acquiring the raw data stream to be stored through the data acquisition interface of a smart chip, and performing segmentation and marking processing on the raw data stream. The segmentation and marking are based on the timestamp or sequence number of the data stream, adding identifiers to ensure the order and traceability of the data segments. A dynamic decomposition algorithm is used to hierarchically decompose the marked data segments, generating multiple data components. The dynamic decomposition algorithm adaptively adjusts the decomposition granularity according to data characteristics. The information density index of each data component is calculated, and the information density index is used to distinguish the importance of the data. Key data components and non-key data components are selected based on the information density index, and the selection threshold is dynamically set based on historical data or real-time requirements. Key data components are input into the high-speed cache of the smart chip, and non-key data components are input into the conventional storage area of ​​the smart chip. The high-speed cache has faster access speed and higher reliability. Redundancy check operation is performed on the key data components in the high-speed cache to generate check results. Redundancy check is implemented by adding check bits or cyclic redundancy codes. Based on the check results, errors are corrected in the key data components, and the corrected key data components are recombined with the non-key data components to generate an optimized data stream. The recombination process ensures data integrity. The optimized data stream is written to the physical storage medium of the smart chip, and storage location information, including logical address and physical address mapping, is recorded. The storage load status of the smart chip is monitored, reflecting the usage of the storage area. Based on the storage load status, the allocation ratio between the cache and regular storage areas is dynamically adjusted, with the adjustment operation based on a predefined strategy. When a data read request is detected, the corresponding optimized data stream is extracted from the physical storage medium according to the storage location information and restored to the original data stream according to a reverse reassembly process. The reverse reassembly process executes the storage operation steps in reverse.

[0025] Example 1: See Figure 2The implementation of the dynamic decomposition algorithm begins with setting the initial decomposition threshold and the incremental step size. The initial decomposition threshold is a configurable numerical parameter whose size is directly related to the overall size and data type of the data segment to be processed. For example, for sensor data streams with high dynamic range, the initial threshold is set to a lower value to achieve finer granular control, while for relatively stable log data, a higher threshold can be used to improve processing efficiency. The incremental step size defines the magnitude and direction of threshold adjustment during the decomposition process. This parameter is not fixed but is dynamically calibrated according to the real-time characteristics of the data stream. When data characteristics change drastically, the step size value automatically increases to accelerate decomposition adaptability, and conversely, it decreases to maintain stability.

[0026] The characteristic parameters of the raw data stream form the basis for calculating the adaptive adjustment coefficients. These parameters include, but are not limited to, the entropy, peak characteristics, and trend statistics of the data segments. These indicators are calculated in real time using a dedicated coprocessor and input into the coefficient generation algorithm. This algorithm employs a multivariate fitting mechanism to map the characteristic parameters to adjustment coefficients, ensuring that the decomposition process closely matches the inherent patterns of the data. For example, for industrial time-series data with significant periodicity, the algorithm outputs a higher adjustment coefficient, prompting the decomposition to focus more on identifying periodic boundaries. The real-time decomposition parameter generation integrates three elements: the initial threshold, the incremental step size, and the adaptive adjustment coefficient. The generation process uses a weighted fusion strategy, with the adjustment coefficient participating as a weighting factor in the calculation. This ensures that the final applied decomposition parameters retain the initial intended settings while incorporating data-driven intelligent adjustments.

[0027] When performing multi-level decomposition operations using real-time decomposition parameters, the system establishes a hierarchical processing pipeline. The first level of decomposition divides data segments into coarse-grained components at the macro level according to the threshold set by the parameters. Subsequent levels of decomposition refine the data progressively based on the output of the previous level. Each data component generated at each level is assigned a unique level identifier. The hierarchical relationship is maintained through a doubly linked list structure. In addition to storing the actual data content, each data component node also contains pointers to parent and child components. This design allows subsequent reassembly operations to quickly trace the data source. The decomposition operation specifically employs an adaptive wavelet transform algorithm, dynamically selecting the wavelet basis function and the number of decomposition levels based on real-time parameters. For data segments containing abrupt changes, Haar wavelets are automatically used for fast edge capture, while Daubechies wavelets are used for smooth data segments to achieve a higher compression ratio.

[0028] In the information density index calculation stage, time-domain feature extraction focuses on the statistical characteristics and variation patterns of data components. The calculation module simultaneously acquires indicators such as the mean, variance, and autocorrelation function of the components. Among these, the autocorrelation function analysis pays particular attention to time delay characteristics to identify repetitive patterns. Frequency-domain feature extraction transforms the data to the frequency domain using a Fast Fourier Transform (FFT) to analyze its power spectral density distribution characteristics, focusing on capturing the energy concentration of the dominant frequency components. The pre-trained density evaluation model employs a deep neural network architecture. During the training phase, this network uses labeled sample data to learn the nonlinear relationship between time-frequency features and information density. The network contains multiple hidden layers for feature abstraction, and the output layer uses the Sigmoid activation function to generate standardized density values.

[0029] The calculation of time-domain density values ​​pays special attention to the correlation between data volatility and information carrying capacity. By analyzing the statistical characteristics of the differential sequence of the time-domain signal, higher density weights are assigned to segments with severe fluctuations. Frequency-domain density values, on the other hand, focus on evaluating the uniformity of spectral energy distribution; concentrated spectra typically correspond to lower frequency-domain density values. A dynamic weighting mechanism is introduced into the weighted fusion process, with weight allocation adaptively adjusted according to the data application scenario. In real-time data processing scenarios, the time-domain weight can account for over 70%, while in archival storage scenarios, the frequency-domain weight is increased to around 60%. The fusion algorithm uses a weighted geometric average to maintain numerical stability. The final generated information density index is normalized, with its value controlled between 0 and 1, facilitating subsequent unified threshold selection. The closer the index value is to 1, the denser the effective information contained in the data component. The entire implementation process utilizes dedicated hardware accelerators to execute feature extraction and model inference in parallel, with computational latency controlled at the microsecond level. Metadata generated during the decomposition process is stored synchronously with the data components, including hierarchical relationship graphs and density index tables. This metadata plays a crucial navigational role in the subsequent reassembly stage. The dynamic threshold adjustment mechanism continuously monitors the decomposition quality indicators and automatically triggers a parameter recalibration cycle when an uneven distribution of component sizes is detected.

[0030] Example 2: See Figure 3The execution of redundancy checks begins with the accurate identification of the data types of key data components. The system's built-in type identification module scans the metadata identifiers and structural features of the data components to distinguish between different categories such as numeric data, character data, and binary stream data. The verification rules configured for numeric data focus on the reasonableness and continuity of numerical ranges, employing a dual mechanism of parity checking combined with range checking. The rule base pre-sets threshold parameters for different numeric types; for example, for 16-bit integer data, a valid range boundary of -32768 to 32767 is set. The verification rules for character data primarily focus on encoding standardization and character set validity, configuring UTF-8 or ASCII encoding verification rules while checking the legal occurrence of control characters. The verification rules for binary stream data are the most complex, employing a multi-layered verification strategy, including header identifier verification, length field consistency checks, and a combination of cyclic redundancy checks.

[0031] When performing block verification on critical data components according to the verification rules, the block size is determined dynamically. The system calculates the optimal block size based on the total size of the data components and the preset error detection granularity requirements, typically controlling it between 512 bytes and 4KB to balance verification overhead and error location accuracy. Each data block is independently sent to the verification calculation engine, which adopts a hardware-accelerated parallel processing architecture. For numerical data blocks, parity calculation and numerical range scanning are performed synchronously, while character data blocks undergo two verification processes: encoding verification and character sequence analysis. The processing of binary stream data blocks is the most time-consuming, requiring a three-stage pipeline verification: verifying the magic number identifier in the block header, checking the matching degree between the length field and the actual data size, and performing polynomial division to calculate the cyclic redundancy check code.

[0032] Different check codes employ different calculation algorithms during check code generation. Parity check codes calculate a single-bit check value modulo 2 by counting the number of binary "1"s in the data block. Range check codes record the maximum and minimum values ​​in the data block to form interval check identifiers. Cyclic Redundancy Check (CRC) codes are calculated using the standard CRC-32 polynomial algorithm, efficiently computed through shift register hardware, generating a 4-byte check digest value. All generated check codes are indexed and associated with their corresponding data blocks. Index information is recorded in the check log table, including metadata such as data block offset address, check code type, and calculation timestamp. The comparison between the check code and the pre-stored baseline check code is performed in a dedicated verification module. The baseline check code originates from the initial calculation result before data storage and is stored in a protected, read-only storage area. The comparison process uses a block-by-block comparison strategy. The verification module locates the corresponding baseline check code based on the data block index and selects the comparison algorithm according to the check code type. Single-bit parity check codes undergo Boolean equality comparison, interval check codes check the inclusion relationship of numerical ranges, and cyclic redundancy check codes undergo a complete 32-bit bit-by-bit matching. The comparison results are recorded in real time in the verification status register, which includes three status indicators: pass, fail, or suspicious. The suspicious status indicates that the check code partially matches and requires further manual review.

[0033] The verification results are output in a structured report format, containing the verification status of each data block, the error location offset, and error type classification information. Error types are subdivided into three main categories: data corruption, data tampering, and data loss. Each type of error has a corresponding severity rating. This detailed information provides a basis for subsequent error correction decisions. The verification process employs a pipelined design, allowing data component verification and storage operations to overlap. While some data blocks are still undergoing verification calculations, data blocks that have passed verification can be stored ahead of time. This asynchronous processing mechanism significantly improves system throughput. The verification module also has a self-checking function, periodically verifying the correctness of the verification calculation engine using a test mode.

[0034] The dynamic update mechanism of the verification rules allows the system to adjust the verification strength based on historical error statistics. When a certain type of data error occurs frequently, the system will automatically increase the verification level of that type of data, such as upgrading from single parity check to double parity check or increasing the verification frequency. The rule base maintains version control information, and each update records a change log for easy problem tracking. This adaptive capability allows the verification system to continuously optimize as data characteristics change. All metadata generated by verification operations is encrypted and stored to prevent the verification information itself from being tampered with, forming a complete secure verification chain. Boundary cases are specially handled during data block verification. When the size of a data component is not an integer multiple of the block size, the last incomplete block adopts a padding verification strategy. A specific padding pattern is used to pad the block size before calculating the checksum, and the padding information is marked in the metadata for subsequent differentiation. The timing of verification calculations is carefully designed to avoid conflicts with peak storage access periods. The system uses a load monitoring mechanism to select relatively idle periods to concentrate on executing resource-intensive verification operations. Verification tasks with multiple data components use a priority scheduling algorithm, with highly critical data components receiving verification resources first, ensuring optimal allocation of system resources.

[0035] The verification result analysis module performs pattern analysis on failure cases, identifying the distribution patterns of systematic and random errors. These analysis results are fed back to the verification rule configuration module to form a closed-loop optimization. For intermittent errors, error rate trend monitoring is established. When the error rate of a data block exceeds a threshold, data migration and deep inspection operations are automatically triggered. The entire verification system works closely with the storage controller to achieve seamless integration of verification, error correction, and storage. Verification results are directly mapped to the error management unit of the physical storage medium, achieving hardware-level error processing acceleration. The fault-tolerant design of the verification operation considers scenarios where the verification module itself fails, employing a design of redundant verification calculation units mutually verifying each other. When the calculation results of the primary verification unit and the backup unit are inconsistent, a third-party arbitration unit is initiated for final adjudication. All verification calculations are timestamped and sequenced, ensuring the auditability of the verification process. These security measures enable the verification system to meet the stringent requirements of high-reliability application scenarios. The verification log storage adopts a circular buffer design to prevent the log from growing indefinitely and occupying storage space. Important verification events are synchronously recorded in non-volatile memory for long-term storage.

[0036] Example 3: The error correction process relies on the detailed information provided by the verification results. The system parses the error identifier field in the verification result data structure to locate the specific data block where the error occurred and its offset address in the data component. The error identifier field includes the error type code, error location index, and severity score. Error type identification uses a pattern matching algorithm to compare the verification results with predefined patterns in the error feature library. The error feature library stores digital fingerprints of various error types such as bit flips, data loss, and data tampering. Bit flip errors are manifested as a reversal of the logical value of one or more bits. By analyzing the difference patterns of the checksum, the flip position can be accurately located. Data loss errors are shown as a complete mismatch of the checksum and an abnormal data block length. The system will check the inconsistency between the data block header length field and the actual data size. Data tampering errors show the characteristic of a mismatched checksum but an intact data structure, requiring further analysis of the rationality of the data content for judgment.

[0037] The selection of the correction strategy is based on a comprehensive assessment of the error type and severity. The direct replacement strategy is suitable for situations where a complete backup exists and the error range is clearly defined. The system retrieves the original copy of the corresponding data block from the backup storage area, which is located on different physical storage media to reduce the risk of simultaneous failure. The interpolation reconstruction strategy is used for scenarios where the backup is unavailable or the error range is large. This strategy reconstructs the data by analyzing the inherent patterns of valid data surrounding the erroneous data block. The strategy selection algorithm uses a decision tree model to determine whether the erroneous data block is in a continuous data sequence, assess the age of the available backup, and output the optimal correction strategy considering the real-time requirements of the correction operation.

[0038] The direct replacement strategy requires access to the backup management system. Backup data blocks are stored in different physical storage areas. The system locates the backup version of the corresponding data block through a backup index table, which uses a hash table structure for fast lookup. The replacement operation employs an atomic write mechanism to ensure data consistency. First, the backup data block is read into a buffer for integrity verification. Then, a single instruction operation overwrites the erroneous data block. The entire process is protected by a write protection mechanism to prevent data inconsistency caused by accidental interruptions. After the replacement is complete, the system recalculates the checksum of the data block and performs a secondary verification with the original baseline checksum to ensure that the replacement operation completely corrects the error. Data blocks that pass verification are marked as corrected.

[0039] The implementation of interpolation reconstruction strategies is more complex, requiring the selection of different reconstruction algorithms based on data type. For numerical data, time-series-based predictive interpolation is used, constructing a regression model using valid data before and after the error point. Given timestamps and numerical pairs corresponding to n valid data points, the reconstructed value is calculated using an interpolation function. The interpolation function is based on a linear assumption or a higher-order polynomial fit; the specific choice depends on the data characteristics. For binary stream data, a context-based pattern filling algorithm is used, analyzing the pattern characteristics of adjacent data blocks for appropriate filling. During the reconstruction process, the interpolation parameters used and the reference data range are recorded; this metadata is stored in a correction log for subsequent verification.

[0040] During the interpolation reconstruction process, when processing numerical data sampled at equal intervals, the system uses linear interpolation to recover the values ​​of erroneous points. The interpolation formula is expressed as:

[0041] in: Indicates the location index of the error point. This represents the value to be reconstructed. and This represents the index of the nearest valid point before and after the erroneous point. and This represents the numerical value of the corresponding valid point. Location index. It is an integer index in a data sequence, a numerical value. This is a sensor reading or an analog-to-digital conversion value. The interpolation process is performed on a dedicated hardware interpolator, capable of processing high-speed data streams in real time.

[0042] The corrected critical data components need to undergo quality verification before entering the reorganization process. This verification process includes two levels: integrity checks and reasonableness checks. Integrity checks confirm that the size and structure of the data components conform to the expected format, while reasonableness checks verify the valid range of data values ​​through the business rule engine. Only data components that pass verification are marked as ready to enter the reorganization phase. The operation logs generated throughout the correction process are recorded in detail, including error occurrence time, the basis for the correction strategy selection, and performance metrics such as correction operation time. This log data is used for subsequent system optimization and analysis. The data reorganization phase begins with the reading of hierarchical relationships. The system retrieves the hierarchical relationship descriptors generated during the decomposition phase from the metadata storage area. These descriptors use a tree structure to record the parent-child and sibling relationships between various data components. The relationship resolution module reconstructs the complete hierarchical topology graph, identifying the relative positions of critical and non-critical data components, particularly determining which non-critical data components have dependencies on the corrected critical data components. The topology graph traversal algorithm uses a breadth-first strategy to ensure that data components are processed in the correct order, while simultaneously checking the integrity flags of the data components during the traversal.

[0043] The alignment operation needs to consider the temporal characteristics and logical dependencies of data components. The timestamp alignment module adjusts the time base of each data component to eliminate timing misalignments caused by processing delays. Logical alignment focuses on the semantic relationships between data components, ensuring that data blocks belong to the same logical unit by verifying the sequence number and identifier in the data header. Anomalies detected during the alignment process trigger alignment correction subprocesses; for example, when a timestamp jump anomaly is detected in a data component, the system automatically inserts a padding data block to maintain temporal continuity. The alignment operation uses multi-channel parallel processing, with each channel responsible for aligning a subset of data components. The data component merging operation is executed according to the marking order. The marking order information is obtained from the metadata saved in the segment marking processing stage. The system establishes a merging queue and processes data in ascending order of marking sequence number. The merging algorithm selects different merging strategies based on the data type. Streaming data uses sequential concatenation, while structured data uses field-level fusion. Integrity verification is continuously performed during the merging process. After each data segment is merged, the checksum of that segment is immediately calculated and compared with the expected value. The merged optimized data stream is given new header information, recording the data stream version number, generation timestamp, and integrity check code. This metadata plays an important reference role in the subsequent reading and restoration stage.

[0044] The entire error correction and reorganization process manages the execution order through a state machine mechanism. Each step's state transition is subject to strict conditional constraints; subsequent operations are only triggered after the preceding step is successfully completed. Temporary data involved in the process is stored in a protected cache area with an automatic cleanup mechanism to prevent memory leaks. The system also includes a rollback mechanism that automatically reverts to the previous stable state if a correction or reorganization step fails, avoiding inconsistencies in partially corrected data. All operations are performed under transaction protection, ensuring data consistency even in the event of unexpected system interruptions. Resource management during error correction employs a dynamic allocation strategy, allocating computing resources based on error severity and correction complexity. Simple error corrections are handled by low-priority threads, while complex error corrections are handled by dedicated hardware accelerators. The resource allocation algorithm monitors system load and adjusts resource allocation ratios in real time to ensure high-priority tasks receive sufficient computing power. The correction task queue uses a fair scheduling algorithm to avoid starvation. Resource usage is recorded in real-time in the resource log for performance analysis and optimization. Data consistency during the reorganization process is ensured through multiple mechanisms. Before the merge operation, the system verifies the version consistency of all data components to ensure they originate from the same point in time. During the merge process, copy-on-write technology is used to prevent accidental modification of the original data. The merged data stream undergoes overall checksum calculation and comparison with the expected checksum stored in the metadata; any mismatch triggers a re-merging process. A final format verification is performed before the data stream is output to ensure it conforms to the specifications of the original data stream.

[0045] See Figure 4 The top image shows a time-series comparison of sensor data. The dark gray line represents the original acquired data, and the light gray line shows the data after error correction. Black crosses are used to clearly mark the locations of various data errors detected by the system. It can be observed that the corrected data curves are smoother and more continuous, effectively eliminating abnormal fluctuations and missing points present in the original data. The bottom bar chart details the distribution of two main error types: bit flip errors manifest as abnormal changes in the value of a single data point, while data tampering errors are reflected in abnormal offsets of data values. The statistical information box on the right side of the chart shows the execution of the two correction strategies employed by the system: the direct replacement strategy is suitable for simple error repair, while the interpolation reconstruction strategy is used for complex error scenarios. The performance metrics area displays the quantitative evaluation results of the correction process, including key parameters such as error measurement and correction success rate.

[0046] Example 4: The implementation process of monitoring storage load status in an embedded memory chip data storage system begins with the periodic activation of the data acquisition module. This module wakes up and scans the physical storage units of the cache and regular storage areas at fixed time intervals. The real-time occupancy rate is calculated based on the ratio of the number of used storage blocks to the total number of storage blocks. For example, in a cache with 1024 storage blocks, when 300 blocks are detected to be occupied, the real-time occupancy rate is recorded as 29.3%. Access frequency statistics are implemented through hardware counters. Each storage area has an independent read / write operation counter, and the statistical window is set to a configurable time length, such as 100 milliseconds. The counter value is reset to zero at the end of each statistical period and starts accumulating again. The collected raw data is filtered to eliminate the influence of instantaneous fluctuations, and a moving average algorithm is used to smooth the data curve, making the monitoring results more reflective of long-term trends.

[0047] The calculation of storage load balancing integrates information from two dimensions: real-time occupancy and access frequency. The algorithm normalizes the occupancy value to the range of 0-1, while standardizing the access frequency according to the maximum designed throughput. The weighted product of the two standardized values ​​constitutes the load balancing index. The weighting coefficient is dynamically adjusted according to the storage area type. The access frequency weight for cache areas is typically set higher, such as 0.7, while the occupancy weight for regular storage areas may be increased to 0.6. This differentiated weighting strategy more accurately reflects the actual load characteristics of different storage areas. The load balancing threshold is set as a configurable parameter, with a typical value of 0.8. When the calculated load balancing exceeds this threshold, the system triggers an allocation ratio adjustment process. The threshold value can be dynamically configured through the system parameter table to adapt to different application scenarios.

[0048] When an allocation ratio adjustment operation is triggered, the system saves a snapshot of the current storage state, including detailed occupancy maps and access heatmaps for each storage area. This data is used for subsequent analysis of the adjustment effect. Before the adjustment process starts, it checks whether the system is currently in an idle window to avoid large-scale storage reorganization operations during peak read / write periods. Idleness detection is achieved by analyzing the distribution of access request intervals in recent time periods. The adjustment operation adopts a gradual execution strategy: first, new storage area space is allocated, then data is gradually migrated, and finally the original storage area is released. This method ensures that the system can still serve read / write requests normally during the adjustment process. The mapping relationship table is constructed based on statistical analysis of historical operating data. The system collects empirical data on the optimal allocation ratio under different load conditions to form a state-ratio correspondence database. The table structure adopts a multi-dimensional index design, with storage load balancing as the primary key, while also considering access pattern characteristics such as random read / write ratio and sequential access ratio. The table update mechanism supports online learning. When it detects that the actual performance under a certain load condition is better than the current table settings, it automatically adjusts the ratio value in the corresponding entry to continuously optimize the mapping relationship.

[0049] When querying the mapping table, the system uses the current storage load status as the input key. The query algorithm employs a nearest neighbor matching strategy, finding the reference state with the smallest Euclidean distance to the current state in the state space and extracting its corresponding target allocation ratio. The query results undergo a rationality check, verifying that the ratio value is within the system's allowed reasonable range. For example, the cache ratio must not be less than 20% of the total capacity to ensure basic performance. Verified ratio values ​​are sent to the storage management unit for actual adjustment. The re-allocation of storage capacity is achieved through the address remapping function of the storage controller, calculating the capacity difference under the old and new allocation ratios to determine the storage areas that need to be expanded or reduced. Expansion operations allocate additional blocks from the spare storage pool to the target area, while reduction operations mark excess blocks as reclaimable and return them to the spare pool. The address mapping table is synchronously updated to reflect the new storage layout. This process requires pausing all storage access operations for several microseconds to ensure data consistency. After adjustment, the system verifies the effectiveness of the new layout by testing performance through simulated access modes and recording adjustment logs for subsequent analysis.

[0050] Referring to Table 1, consider a video surveillance embedded system. Its storage system includes a high-speed cache for storing keyframes of the real-time video stream and a regular storage area for storing complete video data. One weekday afternoon, the system monitored that the high-speed cache utilization rate consistently remained above 85%, while the regular storage area utilization rate was only 40%. Simultaneously, the access frequency of the high-speed cache reached 90% of the design value, while the access frequency of the regular storage area was only 30%. The calculated load balancing score reached 0.92, significantly exceeding the threshold of 0.8. The system queried the mapping table, which records the optimized allocation ratio under different load conditions.

[0051] Table 1: Mapping Table of Storage Load Status and Allocation Ratio Load balancing range Cache area ratio Regular storage area ratio Applicable scenario description 0.0-0.3 30% 70% Low load period, energy saving priority 0.3-0.7 50% 50% Normal load, balanced configuration 0.7-1.0 70% 30% High load period, performance priority Based on the current balance of 0.92, falling within the 0.7-1.0 range, the system determines the target allocation ratio as 70% for the cache and 30% for the regular storage area. The current actual allocation is 50% for the cache and 50% for the regular storage area, requiring a 20% expansion of the cache. The system allocates additional space from the spare storage pool to the cache and initiates a data migration process, migrating frequently accessed data blocks from the regular storage area to the expanded cache. The migration process uses a background pipeline approach, prioritizing the migration of data blocks with the most recent access frequency. After migration, the address mapping table is updated, and the new storage layout takes effect immediately. The system continuously monitors the adjusted performance metrics, including response latency and throughput, to ensure the adjustments achieve the expected results. The accuracy of the monitoring data acquisition is improved with hardware assistance; the storage controller integrates dedicated monitoring circuitry, capable of recording access timestamps and address information with nanosecond-level precision. The monitoring data is buffered in a dedicated cache and periodically uploaded in batches to the processing unit for analysis. This design reduces interrupt interference to the main processor. The acquisition cycle is dynamically adjusted according to the system load; the acquisition interval is extended under light load to save energy, and shortened under heavy load to improve monitoring granularity.

[0052] The load balancing calculation incorporates a time decay factor, giving higher weight to recent monitoring data than historical data, making the calculation results more reflective of the current system state. The decay factor uses an exponentially weighted moving average algorithm, with newly collected values ​​having twice the weight of older values. This design makes the system sensitive to load changes while maintaining stability. The load balancing threshold setting considers the system's operating mode: in performance-priority mode, the threshold is lowered to 0.7 to prompt early system adjustments, while in energy-saving mode, the threshold is raised to 0.9 to reduce the frequency of adjustments. The maintenance of the mapping table includes periodic calibration and outlier cleanup. The calibration process compares actual operating performance with the table's predicted values; when the deviation consistently exceeds the allowable range, the table is rebuilt. An outlier detection algorithm identifies isolated data points that clearly do not conform to the overall trend. These points may originate from measurement errors or transient interference and are excluded from the training data. Table version management records the content and time of each change, supporting quick rollback to previous versions when the new table performs poorly. Data migration during storage capacity adjustments uses copy-on-write technology. At the start of migration, the source data blocks are marked as read-only; read requests are still served normally, and write requests trigger the data blocks to be copied to the new location, updating the mapping. This method ensures data availability throughout the migration process, avoiding service interruptions. Migration priority is dynamically adjusted based on data access patterns, with frequently accessed data blocks being migrated first, while less frequently accessed data blocks can have their migration delayed or even remain in their original location.

[0053] The effectiveness of the adjustment is evaluated by comparing performance metrics before and after the adjustment, including parameters such as average response time and throughput fluctuation coefficient. The evaluation period covers multiple workload cycles after the adjustment to ensure that the evaluation results are statistically significant. The evaluation data is fed back to the mapping table update algorithm, forming a closed-loop optimization system. Long-term operating data is also used to identify periodic changes in load patterns, such as daily peak hours or seasonal variations. These patterns are used for predictive adjustments, adjusting the allocation ratio in advance before anticipated load changes. The entire monitoring and adjustment system is designed with a fault-tolerant architecture. When the monitoring module fails, it automatically switches to degraded mode and continues to operate using a fixed allocation ratio. System status checkpoints are created before adjustment operations, allowing for a quick rollback to the previous state in case of adjustment failure. All critical operations are logged in detail, including the basis for adjustment decisions, execution steps, and result verification data. These logs are used for problem diagnosis and system optimization.

[0054] Example 5: The parsing operation for the optimized data stream begins with the identification of the data container format in the storage medium. The system reads the header metadata area of ​​the optimized data stream, which stores the data component distribution map and decoding parameter information. The metadata parser verifies the header signature and version compatibility, confirms that the data stream format conforms to the system processing specifications, and extracts the location index table of critical and non-critical data components. The index table is stored in a B+ tree structure, with each leaf node recording the starting offset, length, and type identifier of a data component. The parser obtains the physical addresses of all components by traversing the index tree. Based on the index information, the system initiates a series of storage read requests, reading the data component content in parallel from different locations on the physical storage medium. During the reading process, a buffering mechanism is used to preload adjacent data blocks to improve efficiency.

[0055] Integrity verification of data components is performed immediately after reading. The system allocates a temporary buffer for each data component, with parity protection to prevent memory errors. The parser compares the size of the actually read data component with the length value recorded in the index table. If a size mismatch is found, a reread mechanism is triggered, with a maximum of three rereads. Data components that pass verification are marked as ready and moved to the reverse verification queue for processing. Components that fail verification are isolated and an error log is recorded. Simultaneously, a redundancy recovery process is initiated to retrieve replacement data from the backup storage area.

[0056] The reverse verification operation applies stricter verification standards to critical data components. The verification rules are completely symmetrical to those used in the storage phase, but the execution order is reversed. The system checks the checksum field of the critical data components, recalculates the hash value of the data content using the same hash algorithm, and compares it with the stored checksum. For binary stream data, reverse verification includes cyclic redundancy check (CRC) verification, calculating the CRC32 value of the current data component and comparing it with the original checksum stored in the metadata. Any mismatch found during the verification process triggers a correction procedure. The system automatically retrieves a copy of the data component from the redundant storage node for replacement, and then re-verifies until it passes.

[0057] The reconstruction of hierarchical relationships relies on topological descriptors stored in the metadata. These descriptors use a directed acyclic graph (DAG) structure to record the dependencies between data components. The relationship reconstruction engine traverses the entire topology graph starting from the root node, calculating the position number of each data component in the merging sequence. The merging order of critical and non-critical data components strictly follows the hierarchical relationship, with lower-level components merged first and higher-level components merged later. The merging engine ensures data timing correctness through timestamp synchronization and sequence number alignment. The merging operation adopts a pipeline architecture; while the current data group is being merged, the alignment check for subsequent groups has already begun, improving overall processing efficiency. The process of removing data segment marker information requires careful handling of marker header and tail identifiers. The system identifies specific byte patterns for the start and end markers of each data segment, and these patterns are added during the storage phase to define data segment boundaries. The removal operation uses in-memory editing technology, directly concatenating valid data content in the newly allocated buffer, skipping marker identifier bytes. For special marker types such as compressed or encrypted markers, the system calls the corresponding decompression or decryption modules to process the data content and restore it to its original format. The data segments after removing the markers undergo length verification to ensure they match the length of the original data stream record; any length discrepancy will trigger the exception handling process.

[0058] In this specific example, we consider an industrial sensor data storage scenario. The raw data stream contains time-series data collected by three sensors: temperature, pressure, and vibration. During the storage phase, the system identifies the high-frequency sampled vibration data as the critical data component, and the temperature and pressure data as non-critical data components. When a data read request is received, the system locates the corresponding optimized data stream file from the storage medium; the file size is approximately 2MB. The parser reads the file header metadata and finds that the file contains 3 critical data components and 12 non-critical data components. The critical data components store the raw waveform data from the vibration sensors, while the non-critical data components store statistical summaries of the temperature and pressure data.

[0059] During the reverse verification process, the system performs a strict CRC32 check on the vibration data components and finds a mismatch in the checksum of the second critical data component. The system immediately retrieves a redundant copy of this component from the backup storage area, replaces the corrupted data, and recalculates the checksum to pass verification. During hierarchical relationship reconstruction, metadata records indicate that the vibration data components need to be cross-merged with time-aligned temperature and pressure data. The merging engine interleaves vibration sampling points and environmental parameter points at a 1:10 ratio. In the marker removal phase, the system identifies the 0xAA55 marker at the beginning and end of each data segment; these markers are used to separate data from different time windows during the storage phase. After the removal operation, the data stream is restored to a continuous sensor data sequence, with vibration data sampled at 1000Hz and temperature and pressure data sampled at 100Hz, completely restoring the original format before storage. The entire restoration process takes 15 milliseconds, meeting the system's real-time requirements.

[0060] Error handling during data restoration employs a tiered response mechanism. Minor errors, such as single data block verification failures, are resolved automatically. Moderate errors, such as multiple corrupted related blocks, trigger partial reconstruction. Severe errors, such as metadata corruption, require manual intervention. Error handling logs meticulously record the occurrence time, handling method, and final result of each anomaly. These logs are used for system reliability analysis and to refine the restoration algorithm. After restoration, the data stream undergoes a final integrity check. The system compares the restored data with the data fingerprints of the original stored records to ensure no deviations were introduced during the restoration process. The data fingerprints are calculated using the SHA-256 algorithm. Once the comparison results match, the restored data stream is sent to the output buffer for transmission to the requesting party. The output stage uses streaming transmission, allowing the receiver to process data as it is received, reducing overall latency. Performance optimization of the entire reverse reassembly process considers multiple aspects, including storage access mode optimization, memory management strategy adjustment, and computing resource allocation. The system employs pre-fetching technology to load potentially needed data components in advance, a caching mechanism to retain frequently used metadata to reduce disk access, and parallel computing to accelerate verification and merging operations. The monitoring module tracks resource usage at each stage of the restoration process in real time and dynamically adjusts task scheduling strategies to ensure system responsiveness.

[0061] The reliability of the restoration process is enhanced through multiple protection mechanisms. Key operational steps are guaranteed to be atomic, and recovery is possible from the most recent consistent point after unexpected interruption. Important data structures, such as metadata indexes, have redundant backups and undergo regular consistency checks. The system also achieves version compatibility during the restoration process, capable of handling data streams in older storage formats, ensuring access to historical data even after system upgrades. A resource cleanup mechanism during the restoration process ensures timely release of temporary data; each data component's cache space is released immediately after processing, and intermediate results from the merge operation are immediately cleared after the final data stream is output. Resource usage is monitored in real time, and garbage collection is automatically triggered when memory leaks or resource accumulation are detected. A detailed report is generated after the entire restoration process is completed, including information such as the amount of data processed, time consumed, and error statistics. These reports are used for system performance tuning and capacity planning.

[0062] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data storage method for an embedded memory chip, characterized in that, include: The raw data stream to be stored is acquired through the data acquisition interface of the smart chip, and the raw data stream is segmented and marked. A dynamic decomposition algorithm is used to hierarchically decompose the marked data segments, generating multiple data components. Calculate the information density index for each data component, and then filter out key and non-key data components based on the information density index. Key data components are input into the high-speed cache area of ​​the smart chip, while non-key data components are input into the regular storage area of ​​the smart chip. Perform redundancy check operations on critical data components in the cache and generate check results; Based on the verification results, errors are corrected in the key data components, and the corrected key data components are recombined with non-key data components to generate an optimized data stream. The optimized data stream is written into the physical storage medium of the smart chip, and the storage location information is recorded. Monitor the storage load status of the smart chip and dynamically adjust the allocation ratio between the cache area and the regular storage area according to the storage load status; When a data read request is detected, the corresponding optimized data stream is extracted from the physical storage medium based on the storage location information, and restored to the original data stream according to the reverse reconstruction process.

2. The embedded memory chip data storage method according to claim 1, characterized in that, The dynamic decomposition algorithm is used to perform hierarchical decomposition on the marked data segments, generating multiple data components, including: Set the initial decomposition threshold and increment step size for the dynamic decomposition algorithm; The adaptive adjustment coefficients of the dynamic decomposition algorithm are calculated based on the characteristic parameters of the original data stream. Real-time decomposition parameters for a dynamic decomposition algorithm are generated based on an initial decomposition threshold, an incremental step size, and an adaptive adjustment coefficient. By using real-time decomposition parameters, multi-level decomposition operations are performed on the labeled data segments to generate multiple data components with hierarchical relationships.

3. The embedded memory chip data storage method according to claim 2, characterized in that, The calculation of the information density index for each data component includes: Extract the time-domain and frequency-domain features of each data component; The time-domain and frequency-domain features are input into a pre-trained density evaluation model, which outputs the time-domain density value and frequency-domain density value of each data component. The time-domain density value and the frequency-domain density value are weighted and fused to generate the information density index of the data component.

4. The embedded memory chip data storage method according to claim 3, characterized in that, The redundancy check operation performed on the critical data components in the cache includes: Configure corresponding validation rules based on the data type of the key data components; The key data components are checked in blocks according to the verification rules, and a check code is generated for each data block. The verification code is compared with the pre-stored baseline verification code to generate the verification result.

5. The embedded memory chip data storage method according to claim 4, characterized in that, The error correction of key data components based on the verification results includes: Identify erroneous data blocks and their error types in the verification results; Select the corresponding correction strategy based on the error type, including direct replacement strategy or interpolation reconstruction strategy; The error data block is corrected by applying a correction strategy, and the corrected key data components are generated.

6. The embedded memory chip data storage method according to claim 5, characterized in that, The recombination of the corrected critical data components with the non-critical data components includes: Read the hierarchical relationship between critical data components and non-critical data components; Alignment operations are performed on the corrected critical data components and non-critical data components according to the hierarchical relationship. The aligned data components are merged according to the marked order to generate an optimized data stream.

7. The embedded memory chip data storage method according to claim 6, characterized in that, The monitoring of the storage load status of the smart chip includes: Collect real-time occupancy and access frequency of the cache and regular storage areas; Calculate storage load balancing based on real-time occupancy and access frequency; When the storage load balancing exceeds a preset threshold, an allocation ratio adjustment operation is triggered.

8. The embedded memory chip data storage method according to claim 7, characterized in that, The method of dynamically adjusting the allocation ratio of the cache area and the regular storage area according to the storage load status includes: Establish a mapping table between storage load status and allocation ratio. Query the mapping table based on the current storage load status to obtain the target allocation ratio, and then redistribute the storage capacity of the cache area and the regular storage area according to the target allocation ratio.

9. The embedded memory chip data storage method according to claim 8, characterized in that, The process of restoring the data stream to its original state according to the reverse reconstruction process includes: Extract key and non-key data components from the optimized data stream; Perform reverse verification on key data components to ensure data integrity; Based on the hierarchical relationship, key data components and non-key data components are merged into a labeled data segment; Remove the marker information from the data segments to restore the original data stream.

10. An embedded memory chip data storage system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the embedded memory chip data storage method as described in any one of claims 1 to 9.

Images