File system-based key-value data storage method for acquisition terminals

Abstract

This invention discloses a file system-based key-value data storage method for data acquisition terminals. Targeting the usage scenarios of acquisition terminals, a file storage management system is constructed, including a wear-leveling module. Based on the hot data sets of each typical working scenario, the mapping mode between logical and physical addresses is switched. In each mapping mode, the number of erases on storage blocks is recorded, and a pre-defined storage block adjustment strategy is invoked based on the erase count to move the storage blocks. The file storage management system is deployed to each acquisition terminal according to a pre-configured method. This invention reduces disk I / O access, improving system data access performance; it offers a better storage space allocation method with better wear-leveling characteristics for storage media; and it allows for more flexible data item definition. The application layer does not need to concern itself with the physical address allocation of the storage media and can selectively adjust the architecture according to the actual working environment, thus improving the overall performance of the system.

Description

Technical Field

[0001] This invention relates to a data storage method for small data acquisition terminals, and more particularly to a key-value data storage method based on a file system. Background Technology

[0002] In embedded small data acquisition terminals, as the types of business increase, it is inevitable to add data items to store data content when the demand increases. Static key-value databases are gradually unable to meet business needs, while dynamic key-value databases are more in line with the data business development needs of the Internet of Things.

[0003] To address this, engineers developed FlashDB technology. In applications with a limited number of data items, FlashDB is a superior choice as a standard key-value database for small data acquisition terminals.

[0004] However, with the development of the Internet of Things (IoT) industry, the functions of data acquisition devices are becoming increasingly complex, and the number of data items to be stored is also gradually increasing. The performance of MCU embedded products cannot run mature key-value databases like Redis, and technologies such as FlashDB cannot meet performance requirements. Specifically, they suffer from wasted storage space, weak scalability, and susceptibility to fragmentation, making them unable to meet current needs.

[0005] Therefore, a lightweight key-value database storage algorithm based on a file system is needed to meet the application requirements of small embedded devices. Summary of the Invention

[0006] Purpose of the invention: This invention proposes a file system-based key-value data storage method for acquisition terminals to solve the above-mentioned problems existing in the prior art.

[0007] Technical solution: A file system-based key-value data storage method for data acquisition terminals is proposed, including the following steps:

[0008] Step S1: Obtain the pre-stored working scenarios of the acquisition terminal and generate training scenarios using a scenario simulation method; cluster the working scenarios and training scenarios to obtain typical working scenarios of the acquisition terminal;

[0009] Step S2: For each typical work scenario, collect and accumulate the hot data during the statistical reading and writing process to obtain the hot data set of the collection terminal, and sort it in descending order;

[0010] Step S3: For the usage scenarios of the acquisition terminal, a file storage management system is constructed. The file storage management system includes a wear leveling module. The wear leveling module switches the mapping mode between logical address and physical address according to the hot data set of each typical working scenario. In each mapping mode, the number of times the storage block is erased is recorded, and the pre-storage block adjustment strategy is retrieved according to the number of erasures to move the storage block.

[0011] Step S4: Deploy the file storage management system to each acquisition terminal according to the pre-configured method.

[0012] According to one aspect of this application, step S1 is further comprising:

[0013] Step S11: Based on the location information of the acquisition terminal in the working network, form a set of working scenarios based on the location information, and read the data reading status of the acquisition terminal one by one; classify the working data of the acquisition terminal, analyze the reading status of each type of working data, obtain the usage frequency of the data, and sort them in descending order.

[0014] Step S12: Obtain preset scene parameters, generate training scenes using scenario simulation methods, and standardize the data of the working scene and training scene.

[0015] Step S13: Construct working network graphs for working scenarios and training scenarios, generate adjacency matrices, calculate the singular values ​​of each adjacency matrix, and further calculate the root mean square of the singular value sequence of adjacent adjacency matrices and sort them in descending order.

[0016] Step S14: Configure several thresholds for the root mean square (RMS). Based on the relationship between the RMS and the thresholds, classify the working scenarios and training scenarios to obtain several typical working scenarios for the acquisition terminal.

[0017] According to one aspect of this application, step S2 further comprises:

[0018] Step S21: For each typical working scenario, obtain the data access status of each acquisition terminal, analyze whether there are abnormal situations with excessively high or low access frequency, and remove abnormal access data if they exist.

[0019] Step S22: Obtain the hot data of each acquisition terminal during the data reading and writing process and accumulate the data access frequency to obtain the hot data set of each acquisition terminal in this working scenario, and sort them in descending order;

[0020] Step S23: Based on the location of the acquisition terminal in the working network graph, the hot spot data set is divided into a feature hot spot dataset and a non-feature hot spot dataset. Based on the feature hot spot dataset and the location of the acquisition terminal in the working network graph, the working scene of the acquisition terminal is marked to form a scene label set.

[0021] Step S24: Assign weight coefficients to the feature hotspot data in the feature hotspot dataset and the non-feature hotspot data in the non-feature hotspot dataset respectively, and construct a data weight matrix.

[0022] According to one aspect of this application, step S3 further comprises:

[0023] Step S31: Based on the typical working scenarios of the acquisition terminal and the scenario markers of the working scenarios, construct a file storage management system, and switch the mapping mode between logical addresses and physical addresses based on the hot data sets of each typical working scenario; the file storage management system includes at least a wear leveling module.

[0024] The wear balancing module includes:

[0025] The erase counter records the number of erases for each storage block in the acquisition terminal and determines whether the number of erases exceeds the threshold. If it exceeds the threshold, the storage strategy is adjusted.

[0026] The block movement unit adjusts the mapping relationship between the logical address and physical address of storage blocks whose erase count exceeds the threshold, based on the adjusted storage strategy.

[0027] The block recycling unit reclaims the space occupied by data that has been deleted for a predetermined time and invalid data.

[0028] According to one aspect of this application, in step S31, the file storage management system further includes a resilience module, the resilience module including a free list:

[0029] The wear leveling module determines whether a new storage block needs to be allocated. If so, it selects a storage block from the free list that has been used less than a threshold and allocates it to the new data.

[0030] According to one aspect of this application, the idle module further includes an adjustable mapping block:

[0031] Read the data type of the storage block whose erase count exceeds the threshold, and determine whether it is the characteristic hotspot data in this working scenario; if not, find the corresponding working scenario marker based on the characteristic hotspot data, read the mapping mode of the logical address and physical address corresponding to the working scenario marker, and record it as the new working scenario mapping mode; convert the current mapping mode to the new working scenario mapping mode through the adjustable mapping block.

[0032] According to one aspect of this application, the file storage management system further includes:

[0033] The cache unit caches unmodified or synchronized data items in memory when reading data. When the data item is accessed again, the data can be retrieved directly without having to search for storage space again.

[0034] When writing data, the writing method is cached in the write data linked list, and the data item content is directly retrieved from the write cache when the data item is accessed.

[0035] According to one aspect of this application, the file storage management system further includes:

[0036] The lookup unit converts the key value of a data item into a hash value. The data items are then stored in the file in order of hash value size. When searching for a data item in the storage space, the location of the data item is determined by the hash value and a binary search algorithm.

[0037] According to one aspect of this application, step S12 further includes:

[0038] For work and training scenarios, the data is divided into training and validation sets, a neural network is constructed, and the data is standardized based on the output results.

[0039] According to one aspect of this application, step S13 further includes:

[0040] Read the working network diagram of the acquisition terminal and divide it into layers. Assign layer labels to the acquisition terminal based on the layer information. After obtaining the typical working scenarios of the acquisition terminal through clustering, perform secondary division based on the layer labels to form a set of typical working scenarios for each layer.

[0041] Beneficial effects: Compared with existing technologies, this application has the following advantages: higher data retrieval efficiency, reduced disk I / O access, and improved system data access performance; better storage space allocation method with better wear leveling characteristics for storage media; higher degree of freedom in data item definition, allowing application layer development to directly apply for expanded data storage space without worrying about the physical address allocation of the storage media; and the use of sequential search, where binary search offers better time complexity in scenarios with a large number of data items. By statistically analyzing work scenarios and their hot data, a flexible file storage management architecture is designed, which can be selectively adjusted according to the actual work environment, greatly improving the overall performance of the system. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention.

[0043] Figure 2 This is a flowchart of step S1 of the present invention.

[0044] Figure 3 This is a flowchart of step S2 of the present invention.

[0045] Figure 4 This is a flowchart of step S3 of the present invention. Detailed Implementation

[0046] To facilitate the description of the technical features and advantages of this application, the application scenarios and existing solutions of this technology are described first.

[0047] This technology is primarily applied to IoT device terminals. IoT data is characterized by its high dimensionality (i.e., wide rows), but each row may not contain all information, resulting in a sparse matrix. Traditional storage methods waste significant space because they require emptying storage spaces with no data. While this approach isn't particularly disadvantageous for small datasets, the cumulative effect becomes apparent with large datasets. In the era of big data, rapidly changing demands cause database models (table structures) to evolve accordingly. Therefore, databases need to be able to efficiently handle this constant change.

[0048] IoT data acquisition devices have enormous data storage requirements, making data read / write efficiency one of the most crucial factors in data transmission performance. Disk I / O speed is the most critical factor affecting data read / write efficiency, reflecting the data throughput speed between the storage medium and the CPU. The time consumed by the CPU to locate and retrieve data includes the disk read / write process. Therefore, the more disk I / O data exchange, the worse the access performance. In IoT data acquisition devices, key-value databases can be divided into two forms: static key-value databases and dynamic key-value databases.

[0049] Static key-value databases typically use pre-allocation to determine data storage locations. During the code compilation phase, information such as data storage address and data length are allocated to the actual physical address in flash memory. The advantage of this method is extremely low data lookup time; applications only need to read the physical address on the disk to retrieve the data content. However, pre-allocation has a fatal flaw. First, it leads to storage media lifespan issues: because flash memory address space allocation is fixed, the data write location is always the same, and frequent writes affect the storage media's lifespan. Storage media are typically based on sectors; once a sector's lifespan is exhausted, all related data in that area will be corrupted. Second, it leads to wasted storage space: all data has its storage location allocated during the compilation phase, and some space is reserved for future product iterations and expansion needs. This unused reserved space cannot be utilized. Finally, it also suffers from weak expansion capabilities for new data keys: to improve storage space utilization, data keys are usually arranged compactly together. This makes inserting new keys from the middle very inconvenient, as it causes misalignment of the storage address of older data keys, resulting in older data keys not being correctly retrieved after device software updates.

[0050] Dynamic key-value databases typically use a round-robin lookup method to determine the storage location of data values. Taking the FlashDB open-source key-value database as an example, FlashDB allocates a fixed amount of flash memory storage and manages it as a database object. When an application needs to retrieve data, FlashDB searches its cache for historical records to obtain the storage location of the data value. If the search fails, it uses a round-robin approach to query the location of the data key within the storage space. When adding new data, FlashDB appends the new data value to all data keys; if modifying data, it invalidates the old data value, and the data is retrieved from the new storage location on the next read.

[0051] While FlashDB boasts advantages such as excellent wear-leveling performance, strong new data expansion capabilities, and good data caching, it also has some drawbacks. Firstly, it wastes storage space: given a fixed amount of storage space, a certain amount is typically reserved for data refresh to accommodate the addition and modification of data items, resulting in low storage space utilization. For IoT data acquisition devices, different functions or multiple measurement points correspond to multiple data storage spaces, and the consumption of reserved space increases linearly. Secondly, its storage expansion capability is weak: although data items do not have fixed storage locations, data storage space needs to be pre-allocated. The flash memory physical addresses are planned during the code compilation phase. Expanding storage space to increase the number of data items can lead to data misalignment in later storage spaces, causing data loss, or necessitating the addition of new storage space to store new data items. Secondly, fragmentation is a problem: data items inevitably have hot and cold data. Cold data is not frequently modified, while hot data is modified frequently. Since the addition or modification of data items is implemented in a rolling over manner, data items will inevitably become more and more scattered in the storage space over time, resulting in severe fragmentation of available space. If there are many data items, it can easily lead to difficulties in adding and modifying data. Finally, data key polling is inefficient: all data items are distributed discretely in the storage space. The more data items there are and the larger the storage space is, the longer the query time will be. Querying a data item may require reading multiple sectors to locate the data value.

[0052] This application proposes a file system-based key-value data storage method for a data acquisition terminal, comprising the following steps:

[0053] Step S1: Obtain the pre-stored working scenarios of the acquisition terminal and generate training scenarios using a scenario simulation method; cluster the working scenarios and training scenarios to obtain typical working scenarios of the acquisition terminal;

[0054] Step S2: For each typical work scenario, collect and accumulate the hot data during the statistical reading and writing process to obtain the hot data set of the collection terminal, and sort it in descending order;

[0055] Step S3: For the usage scenarios of the acquisition terminal, a file storage management system is constructed. The file storage management system includes a wear leveling module. The wear leveling module switches the mapping mode between logical address and physical address according to the hot data set of each typical working scenario. In each mapping mode, the number of times the storage block is erased is recorded, and the pre-storage block adjustment strategy is retrieved according to the number of erasures to move the storage block.

[0056] Step S4: Deploy the file storage management system to each acquisition terminal according to the pre-configured method.

[0057] This embodiment has the following advantages:

[0058] First, it achieves wear leveling of the storage medium: high-intensity data modifications are distributed as evenly as possible across all sectors, avoiding repeated erasure and writing to one or more fixed sectors, which would shorten the lifespan of the storage medium. Second, it boasts high space utilization, requiring almost no pre-allocated space: all data items within the storage space are compactly arranged in a continuous distribution, and the length of each data item value is variable and customizable. Third, it offers strong data scalability, dynamically expanding data items and storage space: the data storage space exists in the form of files, and the associated data items are stored within those files.

[0059] According to one aspect of this application, step S1 is further comprising:

[0060] Step S11: Based on the location information of the acquisition terminal in the working network, a set of working scenarios is formed, and the data reading status of each acquisition terminal is read one by one; the working data of the acquisition terminal is classified, and the reading status of each type of working data is analyzed to obtain the usage frequency of that type of data, and then sorted in descending order; In this embodiment, based on the location information of the acquisition terminal in the working network, a set of working scenarios is formed, the working data of the acquisition terminal is classified and analyzed to obtain the usage frequency of that type of data, and then sorted in descending order. This allows for a better understanding of the usage of the acquisition terminal and provides a basis for subsequent optimization.

[0061] Step S12: Obtain preset scenario parameters, generate training scenarios using scenario simulation methods, and standardize the data of the working scenario and training scenario; this can better simulate the actual situation and improve prediction accuracy.

[0062] Step S13: Construct working network graphs for working scenarios and training scenarios, generate adjacency matrices, calculate the singular values ​​of each adjacency matrix, and further calculate the root mean square of the singular value sequences of adjacent adjacency matrices and sort them in descending order; this can help to better understand the relationships between different scenarios and find typical scenarios.

[0063] Step S14: Configure several thresholds for the root mean square (RMS). Based on the relationship between the RMS and the thresholds, classify the working scenarios and training scenarios to obtain several typical working scenarios for the acquisition terminal. This allows for better classification, management, and optimization of the acquisition terminal.

[0064] In summary, this embodiment provides a better understanding of the usage of data acquisition terminals, improves prediction accuracy, clarifies the relationships between different scenarios, and identifies typical scenarios, thereby enabling the classification, management, and optimization of data acquisition terminals.

[0065] For example, hotspot data differs across different levels and scenarios, such as homes, factories, and charging stations. Therefore, classifying hotspot data according to usage scenarios allows for a better description of the data and enables the construction of a mapping relationship between logical and physical addresses, thereby achieving better performance in storage wear leveling.

[0066] According to one aspect of this application, step S2 further comprises:

[0067] Step S21: For each typical working scenario, obtain the data access status of each acquisition terminal, analyze whether there are abnormal situations such as excessively high or low access frequency, and remove abnormal access data if they exist; this can better eliminate abnormal data and improve the accuracy of subsequent analysis.

[0068] Step S22: Obtain the hot data of each acquisition terminal during the data reading and writing process and accumulate the data access frequency to obtain the hot data set of each acquisition terminal in this working scenario, and sort it in descending order; this can help to better understand the hot data situation of different acquisition terminals in different scenarios.

[0069] Step S23: Based on the location of the acquisition terminal in the working network diagram, the hot data set is divided into a feature hot data set and a non-feature hot data set. Based on the feature hot data set and the location of the acquisition terminal in the working network diagram, the working scene of the acquisition terminal is marked to form a scene label set; this allows for better weighting of different data and improves the accuracy of subsequent analysis.

[0070] Step S24: Assign weight coefficients to the feature hotspot data in the feature hotspot dataset and the non-feature hotspot data in the non-feature hotspot dataset to construct a data weight matrix. This allows for better weighting of different data and improves the accuracy of subsequent analysis.

[0071] This embodiment allows for better exclusion of abnormal data, understanding of hotspot data from different acquisition terminals in various scenarios, and comprehension of the impact of the location of different acquisition terminals on user behavior in different scenarios. Furthermore, it weights different data, thereby improving the accuracy of subsequent analysis. This provides the most accurate foundation for building a flexible and targeted file storage management architecture.

[0072] According to one aspect of this application, step S3 further comprises:

[0073] Step S31: Based on the typical working scenarios of the acquisition terminal and the scenario markers of the working scenarios, construct a file storage management system. Switch the mapping mode between logical addresses and physical addresses based on the hot data sets of each typical working scenario. The file storage management system includes at least a wear-leveling module. Optimize the performance of the file storage management system and improve data read / write efficiency based on data access conditions under different scenarios.

[0074] The wear balancing module includes:

[0075] The erase counter records the number of erases for each storage block in the acquisition terminal and determines whether the number of erases exceeds the threshold. If it exceeds the threshold, the storage strategy is adjusted to better handle data access anomalies caused by possible wear and tear.

[0076] The block movement unit, based on the adjusted storage strategy, adjusts the mapping relationship between the logical address and physical address of storage blocks whose erase count exceeds the threshold; this can free up space resources and improve flash memory utilization.

[0077] The block recycling unit reclaims the space occupied by data that has been deleted for a predetermined time and invalid data.

[0078] The elastic module includes a free list:

[0079] The wear leveling module determines whether new storage blocks need to be allocated. If so, it selects storage blocks from the free list whose usage count is less than a threshold and allocates them to the new data. This utilizes existing resources and avoids resource waste caused by frequently requesting new resources.

[0080] In this embodiment, the following advantages are provided: First, wear leveling: Storage typically supports a limited number of erase cycles per block, so utilizing the entire storage device is crucial for reliability. In flash memory, due to the limited number of erase cycles per block, uneven wear is prone to occur. To address this issue, the file storage management system employs the following wear leveling strategies: Block erase counter: The file storage management system records the number of times each block is erased and determines the block's usage based on this count. When a block's erase count reaches a certain threshold, the file storage management system marks it as a "hot block" and avoids writing operations to that block as much as possible. Block movement: When a block is marked as a "hot block," the file storage management system attempts to move its data to other unused or less frequently used blocks. This reduces the use of "hot blocks," thereby extending their lifespan. Garbage collection: To avoid wasting flash memory space and fragmentation, the file storage management system performs garbage collection periodically. During garbage collection, the flash memory space occupied by deleted or invalid data is reclaimed and allocated to new data. To improve flash memory utilization and performance, the file storage management system creates a list of free blocks during file system initialization. When a new block needs to be allocated, the system selects an unused or underutilized block from the free list and allocates it to the new data. In summary, by employing wear leveling strategies such as block erase counters, block movement, garbage collection, and free lists, the file storage management system effectively addresses the problem of uneven wear in flash memory and extends its lifespan.

[0081] According to one aspect of this application, the idle module further includes an adjustable mapping block:

[0082] Read the data type of the storage block whose erase count exceeds the threshold, and determine whether it is the characteristic hotspot data in this working scenario; if not, find the corresponding working scenario marker based on the characteristic hotspot data, read the mapping mode of the logical address and physical address corresponding to the working scenario marker, and record it as the new working scenario mapping mode; convert the current mapping mode to the new working scenario mapping mode through the adjustable mapping block.

[0083] The main purpose is to determine whether the storage strategy needs adjustment based on the number of erases of storage blocks in the acquisition terminal, and to find the corresponding working scenario marker based on characteristic hotspot data, and read the mapping mode of the logical address and physical address corresponding to the working scenario marker in order to perform mapping mode conversion. Specifically, the data type of the storage block whose erase count exceeds the threshold can be determined by the data type field recorded in the storage block header information in the file system. If the storage block contains characteristic hotspot data under the working scenario, the mapping mode conversion can be performed directly; otherwise, it is necessary to find the corresponding working scenario marker and read the logical address and physical address mapping mode corresponding to the marker.

[0084] Adjustable mapping blocks can be used to perform mapping mode transitions. An adjustable mapping block is a fixed-size, reusable space used to manage the mapping relationship between logical and physical addresses. During a mapping mode transition, all logical-to-physical address mappings in the current mapping mode block are copied to the new working block, and the current block is released or reinitialized for use in the next mapping mode transition. This embodiment primarily uses elastic mapping modes to adjust storage management strategies, thereby improving wear leveling performance.

[0085] According to one aspect of this application, the file storage management system further includes:

[0086] The caching unit caches unmodified or synchronized data items in memory when reading data. When accessing the same data item again, the data can be retrieved directly without having to search for storage space again. This avoids frequent writing of data to storage space and improves system performance and stability.

[0087] When writing data, the writing method is cached in the write data linked list, and the data item content is directly retrieved from the write cache when the data item is accessed.

[0088] The lookup unit converts the key value of a data item into a hash value. The data items are then stored in the file in order of their hash values. When searching for a data item within the storage space, the hash value and a binary search algorithm are used to locate the data item. This significantly improves search efficiency and reduces system load.

[0089] In this embodiment, the advantages are as follows: Data power loss protection: Data security is ensured when the device experiences a power outage, and a certain data recovery capability is provided; Data cache access: Data is queried synchronously and modified asynchronously, and the latest accessed data items are stored in a cached manner. For frequently accessed and modified hot data, the number of disk I / O accesses is reduced, improving system performance; High data query efficiency: Data keys are represented by hash values ​​as the query ID of data items and are stored sequentially in the storage space. During data query, a search algorithm is used to quickly determine the data storage location, improving system performance.

[0090] Using data items as objects, all data key-value pairs are managed through a linked list, which includes the storage location (file path), the hash value of the data key, the length and content of the data value, etc.

[0091] The accessed data items are divided into two types: read data and write data.

[0092] Reading data: Unmodified or synchronized data items are cached in the device's memory. When the data item is accessed again, it can be retrieved directly without having to search the storage space again.

[0093] Data writing: When an application adds, modifies, or deletes data items, the data is not immediately persisted. Instead, it is cached in a write data list as a write record. When a data item is accessed, its content can be retrieved immediately from the write cache, improving system data access performance. When system data access is idle, the data item will be asynchronously updated to the storage space. If multiple data items in the same storage space exist in the data list, they will also be updated to the storage space together, reducing the number of disk I / O accesses. After the data items are synchronized, they will be automatically converted to read data to meet the application's need for fast reading.

[0094] The total number of cached data objects can be determined based on system configuration requirements. More data cache space can better reduce the number of disk I / O accesses. When accessing a new data item, the oldest cached record is automatically released to make room for the latest data item. The system keeps the most recently accessed data storage space file open to avoid repeatedly opening the file when accessing multiple data items in the same storage space in a sudden rush. The file is only closed when accessing data items in another data storage space or when new data needs to be written and the data storage space file needs to be operated on. This improves data access speed, reduces the number of disk I / O accesses, and improves system performance.

[0095] A hash value is a fixed-length numeric value used to uniquely identify data. Hash values ​​represent large amounts of data in much smaller numerical values, thus they are used in conjunction with digital signatures. Signing a hash value is more efficient than signing a larger value. Hash values ​​are also useful for verifying the integrity of data sent over insecure channels.

[0096] Although hash values ​​have good uniqueness, hash collisions can still occur when calculating data keys for a large number of data items. To resolve this collision, the CRC checksum can be used together as a digital signature for the data key, making the probability of a collision almost zero.

[0097] The data storage space exists in the form of files. The keys of all data items are converted into hash values. The data items are stored in the file in order of hash value size. When searching for a data item in the storage space, the hash value and binary search algorithm are used to quickly locate the data item, improving search efficiency.

[0098] Binary search has a worst-case logarithmic time complexity, requiring O(logn) comparisons. It uses constant space; the space required is the same for input data of any size. Unless the input data is very small, binary search is faster than linear search, but the array must be sorted beforehand. While some specific data structures designed for fast searching are more efficient (such as hash tables), binary search has a wider range of applications.

[0099] According to one aspect of this application, step S12 further includes:

[0100] For work and training scenarios, the data is divided into training and validation sets, a neural network is constructed, and the data is standardized based on the output results.

[0101] The training set is used to build the neural network model, while the validation set is used to test the model's accuracy and generalization ability. When partitioning the dataset, factors such as the number of samples, sample distribution, and feature selection need to be considered to ensure the model has good predictive ability. Then, a deep learning framework (such as TensorFlow, Keras, etc.) can be used to build the neural network model. Appropriate neural network structures and parameter settings are selected according to different work and training scenarios, and the model is trained using the training set. During training, methods such as cross-validation can be used to evaluate model performance, and parameter settings can be adjusted based on the evaluation results. Finally, after obtaining the neural network output, it can be standardized. Standardization ensures that the output results have the same scale and range, facilitating comparison and analysis. Common standardization methods include Z-score standardization and Min-Max standardization.

[0102] According to one aspect of this application, step S13 further includes:

[0103] Read the working network diagram of the acquisition terminal and divide it into layers. Assign layer labels to the acquisition terminal based on the layer information. After obtaining the typical working scenarios of the acquisition terminal through clustering, perform secondary division based on the layer labels to form a set of typical working scenarios for each layer.

[0104] Specifically, nodes in a network graph can be classified according to their position and function within the network, and then divided into different layers. For example, bottom-level nodes (such as household meters and enterprise meters) can be classified as the first layer, intermediate nodes (such as transformers and substations) as the second layer, and top-level nodes (such as area controllers and cloud services) as the third layer.

[0105] After obtaining typical working scenarios of the data acquisition terminals based on clustering algorithms, these scenarios can be further divided based on hierarchical labels. Specifically, each scenario can be assigned to a corresponding hierarchy and set based on the nodes and functional characteristics involved. For example, the first layer may contain multiple sets of typical working scenarios, including household electricity and outdoor charging; the second layer may contain sets of typical working scenarios such as transformers and control stations; and the third layer may contain sets of typical working scenarios such as cloud servers.

[0106] Assign corresponding hierarchical tags to each data acquisition terminal to better understand its data read / write status. For example, home terminal nodes can be tagged according to their hierarchical position and associated with corresponding typical work scenario sets. This allows for easy viewing of the status and performance of each acquisition terminal at different levels, enabling adjustments to the file management system architecture and timely identification and resolution of problems.

[0107] In summary, the present invention has at least the following advantages:

[0108] Power failure protection: It requires robust protection to ensure file system consistency and flush data to the underlying storage.

[0109] Pre-write log: Before writing data, the file storage management system first writes the data to the log area and performs verification and calculation. Only after the verification passes will the data be written to the actual block. This ensures that even if an abnormal situation such as a power outage occurs during the writing process, the data can be recovered by reading the log. Transactions: To ensure the consistency and integrity of the file system, a transaction mechanism is used in the file storage management system. Each transaction includes multiple operation steps and is treated as an atomic operation. When multiple operations are required, the file storage management system encapsulates them into a transaction and ensures that all operations succeed or fail completely during execution. Block erase counter: The file storage management system records the number of times each block is erased and judges the block's usage based on the number. When the number of erases of a block reaches a certain threshold, the file storage management system marks it as a "hot block" and avoids writing operations on that block as much as possible. This reduces the use of "hot blocks" and thus reduces the risk of data loss. Block movement: When a block is marked as a "hot block," the file storage management system attempts to move the data in it to other unused or less used blocks. This reduces the use of "hotspot blocks," thus preventing data loss due to their corruption. In summary, the file storage management system, through mechanisms such as write-before-logging, transactions, block erase counters, and block movement, achieves excellent power-loss resilience. Even in the event of unexpected power outages or other abnormal situations, the file system can quickly recover to a normal state, ensuring data integrity and consistency. It also has a small footprint, and its compact design saves costs.

Claims

1. A file system based key value data storage method for a harvesting terminal, characterized by, The method comprises the following steps: Step S1, obtaining pre-stored working scenarios of the collection terminal, and generating training scenarios by using a scenario simulation method; clustering the working scenarios and the training scenarios to obtain typical working scenarios of the collection terminal; Step S2, for each typical working scenario, counting and accumulating hot data in the data read-write process to obtain a hot data set of the collection terminal, and arranging the hot data in descending order; Step S3, for the use scenario of the collection terminal, a file storage management system is constructed, the file storage management system comprises a wear leveling module, the wear leveling module switches the mapping mode of the logical address and the physical address according to the hot data set of each typical working scenario; and in each mapping mode, the erasing times of the storage block are recorded, and the pre-stored block adjustment strategy is called according to the erasing times to move the storage block; Step S4, deploying the file storage management system to each collection terminal according to a preconfigured method; Wherein, the step S1 is further: Step S11, according to the position information of the collection terminal in the working network, forming a working scenario set based on the position information, and reading the data read situation of the collection terminal one by one; classifying the working data of the collection terminal, and analyzing the read situation of each type of working data to obtain the use frequency of the data of this type, and arranging the use frequency in descending order; Step S12, obtaining pre-set scenario parameters, generating training scenarios by using a scenario simulation method, and standardizing the data of the working scenarios and the training scenarios; Step S13, constructing a working network graph of the working scenarios and the training scenarios, generating an adjacency matrix, calculating the singular values of each adjacency matrix, further calculating the root mean square of the singular value sequence of adjacent adjacency matrices and arranging the root mean square in descending order; Step S14, configuring several threshold values of the root mean square, classifying the working scenarios and the training scenarios according to the relationship between the root mean square and the threshold values to obtain several typical working scenarios of the collection terminal; The step S2 is further: Step S21, for each typical working scenario, obtaining the data access situation of each collection terminal, and analyzing whether there is an abnormal situation of excessively high or low access frequency, if there is, removing the abnormal access data; Step S22, obtaining the hot data of each collection terminal in the data read-write process and accumulating the data access frequency to obtain the hot data set of each collection terminal under this working scenario, and arranging the hot data in descending order; Step S23, based on the positioning of the collection terminal in the working network graph, the hot data set is divided into a characteristic hot data set and a non-characteristic hot data set, based on the characteristic hot data set and the positioning of the collection terminal in the working network graph, the working scenario of the collection terminal is marked to form a scenario marking set; Step S24, respectively assigning weight coefficients to the characteristic hot data in the characteristic hot data set and the non-characteristic hot data in the non-characteristic hot data set to construct a data weight matrix; The step S3 is further: Step S31, according to the typical working scene of the collection terminal, a file storage management system is constructed according to the scene mark of the working scene, and the mapping mode of the logical address and the physical address is switched according to the hot data set of each typical working scene; the file storage management system at least includes a wear leveling module, The wear leveling module includes: an erase counter, which records the number of erasures of each storage block in the collection terminal and judges whether the number of erasures exceeds a threshold value, and if the threshold value is exceeded, the storage strategy is adjusted; a block moving unit, which adjusts the mapping relationship between the logical address and the physical address of the storage block whose number of erasures exceeds the threshold value based on the adjusted storage strategy; a block recycling unit, which recycles the space occupied by the data deleted for a predetermined time and invalid data.

2. The file system based key-value data storage method for a collection terminal according to claim 1, wherein, In step S31, the file storage management system further includes an elastic module, and the elastic module includes an idle list: The wear leveling module judges whether a new storage block needs to be allocated, and if so, selects a storage block with a usage frequency less than a threshold value from the idle list and allocates it to new data.

3. The file system based key-value data storage method for a harvesting terminal according to claim 2, wherein, The idle module further includes an adjustable mapping block: read the data type of the storage block whose number of erasures exceeds the threshold value, and judge whether it is the characteristic hot data under this working scene; if not, find the corresponding working scene mark according to the characteristic hot data, read the mapping mode of the logical address and the physical address corresponding to the working scene mark, and mark it as the new working scene mapping mode; convert the current mapping mode to the new working scene mapping mode through the adjustable mapping block.

4. The file system based key-value data storage method for a harvesting terminal according to any one of claims 1 to 3, characterized in that, The file storage management system further includes: a cache unit, which caches data items that have not been modified or have been synchronized in memory when reading out data, so that when the data items are accessed again, the data can be directly obtained without the need to search the storage space again; when writing data, the writing record mode is cached in the write data linked list, and when the data item is accessed, the data item content is directly obtained from the write cache.

5. The file system based key-value data storage method for a harvesting terminal according to any one of claims 1 to 3, characterized in that, The file storage management system further includes: a search unit, after the key value of the data item is converted into a HASH value, the data item is stored in the file in the order of the HASH value size, and when searching for the data item in the storage space, the position of the data item is located through the HASH value and the binary search algorithm.

6. The file system based key-value data storage method for a collection terminal according to claim 1, wherein, Step S12 further includes: For working scenes and training scenes, they are divided into training sets and validation sets, a neural network is constructed, and the data is standardized according to the output results.

7. The file system based key-value data storage method for a collection terminal according to claim 1, wherein, Step S13 further includes: read the working network diagram of the collection terminal and divide it into levels, assign a level mark to the collection terminal based on the level information, and after obtaining the typical working scenes of the collection terminal through clustering, perform secondary division based on the level mark to form each typical working scene set of each level.

Images