Multi-type data storage management method and system

Abstract

The invention relates to the technical field of data storage, and provides a multi-type data storage management method and system, and the method comprises the steps: building a data coding structure of a virtual KEY based on the key attribute of storage data, and generating a corresponding virtual KEY and calling a corresponding interface layer when a data management instruction is received, so as to enable a core logic layer to carry out data analysis and storage management; thus, through a unified interface of the interface layer only used for data forwarding, a set of APIs (vkset and vkget) can support reading and writing of all structural bodies, and interface redundancy is eliminated; multi-storage adaptation is carried out through the storage requirement of the virtual KEY, efficient temporary storage of a memory and persistent storage of a file can be flexibly switched, and different scenes are adapted; according to the method, the data coding structure of the exclusive virtual KEY is designed to realize low-coupling expansion, so that when a structural body / storage medium is newly added, core logic does not need to be modified, only a metadata interface is supplemented, and good expansibility is achieved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method and system for managing multi-type data storage. Background Technology

[0002] In traditional embedded system development, structured data management methods have obvious shortcomings.

[0003] First, the system suffers from severe interface redundancy. For each independently defined structure data type, developers must write dedicated read and write functions, resulting in a large amount of repetitive logic in the code and increasing the total amount of code.

[0004] Secondly, data storage management is chaotic. The data management logic in temporary running memory and non-volatile Flash memory is separated, lacking a unified scheduling and management mechanism. This easily leads to problems such as data inconsistency and operation asynchrony, and the need for additional coordination code increases system instability.

[0005] Finally, its scalability is extremely poor. Whenever a new business structure needs to be added, developers must make in-depth modifications to the core data access process code. Such frequent and invasive modifications to existing code violate the "open / closed principle" and limit the product's ability to iterate and upgrade in order to adapt to flexible and ever-changing user needs. Summary of the Invention

[0006] This invention provides a multi-type data storage management method and system, which solves the technical problems of interface redundancy, storage chaos and poor scalability in existing data storage management solutions.

[0007] To address the above technical problems, this invention provides a multi-type data storage management method, comprising the following steps: A data encoding structure for creating virtual keys is created based on the key attributes of the stored data; Receive data management instructions and generate a corresponding virtual key based on the data encoding structure; A data management request is generated based on the virtual key, and the corresponding interface layer is called to forward it to the core logic layer; The core logic layer obtains the data management request and parses the virtual key in the data management request to extract storage requirements, and then schedules the storage layer to perform corresponding data management operations according to the storage requirements.

[0008] This basic solution creates a virtual key data encoding structure based on the key attributes of stored data. Upon receiving data management instructions, it generates the corresponding virtual key and calls the corresponding interface layer for the core logic layer to perform data analysis and storage management. Thus, through a unified interface layer used only for data forwarding, a single API (vk_set, vk_get) can support reading and writing all structures, eliminating interface redundancy. Multi-storage adaptation is achieved through the storage requirements of the virtual key, allowing flexible switching between efficient in-memory temporary storage and persistent file storage to suit different scenarios. The design of a dedicated virtual key data encoding structure enables loosely coupled expansion, so when adding new structures / storage media, no modification to the core logic is required; only the metadata interface needs to be supplemented, demonstrating excellent scalability.

[0009] In a further implementation, a data encoding structure for a virtual key is created based on the key attributes of the stored data, including: A global index range mapping table is constructed to centrally manage index resources. The global index range mapping table uses data type and submodule as keys and allocates an exclusive, non-overlapping index range. The high 8 bits are allocated to the data type, the middle 8 bits are allocated to the submodule, and 16 bits are allocated to the storage index for functional segmentation, thus constructing a 32-bit data encoding structure for the virtual key.

[0010] This solution designs and constructs a global index range mapping table to centrally manage index resources. This global index range mapping table uses data type and submodule as keys, allocating a unique, non-overlapping index range. This ensures the uniqueness of the virtual key index during both development and runtime, significantly reducing data redundancy. Through composite encoding, key data attributes (type, ownership, and identifier) ​​are integrated into a unique 32-bit data encoding structure called a virtual key, achieving "one key to locate all information." This provides a foundation for subsequent storage strategy matching and data routing, allowing for preliminary identification of data attributes without additional table lookups, and offering strong interpretability. Reserved field space supports the addition of new data types, modules, or index ranges, providing good scalability. The lightweight 32-bit integer structure results in low memory consumption, facilitating transmission and storage.

[0011] In a further implementation, generating a data management request based on the virtual key and calling the corresponding interface layer to forward it to the core logic layer includes: When the data management instruction is detected to contain a data write instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_write function is called to forward the data management request to the core logic layer. When the data management instruction is detected to contain a data read instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_read function is called to forward the data management request to the core logic layer. When the data management instruction is found to contain a data deletion instruction, a data management request is generated based on the virtual key, and the data_delete function in the interface layer is called to forward the data management request to the core logic layer.

[0012] This solution sets up a unified interface and query language interface layer, which directly forwards requests to the core logic layer without performing additional business processing. By using virtual keys, the details of multimodal data storage are hidden from users, so that users are unaware of the existence of multimodal data during operation. Multiple key-value pairs can be written at once without worrying about whether these key-value pairs are on the same data slice, which improves the flexibility of data writing and simplifies the data management process.

[0013] In a further implementation, the virtual key in the data management request is parsed to extract storage requirements, including; The data encoding structure of the virtual key in the data management request is parsed to extract information data of the corresponding data type, sub-module, and storage index; configuration information, including storage requirements, is obtained by matching the information data of the data type with the metadata.

[0014] This solution integrates storage requirements into the data encoding structure of the virtual key. In actual operation, it can efficiently distribute the storage of various heterogeneous data, such as relational data and file data, according to the storage requirements.

[0015] In a further implementation, when it is detected that the data management instruction contains a data write instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, write the data to the storage location corresponding to the memory pool according to the virtual key. If the storage requirement is "file only", a file path is generated based on the virtual key, and then the data is written to the corresponding file. If the storage requirement is "dual storage", then both the memory storage module and the file storage module are invoked simultaneously, and the data is written to the memory pool and the file respectively according to the virtual key.

[0016] In a further implementation, when the data management instruction is identified to contain a data read instruction, the storage layer is scheduled to perform the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, read data from the storage location corresponding to the memory pool into the buffer according to the virtual key. Identify whether the storage requirement is "file only"; if so, read data from the corresponding file into the buffer based on the virtual key. If the storage requirement is identified as "dual storage", then data is first read from the memory pool into the buffer according to the virtual key. If the read fails, data is read from the corresponding file into the buffer.

[0017] In a further implementation, when the data management instruction is identified to contain a data deletion instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, clear the data in the corresponding storage location in the memory pool according to the virtual key. Identify whether the storage requirement is "file only"; if so, locate the corresponding file based on the virtual key and delete it. Identify whether the storage requirement is "dual storage". If so, clear the corresponding data in the memory pool and delete the corresponding file according to the virtual key.

[0018] This solution includes "memory-only", "file-only", and "dual storage" multi-copy modes to ensure data consistency and robustness. Whether it is relational or file-based data, it can be backed up and stored in the system through multiple copies, ensuring data security and reliability, reducing the risk of data management systems, and avoiding the possibility of data loss or damage.

[0019] In a further implementation, security verification is performed based on the information data obtained from the virtual key, including: When the data management instruction is detected to contain a data write / read instruction, the system determines whether the index is within the valid range based on the virtual key and the global index range mapping table. If so, the security check is passed. When the data management instruction is found to contain a data deletion instruction, the system determines whether the user has deletion permission based on the information data parsed from the virtual key. If so, the security check is passed.

[0020] Before performing data writing / reading / deletion, this solution performs security checks by verifying whether the index is within the legal range and whether the user has the necessary deletion permissions by mapping the virtual key and the global index range table. By intercepting illegal index requests, it effectively prevents serious failures such as data corruption, memory leaks, and even system crashes caused by unauthorized access. At the same time, the pre-verification of permissions ensures the operational security of critical data and significantly enhances the stability of data storage management.

[0021] In a further implementation, before performing the write operation, a write check for the existence of the virtual key is added. Based on the storage requirements, it is determined whether the same virtual key already exists in the corresponding storage medium. If it does, it is considered a conflict and an error is returned; otherwise, the write operation is performed.

[0022] This solution adds a key existence check step before scheduling the storage layer to perform write operations in the core logic layer. This effectively reduces redundant operations on the storage medium, improves write efficiency, and saves system storage space.

[0023] The present invention also provides a multi-type data storage management system, which is applied to implement the multi-type data storage management method described above, including an application layer, an interface layer, and a core logic layer connected in sequence; The core logic layer is used to create a data encoding structure for virtual keys based on the key attributes of the stored data. The application layer is used to receive data management instructions, generate a corresponding virtual key based on the data encoding structure, and then generate a data management request based on the virtual key. The interface layer is used to call the corresponding interface function and forward it to the core logic layer; The core logic layer is also used to obtain the data management request and parse the virtual KEY in the data management request to extract storage requirements, and then schedule the storage layer to perform corresponding data management operations according to the storage requirements. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a multi-type data storage management method provided in an embodiment of the present invention; Figure 2 This is a system flowchart for multi-type data storage management provided by an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0026] This invention provides a method for managing multiple types of data storage, such as... Figure 1 As shown, this embodiment includes the following steps: S1. Create a virtual key data encoding structure based on the key attributes of the stored data, including: A global index range mapping table is constructed to centrally manage index resources. The global index range mapping table uses data type and submodule as keys and allocates an exclusive, non-overlapping index range. The high 8 bits are allocated to the data type, the middle 8 bits are allocated to the submodule, and 16 bits are allocated to the storage index for functional segmentation, thus constructing a 32-bit data encoding structure for the virtual key.

[0027] The core idea behind generating virtual keys is to integrate the key attributes of data (type, ownership, and identifier) ​​into a unique identifier through composite encoding.

[0028] I. Core Design Principles of Virtual Key Uniqueness: Each data instance (such as a CAN message or a module setting) corresponds to a unique virtual key to avoid conflicts.

[0029] The uniqueness guarantee mechanism for virtual keys is as follows: 1) The KEY is 32 bits, designed using data structure (data type: high 8 bits, sub-module: middle 8 bits, index 16 bits). 2) Index management: During system development, a unified index allocation and management tool (such as a configuration verification script or management platform) is used to centrally manage index resources. This tool maintains a global index range mapping table, which uses (data type, submodule ID) as the key to allocate an exclusive, non-overlapping index range (e.g., module A is allocated 1-100, and module B is allocated 101-200). This mapping table is also used by the core logic layer at runtime for "index range verification," thus ensuring the uniqueness of the indexes during both development and runtime.

[0030] II. Encoding Structure Design (32-bit Integer) The 32-bit integer is divided into "functional segments," with each segment corresponding to a key attribute of the data. A typical structure is as follows:

[0031] III. Generation Logic and Implementation The generation logic is encapsulated through macro definitions, simplifying application-layer calls and ensuring consistent coding rules. / / 1. Define a data type enumeration (high 8 bits) typedef enum { DATA_TYPE_CONFIG = 0x01, / / Setting item DATA_TYPE_CAN = 0x02, / / CAN data DATA_TYPE_UPDATE = 0x03 / / Upgrade status DataType; / / 2. Define a submodule ID enumeration (8 bits) typedef enum { MODULE_DISPLAY = 0x10, / / Display module MODULE_POWER = 0x20, / / Power system MODULE_FIRMWARE = ​​0x30 / / Firmware module } SubModule; / / 3. Generate macros from virtual keys (combining three pieces of information) typedef uint32_t VirtualKey; #define MAKE_VK(data_type, sub_module, index) \ ((VirtualKey)( \ ((uint32_t)(data_type)<<24) | / / High 8 bits: data type ((uint32_t)(sub_module)<<16) | / / Middle 8 bits: Submodule ID ((uint32_t)(index)&0xFFFF) / / Lower 16 bits: index (ensure it does not exceed the 16-bit range) )) IV. Generating Examples 1) Display module settings: VirtualKey vk=MAKE_VK(DATA_TYPE_CONFIG,MODULE_DISPLAY, 0x0001); The generated virtual key is 0x01 10 0001 (hexadecimal), which represents "Settings - Display Module - First Configuration".

[0032] 2) CAN data VirtualKey vk=MAKE_VK(DATA_TYPE_CAN,MODULE_POWER,0x0002); Generate virtual key: 0x02 20 0002, which means "CAN data - power system - second message".

[0033] 3) Firmware module upgrade status VirtualKey vk=MAKE_VK(DATA_TYPE_UPDATE,MODULE_FIRMWARE, 0x0003); The generated key is 0x03 30 0003, which means "Upgrade Status - Firmware Module - 3rd Record".

[0034] V. Design Description 1) Self-parsing capability: Information can be directly extracted from the key using the GET_DATA_TYPE / GET_SUB_MODULE / GET_INDEX macros without the need for additional storage of mapping relationships. #define GET_DATA_TYPE(vk)((vk>>24)&0xFF) / / Extract data type #define GET_SUB_MODULE(vk)((vk>>16)&0xFF) / / Extract submodule ID #define GET_INDEX(vk)(vk&0xFFFF) / / Extract index 2) Integration with metadata: The core logic layer can directly match the metadata table (g_meta_table) by parsing the "data type" obtained from the virtual key, and obtain the storage strategy (memory / file / format, etc.) for that type of data, thus realizing full automation of the "key parsing → strategy matching → automatic storage".

[0035] 3) Compatibility extension: If you need to add a new data type (such as Bluetooth parameters), you only need to extend the DataType enumeration (such as DATA_TYPE_BLUETOOTH=0x04) and generate a new type of virtual key without modifying the generation logic. It has extremely strong compatibility.

[0036] Note: The structure of the meta element is defined, and the structure defines the core configuration for each data type. Field design must cover requirements such as "storage, format, structure, and extension". typedef struct { / / 1. Data Structure Attributes size_t data_size; / / Size of the data structure (in bytes), used for memory allocation / file I / O. const char* struct_name; / / Name of the data structure (for debugging, such as "DisplayConfig") / / 2. Storage Policy Configuration StorageRequire storage; / / Storage requirements (memory / file / dual storage) FileFormat default_fmt; / / Default file format (only effective when storage contains FILE) const char* file_suffix; / / Stores the file extension (e.g., ".config" or ".can", used to generate filenames) / / 3. Format extension configuration (custom parameters for different formats) const char* fmt_opts; / / Formatting options (such as INI section, JSON indentation) DataMeta; Metadata table (g_meta_table) / / Global metadata table: index = DataType enumeration value static const DataMeta g_meta_table[DATA_TYPE_MAX] = { / / -------------------------- Line 0: DATA_TYPE_CONFIG (Settings) -------------------------- [DATA_TYPE_CONFIG] = { .data_size = sizeof(DisplayConfig), / / Size of the corresponding DisplayConfig structure .struct_name = "DisplayConfig", .storage = STORAGE_REQUIRE_FILE, / / File-only storage (persistence) .default_fmt = FILE_FORMAT_INI, / / Default to INI format .file_suffix = ".config", / / Filename suffix: e.g., 00100001.config .fmt_opts = "section=DISPLAY" / / INI format option: Specifies the partition as [DISPLAY] }, / / -------------------------- Line 1: DATA_TYPE_CAN (CAN data)------------------ [DATA_TYPE_CAN] = { .data_size = sizeof(CanData), .struct_name = "CanData", .storage = STORAGE_REQUIRE_MEM, / / Memory-only storage (temporary real-time data) .default_fmt = FILE_FORMAT_BIN, / / If archiving is required, use binary by default. .file_suffix = ".can", / / Archive file name: e.g., 01200002.can .fmt_opts = NULL / / No additional options are needed for binary format }, / / -------------------------- Line 2: DATA_TYPE_UPDATE (Upgrade Status) -------------- [DATA_TYPE_UPDATE] = { .data_size = sizeof(UpdateStatus), .struct_name = "UpdateStatus", .storage = STORAGE_REQUIRE_BOTH, / / Dual storage: memory and file (real-time progress + persistent results) .default_fmt = FILE_FORMAT_JSON, / / Defaults to JSON format (easy to parse across platforms) .file_suffix = ".update", / / Filename: e.g., 02300003.update .fmt_opts = "indent=2" / / JSON formatting option: indent by 2 spaces }, / / -------------------------- Line 3: DATA_TYPE_DETECTION (APA Target Detection) ----- [DATA_TYPE_APA_DETECTION] = { .data_size = sizeof(APA_DetectionData), / / Size of the data structure (matching the struct above) .struct_name = "APA_DetectionData", / / Structure name (for debugging / logging) .storage = STORAGE_REQUIRE_BOTH, / / Storage requirements: memory + files (dual storage) / / Memory: Real-time cache of the latest detection results for use by the parking algorithm; / / Files: Persistently store historical data for algorithm optimization / fault backtracking .default_fmt = FILE_FORMAT_BIN, / / Default file format: binary .file_suffix = ".apa_detect", / / Stores the file extension: for easy identification of data types / / Example filename: 03 40 0001.apa_detect (when the virtual key is 0x03400001) / / (03 = data type, 40 = submodule ID, 0001 = index) .fmt_opts = "batch=100" / / Format extension option: Batch writing (writes to the file once every 100 accumulated test data entries) This embodiment designs and constructs a global index range mapping table to centrally manage index resources. The global index range mapping table uses data type and submodule as keys and allocates a unique, non-overlapping index range, thus ensuring the uniqueness of the virtual key index during development and runtime, significantly reducing data redundancy. Through composite encoding, key data attributes (type, ownership, and identifier) ​​are integrated into a unique identifier virtual key 32-bit data encoding structure, achieving "one key to locate all information," providing a foundation for subsequent storage strategy matching and data routing. Data attributes can be initially identified without additional table lookups, exhibiting strong interpretability (the virtual key contains extractable structured information, such as data type and module). Reserved field space supports adding new data types, modules, or index ranges, providing good scalability. The lightweight 32-bit integer structure results in low memory consumption, facilitating transmission and storage.

[0037] S2. Receive data management instructions and generate a corresponding virtual KEY based on the data encoding structure; S3. Generate a data management request based on the virtual key, and call the corresponding interface layer to forward it to the core logic layer, including: When the data management instruction is detected to contain a data write instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_write function is called to forward the data management request to the core logic layer. When the data management instruction is detected to contain a data read instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_read function is called to forward the data management request to the core logic layer. When the data management instruction is found to contain a data deletion instruction, a data management request is generated based on the virtual key, and the data_delete function in the interface layer is called to forward the data management request to the core logic layer.

[0038] This embodiment sets up a unified interface and query language interface layer, which directly forwards requests to the core logic layer without performing additional business processing. By using virtual keys, the details of multimodal data storage are shielded from users, so that users are unaware of the existence of multimodal data during operation. Multiple key-value pairs can be written at once without worrying about whether these key-value pairs are on the same data slice, which improves the flexibility of data writing and simplifies the data management process.

[0039] S4. The core logic layer obtains the data management request and parses the virtual KEY in the data management request to extract the storage requirements, and then schedules the storage layer to perform the corresponding data management operation according to the storage requirements.

[0040] In this embodiment, parsing the virtual key in the data management request to extract storage requirements includes: The data encoding structure of the virtual key in the data management request is parsed to extract information data of the corresponding data type, sub-module, and storage index; configuration information, including storage requirements, is obtained by matching the information data of the data type with the metadata.

[0041] This embodiment integrates storage requirements into the data encoding structure of the virtual key. In actual operation, it can achieve efficient distributed storage of various heterogeneous data such as relational data and file data according to storage requirements.

[0042] In this embodiment, when it is detected that the data management instruction contains a data write instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, write the data to the storage location corresponding to the memory pool according to the virtual key. If the storage requirement is "file only", a file path is generated based on the virtual key, and then the data is written to the corresponding file. If the storage requirement is "dual storage", then both the memory storage module and the file storage module are invoked simultaneously, and the data is written to the memory pool and the file respectively according to the virtual key.

[0043] In this embodiment, when it is detected that the data management instruction contains a data read instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, read data from the storage location corresponding to the memory pool into the buffer according to the virtual key. Identify whether the storage requirement is "file only"; if so, read data from the corresponding file into the buffer based on the virtual key. If the storage requirement is identified as "dual storage", then data is first read from the memory pool into the buffer according to the virtual key. If the read fails, data is read from the corresponding file into the buffer.

[0044] In this embodiment, when it is detected that the data management instruction contains a data deletion instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, clear the data in the corresponding storage location in the memory pool according to the virtual key. Identify whether the storage requirement is "file only"; if so, locate the corresponding file based on the virtual key and delete it. Identify whether the storage requirement is "dual storage". If so, clear the corresponding data in the memory pool and delete the corresponding file according to the virtual key.

[0045] This embodiment includes "memory only", "file only" and "dual storage" multi-copy modes to ensure data consistency and robustness. Whether it is relational data or file data, it can be backed up and stored in the system through multiple copies, ensuring data security and reliability, reducing the risk of data management system, and avoiding the possibility of data loss or damage.

[0046] In this embodiment, security verification is performed based on the information data obtained from the virtual key and the parsed data, including: When the data management instruction is detected to contain a data write / read instruction, the system determines whether the index is within the valid range based on the virtual key and the global index range mapping table. If so, the security check is passed. When the data management instruction is found to contain a data deletion instruction, the system determines whether the user has deletion permission based on the information data parsed from the virtual key. If so, the security check is passed.

[0047] Before performing data writing / reading / deletion, this embodiment performs security checks by comparing the virtual key and the global index range mapping table to determine whether the index is within the legal range and whether the user has the necessary deletion permissions. By intercepting illegal index requests, it effectively prevents serious failures such as data corruption, memory leaks, and even system crashes caused by unauthorized access. At the same time, the pre-verification of permissions ensures the operational security of critical data and significantly enhances the stability of data storage management.

[0048] In this embodiment, before performing the write operation, a write check for the existence of the virtual key is added. Based on the storage requirements, it is determined whether the same virtual key already exists in the corresponding storage medium. If it does, it is considered a conflict and an error is returned. If not, the write operation is performed.

[0049] The write check is performed according to the storage policy defined in the metadata: If it is memory-only or file-only, then you only need to check whether the virtual key already exists in the corresponding storage medium; If dual storage is used, the existence of data in memory is checked first (because the access speed is faster). If the virtual key is detected to exist in either of the requested storage media, it is considered a conflict, and the corresponding error is returned immediately and the write process is terminated.

[0050] This embodiment adds a key existence check step before the core logic layer schedules the storage layer to perform a write operation. This can effectively reduce redundant operations on the storage medium, improve write efficiency, and save system storage space.

[0051] This invention creates a virtual key data encoding structure based on the key attributes of stored data. Upon receiving data management instructions, it generates a corresponding virtual key and calls the corresponding interface layer for the core logic layer to perform data analysis and storage management. Thus, through a unified interface layer used only for data forwarding, a single API (vk_set, vk_get) can support reading and writing all structures, eliminating interface redundancy. By adapting to the storage requirements of the virtual key, it can flexibly switch between efficient temporary memory storage and persistent file storage to suit different scenarios. The design of a dedicated virtual key data encoding structure enables loosely coupled expansion, so when adding a new structure / storage medium, there is no need to modify the core logic; only the metadata interface needs to be supplemented, demonstrating excellent scalability.

[0052] Example 2 The reference numerals included in the embodiments of the present invention include: application layer 1, interface layer 2, core logic layer 3, memory storage module 4, and file storage module 5.

[0053] This invention also provides a multi-type data storage management system, applied to implement the multi-type data storage management method described in Embodiment 1 above. See [link to previous document]. Figure 2 It includes an application layer 1, an interface layer 2, and a core logic layer 3 connected in sequence, and also includes a memory storage module 4 and a file storage module 5 connected to the core logic layer 3. The core logic layer 3 is used to create a data encoding structure for virtual keys based on the key attributes of the stored data. The application layer 1 is used to receive data management instructions, generate a corresponding virtual key based on the data encoding structure, and then generate a data management request based on the virtual key. The interface layer 2 is used to call the corresponding interface function and forward it to the core logic layer 3; The core logic layer 3 is also used to obtain the data management request and parse the virtual KEY in the data management request to extract storage requirements, and then schedule the storage layer to perform corresponding data management operations according to the storage requirements.

[0054] The memory storage module 4 includes an array memory pool for fast read and write operations.

[0055] The file storage module 5 can store data in multiple formats, such as xml / json / ini / binary.

[0056] Referring to Table 1, the specific working principle of data writing / reading / deletion is as follows:

[0057] Table 1 I. Data Writing Process 1.1 The application layer initiates a request Application layer 1 constructs data to be written (such as settings, CAN message), generates the corresponding virtual KEY, calls the interface layer data_write function, and passes in the virtual KEY as the data management request and the data buffer pointer; 1.2 Interface Layer Forwarding Requests Interface layer 2 directly forwards the generated data management request to the core logic layer without performing any additional business processing.

[0058] 1.3 Core Logic Layer Parsing and Scheduling 1) Parse the virtual key to extract information such as data type, submodule, and storage index. 2) Match metadata to obtain configuration information such as storage requirements (memory only, file only, dual storage), data size; 3) Perform security checks, including checking data types, module types, index ranges, and data sizes.

[0059] For example, check whether the storage index is within the valid range (i.e., the valid index range). If the verification fails, return an error and terminate the process; if the verification passes, schedule the storage module according to the storage requirements.

[0060] 1.4. Storage layer performs write operations. 1) If the storage requirement is "memory only", the memory storage module 4 writes the data to the corresponding location in the memory pool according to the data size and storage index.

[0061] 2) If the storage requirement is "file only", the file storage module 5 (based on basic formats such as simple binary) generates a file path according to the virtual key and writes the data to the corresponding file.

[0062] Specifically, the file path generation rule is: root directory + category subdirectory + core file name + file extension.

[0063] The names of the category subdirectories are determined by a predefined string mapping table that corresponds to the enumerated values ​​of the data type fields in the virtual key.

[0064] For example, the file path is: / mnt / storage / APA_DETECTION / 03400001.apa_detect.

[0065] Here, / mnt / storage is the root directory, APA_DETECTION is the data type, 03400001 is the core file association key (the virtual key is 0x03400001), and apa_detect is the data purpose (defined according to metadata).

[0066] 3) If the storage requirement is "dual storage", the memory storage module 4 and the file storage module 5 are called simultaneously to write the data to the memory pool and the file respectively.

[0067] 1.5 Result Return The storage layer will return the write result (success / failure) to the application layer 1 via the core logic layer 3 and the interface layer 2.

[0068] II. Data Reading Process 2.1 Application Layer 1 initiates a request Application layer 1 generates a virtual key for reading data, calls the data_read function of interface layer 2, and passes in the virtual key as a data management request and a data buffer pointer.

[0069] 2.2 Interface Layer 2 Forwarding Requests Interface layer 2 forwards data management requests to core logic layer 3.

[0070] 2.3 Core Logic Layer 3: Analysis and Scheduling 1) Parse the virtual key, match the metadata, and obtain configuration information such as storage requirements and data size.

[0071] 2) Perform security checks, such as whether the storage index is valid. If the check fails, return an error; if the check passes, schedule the storage module according to the storage requirements.

[0072] 2.4 Storage layer performs read operations 1) If the storage requirement is "memory only", the memory storage module 4 reads data from the corresponding location in the memory pool into the buffer provided by the application layer 1.

[0073] 2) If the storage requirement is "file only", the file storage module 5 finds the corresponding file based on the virtual key and reads the data into the buffer.

[0074] 3) If the storage requirement is "dual storage", it will read from the memory storage module 4 first (to ensure real-time performance). If there is no valid data in memory (such as after the system restarts), it will read from the file storage module 5.

[0075] 2.5 Result Return The storage layer returns the read data and operation results to the application layer 1 via the core logic layer 3 and the interface layer 2.

[0076] III. Data Deletion Process 3.1 Application Layer 1 initiates a request Application layer 1 generates a virtual key for the data to be deleted, calls the data_delete function of interface layer 2, and passes in the virtual key as the data management request.

[0077] 3.2 Interface Layer 2 Forwarding Requests Interface layer 2 forwards data management requests to core logic layer 3.

[0078] 3.3 Core Logic Layer 3: Analysis and Scheduling 1) Parse the virtual key, match the metadata, and obtain the storage requirements.

[0079] 2) Perform security checks, such as whether there is permission to delete, and return an error if the check fails; if the check passes, schedule the storage module according to storage needs.

[0080] 3.4 Execute deletion in the storage layer 1) If the storage requirement is "memory only", the memory storage module 4 clears the data in the corresponding position of the memory pool.

[0081] 2) If the storage requirement is "file only", the file storage module 5 will find the corresponding file based on the virtual key and delete it.

[0082] 3) If the storage requirement is "dual storage", clear the corresponding data in the memory pool and delete the corresponding files.

[0083] 3.5 Result Return The storage layer returns the deletion result to the application layer 1 via the core logic layer 3 and the interface layer 2.

[0084] The beneficial effects of this embodiment are as follows: (1) Efficient storage and management of heterogeneous data: For relational data, it can be stored in rows to ensure the consistency and high availability of multiple copies of data. At the same time, it is designed to support efficient distributed query processing and meet ACID constraints. For file data, the value can be separated and stored separately to reduce write amplification, save I / O bandwidth and CPU resources, and provide efficient read and write performance.

[0085] (2) Improve data consistency and accuracy: Virtual keys ensure data consistency across tables through referential integrity constraints, preventing data redundancy and inconsistency. For example, in the association between the order and customer tables, virtual keys can ensure that each order corresponds to a valid customer record, avoiding situations where there are no customers, thus improving data accuracy and consistency.

[0086] (3) Reduce data redundancy: In traditional database design, data redundancy may occur if virtual keys are not used. However, when using virtual keys, common information can be stored in one table, and other tables can reference this information through the virtual key, thereby greatly reducing data redundancy. For example, the order table only needs to store key information such as order number and order amount, while the customer's detailed information is stored in the customer table, and the relationship is established through virtual keys.

[0087] (4) Enhanced data scalability and adaptability: The storage management method based on virtual keys has good scalability and can adapt to the ever-increasing data volume and future changes in data processing needs. For example, by using column-based storage expansion technology, storing data in columns can solve the problem of slow data analysis in row-based storage, while also providing the performance advantages of high-speed updates and batch reads.

[0088] (5) Supports flexible reliability configuration: For example, Samsung's KV-SSD can provide three operating modes: standalone, hybrid and custom through the virtual device manager KVMD. Users can choose the appropriate reliability technology according to different workloads and data characteristics to achieve custom reliability configuration by object, achieve data reliability under the same or lower throughput degradation, and KVMD can provide faster rebuild and recovery speed.

[0089] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for managing multi-type data storage, characterized in that, Including the following steps: A data encoding structure for creating virtual keys is created based on the key attributes of the stored data; Receive data management instructions and generate a corresponding virtual key based on the data encoding structure; A data management request is generated based on the virtual key, and the corresponding interface layer is called to forward it to the core logic layer; The core logic layer obtains the data management request and parses the virtual key in the data management request to extract storage requirements, and then schedules the storage layer to perform corresponding data management operations according to the storage requirements.

2. The multi-type data storage management method as described in claim 1, characterized in that, A data encoding structure for creating virtual keys based on the key attributes of the stored data includes: A global index range mapping table is constructed to centrally manage index resources. The global index range mapping table uses data type and submodule as keys and allocates an exclusive, non-overlapping index range. The high 8 bits are allocated to the data type, the middle 8 bits are allocated to the submodule, and 16 bits are allocated to the storage index for functional segmentation, thus constructing a 32-bit data encoding structure for the virtual key.

3. The multi-type data storage management method as described in claim 2, characterized in that, Generating a data management request based on the virtual key and forwarding it to the core logic layer by calling the corresponding interface layer includes: When the data management instruction is detected to contain a data write instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_write function is called to forward the data management request to the core logic layer. When the data management instruction is detected to contain a data read instruction, a data management request is generated based on the virtual key and the data buffer pointer. The interface layer data_read function is called to forward the data management request to the core logic layer. When the data management instruction is found to contain a data deletion instruction, a data management request is generated based on the virtual key, and the data_delete function in the interface layer is called to forward the data management request to the core logic layer.

4. The multi-type data storage management method as described in claim 1, characterized in that, Parse the virtual key in the data management request to extract storage requirements, including; The data encoding structure of the virtual key in the data management request is parsed to extract information data of the corresponding data type, sub-module, and storage index; configuration information, including storage requirements, is obtained by matching the information data of the data type with the metadata.

5. The multi-type data storage management method as described in claim 4, characterized in that, When the data management instruction is detected to contain a data write instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, write the data to the storage location corresponding to the memory pool according to the virtual key. If the storage requirement is "file only", then generate a file path based on the virtual key and write the data to the corresponding file. If the storage requirement is "dual storage", then the memory storage module and the file storage module are called simultaneously, and the data is written to the memory pool and the file respectively according to the virtual key.

6. The multi-type data storage management method as described in claim 5, characterized in that, When the data management instruction is detected to contain a data read instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, read data from the storage location corresponding to the memory pool into the buffer according to the virtual key. Identify whether the storage requirement is "file only"; if so, read data from the corresponding file into the buffer based on the virtual key. If the storage requirement is identified as "dual storage", then data is first read from the memory pool into the buffer according to the virtual key. If the read fails, data is read from the corresponding file into the buffer.

7. The multi-type data storage management method as described in claim 6, characterized in that, When the data management instruction is detected to contain a data deletion instruction, the storage layer is scheduled to execute the corresponding data management operation according to the storage requirements, including: A security check is performed based on the information data obtained from the virtual key. If the check passes, proceed to the next step; otherwise, an error is returned. Identify whether the storage requirement is "memory only"; if so, clear the data in the corresponding storage location in the memory pool according to the virtual key. Identify whether the storage requirement is "file only"; if so, locate the corresponding file based on the virtual key and delete it. Identify whether the storage requirement is "dual storage". If so, clear the corresponding data in the memory pool and delete the corresponding file according to the virtual key.

8. The multi-type data storage management method as described in claim 7, characterized in that, Security verification is performed based on the information data obtained from the virtual key, including: When the data management instruction is detected to contain a data write / read instruction, the system determines whether the index is within the valid range based on the virtual key and the global index range mapping table. If so, the security check is passed. When the data management instruction is found to contain a data deletion instruction, the system determines whether the user has deletion permission based on the information data parsed from the virtual key. If so, the security check is passed.

9. The multi-type data storage management method as described in claim 5, characterized in that: Before performing a write operation, a write check for the existence of the virtual key is added. Based on the storage requirements, it is determined whether the same virtual key already exists in the corresponding storage medium. If it does, it is considered a conflict and an error is returned; otherwise, the write operation is performed.

10. A multi-type data storage management system, applied to implement the multi-type data storage management method as described in any one of claims 1 to 9, characterized in that: This includes the application layer, interface layer, and core logic layer, which are connected sequentially. The core logic layer is used to create a data encoding structure for virtual keys based on the key attributes of the stored data. The application layer is used to receive data management instructions, generate a corresponding virtual key based on the data encoding structure, and then generate a data management request based on the virtual key. The interface layer is used to call the corresponding interface function and forward it to the core logic layer; The core logic layer is also used to obtain the data management request and parse the virtual KEY in the data management request to extract storage requirements, and then schedule the storage layer to perform corresponding data management operations according to the storage requirements.

Images