Data storage method and device based on partition identifier mapping, equipment and medium

Abstract

The invention relates to the technical field of distributed storage, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a data storage method, device, equipment and medium based on partition identifier mapping, and the method comprises the following steps: receiving a file and dividing the file into data blocks to generate block identifiers; generating a partition identifier based on the file identifier and the block identifier through hash mapping; creating a partition disk pack mapping table and writing an initial relationship; querying the mapping table to obtain a disk group, and writing the data block into a physical disk to generate a copy; monitoring the health of the disk, keeping the partition identifier and the block identifier unchanged when a fault occurs, and updating the mapping relation to a new disk group; and receiving a read-write request of the target partition identifier, querying the mapping table to obtain the target disk pack, and executing access. According to the method, the partition identification and the block identification are kept unchanged, fast fault tolerance is achieved only by updating the mapping relation, metadata updating expenditure is reduced, bottom layer change is shielded through partition identification routing, and access transparency and high availability are achieved.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a data storage method, apparatus, device, and medium based on partition identifier mapping. Background Technology

[0002] In current mainstream distributed file systems, such as HDFS, Ceph, FastDFS, MinIO, and GlusterFS, the high availability and fault tolerance of the system are usually improved by splitting files into data blocks and generating multiple replicas. Although this approach is widely used in practice, it still has significant shortcomings in large-scale storage scenarios.

[0003] In the fintech sector, a large amount of data, including transaction logs, risk monitoring logs, and compliance audit documents, requires high-concurrency writing and reading in a distributed manner. Current technologies suffer from static binding between each data block and the physical disk. When a disk fails, the system needs to rebuild all data blocks on that disk and update the corresponding mappings. This results in thousands of modifications to metadata storage, creating severe high-concurrency metadata write pressure and impacting the stability and consistency of the financial system. In financial scenarios with extremely high business continuity requirements, such as real-time risk control or clearing systems, clients need to retrieve the latest data block location information, which can easily lead to access interruptions or delays, affecting the real-time nature of transactions and the stable operation of financial compliance.

[0004] In the healthcare sector, data such as imaging data, electronic medical records, and monitoring logs also rely on distributed storage to ensure high availability. Existing technologies, due to the strong binding of mapping relationships to physical disks, trigger large-scale data migration and reconstruction upon disk failure. This places immense pressure on the storage system in a short period, easily causing image access delays or interruptions in diagnostic data. In medical image analysis or remote diagnosis scenarios, medical personnel or medical applications must re-query the storage location of data blocks, making transparent access difficult and negatively impacting the continuity of diagnosis and treatment and the real-time availability of medical data. Summary of the Invention

[0005] The main objective of this invention is to provide a data storage method, apparatus, device, and storage medium based on partition identifier mapping. This invention aims to solve the technical problem that in existing distributed file systems, partition identifiers and physical disks have a static binding relationship, which leads to the need to update a large amount of metadata and trigger a full data reconstruction when a disk fails. This results in huge metadata update overhead and opaque upper-layer access.

[0006] To achieve the above objectives, the present invention provides a data storage method based on partition identifier mapping, comprising: Receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block; A partition identifier is generated based on the file identifier and the block identifier through hash mapping; Create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table; Based on the partition identifier, query the partition disk group mapping table to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple copies; Monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group. When a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access.

[0007] Furthermore, to achieve the above objectives, the present invention provides a data storage device based on partition identifier mapping, comprising: The file segmentation module is used to receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block. A partition identifier generation module is used to generate a partition identifier based on the file identifier and the block identifier through hash mapping processing; The mapping table creation module is used to create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table. The replica writing module is used to query the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple replicas. The fault remapping module is used to monitor the health status of the physical disk. When a disk fault is detected, the partition identifier and the block identifier remain unchanged, and the mapping relationship in the partition disk group mapping table is updated to remap the partition identifier associated with the faulty physical disk to a new available disk group. The read / write access module is used to query the partition disk group mapping table using the target partition identifier when a read / write request containing the target partition identifier is received, obtain the currently mapped disk group as the target disk group, and route the read / write request to the target disk group for data access.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a data storage program based on partition identifier mapping stored in the memory and executable on the processor, wherein when the data storage program based on partition identifier mapping is executed by the processor, it implements the steps of the data storage method based on partition identifier mapping as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data storage program based on partition identifier mapping, wherein the data storage program based on partition identifier mapping, when executed by a processor, implements the steps of the data storage method based on partition identifier mapping as described above.

[0010] Beneficial Effects: This invention relates to the field of distributed storage technology and can be applied to business scenarios such as fintech and healthcare. It discloses a data storage method, apparatus, device, and medium based on partition identifier mapping, comprising: receiving a file to be stored; dividing the file into multiple data blocks based on a file identifier and generating a block identifier for each data block; generating a partition identifier based on the file identifier and block identifier through hash mapping; creating a partition disk group mapping table and writing the initial mapping relationship between partition identifiers and disk groups into the partition disk group mapping table; querying the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and writing the data block content to the physical disks in the disk group to generate multiple copies; monitoring the health status of the physical disks; when a disk failure is detected, keeping the partition identifier and block identifier unchanged, updating the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group; when a read / write request containing a target partition identifier is received, querying the partition disk group mapping table using the target partition identifier to obtain the currently mapped disk group as the target disk group, and routing the read / write request to the target disk group for data access. This invention achieves rapid fault tolerance in disk failure scenarios by keeping the partition identifier and block identifier unchanged and only updating the mapping relationship in the partition disk group mapping table, avoiding large-scale metadata updates and reducing system overhead. At the same time, by performing read and write routing based on the partition identifier, upper-layer access does not need to be aware of changes in the underlying disk, thereby improving the system's high availability and access transparency. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a data storage method based on partition identifier mapping according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the data storage method based on partition identifier mapping according to the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data storage device based on partition identifier mapping of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The data storage method based on partition identifier mapping provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive files to be stored from the client, divide the files into multiple data blocks based on file identifiers, and generate a block identifier for each data block; generate partition identifiers based on the file identifiers and block identifiers through hash mapping; create a partition disk group mapping table and write the initial mapping relationship between partition identifiers and disk groups into the partition disk group mapping table; query the partition disk group mapping table based on the partition identifiers to obtain the currently mapped disk group, write the data block content to the physical disks in the disk group to generate multiple copies; monitor the health status of the physical disks, and when a disk failure is detected, keep the partition identifiers and block identifiers unchanged, update the mapping relationship in the partition disk group mapping table to remap the partition identifiers associated with the failed physical disks to new available disk groups; when a read / write request containing a target partition identifier is received, use the target partition identifier to query the partition disk group mapping table, obtain the currently mapped disk group as the target disk group, and route the read / write request to the target disk group for data access. This invention achieves rapid fault tolerance in disk failure scenarios by keeping partition and block identifiers unchanged and only updating the mapping relationship in the partition disk group mapping table, avoiding large-scale metadata updates and reducing system overhead. Simultaneously, by using partition identifier-based read / write routing, upper-layer access does not need to be aware of underlying disk changes, thereby improving system high availability and access transparency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2This is a flowchart illustrating an embodiment of the data storage method based on partition identifier mapping provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the data storage method based on partition identifier mapping proposed in this invention includes the following steps: S10, Receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block; In this embodiment, the system needs to establish an interaction mechanism with external requests during the process of receiving files to be stored. File reception is typically achieved through standardized storage interfaces, such as network file transfer protocols, RESTful APIs, or client SDKs of distributed storage systems. During transmission, files are assigned file identifiers, which often originate from input by the caller or are generated internally by the storage system. These identifiers can be strings, numeric sequences, or encoded globally unique identifiers. The file identifier not only serves to distinguish the uniqueness of files but also provides a logical entry point for subsequent data block partitioning and mapping.

[0016] After receiving the file, the system needs to parse and segment it based on the file identifier. The reason for dividing it into multiple data blocks is that directly storing large-scale files would lead to uneven load distribution and low read / write efficiency. Therefore, splitting the file into fixed- or variable-sized data blocks achieves parallel writing and load balancing. The size of the data block is generally determined based on file attributes, storage media performance, and system configuration parameters, such as 64MB, 128MB, or larger. During the segmentation process, sequential cutting or boundary alignment is used to divide the entire file into consecutive small units, each of which is a data block.

[0017] Each data block must be assigned a block identifier after partitioning. The block identifier is typically generated by combining the file identifier and the block's sequential number within the file, ensuring uniqueness within the system. For example, the file identifier can be a UUID, and the block sequential number can be an incrementing integer; the combination is then hashed or encoded to obtain the block identifier. The block identifier distinguishes different data blocks and serves as a crucial index for subsequent hash mapping and replica management. The generation process of the block identifier needs to be stable and repeatable so that it can be re-derived using the file identifier and block sequence during data reconstruction or verification.

[0018] During implementation, the system also needs to perform integrity checks when dividing data blocks to ensure that the data in each block has not been truncated or tampered with. Verification can be performed using hash digests or CRC checks to guarantee the correctness of block-level data. After successful verification, the system establishes an association index between file identifiers and block identifiers and writes it to the metadata store to ensure subsequent queries and traceability.

[0019] File partitioning can be implemented in various ways. In one implementation, a fixed-size block strategy is used, where a pre-defined block size parameter is set, and all files are split into uniform sizes. This approach facilitates management and scheduling, making it suitable for high-throughput distributed storage scenarios. In another implementation, the partition size can be dynamically adjusted based on file type or access characteristics. For example, large block partitions can be used for video files to reduce the number of indexes, while small block partitions can be used for log files to improve write concurrency. Alternatively, an adaptive algorithm can be used to dynamically calculate the block size, allowing the system to automatically optimize block granularity under different loads.

[0020] The method for generating block identifiers can also be adjusted depending on the implementation. One approach is to directly concatenate the file identifier with the block order as the identifier. Another approach is to perform hash processing on the concatenation result to generate a short string, thereby reducing storage and transmission overhead. Furthermore, adding timestamps or version numbers can enhance the identifier's traceability capabilities, adapting to version control and historical rollback scenarios.

[0021] This embodiment divides a file into multiple data blocks after receiving it and generates a block identifier for each block. This enables the system to achieve parallel file writing and balanced storage load, preventing single-point disks from becoming bottlenecks. Simultaneously, the introduction of block identifiers provides a precise index for subsequent partition mapping and replica management, ensuring good traceability and recoverability of data in a distributed storage environment. This mechanism effectively reduces the complexity of metadata management and improves the scalability and reliability of large-scale data storage.

[0022] S20, a partition identifier is generated based on the file identifier and the block identifier through hash mapping; In this embodiment, the process of generating partition identifiers through hash mapping based on file identifiers and block identifiers essentially establishes a link between logical file units and physical storage partitions. File identifiers typically originate from globally unique identifiers generated during the file receiving phase, while block identifiers are unique numbers generated for each data block after the file is split into data blocks. The combination of these two forms the mapping input, which is a prerequisite for ensuring that each data block has a unique location attribute throughout the entire storage system.

[0023] Hash mapping relies on predefined hash functions. Hash functions are mathematical functions that convert input data into a fixed-length output, such as MD5, SHA series, or MurmurHash. In distributed storage systems, functions with low computational complexity and uniform distribution are often chosen. Input data consists of file identifiers and block identifiers, which can be concatenated, bitwise encoded, or XORed to form a unified input string, which is then used by the hash function to generate a hash value. The hash value is then processed through modulo operation or range mapping and assigned to a partition identifier space. The partition identifier space is a predefined set of logical numbers used to evenly distribute data blocks across different partitions, avoiding data skew.

[0024] To ensure the validity of the hash mapping, the generated partition identifiers need to undergo validity verification. For example, this involves checking whether the partition identifier falls within the valid number range, whether there are conflicts, and whether it overlaps with existing partition identifiers. After successful verification, a one-to-many index relationship is established between the partition identifier and the file identifier and block identifier, ensuring that the corresponding data block can be quickly located using the partition identifier during subsequent queries and copy writes.

[0025] During implementation, it is also necessary to record the version number of the hash function and the mapping parameters used to ensure compatibility and traceability during system upgrades or algorithm replacements. This process is not only a computational step but also a crucial part of metadata maintenance.

[0026] There are different ways to generate partition identifiers. In one implementation, the file identifier and block identifier are concatenated as strings and directly input into the hash function, suitable for structured data and small to medium-sized files. In another implementation, the file identifier and block identifier can be converted into numerical values ​​respectively, and then combined using bitwise operations and shifting before being input into the hash function to improve the uniformity of the hash value. A double hashing approach can also be used, first hashing the input data once, and then hashing the result a second time with the block number to enhance collision resistance.

[0027] In designing the partition identifier space, a fixed-size space can be used, such as mapping hash values ​​to a number range of 0-N. Alternatively, a consistent hash ring approach can be used to map hash values ​​to a logical ring, improving the scalability of distributed nodes. In scenarios where fault tolerance and migration costs need to be balanced, a virtual node mechanism can be introduced to map hash results to multiple logical virtual nodes, which are then uniformly categorized into physical partitions to achieve load balancing.

[0028] This embodiment generates partition identifiers through hash mapping by combining file identifiers and block identifiers. This enables automatic allocation and balanced distribution of data blocks and partitions in a distributed storage system, effectively avoiding the large-scale metadata update problem caused by static binding. The introduction of partition identifiers simplifies the calculation logic of storage paths, allowing upper-layer access to transparently locate data through partition identifiers, thereby reducing additional metadata query overhead and improving the system's scalability and fault tolerance.

[0029] S30, Create a partition disk group mapping table, and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table; In this embodiment, the process of creating a partition disk group mapping table requires establishing a stable key-value relationship structure. Partition identifiers are derived from file identifiers and block identifiers through hash mapping, possessing global uniqueness and repeatable derivability. A disk group consists of multiple physical disks and has replaceable and expandable attributes. In the mapping table, the partition identifier serves as the key, and the disk group identifier set serves as the value, forming key-value pairs. The mapping table must define data organization methods and constraints, such as partition identifier fields, disk group identifier set fields, version fields, write time fields, and consistency check fields. Partition identifiers are stored in fixed-length numeric encoding to avoid differences caused by different character sets and serialization methods; the disk group identifier set is stored in an ordered list, supporting fast updates and differential replacement. The mapping table can be implemented in memory and persistently. The memory part is used to accelerate queries, while the persistent part ensures long-term data reliability. The initial mapping relationship may originate from a pre-set disk allocation list during deployment, a binding relationship generated by scanning available disks in the system, or an immediate allocation by the scheduler when the partition identifier is first generated. Writing the initial mapping relationship must adhere to atomic operation requirements to avoid inconsistencies caused by partial failures. The typical execution sequence includes: generating candidate entries, checking if partition identifiers are unused, allocating disk groups and establishing mappings, writing to the log, updating in-memory table entries, and completing persistence. To support concurrent scenarios, partition-level locking or lease mechanisms can be introduced during creation. To ensure the evolution of the mapping table across different versions, a schema version field can be added during design, allowing new fields to be added in a compatible manner while ensuring that the reader selects the parsing method based on the version number. To enhance query efficiency, the mapping table can also add inverted indexes, i.e., the relationship from disk groups to partition sets, to quickly locate affected partitions during disk health checks or migration tasks.

[0030] In one implementation, the partition disk group mapping table is implemented as a distributed key-value database, with partition identifiers stored as primary keys and disk group identifier sets stored as values. During system initialization, a preset list is loaded, mapping entries are generated in batches and written to the database, and each entry is written and committed within a transaction to avoid partial failures. A snapshot cache is also maintained in memory for high-frequency queries; when a snapshot expires, it is restored to the latest state by pulling incremental logs. In another implementation, the creation and writing of the mapping table are based on a consistent log commit mechanism. Each write to the mapping relationship is first written to the log, and the state machine is updated only after a majority of replicas confirm, ensuring consistency. In yet another implementation, without a preset list, the scheduler allocates disk groups immediately when partition identifiers are first generated, generates initial mappings and writes them to an in-memory table, and then persists them after batch merging, improving initialization efficiency. For multi-environment scenarios, a strongly consistent log mechanism can be used in online environments to ensure linear access, while a delayed persistence strategy can be used in offline scenarios to improve throughput.

[0031] This embodiment creates a partition disk group mapping table and writes the initial mapping relationship, which can quickly determine the target disk group where data blocks should be stored without relying on a large number of blocks directly bound to disks. This significantly reduces the size of metadata, reduces the update pressure under high concurrency conditions, and provides a stable entry point for subsequent fault recovery and load balancing.

[0032] S40, query the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple copies; In this embodiment, upon receiving a request containing file data, the file must first be bound to a unique identifier. The file identifier originates from the file's metadata, such as the filename, timestamp, storage path, or a globally unique identifier generated by the system. The file identifier ensures global consistency and traceability during subsequent data block processing and mapping. The file is divided into multiple data blocks, each segmented into fixed-size bytes or with a segmentation granularity determined by dynamic load conditions. Each data block requires an appended block identifier during generation. This block identifier can be generated by combining the file identifier and the fragment sequence number, or calculated using an independent hash function, and is used to uniquely identify the correspondence between the data block and the file.

[0033] File identifiers and block identifiers are combined and then processed through hash mapping to generate partition identifiers. Hash mapping typically relies on hash functions, such as CRC32, MD5, or consistent hashing, to evenly distribute the identifier space across multiple partition ranges. The implementation of hash mapping is not only used for load balancing but also for distributing data across different storage nodes, reducing single-point bottlenecks and concentrated collisions. The numerical space design of partition identifiers needs to ensure a low collision probability under different numbers of storage nodes and support dynamic expansion.

[0034] The generated partition identifiers need to be mapped to disk groups within the system. This mapping is stored in a partition-disk-group mapping table, which is generated by the system during initialization and stored in the distributed metadata management module. The initial construction process of the mapping table involves randomly or strategically assigning each partition identifier to a disk group. A disk group can consist of multiple physical disks, each containing attributes such as capacity, health status, and IO rate. Updates to the mapping table must ensure consistency, typically using distributed consistency protocols such as Paxos or Raft to avoid errors or conflicts caused by single-point updates.

[0035] When a partition identifier is queried, the system quickly locates the corresponding disk group in the mapping table. Query methods can employ key-value pair indexes, hash indexes, or tree-structured indexes to improve lookup efficiency. After obtaining the target disk group, data blocks are written to the physical disks within that disk group. The write process supports a multi-replica mechanism, allowing the selection of multiple physical disks within the same disk group and writing the same data, or redundant replication across disk groups. The number of replicas generated depends on the system configuration, such as dual or triple replicas, to balance storage overhead and fault tolerance.

[0036] During the writing process, write strategies and consistency guarantees need to be considered. Writing can be done synchronously, where the operation is confirmed only after all replicas have been written, or asynchronously, where a write completion signal is returned early after some replicas have finished writing. To ensure data security, the written replicas need to be verified, for example, through hash checksums or CRC checks to verify the consistency of the written content.

[0037] In one implementation, data blocks are divided into fixed-size segments, such as 64MB each. Block identifiers are generated using sequential numbering, and these identifiers are combined with file identifiers to form the key value for the input hash function, thus generating partition identifiers. The hash mapping uses a consistent hashing algorithm to allocate partition identifiers to disk groups, enabling the reallocation of only some partitions during expansion, thereby improving scalability.

[0038] In another implementation, the size of the data blocks can be dynamically adjusted based on file type or system load; for example, video files can be segmented with a large granularity, while small text files can be segmented with a small granularity. Block identifiers are calculated using a global hash generator to avoid duplicate identifiers across different files. Partition identifiers are generated using a double hash mapping: the first layer distributes to logical partitions, and the second layer maps to specific disk groups, improving distribution balance.

[0039] In another implementation, the mapping table is stored in a distributed database, and query efficiency is optimized using a B-tree index. A three-replica mechanism is used for data writing: two copies are written to the current disk group, and the third copy is written to a remote disk group, enabling cross-datacenter disaster recovery. Writes are asynchronously acknowledged; the client returns success when two copies are written, and the third copy is completed synchronously in the background to improve write performance.

[0040] This embodiment achieves distributed storage and copy redundancy of data blocks across multiple physical disks by mapping partition identifiers to disk groups, thereby improving data reliability and availability. The introduction of hash mapping ensures load balancing of data at the partition level, reducing disk hotspot issues. The dynamically maintained mapping table can quickly update the mapping relationship when disk groups change or disk failures occur, without changing file identifiers and block identifiers, achieving seamless data migration and reducing the risk of data access interruption.

[0041] S50, monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group. In this embodiment, in a distributed storage environment, real-time monitoring of the physical disk's operating status is required. Health status monitoring can be achieved by periodically collecting disk self-test information, IO latency, error rate, temperature, bad block count, and other metrics. Alternatively, it can utilize the SMART (Self-Monitoring, Analysis, and Reporting Technology) mechanism to obtain hardware-provided status information. The monitoring module typically runs at the kernel level or in a daemon process of the storage node's operating system, ensuring that status information is collected and uploaded to the metadata management component with low latency.

[0042] When the monitoring mechanism detects a disk failure, such as consecutive write failures, IO timeouts, or critical parameters in the SMART report exceeding thresholds, the system triggers a fault handling process. In traditional designs, disk failures often necessitate the regeneration of file or block identifiers and the rewriting of global metadata. However, here, the file and block identifiers remain unchanged to ensure that upper-layer applications or clients can still access data using the original logical identifiers, avoiding additional relocation operations.

[0043] Once a disk failure is confirmed, the partition disk group mapping table needs to be updated. The update process involves removing the partition identifier originally mapped to the failed disk from the mapping table and reselecting an available disk group for that partition identifier. The new disk group can be selected using various strategies, such as selecting the disk group with the lowest current write pressure based on load balancing, or selecting the disk group with the largest remaining capacity based on space availability. The updated mapping relationships are written to the mapping table in a transactional manner to ensure that no brief access inconsistencies occur during the update process.

[0044] After the update is completed, the partition identifier still maintains its logical binding relationship with the original file identifier and block identifier. The client or upper-layer application does not need to be aware that the underlying disk has changed. Access requests can be redirected to the new disk group by querying the updated mapping table, thereby achieving seamless switching of data access.

[0045] In one implementation, health status monitoring is based on hardware self-test information. SMART parameters are read every few minutes and compared with preset thresholds. When the disk temperature is too high or the number of remapped sectors continues to increase, it is immediately marked as a fault and remapping is triggered.

[0046] In another implementation, health status monitoring is based on IO performance metrics, collecting read / write latency and failure rate in real time, and using a sliding window for statistical analysis. When IO latency exceeds historical benchmark values ​​by several times and persists for a period of time, it is automatically identified as a potential failure and the disk is taken offline, while the partition identifier is remapped to another healthy disk group.

[0047] In another implementation, disk group selection is accomplished through a distributed consensus protocol, ensuring that different nodes reach a consensus on the new ownership of partition identifiers when the cluster is large. Mapping table updates are stored in a distributed key-value store, such as Etcd or Consul, to ensure rapid propagation within the cluster and avoid distributed inconsistencies.

[0048] This embodiment monitors the health status of the physical disk and maintains the file and block identifiers unchanged in the event of a failure, updating only the partition disk group mapping table. This reduces large-scale metadata updates and minimizes the impact on system performance. Clients continue to access data using the original logical identifiers, achieving transparent shielding against fault migration and ensuring access continuity and consistency.

[0049] S60, when a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access.

[0050] In this embodiment, in a distributed storage system, data access requests initiated by clients typically include logical identification information, such as a target partition identifier. Upon receiving the request, the access scheduling module first parses the partition identifier and performs a query operation in the metadata storage to obtain the current partition disk group mapping relationship. The mapping table stores the corresponding records between partition identifiers and disk groups, thus allowing for quick location of the disk group to which the partition belongs. Query operations need to support high-concurrency access; a common approach is to use hash indexes or skip list index structures to improve retrieval efficiency.

[0051] The query results return the currently mapped disk group, which may contain multiple physical disks, each holding a copy of the same data blocks. Once the target disk group is determined, the request scheduling mechanism routes read and write requests to one or more of its physical disks. Write requests typically need to write to all replicas simultaneously to ensure consistency and redundancy, while read requests can select the disk replica with the lowest latency or lightest load based on a strategy, thereby improving response speed.

[0052] The routing process involves network addressing and disk node selection. The system can use a consistent hashing algorithm to convert the target partition identifier into the address of the target disk node, or it can locate the target disk group through a centralized routing table or a distributed directory service. Finally, the requested data is transmitted to the disk through the interface layer of the storage node (such as an interface based on gRPC or HTTP API) to complete the read and write access. Throughout the entire process, the client only needs to provide the logical partition identifier and does not need to know the specific disk address, thus decoupling logical access from physical storage.

[0053] In one implementation, the mapping table is stored on a centralized metadata server. When a client request arrives, it accesses the mapping table through a unified query interface to obtain the target disk group information, and then forwards the request to the target disk group.

[0054] In another implementation, the mapping table is maintained by a distributed consistent storage system. Each storage node locally caches a portion of the mapping table data, and client requests can be quickly queried on the local node, with read and write operations routed to the correct target disk group.

[0055] In another implementation, read and write routing employs a load-aware strategy, prioritizing read requests to the replica with the lowest latency, while write requests use a parallel pipeline mechanism to write to multiple replicas simultaneously, thereby reducing overall write time.

[0056] This embodiment uses partition identifier lookup and mapping table to route requests, ensuring that client access always depends on logical identifiers rather than physical locations. This avoids the need for clients to re-obtain data locations after disk remapping, thus achieving transparent shielding against data migration. Read requests can improve performance by selecting the best replica, while write requests ensure replica consistency through parallel operations, thereby enhancing the overall system's high availability and access efficiency.

[0057] This invention relates to the field of distributed storage technology and can be applied to business scenarios such as fintech and healthcare. It discloses a data storage method, apparatus, device, and medium based on partition identifier mapping, comprising: receiving a file to be stored; dividing the file into multiple data blocks based on a file identifier and generating a block identifier for each data block; generating a partition identifier based on the file identifier and block identifier through hash mapping; creating a partition disk group mapping table and writing the initial mapping relationship between partition identifiers and disk groups into the partition disk group mapping table; querying the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and writing the data block content to the physical disks in the disk group to generate multiple copies; monitoring the health status of the physical disks, and when a disk failure is detected, keeping the partition identifier and block identifier unchanged, updating the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group; when a read / write request containing a target partition identifier is received, querying the partition disk group mapping table using the target partition identifier to obtain the currently mapped disk group as the target disk group, and routing the read / write request to the target disk group for data access. This invention achieves rapid fault tolerance in disk failure scenarios by keeping the partition identifier and block identifier unchanged and only updating the mapping relationship in the partition disk group mapping table, avoiding large-scale metadata updates and reducing system overhead. At the same time, by performing read and write routing based on the partition identifier, upper-layer access does not need to be aware of changes in the underlying disk, thereby improving the system's high availability and access transparency.

[0058] In one embodiment, step S10 above includes: S101, Receive a file storage request containing the file to be stored and its corresponding file identifier; S102, parse the file attribute information in the file storage request to obtain the file size and file type characteristics; S103, determine the data block partitioning strategy parameters based on the file size and the file type characteristics; S104, Determine the fixed size parameters of the data block according to the partitioning strategy parameters; S105, the file to be stored is sequentially divided into multiple consecutive data blocks according to the fixed size parameter; S106, Verify the integrity of each data block and whether its size meets the fixed size parameter requirements; S107, Generate a globally unique block identifier for each verified data block; S108, Establish an index mapping the file identifier to all corresponding block identifiers; S109, Write the mapping relationship index into the metadata storage system; S110 records metadata information including partition timestamps and partition strategy versions.

[0059] In this embodiment, when receiving a file storage request containing the file to be stored and its identifier, the input payload must include two types of information: a binary content stream and a file identifier. The file identifier can be a globally unique identifier, a path signature, or an encoded business key passed in by the caller. To ensure the determinability and traceability of the request, the receiving stage verifies the file identifier format, uniqueness, and consistency with the content digest. Traffic shaping and rate limiting are performed on the receiving channel to prevent large files from filling the transmission window and affecting subsequent processing. After the file arrives, file attribute information is extracted. The file size is derived from the transmission header or the cumulative number of bytes counted while reading. File type characteristics can be given by the extension, media header, magic number feature, MIME type, or caller tag. When multiple sources conflict, the source with higher priority is used, and the determination criteria are recorded, forming a stable attribute snapshot for subsequent policy calculation and auditing.

[0060] The partitioning strategy parameters take file size and file type characteristics as input, generating control variables such as block granularity and boundary alignment. To balance throughput and indexing cost, the strategy calculation should output upper and lower limits and priority granularity of block granularity, and provide the tail block processing principle (truncated retention or padded retention). When the file type characteristics indicate a high proportion of sequential reads, the strategy can favor larger granularity to reduce the index size; when random access accounts for a high proportion or is subsequently involved in analysis tasks, the strategy can favor smaller granularity to improve concurrency. After obtaining fixed-size parameters based on the strategy calculation, the file content stream is sequentially sliced ​​using a sliding window, with the window step size equal to the fixed-size parameter. During the slicing process, alignment is made to content boundaries or encoded frame boundaries to reduce cross-block parsing costs; when the end is less than a full window, the tail block processing principle is applied, and the actual effective length is recorded in the block element.

[0061] Each generated data block must undergo integrity and size verification. Integrity verification calculates both a quick digest and a strong verification digest for the block content, allowing for cost selection in scenarios involving high-frequency verification and dispute review. Size verification compares the effective block length with fixed size parameters to ensure that all blocks except the last block meet the fixed granularity requirements. Blocks that fail verification are marked as invalid and trigger a re-slicing or retransmission process. The reason for failure is written to a fault record and bound to the file identifier and sequence number for easy location during subsequent retry.

[0062] Verified data blocks require the generation of globally unique block identifiers. These identifiers are constructed using a repeatable derivation method, such as combining a file identifier and block sequence number with hash compression. This ensures uniqueness while reducing storage and transmission overhead. The encoding scheme specifies the character set, delimiter rules, and version bit to prevent ambiguity between different implementations. To support concurrent access and idempotent retries, the block identifier generation function remains deterministic; repeated submissions will not generate different identifiers, avoiding index forks. Regarding the association between block identifiers and data content, both a content digest and the generated version are stored within the block element, ensuring that subsequent verification and reconstruction processes can trace back to the same source.

[0063] When establishing an index mapping file identifiers to all block identifiers, the index entries must maintain a stable and ordered block order to support range queries, range reads, and parallel scheduling. The index structure can employ a two-layer organization: a sparse directory and dense details. The sparse directory records anchor blocks with a fixed span, while the dense details store the complete sequence of consecutive blocks, enabling fast location and segment-by-segment scanning for large files. To improve multi-threaded database insertion efficiency, index construction is divided into two phases: in-memory construction and persistent commit. The in-memory phase uses lock-free or fine-grained lock-based ordered containers, while the commit phase writes to the metadata storage system in a transactional manner, including a pre-write log, commit sequence number, and version flag, ensuring that after crash recovery, the index can be replayed to a consistent state using the commit sequence number.

[0064] When writing mapping indexes to the metadata storage system, consistent visibility of a single file must be guaranteed. This is achieved by assigning a separate commit sequence number range to the file. After all index shards have been written, a commit point is published. Readers only consider the file's index complete and usable after seeing the commit point; intermediate states before reaching the commit point are not visible to the outside world, preventing readers from perceiving incomplete indexes. The metadata storage system should support high-concurrency writes, strong single-key consistency reads, and version-based snapshot reads to meet subsequent remapping and auditing scenarios.

[0065] Recording timestamps and strategy versions is a requirement for auditing and reproducibility. Timestamps use a unified time source combined with a monotonically increasing logical clock to avoid sequence reversal caused by cross-node clock drift. Strategy versions record the version identifier of the strategy calculation logic and parameter templates, along with fixed-size parameters, tail block processing principles, and block checksum algorithms, ensuring that the same set of parameters can be reused to reproduce the same results during subsequent reconstruction, reconciliation, and cross-environment migration. To reduce record bloat, common, invariant parameters can be stored in the strategy version template record, with only references stored in the index entries.

[0066] This embodiment generates fixed-size parameters through strategy calculation driven by file identifiers and file attribute information. With the cooperation of sequential cutting, strict verification, and deterministic block identifier generation, it completes block-level organization and index persistence, forming a stable and consistent file identifier-block identifier mapping index. This enables large-volume data to be accessed and written in parallel, providing a standardized entry point for subsequent partition identifier-based mapping and routing. The pre-write log and commit point mechanism reduce crash recovery costs and the risk of inconsistency at the read end. The strategy version and timestamp recording ensure that the partitioning process is traceable and reproducible, thereby maintaining high-throughput data entry while laying a reliable block-level foundation for subsequent fault-tolerant remapping and transparent access.

[0067] In one embodiment, step S20 above includes: S201, Obtain the numerical code of the file identifier and the sequential number of the block identifier; S202, combine the numerical code of the file identifier and the sequential number of the block identifier to form input data; S203, The input data is hashed using a pre-configured hash function; S204, Map the hash processing result to a predefined partition identifier space, generate a unique partition identifier and verify the validity of the partition identifier in the partition identifier space; S205 establishes an association between the verified valid partition identifier and the corresponding file identifier and block identifier; S206 records the hash function version and mapping parameters used in the partition identifier generation process.

[0068] In this embodiment, when obtaining the digital encoding of the file identifier, a unified encoding protocol is first established to ensure that the same business object generates a consistent binary representation across different access terminals and at different times. The file identifier originates from a unique identifier bound in the preceding steps and can be provided by the caller or generated by the system. The digital encoding is achieved through character normalization, character set unification, case and whitespace handling, time zone and time format normalization, and deterministic serialization, ensuring that the same byte sequence is obtained in any retries and concurrent scenarios. To isolate cross-environment conflicts, a namespace prefix or namespace number can be introduced and incorporated into the encoding header. The namespace list is issued by the configuration center and is version-managed.

[0069] The sequential numbering of block identifiers originates from the block order generated during file partitioning, employing a monotonically increasing numbering system. The numbering is determined in-place during the block partitioning process and participates in subsequent mapping along with the file identifier. To mitigate contention caused by multi-threaded database loading and breakpoint resume uploads, an independent counter is established for each file, with an idempotent generation function at the underlying level ensuring that repeated submissions do not alter the numbering. The binding between the number and block content is recorded through block metadata, including the content digest algorithm fingerprint and the strategy version used during generation, facilitating subsequent verification.

[0070] When combining the numerical encoding of file identifiers with the sequential numbering of block identifiers into input data, it is necessary to avoid concatenation ambiguity and unclear field boundaries. The combination process uses a length prefix or variable-length integer field to label each component, and specifies end-order, alignment, and padding rules. To improve the entropy of the hash input, environment salts or namespace salts can also be mixed in. These salts are distributed by version by the key management component and indexed in the meta-record. The combined input data is presented in an immutable buffer, supporting both pipelining and one-time hashing methods.

[0071] Pre-configured hash functions refer to a family of hash functions that have been evaluated, selected, and included in version control before the system goes live. Selection prioritizes characteristics such as uniform output distribution, low collision rate, simple implementation, and scalable throughput. The computation interface exposes adjustable parameters such as seed, round, and block size, and the parameter set is managed by a mapping parameter template. To ensure cross-platform consistency, the hash library version and compilation options are included in a version list, which corresponds one-to-one with the service image to prevent output differences during upgrades.

[0072] When performing hashing on the input data, the input buffer is first read in blocks and mixed at the bit or byte level. The initial distribution is diffused internally through multiple rounds of mixing and carry perturbation. To reduce edge effects caused by long sequence concatenation, reversible encoding or bit-width rearrangement can be applied to the field order. The hash result is output as a fixed-length digest, and a short index is generated for fast filtering and cache keys.

[0073] When mapping the hashing results to a predefined partition identifier space, a consistent mapping from the digest domain to the partitions needs to be established. The partition identifier space consists of a set of logical numbers defined during the deployment phase; it can be a consecutive numbered field or a logical ring structure. The mapping algorithm can employ integer mapping, modulo mapping, consistent hashing mapping, or distributed mapping with virtual nodes. To suppress data skew and scaling disturbances, a weight table and virtual node multiplier control are introduced during the mapping process, ensuring that different disk groups are allocated partition shares according to their capabilities. During scaling up or down, only a small number of partitions near the mapping boundary are affected, and the already generated partition identifiers remain stable.

[0074] After generating a unique partition identifier, validity verification must be performed. Verification includes number range checking, conflict detection, reserved segment filtering, and quota constraint checks. Conflict detection is achieved through atomic comparison and swapping or distributed leases to prevent the creation of the same partition identifier; reserved segments are marked by governance policies, such as segments in migration freeze or offline windows not accepting new reservations; quota checks ensure that the number of partitions mapped to a single disk group does not exceed the capacity and throughput limits. If any verification fails, a rollback strategy is used to recalculate using an alternative salt value or a secondary perturbation function, and retries are completed within a finite number of attempts.

[0075] When establishing associations between verified partition identifiers and their corresponding file and block identifiers, two types of records are generated: forward association and reverse association. Forward association is used to locate the file and block set from the partition identifier, while reverse association is used to locate the partition identifier from the file and block. Both types of records include the generation time, hash function version, and mapping parameter version. Record writing follows the single transaction commit and pre-write log order, with commit numbers monotonically increasing. The read side provides consistent reading based on the largest visible commit point. To support high-concurrency queries, forward associations are cached in a read-only cache and sharded using the key space.

[0076] Recording hash function versions and mapping parameters is used for traceability and canary-scale evolution. Version records include not only the function name and version number, but also the seed, round, virtual node multiplier, weight table snapshot, and namespace salt index. Version metadata is stored in a parameter repository and associated with the mapping record via a version key, allowing for reproducibility of the output during playback or auditing. During canary-scale switching, the same input data is computed in parallel under both the old and new versions; output differences are entered into a reconciliation queue, and the scale is gradually increased using a threshold strategy until a complete switch is achieved. These steps constitute a closed loop from input normalization, combined encoding, hash transformation, spatial mapping, validity verification, bidirectional association to version records, ensuring that determinism, uniformity, scalability, and reproducibility are simultaneously achieved within the same mechanism.

[0077] This embodiment uses file identifiers and block identifiers as stable inputs and performs controlled hash mapping and partition space projection to form a definite and unique partition identifier. Reading and writing can be directly based on the partition identifier without maintaining a huge block-to-disk direct relationship. Validity verification and placeholder mechanisms avoid concurrent conflicts and hot spot skew. Virtual nodes and weight tables suppress expansion disturbances. Version and parameter records support algorithm iteration and replay traceability without changing the upper-layer identifier. This reduces the pressure of metadata updates, smoothly expands the storage scale, and maintains the stability and verifiability of the access path in fault and change scenarios.

[0078] In one embodiment, step S30 above includes: S301, Initialize the data structure of the partition disk group mapping table, and define a key-value pair format with the partition identifier as the key and the set of identifiers of the disk group as the value; S302, loads the preset mapping relationship between partition identifiers and disk groups during system initialization; S303, The preset mapping relationship is written as the initial mapping relationship into the partition disk group mapping table; S304, Verify the integrity and consistency of the initial mapping relationship; S305, writes the version information of the verified initial mapping relationship into the metadata storage system.

[0079] In this embodiment, the partition disk group mapping table is used to carry static binding information from partition identifiers to disk group identifier sets. The data is organized in key-value pairs, where the key is a fixed-length or variable-length partition identifier code, and the value is an ordered set of disk group identifiers carrying meta-fields such as validity period, status bit, creation time, commit sequence number, and generation source. The partition identifier originates from the preceding hash mapping link and must be globally unique, repeatable, and consistent with the upper-layer access protocol. The disk group identifier set represents a group of physical disk identifiers that can be read and written in parallel; the set order can be used for subsequent replica placement and read priority strategies. The key uses a unified byte order and character set encoding to avoid cross-language parsing differences; the value uses compact serialization to reduce storage and transmission overhead and retains backward-compatible reserved fields to support pattern evolution.

[0080] When initializing the data structure of the mapping table, the layout of the primary and secondary indexes needs to be clearly defined. The primary index uses the partition identifier as the primary key, providing equality queries and range scan capabilities; the secondary index uses the disk group identifier as the key to maintain an inverse set, used to quickly locate associated partitions during health checks and migration assessments. The persistence layer can use a distributed key-value store or a relational store with logs, while the memory layer maintains read-only snapshots to reduce query latency. To support horizontal scaling, the tablespace is sharded according to the partition identifier hash, and the shard directory records the shard range, primary replica members, and routing endpoints. Concurrency control uses an independent optimistic version number for each partition identifier or short leases based on shards to ensure mutually exclusive writes during initialization.

[0081] When loading preset mapping relationships, the input source can be a deployment manifest, a binding table exported by a capacity orchestration tool, or a static allocation file generated during the assembly phase. The loading process first performs a consistency check on the input, including partition identifier encoding verification, duplicate detection, disk group existence verification, and resource quota verification. Simultaneously, the manifest undergoes checksum or signature verification to ensure source trustworthiness. For cross-generation manifests, a matrix of declared schema versions and field compatibility is required. The loading end selects the parsing path based on the version and performs field default filling and value range correction when necessary. To avoid write spikes to the storage layer from a single load, a batch windowing strategy can be adopted, importing in batches and advancing visibility according to the commit sequence number.

[0082] When writing the predefined mapping relationship to the mapping table, atomic visibility and idempotency semantics must be satisfied. The execution path is as follows: generate candidate records, verify that the partition identifier is not occupied, append to the pre-write log, write the primary index, write the secondary index, and publish the commit point. The write operation is completed within a single transaction. In case of failure, the process is restored to the most recent consistent state by comparing the log replay with the commit sequence number. To ensure idempotency, candidate records carry a deduplication token. During re-execution, the deduplication token is compared with the version number to reject duplicate commits. In a clustered environment, write arbitration achieves cross-node fault tolerance through majority confirmation, and the commit point is only visible to the read path after the majority persistence is completed.

[0083] Verifying the integrity and consistency of the initial mapping relationship is a necessary step after loading. Integrity means that all partition identifier sets to be covered have entries, and each entry contains a valid set of disk group identifiers; consistency means that no partition identifier has multiple bindings, no disk group exceeds its capacity and throughput quotas, and the replica placement strategy satisfies the cross-fault domain distribution constraint. Verification can be implemented in two levels: offline batch verification scans all entries and builds a conflict graph, while online rapid verification monitors anomaly thresholds in real time based on incremental events and counters. For any gaps or conflicts found, the loader generates a list of repair suggestions, which are then supplemented or rolled back with new commit numbers after administrator confirmation, ensuring strong consistency is achieved at the end of the initialization phase.

[0084] When writing the version information of the verified initial mapping relationship to the metadata store, two types of versions need to be distinguished. The schema version describes the table structure and field semantics, used for cross-version compatibility and client capability negotiation; the content version describes the generation environment of this initial mapping relationship, including the weight table snapshot, virtual node multiple, namespace salt index, commit time, and committer identifier. Version records and mapping records are stored decoupled but associated with a version key. The read path first parses the version and then reads the mapping, ensuring a consistent context is obtained during reproduction. To avoid the jitter caused by large-scale replacements, new version records can be written first, and alias pointers can be used to point to the new version in a grayscale manner. After observation of stability, the default alias can be switched. The switching process does not affect the stability of existing partition identifiers.

[0085] Regarding observability and recovery capabilities, all write and verification events generated during the initialization phase are entered into the audit pipeline and persisted, including batch number, object range, execution time, and result code. In the event of an abnormal restart, the loader recovers based on the last commit sequence number and batch breakpoint, without needing to repeat batches that have already been successfully written. To reduce query jitter during cold starts, the snapshot cache can be built all at once after loading, and the version pointer is bound to the snapshot, ensuring that the read path reads the mapping view matching the version from the very first moment.

[0086] This embodiment initializes the mapping table structure, performs pre-bound batch loading, atomic writing and dual-index disk writing, combines full and incremental verification, and independently records version information during the startup phase. The initial relationship between partition identifiers and disk groups takes effect in a linear and consistent manner, and the read path only requires a single key lookup to obtain a stable result. The write path is idempotent and replayable, and can be recovered without loss according to the commit sequence number in case of interruption. Version and context snapshots ensure that the mapping remains traceable and reproducible in subsequent expansion and evolution. Thus, without increasing the size of block-level metadata, it provides a low-overhead and low-risk basic environment for subsequent routing and fault remapping.

[0087] In one embodiment, step S40 above includes: S401, query the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group; S402, determine the replica quantity parameters and replica distribution strategy; S403, Select multiple physical disks in the disk group that conform to the replica distribution strategy as replica storage locations; S404 writes the data block content to the selected physical disk in parallel to generate multiple copies; S405 verifies the success status and integrity of data block writes on each physical disk; S406, After all replicas are successfully written, record the correspondence between the partition identifier and the block identifier in the metadata; S407, write the replica storage location information into the metadata storage system; S408 records the execution time and performance metrics of data block write operations to the operation log.

[0088] In this embodiment, when performing a query based on the partition identifier, the access component retrieves the currently mapped disk group in the partition disk group mapping table using the partition identifier as the key. The search path prioritizes hitting the memory snapshot; if no match is found, it falls back to the persistent layer. The return value includes the disk group identifier, a list of member physical disks, member status, and effective version. To avoid reading expired entries, the query carries the read version or commit sequence number, and the reading end only accepts entries with a sequence number not lower than this number. After obtaining the disk group, it is necessary to generate replica quantity parameters and replica distribution strategies. The replica quantity parameter comes from the configuration center or partition-level strategy and can be derived based on data type, number of available domains, and expected durability level. The replica distribution strategy is used to constrain the distribution of replicas across racks, hosts, controllers, and disk cages, and provides write concurrency, pipeline depth, and cross-domain preferred order. During the strategy evaluation phase, the disk group member status, capacity margin, historical failure rate, and real-time load are considered to output a set of available slots and priorities.

[0089] The selection of physical disks is based on a replica distribution strategy, filtering and sorting the candidate list. Filtering conditions ensure that different replicas do not fall into the same fault domain. The sorting rules combine queue length, near-end latency, and throughput window to select a sufficient number of target physical disks and assign a write channel identifier and session lease to each target. To reduce conflicts in concurrent selection, the selection process adds short leases or uses a comparison-based write "placeholder" mark to target physical disks. The lease includes timeout and holder information to prevent multiple sources from simultaneously contending for the same disk location. During the parallel write phase, data blocks are used as the smallest transmission unit. An independent write session is established on each channel, enabling pipelining and fan-out control. Before each piece of data enters the channel, a content digest and length tag are calculated. The digest and length are encapsulated in the block header for peer verification and disk write confirmation. To limit network amplification, the transport layer performs congestion control based on the window size. When the bandwidth limit is reached, channels crossing fault domains are prioritized, while local channels are slowed down or suspended.

[0090] Upon receiving data, each target physical disk performs a two-level verification and atomic write-to-disk operation. The first level compares the block header digest with the recalculated digest in memory. The second level verifies and scans the write area after writing and returns an acknowledgment code; simultaneously, it records the sequence number, offset, and write range to ensure that idempotent retries do not generate duplicate entries. The success status is defined by the acknowledgment code and persistence barriers, returning success only when both the log and data pages meet the persistence conditions. The write scheduler maintains a replica-level acknowledgment convergence condition and timeout table, preventing downstream metadata commits until the required number of acknowledgments is reached. If a single channel fails, a replacement address is selected and rewritten; the replacement position must not violate distribution policy constraints, and the number of rewrites and waiting intervals are given by the policy. After all parallel writes meet the success conditions, the metadata commit phase begins. Metadata records include the mapping between partition identifiers and block identifiers, the physical path of successful replicas, version keys, and commit sequence numbers. Commits employ single-transaction pre-write logging and atomic publishing to ensure that read paths only see a complete, consistent view.

[0091] When replica storage location information is written to the metadata storage system, the forward index stores the replica location array using the block identifier as the key, and the reverse index stores the sparse directory carrying the block set using the physical disk as the key. Both types of indexes are bound with the same commit sequence number, and the reader can obtain the location and consistency context at once through the largest visible commit point. To support subsequent health checks and migrations, the location item includes the generation time, channel fingerprint, and disk media type. After writing is complete, the operation log records the execution time and performance metrics, including end-to-end time, queuing and transmission latency for each channel, verification time, disk write time, acknowledgment fan-out, number of retries, and final address dispersion. The log is entered into a time-series warehouse in a structured format for easy regression analysis and capacity planning. To improve steady-state throughput, adaptive parameters are applied throughout the process: concurrency and pipeline depth are reduced when the average queuing latency increases; cross-domain priority is increased when cross-domain links are idle; and the selection weight of a member is dynamically adjusted when the single disk error rate increases. All adjustments are controlled by policy versioning and referenced in the metadata to ensure that replay and auditing can reproduce experimental conditions.

[0092] This embodiment locates disk groups using partition identifiers and completes replica addressing, parallel disk writes, dual verification, and atomic commits within the same transaction domain. The read / write path does not rely on static binding from block to single disk to obtain a stable location view. The distribution strategy and backup addressing ensure cross-fault domain dispersion while suppressing hotspots. The success condition and commit point mechanism avoid visibility splits caused by partial success. Forward and reverse indexes work together to decouple the cost of location queries and health checks. The structured indicators of the operation log provide a calculable basis for subsequent expansion and parameter tuning. This reduces the pressure on metadata updates in high-concurrency scenarios, improves write success rate and data availability, and provides a consistent context for subsequent remapping and transparent access.

[0093] In one embodiment, step S50 above includes: S501 continuously monitors the health status of the physical disk through a heartbeat detection mechanism; S502, when a heartbeat signal loss or abnormality is detected, it is confirmed that the physical disk has failed; S503, Identify all partition identifiers associated with the faulty physical disk in the partition disk group mapping table; S504 selects a new disk group that meets the capacity and performance requirements from the pool of available disk resources as a new available disk group; S505, Update the mapping relationship in the partition disk group mapping table, and remap the partition identifier from the original disk group to the new available disk group; S506, trigger the data reconstruction task to reconstruct all data block contents corresponding to the partition identifier from the healthy copy to the physical disk in the new available disk group to generate a new copy.

[0094] In this embodiment, the heartbeat detection mechanism continuously collects the online status and response behavior of the physical disk. Detection requests are sent by the node agent at a fixed rhythm, and the monitoring end aggregates packet loss, latency, timeout, error codes, and hardware self-test signals by window to form a multi-indicator evidence set. The state machine progresses through four stages: normal, suspicious, degraded, and fault. A single instantaneous anomaly enters the suspicious stage, while multiple continuous anomalies or critical alarms enter the fault stage, thereby reducing misjudgments caused by jitter. Upon entering the fault stage, new writes to the target disk are immediately frozen, incomplete channels are suspended and marked in the queue, and a unique event marker is generated for end-to-end tracing.

[0095] After confirming the fault, it is necessary to locate the set of affected partition identifiers. The partition disk group mapping table maintains forward mappings and inverted indexes. The inverted index records the list of partition identifiers hosted by the disk group as the key. By combining this with the group member table, the associated set can be quickly filtered out. To avoid concurrent migration conflicts, a short-term lease or a version number-based placeholder is requested for each partition identifier. After authorization, it is added to the migration task queue. Throughout the process, the partition identifier and block identifier remain unchanged, and the logical entry point on the read and write ends remains consistent. Subsequently, only the hosting location is changed; the block-level meta-records are not rewritten.

[0096] New available disk groups come from the available disk resource pool. The resource pool is hierarchically arranged by availability domain, host, controller, and disk cage. The selection strategy comprehensively considers capacity margin, historical stability, current queue length, cross-domain dispersion constraints, and network path quality to produce a candidate set and priority weights. To suppress hotspots, the strategy allows for weighted randomization, ensuring that the migration of multiple partitions at the same time is spread across a wider range of members. If the resource pool is insufficient in the short term, the scheduler first releases or delays some migration batches occupied by lower priority partitions, ensuring that high-risk partitions are taken over first.

[0097] Mapping updates are performed using a transactional model. Each update patch generates a single replacement record for each partition identifier, containing the original disk group, target disk group, event flag, version key, and commit number. Logs are written to disk before writing, primary and inverted indexes are switched within the same transaction, and the commit point is only visible to the query path after publication. To ensure idempotency, duplicate commits are deduplicated using event flags, and published commit numbers cannot be rolled back. Batch processing uses small-batch advancement and intermittent refresh to avoid directory jitter and cache mismatch caused by one-time switching; under the constraint of monotonically increasing commit numbers, the query end only reads entries not less than its own read barrier, ensuring a consistent view.

[0098] The data reconstruction task is triggered immediately after the mapping update. The task scheduler collects the physical location and checksum of the healthy replica for each partition identifier, establishes a fan-out channel from the healthy replica to the target disk group, and writes the data block content to the target physical disk using segmented parallel and pipelined transmission. Each data segment is accompanied by a length and digest tag before entering the channel. After the target end completes the disk write, it performs a secondary check and returns confirmation. The scheduler maintains replica-level convergence conditions and timeout tables. If a single channel fails, a replacement location is selected for retry according to distributed constraints. Retry follows backoff and maximum number limits to prevent amplification of failures. To reduce the impact on online services, the reconstruction task dynamically adjusts the concurrency and bandwidth limit according to the production load. If necessary, the reconstruction window is divided into multiple time slices for interleaved execution. After reconstruction is completed, the replica status and location index are updated, the event tag and completion snapshot are written to the audit pipeline, the partition lease is released, and the temporary routing exception is turned off.

[0099] To ensure link closure and reproducibility, the monitoring, selection, update, and reconstruction processes all record version information and key parameter snapshots, including the address weight table, read / write throttling parameters, lease duration, read barriers, and commit sequence numbers. During anomaly rollback, the system restores to the most recent consistent point based on the pre-write logs and commit sequence numbers; during cross-node recovery, event-driven continuation is used instead of a full redo, shortening the recovery path. The entire mechanism completes the transition from the failed bearer to the new bearer without changing the partition and block identifiers, and exposes a continuous, monotonous mapping view to the read / write path.

[0100] This embodiment achieves stable fault determination through a heartbeat detection mechanism and multi-source evidence verification, reduces migration concentration through resource pool grouping and weight distribution, provides a consistent view to the query end through atomic switching of mapping relationships and read barrier control, and restores redundancy without modifying partition and block identifiers through segmented parallel reconstruction based on healthy replicas. This limits migration costs to the partition to disk group level, avoids rewriting massive block-level meta records, reduces access interruptions caused by changes in client location, reduces the disturbance of reconstruction to online services, and makes fault handling traceable and replayable.

[0101] In one embodiment, step S60 above includes: S601, Receive a read / write request containing the target partition identifier; S602, parse the read / write request and extract the target partition identifier and operation type; S603, use the target partition identifier to query the partition disk group mapping table to obtain the currently mapped disk group information; S604, the obtained disk group information is identified as the target disk group; S605, Verify whether the physical disks in the target disk group are available; S606, If the target disk group is available, then select the target physical disk in the target disk group according to the operation type and the current disk load status; S607, the read / write request is routed to the target physical disk for data access operation; S608 records the processing status, execution time, and performance metrics of read and write requests to the access log system; S609 returns the operation result and status information to the requester after the read / write operation is completed.

[0102] In this embodiment, when receiving a read / write request containing a target partition identifier, the entry component performs integrity verification and length boundary checks on the request body and header fields to ensure that the minimum set of the target partition identifier, operation type, and request number is complete and encoded consistently. The target partition identifier uses a unified byte order and character set. The entry component performs format reduction comparison and namespace verification on it and registers the request number in the idempotent window to avoid duplicate execution caused by replay. During the parsing phase, the target partition identifier and operation type are extracted from the request body. The operation type is limited to read or write, and is accompanied by an additional set of parameters, such as read offset and length, write position and content digest, etc. After extraction, a structured access intent is generated to enter the query path.

[0103] The query path retrieves the current mapping record in the partition disk group mapping table using the target partition identifier as the key. The mapping table is maintained in memory as a read-only snapshot with a commit sequence number. The query carries a read barrier to ensure visibility does not rollback; if a memory miss occurs, it rolls back to the persistent layer for reading and cache refilling. The mapping record contains the disk group identifier, member list, effective version, and validity period. Upon successful retrieval, the record is promoted to a candidate view of the target disk group. The candidate view enters the availability verification process, checking the online status, I / O error count, SMART metric threshold, remaining capacity level, current load, and fault domain label of each member in the member list. This generates a set of available members and a list of reasons for unavailability, while simultaneously calculating the view's freshness to determine whether a background refresh needs to be triggered.

[0104] Once the target disk group is verified, the scheduler selects a set of target physical disks based on the operation type and current load. For read paths, members with the lowest latency and located in different fault domains are prioritized, supporting both single-replica direct reads and multi-replica race reads. For write paths, multiple members are selected based on the number and distribution constraints of replicas, ensuring that different replicas are distributed across racks, hosts, or controllers. The selection algorithm scores members based on a combination of queue length, bandwidth utilization, historical failure rate, and link health, reducing the weight of high-load members and removing members in the recovery window from participation. After selection, a session lease and transmission channel identifier are assigned to each target physical disk. The lease includes timeout and holder information to prevent concurrent contention.

[0105] The routing phase maps read and write requests to the network endpoints of the target physical disks. For read paths, read-only sessions are established, and requests are sent in parallel or sequentially according to a strategy. The first result that satisfies consistency and verification constraints is returned, and the remaining sessions terminate and latency is recorded. For write paths, independent sessions are established for each target physical disk for parallel transmission. Data is accompanied by a length and content digest before entering the channel, and an acknowledgment code is sent back after the target end completes the write operation. The transport layer employs congestion window and pipeline depth adaptive control, reducing concurrency when the link is saturated and increasing fan-out to shorten tail latency when idle. Write convergence conditions are defined based on the number of acknowledgments and persistence barriers; no success signal is issued if convergence is not achieved. If a single channel fails, the scheduler selects a replacement position to retry without violating distribution constraints and writes the reason for the failure to a health counter for subsequent weighting.

[0106] After the access is completed, the results are processed and recorded. The scheduler generates metrics such as processing status, execution time, queuing and transmission latency for each channel, verification time, acknowledgment distribution, and retries, and writes them to the time-series storage of the access log system. The read path additionally records the list of hit replicas and abandoned replicas, and the write path additionally records the location of successful replicas and the session fingerprint. All entries are associated with the request number to support end-to-end tracing. The result return channel follows the same idempotent constraints as the entry point; if a duplicate request is detected and the previous request was successful, the result is replayed directly without repeating the execution. The entire chain, without exposing physical location information to the client, uses the target partition identifier to drive queries and routing, ensuring decoupling between logical addressing and physical bearer, and uses versioning and read barrier mechanisms to ensure consistency between the query view and the disk state.

[0107] This embodiment achieves location resolution and efficient access by sequentially connecting the target partition identifier with the partition disk group mapping table, target disk group availability verification, target physical disk selection based on operation type and load, parallel routing and convergence confirmation, and structured log recording, all while requiring only a logical identifier from the client. Reads reduce tail latency through replica racing, writes improve redundancy reliability through distribution constraints and convergence conditions, adaptive transmission control mitigates the amplification of latency caused by congestion, and read barriers and commit sequence numbers ensure read reliability. Figure 1 Therefore, idempotent windows eliminate the side effects of replay, thereby reducing the impact of location changes on upper layers, stabilizing high-concurrency access performance, and providing quantifiable basis for subsequent capacity planning and fault diagnosis.

[0108] In one embodiment, a data storage device based on partition identifier mapping is provided, which corresponds one-to-one with the data storage method based on partition identifier mapping in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data storage device based on partition identifier mapping of the present invention. The modules include a file segmentation module 10, a partition identifier generation module 20, a mapping table creation module 30, a copy writing module 40, a fault remapping module 50, and a read / write access module 60. Detailed descriptions of each functional module are as follows: The file segmentation module 10 is used to receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block. Partition identifier generation module 20 is used to generate a partition identifier based on the file identifier and the block identifier through hash mapping processing; The mapping table creation module 30 is used to create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table. The replica writing module 40 is used to query the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple replicas. The fault remapping module 50 is used to monitor the health status of the physical disk. When a disk fault is detected, the partition identifier and the block identifier remain unchanged, and the mapping relationship in the partition disk group mapping table is updated to remap the partition identifier associated with the faulty physical disk to a new available disk group. The read / write access module 60 is used to query the partition disk group mapping table using the target partition identifier when a read / write request containing the target partition identifier is received, obtain the currently mapped disk group as the target disk group, and route the read / write request to the target disk group for data access.

[0109] For specific limitations regarding the data storage device based on partition identifier mapping, please refer to the foregoing limitations on the data storage method based on partition identifier mapping, which will not be repeated here. Each module in the aforementioned data storage device based on partition identifier mapping can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0110] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a data storage method based on partition identifier mapping.

[0111] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a data storage method based on partition identifier mapping.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block; A partition identifier is generated based on the file identifier and the block identifier through hash mapping; Create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table; Based on the partition identifier, query the partition disk group mapping table to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple copies; Monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group. When a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access.

[0113] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block; A partition identifier is generated based on the file identifier and the block identifier through hash mapping; Create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table; Based on the partition identifier, query the partition disk group mapping table to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple copies; Monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group. When a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access.

[0114] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0117] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0118] The user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various open, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A data storage method based on partition identification mapping, characterized by, Includes the following steps: Receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block; A partition identifier is generated based on the file identifier and the block identifier through hash mapping; Create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table; Based on the partition identifier, query the partition disk group mapping table to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple copies; Monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group. When a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access.

2. The partition identification map based data storage method of claim 1, wherein, Receive a file to be stored, divide the file into multiple data blocks based on a file identifier, and generate a block identifier for each data block, including: Receive a file storage request containing the file to be stored and its corresponding file identifier; Parse the file attribute information in the file storage request to obtain the file size and file type characteristics; The data block partitioning strategy parameters are determined based on the file size and file type characteristics; The fixed size parameters of the data blocks are determined according to the partitioning strategy parameters; The file to be stored is sequentially divided into multiple consecutive data blocks according to the fixed size parameter; Verify the integrity of each data block and whether its size meets the fixed size parameter requirements; Generate a globally unique block identifier for each verified data block; Establish an index mapping the file identifier to all corresponding block identifiers; Write the mapping relationship index into the metadata storage system; The record includes metadata information such as the partition timestamp and the partitioning strategy version.

3. The data storage method based on partition identifier mapping as described in claim 1, characterized in that, A partition identifier is generated based on the file identifier and the block identifier through a hash mapping process, including: Obtain the numerical code of the file identifier and the sequential number of the block identifier; The input data is formed by combining the numerical code of the file identifier and the sequential number of the block identifier; The input data is hashed using a pre-configured hash function; The hashing result is mapped to a predefined partition identifier space to generate a unique partition identifier and verify the validity of the partition identifier in the partition identifier space. Establish a relationship between the verified valid partition identifier and the corresponding file identifier and block identifier; Record the hash function version and mapping parameters used in the partition identifier generation process.

4. The data storage method based on partition identifier mapping as described in claim 1, characterized in that, Create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table, including: Initialize the data structure of the partition disk group mapping table, and define a key-value pair format with the partition identifier as the key and the set of identifiers of the disk group as the value; Load the preset mapping relationship between partition identifiers and disk groups during system initialization; Write the preset mapping relationship as the initial mapping relationship into the partition disk group mapping table; Verify the integrity and consistency of the initial mapping relationships; The version information of the verified initial mapping relationship is written to the metadata storage system.

5. The data storage method based on partition identifier mapping as described in claim 1, characterized in that, Based on the partition identifier, the partition disk group mapping table is queried to obtain the currently mapped disk group. The data block content is written to the physical disk in the disk group to generate multiple copies, including: The partition disk group is obtained by querying the partition disk group mapping table based on the partition identifier; Determine the replica count parameters and replica distribution strategy; Select multiple physical disks in the disk group that conform to the replica distribution strategy as replica storage locations; The data block contents are written in parallel to the selected physical disk to generate multiple copies; Verify the success status and integrity of data block writes on each physical disk; Once all replicas have been successfully written, the correspondence between the partition identifier and the block identifier is recorded in the metadata. Write the copy storage location information into the metadata storage system; Record the execution time and performance metrics of data block write operations to the operation log.

6. The data storage method based on partition identifier mapping as described in claim 1, characterized in that, Monitor the health status of the physical disk. When a disk failure is detected, keep the partition identifier and the block identifier unchanged, and update the mapping relationship in the partition disk group mapping table to remap the partition identifier associated with the failed physical disk to a new available disk group, including: The health status of the physical disk is continuously monitored through a heartbeat detection mechanism; When a loss or abnormality in the heartbeat signal is detected, it is confirmed that the physical disk has failed. Identify all partition identifiers associated with the faulty physical disk in the partition disk group mapping table; Select a new disk group that meets the capacity and performance requirements from the pool of available disk resources; Update the mapping relationship in the partition disk group mapping table, and remap the partition identifier from the original disk group to the new available disk group; Trigger a data reconstruction task to reconstruct all data block contents corresponding to the partition identifier from the healthy copy to the physical disks in the new available disk group to generate a new copy.

7. The data storage method based on partition identifier mapping as described in claim 1, characterized in that, When a read / write request containing a target partition identifier is received, the target partition identifier is used to query the partition disk group mapping table to obtain the currently mapped disk group as the target disk group, and the read / write request is routed to the target disk group for data access, including: Receive read / write requests containing the target partition identifier; Parse the read / write request to extract the target partition identifier and operation type; Use the target partition identifier to query the partition disk group mapping table to obtain the currently mapped disk group information; The obtained disk group information is identified as the target disk group; Verify whether the physical disks in the target disk group are available; If the target disk group is available, then select the target physical disk in the target disk group according to the operation type and the current disk load status; The read / write request is routed to the target physical disk for data access operations; Record the processing status, execution time, and performance metrics of read and write requests to the access log system; Once the read / write operation is complete, the operation result and status information are returned to the requester.

8. A data storage device based on partition identifier mapping, characterized in that, The data storage device based on partition identifier mapping includes: The file segmentation module is used to receive the file to be stored, divide the file to be stored into multiple data blocks based on the file identifier, and generate a block identifier for each data block. A partition identifier generation module is used to generate a partition identifier based on the file identifier and the block identifier through hash mapping processing; The mapping table creation module is used to create a partition disk group mapping table and write the initial mapping relationship between the partition identifier and the disk group into the partition disk group mapping table. The replica writing module is used to query the partition disk group mapping table according to the partition identifier to obtain the currently mapped disk group, and write the data block content to the physical disk in the disk group to generate multiple replicas. The fault remapping module is used to monitor the health status of the physical disk. When a disk fault is detected, the partition identifier and the block identifier remain unchanged, and the mapping relationship in the partition disk group mapping table is updated to remap the partition identifier associated with the faulty physical disk to a new available disk group. The read / write access module is used to query the partition disk group mapping table using the target partition identifier when a read / write request containing the target partition identifier is received, obtain the currently mapped disk group as the target disk group, and route the read / write request to the target disk group for data access.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a partition identifier mapping-based data storage program stored on the memory and executable on the processor, wherein the partition identifier mapping-based data storage program, when executed by the processor, implements the steps of the partition identifier mapping-based data storage method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a data storage program based on partition identifier mapping, which, when executed by a processor, implements the steps of the data storage method based on partition identifier mapping as described in any one of claims 1-7.

Images