File fragment storage method, device, equipment, storage medium and program product

Abstract

The application provides a file fragment storage method, device, equipment, storage medium and program product, and relates to the field of financial technology or other related fields. The method comprises the following steps: distributing each fragment data of a plurality of original files to a working node in a distributed cluster respectively through a consistent hashing algorithm, and recording a metadata triple of the fragment data through the working node; after storing the fragment data in the working node, pulling a plurality of fragment data of each original file, and performing consistent verification on each fragment data through the pulled fragment data and the metadata triple; for each original file, after each fragment data of the original file passes the consistent verification, performing in-memory mapping on each fragment data, determining a fragment mapping table of the original file, and using the fragment mapping table to determine an in-memory mapping area of each file to directly access each fragment data. The method provided in the application improves the storage space utilization rate and stability of the fragment data.

Description

Technical Field

[0001] This application relates to the field of financial technology or other related fields, and in particular to a file fragmentation storage method, apparatus, device, storage medium and program product. Background Technology

[0002] In distributed data processing scenarios, batch files are typically stored, transmitted, and archived using distributed file systems, centralized metadata indexes, and conventional integrity verification methods. Taking the financial industry as an example, banks handle massive amounts of data such as transaction statements, customer credentials, and log files in their daily operations.

[0003] However, when faced with a large number of small files with high concurrency, the above solutions are prone to low space utilization and significant metadata management pressure due to fixed-granularity storage. Data processing based on conventional memory mechanisms is also prone to fragmentation and recycling pauses, which further affect throughput, processing latency and system stability.

[0004] Therefore, how to balance storage utilization, processing stability, and data integrity in scenarios involving the processing of massive amounts of small files has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a file fragment storage method, apparatus, device, storage medium, and program product to solve the technical problems of low storage space utilization and insufficient metadata management efficiency in file fragment storage.

[0006] Firstly, this application provides a file fragmentation storage method, including:

[0007] Using a consistent hashing algorithm, the data of each fragment of multiple original files is distributed to the worker nodes in the distributed cluster, and the worker nodes record the metadata triples of the fragment data. The multiple fragments are obtained by dynamically fragmenting the original files.

[0008] After the worker nodes have stored the sharded data, they pull multiple sharded data from each original file and perform consistency checks on each sharded data using the pulled sharded data and metadata triples. The consistency checks include file-level checks and metadata-level checks.

[0009] For each original file, after the data in each segment of the original file passes the consistency check, memory mapping is performed on each segment of the data to determine the segment mapping table of the original file. The segment mapping table is used to determine the memory mapping area of ​​each file so as to directly access each segment of the data.

[0010] Secondly, this application provides a file fragmentation storage device, comprising:

[0011] The dynamic sharding module is used to distribute the shard data of multiple original files to the worker nodes in the distributed cluster using a consistent hashing algorithm, and to record the metadata triples of the shard data through the worker nodes. The multiple shard data are obtained by dynamically sharding the original files.

[0012] The consistency verification module is used to pull multiple fragments of each original file after the worker node has stored the fragment data, and to perform consistency verification on each fragment data by using the pulled fragment data and the metadata triple. The consistency verification includes file-level verification and metadata-level verification.

[0013] The memory optimization module is used to perform memory mapping on each of the original files after the consistency check of each of the original file's fragment data, and to determine the fragment mapping table of the original file. The fragment mapping table is used to determine the memory mapping region of each file so as to directly access each of the fragment data.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0019] The file fragmentation storage method, apparatus, device, storage medium, and program products provided in this application generate multiple fragmented data through dynamic fragmentation, allocate the fragments to worker nodes in a distributed cluster using a consistent hashing algorithm, and have the worker nodes synchronously save the metadata triples associated with the fragments. After fragmentation storage is completed, file-level verification and metadata-level verification are performed. Finally, a fragmentation mapping table is established based on the verified fragments, linking the fragmented data, metadata, and access relationships, thereby completing the fragmented storage, verification, and direct access processing of files. Based on the combination of current technical features, this implementation method ensures that the fragmentation location is determined, the verification basis is clear, and the access path can be directly located. It is suitable for stable processing in high-concurrency, massive small file scenarios, improves file storage, achieves zero-copy file transfer, and avoids repeated copying of data between user space and kernel space. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 A flowchart illustrating a file fragment storage method provided in this application embodiment. Figure 1 ;

[0022] Figure 2 A flowchart illustrating a file fragment storage method provided in this application embodiment. Figure 2 ;

[0023] Figure 3 A flowchart illustrating a file fragment storage method provided in this application embodiment. Figure 3 ;

[0024] Figure 4 A schematic diagram of the structure of a file fragmentation storage device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] It should be noted that the file fragmentation storage method, apparatus, device, storage medium and program product provided in this application can be used in the field of financial technology, or in any field other than financial technology. The application field of the file fragmentation storage method, apparatus, device, storage medium and program product in this application is not limited.

[0029] Massive small file processing technology is typically used in distributed data processing scenarios. Taking the financial industry as an example, in daily operations, account statement data needs to be stored in a distributed file system according to channel and protocol. At the end of the month, the data is retrieved from the distributed file system to the local server, decompressed, and then recompressed before being sent.

[0030] In the above scenario, existing solutions typically first split the files and store them in a distributed file system, then use a centralized metadata index to record information such as file path, size, and storage location. When it is necessary to collect or archive the files, the file data is retrieved from the corresponding node based on the index information.

[0031] However, fixed-granularity fragmentation and storage methods are difficult to adapt to the actual characteristics of files from different sources and of different sizes. Files rely on conventional memory processing mechanisms, which are prone to memory fragmentation and garbage collection pauses, resulting in low storage space utilization.

[0032] In view of this, how to balance the efficiency of sharded storage, the reliability of verification, and the performance of subsequent access in the scenario of processing massive small files has become an urgent technical problem to be solved.

[0033] To address the aforementioned issues, a file sharding storage method is provided. This method uses a consistent hashing algorithm to dynamically shard multiple original files, distributing the sharded data to worker nodes in a distributed cluster. Each worker node records the corresponding metadata triples. After the worker nodes complete the sharded data storage, they retrieve multiple shards from each original file and perform file-level and metadata-level verification based on the retrieved shard data. After each shard passes the consistency verification, it is then memory-mapped to determine the sharding mapping table of the original files, enabling direct access to each shard. This provides a technical foundation for the stable processing and consistency assurance of massive amounts of small files.

[0034] Figure 1 This is a flowchart illustrating a file fragmentation storage method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0035] S101. Using a consistent hashing algorithm, the data fragments of multiple original files are distributed to the worker nodes in the distributed cluster, and the worker nodes record the metadata triplet of the data fragments.

[0036] In this embodiment of the application, the entity executing this step can be a file sharding storage system deployed in a distributed cluster. The file sharding storage system is connected to multiple worker nodes to receive multiple original files to be processed, complete sharding planning, determine sharding locations, and issue storage instructions.

[0037] The consistent hashing algorithm is used to distribute the data shards of multiple original files to worker nodes in a distributed cluster. These multiple data shards are obtained by dynamically sharding the original files. Dynamic sharding is used to shard the original files to generate multiple data shards.

[0038] A worker node is a computing node in a distributed cluster used to store and process sharded data. For example, a server node in a distributed cluster is responsible for storing specific sharded data. A metadata triple is a set of metadata describing the file storage state, used to record the association information of sharded data, and serves as the basis for consistency verification when subsequently pulling sharded data.

[0039] In this step, the data of each fragment of multiple original files is evenly distributed to the worker nodes in the distributed cluster using a consistent hashing algorithm, and the metadata triples of the fragment data are recorded by the worker nodes.

[0040] In practice, the original file can be received first, and then dynamically segmented according to the preset segmentation method to form multiple segmented data, and a unique segment identifier can be generated for each segmented data.

[0041] The consistent hashing algorithm determines the hash value corresponding to the unique identifier of each data shard and identifies the corresponding target worker node. The file shard storage system sends the shard data to the target worker node over the network, and the target worker node writes the shard data to the storage medium. To ensure the feasibility of subsequent verification, before or during the transmission of shard data, corresponding verification information can be saved for each shard, and the attribute information corresponding to that shard can be extracted to form a metadata triple.

[0042] In one possible implementation, the metadata triple includes the file path, fragment size, and modification time. After receiving the fragment data, the worker node establishes an association between the fragment data and the metadata triple, and records the storage completion status.

[0043] In an exemplary implementation, an original file is parsed into multiple data fragments. The file fragment storage system distributes these fragments to multiple worker nodes using a consistent hashing algorithm. After completing the write operation, each worker node sends back storage confirmation information, and the control service updates the fragment records accordingly.

[0044] By combining dynamic sharding, consistent hashing allocation, and synchronous recording of metadata triples, each shard forms a correspondence between content information, attribute information, and node location when it enters the distributed cluster, thus providing basic data for subsequent pull verification and memory mapping.

[0045] S102. After storing the sharded data on the worker node, pull multiple sharded data from each original file, and perform consistency verification on each sharded data using the pulled sharded data and metadata triplet.

[0046] Consistency checks include file-level checks and metadata-level checks. Consistency checks are used to verify the retrieved fragment data against the metadata triples after the fragment data is stored on the worker nodes, determining whether the fragment data has remained consistent during transmission and storage. File-level checks verify the content of the fragment data to confirm that its content has not changed. Metadata-level checks verify the attribute information of the fragment data against the metadata triples to confirm that the attribute information is consistent.

[0047] Specifically, after receiving the storage completion status from each target worker node, the file sharding storage system initiates a shard retrieval process based on the shard records. Multiple shard unique identifiers for the same original file are aggregated according to the original file identifier. Then, based on the worker node location corresponding to each shard unique identifier, a read request is sent to the corresponding worker node. Upon receiving the read request, the worker node reads the corresponding shard data and the metadata triple associated with that shard, and returns the shard data and metadata triple to the file sharding storage system.

[0048] In practice, file-level verification can be performed on the retrieved shard data, and metadata-level verification can be performed on its associated metadata triples. When both file-level and metadata-level verifications pass, the shard is determined to have passed the consistency check; when either verification fails, the shard is determined to be abnormal. File-level and metadata-level verifications can be performed in parallel or in a preset order.

[0049] For sharded data that fails verification, the abnormal shard information can be recorded for subsequent re-fetching, rewriting, or further processing.

[0050] For the same original file, when all its fragment data passes file-level verification and metadata-level verification, the original file is marked as consistent and the next step of memory mapping processing is triggered.

[0051] For example, in the case where a certain original file contains multiple fragments of data, the file fragment storage system pulls all fragment data and their metadata triples from multiple worker nodes, and performs file-level verification and metadata-level verification respectively.

[0052] By verifying the fetched results through two dimensions—fragmented content and fragmented attributes—it is possible to identify abnormal content or attributes before fragment reassembly, thereby ensuring that the data and metadata entering the subsequent access process remain consistent.

[0053] Based on the aforementioned embodiments, further, multiple fragments of data from each original file are retrieved, and consistency checks are performed on each fragment, including: determining the retrieval path for each original file according to its original file size, with each retrieval path carrying a different number of original files; retrieving each original file in parallel through its retrieval path, and performing consistency checks on each fragment.

[0054] Different pull paths can correspond to different concurrent handling capabilities. Parallel pull refers to initiating retrieval requests for each original file fragment data simultaneously through multiple pull paths, so that the fragment data enters the consistency verification process after retrieval.

[0055] Before fetching each original file, the file size of each original file is read, and the original files are divided into different levels according to a preset size threshold. Larger files (those larger than the preset size threshold) are assigned to fetch paths that carry fewer original files, while smaller files (those smaller than the preset size threshold) are assigned to fetch paths that carry more original files. For example, the first fetch path might carry multiple smaller files (e.g., 100), while the second fetch path might carry a small number of larger files (e.g., two).

[0056] For data shards that pass the consistency check, the subsequent memory optimization process begins. For data shards that fail the consistency check, the original file corresponding to the data shard is re-sharded, and the re-sharded data shards are redistributed to the worker nodes on the distributed cluster.

[0057] By using different path partitioning and parallel fetching methods, the fetching tasks of each original file are distributed to multiple paths with different carrying capacities, thereby improving the continuity and stability of subsequent reassembly processes.

[0058] Based on the aforementioned embodiments, a detailed description is provided of the consistency verification of each data shard, including: performing a hash calculation on the data content of the data shard to obtain a second hash value; if the second hash value is consistent with the third hash value, the file-level verification is determined to be successful; comparing the first metadata of the fetched data shard with the second metadata in the metadata triple stored on the worker node; if the first metadata and the second metadata are consistent, the metadata-level verification is determined to be successful.

[0059] In this embodiment, file verification and metadata-level verification are performed in parallel.

[0060] The second hash value represents the result of hashing the retrieved shard data, while the third hash value serves as the base hash value for file-level verification. It is calculated based on the shard data before it is allocated to the worker nodes and is saved along with the index information of the shard data.

[0061] In the specific implementation, the content of the retrieved allocation data is hashed to obtain a second hash value. The hash algorithm can be any of SHA-256 (Secure Hash Algorithm 256), SHA-1 (Secure Hash Algorithm 1), or MD5 (Message-Digest Algorithm 5). Preferably, the hash algorithm is SHA-256.

[0062] The second hash value is compared with the third hash value calculated based on the fragmented data before it was allocated to the worker node. If the second hash value and the third hash value are consistent, it is determined that the fragmented data has not undergone content changes during transmission, storage and retrieval, and thus the file-level verification result is output.

[0063] If the second hash value and the third hash value are inconsistent, the file-level verification is determined to have failed. It is determined that the content of the data fragment has changed during transmission, storage, and retrieval, thus confirming the file-level verification failure.

[0064] In metadata-level validation, the metadata triples stored by the worker node must include at least three attributes: file path, file size, and file modification time.

[0065] At the metadata level, a B+ tree index list is constructed. After the sharded data is pulled and carries the first metadata, it is compared with the second metadata in the text metadata triple. When the two are consistent, the consistency of the second metadata corresponding to the first metadata is further confirmed.

[0066] If the first and second metadata are consistent, the fragment is determined to be consistent in file path, file size, and file modification time, and the metadata-level verification result is output as passed. If the first and second metadata are inconsistent, the metadata-level verification is determined to have failed.

[0067] By splitting consistency verification into file-level verification and metadata-level verification, the retrieved sharded data can be verified for content consistency through hash values, thus improving the accuracy and reliability of sharded data verification.

[0068] S103. For each original file, after the data of each fragment of the original file passes the consistency check, perform memory mapping on the data of each fragment to determine the fragment mapping table of the original file.

[0069] The fragment mapping table is used to determine the memory mapping region for each original file, enabling direct access to each fragment of data. The memory mapping is used to determine the fragment mapping table for the original file after the fragment data passes consistency verification, and to directly access each fragment of data accordingly. The fragment mapping table records the memory mapping region corresponding to each fragment of data in the original file, supporting subsequent direct access to the fragment data.

[0070] In practice, memory mapping (mmap) technology can be used to map each data segment to memory and determine the corresponding memory mapping region based on each data segment. After mapping, the data segments of the same original file are combined to form a segment mapping table of the original file, which realizes zero-copy file transfer and avoids repeated copying of data between user space and kernel space.

[0071] When accessing a certain original file later, the access module can locate the corresponding memory mapping area of ​​each piece of data in the original file according to the original file's fragment mapping table, and directly read the corresponding fragment data.

[0072] This application provides a file fragment storage method. Multiple fragments are generated through dynamic fragmentation. A consistent hashing algorithm is used to allocate these fragments to worker nodes in a distributed cluster. Each worker node synchronously stores the metadata triples associated with its fragment. After fragment storage is complete, file-level and metadata-level verification are performed. Finally, a fragment mapping table is established based on the verified fragments, linking fragment data, metadata, and access relationships. This completes the fragment storage, verification, and direct access processing of the file. Based on the combination of current technical features, this implementation ensures that the fragment location is determined, the verification basis is clear, and the access path can be directly located. It is suitable for stable processing in high-concurrency scenarios with massive amounts of small files, improves file storage, achieves zero-copy file transfer, and avoids repeated copying of data between user space and kernel space.

[0073] Based on the aforementioned embodiments, the memory mapping of each data segment to determine the original file's segment mapping table is further described in detail, including: allocating direct memory outside the Java Virtual Machine heap as a data buffer through the buffer allocator provided by the network communication framework; storing the segmented data in the data buffer; and performing memory mapping on each data segment to determine the original file's segment mapping table.

[0074] In this embodiment, the buffer allocator provided by the network communication framework is used to request direct memory located outside the Java Virtual Machine heap and serve as a data buffer to carry fragmented data.

[0075] The received fragmented data is written directly to the data buffer, so that the fragmented data is temporarily stored in the data buffer as a non-heap object.

[0076] Establish memory mapping relationships using the fragmented data in the data buffer as objects, and generate a fragment mapping table to identify the storage paths and order relationships of each fragment of the original file.

[0077] By storing and mapping fragmented data around off-heap direct memory, the heap data usage is reduced, and direct access to the original file is enabled, thereby improving the continuity of fragmented data access and the efficiency of reassembly processing.

[0078] In one possible implementation, memory mapping is performed on each data segment, and the segment mapping table of the original file is described in detail, including: mapping the data content of the data segment to virtual addresses to obtain the mapping area of ​​the data storage address of the data segment; recording the area information of each mapping area according to the logical order of each data segment in the original file, and constructing the segment mapping table.

[0079] Through the above processing, the sharding mapping table can be directly associated with sharded data.

[0080] Specifically, for each data segment that has passed consistency verification, the file content corresponding to the data segment is mapped into the process address space, giving each data segment a corresponding virtual address range, forming a directly accessible mapped region. Region information is written according to the original file's order, so that the segment mapping table contains both address location and order organization information. Therefore, when accessing the original file is needed, the memory-mapped region of the target segment can be located directly based on the segment mapping table and read, without the need for complex reorganization.

[0081] Accessing fragmented data directly via virtual addresses makes the fragment location of the original file clearer, simplifying the address lookup process during subsequent retrieval and reassembly.

[0082] Figure 2 A flowchart illustrating a file fragment storage method provided in this application embodiment. Figure 2 In this embodiment Figure 1 Building upon the previous examples, this paper further details how multiple data shards are distributed to worker nodes in a distributed cluster using a consistent hashing algorithm. Figure 2 As shown, the method includes:

[0083] S201. For each data shard, perform a hash calculation on the unique identifier of the data shard to obtain the first hash value.

[0084] The first hash value is used for subsequent virtual node lookup.

[0085] Specifically, after receiving multiple data shards, a hash operation is first performed on the unique identifier of each data shard to obtain the first hash value.

[0086] S202. Based on the first hash value, the sharded data is distributed to the corresponding worker nodes in the distributed cluster using a consistent hashing algorithm with virtual nodes.

[0087] There is a relationship between virtual nodes and worker nodes. Virtual nodes are logical nodes pre-deployed on a consistent hashing ring. The same worker node can correspond to multiple virtual nodes. The relationship is updated synchronously when the distributed cluster is scaled up or down, or when virtual nodes are added or deleted.

[0088] The first hash value is mapped to the corresponding virtual node. Based on the association between the virtual node and the physical node, the corresponding worker node is determined, and the sharded data is allocated to that worker node. Specifically, the consistent hashing algorithm can be implemented using consistent hash rings, ordered hash tables, hash slots, etc.

[0089] In one possible implementation, based on the first hash value, the sharded data is allocated to the corresponding worker nodes in the distributed cluster using a consistent hashing algorithm with virtual nodes. This includes: mapping the first hash value to a consistent hashing ring with multiple pre-distributed virtual nodes to determine the initial position corresponding to the first hash value; searching for the first virtual node in the consistent hashing ring in a clockwise direction according to the initial position; and allocating the sharded data to the worker nodes in the distributed cluster associated with the virtual nodes.

[0090] The initial position is the location of the first hash value on the consistent hash ring, which serves as the starting point for clockwise searching.

[0091] In practice, the unique identifier of each shard is hashed to obtain a first hash value, which is then mapped to coordinates on the consistent hash ring. After mapping, using this coordinate as the initial position, virtual nodes on the consistent hash ring are searched sequentially in a clockwise direction along the hash value until the first virtual node is found. The worker node identifier corresponding to this virtual node is read, and the current shard data is written to the worker node corresponding to this worker node identifier.

[0092] In one implementation, the number of virtual nodes can be configured based on the number of worker nodes and the size of the sharded data. For example, each worker node may correspond to multiple virtual node mapping points to reduce the skew caused by a single physical node carrying a concentrated load.

[0093] By associating virtual nodes with worker nodes, the target node for the sharded data is determined based on the location result of the first hash value, thereby outputting a stable allocation result.

[0094] This application provides a file sharding storage method. For each shard, a first hash value is obtained by hashing the unique identifier of the shard data. Based on the first hash value, a consistent hashing algorithm with virtual nodes is used to allocate the shard data to corresponding worker nodes in a distributed cluster. The allocation is determined through the association mapping between virtual nodes and worker nodes, improving the balance of shard data allocation. This reduces allocation fluctuations and improves the continuity of distributed cluster storage and processing.

[0095] Based on the aforementioned embodiments, before allocating the fragment data of multiple original files to the worker nodes in the distributed cluster using a consistent hashing algorithm, the method further includes: parsing the original files to obtain the source characteristics of the original files; determining a multi-dimensional weighting algorithm for the original files according to the source characteristics of the original files; generating fragmentation parameters of the original files using the multi-dimensional weighting algorithm based on the original file type of the original files and the current network state; and dynamically fragmenting the original files to be processed according to the fragmentation parameters to obtain multiple fragment data.

[0096] The original file source features characterize the file's source category, and may be specific attributes of common protocols such as Simple Storage Service (S3) and File Transfer Protocol (FTP). The current network status includes at least one of bandwidth usage, latency, jitter, and packet loss rate. Multi-dimensional weighting algorithms may include, for example, neural network algorithms or support vector machines.

[0097] In the specific implementation, the original file is parsed to obtain its original file source characteristics. A dynamic sharding parameter calculator corresponding to these original file source characteristics is selected. Different dynamic sharding parameter calculators correspond to different weights in the multi-dimensional weighting algorithm. The multi-dimensional weighting algorithm maps the original file type and the current network state to a computable weight combination and outputs sharding parameters accordingly. The original file is then sharded based on these sharding parameters to obtain multiple shard data.

[0098] In one possible implementation, the parameters of the dynamic sharding parameter calculator corresponding to the multi-dimensional weighting algorithm can be adjusted based on the successful verification results of the original files that have been sharded in the past, so as to improve the accuracy of sharding parameter determination and file sharding storage success rate.

[0099] By dynamically determining the sharding parameters based on the file source, current network status, and original file type, the flexibility of sharding and the stability of cluster processing in scenarios with massive amounts of small files are improved.

[0100] For example, Figure 3 A flowchart illustrating a file fragment storage method provided in this application embodiment. Figure 3 ,like Figure 3 As shown, the method includes: a fragmented storage stage, a consistency verification stage, and a memory optimization stage.

[0101] In the fragmented storage stage: for multiple original files to be processed, each original file is first parsed to obtain the source characteristics of the original files, and the corresponding multi-dimensional weight algorithm is determined according to the source characteristics of the original files.

[0102] By combining the original file type and the current network status, a multi-dimensional weighted algorithm is used to generate sharding parameters for each original file. Then, dynamic sharding is performed on the original files to be processed according to the sharding parameters, resulting in multiple sharded data. The working node for each sharded data is determined by a consistent hashing ring.

[0103] After receiving the allocated shard data, each worker node stores the shard data and records the corresponding metadata triplet.

[0104] Consistency verification phase: After the distributed storage of the original file fragments is completed, the pull path of each original file is determined according to the original file size, and multiple fragments of each original file are pulled in parallel through their respective pull paths.

[0105] After retrieving the sharded data, file-level and metadata-level verifications are performed on each shard.

[0106] If the verification is successful, the memory optimization phase begins. If the verification fails, the original file enters the intelligent retry queue, waiting to be re-sharded and stored.

[0107] Memory optimization phase: The original files are optimized for memory using mmap memory mapping technology and off-heap memory management. A fragment mapping table is established for each original file, storing the memory mapping region corresponding to each file, and the data in each fragment is accessed directly accordingly. After memory optimization, the original files are persistently stored.

[0108] Figure 4 This is a schematic diagram of the structure of a file fragmentation storage device provided in an embodiment of this application, as shown below. Figure 4 As shown, the file fragment storage device 40 provided in this embodiment includes: a dynamic fragmentation module 401, a consistency verification module 402, and a memory optimization module 403;

[0109] The dynamic sharding module 401 is used to distribute the shard data of multiple original files to the worker nodes in the distributed cluster using a consistent hashing algorithm, and to record the metadata triples of the shard data through the worker nodes. The multiple shard data are obtained after the original files are dynamically sharded. The consistency verification module 402 is used to pull the multiple shard data of each original file after the worker nodes have stored the shard data, and to perform consistency verification on each shard data using the pulled shard data and metadata triples. The consistency verification includes file-level verification and metadata-level verification. The memory optimization module 403 is used to perform memory mapping on each shard data of each original file after the shard data of the original file has passed the consistency verification, and to determine the shard mapping table of the original file. The shard mapping table is used to determine the memory mapping area of ​​each file so as to directly access each shard data.

[0110] In one possible implementation, the memory optimization module 403 is also used to allocate direct memory outside the Java Virtual Machine heap as a data buffer through the buffer allocator provided by the network communication framework; store the fragmented data into the data buffer; perform memory mapping on each fragmented data; and determine the fragmented mapping table of the original file.

[0111] In one possible implementation, the memory optimization module 403 is also used to map the data content of the fragmented data to the virtual address to obtain the mapping region of the fragmented data storage address; according to the logical order of each fragmented data in the original file, the region information of each mapping region is recorded to construct a fragmented mapping table.

[0112] In one possible implementation, the dynamic sharding module 401 is further used to perform hash calculation on the unique identifier of each shard data to obtain a first hash value; based on the first hash value, the shard data is allocated to the corresponding worker nodes in the distributed cluster through a consistent hashing algorithm with virtual nodes, and there is an association relationship between the virtual nodes and the worker nodes.

[0113] In one possible implementation, the dynamic sharding module 401 is further configured to map the first hash value to a consistent hash ring pre-distributed with multiple virtual nodes, determine the initial position corresponding to the first hash value; search for the first virtual node in the consistent hash ring in a clockwise direction according to the initial position; and allocate the sharded data to the worker nodes in the distributed cluster associated with the virtual nodes.

[0114] In one possible implementation, the dynamic sharding module 401 is further used to determine the pull path of each original file according to the original file size of each original file, and the pull path carries a different number of original files; through the pull path of each original file, each original file is pulled in parallel, and the consistency of each shard data is checked.

[0115] In one possible implementation, the consistency verification module 402 is also used to perform hash calculation on the data content of the sharded data to obtain a second hash value. If the second hash value is consistent with the third hash value, the file-level verification is determined to be successful. The third hash value is calculated based on the sharded data before it is allocated to the worker node. The first metadata of the pulled sharded data is compared with the second metadata in the metadata triple stored on the worker node. If the first metadata and the second metadata are consistent, the metadata-level verification is determined to be successful.

[0116] In one possible implementation, the dynamic sharding module 401 is further configured to: parse the original files to obtain the source characteristics of the original files; determine the multi-dimensional weighting algorithm of the original files according to the source characteristics of the original files; generate sharding parameters of the original files using the multi-dimensional weighting algorithm based on the original file type of the original files and the current network state; and dynamically shard the original files to be processed according to the sharding parameters to obtain multiple shard data.

[0117] This embodiment provides a file fragment storage device that can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0118] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0119] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0120] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0121] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0122] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0123] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0124] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0125] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0126] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0127] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0128] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0129] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0130] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0131] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0132] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0133] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0134] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file fragment storage method, characterized in that, include: Using a consistent hashing algorithm, the data fragments of multiple original files are distributed to worker nodes in a distributed cluster, and the metadata triples of the data fragments are recorded by the worker nodes. The multiple data fragments are obtained by dynamically splitting the original files. After the working node has stored the shard data, it pulls multiple shard data of each original file, and performs consistency verification on each shard data by using the pulled shard data and the metadata triple. The consistency verification includes file-level verification and metadata-level verification. For each of the original files, after each of the fragments of the original file passes the consistency check, memory mapping is performed on each of the fragments to determine the fragment mapping table of the original file. The fragment mapping table is used to determine the memory mapping region of each of the files so as to directly access each of the fragments.

2. The method according to claim 1, characterized in that, The step of memory mapping each of the data segments to determine the segment mapping table of the original file includes: The buffer allocator provided by the network communication framework allocates direct memory from outside the Java Virtual Machine heap as a data buffer. The fragmented data is stored in the data buffer; Memory mapping is performed on each of the data segments to determine the segment mapping table of the original file.

3. The method according to claim 2, characterized in that, The step of memory mapping each of the data segments to determine the segment mapping table of the original file includes: The data content of the fragmented data is mapped to a virtual address to obtain the mapping area of ​​the fragmented data storage address; Based on the logical order of each fragment of data in the original file, the region information of each mapping region is recorded to construct a fragment mapping table.

4. The method according to claim 1, characterized in that, The process of distributing multiple data shards to worker nodes in a distributed cluster using a consistent hashing algorithm includes: For each of the aforementioned data shards, a hash calculation is performed on the unique identifier of the data shard to obtain a first hash value; Based on the first hash value, the sharded data is allocated to the corresponding worker nodes in the distributed cluster using a consistent hashing algorithm with virtual nodes, and there is an association between the virtual nodes and the worker nodes.

5. The method according to claim 4, characterized in that, The step of allocating the sharded data to the corresponding worker nodes in the distributed cluster based on the first hash value using a consistent hashing algorithm with virtual nodes includes: The first hash value is mapped onto a consistent hash ring pre-distributed with multiple virtual nodes to determine the initial position corresponding to the first hash value; Based on the initial position, find the first virtual node in the consistent hash ring in a clockwise direction; The sharded data is distributed to worker nodes in the distributed cluster associated with the virtual node.

6. The method according to claim 1, characterized in that, The step of retrieving multiple data fragments from each of the original files and performing consistency checks on each data fragment includes: The pull path for each original file is determined according to its original file size, and the pull path carries a different number of original files. The original files are retrieved in parallel through their respective retrieval paths, and the consistency of each shard data is verified.

7. The method according to claim 6, characterized in that, The consistency check of each of the data fragments includes: The data content of the sharded data is hashed to obtain a second hash value. If the second hash value is consistent with the third hash value, the file-level verification is determined to be successful. The third hash value is calculated based on the sharded data before the sharded data was allocated to the worker node. The first metadata of the retrieved shard data is compared with the second metadata in the metadata triple stored on the worker node. If the first metadata and the second metadata are consistent, the metadata-level verification is determined to be successful.

8. The method according to claim 1, characterized in that, Before allocating the data fragments of multiple original files to the worker nodes in the distributed cluster using the consistent hashing algorithm, the method further includes: For each of the original files, the original files are parsed to obtain the source characteristics of the original files; Based on the source characteristics of the original file, a multi-dimensional weighting algorithm is determined for the original file. Based on the original file type and the current network status, a multi-dimensional weighting algorithm is used to generate the fragmentation parameters of the original file. According to the fragmentation parameters, the original file to be processed is dynamically fragmented to obtain multiple fragment data.

9. A file fragment storage device, characterized in that, include: The dynamic sharding module is used to distribute the shard data of multiple original files to the worker nodes in the distributed cluster using a consistent hashing algorithm, and to record the metadata triples of the shard data through the worker nodes. The multiple shard data are obtained by dynamically sharding the original files. The consistency verification module is used to pull multiple fragments of each original file after the worker node has stored the fragment data, and to perform consistency verification on each fragment data by using the pulled fragment data and the metadata triple. The consistency verification includes file-level verification and metadata-level verification. The memory optimization module is used to perform memory mapping on each of the original files after the consistency check of each of the original file's fragment data, and to determine the fragment mapping table of the original file. The fragment mapping table is used to determine the memory mapping region of each file so as to directly access each of the fragment data.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images