Data processing method, device, storage medium and computer equipment

Abstract

The embodiments of the present application provide a data processing method, apparatus, storage medium and computer equipment, which store original data in the form of data blocks along with data stripes, determine the stripe information of each data stripe after storage, and store the data block storage space of each real data block in a data block storage space table, so that in the subsequent access to the original data, data access can be performed according to the stripe information and the data block storage space table, avoiding the adjustment of the disk and data structure before managing the disk, thereby improving data integrity.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, device, storage medium and computer equipment. Background Art

[0002] With the rapid development of information technology, it is very important to preserve the original data completely in high-speed real-time data acquisition application scenarios. Data storage systems usually need to have powerful high-throughput data storage capabilities. From a hardware perspective, the storage method that supports increasing or decreasing the number of removable disks is the most convenient and flexible.

[0003] In the related technology, high-throughput data storage methods include Redundant Arrays of Independent Disk (RAID) and Distributed File System (DFS). When RAID or DFS manages the disk, it usually involves formatting the disk or making large-scale data structure adjustments, thereby changing the file system format, storage layout, etc. of the disk, making the original data storage method no longer applicable, thereby destroying existing data. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, device, storage medium and computer equipment, aiming to provide a new data processing method to avoid adjusting the disk and data structure before RAID or DFS manages the disk, thereby improving data integrity. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Acquire data to be stored, and perform data striping processing on the data to be stored to obtain a plurality of data stripes, wherein the data stripes include a plurality of data blocks;

[0007] Respectively storing each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and recording data block index information of each of the data blocks, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located;

[0008] Based on the data block index information of each data block and the corresponding data block storage space, constructing stripe information of each data stripe, the stripe information including a partition identifier, a stripe identifier, a number of data blocks and a starting offset of the data blocks contained in the data stripe;

[0009] Each of the stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0011] A striping processing unit, used for acquiring data to be stored, performing data striping processing on the data to be stored, and obtaining a plurality of data stripes, wherein the data stripes include a plurality of data blocks;

[0012] A recording unit, used to store each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and record data block index information of each data block, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located;

[0013] A construction unit, configured to construct stripe information of each of the data stripes based on the data block index information of each of the data blocks and the corresponding data block storage space, wherein the stripe information includes a partition identifier, a stripe identifier, a number of data blocks, and a starting offset of the data blocks contained in the data stripe;

[0014] The storage unit is used to store each of the stripe information into the stripe information table of the partition file corresponding to the primary partition, and sequentially store the data block storage space of the data blocks whose data block storage space is not zero into the data block storage space table of the partition file corresponding to the primary partition.

[0015] In some embodiments, the recording unit includes:

[0016] An acquisition subunit is used to acquire the storage speed of the disk corresponding to each partition and the unstored number of data stripes that are not currently stored in each partition;

[0017] A first determining subunit, configured to determine an allocation weight of each partition based on a storage speed and a corresponding unstored quantity corresponding to each partition;

[0018] A second determining subunit, configured to determine the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes;

[0019] The storage subunit is used to store each of the data stripes in the data stripe table of the partition file corresponding to the different partitions according to the allocation quantity corresponding to each partition.

[0020] In some embodiments, the first determining subunit is configured to:

[0021] Get the sum of the storage speeds corresponding to each partition to get the total storage speed;

[0022] Calculate the ratio of the storage speed corresponding to each partition to the total storage speed to obtain the allocation ratio corresponding to each partition;

[0023] Determine the ratio of the unstored quantity corresponding to each partition to the corresponding storage speed, and obtain the storage time length of each partition storing the unstored data stripe;

[0024] Determine the ratio of the allocation ratio corresponding to each partition to the corresponding storage duration, and obtain the allocation weight of each partition.

[0025] In some embodiments, the second determining subunit is configured to:

[0026] Get the sum of the allocation weights of each partition to get the total allocation weight;

[0027] Determine the ratio of the allocation weight of each partition to the total allocation weight to obtain the allocation weight ratio of each partition;

[0028] Determine the product of the target number of data stripes and the allocation weight ratio of each partition to obtain a preliminary allocation number of data stripes allocated to each partition;

[0029] When the sum of the preliminary allocated quantities of each partition exceeds the target quantity, the preliminary allocated quantities of the partitions are adjusted according to the partition order between each partition to obtain the allocated quantity of the data stripes allocated to each partition.

[0030] In some embodiments, the striping processing unit is configured to:

[0031] Writing the data to be stored into a data cache, obtaining a current data volume of the data cache, and when the current data volume is greater than or equal to a preset data volume, taking out target data of the preset data volume from the data cache, and updating the current data volume;

[0032] compressing the target data to obtain a first data block, and recording a data block storage space of the first data block;

[0033] Writing the first data block into a stripe cache;

[0034] When the cache amount of the stripe cache is greater than or equal to the preset data stripe storage space, extracting a plurality of first data blocks whose sum of data block storage space is the preset data stripe storage space from the stripe cache, and the plurality of first data blocks form a first data stripe;

[0035] Acquire the updated current data amount, and when the current data amount is greater than or equal to the preset data block depth, return to execute the step of taking out target data of the preset data amount from the data cache until the current data amount is less than the preset data block depth or the current time reaches the specified end time, thereby obtaining a plurality of first data stripes;

[0036] When the updated current data amount is zero, each of the first data stripes is determined as a data stripe.

[0037] In some embodiments, the striping processing unit is further configured to:

[0038] When the current time reaches the specified end time and the current data amount is less than the preset data amount, compressing the remaining data in the data cache to obtain a second data block;

[0039] Writing the second data block into the stripe cache;

[0040] When the cache amount of the stripe cache is less than the preset data stripe storage space, calculating the difference between the preset data stripe storage space and the cache amount of the stripe cache to obtain the number of filling times;

[0041] Using a preset filling character, filling the stripe buffer according to the filling number of times;

[0042] Extracting cache content in the stripe cache, and determining the cache content as a second data stripe;

[0043] Each of the first data stripes and the second data stripes formed is determined as a data stripe.

[0044] In some embodiments, a construction unit is used to:

[0045] Traversing each of the data blocks, determining the partition identifier and the stripe identifier of the current data block as the partition identifier and the stripe identifier of the data stripe where the current data block is located;

[0046] When the data block storage space of the current data block is not zero, obtaining the current record count and the total storage space of the current data block;

[0047] Increment the current record count by one;

[0048] Determine a sum of the total storage space of the current data block and the data block storage space of the current data block, and update the total storage space of the current data block according to the sum;

[0049] When the updated total storage space of the current data block is greater than or equal to the preset stripe storage space, the updated current record count is determined as the number of data blocks in the data stripe where the current data block is located, and the current offset is determined as the starting offset of the data stripe where the current data block is located;

[0050] Reset the current record count to zero, and determine the difference between the current data block total storage space and the preset stripe storage space as the current data block total storage space and the current offset;

[0051] The next data block after the current data block is determined as the current data block, and the step of determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located is returned to be executed until each of the data blocks is traversed and the stripe information of each of the data stripes is obtained.

[0052] In some embodiments, the apparatus further comprises:

[0053] A first acquisition unit, configured to acquire information of each target stripe stored in a stripe information table of a partition file corresponding to the primary partition in response to a data restoration instruction;

[0054] A second acquisition unit, configured to acquire, for each target stripe information, a target partition identifier and a target stripe identifier included in the target stripe information;

[0055] A first determining unit, configured to determine a target partition corresponding to the target partition identifier, and extract a target data stripe corresponding to the target stripe identifier from a data stripe table of a partition file corresponding to the target partition;

[0056] A second determining unit, configured to determine other target stripe information in the stripe information table that is before the target stripe information;

[0057] A third determining unit, configured to determine a target number of a first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information;

[0058] A fourth determining unit, configured to determine a target data block storage space of the data block corresponding to the target number from the data block storage space table;

[0059] A fifth determining unit is configured to determine original data based on a target start offset included in the target stripe information, a target stripe storage space of the target data stripe, the target number, and the data block storage space table.

[0060] In some embodiments, the third determining unit is configured to:

[0061] Determine the sum of the number of data blocks included in each of the other target stripe information to obtain the target number of data blocks;

[0062] The sum of the number of target data blocks and one is determined as the target number of the first target data block included in the target data stripe.

[0063] In some embodiments, the fifth determining unit is configured to:

[0064] When the target start offset included in the target stripe information is not zero, extracting a third data block from the target data stripe according to the target start offset;

[0065] Determine a previous data block to be merged whose number is before the target number, and merge the data block to be merged with the third data block to obtain a complete data block;

[0066] Decompressing the complete data block to obtain part of original data corresponding to the complete data block;

[0067] Determine a target data block storage space numbered with the target number from the data block storage space table;

[0068] When the remaining stripe storage space of the target data stripe is greater than or equal to the target data block storage space, extracting the target data block corresponding to the target number from the target data stripe according to the target data block storage space;

[0069] Decompressing the target data block to obtain a portion of original data corresponding to the target data block;

[0070] Calculate the sum of the target number and one, determine the sum as the target number, and return to the step of determining the target data block storage space of the data block corresponding to the target number from the data block storage space table until the stripe storage space of the target data stripe is smaller than the target data block storage space, thereby obtaining a plurality of partial original data;

[0071] The multiple parts of the original data are combined to obtain the original data.

[0072] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute any of the above data processing methods.

[0073] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above data processing methods when executing the computer program.

[0074] In an embodiment of the present application, by acquiring data to be stored, data striping processing is performed on the data to be stored to obtain multiple data stripes, and the data stripes include multiple data blocks; each of the data stripes is stored in a data stripe table of a partition file corresponding to a different partition, and data block index information of each data block is recorded, the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located; based on the data block index information of each data block and the corresponding data block storage space, stripe information of each data stripe is constructed, the stripe information includes a partition identifier, a stripe identifier, a number of data blocks and a starting offset of the data blocks contained in the data stripe; each of the stripe information is stored in a stripe information table of a partition file corresponding to a primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence. In this way, by storing the original data in the form of data blocks along with data stripes, determining the stripe information of each data stripe after storage, and storing the data block storage space of each real data block in the data block storage space table, in the subsequent access to the original data, storage management software such as RAID or DFS can access the data based on the stripe information and the data block storage space table, avoiding the adjustment of the disk and data structure before RAID or DFS manages the disk, thereby improving data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A schematic diagram of a data processing system according to an embodiment of the present invention.

[0077] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application.

[0078] Figure 3 A schematic diagram of a partition file for a primary partition provided in an embodiment of the present application.

[0079] Figure 4 A schematic diagram of a multi-disk parallel storage data storage format provided in an embodiment of the present application.

[0080] Figure 5A schematic diagram of the structure of a data processing device provided in an embodiment of the present application.

[0081] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0083] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0084] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0085] Redundant Array of Independent Disks (RAID) is a technology that combines multiple independent hard disks (physical hard disks) in different ways to form a hard disk group (logical hard disk), thereby providing higher storage performance and data redundancy than a single hard disk.

[0086] The main functions and features of RAID include: 1. Improve storage performance by dividing data into stripes and distributing them on multiple disks. This allows data to be read and written in parallel, greatly improving the data storage speed. For example, in the RAID0 level, data is divided into stripes and written to different disks respectively, and can also be read from multiple disks at the same time when reading, thereby significantly improving the data transmission rate. Since the data is scattered on multiple disks, for random read and write operations, it can seek and read from multiple disks at the same time, reducing the seek time and rotation delay of a single disk and improving random read and write performance. 2. Provide data redundancy, by mirroring (RAID1), the data is completely copied to another disk. When one disk fails, the other disk can immediately take over the work to ensure data availability. Although this method will take up more disk space, it provides the highest data reliability. The parity information of the data is calculated through parity (RAID5, RAID6, etc.) and stored on multiple disks in a distributed manner. When a disk fails, the data on the failed disk can be restored through the data and parity information on other disks. RAID5 requires at least three disks, one of which is used to store parity information. RAID6 can tolerate two disk failures at the same time. 3. Scalability: New disks can be easily added to the RAID array to expand storage capacity or improve performance. For example, in some enterprise-level storage environments, as the amount of data continues to grow, disks can be gradually added to the RAID array without the need for large-scale upgrades to the entire storage system.

[0087] A distributed file system (DFS) is a file system that stores files in multiple nodes (servers or storage devices) in a dispersed manner, connects these nodes through a network, and provides users with a unified file access interface, allowing users to access files in the distributed file system just like accessing a local file system.

[0088] The main functions and features of DFS include: 1. High reliability. Distributed file systems usually copy files to multiple nodes to prevent data loss due to a single node failure. For example, Hadoop's distributed file system HDFS copies each file block to multiple data nodes, with three copies by default. When a data node fails, the system can automatically recover data from other copies to ensure data availability. In addition, distributed file systems can automatically detect node failures and perform failover and recovery. If a node fails, the system will automatically redistribute tasks on the node to other normal nodes to ensure the continuous operation of the system. 2. High scalability. New storage nodes can be easily added to expand storage capacity and performance. As the amount of data grows and business needs change, new servers or storage devices can be continuously added to the distributed file system without large-scale reconstruction of the entire system. Distributed file systems can automatically balance the load between nodes and distribute files to different nodes for storage and access, avoiding excessive load on a node affecting system performance. Efficient operation of the system is achieved by dynamically adjusting the storage location and access path of files. 3. High performance, supporting multiple clients to access files at the same time, can realize parallel data reading and writing operations, and improve the throughput and response speed of the system. For example, in the big data processing scenario, multiple computing nodes can read data from the distributed file system for processing at the same time, which greatly improves the efficiency of data processing. The distributed file system will try to store files close to the nodes that use it to improve the access speed of data. For example, in the Hadoop ecosystem, MapReduce tasks will be preferentially run on nodes that store input data, reducing the network transmission overhead of data and improving the execution efficiency of tasks. 4. Unified namespace, providing users with a unified file namespace, users can access files in the distributed file system through a unified path without worrying about which node the file is actually stored on. This unified namespace makes the distributed file system easy to manage and use, and users can operate files in the distributed file system like using a local file system.

[0089] Currently, initialization configuration before RAID or DFS manages the disk will change the disk's file system format, storage layout, etc., thereby destroying existing data. This is because the initialization process usually involves formatting the disk or making large-scale data structure adjustments, making the original data storage method no longer applicable. Therefore, a data processing method is urgently needed to solve the above technical problems.

[0090] In order to solve the above-mentioned problem, the embodiment of the present application proposes to obtain data to be stored, perform data striping processing on the data to be stored, and obtain multiple data stripes, wherein the data stripes include multiple data blocks; each of the data stripes is stored in a data stripe table of a partition file corresponding to a different partition, and data block index information of each data block is recorded, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located; based on the data block index information of each data block and the corresponding data block storage space, stripe information of each data stripe is constructed, wherein the stripe information includes a partition identifier, a stripe identifier, the number of data blocks, and a starting offset of the data blocks contained in the data stripe; each of the stripe information is stored in a stripe information table of a partition file corresponding to a primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence. In this way, by storing the original data in the form of data blocks along with data stripes, determining the stripe information of each data stripe after storage, and storing the data block storage space of each real data block in the data block storage space table, in the subsequent access to the original data, storage management software such as RAID or DFS can access the data according to the stripe information and the data block storage space table, avoiding the need to adjust the disk and data structure before RAID or DFS manages the disk, thereby improving data integrity. Please continue to refer to the following specific embodiments for details.

[0091] See also Figure 1 , Figure 1 A schematic diagram of a data processing system provided in an embodiment of the present application, which includes a terminal 140, the Internet 130, a gateway 120, a computer device 110, etc.

[0092] The terminal 140 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. In addition, it can be a single device or a collection of multiple devices. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0093] Computer equipment refers to a computer system that can provide certain services to the terminal 140. Compared with ordinary terminals 140, computer equipment 110 has higher requirements in terms of stability, security, performance, etc. Computer equipment 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0094] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the computer device 110 must be sent to the corresponding computer device 110 through the gateway 120. The message sent by the computer device 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0095] The data transmission method according to the embodiment of the present disclosure may be implemented on the computer device 110 .

[0096] It should be noted that Figure 1 The scenario diagram of the data processing system shown is merely an example. The data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of data processing technology and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0097] For details, please refer to Figure 2 , Figure 2 A flow chart of a data processing method provided in an embodiment of the present application. The data processing method is applied to a verification device and includes:

[0098] In step 201, data to be stored is obtained, and data striping is performed on the data to be stored to obtain a plurality of data stripes, each of which includes a plurality of data blocks.

[0099] The data to be stored is data that needs to be stored in the storage system of the electronic device, which can be various types of data, such as files, database records, etc. The data to be stored can be data that the user wishes to store in the storage system through an external device (such as a mobile hard disk, a USB flash drive, etc.), or data sent by one or more terminals and received by the electronic device in a network transmission scenario, which is not limited here. The data stripe is a logical unit into which the data to be stored is divided, and is composed of multiple data blocks. The data block is the smallest storage unit in the data stripe, which is obtained by compressing part of the data to be stored, and the compression method is lossless compression.

[0100] Specifically, the data to be stored is divided into smaller data stripes to better manage storage and improve the parallel access capability of data.

[0101] In some implementations, obtaining data to be stored, and performing data striping processing on the data to be stored to obtain a plurality of data stripes includes:

[0102] (1) writing the data to be stored into a data cache, obtaining a current data volume of the data cache, and when the current data volume is greater than or equal to a preset data volume, taking out target data of the preset data volume from the data cache, and updating the current data volume;

[0103] (2) compressing the target data to obtain a first data block, and recording a data block storage space of the first data block;

[0104] (3) writing the first data block into the stripe cache;

[0105] (4) when the cache amount of the stripe cache is greater than or equal to the preset data stripe storage space, extracting from the stripe cache a plurality of first data blocks whose sum of data block storage space is equal to the preset data stripe storage space, and the plurality of first data blocks form a first data stripe;

[0106] (5) obtaining the updated current data amount, and when the current data amount is greater than or equal to the preset data block depth, returning to the step of retrieving the target data of the preset data amount from the data cache, until the current data amount is less than the preset data block depth, or the current time reaches the specified end time, thereby obtaining a plurality of first data stripes;

[0107] (6) When the updated current data amount is zero, each of the first data stripes is determined as a data stripe.

[0108] The specific implementation method of striping the stored data to obtain multiple data stripes can refer to the following code:

[0109] while ≥presetBDepth;

[0110] blk=pop( ,presetBDepth);

[0111] if PTABLE[comp]≠None;

[0112] blk=compress(blk,PTABLE[comp]);

[0113] end if

[0114] bIndex[compSize]=||blk||;

[0115] ;

[0116] ;

[0117] if || |≥

[0118] ;

[0119] end

[0120] end while

[0121] Among them, by putting the data to be stored into the data cache at the current moment , read The current amount of data in the data to form a data block. The preset data volume (presetBDepth) is the pre-set data block depth required to form a data block. When the amount of data is greater than or equal to the preset amount, it indicates that the amount of data in the data cache can form at least one data block. Then, the target data of the preset amount of data is retrieved from the data cache in a manner of fetching from front to back. blk=pop( , presetBDepth); where pop(a,b) means taking b elements from a from the front to the back and returning them as an element sequence, blk is the target data, since it is from the data cache If the target data is extracted from the data, the current data volume needs to be updated so that data can be extracted according to the current data volume in the future.

[0122] When the preset compression method PTABLE[comp] is not empty (None), the target data blk is compressed according to the preset compression method to obtain the first data block. compress(a,b) means using method b to compress a, and returns the compressed data, and records the data block storage space bIndex[compSize] =||blk|| of the first data block, wherein the norm calculation of the target data blk is to determine the data block storage space (that is, the data block size, such as the number of bytes occupied) of the target data blk corresponding to the first data block.

[0123] Among them, the preset data stripe storage space The stripe depth required to form a data stripe is preset. When each first data block is obtained, each first data block is written into the stripe cache corresponding to the current moment ( ), when the stripe cache The amount of cache Greater than or equal to the preset data stripe storage space When the stripe cache The sum of the data block storage space extracted from the preset data stripe storage space The first data blocks are multiple, and the first data blocks form a first data stripe ( ). bIndex is the data block index information set corresponding to the current moment The index information in a[b] means to get the b element in a. b means adding b as a whole to a. b means add b to a in an element-by-element manner.

[0124] In this way, the updated total amount of data, i.e., the current amount of data, is obtained again by looping the data cache. Check whether the current amount of data is greater than or equal to the preset data block depth presetBDepth. If so, repeat steps (1) to (4), continuously fetching target data from the data cache, compressing it, writing it into the stripe cache, and forming data stripes, until the remaining amount of data is less than the preset data block depth, or the current time reaches the specified end time ( ), the specified end time is the end time manually input by the user when requesting to store the data to be stored, or the time when the user's termination storage instruction is received during the storage of the data to be stored, which is not limited here.

[0125] For example, the preset data block depth is 30, and the current data volume is 40. At this time, steps (1) to (4) are executed again to continue processing the remaining data to be stored in the data cache until the remaining data volume is less than 30.

[0126] When the updated current data amount is zero, it indicates that all the data to be stored have been processed. Then each first data stripe formed previously is determined as a data stripe.

[0127] In addition, when the updated current data volume is less than the preset data block depth, the remaining data in the data cache is retained until the next data storage request is received, and the remaining data is processed together, that is, Assign to , .

[0128] In some embodiments, the method further comprises:

[0129] (7) When the current time reaches the specified end time and the current data amount is less than the preset data amount, compressing the remaining data in the data cache to obtain a second data block;

[0130] (8) writing the second data block into the stripe cache;

[0131] (9) When the cache capacity of the stripe cache is less than the preset data stripe storage space, calculating the difference between the cache capacity of the preset data stripe storage space and the cache capacity of the stripe cache to obtain the number of fill times;

[0132] (10) using a preset filling character to fill the stripe cache according to the filling number;

[0133] (11) extracting cache content in the stripe cache, and determining the cache content as a second data stripe;

[0134] (12) Determine each of the first data stripes and the second data stripes formed as a data stripe.

[0135] The specific implementation method of forming data stripes can refer to the following code:

[0136] if

[0137] if

[0138] ;

[0139] end if

[0140] ;

[0141] ;

[0142] ;

[0143] else

[0144] ;

[0145] end if

[0146] if

[0147] ;

[0148] ;

[0149] end if

[0150] in, If it is empty, the current time reaches the specified end time ( ), and the current data amount is less than the preset data amount, it means that the remaining data in the data cache can no longer fill up a complete data block. At this time, the remaining data in the data cache is compressed and converted into a second data block for subsequent processing. This step ensures that the remaining data can be processed as effectively as possible even when the data amount is insufficient.

[0151] Specifically, That is, to cache the data The remaining data in the Compress to obtain the second data block, and record the data block storage space of the second data block. ) is not empty, indicating that there are data blocks inside, then calculate the preset data stripe storage space Cache size with stripe cache The difference between the values ​​of a and b is used to obtain the number of padding times. In order to form a complete data stripe from the data blocks that cannot form a data stripe in the stripe cache, it is necessary to fill the stripe cache with a preset padding character (such as "0") according to the number of padding times; padding(a,b,c) means using element c to append b times to the end of a and return the appended result. In particular, if b is 0, a is returned directly. When the padding is completed, the stripe cache is extracted ( ), determining the cache content as a second data stripe; and determining each of the first data stripe and the second data stripe formed as a data stripe.

[0152] Specifically, if the current time reaches the specified end time ( ), and when the current data amount is greater than or equal to the preset data amount, the step of retrieving the target data of the preset data amount from the data cache is returned to be executed until the current data amount is less than the preset data block depth. At this time, if the current data amount is not 0, the step of compressing the remaining data in the data cache to obtain the second data block is executed, so that all the data in the data cache are compressed into the data block.

[0153] In step 202, each data stripe is stored in a data stripe table of a partition file corresponding to a different partition, and data block index information of each data block is recorded. The data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located.

[0154] In the case where the storage system includes multiple partitions, each data stripe obtained in step 201 can be stored in a data stripe table of a partition file corresponding to a different partition. In the data stripe table, since the data blocks are stored in the data stripe table along with the data stripe in which they are located, the partition identifier of the partition in which each data block is located and the stripe identifier of the data stripe in which the data block is located can be recorded.

[0155] Specifically, the index information of the data block can be recorded by referring to the following code:

[0156] for i from 1 to n

[0157] if

[0158] for j from 1 to

[0159] ;

[0160] ;

[0161] for each data block

[0162] ;

[0163] ;

[0164] end for

[0165] end for

[0166] ;

[0167] end if

[0168] end for

[0169] Where n is the number of partitions, is the number of data stripes of partition i at time t, from each data stripe Extract a data stripe and assign it to data stripe j ( ), for each data block u in data stripe j, the data stripe j ( ) The partition identifier "i" of the partition i stored in is recorded as the partition identifier of each data block u, and the current number of data stripes in the data stripe table of partition i As the stripe identifier cID of each data block u. And update the number of data stripes in the data stripe table of partition i .

[0170] In some implementations, storing each of the data stripes in a data stripe table of a partition file corresponding to a different partition includes:

[0171] (1) Obtain the storage speed of the disk corresponding to each partition and the number of unstored data stripes currently not stored in each partition;

[0172] (2) Determine the allocation weight of each partition based on the storage speed and the corresponding unstored quantity of each partition;

[0173] (3) determining the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes;

[0174] (4) According to the allocation quantity corresponding to each partition, each of the data stripes is stored in the data stripe table of the partition file corresponding to the different partitions.

[0175] Among them, since different partitions may belong to different disks, and different disks have different storage speeds when storing data, the corresponding number of data stripes can be allocated to different partitions according to the storage speed of the disks corresponding to different partitions, so as to achieve dynamic balancing of each data stripe to multiple disks for parallel writing of the stripe data tables of each partition file.

[0176] Specifically, all partitions are traversed, and for each partition, the storage speed of the corresponding disk can be obtained by querying the relevant information of the storage system or using a specific tool. The storage speed can be expressed by the amount of data that can be written per unit time, such as the number of bytes written per second.

[0177] At the same time, the number of data stripes that are not currently stored in each partition is recorded. The number of unstored data stripes can be determined by checking the data stripe table of the partition file or maintaining a special counter. According to the storage speed corresponding to each partition and the corresponding number of unstored data stripes, the allocation weight of partition i at the current time t is determined. , thereby obtaining the allocation weight of each partition; and determining the allocation quantity of partition i according to the allocation weight of each partition and the target number of data stripes, for example Finally, each of the data stripes is stored in the data stripe table of the partition file corresponding to the different partitions according to the allocation quantity corresponding to each partition.

[0178] In some implementations, determining the allocation weight of each partition based on the storage speed and the corresponding unstored quantity corresponding to each partition includes:

[0179] (2.1) Obtain the sum of the storage speeds corresponding to each partition to obtain the total storage speed;

[0180] (2.2) Calculating the ratio of the storage speed corresponding to each partition to the total storage speed, and obtaining the allocation ratio corresponding to each partition;

[0181] (2.3) determining a ratio of the unstored quantity corresponding to each partition to the corresponding storage speed, and obtaining a storage duration for each partition to store the unstored data stripe;

[0182] (2.4) Determine the ratio of the allocation ratio corresponding to each partition to the corresponding storage duration, and obtain the allocation weight of each partition.

[0183] Taking the calculation of the allocation ratio of partition i as an example, the allocation ratio can be calculated by the following formula:

[0184]

[0185] in, is the storage speed corresponding to partition i, and the denominator is the sum of the storage speeds of each partition from partition 1 to partition n, which is the total storage speed. The ratio of the total storage speed to the allocation ratio of partition i can be obtained. .

[0186] The number of unstored data stripes is the number of unstored data stripes belonging to historical moments that are still not stored in each partition at the current time t. Taking partition i as an example, the number of unstored data stripes corresponding to it is , the allocation weight can be calculated by the following formula:

[0187]

[0188] Among them, partition i stores unstored data stripes The storage duration is, is a small positive number to prevent the division by zero problem. By calculating the allocation ratio and storage duration corresponding to partition i Ratio (ignore ), thereby obtaining the allocation weight corresponding to partition i. In this way, the allocation weight corresponding to each partition can be calculated.

[0189] In some implementations, determining the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes includes:

[0190] (3.1) Obtain the sum of the allocation weights of each partition to obtain the total allocation weight;

[0191] (3.2) Determine the ratio of the allocation weight of each partition to the total allocation weight to obtain the allocation weight ratio of each partition;

[0192] (3.3) determining the product of the target number of data stripes and the allocation weight ratio of each partition, and obtaining the preliminary allocation number of data stripes allocated to each partition;

[0193] (3.4) When the sum of the preliminary allocation quantities of each partition exceeds the target quantity, the preliminary allocation quantities of the partitions are adjusted according to the partition order between each partition to obtain the allocation quantity of the data stripes allocated to each partition.

[0194] Taking the determination of the allocation quantity of partition i as an example, the allocation quantity of each partition can be determined by referring to the following formula:

[0195]

[0196] in, That is, the sum of the allocation weights for each partition, that is, the total allocation weight, calculates the allocation weight of partition i The ratio of the target number of data stripes to the total allocation weight is used to obtain the allocation weight ratio of partition i. Then the product of the target number of data stripes and the allocation weight ratio of partition i is calculated to obtain the preliminary allocation number of data stripes allocated to partition i. round() represents the rounding function.

[0197] Specifically, since round() is used to determine the preliminary allocation quantity of each partition, the sum of the preliminary allocation quantities of each partition may exceed the target quantity. In this case, the preliminary allocation quantity of each partition can be adjusted by sorting the partitions between each partition to obtain the allocation quantity of the data stripes allocated to each partition.

[0198] The method of adjusting the initial allocation number of partitions can be implemented by referring to the following code:

[0199] while

[0200] for i from n to 1

[0201] if

[0202] ;

[0203] break;

[0204] end if

[0205] end for

[0206] end while

[0207] Among them, each partition is traversed in order from back to front, and for each partition i, check >0, that is, whether the partition is assigned to a stripe unit. If the partition is assigned to a stripe unit, The value of is reduced by 1, which means reducing the number of allocations for the partition. Then use the break statement to jump out of the inner loop and start the judgment of the next outer loop.

[0208] For example, there are three partitions (n=3), and in the initial state:

[0209] is 10, that is, the stripe unit set requested to be written at the current moment has 10 stripe units. Assume that the initial =5, =4, =3. Then in the first outer loop: =5+4+3=12, greater than || ||=10, the loop condition is met. Enter the inner loop and start checking from i=3. When i=3, =3>0, Subtract 1, become 2, and then jump out of the inner loop. At this time, =5, =4, =2.

[0210] Second outer loop:

[0211] =5+4+2=11, still greater than 10, continue the loop. Enter the inner loop again and start checking from i=3. When i=3, =2>0, Subtract 1 again, it becomes 1, and then jumps out of the inner loop. At this time, ==5, =4, = 1. Continue looping until equals|| || until now.

[0212] In some implementations, before storing each of the data stripes in the data stripe table of the partition file corresponding to the different partitions and recording the data block index information of each of the data blocks, the method further includes:

[0213] Obtain disk information of target disks whose disk type is a specified disk type and whose remaining capacity is greater than or equal to a preset capacity among the mounted disks, wherein each of the disk information includes a disk number dID, a corresponding root directory root in the file system, and a read / write speed s;

[0214] In response to a selection operation on disk information, determining selected target disk information;

[0215] Get the expected throughput and the specified primary partition storage path;

[0216] Determine the disk identifier of the disk corresponding to each partition, the corresponding partition identifier, and the corresponding partition path according to the expected throughput, the primary partition storage path, and the information of each disk;

[0217] The disk identifier and the partition identifier corresponding to each partition are written into the corresponding partition information table, and the disk identifier and the partition path corresponding to each partition are written into the partition table corresponding to the primary partition.

[0218] Among them, the information set DINFO of the available disks of the current system is obtained. First, the basic information of the currently mounted disk is obtained: disk number dID, available capacity vol, disk type p (p=r or w, r means that the disk is readable, w means that the disk is readable and writable), and file system root directory root. Disks that satisfy the conditions of vol≥presetV (presetV is an adjustable preset capacity value) and disk type p=w (the readable and writable disk type is determined to be the specified disk type) are called available disks. Then, the storage speed s of the available disk is searched according to the disk number from the global configuration file config. If not found, the write speed of the disk is tested and the test results are written to config. Next, the storage speed s of the disk is merged with some basic information to obtain the disk information set dInfo={dID,root,s}. Assuming there are n available disks, the information set of the i-th available disk is , where i∈[1,n]. Finally, sort the n disk information sets in descending order according to the writing speed, and obtain the available disk information set DINFO={ }(i=1~n).

[0219] Specifically, when the user needs to partition the disk, the expected maximum storage throughput reqT and the primary partition storage path mPath are entered. For the primary partition, the dID and pPath of the primary partition are added to the pTable of PTABLE: ; Indicates the disk information of the disk where the primary partition is located, thereby constructing the partition information table of the primary partition: ; , where main is the partition identifier of the primary partition.

[0220] For each secondary partition, you can refer to the following code to determine the disk ID, partition ID, and partition path of each partition, and write the disk ID and partition ID corresponding to each partition into the corresponding partition information table, and write the disk ID and partition path corresponding to each partition into the partition table corresponding to the primary partition.

[0221] if reqT<0

[0222] for each ∈DINFO(i≠main)

[0223] ; (concat represents the element concatenation function, pdrPath represents the relative path of the directory used to store the secondary partition files (relative to the root directory of the disk))

[0224] ;

[0225] end for

[0226] else

[0227] totalT= ;

[0228] for each ∈ DINFO (i≠main)

[0229] if totalT ≥ reqT

[0230] break;

[0231] end if

[0232]

[0233] ;

[0234] totalT=totalT+ ;

[0235] end for

[0236] if totalT <reqT

[0237] Error message; Report an error

[0238] end if

[0239] end if

[0240] Update pCount of PTABLE: PTABLE[pCount]=||PTABLE[pTable]||; (||a|| means getting the number of elements of a (the same below))

[0241] Update the comp of PTABLE: PTABLE[comp]=presetComp; (presetComp indicates the lossless compression method, which is configurable)

[0242] According to PTABLE, generate PINFO for all sub-partitions. The generation method is as follows:

[0243] for each

[0244] ;

[0245] ; (secondary means secondary partition)

[0246] end for

[0247] Create corresponding empty partition files for all partitions in PTABLE[pTable] and convert them into Write the empty partition file of partition i (i=1,…,PTABLE[pCount]);

[0248] Write PTABLE to the primary partition file;

[0249] Initialize the data stripe table for partition i (i=1,…,PTABLE[pCount]), and Write the partition file for partition i. The initialization method is as follows:

[0250] ; (presetCDepth indicates the strip depth, which is configurable)

[0251] ;

[0252] ;

[0253] Specifically, if the user expects to use the maximum storage space, the expected throughput reqT is not input, so that reqT<0. For each secondary partition, the disk information list DINFO in the storage system is traversed (excluding the disk information corresponding to the primary partition).

[0254] Disk information for each sub-partition , by concatenating the root directories of the disks , relative path pdrPath and globally unique identifier gID, get the partition path ,Right now =concat( ,pdrPath,gID).

[0255] Identify the disk and partition path Write into the partition table pTable of the partition table set PTABLE corresponding to the main partition, that is, PTABLE[pTable].

[0256] For the case where the user inputs the expected throughput reqT, that is, reqT > 0, first assign the storage speed s in the disk information corresponding to the main partition to totalT, that is, totalT = .

[0257] Traverse the list of disk information DINFO in the storage system (excluding the disk information corresponding to the main partition).

[0258] During the traversal, if totalT ≥ reqT, it means that the total storage speed of the currently traversed partition reaches the expected throughput reqT, and the loop is exited.

[0259] If the exit condition is not met, perform operations similar to the case where reqT < 0, that is, concatenate the partition path , write the disk identifier and the partition path into PTABLE[pTable], and update the value of totalT to totalT = totalT + .

[0260] If finally totalT < reqT, it means an error occurs, and an error message is output.

[0261] Update the partition table related information:

[0262] Update pCount of PTABLE, that is, the number of partitions, by obtaining the number of elements of PTABLE[pTable], that is, PTABLE[pCount] = ||PTABLE[pTable]||.

[0263] Set the compression method of PTABLE to the preset lossless compression method presetComp, that is, PTABLE[comp] = presetComp.

[0264] Generate secondary partition information and write it into the partition file:

[0265] Traverse each partition in the partition table PTABLE[pTable] of the partition table set except for the first partition that serves as the main partition.

[0266] For each secondary partition, set the secondary partition information The disk identifier of is the disk identifier of the corresponding partition, that is, = , and set the flag to secondary partition, that is, [mFlag] = secondary.

[0267] Create a corresponding empty partition file for each partition and convert it into Write partition file.

[0268] Write PTABLE to the primary partition file.

[0269] Initialize the partitioned data stripe table:

[0270] For each partition (indexed from 1 to PTABLE[pCount]), initialize the data stripe table .

[0271] Set the strip depth to the preset value presetCDepth, that is .

[0272] Set the number of data stripes to 0, that is, .

[0273] Set the data stripe table content to empty, that is .

[0274] After initialization Write the corresponding partition file.

[0275] In step 203, based on the data block index information of each data block and the corresponding data block storage space, the stripe information of each data stripe is constructed, and the stripe information includes the partition identifier, stripe identifier, data block quantity and starting offset of the data blocks contained in the data stripe.

[0276] In step 202, after obtaining the index information of each data block through recording, the stripe information of each data stripe is determined through the index information of each data block, and the stripe information specifically includes the partition identifier, stripe identifier, number of data blocks and starting offset of the data blocks included in the data stripe. The starting offset refers to the offset of the first data block relative to the starting position of the data stripe among the multiple data blocks included in the data stripe.

[0277] Specifically, since each data block in the data stripe is stored along with the data stripe, the partition identifier and stripe identifier of each data block are the same and can be directly used as the partition identifier and stripe identifier of the data stripe to which it belongs.

[0278] In some implementations, constructing stripe information of each of the data stripes based on the data block index information of each of the data blocks and the corresponding data block storage space includes:

[0279] (1) traversing each of the data blocks, and determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located;

[0280] (2) When the data block storage space of the current data block is not zero, obtaining the current record count and the total storage space of the current data block;

[0281] (3) Increment the current record count by one;

[0282] (4) determining a sum of the total storage space of the current data block and the data block storage space of the current data block, and updating the total storage space of the current data block according to the sum;

[0283] (5) when the total storage space of the updated current data block is greater than or equal to the preset stripe storage space, the updated current record count is determined as the number of data blocks in the data stripe where the current data block is located, and the current offset is determined as the starting offset of the data stripe where the current data block is located;

[0284] (6) resetting the current record count to zero, and determining the difference between the current data block total storage space and the preset stripe storage space as the current data block total storage space and the current offset;

[0285] (7) Determine the next data block after the current data block as the current data block, and return to execute the step of determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located, until each of the data blocks is traversed and the stripe information of each of the data stripes is obtained.

[0286] The stripe information of the data stripe can be determined by referring to the following code:

[0287] for each ∈

[0288] index[pID]= ;

[0289] ;

[0290] if

[0291] index[bCount]=index[bCount]+1;

[0292] ;

[0293] end if

[0294] total=total+ ;

[0295] if total ≥

[0296] index[start]=offset;

[0297] ;

[0298] ;

[0299] offset=total;

[0300] index[bCount]=0;

[0301] end if

[0302] end for

[0303] if total>0

[0304] ;

[0305] end if

[0306] Update BINDEX's iCount: BINDEX[iCount]=||BINDEX[iTable]||;

[0307] Update bCount of BDEPTH: BDEPTH[bCount]=||BDEPTH[bdTable]||;

[0308] in, From time t to time The index information set composed of the index information of each data block in for The index information of a data block in is the data block storage space for data block i, Current record count, The data block storage space table for the data block storage space domain of the partition file corresponding to the primary partition. is the total storage space of the current data block, For the preset stripe storage space, The number of stripe information in the stripe information table of the partition file corresponding to the primary partition, and BDEPTH[bCount] is the number of data block storage spaces in the data block storage space domain of the partition file corresponding to the primary partition, that is, the number of data blocks.

[0309] Specifically, since the target data is taken out from the data cache in a front-to-back order to be compressed to form the data block, the formed data blocks are also formed in a front-to-back order, resulting in that when forming the data stripe, the data blocks forming the data stripe are also in order. Therefore, each data block can be traversed in the order of the data blocks, and the index information set of the data block can be traversed. Each data block index information bIndex in the data block is extracted to extract the partition identifier pID and the stripe identifier cID. These identifiers will be used to determine the identification information of the data stripe to which the current data block belongs. The purpose of this is to accurately associate the data block with the data stripe to which it belongs in the future, so as to perform more detailed stripe information statistics and management.

[0310] For example, there are three data block index information, namely {pID:1,cID:2,compSize:10}, {pID:1,cID:2,compSize:20} and {pID:2,cID:3,compSize:15}. When traversing the first data block, the partition identifier 1 and the stripe identifier 2 are determined as the partition identifier and the stripe identifier of the data stripe where the data block is located.

[0311] Check the storage space size of the current data block ( ) is greater than zero. If so, it means that this data block actually occupies storage space and needs further processing. At this time, obtain the current record count (used to count the number of data blocks that have been processed) and the total storage space of the current data block (an accumulator used to count the total storage space of all processed data blocks). Increase the current record count by one, indicating that a new data block has been processed. This record count will be used to subsequently count the number of data blocks in the data stripe. Add the storage space size of the current data block to the total storage space of the current data block to obtain the new total storage space. In this way, the storage space of the processed data blocks can be continuously accumulated, so as to determine whether the preset stripe storage space is reached later. Check whether the updated total storage space of the current data block is greater than or equal to the preset stripe storage space. If so, it means that enough data blocks have been accumulated to form a data stripe. At this time, determine the current record count as the number of data blocks in this data stripe, and determine the current offset as the starting offset of the first data block in the data stripe. The starting offset can be used to quickly locate the starting position of the data block in the storage system. After a data stripe is determined, preparations need to be made for the statistics of the next data stripe. The current record count is reset to zero so that the number of data blocks in the next data stripe can be counted again. At the same time, the difference between the total storage space of the current data block and the preset stripe storage space is determined as the new total storage space of the current data block, and the current offset is updated. This ensures that the statistics of the next data stripe start from the correct position. After processing a data block, the next data block is used as the current data block, and steps (1) to (6) are repeated until all data blocks have been traversed. In this way, the stripe information of each data stripe can be gradually counted, including the partition identifier, stripe identifier, number of data blocks, and starting offset.

[0312] In step 204, each stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition.

[0313] After determining the stripe information of each data stripe in step 203, each stripe information can be stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data block whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence, and finally the following is formed: Figure 3 The partition file of the primary partition shown in the figure lacks the partition table set and index information table compared to the partition file of the primary partition, and the partition information table is filled with the .

[0314] In some implementations, after storing each of the stripe information in the stripe information table of the partition file corresponding to the primary partition, and sequentially storing the data block storage space of the data blocks whose data block storage space is not zero in the data block storage space table of the partition file corresponding to the primary partition, the method further includes:

[0315] (1) In response to a data restoration instruction, obtain information about each target stripe stored in a stripe information table of a partition file corresponding to the primary partition;

[0316] (2) for each target stripe information, obtaining a target partition identifier and a target stripe identifier included in the target stripe information;

[0317] (3) determining a target partition corresponding to the target partition identifier, and extracting a target data stripe corresponding to the target stripe identifier from a data stripe table of a partition file corresponding to the target partition;

[0318] (4) determining other target stripe information that is before the target stripe information in the stripe information table;

[0319] (5) determining the target number of the first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information;

[0320] (6) determining the target data block storage space of the data block corresponding to the target number from the data block storage space table;

[0321] (7) Determine the original data based on the target start offset, the target stripe storage space of the target data stripe, the target number, and the data block storage space table included in the target stripe information.

[0322] When the data restoration instruction is received, the system starts to prepare for the data recovery operation. First, all the stripe information is obtained from the stripe information table of the partition file corresponding to the primary partition. These stripe information describe the properties of each data stripe, such as partition ID, stripe ID, number of data blocks, and starting offset. This information will be used in the subsequent data restoration process. For example, suppose that the stripe information table of the primary partition stores multiple stripe information, such as {partition ID: 1, stripe ID: 2, number of data blocks: 5, starting offset: 100}, {partition ID: 2, stripe ID: 3, number of data blocks: 3, starting offset: 200}, etc.

[0323] After responding to the data restore instruction, all of these stripe information are obtained as input for subsequent steps. For each target stripe information obtained from the stripe information table, the target partition identifier and target stripe identifier are extracted. These identifiers will be used to determine the location of the specific data stripe in the storage system. For example, for the stripe information {partition identifier: 1, stripe identifier: 2, number of data blocks: 5, starting offset: 100} mentioned above, the target partition identifier is 1 and the target stripe identifier is 2.

[0324] According to the target partition identifier, find the corresponding target partition. Then, in the data stripe table of the partition file corresponding to the target partition, extract the target data stripe according to the target stripe identifier. This target data stripe contains multiple data blocks and is the data unit that needs to be restored. For example, assuming that partition identifier 1 corresponds to partition P1 in the storage system, in the data stripe table of P1, find the corresponding target data stripe according to stripe identifier A, assuming that this data stripe contains a series of data blocks, such as data block 1, data block 2, etc.

[0325] In the stripe information table, all other target stripe information before the current target stripe information is determined. These stripe information will be used to calculate the number of the first data block in the current target data stripe.

[0326] For example, if there are three stripe information in the stripe information table, namely stripe information 1, stripe information 2 and stripe information 3, and the current target stripe information is stripe information 3, then stripe information 1 and stripe information 2 are other target stripe information before the target stripe information.

[0327] By accumulating the number of data blocks in all other target stripe information, the number of the first data block in the current target data stripe can be determined. This number will be used to accurately find the corresponding target data block storage space from the data block storage space table in the subsequent steps. For example, the number of data blocks in stripe information 1 is 5, and the number of data blocks in stripe information 2 is 3. When processing stripe information 3, the number of its first data block is 5+3+1=9.

[0328] After the target number is determined, the corresponding target data block storage space is found in the data block storage space table. This storage space information will be used in subsequent data restoration operations to determine the size and location of the data block.

[0329] For example, the data block storage space table stores the storage space information of each data block. According to the target number 8, the corresponding target data block storage space is found, which is assumed to be 20 bytes.

[0330] By combining the starting offset in the target stripe information, the storage space of the target data stripe, the target number, and the information in the data block storage space table, the location and content of the original data can be gradually determined. First, the starting position of the target data stripe in the storage system is determined according to the starting offset. Then, the location of the target data block in the target data stripe can be determined by combining the target stripe storage space and the target number. Finally, the target data block can be accurately read and restored through the information in the data block storage space table, thereby gradually restoring the original data.

[0331] For example, assume that the starting offset in the target stripe information is 1000, the target stripe storage space is 500 bytes, and the target number is 8. Determine the starting position of the target data stripe based on the starting offset, and then determine the position of the target data block in the stripe based on the target stripe storage space and the target number. Combined with the information in the data block storage space table, assume that the target data block storage space is 120 bytes, read the 120-byte data block content from this position, and gradually restore the original data.

[0332] In some implementations, determining the target number of the first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information includes:

[0333] (5.1) determining the sum of the number of data blocks included in each of the other target stripe information to obtain the target number of data blocks;

[0334] (5.2) The sum of the number of target data blocks and one is determined as the target number of the first target data block included in the target data stripe.

[0335] The total number of data blocks included in all other stripe information before the current target stripe information is counted.

[0336] By traversing other target stripe information, the number of data blocks in each stripe information is extracted and added up to obtain a sum, which represents the number of data blocks that have been processed before the current target stripe.

[0337] For example, there are three stripe information 1, 2, and 3, and the current target stripe information is 3. Stripe information A contains 3 data blocks, and stripe information B contains 2 data blocks. Then when processing stripe information C, for step (5.1), add the number of data blocks of A and B, and the target number of data blocks is 3+2=5.

[0338] After obtaining the target data block number in the previous step, this step adds 1 to it to obtain the number of the first data block in the current target data stripe. Because the data block number usually starts from 1, and the target data block number counted previously is the total number of data blocks in all previous stripes, in order to determine the first data block number of the current target stripe, it is necessary to add 1 to the target data block number.

[0339] In some implementations, determining the original data based on the target start offset, the target stripe storage space of the target data stripe, the target number, and the data block storage space table included in the target stripe information includes:

[0340] (7.1) when the target start offset included in the target stripe information is not zero, extracting a third data block from the target data stripe according to the target start offset;

[0341] (7.2) determining a previous data block to be merged whose number is before the target number, and merging the data block to be merged with the third data block to obtain a complete data block;

[0342] (7.3) decompressing the complete data block to obtain a portion of original data corresponding to the complete data block;

[0343] (7.4) determining a target data block storage space numbered with the target number from the data block storage space table;

[0344] (7.5) when the remaining stripe storage space of the target data stripe is greater than or equal to the target data block storage space, extracting the target data block corresponding to the target number from the target data stripe according to the target data block storage space;

[0345] (7.6) decompressing the target data block to obtain a portion of original data corresponding to the target data block;

[0346] (7.7) calculating the sum of the target number and one, determining the sum as the target number, and returning to the step of determining the target data block storage space of the data block corresponding to the target number from the data block storage space table until the stripe storage space of the target data stripe is smaller than the target data block storage space, thereby obtaining a plurality of partial original data;

[0347] (7.8) Merge the plurality of partial original data to obtain original data.

[0348] Among them, the original data can be obtained through the following code:

[0349] ;

[0350] if PTABLE[comp]≠None

[0351] ;

[0352] end if

[0353] ;

[0354] According to BDEPTH[bdTable], from All data blocks are read one by one, the data blocks are optionally decompressed, and DATA is updated. The specific method is as follows:

[0355] while ≠∅

[0356] if

[0357] ;

[0358] if PTABLE[comp]≠None

[0359] ;

[0360] end if

[0361] ;

[0362] u=u+1;

[0363] else

[0364] ;

[0365] ;

[0366] end if

[0367] end while

[0368] First, the starting offset in the target stripe information is checked. If the starting offset is not zero, it means that the data stripe does not start from the absolute starting position of the storage system. Based on this starting offset, a specific position in the target data stripe can be determined. A data block is extracted from this position, which is called the third data block. The purpose of this step is to start reading the data block from the correct position during the data restoration process.

[0369] For example, the starting position of the target data stripe in the storage system is offset 100, and the target starting offset is 50. Then, after the target data stripe is found in the storage system, the third data block is extracted according to the position of offset 50.

[0370] When the starting offset of the stripe information is not 0, it means that the third data block extracted from the stripe information is not a complete data block. It is a complete data block only when it is merged with the previous data block. Therefore, after the third data block is extracted, the previous data block to be merged whose number is before the target number will be determined according to the target number. This data block is another incomplete data block. Therefore, the third data block is merged with the data block to be merged to obtain a complete data block. By decompressing the complete data block, the corresponding part of the original data is obtained.

[0371] If the storage system compresses data when storing it, the data block needs to be decompressed when restoring the data. decompress(a,b) means using method b to decompress a and return the decompressed data. Here, the third data block is decompressed to obtain the partial original data corresponding to the data block. This partial original data is part of the original data. By gradually restoring each data block, the complete original data can be finally obtained.

[0372] For example, the third data block is compressed by a compression algorithm, and is decompressed using a decompression method corresponding to the compression algorithm to obtain a portion of original data after decompression.

[0373] During data restoration, you need to know the size of each data block in order to correctly read and restore the data. In the example, find the corresponding target data block storage space according to the target number. This storage space indicates the size of the space occupied by the target data block in the storage system.

[0374] The data block storage space table stores the number of each data block and the corresponding storage space information. According to the current target number, the storage space of the target data block is searched and determined.

[0375] After determining the storage space of the target data block, check the remaining storage space of the target data stripe. If the remaining storage space is greater than or equal to the storage space of the target data block, it means that the target data block can be extracted from the target data stripe. According to the size of the target data block storage space, accurately extract the target data block corresponding to the target number from the target data stripe.

[0376] For example, if the remaining storage space of the target data stripe is 200 bytes and the storage space of the target data block is 100 bytes, the target data block can be extracted from the target data stripe.

[0377] If the storage system has compressed the data, the extracted target data block is decompressed to obtain part of the original data corresponding to the target data block. The target data block is decompressed using a decompression method corresponding to the compression algorithm to obtain part of the original data after decompression.

[0378] After extracting a target data block and obtaining part of the original data, the target number is updated. The sum of the target number and one is used as the new target number, and then the data block storage space corresponding to the new target number is determined from the data block storage space table again. This process is repeated until the stripe storage space of the target data stripe is smaller than the storage space of the target data block. In this way, all data blocks in the target data stripe can be gradually extracted and multiple parts of the original data can be obtained.

[0379] For example, if the current target number is 5, calculate 5+1=6 and use 6 as the new target number. Then continue to search for the data block storage space with the target number 6 from the data block storage space table and repeat the subsequent steps until the storage space of the target data stripe is insufficient to extract the next data block.

[0380] After obtaining multiple parts of original data through the previous steps, these parts of original data are merged in the correct order. The result of the merger is the complete original data, completing the data restoration process.

[0381] For example: Assume that three parts of original data are obtained through the previous steps, which are different parts of the original data. Combine these three parts of original data in the correct order to obtain the complete original data.

[0382] As can be seen from the above, the embodiment of the present application obtains the data to be stored, performs data striping processing on the data to be stored, and obtains multiple data stripes, wherein the data stripes include multiple data blocks; each of the data stripes is stored in a data stripe table of a partition file corresponding to a different partition, and the data block index information of each data block is recorded, wherein the data block index information includes the partition identifier of the partition where the data block is located and the stripe identifier of the data stripe where the data block is located; based on the data block index information of each data block and the corresponding data block storage space, the stripe information of each data stripe is constructed, wherein the stripe information includes the partition identifier, stripe identifier, number of data blocks and starting offset of the data blocks contained in the data stripe; each of the stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence. In this way, by storing the original data in the form of data blocks along with data stripes, determining the stripe information of each data stripe after storage, and storing the data block storage space of each real data block in the data block storage space table, in the subsequent access to the original data, storage management software such as RAID or DFS can access the data based on the stripe information and the data block storage space table, avoiding the adjustment of the disk and data structure before RAID or DFS manages the disk, thereby improving data integrity.

[0383] For details, please refer to Figure 4 , Figure 4 The schematic diagram of the multi-disk parallel storage data storage format provided in the embodiment of the present application, in order to realize multi-disk data parallel storage, the set containing the original data DATA and the index information table is defined as a logical file, and its structure is shown in Figure 1. The logical file is divided into x partition files at the top level, and each is stored in the form of a file on different disks. Among them, the primary partition is a partition file with a path specified by the user, which serves as the entry of the logical file.

[0384] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0385] In order to better implement the data transmission method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data transmission method. The meanings of the terms are the same as those in the above data transmission method, and the specific implementation details can refer to the description in the method embodiment.

[0386] See also Figure 5 , Figure 5The data transmission method device may include a striping processing unit 601, a recording unit 602, a construction unit 603, and a storage unit 604.

[0387] The striping processing unit 601 is used to obtain data to be stored, and perform data striping processing on the data to be stored to obtain multiple data stripes, where the data stripes include multiple data blocks;

[0388] A recording unit 602 is used to store each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and record data block index information of each data block, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located;

[0389] A construction unit 603 is used to construct stripe information of each of the data stripes based on the data block index information of each of the data blocks and the corresponding data block storage space, wherein the stripe information includes a partition identifier, a stripe identifier, a number of data blocks, and a starting offset of the data blocks included in the data stripe;

[0390] The storage unit 604 is used to store each of the stripe information into the stripe information table of the partition file corresponding to the primary partition, and store the data block storage space of the data blocks whose data block storage space is not zero into the data block storage space table of the partition file corresponding to the primary partition.

[0391] In some embodiments, the recording unit includes:

[0392] An acquisition subunit is used to acquire the storage speed of the disk corresponding to each partition and the unstored number of data stripes that are not currently stored in each partition;

[0393] A first determining subunit, configured to determine an allocation weight of each partition based on a storage speed and a corresponding unstored quantity corresponding to each partition;

[0394] A second determining subunit, configured to determine the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes;

[0395] The storage subunit is used to store each of the data stripes in the data stripe table of the partition file corresponding to the different partitions according to the allocation quantity corresponding to each partition.

[0396] In some embodiments, the first determining subunit is configured to:

[0397] Get the sum of the storage speeds corresponding to each partition to get the total storage speed;

[0398] Calculate the ratio of the storage speed corresponding to each partition to the total storage speed to obtain the allocation ratio corresponding to each partition;

[0399] Determine the ratio of the unstored quantity corresponding to each partition to the corresponding storage speed, and obtain the storage time length of each partition storing the unstored data stripe;

[0400] Determine the ratio of the allocation ratio corresponding to each partition to the corresponding storage duration, and obtain the allocation weight of each partition.

[0401] In some embodiments, the second determining subunit is configured to:

[0402] Get the sum of the allocation weights of each partition to get the total allocation weight;

[0403] Determine the ratio of the allocation weight of each partition to the total allocation weight to obtain the allocation weight ratio of each partition;

[0404] Determine the product of the target number of data stripes and the allocation weight ratio of each partition to obtain a preliminary allocation number of data stripes allocated to each partition;

[0405] When the sum of the preliminary allocated quantities of each partition exceeds the target quantity, the preliminary allocated quantities of the partitions are adjusted according to the partition order between each partition to obtain the allocated quantity of the data stripes allocated to each partition.

[0406] In some embodiments, the striping processing unit is configured to:

[0407] Writing the data to be stored into a data cache, obtaining a current data volume of the data cache, and when the current data volume is greater than or equal to a preset data volume, taking out target data of the preset data volume from the data cache, and updating the current data volume;

[0408] compressing the target data to obtain a first data block, and recording a data block storage space of the first data block;

[0409] Writing the first data block into a stripe cache;

[0410] When the cache amount of the stripe cache is greater than or equal to the preset data stripe storage space, extracting a plurality of first data blocks whose sum of data block storage space is the preset data stripe storage space from the stripe cache, and the plurality of first data blocks form a first data stripe;

[0411] Acquire the updated current data amount, and when the current data amount is greater than or equal to the preset data block depth, return to execute the step of taking out target data of the preset data amount from the data cache until the current data amount is less than the preset data block depth or the current time reaches the specified end time, thereby obtaining a plurality of first data stripes;

[0412] When the updated current data amount is zero, each of the first data stripes is determined as a data stripe.

[0413] In some embodiments, the striping processing unit is further configured to:

[0414] When the current time reaches the specified end time and the current data amount is less than the preset data amount, compressing the remaining data in the data cache to obtain a second data block;

[0415] Writing the second data block into the stripe cache;

[0416] When the cache amount of the stripe cache is less than the preset data stripe storage space, calculating the difference between the preset data stripe storage space and the cache amount of the stripe cache to obtain the number of filling times;

[0417] Using a preset filling character, filling the stripe buffer according to the filling number of times;

[0418] Extracting cache content in the stripe cache, and determining the cache content as a second data stripe;

[0419] Each of the formed first data stripes and the second data stripes is determined as a data stripe.

[0420] In some embodiments, a construction unit is used to:

[0421] Traversing each of the data blocks, determining the partition identifier and the stripe identifier of the current data block as the partition identifier and the stripe identifier of the data stripe where the current data block is located;

[0422] When the data block storage space of the current data block is not zero, obtaining the current record count and the total storage space of the current data block;

[0423] Increment the current record count by one;

[0424] Determine a sum of the total storage space of the current data block and the data block storage space of the current data block, and update the total storage space of the current data block according to the sum;

[0425] When the updated total storage space of the current data block is greater than or equal to the preset stripe storage space, the updated current record count is determined as the number of data blocks in the data stripe where the current data block is located, and the current offset is determined as the starting offset of the data stripe where the current data block is located;

[0426] Reset the current record count to zero, and determine the difference between the current data block total storage space and the preset stripe storage space as the current data block total storage space and the current offset;

[0427] The next data block after the current data block is determined as the current data block, and the step of determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located is returned to be executed until each of the data blocks is traversed and the stripe information of each of the data stripes is obtained.

[0428] In some embodiments, the apparatus further comprises:

[0429] A first acquisition unit, configured to acquire information of each target stripe stored in a stripe information table of a partition file corresponding to the primary partition in response to a data restoration instruction;

[0430] A second acquisition unit, configured to acquire, for each target stripe information, a target partition identifier and a target stripe identifier included in the target stripe information;

[0431] A first determining unit, configured to determine a target partition corresponding to the target partition identifier, and extract a target data stripe corresponding to the target stripe identifier from a data stripe table of a partition file corresponding to the target partition;

[0432] A second determining unit, configured to determine other target stripe information in the stripe information table that is before the target stripe information;

[0433] A third determining unit, configured to determine a target number of a first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information;

[0434] A fourth determining unit, configured to determine a target data block storage space of the data block corresponding to the target number from the data block storage space table;

[0435] A fifth determining unit is configured to determine original data based on a target start offset included in the target stripe information, a target stripe storage space of the target data stripe, the target number, and the data block storage space table.

[0436] In some embodiments, the third determining unit is configured to:

[0437] Determine the sum of the number of data blocks included in each of the other target stripe information to obtain the target number of data blocks;

[0438] The sum of the number of target data blocks and one is determined as the target number of the first target data block included in the target data stripe.

[0439] In some embodiments, the fifth determining unit is configured to:

[0440] When the target start offset included in the target stripe information is not zero, extracting a third data block from the target data stripe according to the target start offset;

[0441] Determine a previous data block to be merged whose number is before the target number, and merge the data block to be merged with the third data block to obtain a complete data block;

[0442] Decompressing the complete data block to obtain part of original data corresponding to the complete data block;

[0443] Determine a target data block storage space numbered with the target number from the data block storage space table;

[0444] When the remaining stripe storage space of the target data stripe is greater than or equal to the target data block storage space, extracting the target data block corresponding to the target number from the target data stripe according to the target data block storage space;

[0445] Decompressing the target data block to obtain a portion of original data corresponding to the target data block;

[0446] Calculate the sum of the target number and one, determine the sum as the target number, and return to the step of determining the target data block storage space of the data block corresponding to the target number from the data block storage space table until the stripe storage space of the target data stripe is smaller than the target data block storage space, thereby obtaining a plurality of partial original data;

[0447] The multiple parts of the original data are combined to obtain the original data.

[0448] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0449] As can be seen from the above, the embodiment of the present application obtains the data to be stored through the striping processing unit 601, performs data striping processing on the data to be stored, and obtains multiple data stripes, wherein the data stripes include multiple data blocks; the recording unit 602 stores each of the data stripes in the data stripe table of the partition file corresponding to the different partitions, and records the data block index information of each of the data blocks, wherein the data block index information includes the partition identifier of the partition where the data block is located and the stripe identifier of the data stripe where the data block is located; the construction unit 603 constructs the stripe information of each of the data stripes based on the data block index information of each of the data blocks and the corresponding data block storage space, wherein the stripe information includes the partition identifier, stripe identifier, number of data blocks and starting offset of the data blocks contained in the data stripe; the storage unit 604 stores each of the stripe information in the stripe information table of the partition file corresponding to the primary partition, and sequentially stores the data block storage space of the data blocks whose data block storage space is not zero in the data block storage space table of the partition file corresponding to the primary partition.

[0450] In this way, by storing the original data in the form of data blocks along with data stripes, determining the stripe information of each data stripe after storage, and storing the data block storage space of each real data block in the data block storage space table, in the subsequent access to the original data, storage management software such as RAID or DFS can access the data based on the stripe information and the data block storage space table, avoiding the adjustment of the disk and data structure before RAID or DFS manages the disk, thereby improving data integrity.

[0451] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0452] Reference Figure 6 , Figure 6 The block diagram of the structure of part of the computer device 110 for implementing the embodiment of the present disclosure. The computer device 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

[0453] The computer device 110 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0454] The central processor 622 in the computer device 110 may be used to execute the data processing method of the embodiment of the present disclosure, for example:

[0455] Acquire data to be stored, and perform data striping processing on the data to be stored to obtain a plurality of data stripes, wherein the data stripes include a plurality of data blocks;

[0456] Respectively storing each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and recording data block index information of each of the data blocks, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located;

[0457] Based on the data block index information of each data block and the corresponding data block storage space, constructing stripe information of each data stripe, the stripe information including a partition identifier, a stripe identifier, a number of data blocks and a starting offset of the data blocks contained in the data stripe;

[0458] Each of the stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence.

[0459] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the data processing methods of the aforementioned embodiments.

[0460] The present disclosure also provides a computer program product, which includes a computer program. The processor of the computer device reads and executes the computer program, so that when the computer device is a verification device, the above-mentioned data processing method is implemented. For example:

[0461] Acquire data to be stored, and perform data striping processing on the data to be stored to obtain a plurality of data stripes, wherein the data stripes include a plurality of data blocks;

[0462] Respectively storing each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and recording data block index information of each of the data blocks, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located;

[0463] Based on the data block index information of each data block and the corresponding data block storage space, constructing stripe information of each data stripe, the stripe information including a partition identifier, a stripe identifier, a number of data blocks and a starting offset of the data blocks contained in the data stripe;

[0464] Each of the stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence.

[0465] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0466] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0467] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0468] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0469] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0470] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0471] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0472] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0473] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0474] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire data to be stored, and perform data striping processing on the data to be stored to obtain a plurality of data stripes, wherein the data stripes include a plurality of data blocks; Respectively storing each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and recording data block index information of each of the data blocks, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located; Traversing each of the data blocks, determining the partition identifier and the stripe identifier of the current data block as the partition identifier and the stripe identifier of the data stripe where the current data block is located; When the data block storage space of the current data block is not zero, obtaining the current record count and the total storage space of the current data block; Increment the current record count by one; Determine a sum of the total storage space of the current data block and the data block storage space of the current data block, and update the total storage space of the current data block according to the sum; When the updated total storage space of the current data block is greater than or equal to the preset stripe storage space, the updated current record count is determined as the number of data blocks in the data stripe where the current data block is located, and the current offset is determined as the starting offset of the data stripe where the current data block is located; Reset the current record count to zero, and determine the difference between the current data block total storage space and the preset stripe storage space as the current data block total storage space and the current offset; Determine the next data block after the current data block as the current data block, and return to execute the step of determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located, until each of the data blocks is traversed and stripe information of each of the data stripes is obtained, wherein the stripe information includes the partition identifier, stripe identifier, number of data blocks and starting offset of the data blocks contained in the data stripe; Each of the stripe information is stored in the stripe information table of the partition file corresponding to the primary partition, and the data block storage space of the data blocks whose data block storage space is not zero is stored in the data block storage space table of the partition file corresponding to the primary partition in sequence.

2. The data processing method according to claim 1, characterized in that: The storing each of the data stripes in the data stripe table of the partition file corresponding to the different partitions respectively includes: Get the storage speed of the disk corresponding to each partition and the number of unstored data stripes currently not stored in each partition; Determine the allocation weight of each partition based on the storage speed corresponding to each partition and the corresponding unstored quantity; Determine the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes; According to the allocation quantity corresponding to each partition, each of the data stripes is stored in the data stripe table of the partition file corresponding to the different partitions.

3. The data processing method according to claim 2, characterized in that: The step of determining the allocation weight of each partition based on the storage speed and the corresponding unstored quantity corresponding to each partition includes: Get the sum of the storage speeds corresponding to each partition to get the total storage speed; Calculate the ratio of the storage speed corresponding to each partition to the total storage speed to obtain the allocation ratio corresponding to each partition; Determine the ratio of the unstored quantity corresponding to each partition to the corresponding storage speed, and obtain the storage time length of each partition storing the unstored data stripe; Determine the ratio of the allocation ratio corresponding to each partition to the corresponding storage duration, and obtain the allocation weight of each partition.

4. The data processing method according to claim 2, characterized in that: The step of determining the number of data stripes to be allocated to each partition based on the allocation weight and the target number of data stripes includes: Get the sum of the allocation weights of each partition to get the total allocation weight; Determine the ratio of the allocation weight of each partition to the total allocation weight to obtain the allocation weight ratio of each partition; Determine the product of the target number of data stripes and the allocation weight ratio of each partition to obtain a preliminary allocation number of data stripes allocated to each partition; When the sum of the preliminary allocated quantities of each partition exceeds the target quantity, the preliminary allocated quantities of the partitions are adjusted according to the partition order between each partition to obtain the allocated quantity of the data stripes allocated to each partition.

5. The data processing method according to claim 1, characterized in that: The step of obtaining the data to be stored and performing data striping processing on the data to be stored to obtain a plurality of data stripes includes: Writing the data to be stored into a data cache, obtaining a current data volume of the data cache, and when the current data volume is greater than or equal to a preset data volume, taking out target data of the preset data volume from the data cache, and updating the current data volume; compressing the target data to obtain a first data block, and recording a data block storage space of the first data block; Writing the first data block into a stripe cache; When the cache amount of the stripe cache is greater than or equal to the preset data stripe storage space, extracting a plurality of first data blocks whose sum of data block storage space is the preset data stripe storage space from the stripe cache, and the plurality of first data blocks form a first data stripe; Acquire the updated current data amount, and when the current data amount is greater than or equal to the preset data block depth, return to execute the step of taking out target data of the preset data amount from the data cache until the current data amount is less than the preset data block depth or the current time reaches the specified end time, thereby obtaining a plurality of first data stripes; When the updated current data amount is zero, each of the first data stripes is determined as a data stripe.

6. The data processing method according to claim 5, characterized in that: The method further comprises: When the current time reaches the specified end time and the current data amount is less than the preset data amount, compressing the remaining data in the data cache to obtain a second data block; Writing the second data block into the stripe cache; When the cache amount of the stripe cache is less than the preset data stripe storage space, calculating the difference between the preset data stripe storage space and the cache amount of the stripe cache to obtain the number of filling times; Using a preset filling character, filling the stripe buffer according to the filling number of times; Extracting cache content in the stripe cache, and determining the cache content as a second data stripe; Each of the first data stripes and the second data stripes formed is determined as a data stripe.

7. The data processing method according to claim 1, characterized in that: After storing each of the stripe information in the stripe information table of the partition file corresponding to the primary partition, and sequentially storing the data block storage space of the data block whose data block storage space is not zero in the data block storage space table of the partition file corresponding to the primary partition, the method further includes: In response to the data restoration instruction, obtaining each target stripe information stored in the stripe information table of the partition file corresponding to the primary partition; For each target stripe information, obtaining a target partition identifier and a target stripe identifier included in the target stripe information; Determine a target partition corresponding to the target partition identifier, and extract a target data stripe corresponding to the target stripe identifier from a data stripe table of a partition file corresponding to the target partition; Determine other target stripe information before the target stripe information in the stripe information table; Determine the target number of the first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information; Determine the target data block storage space of the data block corresponding to the target number from the data block storage space table; The original data is determined based on the target start offset, the target stripe storage space of the target data stripe, the target number and the data block storage space table included in the target stripe information.

8. The data processing method according to claim 7, characterized in that: The step of determining the target number of the first target data block included in the target data stripe based on the number of data blocks included in each of the other target stripe information comprises: Determine the sum of the number of data blocks included in each of the other target stripe information to obtain the target number of data blocks; The sum of the number of target data blocks and one is determined as the target number of the first target data block included in the target data stripe.

9. The data processing method according to claim 7, characterized in that: The determining of the original data based on the target start offset, the target stripe storage space of the target data stripe, the target number and the data block storage space table included in the target stripe information includes: When the target start offset included in the target stripe information is not zero, extracting a third data block from the target data stripe according to the target start offset; Determine a previous data block to be merged whose number is before the target number, and merge the data block to be merged with the third data block to obtain a complete data block; Decompressing the complete data block to obtain part of original data corresponding to the complete data block; Determine a target data block storage space numbered with the target number from the data block storage space table; When the remaining stripe storage space of the target data stripe is greater than or equal to the target data block storage space, extracting the target data block corresponding to the target number from the target data stripe according to the target data block storage space; Decompressing the target data block to obtain a portion of original data corresponding to the target data block; Calculate the sum of the target number and one, determine the sum as the target number, and return to the step of determining the target data block storage space of the data block corresponding to the target number from the data block storage space table until the stripe storage space of the target data stripe is smaller than the target data block storage space, thereby obtaining a plurality of partial original data; The multiple parts of the original data are combined to obtain the original data.

10. A data processing device, characterized in that: include: A striping processing unit, used for acquiring data to be stored, performing data striping processing on the data to be stored, and obtaining a plurality of data stripes, wherein the data stripes include a plurality of data blocks; A recording unit, used to store each of the data stripes in a data stripe table of a partition file corresponding to a different partition, and record data block index information of each data block, wherein the data block index information includes a partition identifier of the partition where the data block is located and a stripe identifier of the data stripe where the data block is located; Building blocks for: Traversing each of the data blocks, determining the partition identifier and the stripe identifier of the current data block as the partition identifier and the stripe identifier of the data stripe where the current data block is located; When the data block storage space of the current data block is not zero, obtaining the current record count and the total storage space of the current data block; Increment the current record count by one; Determine a sum of the total storage space of the current data block and the data block storage space of the current data block, and update the total storage space of the current data block according to the sum; When the updated total storage space of the current data block is greater than or equal to the preset stripe storage space, the updated current record count is determined as the number of data blocks in the data stripe where the current data block is located, and the current offset is determined as the starting offset of the data stripe where the current data block is located; Reset the current record count to zero, and determine the difference between the current data block total storage space and the preset stripe storage space as the current data block total storage space and the current offset; Determine the next data block after the current data block as the current data block, and return to execute the step of determining the partition identifier and stripe identifier of the current data block as the partition identifier and stripe identifier of the data stripe where the current data block is located, until each of the data blocks is traversed and stripe information of each of the data stripes is obtained; The storage unit is used to store each of the stripe information into the stripe information table of the partition file corresponding to the primary partition, and sequentially store the data block storage space of the data blocks whose data block storage space is not zero into the data block storage space table of the partition file corresponding to the primary partition.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the data processing method according to any one of claims 1 to 9.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 9 is implemented.

Images