Data file processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data file processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a to-be-stored data file, performing file partitioning processing on the data file to obtain a plurality of file blocks, respectively writing the file blocks into a plurality of first storage spaces of a first storage medium, generating a plurality of temporary verification blocks for the file blocks, writing the temporary verification blocks into a second storage space of the first storage medium, and storing the temporary verification blocks in the second storage space of the first storage medium. By the adoption of the method, on one hand, under the condition that writing of the temporary check blocks is completed, the temporary check blocks are migrated into the second storage medium, so that the data blocks stored in the first storage medium are multiple file blocks, and the storage space utilization rate is increased; the file block is stored in the first storage medium, the temporary verification block is stored in the second storage medium, the read-write speed of the first storage medium is higher than that of the second storage medium, the overall read-write efficiency of data storage is improved, and then the storage performance is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data file processing method, device, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of distributed storage, the storage of data files is a key process. The rationality of the storage strategy directly affects the subsequent data recovery process.

[0003] In the traditional way, by replicating data to multiple nodes, when a node fails, the data can be restored from other nodes. This strategy provides high availability and data redundancy, but requires more storage space and has the problem of poor storage performance. Summary of the invention

[0004] Based on this, it is necessary to provide a data file processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve storage performance in response to the above technical problems.

[0005] In the first aspect, the present application provides a data file processing method. The method includes: obtaining a data file to be stored, performing file block processing on the data file to obtain multiple file blocks; writing each of the file blocks into multiple first storage spaces of a first storage medium; for multiple temporary check blocks generated for each of the file blocks, writing each of the temporary check blocks into a second storage space of the first storage medium; when each of the temporary check blocks is written, migrating each of the temporary check blocks from the first storage medium to the second storage medium; the read and write speed of the first storage medium is greater than that of the second storage medium.

[0006] In one of the embodiments, the method further includes: determining the number of check blocks configured for each of the file blocks; determining a plurality of sub-matrices matching the number of check blocks; and generating a plurality of temporary check blocks corresponding to each of the file blocks based on each of the sub-matrices and each of the file blocks.

[0007] In one of the embodiments, determining multiple sub-matrices that match the number of check blocks includes: obtaining a unit matrix and a Vandermonde matrix configured for each of the file blocks; combining the unit matrix and the Vandermonde matrix to obtain a generator matrix; extracting the generator matrix according to a sub-matrix extraction method that matches the number of check blocks to obtain a sub-matrix corresponding to each of the file data matrices.

[0008] In one of the embodiments, when the writing of each of the temporary check blocks is completed, migrating each of the temporary check blocks from the first storage medium to the second storage medium includes: obtaining the writing status of each of the temporary check blocks; when each of the writing statuses is that the writing is completed, determining the data block integrity of each of the temporary check blocks; when the integrity of each of the data blocks meets the data migration condition, migrating each of the temporary check blocks from the first storage medium to the second storage medium.

[0009] In one embodiment, the migrating of each of the temporary check blocks from the first storage medium to the second storage medium includes: obtaining the first computing capability of each storage node to which each of the first storage media belongs, the second computing capability of the storage node to which the second storage medium belongs, the first read-write capability of the first storage medium, and the second read-write capability of the second storage medium; determining a read-write frequency that matches the first computing capability, the second computing capability, the first read-write capability, and the second read-write capability; and migrating each of the temporary check blocks from the first storage medium to the second storage medium according to the read-write frequency.

[0010] In one of the embodiments, the method further includes: determining the data block integrity of each of the file blocks; determining the file block whose data block integrity meets the data recovery condition as a target block; and performing data recovery on the target block based on each file block other than the target block and a temporary check block to obtain a recovered target data block.

[0011] In a second aspect, the present application also provides a data file processing device. The device comprises:

[0012] An acquisition module is used to acquire a data file to be stored, and perform file block processing on the data file to obtain multiple file blocks; a first writing module is used to write each of the file blocks into multiple first storage spaces of a first storage medium respectively; a second writing module is used to write each of the multiple temporary check blocks generated for each of the file blocks into a second storage space of the first storage medium; a migration module is used to migrate each of the temporary check blocks from the first storage medium to the second storage medium when the writing of each of the temporary check blocks is completed; the reading and writing speed of the first storage medium is greater than that of the second storage medium.

[0013] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0014] Acquire a data file to be stored, perform file block processing on the data file to obtain multiple file blocks; write each of the file blocks into multiple first storage spaces of a first storage medium respectively; for multiple temporary check blocks generated for each of the file blocks, write each of the temporary check blocks into a second storage space of the first storage medium; when the writing of each of the temporary check blocks is completed, migrate each of the temporary check blocks from the first storage medium to the second storage medium; the read and write speed of the first storage medium is greater than that of the second storage medium.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0016] Acquire a data file to be stored, perform file block processing on the data file to obtain multiple file blocks; write each of the file blocks into multiple first storage spaces of a first storage medium respectively; for multiple temporary check blocks generated for each of the file blocks, write each of the temporary check blocks into a second storage space of the first storage medium; when the writing of each of the temporary check blocks is completed, migrate each of the temporary check blocks from the first storage medium to the second storage medium; the read and write speed of the first storage medium is greater than that of the second storage medium.

[0017] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0018] Acquire a data file to be stored, perform file block processing on the data file to obtain multiple file blocks; write each of the file blocks into multiple first storage spaces of a first storage medium respectively; for multiple temporary check blocks generated for each of the file blocks, write each of the temporary check blocks into a second storage space of the first storage medium; when the writing of each of the temporary check blocks is completed, migrate each of the temporary check blocks from the first storage medium to the second storage medium; the read and write speed of the first storage medium is greater than that of the second storage medium.

[0019] The above-mentioned data file processing method, device, computer equipment, storage medium and computer program product obtain the data file to be stored, perform file block processing on the data file to obtain multiple file blocks, write each file block into multiple first storage spaces of the first storage medium respectively, and write each temporary check block into the second storage space of the first storage medium for the multiple temporary check blocks generated for each file block. Using the above-mentioned method, on the one hand, when each temporary check block is written, each temporary check block is migrated from the first storage medium to the second storage medium, so that the data blocks stored in the first storage medium are multiple file blocks, thereby improving the storage space utilization rate of the first storage medium. On the other hand, the file blocks are stored in the first storage medium and the temporary check blocks are stored in the second storage medium. The read and write speed of the first storage medium is greater than that of the second storage medium, thereby improving the overall read and write efficiency of data storage, thereby improving the storage performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An application environment diagram of a data file processing method in an embodiment;

[0021] Figure 2 A schematic diagram of a flow chart of a data file processing method in one embodiment;

[0022] Figure 3 A schematic diagram of a process of an erasure code storage method using a hybrid medium of SSD and HDD in one embodiment;

[0023] Figure 4 A schematic diagram of mixed media storage in one embodiment;

[0024] Figure 5 A schematic diagram of data distribution after steady state in one embodiment;

[0025] Figure 6 A schematic diagram of a first type of data repair in an embodiment;

[0026] Figure 7 is a schematic diagram of a second data repair in an embodiment;

[0027] Figure 8 is a structural block diagram of a data file processing device in one embodiment;

[0028] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The data file processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0031] The terminal 102 can obtain a data file to be stored, perform file block processing on the data file, and obtain multiple file blocks. The terminal 102 can write each file block into multiple first storage spaces of the first storage medium. The terminal 102 can write each temporary check block into the second storage space of the first storage medium for multiple temporary check blocks generated for each file block. The terminal 102 can migrate each temporary check block from the first storage medium to the second storage medium when each temporary check block is written. The read and write speed of the first storage medium is greater than that of the second storage medium. The second storage medium can be a storage medium in the server 102, and the second storage medium can also be a storage medium in other terminals.

[0032] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0033] In one embodiment, Figure 2 As shown, a data file processing method is provided, which is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation, including:

[0034] S202, obtaining a data file to be stored, and performing file block processing on the data file to obtain a plurality of file blocks.

[0035] The data file to be stored can be a file format used to store data in a computer system, and can contain various types of data, such as text, numbers, images, audio, and video. Data files usually exist in the form of operating system files and are physical components of database tablespaces. In a database environment, data files are used to store actual data, including user data, index data, temporary data, and undo data.

[0036] Data files can be stored in a variety of ways, either as text files or binary files. Text files usually store data in the form of lines, each line containing a series of characters, while binary files store non-text data, such as images, video, and audio files, where data is stored in a binary format of 0s and 1s. In addition, data files can also be in CSV (comma-separated value) format, where each line represents a record and fields are separated by commas.

[0037] The storage location of data files can be a local storage device, such as a built-in hard disk, an external USB drive or SSD, or a remote storage device, such as a cloud storage service or an FTP server. In a database, data files are usually evenly distributed on different disks to improve I / O performance and reliability.

[0038] The file blocks may be data blocks obtained by dividing the data files into blocks.

[0039] There are many ways of block processing. Specifically, after obtaining the data file to be stored, the terminal can block the data file according to any one of the block processing methods of fixed-length block, dynamic block, and sliding block to obtain multiple file blocks.

[0040] Fixed-length chunking may mean that a file is split into blocks of a predefined size. For example, a file may be split into multiple data blocks of a fixed size (such as 64 MB).

[0041] Dynamic block division can mean dynamically dividing data blocks according to the actual content of the file. This method can flexibly adjust the size of the block according to the changes in the file content, thereby improving the flexibility and efficiency of data.

[0042] Sliding blocking can be said to combine the advantages of fixed-length blocking and dynamic blocking. First, the length of the data block is fixed, and then the weak checksum of the data block after blocking is calculated. If the checksum matches successfully, the hash value of the data block is further calculated as the strong checksum to detect duplicate data blocks.

[0043] In one embodiment, the terminal may divide the data file into blocks according to the block division method in the erasure code strategy to obtain multiple file blocks. For example, the file may be divided into 6 file blocks or 4 file blocks using the erasure code strategy.

[0044] S204: Write each file block into a plurality of first storage spaces of the first storage medium respectively.

[0045] The first storage medium may be a specific data storage medium.

[0046] Storage media can include solid-state drives, mechanical hard disks, optical disks, tapes, flash memory, floppy disks, cloud storage, etc. Among them, solid-state drives (SSD) are composed of flash memory chips, have no mechanical structure, fast read and write speeds, and good shock resistance, but are relatively expensive. Mechanical hard disks (HDD) have the characteristics of large capacity and low cost through high-speed rotation of the disk and the magnetic head reading and writing data, but the read and write speeds are slow and they are easily damaged by vibration. Optical disks include CDs, DVDs, and Blu-ray discs, which use laser technology to read and write data and are suitable for long-term storage of media content, but the read and write speeds are slow. Tapes have high storage capacity and low cost, and are suitable for long-term data backup and archiving, but the read and write speeds are slow. Flash memory includes USB flash drives, CF cards, SD cards, etc., which are based on semiconductor flash memory, are small in size, light in weight, and fast in speed, and are widely used in mobile devices and portable storage. Floppy disks: removable storage media that were widely used in the early days, with small capacity, and are gradually replaced by other storage media with higher capacity. Cloud storage: data storage services are provided through the Internet, which are flexible and scalable, and users can adjust storage capacity at any time according to their needs.

[0047] The first storage space may be a virtual space in the first storage medium allocated for storing data. The first storage space may be represented by a storage location. For example, addresses between a start storage location and an end storage location may be considered as storage space for data.

[0048] It should be noted that in order to ensure high availability and data redundancy, and also facilitate data recovery, the terminal may write each file block to the first storage space of each different first storage medium. The different first storage media may represent the first storage media of different storage nodes.

[0049] Specifically, the terminal may also write each file block into a plurality of first storage spaces of the first storage medium respectively, and each first storage space is isolated from each other, thereby ensuring high availability and data redundancy as well as efficiency and accuracy of possible data recovery in the later stage.

[0050] S206: For each of the multiple temporary check blocks generated for each file block, write each temporary check block into the second storage space of the first storage medium.

[0051] The temporary check block may be a temporary check block obtained by converting each file block into a check block. It should be noted that when a file block is incomplete or lost, the abnormal file block can be restored based on the temporary check block and other file blocks.

[0052] The second storage space may be a storage space in the first storage medium that is isolated from the first storage space.

[0053] There are many ways to convert the check blocks. Specifically, the terminal can determine the number of check blocks configured for each file block and the file block matrix corresponding to each file block. The terminal can determine multiple sub-matrices that match the number of check blocks. The terminal can generate multiple temporary check blocks corresponding to each file block based on each sub-matrix and each file data matrix. Specifically, the terminal can also directly perform encoding calculations on each file block to obtain a temporary check block.

[0054] Before writing each temporary check block into the second storage space of the first storage medium, it is necessary to determine the second storage space first. Specifically, the terminal can determine the physical node described in the first storage medium to which the first storage space belongs. The terminal determines the storage space of the first storage medium in other physical nodes as the second storage space. In one embodiment, the physical nodes to which the first storage spaces and the first storage media to which the second storage spaces belong may be different. For example, the physical node may be a terminal or a server. The first first storage space is allocated in the first storage medium in terminal 1, the second first storage space is allocated in the first storage medium in terminal 2, and the first second storage space is allocated in the first storage medium in terminal 3.

[0055] S208: When the writing of each temporary check block is completed, migrate each temporary check block from the first storage medium to the second storage medium.

[0056] The second storage medium may be a specific data storage medium. The read and write speed of the first storage medium is greater than that of the second storage medium. For example, the second storage medium may be a mechanical hard disk HDD, and the first storage medium may be a solid state disk SSD.

[0057] Data migration may refer to transferring data from a first storage medium to a second storage medium and deleting corresponding data in the first storage medium.

[0058] Specifically, the terminal can migrate each temporary check block from the first storage medium to a different storage space in the second storage medium when each temporary check block is written, so as to facilitate subsequent data recovery and improve the accuracy and efficiency of data recovery.

[0059] Specifically, the terminal may determine the data block integrity of each file block. The terminal may determine the file block whose data block integrity meets the data recovery condition as the target block. The terminal may perform data recovery on the target block based on each file block other than the target block and the temporary check block to obtain a recovered target data block.

[0060] Since the file blocks and temporary file blocks are written into the first storage medium, the writing efficiency of the data blocks is improved. Moreover, the temporary check blocks are migrated to the second storage medium, ensuring that the proportion of the file blocks in the first storage medium can reach 100%, thereby improving the storage space utilization.

[0061] In the above data file processing method, a data file to be stored is obtained, and the data file is processed into file blocks to obtain multiple file blocks. Each file block is written into multiple first storage spaces of a first storage medium respectively. For multiple temporary check blocks generated for each file block, each temporary check block is written into a second storage space of the first storage medium. With the above method, on the one hand, after each temporary check block is written, each temporary check block is migrated from the first storage medium to the second storage medium, so that the data blocks stored in the first storage medium are multiple file blocks, thereby improving the storage space utilization rate of the first storage medium. On the other hand, file blocks are stored in the first storage medium and temporary check blocks are stored in the second storage medium. The read and write speed of the first storage medium is greater than that of the second storage medium, thereby improving the overall read and write efficiency of data storage, thereby improving storage performance.

[0062] In one embodiment, the data file processing method further includes: determining the number of check blocks configured for each file block, determining multiple sub-matrices matching the number of check blocks, and generating multiple temporary check blocks corresponding to each file block based on each sub-matrix and each file block.

[0063] The number of check blocks may represent the expected number of check blocks obtained by calculating the file blocks. For example, the expected number of check blocks obtained by calculating four file blocks may be two. The file blocks may be in a matrix form.

[0064] The submatrix may be a matrix used for transposition calculation. The number of submatrices may correspond to the number of check blocks one by one.

[0065] Specifically, for each sub-matrix, the terminal may determine the product result between each file block and the sub-matrix as a temporary check block. In one embodiment, the terminal may determine the product result between the transposed matrix of each file block and the sub-matrix as a temporary check block.

[0066] Specifically, the terminal may also obtain the identity matrix and the Vandermonde matrix configured for each file block. The terminal may combine the identity matrix and the Vandermonde matrix to obtain a generator matrix. The terminal may extract the generator matrix according to the submatrix extraction method matching the number of check blocks to obtain a submatrix corresponding to each file block.

[0067] Exemplarily, it is determined that for each file block ( ) The number of check blocks configured (2). Determine multiple sub-matrices that match the number of check blocks ( and ) Based on each sub-matrix and each file block, generate multiple temporary check blocks corresponding to each file block ( and ), the formula is as follows:

[0068]

[0069] In this embodiment, the number of check blocks configured for each file block is determined, and multiple sub-matrices matching the number of check blocks are determined. Based on each sub-matrix and each file block, multiple temporary check blocks corresponding to each file block are generated. Multiple temporary check blocks are calculated based on each file block. The temporary check blocks can be used to recover data from the file blocks when an abnormality occurs in subsequent file blocks, thereby improving the efficiency and accuracy of data recovery.

[0070] In one embodiment, determining multiple sub-matrices that match the number of check blocks includes: obtaining a unit matrix and a Vandermonde matrix configured for each file block, combining the unit matrix and the Vandermonde matrix to obtain a generator matrix, extracting the generator matrix according to a sub-matrix extraction method that matches the number of check blocks, and obtaining a sub-matrix corresponding to each file block.

[0071] The identity matrix may represent a matrix in which all elements on the main diagonal are 1 and all other elements are 0.

[0072] The elements of each column of the Vandermonde matrix are the power of the elements of the previous column, that is, each column element is 1 once. , ,……, ,i is a natural number.

[0073] The generator matrix may be a linear combination of the identity matrix and the Vandermonde matrix.

[0074] Exemplarily, the terminal may linearly combine the identity matrix and the Vandermonde matrix to obtain a generator matrix. The generator matrix may be represented as GT. GT=(k+m)×k. The first k rows of the generator matrix are the identity matrix, and the last m rows may be the Vandermonde matrix.

[0075] The submatrix extraction method for matching the number of check blocks can represent the number of rows or columns extracted from the generator matrix. For example, the submatrix can be obtained by combining the elements of the 5th row, the 6th row, the 1st column and the 4th column extracted from the generator matrix. .

[0076] In this embodiment, the unit matrix and the Vandermonde matrix configured for each file block are obtained, and the unit matrix and the Vandermonde matrix are combined to obtain a generator matrix. The generator matrix is ​​extracted according to the submatrix extraction method that matches the number of check blocks to obtain a submatrix corresponding to each file block. Obtaining an accurate submatrix is ​​beneficial for subsequent data recovery of the file block in the event of an abnormality in the subsequent file block, thereby improving the efficiency and accuracy of data recovery.

[0077] In one embodiment, when the writing of each temporary check block is completed, each temporary check block is migrated from the first storage medium to the second storage medium, including: obtaining the writing status of each temporary check block, and when each writing status is that the writing is completed, determining the integrity of the data block of each temporary check block, and when the integrity of each data block meets the data migration condition, migrating each temporary check block from the first storage medium to the second storage medium.

[0078] The writing status may indicate a status of the temporary check block being written into the first storage medium.

[0079] The write status may include initial status, receiving status, receiving completion status, submission status, copy status, etc.

[0080] Initial state (uc): When a client requests a new data block, it is assigned a data block with an initial state of "uncompleted" (uc). Receiving state: When a data block starts to be written, it is in this state. At this stage, the data of the data block is being received. Receiving complete state: When all the data of the data block has been successfully written to one or more storage media, the state of the data block changes to "receiving complete". Committed state: After the client completes all write operations on the data block, the state of the data block is set to "committed". .

[0081] The situation where all write states are complete writing can indicate that all temporary check blocks are in the committed state.

[0082] The data block integrity may represent the integrity of the temporary check block.

[0083] The data migration condition may be an integrity condition configured for the integrity of the data block. Meeting the data migration condition may mean that each temporary check block may be migrated from the first storage medium to the second storage medium, and the temporary check block in the first storage medium is deleted. Not meeting the data migration condition may mean that each temporary check block is not migrated from the first storage medium to the second storage medium.

[0084] It is necessary to determine the data block integrity of each temporary check block. Specifically, the terminal can perform integrity check on each temporary check block to obtain the data integrity of each temporary check block.

[0085] In one of the embodiments, the integrity check method can be based on a checksum method or a preset check mechanism. Taking the checksum method as an example, the terminal can use the CRC-32 algorithm to calculate the checksum of the data block and verify it when writing and reading data. When the data is written, the terminal calculates the checksum and stores it on the metadata node. When reading data, the system recalculates the checksum and compares it with the stored checksum. If the two do not match, it indicates that the integrity of the temporary check block is lower than the preset value, or it may indicate that the data block integrity of the temporary check block does not meet the data migration conditions.

[0086] In this embodiment, the write status of each temporary check block is obtained. When the write status of each temporary check block is complete, the integrity of the data block of each temporary check block is determined. When the integrity of each data block meets the data migration conditions, each temporary check block is migrated from the first storage medium to the second storage medium, thereby ensuring that the temporary check blocks are complete data blocks, improving the accuracy of subsequent data recovery, and reducing the time for manual troubleshooting and manual data recovery, thereby improving the efficiency of data recovery.

[0087] In one embodiment, migrating each temporary check block from a first storage medium to a second storage medium includes: obtaining the first computing capability of each storage node to which each first storage medium belongs, the second computing capability of the storage node to which the second storage medium belongs, the first read-write capability of the first storage medium, and the second read-write capability of the second storage medium, determining a read-write frequency that matches the first computing capability, the second computing capability, the first read-write capability, and the second read-write capability, and migrating each temporary check block from the first storage medium to the second storage medium according to the read-write frequency.

[0088] The storage node may represent a physical node to which the storage medium belongs. The storage node may be a terminal or a server.

[0089] The first computing capability may represent the computing capability of the storage node to which the first storage medium belongs. The second computing capability may represent the computing capability of the storage node to which the second storage medium belongs.

[0090] The first read / write capability may indicate the read / write capability of the first storage medium, and the second read / write capability may indicate the read / write capability of the second storage medium.

[0091] It should be noted that computing power can be determined by factors such as CPU speed and memory capacity. The faster the CPU speed and the higher the memory capacity, the stronger the computing power. Similarly, the stronger the read and write capabilities, the stronger the computing power.

[0092] The read / write frequency may indicate the number of read / write operations that a device such as a hard disk, memory, or RFID reader / writer can handle in a unit of time. Different devices and application scenarios have different requirements and performances for the read / write frequency. Specifically, the read / write frequency may indicate the read / write frequency of the first storage medium.

[0093] There are many ways to determine the read / write frequency. Specifically, after determining the first computing capability, the second computing capability, the first read / write capability, and the second read / write capability, the terminal can calculate the estimated read / write capability based on the first computing capability, the second computing capability, the first read / write capability, and the second read / write capability. The terminal determines the read / write frequency corresponding to the estimated read / write capability according to the mapping relationship between the read / write capability and the read / write frequency.

[0094] In one embodiment, the terminal may calculate an estimated reading and writing ability (score) according to the first computing ability, the second computing ability, the first reading and writing ability, and the second reading and writing ability, and the respective weights of the first computing ability, the second computing ability, the first reading and writing ability, and the second reading and writing ability.

[0095] In one embodiment, the terminal may calculate the score according to the reading and writing ability and the weight of the reading and writing ability. The terminal may calculate the score according to the computing ability and the weight of the computing ability. After obtaining the scores of the first computing ability, the second computing ability, the first reading and writing ability, and the second reading and writing ability, the terminal calculates the scores to obtain the estimated reading and writing ability.

[0096] In one embodiment, after obtaining the scores of the first computing capability, the second computing capability, the first reading and writing capability, and the second reading and writing capability, the terminal uses the lowest score as the estimated reading and writing capability.

[0097] In this embodiment, the first computing capacity of each storage node to which each first storage medium belongs, the second computing capacity of the storage node to which the second storage medium belongs, the first read-write capacity of the first storage medium, and the second read-write capacity of the second storage medium are obtained, and the read-write frequency matching the first computing capacity, the second computing capacity, the first read-write capacity, and the second read-write capacity is determined. According to the read-write frequency, each temporary check block is migrated from the first storage medium to the second storage medium, and each read-write frequency is accurately determined to improve the accuracy and efficiency of data migration.

[0098] In one embodiment, the data file processing method further includes: determining the data block integrity of each file block, determining the file block whose data block integrity meets the data recovery condition as the target block, and performing data recovery on the target block based on each file block other than the target block and the temporary check block to obtain a recovered target data block.

[0099] The data block integrity may be the real-time integrity of the file block. If the data block integrity is lower than a certain value, it may indicate that data loss has occurred in the file block, and data recovery of the file block is required.

[0100] The data recovery condition may be a completeness filter condition configured for data block completeness. The data recovery condition may be that the data block completeness is less than 99.98% or other values.

[0101] The target block may represent a file block whose data block integrity is lower than a preset value.

[0102] Specifically, the terminal may read each file block except the target block from the first storage medium. The terminal may read the temporary check block from the second storage medium. The terminal may perform data recovery on the target block based on each file block except the target block and the temporary check block to obtain a recovered target data block.

[0103] Specifically, the terminal may also read the temporary verification module from the first storage medium. The terminal may perform data recovery on the target block based on each file block except the target block and the temporary verification block to obtain a recovered target data block.

[0104] In this embodiment, the data block integrity of each file block is determined, and the file block whose data block integrity meets the data recovery condition is determined as the target block. Based on each file block other than the target block and the temporary check block, data recovery is performed on the target block to obtain the recovered target data block, thereby improving the accuracy of data recovery.

[0105] In one embodiment, 1. In the field of distributed storage, using erasure codes to reduce storage costs has become a widely used technology. Especially for storage systems using SSD as media, using erasure code technology has become an effective way to effectively reduce storage costs. 2. Commonly used erasure codes are generally in the form of 4+2, 8+3, etc. How to further reduce the use cost and improve storage utilization has become an important issue. In order to further improve the storage space utilization (to 80% or above 90%), large-scale EC systems (25+3 or even 100+3) are widely used. 3. The problems brought by large-scale EC are also obvious: a. More computing resources are required, and b. The cost of data repair after failure is also doubled. 4. Through the analysis of the erasure code reading and writing mechanism, we found that data with fast verification will not be read in normal scenarios. Therefore, we propose a hybrid storage method of SSD and HDD, where SSD is used to store data blocks and HDD users store verification blocks. In this way, the space utilization rate of SSD can reach 100%. Since HDD is only used to store verification blocks, it will not affect the reading of data.

[0106] In view of this, if Figure 3As shown, a method for erasure code storage using a hybrid medium of SSD and HDD is provided, comprising:

[0107] S301, obtaining a data file to be stored, and performing file block processing on the data file to obtain multiple file blocks.

[0108] S302: Write each file block into a plurality of first storage spaces of a first storage medium respectively.

[0109] S303: For each of the multiple temporary check blocks generated for each file block, write each temporary check block into the second storage space of the first storage medium.

[0110] Among them, the number of check blocks configured for each file block is determined.

[0111] S304, obtaining a unit matrix and a Vandermonde matrix configured for each file block.

[0112] S305, combining the identity matrix and the Vandermonde matrix to obtain a generator matrix.

[0113] S306, extracting the generation matrix according to the sub-matrix extraction method matching the number of check blocks, and obtaining the sub-matrix corresponding to each file block.

[0114] S307: Based on each sub-matrix and each file block, generate multiple temporary check blocks corresponding to each file block.

[0115] S308, obtaining the writing status of each temporary check block.

[0116] S309: When all writing states are complete, determine the data block integrity of each temporary check block.

[0117] S310, when the integrity of each data block meets the data migration conditions, obtain the first computing capacity of the storage node to which each first storage medium belongs, the second computing capacity of the storage node to which the second storage medium belongs, the first read / write capacity of the first storage medium, and the second read / write capacity of the second storage medium.

[0118] S311, determining a read / write frequency that matches the first computing capability, the second computing capability, the first read / write capability, and the second read / write capability.

[0119] S312: Migrate each temporary check block from the first storage medium to the second storage medium according to the read and write frequency.

[0120] The read and write speed of the first storage medium is greater than that of the second storage medium.

[0121] Unlike the traditional 4+2 which uses 6 disks, Hybrid EC requires 8 disks to participate. The corresponding PGs are:

[0122] [ssd1,ssd2,ssd3,ssd4,ssd5,ssd6,hdd7,hdd8]. Among them, ssd1~ssd4 store data shards, ssd5~ssd6 store temporary check shards. hdd7~hdd8 store final check shards. Figure 4 As shown in the hybrid media storage schematic, the first storage medium may be an SSD, and the second storage medium may be an HDD. Chunk1, Chunk2, Chunk3, and Chunk4 may represent file blocks. Chunk5 and Chunk6 may represent temporary check blocks. After Chunk5 and Chunk6 in the first storage medium are migrated to the second storage medium, the temporary check blocks may be referred to as check blocks.

[0123] The temporary check block may be a LogicalChunk, and the check block may be a LogicalChunk Seal.

[0124] LogicalChunk: Logical data block, only supports appending, each chunk has a uint64 ChunkId as a unique identifier. By taking the modulus of ChunkId and the number of PGs, you can calculate which PG the chunk belongs to. Generally, a Chunk is at least 128MB in size. LogicalChunk Seal: refers to sealing the logical chunk. After sealing, the chunk becomes read-only and users can no longer write to the chunk. At this time, we believe that the chunk data is stable.

[0125] like Figure 4 As shown, a. First, the client applies for a Logical Chunk Id, calculates the PG corresponding to the LogicalChunk, and obtains the DiskGroup on the PG.

[0126] DiskGroup is [ssd1,ssd2,ssd3,ssd4,ssd5,ssd6,hdd7,hdd8].

[0127] b. The client then slices the data and calculates two check blocks. The four data blocks are written to the first four SSDs of the PG, and the two check blocks are written to the two temporary SSDs of the PG. At this time, there is no data on the HDD. This is the case for every subsequent write.

[0128] After LogicalChunk Seal, migrate the check shard data chunk5 and chunk6 on the temporary SSD to hdd7 and hdd8. Figure 5The distribution diagram of the data after steady state is shown. After the parity block data migration is completed, the parity data chunk5 and chunk6 on ssd5 and ssd6 are deleted. In this way, the SSD space utilization rate reaches 100%.

[0129] S313, determining the data block integrity of each file block.

[0130] S314: determine the file blocks whose data block integrity meets the data recovery condition as target blocks.

[0131] S315, based on each file block except the target block and the temporary check block, data recovery is performed on the target block to obtain a recovered target data block.

[0132] Among them, Figure 6 As shown in the schematic diagram of the first data repair, when the data on the temporary shard has not yet been migrated to the HDD, chunk3 can be directly recovered through the data of the temporary shard ssd5.

[0133] like Figure 7 As shown in the schematic diagram of the second data repair, when the data on the temporary shard has been migrated to the HDD, chunk3 is recovered through the data on the HDD and the remaining data shards on the SSD.

[0134] In this embodiment, a data file to be stored is obtained, and the data file is divided into file blocks to obtain multiple file blocks. Each file block is written into multiple first storage spaces of a first storage medium, and multiple temporary check blocks generated for each file block are written into a second storage space of the first storage medium. Using the above method, on the one hand, when each temporary check block is written, each temporary check block is migrated from the first storage medium to the second storage medium, so that the data blocks stored in the first storage medium are multiple file blocks, thereby improving the storage space utilization rate of the first storage medium. On the other hand, the file blocks are stored in the first storage medium and the temporary check blocks are stored in the second storage medium. The read and write speed of the first storage medium is greater than that of the second storage medium, thereby improving the overall read and write efficiency of data storage, thereby improving the storage performance.

[0135] The above method can also bring the following beneficial effects: 1. Adopting the erasure code scheme of HDD and SSD hybrid media, the data shards are stored on SSD and the check shards are stored on HDD, thus improving the data reading and writing efficiency.

[0136] 2. By adding a temporary check shard (SSD data disk) in EC, it is ensured that the HDD disk is not passed during the writing process, so as to achieve the purpose of not reducing the writing performance. Since the reading itself does not read the check data, the reading performance is not reduced.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a data file processing device for implementing the data file processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more data file processing device embodiments provided below can refer to the limitations of the data file processing method above, and will not be repeated here.

[0139] In one embodiment, Figure 8 As shown, a data file processing device is provided, including: an acquisition module 801, a first writing module 802, a second writing module 803 and a migration module 804, wherein:

[0140] The acquisition module 801 is used to acquire the data file to be stored, and perform file block processing on the data file to obtain multiple file blocks;

[0141] A first writing module 802, used to write each file block into a plurality of first storage spaces of a first storage medium;

[0142] The second writing module 803 is used for writing each temporary check block generated for each file block into the second storage space of the first storage medium;

[0143] The migration module 804 is used to migrate each temporary check block from the first storage medium to the second storage medium when each temporary check block is written. The read and write speed of the first storage medium is greater than that of the second storage medium.

[0144] In one of the embodiments, the data file processing device also includes: a temporary check block determination module, used to determine the number of check blocks configured for each file block; determine multiple sub-matrices matching the number of check blocks; and generate multiple temporary check blocks corresponding to each file block based on each sub-matrix and each file block.

[0145] In one of the embodiments, the temporary check block determination module is further used to obtain the unit matrix and the Vandermonde matrix configured for each file block; combine the unit matrix and the Vandermonde matrix to obtain the generation matrix; extract the generation matrix according to the sub-matrix extraction method matching the number of check blocks to obtain the sub-matrix corresponding to each file block.

[0146] In one embodiment, the migration module 804 is also used to obtain the write status of each temporary check block; when each write status is complete, determine the data block integrity of each temporary check block; when each data block integrity meets the data migration conditions, migrate each temporary check block from the first storage medium to the second storage medium.

[0147] In one embodiment, the migration module 804 is also used to obtain the first computing capacity of each storage node to which each first storage medium belongs, the second computing capacity of the storage node to which the second storage medium belongs, the first read and write capacity of the first storage medium, and the second read and write capacity of the second storage medium; determine the read and write frequency that matches the first computing capacity, the second computing capacity, the first read and write capacity, and the second read and write capacity; and migrate each temporary check block from the first storage medium to the second storage medium according to the read and write frequency.

[0148] In one of the embodiments, the data file processing device also includes: a data recovery module, used to determine the data block integrity of each file block; determine the file block whose data block integrity meets the data recovery condition as the target block; based on each file block other than the target block and the temporary check block, perform data recovery on the target block to obtain a recovered target data block.

[0149] Each module in the above data file processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0150] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file block data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data file processing method is implemented.

[0151] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method steps are implemented when the processor executes the computer program.

[0153] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0154] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0156] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data file processing method, characterized in that: The method comprises: Acquire a data file to be stored, and perform file block processing on the data file to obtain multiple file blocks; Writing each of the file blocks into a plurality of first storage spaces of the first storage medium respectively; For each of the file blocks, a plurality of temporary check blocks generated are written into a second storage space of the first storage medium; When the writing of each temporary check block is completed, each temporary check block is migrated from the first storage medium to the second storage medium; the reading and writing speed of the first storage medium is greater than that of the second storage medium.

2. The method according to claim 1, characterized in that The method further comprises: Determining the number of check blocks configured for each of the file blocks; Determine a plurality of sub-matrices matching the number of the check blocks; Based on each of the sub-matrices and each of the file blocks, a plurality of temporary check blocks corresponding to each of the file blocks are generated.

3. The method according to claim 2, characterized in that The determining of a plurality of sub-matrices matching the number of check blocks comprises: Obtaining the identity matrix and the Vandermonde matrix configured for each of the file blocks; Combining the identity matrix and the Vandermonde matrix to obtain a generator matrix; The generator matrix is ​​extracted according to the submatrix extraction method matching the number of check blocks to obtain the submatrix corresponding to each of the file data matrices.

4. The method according to claim 1, characterized in that: When the writing of each temporary check block is completed, migrating each temporary check block from the first storage medium to the second storage medium includes: Obtaining the writing status of each of the temporary check blocks; When the writing status of each of the temporary check blocks is writing completed, determining the integrity of the data blocks of each of the temporary check blocks; When the integrity of each data block meets the data migration condition, each temporary check block is migrated from the first storage medium to the second storage medium.

5. The method according to any one of claims 1 to 4, characterized in that: The step of migrating each of the temporary check blocks from the first storage medium to the second storage medium includes: Obtaining first computing capabilities of storage nodes to which each of the first storage media belongs, second computing capabilities of storage nodes to which the second storage media belongs, first read / write capabilities of the first storage media, and second read / write capabilities of the second storage media; Determining a read / write frequency that matches the first computing capability, the second computing capability, the first read / write capability, and the second read / write capability; According to the read and write frequency, each of the temporary check blocks is migrated from the first storage medium to the second storage medium.

6. The method according to claim 5, characterized in that The method further comprises: Determining the data block integrity of each of the file blocks; Determine the file block whose data block integrity meets the data recovery condition as the target block; Based on each file block except the target block and the temporary check block, data recovery is performed on the target block to obtain a recovered target data block.

7. A data file processing device, characterized in that: The device comprises: An acquisition module is used to acquire a data file to be stored, and perform file block processing on the data file to obtain multiple file blocks; A first writing module, used for writing each of the file blocks into a plurality of first storage spaces of a first storage medium respectively; A second writing module, configured to write each of the temporary check blocks generated for each of the file blocks into a second storage space of the first storage medium; A migration module is used to migrate each temporary check block from the first storage medium to a second storage medium when each temporary check block is written. The read and write speed of the first storage medium is greater than that of the second storage medium.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images