Erasure correction storage method, system and device of blue light storage device and medium

Abstract

The invention discloses an erasure storage method, system, device and medium for a blue-ray storage device, and relates to the technical field of data storage, and the method comprises the following steps: constructing a layer of independent disk buffer area; when the file is uploaded, the size of an uploaded object is obtained from the uploading request parameters, and Chunk is selected according to the size of the object to store the file content; meanwhile, the object size, the uploading time and the address information in the Chunk serve as metadata to be stored in an object metadatabase, and file content is stored in a disk buffer area; acquiring erasure codes from the Chunk in the disk buffer area, and storing the erasure codes in the disk buffer area; when the file is downloaded, the file is read from the disk buffer area, file content is generated through the Chunk corresponding to the file and the check Chunk, and the file content is downloaded to the blue-ray storage device. In the recording process, only the work of recording the data slices and the erasure slices to the optical disc is completed, and therefore the recording performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to an erasure storage method, system, device and medium for a Blu-ray storage device. Background Art

[0002] Blu-ray storage devices can migrate data that needs to be preserved for a long time from online storage (such as distributed storage systems based on disk media) to read-only, offline optical media and preserve it for a long time. Therefore, they play a key role in modern large-capacity, long-term, secure data storage systems.

[0003] Existing Blu-ray storage management technology generally introduces a stand-alone storage management server, which first caches the files to be burned onto the disc on the local disk of the storage management server. The storage management program then executes a dynamic "accumulation" algorithm to accumulate several files to a capacity close to that of a disc (generally 80-85% of 100GB), and then burns these files onto the disc in sequence.

[0004] When burning files on a disc, it is necessary to calculate the erasure code, connect the accumulated files in order, and then slice them according to a fixed size (such as 10MB). The slices on different discs form an "erasure stripe", calculate the erasure slices according to the preset erasure mode (such as 5+1, 11+1, etc.), and then cache the erasure slices on the local disk. When all data discs are burned, all erasure slices are burned, but the calculation process of the erasure code and the burning process are mixed together, or the erasure code is dynamically calculated during the burning process; due to the burning characteristics of the disc, it is possible that the slice fails to burn during the burning process. However, the above accumulation process has fixed the relationship between the slice and the disc, that is, a slice must be burned to a specific disc, resulting in all the space of the disc being fully occupied, which will cause the entire burning process to fail. Therefore, in the accumulation process, all the space of the disc cannot be fully occupied, and a buffer must be given for the slice to be burned again due to failure. Under the existing technology, this buffer is 15-20%. This space is basically wasted. Summary of the invention

[0005] The purpose of the present invention is to provide an erasure storage method, system, device and medium for a Blu-ray storage device in view of the deficiencies of the above-mentioned prior art, so as to solve the problem that the accumulation process in the prior art has fixed the relationship between slices and optical discs, that is, a slice must be burned to a specific optical disc, resulting in full occupation of all the space of the optical disc, which will cause the failure of the entire burning process. Therefore, in the accumulation process, all the space of the optical disc cannot be fully occupied, and a buffer must be provided for slices that fail to burn and need to be re-engraved. Under the prior art, this buffer is 15-20%. This space is basically wasted.

[0006] The present invention specifically provides the following technical solutions: A method for erasure correction storage of a Blu-ray storage device comprises the following steps: Build an independent disk buffer layer; When uploading a file, the size of the object to be uploaded is obtained from the upload request parameters according to the S3 object storage protocol, and the corresponding Chunk is selected to save the file content according to the object size; at the same time, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, and the Chunk corresponding to the file content is saved to the disk buffer; Use the erasure batch processing task to process the chunks in the disk buffer to obtain the erasure code, and save the erasure code in the disk buffer in the form of verification chunks; When downloading a file, metadata is read from the object metadata database to determine the location of the object, the chunk and verification chunk corresponding to the file are read from the disk buffer, the file content is generated through the chunk and verification chunk corresponding to the file, and downloaded to the Blu-ray storage device.

[0007] Preferably, the step of constructing an independent disk buffer layer includes: Select a local file system, NAS space, or distributed file system as the disk buffer.

[0008] Preferably, the erasure batch processing task is used to obtain the erasure code for the Chunk in the disk buffer, and the erasure code is stored in the disk buffer in the form of a verification Chunk, including: Execute the erasure batch processing task in the form of a scheduled task, batch scan the un-erased chunks, and perform erasure operations on the un-erased chunks; Based on the erasure operation, K chunks are selected to form an EC-Set through the specified erasure mode, and M check chunks in the EC-Set are calculated, and the calculated check chunks are written back to the disk buffer; Among them, the unerased chunk is stored in the object metadata database, and the unerased chunk only has the key value but no actual data content Value.

[0009] Preferably, before reading metadata from the object metadata database and determining the location of the object, the method further includes: The Chunk corresponding to the object has been recorded on the disc, but a copy is stored in the disk buffer. In this case, the Chunk is directly read to generate the file content. The Chunk corresponding to the object has been burned on the CD, and there is no copy in the disk buffer. At this time, a failure response is returned to the client in accordance with the S3 object storage protocol, prompting the business system to perform a file migration operation first. After the migration is completed, the Chunk is directly read to generate the file content.

[0010] Preferably, the downloading to a Blu-ray storage device includes: The data chunks and check chunks generated by the erasure batch processing task are burned from the disk buffer to the optical disc of the Blu-ray device in the form of files, and the data chunks and check chunks are cleared from the disk buffer.

[0011] The present invention provides an erasure correction storage system for a Blu-ray storage device, comprising: Buffer building module, used to build an independent disk buffer layer; The file upload module is used to obtain the size of the uploaded object from the upload request parameters according to the S3 object storage protocol when uploading files, and select the corresponding Chunk to save the file content according to the object size; at the same time, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, and the Chunk corresponding to the file content is saved to the disk buffer; The erasure processing module is used to process the Chunks in the disk buffer using the erasure batch processing task to obtain the erasure code, and save the erasure code in the disk buffer in the form of a check Chunk; The file download module is used to read metadata from the object metadata database when downloading files, determine the location of the object, read the chunk and verification chunk corresponding to the file from the disk buffer, generate file content through the chunk and verification chunk corresponding to the file, and download it to the Blu-ray storage device.

[0012] The present invention provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the erasure correction storage method of the above-mentioned Blu-ray storage device.

[0013] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the erasure correction storage method of a Blu-ray storage device are implemented.

[0014] Compared with the prior art, the present invention has the following significant advantages: The present invention constructs a layer of independent disk buffer. When a file is written to the disk buffer, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, the Chunk corresponding to the file content is saved in the disk buffer, and the Chunk in the disk buffer is processed by an erasure batch processing task to obtain an erasure code, and the erasure code is saved in the disk buffer in the form of a verification Chunk. That is, during the calculation process and burning process of erasure, only the work of burning data slices and erasure slices to an optical disc is completed without other calculation steps, thereby improving the burning performance and the space utilization rate of optical disc burning. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The overall flow chart of erasure storage of a Blu-ray storage device of the present invention is as follows; Figure 2 This is a corresponding relationship diagram between the file and Chunk of the present invention. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0017] like Figure 1 As shown, the present invention provides an erasure correction storage method for a Blu-ray storage device, which specifically includes the following steps: Step 1: Build a layer of independent disk buffer.

[0018] Select a local file system, NAS space, or a distributed file system and object storage as the disk buffer. The disk buffer is used to store file contents. Chunks and checksum chunks are stored in the disk buffer in the form of files.

[0019] Step 2: When uploading a file, obtain the size of the object from the upload request parameters according to the S3 object storage protocol, and select the corresponding Chunk to save the file content based on the object size; at the same time, save the object size, upload time and address information in the Chunk as metadata in the object metadata database, save the Chunk corresponding to the file content to the disk buffer, and use the Chunk's unique identifier ChunkID as the file name.

[0020] The S3 object storage gateway provides an S3-based storage access protocol for external business systems. External business systems upload and download files through the S3 object storage gateway in the figure.

[0021] The object metadata database is used to store the four data tables shown in Tables 2 to 5. In order to ensure that the system has high response performance and data consistency, the object metadata database in the present invention is implemented by a Key-Value database that supports distributed transactions, and a typical transaction-type Key-Value database is TIKV. In the following Tables 2 to 5, the first column represents the key Key, and the following columns represent the various fields in the actual data content Value.

[0022] Step 3: Use the erasure batch processing task to process the chunks in the disk buffer, obtain the erasure code, and save the erasure code in the form of verification chunks in the disk buffer.

[0023] The erasure batch processing task is executed in the form of a scheduled task, which scans the un-erased Chunk table in batches and performs erasure operations on the un-erased Chunk table. Among them, the un-erased Chunk table is stored in the object metadata database, and the un-erased Chunk table only has the key value, and no actual data content Value value. The main part of the Key is the ChunkID.

[0024] Based on the erasure operation, through the specified erasure mode (K+M), K chunks are selected to form an EC-Set, and the M check chunks in this EC-Set are calculated, and the calculated check chunks are written back to the disk buffer.

[0025] Step 4: When downloading a file, metadata is read from the object metadata database to determine the location of the object, the chunk and verification chunk corresponding to the file are read from the disk buffer, the file content is generated through the chunk and verification chunk corresponding to the file, and downloaded to the Blu-ray storage device.

[0026] When downloading a file, the object metadata is read from the object metadata database, and the chunk on the disc cannot be read directly. This is because reading the chunk on the disc is a slow operation that requires a robot to move the disc, which takes an average of about 5 minutes of delay.

[0027] Read metadata from the object metadata database to determine the location of the object, and also include: The Chunk corresponding to the object has been recorded on the CD, but a copy is stored in the disk buffer. At this time, the Chunk is directly read and merged into the file content.

[0028] The chunk corresponding to the object has been burned on the CD, and there is no copy in the disk buffer. At this time, a failure response is returned to the client in accordance with the S3 object storage protocol, prompting the business system to perform the file migration operation first. After the migration is completed, the chunk is directly read and merged into the file content.

[0029] When downloading to a Blu-ray storage device, also includes: The data chunks and check chunks generated by the erasure batch processing task are burned from the disk buffer to the optical disc of the Blu-ray device in the form of files, and then the data chunks and check chunks are cleared from the disk buffer.

[0030] Figure 2 This shows how the file contents are stored in Chunks. Figure 2 In the example shown, the upper part shows the logical structure of a subdirectory in the file system. In the tmp subdirectory, there are two subdirectories, dir1 and dir2, and the QQ.exe file. There are also subdirectories and files in dir1 and dir2 respectively. When all the contents in the tmp subdirectory are uploaded to the system described in the present invention, the above tmp subdirectory tree is first converted into the following file list according to the absolute path of each file: Table 1 Example file list

[0031] Then, the file contents of each file in the above file list are loaded into the Chunk in turn. Chunk (data block) in the present invention refers to the carrier of file content, which has a fixed size (such as 10MB) and a unique coding identifier. The file contents of small files less than 10MB are merged to form a Chunk. Large files exceeding 10MB are cut and loaded into different Chunks.

[0032] exist Figure 2 In the example, / tmp / dir1 / a.mov is first placed in Chunk-1, and / tmp / dir2 / dir3 / bcd.dat is then placed in the remaining space of Chunk-1. The / tmp / dir2 / M.dat file is relatively large, so it is divided into three parts and placed in Chunk-1, Chunk-2, and Chunk-3. Then, / tmp / dir2 / N.dat is divided into two parts and placed in the remaining space of Chunk-3 and Chunk-4. Finally, / tmp / QQ.exe is placed in the remaining space of Chunk-4.

[0033] The metadata of each file in the above file list is recorded in the metadata table. According to the metadata table, the absolute path of the file can be used to query which chunk each fragment of the file is in, as well as the specific position (offset, length) in the chunk.

[0034] As shown in Table 2, each row in the metadata table records the metadata of a file. The metadata table uses the absolute path of the file as the key (the first column in the table), and the remaining columns represent the fields in the Value. The Value field includes the file size, creation time, and the Chunk tuple list. The Chunk tuple records (Chunk-ID, offset byte number in the Chunk, length).

[0035] Table 2 Metadata table

[0036] As mentioned above, Figure 1 After receiving the file upload request, the S3 object storage gateway allocates the corresponding Chunk list according to the size of the requested file, slices the file content according to the space in the allocated Chunk, and writes it to each Chunk (writes Chunk file). Then the file metadata information is updated to Figure 1 The object metadata database.

[0037] The object metadata database is a key-value database that supports transactions. Therefore, multiple key-value pair records can be written in one transaction. The writing of these records meets the indivisibility of the transaction. That is, if any of the records fails to be written, the entire transaction fails and is rolled back. The data consistency of the object metadata database can be guaranteed through transactionality.

[0038] During the file writing process, in addition to updating the Table 2 metadata table, Table 3 must also be updated in the same transaction.

[0039] In Table 3, the newly written Chunk has its status field marked as NOT_FULL, indicating that the Chunk is not yet full. If all the space in the Chunk has been used up, it is marked as FULL.

[0040] As shown in Table 3, each row in the Chunk information table records a Chunk information. The Chunk information table uses ChunkID as the key (the first column in the table), and the remaining columns represent the fields in the Value. The Value field includes the file slice list contained in the Chunk, Chunk type, status, EC-SetID, and CD address (not shown in the table).

[0041] Table 3 Chunk information table

[0042] Among these fields, the Chunk type field records the type of the Chunk. The Chunk can be a data Chunk (indicated by D) or a check Chunk (indicated by P).

[0043] If the erasure mode set by the system is K+M, the type of data Chunk is D-1, D-2, D-3, …, DK, and the number after the horizontal line indicates that the Chunk is located on the erasure line; similarly, the type of check Chunk is P-1, P-2, …, PM, and the number after the horizontal line indicates that the Chunk is located on the erasure line.

[0044] The status field records the status of the Chunk. NOT_FULL means that the Chunk can still accept new file content; FULL means that the Chunk is full but has not yet been erasure calculated. EC means that the Chunk has been erasure calculated and included in an EC-Set. The ID of the EC-Set is recorded in the EC-Set ID field.

[0045] Explanation of erasure codes: Erasure codes are a data protection method. In the present invention, the basic unit of erasure is Chunk. N Chunks are arranged together in order, and M check Chunks are calculated according to a specific algorithm (such as the Reed-Solomon algorithm) and a preset erasure mode (K+M). K data Chunks and M check Chunks form a tuple, called an EC-Set. Within an EC-Set, it is allowed to lose any M data Chunks or check Chunks. The lost M Chunks can be restored from the remaining Chunks through the erasure algorithm.

[0046] Figure 1 The erasure batch processing task in periodically scans Table 3 and selects the chunks with the status field as FULL. According to the erasure mode (K+M) specified by the optical disk logical pool, K chunks with the status as FULL are placed in an EC-set, and M check chunks are calculated according to the erasure algorithm (such as the Reed-Solomon algorithm) specified by the optical disk logical pool. The calculated check chunks are also written Figure 1 Afterwards, in a metadata database transaction, the information of the check chunk is updated to Table 3, and the information of the newly created EC-set is updated to Table 4. After the transaction is completed, the erasure calculation process of an EC-Set is completed.

[0047] As shown in Table 4, each row in the EC-Set information table records the information of an EC-Set. The EC-Set information table uses the EC Set ID as the key (the first column in the table), and the remaining columns represent the fields in the Value. The Value field includes the Chunk list contained in the ECSet and the EC-Group ID. When an EC-Set is newly created, it is not assigned to any EC-Group. Afterwards, the Blu-ray migration batch task assigns the EC-Set to an EC-Group and updates the EC-Set information table.

[0048] Table 4 EC-Set Information Table

[0049] EC-Group is a collection of N optical disks combined according to the erasure mode N=K+M. EC-Group is also a collection of several EC-Sets, and each Chunk in these EC-Sets is stored on the N optical disks specified by the EC-Group. For example, if the erasure mode is set to N=K+M), then an EC-Set will store its Chunks on N optical disks.

[0050] As shown in Table 5, each row in the EC-Group information table records the information of an EC-Group. The EC-Group information table uses the EC Group ID as the key (the first column in the table), and the remaining columns represent the fields in the Value.

[0051] The Value field includes the list of optical disks assigned to the EC Group and the list of EC-Sets contained in the Group.

[0052] Table 5 EC-Group information table

[0053] The Blu-ray migration batch task periodically scans Table 4 and selects EC-Sets with empty EC-Group IDs. After scanning these EC-Sets, the K+M Chunks contained in the EC-Set can be burned in parallel to the K+M optical discs specified by the EC-Group. Since the batch task always burns Chunk files of fixed length (10MB) to the optical disc in batches each time, and there is no need to calculate the checksum in real time, the K+M optical drives can always burn at a uniform speed, so that multiple optical drives in the optical disc library can work in parallel and at a uniform speed at a high speed, so that the optical disc library can run at or close to the maximum throughput performance.

[0054] The terms used in the present invention are shown in Table 6 and Table 7: Table 6 Terminology used in the present invention 1

[0055] Table 7 Terminology used in the present invention 2

[0056] Based on the above method, the present invention provides an erasure correction storage system for a Blu-ray storage device, comprising: a buffer construction module, a file upload module, an erasure correction processing module and a file download module.

[0057] Among them, the buffer construction module is used to build a layer of independent disk buffer; the file upload module is used to obtain the size of the uploaded object from the upload request parameters according to the S3 object storage protocol when uploading files, and select the corresponding Chunk to save the file content according to the object size; at the same time, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, and the Chunk corresponding to the file content is saved in the disk buffer; the error correction and erasure processing module is used to obtain the erasure code for the Chunk in the disk buffer using the error correction and erasure batch processing task, and save the erasure code in the disk buffer in the form of a verification Chunk; the file download module is used to read the metadata from the object metadata database when downloading files, determine the location of the object, read the Chunk and verification Chunk corresponding to the file from the disk buffer, generate the file content through the Chunk and verification Chunk corresponding to the file, and download it to the Blu-ray storage device.

[0058] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of an erasure correction storage method of a Blu-ray storage device.

[0059] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0060] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, steps of an erasure correction and storage method of a Blu-ray storage device are implemented.

[0061] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for erasure correction storage of a Blu-ray storage device, characterized in that: The steps include: Build an independent disk buffer layer; When uploading a file, the size of the object to be uploaded is obtained from the upload request parameters according to the S3 object storage protocol, and the corresponding Chunk is selected to save the file content according to the object size; at the same time, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, and the Chunk corresponding to the file content is saved to the disk buffer; Use the erasure batch processing task to process the chunks in the disk buffer to obtain the erasure code, and save the erasure code in the disk buffer in the form of verification chunks; When downloading a file, metadata is read from the object metadata database to determine the location of the object, the chunk and verification chunk corresponding to the file are read from the disk buffer, the file content is generated through the chunk and verification chunk corresponding to the file, and downloaded to the Blu-ray storage device.

2. The erasure correction storage method of a Blu-ray storage device according to claim 1, characterized in that: The construction of an independent disk buffer layer includes: Select a local file system, NAS space, or distributed file system as the disk buffer.

3. The erasure correction storage method of a Blu-ray storage device according to claim 1, characterized in that: The erasure batch processing task is used to process the Chunks in the disk buffer to obtain the erasure codes, and the erasure codes are stored in the disk buffer in the form of verification Chunks, including: Execute the erasure batch processing task in the form of a scheduled task, batch scan the un-erased chunks, and perform erasure operations on the un-erased chunks; Based on the erasure operation, K chunks are selected to form an EC-Set through the specified erasure mode, and M check chunks in the EC-Set are calculated, and the calculated check chunks are written back to the disk buffer; Among them, the unerased Chunk is stored in the object metadata database, and the unerased Chunk only has the key value, but no actual data content Value.

4. The erasure correction storage method of a Blu-ray storage device according to claim 1, characterized in that: Before reading metadata from the object metadata database and determining the location of the object, the method further includes: The Chunk corresponding to the object has been recorded on the disc, but a copy is stored in the disk buffer. In this case, the Chunk is directly read to generate the file content. The Chunk corresponding to the object has been burned on the CD, and there is no copy in the disk buffer. At this time, a failure response is returned to the client in accordance with the S3 object storage protocol, prompting the business system to perform a file migration operation first. After the migration is completed, the Chunk is directly read to generate the file content.

5. The erasure correction storage method of a Blu-ray storage device according to claim 1, characterized in that: The downloading to the Blu-ray storage device includes: The data chunks and check chunks generated by the erasure batch processing task are burned from the disk buffer to the optical disc of the Blu-ray device in the form of files, and the data chunks and check chunks are cleared from the disk buffer.

6. An erasure storage system for a Blu-ray storage device, characterized in that: include: Buffer building module, used to build a layer of independent disk buffer; The file upload module is used to obtain the size of the uploaded object from the upload request parameters according to the S3 object storage protocol when uploading files, and select the corresponding Chunk to save the file content according to the object size; at the same time, the object size, upload time and address information in the Chunk are saved as metadata in the object metadata database, and the Chunk corresponding to the file content is saved to the disk buffer; The erasure processing module is used to process the Chunks in the disk buffer using the erasure batch processing task to obtain the erasure code, and save the erasure code in the disk buffer in the form of a check Chunk; The file download module is used to read metadata from the object metadata database when downloading files, determine the location of the object, read the chunk and verification chunk corresponding to the file from the disk buffer, generate file content through the chunk and verification chunk corresponding to the file, and download it to the Blu-ray storage device.

7. A computer device, characterized in that: It comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the erasure correction storage method of a Blu-ray storage device as claimed in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the erasure correction storage method of a Blu-ray storage device according to any one of claims 1 to 5 are implemented.

Images