File storage method and device based on MinIO and MinIO distributed storage system
By constructing a bucket in the MINIO distributed storage system and encrypted the file sharding, it solves the problem of waste and operational inconvenience during the expansion of the system, realized flexible capacity expansion and disaster recovery, saving costs.
Patent Information
- Application Number
- CN202411633224.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-01-28
AI Technical Summary
MinIO distributed storage system needs peer-to-peer capacity expansion when expanding capacity. Adding a single disk cannot achieve capacity expansion, resulting in waste of disks and machines, and inconvenient operation during capacity reduction and disaster recovery scenarios.
In the Minio distributed storage system, by constructing a barrel by building a single -mounted disk, the files to be stored are divided into shards and encrypted storage, so as to achieve centralized storage of files. , Avoid the needs of re -deployment.
The flexible expansion, contraction and disaster recovery of the Minio distributed storage system, saved manpower and time costs, and improved the practicality of the system.
Smart Images
Figure CN119376636A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a MinIO-based file storage method and device, and a MinIO distributed storage system. Background Art
[0002] The MinIO distributed storage system uses erasure codes to divide files into data blocks and check blocks when storing files on the server, and stores them on different disks to facilitate the subsequent correction of missing or incorrect files. This storage method divides files into multiple small blocks, which is convenient for cyclic redundancy checks (CRC) and file recovery. In the case of a distributed deployment, the uploaded files of the Minio distributed storage system will be split onto each mounted disk. When downloading, each small block of data will be obtained from each download disk in turn and organized into a file. This will result in the need for peer-to-peer expansion when storage capacity is needed. It is not possible to add only a single disk, which is likely to cause waste of disks and machines when expanding capacity. Summary of the invention
[0003] The present invention provides a MinIO-based file storage method and device, and a MinIO distributed storage system, so as to overcome or at least partially overcome the deficiencies of the prior art.
[0004] In a first aspect, the present application provides a file storage method based on MinIO, the method is implemented based on a MinIO distributed storage system, the MinIO distributed storage system includes multiple mounted disks, and the method includes:
[0005] Determine whether any of the mounted disks has a storage bucket that has been used in the past. If so, store the file to be stored in the storage bucket in a fragmented and encrypted form.
[0006] If it does not exist, a storage bucket is constructed in any of the mounted disks based on the disk occupancy status, and the files to be stored are stored in the storage bucket in a fragmented and encrypted form.
[0007] In a second aspect, the present application further provides a file storage device based on MinIO, the device comprising:
[0008] The first storage unit is used to determine whether any of the mounted disks has a storage bucket that has been used in the historical period. If not, a storage bucket is constructed in any of the mounted disks based on the disk occupancy status, and the files to be stored are stored in the storage bucket in a fragmented and encrypted form;
[0009] The second storage unit is used to store the file to be stored in the storage bucket in a fragmented and encrypted form if it exists.
[0010] In a third aspect, the present application also provides a MinIO distributed storage system, which includes multiple mounted disks, at least one storage bucket is constructed on at least one of the mounted disks, and at least one file is stored in the storage bucket in a fragmented and encrypted form.
[0011] In a fourth aspect, the present application further provides a file reading method, wherein the file is stored in the aforementioned MinIO distributed storage system by any of the aforementioned MinIO-based file storage methods, and the file reading method comprises:
[0012] According to the file read request, locate the storage bucket where the target file is located;
[0013] Download multiple data blocks corresponding to the target file from the mounted disk where the storage bucket is located;
[0014] Decrypting the data block based on an inverse operation of the encryption algorithm;
[0015] Then the decrypted multiple data blocks are spliced together to obtain the target file.
[0016] In a fifth aspect, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of any of the above-mentioned MinIO-based file storage methods, or the steps of the above-mentioned file reading method.
[0017] In a sixth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned MinIO-based file storage methods, or the steps of the above-mentioned file reading method.
[0018] When storing files, this application builds a storage bucket in a single mounted disk in the MinIO distributed storage system, divides the files to be stored into several relatively large shards, and stores them in the storage bucket. In this way, all the files to be stored are located on one mounted disk. When expanding the system, you only need to add mounted disks without redeployment, which saves a lot of manpower and time costs. It is highly practical and convenient for the expansion, reduction and disaster recovery of the MinIO distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0020] Figure 1 A schematic diagram of a process of a file storage method based on MinIO according to an embodiment of the present application is shown;
[0021] Figure 2 A schematic diagram of the structure of a MinIO distributed storage system according to an embodiment of the present application is shown;
[0022] Figure 3 A schematic diagram of the structure of a file storage device based on MinIO according to an embodiment of the present application is shown;
[0023] Figure 4 is a schematic diagram of the structure of a computer device in one embodiment of the present invention;
[0024] Figure 5 It is a schematic diagram of the structure of a computer device in another embodiment of the present invention. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] MinIO is a high-performance object storage solution designed for cloud-native applications. It supports Amazon S3-compatible APIs, allowing users to easily migrate applications to MinIO. MinIO has high availability and scalability, and is suitable for storing massive amounts of data, especially in the fields of big data and machine learning.
[0027] The Minio distributed storage system uses erasure codes to divide files into data blocks and check blocks when storing files on the server in order to facilitate the subsequent correction of file missing or error problems, and stores them on different disks. In addition, this type of storage divides files into multiple small blocks to facilitate the calculation of file crc and file recovery. In the case of distributed deployment, the uploaded files of the Minio distributed storage system will be split into each mounted disk. When downloading, each small block of data will be obtained from each download disk in turn and organized into a file. This will result in the need for peer-to-peer expansion when the library capacity is needed. It is not possible to add only a single disk, and redeployment is required, which is easy to cause waste of disks and machines when expanding capacity. It is also inconvenient to operate in the face of capacity reduction and disaster recovery scenarios.
[0028] In this regard, the present application provides a file storage method based on MinIO, which uses storage buckets to store the files to be stored in a single mounted disk of the MinIO distributed storage system to facilitate the handling of capacity reduction, capacity expansion and disaster recovery.
[0029] Figure 1 A schematic flow chart of a MinIO-based file storage method according to an embodiment of the present application is shown. The MinIO-based file storage method can be implemented in a MinIO distributed storage system.
[0030] Figure 2 A schematic diagram of the structure of a MinIO distributed storage system according to an embodiment of the present application is shown. Figure 2 It can be seen that the MinIO distributed storage system 100 includes multiple mounted disks 110 . If necessary, the MinIO distributed storage system 100 also includes a central server 120 . Each mounted disk 110 is communicatively connected to the central server 120 . The central server 120 is used to manage each mounted disk 110 .
[0031] It should be noted that the MinIO-based file storage method of this application can be based on Figure 2 The structure shown is implemented, but is not limited to this. Any system and architecture that can implement the business logic of the MinIO-based file storage method of this application can be used.
[0032] from Figure 1 As shown, this embodiment includes steps S110 to S120:
[0033] Step S110: Determine whether any of the mounted disks has a storage bucket that has been used in the historical period. If not, build a storage bucket in any of the mounted disks based on the disk occupancy status, and store the files to be stored in the storage bucket in a fragmented and encrypted form.
[0034] When a user stores files in the MinIO distributed storage system, the files to be stored are recorded as files to be stored. Here, a user is used to store files for the first time. When a user stores files, if it is the first time, no storage bucket has been built for it in the mounted disk of the MinIO distributed storage system. For this user, there is no storage bucket used in the historical time. In this case, a storage bucket is built for any mounted disk with storage space in the MinIO distributed storage system. Then the files to be stored are stored in the storage bucket in a fragmented and encrypted form, so that all allocations of the files to be stored are stored in a single mounted disk.
[0035] For the construction of a storage bucket, each mounted disk can be traversed to find a mounted disk with storage space for construction. Specifically, in some embodiments, the construction of a storage bucket in any of the mounted disks based on the disk occupancy status includes: traversing the multiple mounted disks to determine whether each of the mounted disks has storage space; if the traversed mounted disk has storage space, then constructing the storage bucket in the traversed mounted disk; or; after traversing all mounted disks, selecting the mounted disk with the largest storage space to construct the storage bucket.
[0036] That is, in some embodiments, each mounted disk can be traversed, and when a mounted disk is traversed, if it has enough storage space to store the file to be stored, a storage bucket is constructed for the user in the current mounted disk. For example, when the first mounted disk is traversed, it has enough storage space, so a storage bucket is constructed in the first mounted disk, and the traversal ends.
[0037] In other embodiments, each mounted disk is traversed, and after all mounted disks are traversed, the remaining storage space of each mounted disk is read, and the mounted disk with the largest storage space is selected to construct a storage bucket.
[0038] After the storage bucket is built, the files to be stored are stored in the storage bucket.
[0039] Specifically, the storage is performed in the form of fragmentation and encryption. Fragmentation is to divide the file to be stored into multiple file fragments, and then encrypt each file fragment. The encryption can be performed using any of the existing technologies, and a check block (check code) will be formed after encryption. Then, each encrypted file fragment and the corresponding check block are stored in the storage bucket in a one-to-one correspondence.
[0040] In the prior art, the deployment method of the MinIO distributed storage system divides the files into multiple small blocks, which is convenient for cyclic redundancy checking and file recovery. However, in this way, a file will be split into many small blocks. When there are many files to be stored, it is easy to cause small fragmented files, which occupy the number of disk files and cause the disk space to be surplus, but there is no space to store files. In addition, the Minio distributed storage system adopts encrypted storage solutions such as SSE-S3, SSE-C, and SSE-KMS. This storage solution relies on the server to deploy TLS, the client to support the https protocol, and the encryption of the entire file. This method is complex to deploy and has a loss on system overhead.
[0041] In this regard, in some embodiments of the present application, the above-mentioned defects can be overcome by using a preset quantity threshold and partial encryption.
[0042] Specifically, in some embodiments of the present application, storing the file to be stored in the storage bucket in a fragmented and encrypted form includes: dividing the file to be stored into data blocks and check blocks with a preset number threshold; storing the data blocks in a data directory of the mounted disk where the storage bucket is located, and storing the check blocks in a check block directory of the mounted disk where the storage bucket is located; and encrypting and replacing at least part of each of the data blocks and each check block based on an encryption algorithm.
[0043] First, the file to be stored is divided into data blocks and check blocks with a preset number threshold.
[0044] The process of dividing a file into data blocks and parity blocks usually involves erasure coding technology. Data blocks are the parts into which the file to be stored is divided, and parity blocks are additional blocks used to recover the original data. If some data blocks are lost, they can be recovered using the existing parity blocks and other non-lost data blocks.
[0045] In some embodiments of the present application, the number of data blocks can be controlled by setting a preset number threshold, such as setting the number threshold to 4 (number of data blocks) and 4 (number of check blocks), and then dividing the file to be stored into 4 data blocks and 4 check blocks. This control method of setting the number threshold prevents the number of files from being too large, thereby avoiding the situation where there is excess disk space but no space to store files due to too many disk files.
[0046] In some embodiments, the preset number threshold can be dynamically adjusted according to the size of the file to be stored. For example, when a file to be stored is detected, the set number threshold can be automatically adjusted to 8 and 8, so that each file fragment is not too large. Once a data block is lost, a lot of data information is lost, and recovery is also difficult.
[0047] Then, the data block is stored in the data directory of the mounted disk where the storage bucket is located, and the check block is stored in the check block directory of the mounted disk where the storage bucket is located. In this way, multiple data blocks and multiple check blocks are stored in a single mounted disk.
[0048] Finally, there is the encryption process of the file. Specifically, based on the encryption algorithm, at least part of the file to be stored can be encrypted and replaced. The encryption algorithm can be any one of the existing technologies, but when encrypting, the entire text is not encrypted. Taking a data block as an example, in the preferred embodiment of the present application, only part of the data block is encrypted. Only part of the encryption is convenient for deployment and optimizes system performance. For more convenient execution, in some embodiments, based on the encryption algorithm, at least part of each of the data blocks is encrypted and replaced, including: based on the encryption algorithm, encrypting and replacing multiple bytes of the header of each of the data blocks and each check block respectively.
[0049] Step S120: If it exists, the file to be stored is stored in the storage bucket in a fragmented and encrypted form.
[0050] As mentioned above, if there is a storage bucket used by a user in the past, the files to be stored are directly stored in the storage bucket in a fragmented and encrypted form. The storage method is the same as before and will not be repeated here.
[0051] If the mounted disk where the storage bucket used by the user in the historical time is located does not have enough storage space, a storage bucket is constructed in the remaining mounted disks to store files for the user. The method of constructing the storage bucket in the remaining mounted disks is the same as before and will not be repeated here.
[0052] When reading a file, the storage bucket where the target file is located is located according to the file reading request; multiple data blocks corresponding to the target file are downloaded from the mounted disk where the storage bucket is located; the data blocks are decrypted based on the inverse operation of the encryption algorithm; and the decrypted multiple data blocks are spliced to obtain the target file.
[0053] The mechanism for recovering files is similar to that for reading files, that is, according to a file read recovery request, the storage bucket where the target file is located is located; one or more check blocks corresponding to the target file are downloaded from the mounted disk where the storage bucket is located; the check blocks are decrypted based on the inverse operation of the encryption algorithm; and data is recovered from at least one decrypted check block to obtain at least part of the lost file.
[0054] from Figure 1 It can be seen from the method shown that when storing files, the present application constructs a storage bucket in a single mounted disk in the MinIO distributed storage system, divides the files to be stored into several relatively large shards, and stores them in the storage bucket. In this way, all the files to be stored are located on one mounted disk. When expanding the system, only the mounted disk needs to be added without redeployment, which saves a lot of manpower and time costs. It is highly practical and convenient for the expansion, reduction and disaster recovery of the MinIO distributed storage system.
[0055] Figure 3 A schematic diagram of the structure of a file storage device based on MinIO according to an embodiment of the present application is shown. Figure 3 It can be seen that the file storage device 300 based on MinIO includes:
[0056] The first storage unit 310 is used to determine whether any of the mounted disks has a storage bucket that has been used in the historical time. If not, a storage bucket is constructed in any of the mounted disks based on the disk occupancy status, and the files to be stored are stored in the storage bucket in a fragmented and encrypted form;
[0057] The second storage unit 320 is used to store the file to be stored in the storage bucket in a fragmented and encrypted form if it exists.
[0058] In some embodiments, in the above-mentioned device, the first storage unit 310 is used to traverse the multiple mounted disks to determine whether each of the mounted disks has storage space; if the traversed mounted disk has storage space, then the storage bucket is constructed in the traversed mounted disk; or; after traversing all the mounted disks, the mounted disk with the largest storage space is selected to construct the storage bucket.
[0059] In some embodiments, in the above-mentioned device, the first storage unit 310 is used to divide the file to be stored into data blocks and check blocks of a preset number threshold; store the data blocks in the data directory of the mounted disk where the storage bucket is located, and store the check blocks in the check block directory of the mounted disk where the storage bucket is located; and based on the encryption algorithm, encrypt and replace at least part of each of the data blocks and each of the check blocks.
[0060] In some embodiments, in the above device, the first storage unit 310 is used to encrypt and replace multiple bytes of the header of each data block and each check block based on an encryption algorithm.
[0061] In some embodiments, in the above device, the preset quantity threshold can be dynamically adjusted according to the size of the file to be stored.
[0062] It should be noted that the above-mentioned MinIO-based file storage device can implement the above-mentioned MinIO-based file storage method one by one, which will not be repeated here.
[0063] In one embodiment, a computer device is provided. The computer device may be a service line server. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the steps of a file storage method based on MinIO are implemented, or the steps of the file reading method as described above are implemented.
[0064] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the steps of a file storage method based on MinIO are implemented, or the steps of the file reading method as described above are implemented.
[0065] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps of a file storage method based on MinIO are implemented, or the steps of the file reading method as described above are implemented.
[0066] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the following MinIO-based file storage method or the steps of the above-mentioned file reading method are implemented.
[0067] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions of the business line service side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0069] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0070] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A file storage method based on MinIO, characterized in that: The method is implemented based on the MinIO distributed storage system, the MinIO distributed storage system includes multiple mounted disks, and the method includes: Determine whether any of the mounted disks has a storage bucket that has been used in the past. If not, build a storage bucket in any of the mounted disks based on the disk occupancy status, and store the file to be stored in the storage bucket in a fragmented and encrypted form; If it exists, the file to be stored is stored in the storage bucket in a fragmented and encrypted form.
2. The MinIO-based file storage method according to claim 1, characterized in that: The step of constructing a storage bucket in any of the mounted disks based on the disk occupancy status includes: Traversing the plurality of mounted disks to determine whether storage space exists on each of the mounted disks; If the traversed mounted disk has storage space, construct the storage bucket in the traversed mounted disk; or; After traversing all mounted disks, select the mounted disk with the largest storage space to build the storage bucket.
3. The file storage method based on MinIO according to claim 1, characterized in that: The storing the file to be stored in the storage bucket in a fragmented and encrypted form includes: Dividing the file to be stored into data blocks and check blocks with a preset number threshold; The data block is stored in the data directory of the mounted disk where the storage bucket is located, and the check block is stored in the check block directory of the mounted disk where the storage bucket is located; Based on the encryption algorithm, at least a part of each of the data blocks and each of the check blocks is encrypted and replaced respectively.
4. The MinIO-based file storage method according to claim 3, characterized in that: The step of encrypting and replacing at least a portion of each of the data blocks and each of the check blocks based on an encryption algorithm comprises: Based on the encryption algorithm, multiple bytes of the header of each data block and each check block are encrypted and replaced respectively.
5. The method according to claim 3, characterized in that: The preset quantity threshold may be dynamically adjusted according to the size of the file to be stored.
6. A file storage device based on MinIO, characterized in that: The device comprises: The first storage unit is used to determine whether any of the mounted disks has a storage bucket that has been used in the historical period. If not, a storage bucket is constructed in any of the mounted disks based on the disk occupancy status, and the files to be stored are stored in the storage bucket in a fragmented and encrypted form; The second storage unit is used to store the file to be stored in the storage bucket in a fragmented and encrypted form if it exists.
7. A MinIO distributed storage system, characterized in that: The Minio distributed storage system includes multiple mounted disks, at least one storage bucket is constructed on at least one of the mounted disks, and at least one file is stored in the storage bucket in a fragmented and encrypted form.
8. A file reading method, characterized in that: The file is stored in the MinIO distributed storage system according to claim 7 by the MinIO-based file storage method according to any one of claims 1 to 5, and the file reading method includes: According to the file read request, locate the storage bucket where the target file is located; Download multiple data blocks corresponding to the target file from the mounted disk where the storage bucket is located; Decrypting the data block based on an inverse operation of the encryption algorithm; Then the decrypted multiple data blocks are spliced together to obtain the target file.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the MinIO-based file storage method according to any one of claims 1 to 5 are implemented, or the steps of the file reading method according to claim 8 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the MinIO-based file storage method according to any one of claims 1 to 5 are implemented, or the steps of the file reading method according to claim 8 are implemented.