Distributed file storage method

Abstract

The invention relates to the technical field of data storage, in particular to a distributed file storage method. In particular to a distributed file storage method based on a ShardingSphere distributed SQL (Structured Query Language) transaction and query engine, an object storage service of a MinIO open source protocol and a Mysql database. The advantages of all middleware are comprehensively integrated, MinIO serves as the basis of file storage, ShardingSphere and Mysql serve as auxiliary storage file meta-information, the capacity of uploading, querying and deleting open files is achieved through a storage service application API, and the distributed file storage method which is expansible, easy to backup and high in performance is provided for mass file storage.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a distributed file storage method, which is specifically a distributed file storage method based on ShardingSphere distributed SQL transaction and query engine, MinIO open source protocol object storage service, and MySQL database. Background Art

[0002] With the development of the Internet and big data, the storage of massive unstructured data on the Internet has exploded, such as: massive product images on e-commerce websites, massive video files on video websites, massive images on social networking websites, etc., all of which require massive file storage technology. However, file storage in traditional industries is integrated with application services and deployed on the same server, which cannot meet the storage needs of massive files. When the number of files reaches a certain level, there will be a series of problems such as serious degradation of access performance, insufficient storage space and difficulty in expansion, and difficulty in sharing files across applications.

[0003] Mini0 is an object storage service based on the Apache license v2.0 open source protocol. It is very suitable for storing large-capacity unstructured data, such as pictures, videos, files, backup data, etc. An object file can be of any size, ranging from a few KB to a maximum of 5T.

[0004] ShardingSphere is an open source distributed database middleware, which aims to fully and reasonably utilize the computing and storage capabilities of relational databases in distributed scenarios. By splitting data horizontally or vertically and then storing the split data in different databases or tables, it can reduce the data storage and query pressure of a single database and table, thereby improving the performance of data operations. Therefore, we combine the technical advantages of MinI0 and ShardingSphere to propose a distributed file storage method that can well solve the problems of storage, query, and deletion of massive files and file meta-information in distributed scenarios. Summary of the invention

[0005] The present invention solves the deficiencies of the prior art and provides a distributed file storage method; the purpose is to solve the problems of complexity, scalability, data backup, data consistency and reliability faced by mass file storage.

[0006] A distributed file storage method, comprising: Step 1: Call the file upload interface provided by the file storage service, submit the file and file meta-information data through the interface, and forward it to the file storage service application through the ngnix proxy; the file upload interface provided by the file storage service application is an Http Post request interface, and the file and file meta-information data are submitted through the Json string interface background processing; Step 2: The file storage service application communicates with the client node in the MinIO cluster, initiates a file upload request, and routes it to the nearest available storage server RAMacher. RAMacher is responsible for sharding and replicating the file, dividing the file into 10MB fragments, and each fragment is independently uploaded to different servers in the cluster; when all fragments are uploaded, RAMacher merges these fragments into a complete file and uses the file meta information in the client node, which consists of the file name, size, and type, to generate the file's unique identifier id; Before uploading files between the file storage service application and the MinIO cluster, you need to build a MinIO file server cluster to establish a file storage service application, and use the springboot environment to build an application service for uploading files, querying, deleting, and saving file metadata. When building the file storage service application, you need to introduce the minio java client to operate files in the MinIO cluster. Secondly, the file storage service application integrates the ShardingSphere table and library sharding component. Step 3: Call the upload interface of the file storage service application through Http to upload the file. If the file upload fails, the file storage service application will throw an exception and return a prompt message indicating that the upload failed. Step 4: If the file is uploaded successfully, the MinIO cluster will return the file metadata including the file path; Step 5: The file storage service application receives the file storage path returned by the MinIO cluster client, generates a unique file ID for each file using the snowflake algorithm, calls the ShardingSphere component API to perform data sharding of the file meta information, uses the file meta information as the unique ID, and saves the file name, size, and file storage path to the Mysql database; Step 6: The file storage service application will eventually return the file storage result to the interface caller. If the storage is successful, the file unique identification ID will be returned. The file viewing and deletion interfaces provided by the file storage service application will be called according to the file unique identification ID to realize file viewing and deletion.

[0007] In step 2, the MinIO cluster will divide the file into 10MB segments and copy them to other servers.

[0008] In step 5, data sharding is performed by ShardingSphere according to the file unique identification id modulo the hash algorithm, and the result range is set to 0-N, where N represents the number of servers in the Mysql cluster. If the number of servers in the Mysql cluster is 2, then N=2; if the file unique identification id modulo the hash algorithm result is 0, it is stored in the database db1, and if the file unique identification id modulo the hash algorithm result is 1, it is stored in the database db2; create two tables, table1 and table2, in db1 and db2 respectively, divide the total number of file meta information ids stored in db1 or db2 by 2 and take the remainder. If the remainder is 0, it is stored in table 1, otherwise it is stored in table 2.

[0009] Beneficial Effects 1. This patent comprehensively integrates the advantages of various middlewares, takes MinIO as the basis for file storage, uses ShardingSphere and Mysql as auxiliary storage for file metadata, and opens up the ability to upload, query, and delete files to the outside world through the storage service application API. It creates a combined storage method of ShardingSphere, MinIO, Mysql, and storage service applications, providing a scalable, easy-to-backup, and high-performance distributed file storage method for massive file storage.

[0010] 2. This patent uses ShardingSphere combined with MySQL database to store file meta-information data, and utilizes ShardingSphere's flexible data sharding, high availability and horizontal expansion capabilities to improve file meta-information query performance and simplify database processing capabilities for massive file meta-information.

[0011] 3. This patent provides the ability to upload, query, and delete files through storage services, which reduces usage costs and provides quick access capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of the file storage architecture of the present invention; Figure 2 This is a schematic diagram of the traditional file architecture. DETAILED DESCRIPTION

[0013] A distributed file storage method, comprising: Step 1: Call the file upload interface provided by the file storage service, submit the file and file meta-information data through the interface, and forward it to the file storage service application through the ngnix proxy; the file upload interface provided by the file storage service application is an Http Post request interface, and the file and file meta-information data are submitted through the Json string interface background processing; Step 2: The file storage service application communicates with the client node in the MinIO cluster, initiates a file upload request, and routes it to the nearest available storage server RAMacher. RAMacher is responsible for sharding and replicating the file, dividing the file into 10MB fragments, and each fragment is independently uploaded to different servers in the cluster; when all fragments are uploaded, RAMacher merges these fragments into a complete file and uses the file meta information in the client node, which consists of the file name, size, and type, to generate the file's unique identifier id; Before uploading files to the file storage service application and MinIO cluster, it is necessary to build a MinIO file server cluster to establish a file storage service application, and use the springboot environment to build an application service for uploading files, querying, deleting and saving file metadata; when building the file storage service application, it is necessary to introduce the minio java client to operate files in the MinIO cluster; secondly, the file storage service application integrates the ShardingSphere table and library sharding component; in step 2, the MinIO cluster will divide the file into 10MB fragments and copy them to other servers.

[0014] Step 3: Call the upload interface of the file storage service application through Http to upload the file. If the file upload fails, the file storage service application will throw an exception and return a prompt message indicating that the upload failed. Step 4: If the file is uploaded successfully, the MinIO cluster will return the file metadata including the file path; Step 5: The file storage service application receives the file storage path returned by the MinIO cluster client, generates a file unique identification id for each file using the snowflake algorithm, calls the ShardingSphere component API to perform data sharding of the file meta information, uses the file meta information as the unique identification id, and saves the file name, size, and file storage path to the Mysql database; the data sharding in step 5 is performed by ShardingSphere according to the file unique identification id using the hash algorithm modulo, and the result range is specified to 0-N, where N represents the number of servers in the Mysql cluster. If the number of servers in the Mysql cluster is 2, then N=2; if the file unique identification id is 0 when the hash algorithm is modulo, it is stored in the database db1; if the file unique identification id is 1 when the hash algorithm is modulo, it is stored in the database db2; create two tables, table1 and table2, in db1 and db2 respectively, divide the total number of file meta information ids stored in db1 or db2 by 2 and take the remainder. If the remainder is 0, it is stored in table table1, otherwise it is stored in table2.

[0015] Step 6: The file storage service application will eventually return the file storage result to the interface caller. If the storage is successful, the file unique identification ID will be returned. The file viewing and deletion interfaces provided by the file storage service application will be called according to the file unique identification ID to realize file viewing and deletion.

[0016] The difference between this invention patent and other similar public patents 1. The difference between the present invention patent and a data processing method, device and system of a distributed version control system; Patent publication number CN103647850A: Different application scenarios: The main application scenario of the present invention patent is to provide a fast file storage method for massive file storage, solving various complex problems encountered in the file storage process. The data processing method, device and system patent of a distributed version control system is mainly used for the data processing of the Git version control system on the Git server.

[0017] Different technical solutions: The present invention uses ShardingSphere combined with MySQL database to store file metadata for the storage of massive files, and uses MinI0 to build a cluster to store files. The patented technical solution of a data processing method, device and system of a distributed version control system is to process the data of the Git version control system with Hbase and HDFS.

[0018] Different technical problems solved: The main pain points solved by the present invention patent are the complexity, scalability, data backup, data consistency and reliability faced by massive file storage. The main pain points solved by the patent of a data processing method, device and system of a distributed version control system are the input and output contention and low efficiency of data serial processing caused by the storage process of the packaged files received by the Git client when the Git server is hung on multiple server nodes.

[0019] 2. The difference between the present invention patent and the distributed file storage system and its implementation method; Patent No. CN103092891A: Different technical problems solved: The present invention patent mainly solves the pain points of complexity, scalability, data backup, data consistency and reliability faced by massive file storage. The distributed file storage system and its implementation method mainly solve the pain point of dynamically realizing the on-demand startup of the distributed file storage system to avoid many problems caused by starting the distributed file storage service in advance. The two patents are completely different in their research and solution directions.

[0020] Embodiment: Further explanation is given below in conjunction with the illustrations: File storage mainly consists of two parts: 1. Files are stored in a Mini0 cluster, storing files in one or more partitions of the cluster server.

[0021] 2. File meta information, i.e., the descriptive information of the file, mainly including file ID (file unique identifier), file name, file type, file size, file storage path, etc. File meta information is mainly stored in relational databases, such as MySQL; Figure 1 is a schematic diagram of the architecture of the present invention, Figure 2 The following is a schematic diagram of the architecture of traditional file storage. Based on the comparison of the two architecture diagrams, the traditional file storage architecture has the following three main problems: 1. File storage space bottleneck problem: Figure 2 We can clearly see that traditional file storage and business application services share the same server, and business application services and file storage services are deeply integrated. This architecture makes it difficult to expand file storage services and file storage disk space. The performance degradation of either the file storage service or the business application service will affect the performance of the entire application.

[0022] 2. Bottleneck problem of file metadata storage: Figure 2 As we can see, the traditional file storage architecture uses a single database table to store file metadata, and even the same database is used for business data storage and file metadata storage. When business data or file metadata data reaches a certain level, it seriously affects the efficiency of data addition, deletion, modification and query, and even causes application downtime.

[0023] 3. Difficulties in file backup: Figure 1-2 In the traditional architecture, we can clearly see that the uploaded files are directly stored in the disk directory of the server without any backup or security measures. If backup is required, programmers need to write program code for backup, or operation and maintenance personnel need to perform manual backup regularly. Manual backup has time difference, and the time period between backups can easily cause file loss.

[0024] In view of the above problems, this method uses ShardingSphere+MinI0+Mysql to optimize the overall architecture, solve the bottleneck problem of file storage, the bottleneck of file metadata storage and the difficulty of file backup. The main advantages of the method of the present invention are as follows: (1) Distributed file storage: Use MinIO to build a cluster to store files; MinIO uses erasure coding technology to distribute and store data and verification information on multiple servers; by default, multiple copies of the data will be created, and these copies will also be stored on different servers to prevent data loss. The high availability of files is guaranteed through the distributed storage and copy mechanism of data; MinIO supports horizontal expansion, and more servers can be added according to demand to improve storage and performance, solve the bottleneck problem of file storage, and make capacity expansion very simple.

[0025] (2) File meta-information data storage capability: ShardingSphere is used in combination with MySQL database to store file meta-information data. Through flexible data sharding, high availability and horizontal expansion capabilities, the file meta-information query performance is improved and the database processing capability of massive file meta-information is simplified.

[0026] (3) MinIO snapshot backup file data: MinIO supports snapshots of the entire storage file or specific objects (files) so that the data can be restored to the state at a specific point in time when it is damaged or lost.

[0027] Based on the problems existing in file storage of traditional industry applications, the present invention is a distributed file storage method based on ShardingSphere+MinIO+Mysql: This method combines ShardingSphere sub-library and sub-table middleware, MinIO file storage service, Mysq database and intermediate application service to realize a method of distributed storage of massive files.

[0028] A distributed file storage method based on ShardingSphere+MinIO+Mysql. First, the present invention uses MinIO's erasure coding technology to distribute and store data and verification information on multiple servers. When storing file data, it is stored in an object-oriented manner. At the same time, MinIO defaults to creating multiple copies of the stored file data, and these copies will also be stored on servers that cannot be reached to prevent data loss and ensure the high availability of file storage services. Secondly, the ShardingSphere middleware and Mysql database are integrated to store file meta information. Through ShardingSphere's sub-library and sub-table technology, the file meta information is split and stored in tables of different databases, thereby realizing multiple tables of multiple databases to store file meta information and improve storage capacity. Finally, a file operation service application is developed on the basis of this architecture, providing API capabilities such as file storage, downloading, querying and deletion, and building a cluster of file operation service applications based on Ngnix to ensure high availability of the service layer. The main core of this method is to provide a distributed independent solution for massive file storage by developing applications in combination with ShardingSphere+MinIO+Mysql components.

[0029] This patent is attached Figure 1 This is the architecture diagram of the patent using ShardingSphere+MinIO+Mysql and file storage service application, which aims to explain how to build the ShardingSphere+MinIO+Mysql and file storage service architecture, how the files flow among the components when storing files, and what capabilities each component is responsible for in the overall architecture. Figure 2 This is a schematic diagram of the architecture of traditional file storage. This diagram is for comparison with the file storage of this patent, and the advantages of the patent of this invention are reflected by comparing the architecture diagram.

[0030] Comparison of the efficiency of traditional file storage and the patented file storage of this invention This table is a comparison table of the efficiency of traditional file storage methods and the file storage method using the patent of this invention. The specific data values ​​in the table are obtained by taking the average and the maximum and minimum values ​​of the data obtained by interface stress testing of the two storage solutions. A file query function in a distributed file storage method based on ShardingSphere+MinIO+Mysql is implemented by calling the API capability of the file storage service application, sending the unique identification id of the file to be queried, and the file storage service application queries the Mysql database table according to the unique identification id of the file, queries the file meta information, and then queries MinIO to obtain the file through the unique identification id of the file, and updates the query record of the file.

[0031] The file deletion function is implemented by calling the API capability of the file storage service application and sending the unique identification id of the file to be queried. The file storage service application queries the database table for file meta information based on the unique identification id of the file, and then deletes the file saved in MinIO based on the unique identification id of the file. If the file in MinIO is deleted successfully, the file meta information in the MySQL database is deleted and the file deletion record is updated; otherwise, the MySQL database does not perform any operation.

[0032] Glossary ShardingSphere: is a distributed SQL transaction and query engine that can enhance any database through data sharding, elastic scaling, encryption and other capabilities.

[0033] MinIO: is an object storage service based on the Apache License v2.0 open source agreement. It is very suitable for storing large-capacity unstructured data, such as pictures, videos, log files, backup data, and container / virtual machine images. An object file can be of any size, ranging from a few KB to a maximum of 5T.

Claims

1. A distributed file storage method, characterized in that The steps include: Step 1: Call the file upload interface provided by the file storage service, submit the file and file metadata data through the interface, and forward it to the file storage service application through the ngnix proxy; The file upload interface provided by the file storage service application is an Http Post request interface. The file and file meta-information data are submitted through the Json string and processed by the interface background. Step 2: The file storage service application communicates with the client node in the MinIO cluster, initiates a file upload request, and routes it to the nearest available storage server RAMacher. RAMacher is responsible for sharding and replicating the file, dividing the file into 10MB fragments, and each fragment is independently uploaded to different servers in the cluster; when all fragments are uploaded, RAMacher merges these fragments into a complete file and uses the file meta information in the client node, which consists of the file name, size, and type, to generate the file's unique identifier id; Before uploading files to the file storage service application and MinIO cluster, you need to build a MinIO file server cluster to establish a file storage service application, and use the springboot environment to build an application service to upload files, query, delete, and save file metadata; When building a file storage service application, you need to introduce minio's Java client to operate files in the MinIO cluster; secondly, the file storage service application integrates the ShardingSphere table and library sharding component; Step 3: Call the upload interface of the file storage service application through Http to upload the file. If the file upload fails, the file storage service application will throw an exception and return a prompt message indicating that the upload failed. Step 4: If the file is uploaded successfully, the MinIO cluster will return the file metadata including the file path; Step 5: The file storage service application receives the file storage path returned by the MinIO cluster client, generates a unique file ID for each file using the snowflake algorithm, calls the ShardingSphere component API to perform data sharding of the file meta information, uses the file meta information as the unique ID, and saves the file name, size, and file storage path to the Mysql database; Step 6: The file storage service application will eventually return the file storage result to the interface caller. If the storage is successful, the file unique identification ID will be returned. The file viewing and deletion interfaces provided by the file storage service application will be called according to the file unique identification ID to realize file viewing and deletion.

2. A distributed file storage method according to claim 1, characterized in that In step 2, the MinIO cluster will divide the file into 10MB segments and copy them to other servers.

3. A distributed file storage method according to claim 1, characterized in that In step 5, data sharding is performed by ShardingSphere according to the file unique identification id modulo the hash algorithm, and the result range is set to 0-N, where N represents the number of servers in the Mysql cluster. If the number of servers in the Mysql cluster is 2, then N=2; if the file unique identification id modulo the hash algorithm result is 0, it is stored in the database db1, and if the file unique identification id modulo the hash algorithm result is 1, it is stored in the database db2; create two tables, table1 and table2, in db1 and db2 respectively, divide the total number of file meta information ids stored in db1 or db2 by 2 and take the remainder. If the remainder is 0, it is stored in table 1, otherwise it is stored in table 2.

Images