Large File Chunk Upload and Encrypted Storage Method Based on FastDFS

Abstract

The present invention relates to a method for large file chunked uploading and encrypted storage based on FastDFS, which includes the following steps: obtaining a file selected by a user for uploading, chunking the file into certain sizes, recording the number of chunks and unique file identification information, and then making multi-threaded concurrent requests to the server; uploading the chunked files; performing AES file stream encryption on the uploaded chunked files; and performing the final FastDFS operation on the encrypted chunked files. The method for large file chunked uploading and encrypted storage based on FastDFS provided by the present invention, based on the principle of large file chunked uploading, adapts to the FastDFS file service, maximizing the error tolerance rate and uploading speed of large file uploading. At the same time, the present invention performs AES encryption storage on each chunked file. Combining the storage characteristics of FastDFS, integrating chunking, encryption, unreadable file names and random storage paths, the security of file storage is fully guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of file encryption and upload, and particularly relates to a method for large file chunked upload and encrypted storage based on fastdfs. Background Art

[0002] As an open-source lightweight distributed file system, the fastdfs file service has characteristics such as grouped storage, peer-to-peer structure, and good support for large, medium, and small files, so it is favored by many companies and enterprises. However, for the chunked upload of large files, it only supports sequential calls to its internal append interface method to perform ordered appending and merging of file chunks, solving the fault tolerance rate of large file uploads under weak network conditions. But in a high-bandwidth environment, it cannot concurrently upload object chunks to fully utilize network bandwidth and maximize the file upload speed.

[0003] To optimize the upload rate and further improve the user experience of uploading files, this patent abandons the append file splicing interface provided by fastdfs, records the unique identification information of each chunked file and the storage path information returned during the large file chunked upload process, and performs AES encryption on the byte stream of each chunked file. The front end can concurrently upload chunked files, and the server returns the number of uploaded chunks of the uniquely identified file. When all chunked files are uploaded, the server is notified to write the actual business data of the file and associate the unique identifier. When downloading the file, query the chunked files based on the unique identifier for symmetric key decryption and download, generate a binary file, and finally append and write to the response stream.

[0004] In the prior art, the front end uses the vue-simple-uploader upload component and encapsulates it. After chunking a large file, it carries the chunked file and its related information to concurrently request the server. The server uses MultipartFile to receive the chunked file, performs AES encryption on the byte stream of the chunked file, and uploads it to the fastdfs server. After fastdfs returns the upload storage path, it persistently records the information of the chunked files uploaded this time, including the unique identifier indicating that the chunks belong to the same file, the storage path, MD5, etc. information. Then it returns the information on the number of uploaded slices of the file.

[0005] The front end determines whether the entire file upload is completed based on the returned information on the number of uploaded slices. If all the chunked files of the entire file are uploaded successfully, it requests the business interface to write the main file information and associate the unique identifier of the file.

[0006] When downloading the file, query the chunked files based on the unique identifier for symmetric key decryption and download, generate a binary file, and finally append and write to the response stream.

[0007] During the entire process of chunked uploading of files, the chunked files are not merged, which saves the server's memory resources. At the same time, it maximally improves the upload speed and fault tolerance of large files, and AES encryption is performed on each chunked file. Moreover, due to the characteristics of fastdfs' volume storage and fileId naming, it combines chunking, encryption, unreadable file names, and random storage paths, fully guaranteeing the security of file storage.

[0008] However, in the existing technology, there will be a lot of structured information record data for chunked files. And after the user cancels the upload, the chunked files of the previous files are not deleted, which easily forms unstructured dirty data of sharded files in the server. Summary of the Invention

[0009] To solve the deficiencies of the existing technology, the present invention provides a method for chunked uploading and encrypted storage of large files based on fastdfs, including the following steps:

[0010] Step S1: Obtain the file selected by the user for upload, cut the file into chunks of a certain size, record the number of chunks and the unique file identification information, and then make a multi-threaded concurrent request to the server;

[0011] Step S2: Upload the chunked files;

[0012] Step S3: Perform AES file stream encryption on the uploaded chunked files;

[0013] Step S4: Perform the final fastdfs upload operation on the encrypted chunked files.

[0014] Among them, the step S2 includes the following steps:

[0015] Step S21: Construct a binary file inside the chunk entity class;

[0016] Step S22: After obtaining the binary of the uploaded file through the getBytes() function, calculate its MD5 value and compare it with the MD5 value of the binary of the file before upload passed from the front end to ensure that the file is complete.

[0017] Among them, in the step S2, during the upload process, the MD5 value is verified for each chunked file to determine whether the uploaded file exists in the entire fastdfs to avoid duplicate storage of the same file.

[0018] Among them, the step S3 includes the following steps:

[0019] Step S31: Generate a KeyGenerator object of the specified algorithm key generator, specify the secret key encryption algorithm "SHA1PRNG" and the key, and obtain the converted AES private key;

[0020] Step S32: construct an AES encryption instance of Cipher;

[0021] Step S33: construct CipherInputStream encryption stream through Cipher's AES encryption instance and file input stream;

[0022] Step S34: After writing the encrypted stream into the file, the binary byte data of the file is obtained. At this point, the file stream encryption is completed.

[0023] The fastdfs-based large file block upload and encrypted storage method provided by the present invention is based on the block upload principle of large files, adapted to the fastdfs file service, and maximizes the fault tolerance and upload rate of large file upload. At the same time, the present invention performs AES encryption storage on each block file, combines the fastdfs storage characteristics, integrates block, encryption, unreadable file name and random storage path, so that the security of file storage is fully guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of the implementation of the fastdfs-based large file block uploading and encrypted storage method. DETAILED DESCRIPTION

[0025] In order to have a further understanding of the technical solution and beneficial effects of the present invention, the technical solution and beneficial effects produced by the present invention are described in detail below with reference to the accompanying drawings.

[0026] In this invention, the terms involved are explained and agreed upon as follows:

[0027] Fastdfs: An open source lightweight distributed file system that manages files. Its functions include: file storage, file synchronization, file access (file upload, file download), etc. It solves the problems of large-capacity storage and load balancing.

[0028] Block upload: The file to be uploaded is cut into small blocks for uploading. The server then receives these small blocks uploaded by the front end and stores them in the directory where the server stores files. When the upload is complete, these small blocks are merged into one file and the uploaded blocks are deleted. At this point, the upload process ends.

[0029] Encrypted storage: For the files uploaded to the file service, during the system upload process, the file stream is read and encrypted, and then uploaded to the final file storage service.

[0030] Figure 1 This is the implementation flowchart of the large file chunked upload and encrypted storage method based on fastdfs of the present invention. The system established based on fastdfs of the present invention provides a flexible, efficient, safe and reliable large file upload strategy, and mainly realizes the following functions:

[0031] 1. Customized Uploader component encapsulated based on Vue

[0032] Obtain the file selected by the user for upload, cut the file into chunks of a certain size, record information such as the number of chunks and the unique file identifier, and then make multi-threaded concurrent requests to the server. The unique file identifier is obtained from the server before upload and is used to identify that certain chunk files belong to the same whole file. After the upload is completed, it is recorded in the chunk information table and the business file table as the association of file information. That is, through this identifier, the business file table can obtain all the chunk file information of the whole file in the chunk information table for file download.

[0033] 2. Chunk file upload

[0034] Construct a chunk entity class, and borrow the "MultipartFile" interface in the spring framework of java to receive the binary file in the request body (the placement positions of the received file and file information in the request body involve: the http request body body, the binaryBody stores the file, and the textBody stores the file-related information). Calculate its MD5 value after obtaining the file binary through the getBytes() function, and compare it with the MD5 value of the file binary before upload passed by the front end to ensure that the file is complete.

[0035] During the upload process, the MD5 value is verified for each chunk file to determine whether the uploaded file exists in the entire FastDFS to avoid duplicate storage of the same file. Specifically, it is determined whether the currently required file has been uploaded by querying the MD5 value of the files stored in the system. If it has been uploaded, the relevant file upload is no longer performed, but the business data of the file system is directly written, that is, instant upload is achieved. The theoretical basis is that when the chunk file information is stored on the server side, the MD5 of the entire chunk file is included in the chunk file information. Therefore, by matching the MD5 value of the file to be uploaded with the MD5 of the entire file uploaded previously recorded in the database, it can be known whether there is an entire file in the system with the same MD5 value as the currently required file to be uploaded. If it exists, the information of the file to be uploaded is obtained, and after obtaining the file storage path and other information, data is written, and the upload operation of the currently required file is no longer performed. At the same time, the front end is notified that the file to be uploaded has been instant uploaded, and the continuous request is stopped. After passing the business verification in other systems, the reconstructed FastDFS file upload interface is called to encrypt and upload the file chunk (this step is described in detail below).

[0036] Based on the returned uploaded chunk information, the front end compares the total number of chunks of the file uploaded during this process. When they are equal, the business data write interface is called to write the relevant information of the entire file (that is, the entire file uploaded previously recorded in the database above) into the system. At this time, the file is displayed in the system list.

[0037] 3. AES file stream encryption

[0038] The AES encryption process is one step before the actual FastDFS upload. The file is encrypted and then the final FastDFS upload operation is performed.

[0039] The encryption process is as follows: (1) First, generate a KeyGenerator object of the specified algorithm key generator, then specify the secret key encryption algorithm "SHA1PRNG" and the key (file MD5 value, supplemented if the number of bits is insufficient). This process uses a 128-bit key encryption, and finally obtains the converted AES private key; (2) Then construct the Cipher AES encryption instance. Specifically, call the initialization method to pass in the converted AES private key and encryption mode parameters to complete the Cipher instance initialization; (3) Then construct the CipherInputStream encryption stream through the Cipher AES encryption instance and the file input stream (the file to be encrypted); (4) Finally, write the encrypted stream to the file and obtain the file binary byte data. At this point, the file stream encryption is completed. Finally, call the fastdfs file upload interface and return an unreadable file name and random storage path in the format of "group1 / M00 / 04 / E3 / rB7U_WNpxrOAYeDLAAAEMKT_KlA520.txt".

[0040] 4. Download the decrypted file

[0041] To download a file from the business table, first use the unique file identifier in the business table to associate the block information table to obtain a list of all the block files of the file (the block files are arranged in order of fragment numbers, which is convenient for appending one by one after decryption), including the storage address of each block file. Then call the fastdfs file download interface to obtain the file binary. At this time, the block file binary is encrypted binary, so it is also necessary to call the system-encapsulated AES decryption for the file binary of each block file. Finally, the decrypted file binary data is written to the response stream in an appendable manner.

[0042] The fastdfs-based large file block upload and encrypted storage method provided by the present invention is based on the block upload principle of large files, adapted to the fastdfs file service, and maximizes the fault tolerance and upload rate of large file upload. At the same time, the present invention performs AES encryption storage on each block file, combines the fastdfs storage characteristics, integrates block, encryption, unreadable file name and random storage path, so that the security of file storage is fully guaranteed.

[0043] Although the present invention has been described using the above preferred embodiments, they are not intended to limit the scope of protection of the present invention. Any person skilled in the art may make various changes and modifications to the above embodiments without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A method for large file chunked uploading and encrypted storage based on fastdfs, characterized in that It includes the following steps: Step S1: Obtain the file selected and uploaded by the user, cut the file into chunks of a certain size, record the number of chunks and the unique file identification information, and then make multi-threaded concurrent requests to the server; Step S2: Upload the chunked file; Step S3: Perform AES file stream encryption on the uploaded chunked file; Step S4: Perform the final fastdfs upload operation on the encrypted chunked file; The said Step S2 includes the following steps: Step S21: Construct a binary file within the chunk entity class; Step S22: Obtain the MD5 value of the uploaded file binary through the getBytes() function and compare it with the MD5 value of the file binary before upload passed by the front end to ensure that the file is complete.

2. The large file chunk upload and encrypted storage method based on fastdfs according to claim 1, characterized in that In the said Step S2, during the upload process, the MD5 value is verified for each chunked file to determine whether the uploaded file exists in the entire fastdfs, so as to avoid duplicate storage of the same file.

3. The method for large file chunked uploading and encrypted storage based on fastdfs according to claim 1, characterized in that, The said Step S3 includes the following steps: Step S31: Generate a KeyGenerator object for the specified algorithm key generator, specify the secret key encryption algorithm "SHA1PRNG" and the secret key, and obtain the converted AES special key; Step S32: Construct an AES encryption instance of Cipher; Step S33: Construct a CipherInputStream encryption stream through the AES encryption instance of Cipher and the file input stream; Step S34: Write the encrypted stream to the file and obtain the file binary byte data, thus completing the file stream encryption.

Images