A file storage method, device, system and computer readable storage medium

Abstract

The application discloses a file storage method, device, system and computer readable storage medium, comprising: performing permission authentication on a storage account corresponding to a to-be-stored file based on preset path permission data to obtain a first authentication result; determining that the first authentication result represents successful authentication, detecting file content of the to-be-stored file to obtain a detection result; when the detection result represents that the detection is passed, storing the file content of the to-be-stored file to a lightweight distributed file system and returning a file path of the to-be-stored file; generating metadata of the to-be-stored file based on the file path of the to-be-stored file, and storing the metadata of the to-be-stored file and configuration data of the to-be-stored file to a remote dictionary service and a column-oriented storage distributed database. In this way, the lightweight distributed file system, the remote dictionary service and the column-oriented storage distributed database can improve the performance of the file storage system and realize analysis on the to-be-stored file.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and includes, but is not limited to, a file storage method, apparatus, system, and computer-readable storage medium. Background Technology

[0002] With the rapid development of cloud computing and big data, the operation of various application systems generates massive amounts of data. This data is crucial, generated by each system at various times or during important events, providing a source for subsequent big data mining. To address the needs of massive data storage and rapid data analysis, many distributed storage systems have emerged. Among them, the Hadoop Distributed File System (HDFS) and Ceph are two of the most classic distributed file storage systems.

[0003] When using HDFS and Ceph to process massive amounts of small files (such as log data, image data, etc.), a sharp drop in performance and throughput can occur. In severe cases, service may even be interrupted, the system may freeze, and normal service may be unavailable. Furthermore, Ceph does not analyze the file content when storing files, and cannot identify abnormal data, abnormal images, or abnormal videos, thus lacking file data analysis. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a file storage method, apparatus, system, and computer-readable storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a file storage method applied to a file storage system, the file storage system including: a lightweight distributed file system, a remote dictionary service, and a columnar storage-oriented distributed database, the method including:

[0007] Obtain the file to be stored and the preset path permission data in the column-oriented distributed database;

[0008] Based on the preset path permission data, the storage account corresponding to the file to be stored is authenticated to obtain a first authentication result;

[0009] Once the first authentication result indicates successful authentication, the file content of the file to be stored is detected to obtain the detection result.

[0010] When the detection result indicates that the detection is passed, the file content of the file to be stored is stored in the lightweight distributed file system, and the file path of the file to be stored is returned;

[0011] The metadata of the file to be stored is generated based on the file path of the file to be stored, and the metadata and configuration data of the file to be stored are stored in the remote dictionary service and the column-oriented distributed database.

[0012] This application provides a file storage device, the file storage device comprising:

[0013] The acquisition module is used to acquire the file to be stored and the preset path permission data in the column-oriented distributed database;

[0014] The authentication module is used to perform permission authentication on the storage account corresponding to the file to be stored based on the preset path permission data, and obtain a first authentication result;

[0015] The detection module is used to determine that the first authentication result indicates successful authentication, and to detect the file content of the file to be stored to obtain the detection result;

[0016] The first storage module is used to store the file content of the file to be stored in the lightweight distributed file system and return the file path of the file to be stored when the detection result indicates that the detection is passed.

[0017] The second storage module is used to generate metadata of the file to be stored based on the file path of the file to be stored, and to store the metadata and configuration data of the file to be stored in the remote dictionary service and the column-oriented distributed database.

[0018] This application provides a file storage system, which includes a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database. The file storage system stores target files, which are files whose access count exceeds a threshold.

[0019] The lightweight distributed file system is used to store the file content of all stored files;

[0020] The remote dictionary service is used to store the basic data of the target file, wherein the basic data includes metadata and configuration data;

[0021] The column-oriented distributed database is used to store the basic data of all stored files.

[0022] This application provides a computer-readable storage medium storing computer-executable instructions configured to execute the above-described file storage method.

[0023] This application provides a file storage method, apparatus, system, and computer-readable storage medium. The file storage method is applied to a file storage system, which includes a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database. Based on this, the file storage method includes: first, obtaining the file to be stored and preset path permission data from the column-oriented distributed database, wherein the preset path permission data is pre-set to specify storage permissions for each path; then, performing permission authentication on the storage account corresponding to the file to be stored based on the preset path permission data to determine whether the storage account has storage permissions for the file to be stored, thereby obtaining a first authentication result, and in the first... If the authentication result indicates successful authentication, meaning the storage account has the necessary storage permissions for the file to be stored, the file content will be further inspected to check for any abnormal content. If the inspection result indicates the file passes inspection (i.e., it does not contain abnormal content), the file content will be stored in a lightweight distributed file system, and the file path will be returned. Finally, metadata for the file will be generated based on its file path, and this metadata, along with the configuration data, will be stored in a remote dictionary service and a column-oriented distributed database. This eliminates the need to store the metadata in the lightweight distributed file system, saving memory and improving the overall performance of the file storage system. Furthermore, the authentication and inspection processes during storage allow for the timely detection of abnormal storage accounts or files, enabling analysis of the file and ensuring the secure and stable operation of the file storage system. Attached Figure Description

[0024] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. The drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0025] Figure 1 This is a schematic diagram illustrating an implementation process of the file storage method provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram illustrating an implementation process for storing the file content of a file to be stored, provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating an implementation process of the method for deleting expired files provided in an embodiment of this application;

[0028] Figure 4A schematic diagram illustrating an implementation flow of the method for determining target files provided in this application embodiment;

[0029] Figure 5 This is a schematic diagram illustrating an implementation flow of the file download method provided in an embodiment of this application;

[0030] Figure 6 This application provides a schematic diagram of an implementation process for a method of returning a file to be downloaded to the client, as illustrated in an embodiment of the present application.

[0031] Figure 7 This is a schematic diagram illustrating an implementation flow of the file upload method provided in an embodiment of this application;

[0032] Figure 8 A schematic diagram illustrating another implementation flow of the file download method provided in this application embodiment;

[0033] Figure 9 A schematic diagram illustrating an implementation flow of the multi-level caching method provided in this application embodiment;

[0034] Figure 10 This is a schematic diagram of the composition structure of the file storage device provided in the embodiments of this application;

[0035] Figure 11 This is a schematic diagram of the composition structure of a file storage system provided in an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0040] To address the problems existing in related technologies, this application provides a file storage method. This method can be implemented by a computer program, which, when executed, performs the file storage method provided in this application. In some embodiments, the computer program can be executed on a processor in a file storage system. Figure 1 This application provides an implementation flow of a file storage method, which is applied to a file storage system. The file storage system includes a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database. Based on this, as... Figure 1 As shown, the file storage methods include:

[0041] Step S101: Obtain the file to be stored and the preset path permission data in the column-oriented distributed database.

[0042] Here, the file to be stored can be running data, images, etc. In this embodiment of the application, the file to be stored can also refer to a small file, wherein a small file refers to a file that occupies less than 2 megabytes of storage space.

[0043] In this embodiment, preset path permission data is pre-stored in the column-oriented distributed database (HBase). This preset path permission data specifies the storage permissions for each path. For example, for the "Customer Project 1" folder under disk D, employees of the company's project department have permission to store files under this path; and for the "Employee Management" folder under disk D, employees of the company's general affairs department have permission to store files under this path.

[0044] Step S102: Based on the preset path permission data, perform permission authentication on the storage account corresponding to the file to be stored, and obtain the first authentication result.

[0045] Here, while obtaining the file to be stored, the system also obtains the storage account corresponding to the file. Based on this, the file storage system will use preset path permission data to authenticate the storage account, thereby determining whether the storage account has the permission to store the file to be stored.

[0046] Continuing with the example above, suppose Zhang San stores a file to be stored in the "Customer Project 1" folder on disk D. The file storage system will then perform permission authentication on Zhang San. If Zhang San is an employee of the company's project department, he will receive the first authentication result indicating successful authentication; if Zhang San is an employee of the company's general affairs department, he will receive the first authentication result indicating failed authentication.

[0047] Step S103: Determine that the first authentication result indicates successful authentication, and then inspect the file content of the file to be stored to obtain the inspection result.

[0048] In this embodiment, if a first authentication result indicating successful authentication is obtained, the file content of the file to be stored will continue to be detected to obtain a detection result. The purpose of detection is to promptly identify abnormal content, thereby avoiding the storage of abnormal content or the continued transmission of abnormal content, thus ensuring the safe and efficient operation of the file storage system.

[0049] In some embodiments, if the first authentication result of the characterization authentication failure is obtained, the characterization storage account does not have permission to store the file to be stored, and the storage process will be terminated directly without continuing to detect and process the file content of the file to be stored.

[0050] Step S104: When the detection result indicates that the detection is passed, the file content of the file to be stored is stored in the lightweight distributed file system, and the file path of the file to be stored is returned.

[0051] Here, if the detection result indicates that the detection is successful and the content of the file to be stored does not contain any abnormal content, then the content of the file to be stored will be stored in a lightweight distributed file system, and the storage path of the file to be stored will also be returned, which is also called the file path.

[0052] In some embodiments, if the detection result indicates that the detection fails, it means that the file content of the file to be stored contains abnormal content. To ensure the security of the file storage system, the storage process will be terminated directly, and the file to be stored will not be stored.

[0053] Step S105: Generate metadata of the file to be stored based on the file path of the file to be stored, and store the metadata and configuration data of the file to be stored in a remote dictionary service and a column-oriented distributed database.

[0054] In this embodiment of the application, the filename of the file to be stored and the file path of the file to be stored are associated or mapped to generate the metadata of the file to be stored.

[0055] Here, the configuration data includes file expiration information, file encryption information, file size information, etc. In this embodiment, the configuration data can be used to analyze the files to be stored.

[0056] Based on this, the metadata and configuration data of the files to be stored are stored in a remote dictionary service, and simultaneously stored in a column-oriented distributed database. This enhances data security. During actual data access, data is typically accessed from the remote dictionary service; if the data in the remote dictionary service is corrupted, the uncorrupted data can be retrieved from the column-oriented distributed database.

[0057] In some embodiments, the remote dictionary service stores basic data of frequently accessed files, while the column-oriented distributed database stores basic data of all files. When storing a file to be stored, the metadata and configuration data of the file to be stored can be stored simultaneously in both the remote dictionary service and the column-oriented distributed database; alternatively, only the metadata and configuration data of the file to be stored can be stored in the column-oriented distributed database; or the storage location of the metadata and configuration data of the file to be stored can be determined based on the storage tag information carried by the file.

[0058] This application provides a file storage method applied to a file storage system, which includes a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database. Based on this, the file storage method includes: first, obtaining the file to be stored and preset path permission data from the column-oriented distributed database, wherein the preset path permission data is pre-set to specify storage permissions for each path; then, performing permission authentication on the storage account corresponding to the file to be stored based on the preset path permission data to determine whether the storage account has storage permissions for the file to be stored, thereby obtaining a first authentication result, and the first authentication result characterizing authentication. If successful (i.e., the storage account has the necessary storage permissions for the file to be stored), the file content will be further inspected to check for any abnormal content. If the inspection result indicates that the file passes inspection and does not contain abnormal content, the file content will be stored in a lightweight distributed file system, and the file path will be returned. Finally, metadata for the file will be generated based on its file path, and this metadata, along with the configuration data, will be stored in a remote dictionary service and a column-oriented distributed database. This eliminates the need to store the metadata of the file through the lightweight distributed file system, saving memory and improving the overall performance of the file storage system. Furthermore, during file storage, permission authentication and inspection operations are performed to promptly identify abnormal storage accounts or files, thus achieving the goal of analyzing the file and ensuring the secure and stable operation of the file storage system.

[0059] In some embodiments, to enhance the security of the file to be stored, the file will be encrypted. Based on this, such as Figure 2 As shown, the step S104 above, "storing the file content of the file to be stored to a lightweight distributed file system", can be achieved through the following steps S1041 to S1046:

[0060] Step S1041: Obtain the configuration data of the file to be stored.

[0061] Here, the configuration data of the file to be stored can be obtained through location information or identification information. The configuration file includes the file encryption information of the file to be stored.

[0062] Step S1042: Based on the configuration data of the file to be stored, perform encryption detection on the file to be stored to obtain the first encryption detection result.

[0063] Here, the configuration data of the file to be stored is parsed, that is, the file encryption information of the file to be stored is parsed, to perform encryption detection on the file to be stored, and obtain the first encryption detection result, which can reflect the encryption status of the file to be detected.

[0064] For example, if the file encryption information of the file to be stored is empty, a first encryption detection result indicating that the file to be stored does not need to be encrypted is obtained; and if the file encryption information of the file to be stored is not empty, a first encryption detection result indicating that the file to be stored needs to be encrypted is obtained.

[0065] Step S1043: Determine whether the first encryption result indicates that the file to be stored needs to be encrypted.

[0066] Here, based on the first encryption result, it is determined whether the file to be stored needs to be encrypted. If the first encryption result indicates that the file to be stored needs to be encrypted, then proceed to step S1044; otherwise, if the first encryption result indicates that the file to be stored does not need to be encrypted, then proceed to step S1046.

[0067] Step S1044: Encrypt the file to be stored to obtain the encrypted file.

[0068] Here, the first encryption result indicates that the file to be stored needs to be encrypted, so the file to be stored is encrypted.

[0069] In this embodiment of the application, the first encryption detection result also includes a corresponding encryption method. Based on this, the corresponding encryption method is used to encrypt the file to be stored, thereby obtaining the encrypted file to be stored.

[0070] Step S1045: Store the encrypted file to be stored in a lightweight distributed file system.

[0071] Here, the encrypted file content is stored in a lightweight distributed file system.

[0072] Step S1046: Store the file to be stored to a lightweight distributed file system.

[0073] Here, the file content of the file to be stored is stored in a lightweight distributed file system.

[0074] In this embodiment, through steps S1041 to S1046, when storing the file content of the file to be stored to the lightweight distributed file system, the configuration data of the file to be stored is first obtained, which includes the file encryption information of the file to be stored; then, based on the configuration data of the file to be stored, the file to be stored is encrypted to obtain a first encryption detection result; when the first encryption detection result indicates that the file to be stored needs to be encrypted, the file to be stored is encrypted to obtain the encrypted file to be stored; finally, the file content of the encrypted file to be stored is stored to the lightweight distributed file system, thereby improving the security of the file to be stored.

[0075] In some embodiments, to improve the operating efficiency of the file storage system, expired files in the file storage system are also periodically deleted. Based on this, such as Figure 3 As shown, after step S105, "generating metadata of the file to be stored based on the file path of the file to be stored, and storing the metadata and configuration data of the file to be stored in a remote dictionary service and a column-oriented distributed database", the following steps S106 to S109 can also be performed:

[0076] Step S106: Obtain the current time information, the first expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the columnar storage-oriented distributed database.

[0077] Here, the file storage system can obtain current time information online based on the communication link. In this embodiment, the basic data in the remote dictionary service and the basic data in the column-oriented distributed database are also parsed to obtain the first expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the column-oriented distributed database can also be obtained. The basic data includes configuration data, and the configuration data includes file expiration information.

[0078] In this embodiment of the application, the first expiration time of the basic data in the remote dictionary service is used to characterize the expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the columnar storage-oriented distributed database is used to characterize the expiration time of the basic data in the columnar storage-oriented distributed database.

[0079] Step S107: Based on the current time information, the first expiration time, and the second expiration time, determine the first expired file in the remote dictionary service and the second expired file in the columnar storage-oriented distributed database.

[0080] Here, a first comparison result can be obtained by comparing the current time information and the first expiration time. If the first comparison result indicates that the first expiration time is before the current time, then the basic data corresponding to the first expiration time is determined as the first expired file.

[0081] Similarly, a second comparison result can be obtained by comparing the current time information and the second expiration time. If the second comparison result indicates that the second expiration time is before the current time, then the basic data corresponding to the second expiration time is determined as the second expired file.

[0082] Step S108: Based on the first and second expired files, determine the corresponding third expired file in the lightweight distributed file system.

[0083] Here, since the remote dictionary service and the column-oriented distributed database store the basic data of the files, while the lightweight distributed file system stores the file content, the file content corresponding to the first and second expired files will also be determined from the lightweight distributed file system, which will be referred to as the third expired file.

[0084] Step S109: Delete the first expired file, the second expired file, and the third expired file.

[0085] Here, you can use the delete command to delete the first expired file, the second expired file, and the third expired file.

[0086] In this embodiment, through steps S106 to S109 above, the current time information, the first expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the columnar storage-oriented distributed database are first obtained. Then, the first expired file in the remote dictionary service and the second expired file in the columnar storage-oriented distributed database are determined. Based on the first and second expired files, the corresponding third expired file in the lightweight distributed file system is also determined. Finally, the first, second, and third expired files are deleted. In this way, expired files are cleaned up in a timely manner, improving access efficiency and the running speed of the file storage system.

[0087] When actually storing files using a file storage system, the underlying data stored in the remote dictionary service is contained within the underlying data stored in the column-oriented distributed database. The underlying data stored in the remote dictionary service is the underlying data of the target file, also known as the hot file; while the underlying data stored in the column-oriented distributed database is the underlying data of all stored files. Based on this, as... Figure 4As shown, after step S105 above, "generating metadata of the file to be stored based on the file path of the file to be stored, and storing the metadata and configuration data of the file to be stored in a remote dictionary service and a column-oriented distributed database", the following steps S106' to S108' can also be performed:

[0088] Step S106': Obtain the access count of each file in the lightweight distributed file system.

[0089] Here, the initial value of the access count for each file is 0. Each time a file is accessed, the access count for that file is incremented by 1.

[0090] Step S107': Based on the number of accesses and the access threshold, determine the target file from each file.

[0091] Here, the number of times threshold can be a default value or a custom setting. For example, the number of times threshold can be 10, 15, 20, etc.

[0092] In this embodiment, the relationship between each access count and a threshold value can be compared sequentially. If a target access count exists that exceeds the threshold value, the file corresponding to the target access count is identified as the target file. The target file's access count exceeding the threshold value indicates that it is a frequently accessed file.

[0093] In some embodiments, if all access counts are less than the access threshold, that is, there is no target access count higher than the access threshold, it indicates that there is no target file in any of the files.

[0094] Step S108': Store the basic data of the target file in a remote dictionary service.

[0095] Here, the basic data of the target file is stored in a remote dictionary service to facilitate the reading or manipulation of the target file's basic data when accessing the target file. The efficient caching technology of the remote dictionary service is used to improve access efficiency.

[0096] In some embodiments, the lightweight distributed file system also supports merging multiple files to be stored to save storage space. Therefore, before step S101 "obtaining the files to be stored and the preset path permission data in the column-oriented distributed database", the following steps S001 and S002 can also be performed:

[0097] Step S001: Obtain multiple files to be stored and determine the total storage space occupied by the multiple files to be stored.

[0098] Here, we first determine the storage space corresponding to each file to be stored, and then add up the storage spaces to get the total storage space.

[0099] Step S002: Determine that the total storage space is less than the storage space threshold, merge multiple files to be stored, and obtain the merged file to be stored.

[0100] Here, the storage space threshold can be a default value or a custom value. For example, the storage space threshold can be 5 MB, 6 MB, 7 MB, etc.

[0101] In this embodiment, the relationship between the total storage space and the storage space threshold is first determined. If the total storage space is less than the storage space threshold, multiple files to be stored are merged in sequence; or the multiple files to be stored are sorted according to their category, format and other attribute information, and then merged in the sorted order.

[0102] In this embodiment of the application, the merged file to be stored includes location indication information, which is used to indicate the location of different files to be stored.

[0103] In some embodiments, if the total storage space is greater than the storage space threshold, indicating that the total storage space occupied by the multiple files to be stored is large, then the multiple files to be stored are not merged; or, the number of multiple files to be stored is reduced until the total storage space occupied by the reduced files to be stored is less than the storage space threshold, and then the reduced files to be stored are merged.

[0104] In this embodiment of the application, through the above steps S001 and S002, when the total storage space occupied by multiple files to be stored is less than the storage space threshold, the multiple files to be stored are merged, thereby achieving the purpose of saving storage space, thus enabling the storage of more files to be stored and improving the performance of the file storage system.

[0105] In this embodiment of the application, the file to be downloaded can also be downloaded from the file storage system. Therefore, based on the above-mentioned file storage system, such as Figure 5 As shown, when downloading the file to be downloaded, steps S501 to S504 can also be performed, in some embodiments. Steps S501 to S504 can also be performed after step S105 above.

[0106] Step S501: In response to the download request sent by the client, obtain the basic data of the file to be downloaded corresponding to the download request.

[0107] Here, a download request is used to request the download of a file to be downloaded, where the underlying data of the file to be downloaded is stored in a file storage system.

[0108] Based on the above embodiments, after receiving a download request from the client, the basic data of the file to be downloaded is obtained from a remote dictionary service or a column-oriented distributed database. In actual implementation, the basic data of the file to be downloaded is first obtained from the remote dictionary service; if it cannot be obtained, it will be obtained from the column-oriented distributed database.

[0109] Step S502: Based on the basic data of the file to be downloaded, perform permission authentication on the download account corresponding to the download request to obtain the second authentication result.

[0110] Here, the basic data also includes path permission information, which is used to restrict the account that can download the file to be downloaded.

[0111] In this embodiment of the application, the download account corresponding to the download request will also be obtained, and the download account will be authenticated using the path permission information of the file to be downloaded, so as to obtain a second authentication result indicating whether the download account has download permission.

[0112] Step S503: Determine that the second authentication result indicates successful authentication, and use the basic data of the file to be downloaded to perform an expiration check on the file to be downloaded, and obtain the expiration check result.

[0113] Here, if the second authentication result indicates that the downloading account has download permissions, then the second authentication result indicates successful authentication. Based on this, the file to be downloaded will also be checked for expiration. That is, based on the file expiration information in the basic data, the file to be downloaded will be checked for expiration and the expiration result will be obtained.

[0114] In some embodiments, if the second authentication result indicates that the download account does not have download permissions, then the second authentication result indicates that the authentication has failed, and the process ends directly without checking the expiration date of the file to be downloaded.

[0115] Step S504: Determine the expiration date detection result to indicate that the file to be downloaded has not expired, and return the file to be downloaded to the client.

[0116] In actual implementation, such as Figure 6 As shown, "returning the file to be downloaded to the client" can be achieved through the following steps S5041 to S5044:

[0117] Step S5041: Based on the basic data of the file to be downloaded, perform encryption detection on the file to be downloaded to obtain the second encryption detection result.

[0118] Here, encryption detection can be performed based on the file encryption information in the basic data to obtain a second encryption detection result that characterizes whether the file to be downloaded is encrypted.

[0119] For example, if the file encryption information is not empty, a second encryption detection result indicating that the file to be downloaded is encrypted is obtained; and if the file encryption information is empty, a second encryption detection result indicating that the file to be downloaded is not encrypted is obtained.

[0120] Step S5042: Determine that the second encryption detection result indicates that the file to be downloaded is an encrypted file, and obtain the key sent by the client.

[0121] Here, if the second encryption detection result indicates that the file to be downloaded is an encrypted file, the download request will also be parsed to obtain the key.

[0122] In some embodiments, if the second encryption detection result indicates that the file to be downloaded is an unencrypted file, the file to be downloaded is directly returned to the client without decryption processing.

[0123] Step S5043: Use the key to decrypt the file to be downloaded to obtain the decrypted file.

[0124] Here, the encrypted file to be downloaded is decrypted using a key, thus obtaining the decrypted file to be downloaded, which is also known as the decrypted file.

[0125] Step S5044: Return the decrypted file to the client.

[0126] In some embodiments, to handle high-concurrency download requests, the file storage system employs multi-level caching to reduce the pressure on the underlying file system. Therefore, a cache machine can be set up between the client and the file storage system. This cache machine caches the file content from the lightweight distributed file system. It interacts directly with the client; that is, the client retrieves the file content to be downloaded from the cache machine. If the file content to be downloaded is not present in the cache machine, it will retrieve it from the lightweight distributed file system by receiving instructions.

[0127] In some embodiments, if the expiration detection result indicates that the file to be downloaded has expired, the process can be terminated directly, and a download failure result can be returned to the client.

[0128] In this embodiment, through steps S501 to S504, after receiving a download request, the basic data of the file to be downloaded corresponding to the download request is obtained. This basic data is used to authenticate the download account corresponding to the download request. If the authentication is successful, the download file's expiration date is checked. Furthermore, if the download file has not expired, encryption is checked, and the decrypted file is sent to the client for use. This improves the analysis of the download file during download.

[0129] Based on the above embodiments, this application further provides a file storage method. This file storage method is applied to a file storage system. Taking the storage of a massive number of small files in a file storage system as an example, this application uses a lightweight distributed file system (FastDFS) to store file data. A self-developed client is provided for uploading various files. The self-developed client refers to a client that can match the file processing method in this application embodiment. FastDFS lacks metadata management, which can cause problems. Compared to the distributed file system (Ceph), it has fewer functions. Metadata for small files (such as log data, image data, etc.) can be stored using a remote dictionary server (Redis) and a column-oriented distributed database (HBase). Redis's high read / write performance is used to store the metadata of frequently accessed files, while HBase's high performance for massive data read / write operations is used to store the metadata of smaller files.

[0130] The file storage system provided in this application embodiment has the following functions:

[0131] Function 1: Users can store files into the FastDFS cluster and metadata into Redis and HBase using a self-developed client.

[0132] Function 2: Even if any server or disk in the FastDFS cluster fails, the data can still be accessed because the data is backed up.

[0133] Function 3: Frequently accessed files are cached through Nginx to meet the needs of high-concurrency file download requests. Nginx is equivalent to the cache server in the above embodiment.

[0134] Feature 4 provides multiple advanced features to meet various user needs.

[0135] In some embodiments, the file processing method and file storage system provided in this application can solve the following four problems:

[0136] Problem 1: Solving the problem of excessive metadata management when storing a large number of small files.

[0137] FastDFS does not store metadata when storing massive amounts of small files. Instead, it stores metadata information in encoded form within the filename, resulting in extremely high performance for storing large numbers of small files. For users who need to retain metadata information, Redis's efficient caching technology and HBase's massive columnar data storage capabilities perfectly solve the problem of excessive system memory consumption due to excessive metadata.

[0138] Problem 2: Solve the problem of random allocation of data blocks in files, which leads to a large amount of disk fragmentation and low performance when reading a large number of files.

[0139] FastDFS supports small file merging operations. A file occupies one space when stored in a distributed file system, but the operating system always has a certain limit on the number of files. If small file merging is not supported, a situation will occur where there is still plenty of disk space, but due to the storage of a large number of small files, no more data can be stored. FastDFS perfectly solves this problem.

[0140] Question 3: Enrich the storage system with some advanced features that are lacking, such as overwriting original files, timed file deletion, file encryption and decryption, and AI-powered identification (image recognition, file content recognition) for uploading and downloading prohibited files.

[0141] Thus, the value of these advanced features is reflected in:

[0142] First, the function of overwriting the original file successfully solved the problem of the system being unable to detect the updated file in a timely manner during file upgrades.

[0143] Secondly, the scheduled file deletion function reduces the pressure on the storage system and successfully cleans up junk files in the storage system.

[0144] Third, the storage file encryption and decryption functions effectively protect important files. Even if a user steals files from the file system, they will not be able to steal the information within those files.

[0145] Fourth, artificial intelligence identifies prohibited file uploads and downloads, reducing the upload of prohibited files and the download of prohibited files from external networks.

[0146] Question 4: Provide a multi-level caching mechanism to handle high-concurrency file download requests. Without caching, all requests to the FastDFS file system during high-concurrency file system requests would result in a surge in disk I / O and a degraded file system performance. Multi-level caching reduces the pressure on the underlying file system and also meets the needs of high-concurrency scenarios.

[0147] In this embodiment of the application, the file storage system architecture enables file upload and download operations.

[0148] Figure 7 This is a schematic diagram of the file upload process provided in an embodiment of this application, such as... Figure 7 As shown, the file upload process includes the following steps S701 to S713:

[0149] Step S701, Begin.

[0150] Here, customers can use a self-developed client to store data into the storage system and initiate the file upload process.

[0151] Step S702: The data arrives at Nginx.

[0152] In this embodiment of the application, after the data reaches Nginx, Nginx will then call a corresponding self-developed execution program, which can be written in Lua or Go language.

[0153] Step S703: Run the self-developed execution program.

[0154] Step S704: Obtain basic data (including metadata and advanced configuration data).

[0155] Here, the self-developed program will retrieve basic data from Redis or HBase, which includes metadata and advanced configuration data.

[0156] Step S705: Check if you have permission to upload files to the specified path.

[0157] Here, the upload path information and key provided by the user are compared with the corresponding data in the path permission table in the basic data for permission authentication. If the authentication result indicates that the user has permission to upload files under the specified path, the process proceeds to step S706; otherwise, if the authentication result indicates that the user does not have permission to upload files under the specified path, the process ends, i.e., the process proceeds to step S712.

[0158] Step S706: Use artificial intelligence methods to check whether the file meets the requirements.

[0159] Here, the self-developed execution program uses algorithms such as identification and detection to retrieve the file content to determine whether the file meets the requirements. These requirements mainly include compliance requirements, that is, detecting whether there are abnormal files.

[0160] Taking a file as an image as an example, if the image meets the requirements, that is, if the image contains compliant content, then proceed to step S707; otherwise, end the process, that is, proceed to step S712.

[0161] Step S707: Check whether the file needs to be encrypted.

[0162] Here, the self-developed execution program checks whether the file requires encryption. If encryption is required, the uploaded file is encrypted, and the process proceeds to step S708; otherwise, there is no need to encrypt the uploaded file, and the process proceeds to step S709.

[0163] Step S708, encryption processing.

[0164] Here, the file is encrypted to obtain the encrypted file.

[0165] Step S709: Upload the file to FastDFS.

[0166] Here, the self-developed executable calls the FastDFS interface to upload the file to FastDFS, and FastDFS returns the stored FileID.

[0167] In some embodiments, if file upload fails, the process ends directly.

[0168] Step S710: Determine the basic data of the file.

[0169] Here, the file's metadata can be determined based on the FileID, and then the file's metadata and the file's advanced configuration data can be used as the file's basic data.

[0170] Step S711: Store the basic data of the file in Redis and HBase.

[0171] Step S712, End.

[0172] Here, the process ends if there is no permission to upload a file to the specified path or if the file does not meet the requirements.

[0173] In some embodiments, the self-developed executable program will also determine whether the Redis and HBase data have been written successfully. If the write is successful, it will return a successful upload result to the client; if the write fails, it will return a failed upload result (or an error result) to the client.

[0174] In some embodiments, during file uploads, to alleviate storage pressure and improve processing speed, refer to Figure 7 Following step S712 above, expired files will be periodically identified and deleted, which is step S713, "Execute Scheduled Task". During the execution of the scheduled task, expired files will be retrieved from Redis and HBase and then deleted.

[0175] Figure 8This is a schematic diagram of the file download process provided in the embodiments of this application, such as... Figure 8 As shown, the file download process includes the following steps S801 to S817:

[0176] Step S801, Begin.

[0177] Here, the client initiates the file download by requesting the Uniform Resource Locator (URL) returned by the uploaded file.

[0178] Step S802: The requested data arrives at Nginx.

[0179] Here, the requested data first reaches Nginx; then, Nginx calls a custom executable program, which can be written in Lua or Go.

[0180] Step S803: Run the self-developed execution program.

[0181] Step S804: Obtain basic data from Redis.

[0182] Here, the self-developed executable program will be used to obtain basic data from Redis.

[0183] Step S805: Determine whether the basic data has been obtained from Redis.

[0184] If the basic data cannot be obtained from Redis, proceed to step S806; if the basic data can be obtained from Redis, proceed to step S807.

[0185] Step S806: Obtain the basic data from HBase.

[0186] Step S807: Determine the basic data of the corresponding file.

[0187] Here, the corresponding file refers to the file to be downloaded, which is the basic data obtained from Redis or HBase and is determined as the basic data of the corresponding file.

[0188] Step S808: Check whether the file has permission to be downloaded using the advanced configuration data in the basic data.

[0189] Here, the advanced configuration data is part of the basic data. This advanced configuration data can be used to detect whether the current user has permission to download files. If the user has permission to download, proceed to step S809; otherwise, the process ends and proceeds to step S817.

[0190] Step S809: Check if the file has expired using advanced configuration data.

[0191] Here, the self-developed execution program checks whether the file has expired by checking the corresponding advanced configuration data in the basic data. If it has not expired, it proceeds to step S810; if it has expired, the process ends and proceeds to step S817. It can also return an error result to the customer.

[0192] Step S810: Obtain the corresponding FastDFS FileId through the metadata information in the basic data.

[0193] Step S811: Detect whether a corresponding cache file exists in the Nginx cache files based on the FileId.

[0194] Here, the self-developed executable checks whether the file corresponding to the FileId is cached in the corresponding cache folder of the Nginx machine. If the corresponding cached file does not exist, it proceeds to step S812, which downloads the file from FastDFS to the corresponding cache folder of Nginx. If the corresponding cached file exists, it proceeds to step S814.

[0195] Step S812: Download the file corresponding to FileId from FastDFS.

[0196] Step S813: Store the file corresponding to FileId into the corresponding cache file of nginx.

[0197] Step S814: Obtain the file corresponding to the FileId from the corresponding Nginx cache file.

[0198] Step S815: Check whether the file is encrypted using the corresponding advanced configuration data in the basic data.

[0199] Here, the advanced configuration data includes encryption-related information. Therefore, the file can be detected by the advanced configuration data to determine whether it has been encrypted. If the file is encrypted, proceed to step S816; otherwise, proceed to step S817.

[0200] Step S816: Obtain the upload key and decrypt the file.

[0201] Here, the key uploaded by the user is first detected, and then the file is decrypted using the key to obtain the decrypted file. Finally, the decrypted file is also sent to the user.

[0202] Step S817, End.

[0203] Here, if the user lacks download permissions or the file has expired, the process ends directly. Additionally, if the file is not encrypted, the unencrypted file can be directly sent to the user, ending the process.

[0204] In this embodiment, FastDFS is an open-source, lightweight distributed file system that manages files, including file storage, file synchronization, and file access (file upload and download), solving the problems of large-capacity storage and load balancing. Redis, or Remote Dictionary Service, is an open-source, ANSI C-written, network-enabled, in-memory or persistent log-structured key-value database that provides application programming interfaces (APIs) in multiple languages. Nginx (also known as engine x) is a high-performance Hypertext Transfer Protocol (HTTP) and reverse proxy World Wide Web (Web) server, also providing IMAP / POP3 / SMTP services.

[0205] In some embodiments, the underlying data stored in Redis and HBase includes a metadata table, as shown in Table 1, in which each path filename corresponds to a FileId.

[0206] Table 1 Metadata Table

[0207]

[0208] In Redis, the pathname is used as the key in Redis key-value storage. In HBase, a table is created to store the project, with 'pro' as the column family, 'base' as the column, and 'FileId' as the cell in the 'base' column.

[0209] When a user accesses pro_base_ / apps / test / test.txt, the FileId is obtained through metadata conversion, and then the actual file data is downloaded using Nginx or FastDFS.

[0210] Table 2 is a path permission table provided in an embodiment of this application. Referring to Table 2, the path name is the path specified when uploading the file, and it also includes the corresponding key, department and responsible person for transmission.

[0211] Table 2 Path Permission Table

[0212]

[0213] In this embodiment, the pathname is used as the key in Redis key-value storage. A storage table is created in HBase, with "pro" as the column family, "auth" as the column, and the authentication token as a cell in the "auth" column.

[0214] Table 3 is an advanced configuration data table provided in an embodiment of this application. Referring to Table 3, the advanced configuration data table includes information such as path file name, expiration time, encryption, and memory size.

[0215] Table 3 Advanced Configuration Data Table

[0216]

[0217] In Redis, the pathname is used as the key in Redis key-value storage. In HBase, a table is created to store the project, with "pro" as the column family, "adv" as the column, and advanced configuration as the cells in the "adv" column.

[0218] In some embodiments, advanced configuration data is not stored together with metadata because advanced configuration data can facilitate later troubleshooting and problem-solving.

[0219] In this embodiment, Redis and HBase also store expired data tables. Each time data is stored, the expiration time is obtained, and the data is converted to a specific day of expiration. Taking Redis as an example, when data is stored in the advanced configuration, it is also stored in a specific expired list collection in Redis. For instance, if the data expires on August 13, 2021, it will be stored in the Redis list collection corresponding to `pro_delete_2021-8-13`. A scheduled task searches the specified collection daily and deletes expired files, thus clearing junk files.

[0220] By storing basic data in the above manner, the metadata satisfies the user's need to retain the name of uploaded files, the path permission table effectively manages user-uploaded files and allows for the identification of responsible parties when files are corrupted, while the advanced configuration data table and expired data table effectively provide advanced functions and provide a data foundation for future data analysis and billing.

[0221] In this embodiment, a multi-level caching technique is also employed. This technique is used in scenarios with massive concurrent access. Without multi-level caching, all data requests would go to FastDFS, causing FastDFS to fetch a large number of files from the disk, leading to increased machine I / O and request timeouts that impact user requests. Therefore, multi-level caching is needed to reduce FastDFS requests.

[0222] First-level caching refers to using Nginx's built-in caching mechanism to temporarily cache the results of the same access requests. Since Nginx caching also consumes machine memory, if the caching time is too long, a large number of different concurrent accesses will consume too much machine memory.

[0223] Because the first-level cache has a short retention period, without a second-level cache, a large number of requests will still reach FastDFS. Therefore, it is necessary to design a second-level cache to handle frequently accessed files.

[0224] In practice, the principle of the second-level cache is to use FastDFS file storage to create an identical copy in the nginx-related folders.

[0225] Taking FastDFS file download as an example: A file with FileId: / group1 / M00 / B3 / 62 / Cii-s1rLKaWIbiB7AFFxKmeasnYAAFXDAATYUkAUXFC667.txt, the download steps include steps 1 to 4:

[0226] Step 1: Obtain the address information of the storage machine in group1 from the Tracker.

[0227] Step 2: Access the corresponding storage directory on the storage machine.

[0228] Step 3: Go to the / M00 / B3 / 62 / directory again.

[0229] Step 4: Download the file Cii-s1rLKaWIbiB7AFFxKmeasnYAAFXDAATYUkAUXF-C667.txt.

[0230] Through steps 1 to 4 above, FastDFS data storage is very simple. The corresponding directory can be easily found by FileId, and the file can be downloaded. Since this is the case, a file system can also be simulated on the Nginx machine, and the same request can be retrieved from the Nginx file system.

[0231] Taking a second-level cache created with Nginx as an example: A file with FileId: / group1 / M00 / B3 / 62 / Cii-s1rLKaWIbiB7AFFxKmeasnYAAFXDAATYUkAUXFC667.txt, the download steps include steps 1 to 3:

[0232] Step 1: Locate the file group1 / M00 / B3 / 62 / Cii-s1rLKaWIbiB-7AFFxKmeasnYAAFXDAATYUkAUXFC667.txt in the nginx cache file directory.

[0233] Step 2: If data is found, return directly. If data is not found, download the file from FastDFS and store it in the nginx cache file. The relative path is: group1 / M00 / B3 / 62 / Cii-s1rLKaWIbiB7AFFx-KmeasnYAAFXDAATYUkAUXFC667.txt.

[0234] Step 3: Return the cached file data.

[0235] In some embodiments, a script periodically scans the cache folder on the Nginx machine and checks whether the Nginx second-level cache has expired (the expiration time is set to 7 days) based on the file creation time. If it has expired, the file is deleted.

[0236] Based on the above embodiments, this application proposes a multi-level caching method applied to Nginx, where Nginx is the object that directly interacts with the client, such as... Figure 9 As shown, the multi-level caching method includes the following steps S901 to S907:

[0237] Step S901, Begin.

[0238] Step S902: Obtain the requested data.

[0239] Here, the requested data comes from the client.

[0240] Step S903: Does the cache folder contain the file corresponding to the requested data?

[0241] Here, if the file corresponding to the requested data does not exist in the cache folder, the file needs to be downloaded, proceeding to step S904. If the file corresponding to the requested data exists in the cache folder, proceed to step S905.

[0242] Step S904: Download the file corresponding to the requested data from FastDFS and store it in the cache folder.

[0243] Here, since the file required by the client does not exist in Nginx's cache folder, the file corresponding to the requested data will be downloaded from FastDFS, that is, the file required by the client will be downloaded; then, the downloaded file will be stored in its own cache folder for the client to access.

[0244] Step S905: Obtain the file corresponding to the requested data from the cache folder.

[0245] Step S906: Regularly clean up expired files in the Nginx cache folder.

[0246] To improve access speed, Nginx also periodically cleans up expired files in its own cache folder. For example, a 7-day time limit can be used. When a file's storage time reaches 7 days, the file is identified as expired and cleaned up.

[0247] Step S907, End.

[0248] Through steps S901 to S907 above, a storage folder format exactly the same as FastDFS is burned onto the Nginx machine. This simply and efficiently blocks some requests at the Nginx layer. Through Nginx's first-level cache, a large number of identical requests are avoided, thus perfectly solving the problem of high concurrency access.

[0249] Using the file storage method provided in this application, FastDFS stores metadata in the filename in a special way, rather than in the data itself, thus improving the upload speed of massive numbers of small files. Tests show that FastDFS, through socket communication, can achieve a single-machine upload throughput of 50,000 small files, far exceeding the throughput of Ceph (less than 10,000) and HDFS (less than 20,000). Furthermore, using a FastDFS cluster is more efficient for data storage compared to not using one. The FastDFS cluster's small file merging function prevents physical machines from running out of handles due to storing too many small files, thus avoiding situations where the physical machine has plenty of disk space but can't store any more files. It also utilizes two high-efficiency middleware, Redis and HBase, to compensate for the sacrifices FastDFS makes for efficient storage, satisfying users who need to store large amounts of file metadata while also preserving the original filenames. By leveraging Redis and HBase to store advanced configuration data and permission management data, various advanced functions can be implemented conveniently and effectively. The permission table effectively manages user permissions when uploading and downloading files. The advanced configuration data table provides various file encryption algorithms, allowing users to selectively choose different encryption algorithms. When retrieving data, the corresponding encryption algorithm is used for decryption, enriching the functionality and making subsequent billing statistics more feasible. This application also designs a multi-level caching technology to solve the problem of excessive disk I / O when high-concurrency access to various files in a distributed file system. Finally, various advanced file system functions are designed, such as detecting file anomalies, overwriting original files, periodic file deletion, and file encryption, addressing the shortcomings of ordinary file systems.

[0250] The file storage method provided by the embodiments of this application can solve the shortcomings of the following two application scenarios:

[0251] The first scenario: cold backup storage of massive log files (the files are very small), which reasonably solves the problem of inefficiency in storing such data in Ceph and HDFS.

[0252] The second scenario involves a system storing massive amounts of images and supporting efficient random image retrieval. Since image file sizes are generally small, this perfectly addresses the issue of small file storage. However, if Ceph is used for storage, its performance is poor due to its RESTful interface relying on the AWS protocol. Furthermore, during efficient concurrent file retrieval, searching for the corresponding file within a large amount of metadata and reading the file data will consume excessive time.

[0253] Based on the foregoing embodiments, this application provides a file storage device. The modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by corresponding logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0254] This application embodiment further provides a file storage device, Figure 10 This is a schematic diagram of the composition structure of the file storage device provided in the embodiments of this application, such as... Figure 10 As shown, the file storage device 1000 includes:

[0255] The acquisition module 1001 is used to acquire the file to be stored and the preset path permission data in the column-oriented distributed database;

[0256] The authentication module 1002 is used to perform permission authentication on the storage account corresponding to the file to be stored based on the preset path permission data, and obtain a first authentication result;

[0257] The detection module 1003 is used to determine that the first authentication result indicates successful authentication, and to detect the file content of the file to be stored to obtain a detection result;

[0258] The first storage module 1004 is used to store the file content of the file to be stored in the lightweight distributed file system and return the file path of the file to be stored when the detection result indicates that the detection is passed.

[0259] The second storage module 1005 is used to generate metadata of the file to be stored based on the file path of the file to be stored, and to store the metadata and configuration data of the file to be stored in the remote dictionary service and the column-oriented distributed database.

[0260] In some embodiments, the first storage module 1004 includes:

[0261] The first acquisition submodule is used to acquire the configuration data of the file to be stored;

[0262] The first encryption detection submodule is used to perform encryption detection on the file to be stored based on the configuration data of the file to be stored, and obtain the first encryption detection result.

[0263] An encryption submodule is used to determine that the first encryption detection result indicates that the file to be stored needs to be encrypted, and to encrypt the file to be stored to obtain the encrypted file to be stored.

[0264] The first storage submodule is used to store the encrypted file to be stored in the lightweight distributed file system.

[0265] In some embodiments, the acquisition module 1001 is further configured to acquire current time information, the first expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the column-oriented distributed database; the file storage device 1000 further includes:

[0266] The first determining module is used to determine the first expired file in the remote dictionary service and the second expired file in the column-oriented distributed database based on the current time information, the first expiration time, and the second expiration time.

[0267] The second determining module is used to determine the corresponding third expired file in the lightweight distributed file system based on the first expired file and the second expired file;

[0268] The deletion module is used to delete the first expired file, the second expired file, and the third expired file.

[0269] In some embodiments, the acquisition module 1001 is further configured to acquire the access count of each file in the lightweight distributed file system; the file storage device 1000 further includes:

[0270] The third determining module is used to determine the target file from the files based on each access count and a count threshold, wherein the access count of the target file is higher than the count threshold;

[0271] The third storage module is used to store the basic data of the target file in the remote dictionary service.

[0272] In some embodiments, the acquisition module 1001 is further configured to acquire a plurality of files to be stored and determine the total storage space occupied by the plurality of files to be stored; the file storage device 1000 further includes:

[0273] The merging module is used to determine that the total storage space is less than the storage space threshold, and merge the multiple files to be stored to obtain a merged file to be stored. The merged file to be stored contains location indication information, which is used to indicate the location of different files to be stored.

[0274] In some embodiments, the authentication module 1002 is further configured to perform permission authentication on the download account corresponding to the download request based on the basic data of the file to be downloaded, and obtain a second authentication result; the detection module 1003 is further configured to determine that the second authentication result indicates successful authentication, and perform time limit detection on the file to be downloaded using the basic data of the file to be downloaded, and obtain a time limit detection result;

[0275] The file storage device 1000 further includes:

[0276] The response module is used to respond to a download request sent by the client and obtain the basic data of the file to be downloaded corresponding to the download request, wherein the basic data of the file to be downloaded is stored in the file storage system;

[0277] The return module is used to determine that the expiration date detection result indicates that the file to be downloaded has not expired, and then returns the file to be downloaded to the client.

[0278] In some embodiments, the return module includes:

[0279] The second encryption detection submodule is used to perform encryption detection on the file to be downloaded based on the basic data of the file to be downloaded, and obtain a second encryption detection result;

[0280] The second acquisition submodule is used to determine that the second encryption detection result indicates that the file to be downloaded is an encrypted file, and to acquire the key sent by the client;

[0281] The decryption submodule is used to decrypt the file to be downloaded using the key to obtain a decrypted file;

[0282] The return submodule is used to return the decrypted file to the client.

[0283] It should be noted that the description of the file storage device in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment of this application for understanding.

[0284] It should be noted that, in the embodiments of this application, if the above-described file storage method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0285] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the file storage method provided in the above embodiments.

[0286] This application provides a file storage system. Figure 11 This is a schematic diagram of the composition structure of the file storage system provided in the embodiments of this application, such as... Figure 11 As shown, the file storage system 1100 includes: a lightweight distributed file system 1101, a remote dictionary service 1102, and a column-oriented distributed database 1103. The files stored in the file storage system 1100 include target files, which are files whose access count exceeds a threshold.

[0287] The lightweight distributed file system 1101 is used to store the file contents of all stored files;

[0288] The remote dictionary service 1102 is used to store the basic data of the target file, which includes metadata and configuration data;

[0289] The column-oriented distributed database 1103 is used to store the basic data of all stored files.

[0290] The descriptions of the above file storage system and storage medium embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the file storage system and storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0291] It should be noted that the descriptions of the above embodiments of storage media and file storage systems are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of storage media and file storage systems of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0292] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0293] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0294] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0295] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0296] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0297] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0298] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an AC to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0299] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A file storage method, applied to a file storage system, characterized in that, The file storage system includes: a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database; the method includes: Obtain the file to be stored and the preset path permission data in the column-oriented distributed database; Based on the preset path permission data, the storage account corresponding to the file to be stored is authenticated to obtain a first authentication result; Once the first authentication result indicates successful authentication, the file content of the file to be stored is detected to obtain the detection result. When the detection result indicates that the detection is passed, the file content of the file to be stored is stored in the lightweight distributed file system, and the file path of the file to be stored is returned; Based on the file path of the file to be stored, the metadata of the file to be stored is generated, and the metadata and configuration data of the file to be stored are stored in the remote dictionary service and the column-oriented distributed database. Storing the file content of the file to be stored in the lightweight distributed file system includes: Obtain the configuration data of the file to be stored; Based on the configuration data of the file to be stored, an encryption detection is performed on the file to be stored to obtain a first encryption detection result; The first encryption detection result indicates that the file to be stored needs to be encrypted. The file to be stored is then encrypted to obtain the encrypted file to be stored. The encrypted file to be stored is stored in the lightweight distributed file system.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the current time information, the first expiration time of the basic data in the remote dictionary service, and the second expiration time of the basic data in the column-oriented distributed database; Based on the current time information, the first expiration time, and the second expiration time, the first expired file in the remote dictionary service and the second expired file in the column-oriented distributed database are determined. Based on the first expired file and the second expired file, determine the corresponding third expired file in the lightweight distributed file system; Delete the first expired file, the second expired file, and the third expired file.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the access count of each file in the lightweight distributed file system; Based on the number of accesses and the access threshold, a target file is determined from the files, wherein the number of accesses to the target file is higher than the access threshold. The basic data of the target file is stored in the remote dictionary service.

4. The method according to claim 1, characterized in that, The method further includes: Obtain multiple files to be stored, and determine the total storage space occupied by the multiple files to be stored; If the total storage space is determined to be less than the storage space threshold, the multiple files to be stored are merged to obtain a merged file to be stored. The merged file to be stored contains location indication information, which is used to indicate the location of different files to be stored.

5. The method according to any one of claims 1 to 4, characterized in that, The method includes: In response to a download request sent by a client, the basic data of the file to be downloaded corresponding to the download request is obtained, wherein the basic data of the file to be downloaded is stored in the file storage system; Based on the basic data of the file to be downloaded, the download account corresponding to the download request is authenticated to obtain a second authentication result; Once the second authentication result indicates successful authentication, the file to be downloaded is subjected to an expiration check using the basic data of the file to be downloaded, and an expiration check result is obtained. If the result of the expiration date detection indicates that the file to be downloaded has not expired, the file to be downloaded will be returned to the client.

6. The method according to claim 5, characterized in that, Returning the file to be downloaded to the client includes: Based on the basic data of the file to be downloaded, an encryption detection is performed on the file to be downloaded to obtain a second encryption detection result; The second encryption detection result indicates that the file to be downloaded is an encrypted file, and the key sent by the client is obtained; The file to be downloaded is decrypted using the key to obtain a decrypted file; The decrypted file is returned to the client.

7. A file storage device, characterized in that, The file storage device includes: The acquisition module is used to acquire the file to be stored and the preset path permission data in the column-oriented distributed database; The authentication module is used to perform permission authentication on the storage account corresponding to the file to be stored based on the preset path permission data, and obtain a first authentication result; The detection module is used to determine that the first authentication result indicates successful authentication, and to detect the file content of the file to be stored to obtain the detection result; The first storage module is used to store the file content of the file to be stored to a lightweight distributed file system and return the file path of the file to be stored when the detection result indicates that the detection is passed. The second storage module is used to generate metadata of the file to be stored based on the file path of the file to be stored, and to store the metadata and configuration data of the file to be stored in the remote dictionary service and the column-oriented distributed database. The first storage module is further configured to: acquire configuration data of the file to be stored; perform encryption detection on the file to be stored based on the configuration data of the file to be stored to obtain a first encryption detection result; determine that the first encryption detection result indicates that the file to be stored needs to be encrypted, perform encryption processing on the file to be stored to obtain an encrypted file to be stored; and store the encrypted file to be stored in the lightweight distributed file system.

8. A file storage system, characterized in that, The file storage system includes a lightweight distributed file system, a remote dictionary service, and a column-oriented distributed database. The files stored in the file storage system include target files, which are files whose access count exceeds a threshold. The lightweight distributed file system is used to store the file content of all stored files; The remote dictionary service is used to store the basic data of the target file, wherein the basic data includes metadata and configuration data; The column-oriented distributed database is used to store the basic data of all stored files; The lightweight distributed file system is also used to store encrypted files to be stored. The encrypted files to be stored are obtained by acquiring the configuration data of the files to be stored, performing encryption detection on the files to be stored based on the configuration data, obtaining a first encryption detection result, determining that the first encryption detection result indicates that the files to be stored need to be encrypted, and then performing encryption processing on the files to be stored.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions configured to perform the file storage method according to any one of claims 1 to 6.

Images