File storage and retrieval method, device, storage medium and server
A file storage and file retrieval technology, applied in the field of big data processing, can solve the problems of consuming a lot of time, consuming startup tasks and release tasks, etc., and achieve the effect of reducing pressure and effectively utilizing storage resources
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0054] figure 1 It is a schematic flow chart of the file storage method provided by the embodiment of the present invention. This method is applicable to the situation where the server manages small files reasonably, and can be executed by a file storage device. The device can be implemented by software and / or hardware, and generally can be integrated in the server. Such as figure 1 As shown, the method includes:
[0055] S101. Acquire file information of a small file, where the size of the small file is smaller than a preset threshold.
[0056] In the embodiment of the present invention, when the size of a certain file is smaller than a preset threshold, the file may be called a small file. For example, if the preset threshold is 64K, a file whose size is smaller than 64K may be called a small file. The file information of the small file is acquired, wherein the file information of the small file may include any information related to the small file. Exemplarily, the file...
Embodiment 2
[0066] figure 2 It is a flowchart of a file storage method provided in Embodiment 2 of the present invention. This embodiment is optimized on the basis of the foregoing embodiments. In this embodiment, the file storage method for obtaining the file information of the small file includes: judging whether the file in the HDFS is a small file, and if so, obtaining the small file in the HDFS The file information; Based on the primary key, writing the small file into the pre-created database Hbase table includes: determining the position of the small file in HDFS, and based on the primary key and the location of the small file in HDFS location, write the small files in the HDFS into the pre-created database Hbase table in the form of a binary stream. After writing the small files in the HDFS into the pre-created database Hbase table in the form of a binary stream based on the primary key and the position of the small files in the HDFS, it also includes: closing the binary stream,...
Embodiment 3
[0076] image 3 It is a flowchart of a file storage method provided by Embodiment 3 of the present invention. The present embodiment optimizes on the basis of the foregoing embodiments. In the present embodiment, the Hbase table of the database includes memory space; writing the small file into the Hbase table of the database created in advance based on the primary key includes: The primary key writes the small file into the memory space in the pre-created database Hbase table; when the current available space of the memory space is less than the preset memory threshold, writes the small file into HDFS in the form of a storage file . Correspondingly, the method in this embodiment specifically includes:
[0077] S301. Acquire file information of a small file, where the size of the small file is smaller than a preset threshold.
[0078] S302. Generate a primary key uniquely corresponding to the small file according to the file information according to preset rules.
[0079] ...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More