File storage system, method and device, electronic equipment and storage medium
By introducing storage driver modules, storage management modules and access gateway modules into the data storage system, combined with category identification technology, the problems of low storage space utilization and low deletion efficiency in the existing technology are solved, and efficient file deletion and storage space management are achieved.
Patent Information
- Application Number
- CN202411973040.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
When existing data storage systems process large amounts of small files, the storage space utilization rate is low and the deletion efficiency is not high. They need to frequently deploy multiple storage management system instances for management.
A file storage system is proposed, including a storage driver module, a storage management module and an access gateway module, which can quickly delete files through category identification and improve deletion efficiency.
Through modular design and category identification, the utilization rate of storage space and file deletion efficiency are improved, and massive small files can be quickly deleted according to business and time.
Smart Images
Figure CN119884035A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data storage, and in particular to a file storage system, method, device, electronic device and storage medium. Background Art
[0002] With the development of informatization of various businesses, various types of collected and retained data have shown explosive growth. Therefore, data storage has the characteristics of large total volume and small individual files. In the case of limited overall storage space, a retention period is usually set for the collected and retained data based on business characteristics, and the data is deleted after the retention period expires.
[0003] In response to this situation, a common practice is to deploy multiple storage management system instances, each of which corresponds to a smaller amount of storage space. Data is stored in different storage management system instances according to time and business. After the retention period expires, the corresponding storage management system instance is deleted and the storage space is formatted. In this implementation, the storage cycle corresponding to a single storage instance is usually one month. After the retention period expires, the data is deleted and cleaned up on a monthly basis, resulting in low storage space utilization. Summary of the invention
[0004] In view of this, the purpose of the present disclosure is to provide a file storage system, method, device, electronic device and storage medium, which can solve the existing problems in a targeted manner.
[0005] Based on the above-mentioned purpose, in a first aspect, the present disclosure proposes a file storage system, wherein the storage system includes a storage driver module, a storage management module and an access gateway module, wherein folders are stored in the storage system, and the folders have category identifiers; the storage driver module is used to provide operation functions for files in the storage system; the storage management module is used to call the storage driver module to perform the operation function, and return the file identifier of the stored file to a target electronic device requesting to implement the operation function; the access gateway module is used to provide the target electronic device with the operation function for files in the storage system.
[0006] In a second aspect, a method for storing files is also provided, wherein the method adopts the storage system as described in the first aspect.
[0007] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the second aspect.
[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the method described in the second aspect.
[0009] In a fifth aspect, a computer program product is also provided, comprising a computer program, wherein the computer program is executed by a processor to implement the method described in the second aspect.
[0010] In general, the present disclosure has at least the following beneficial effects: the access of external devices to data in the storage system can be realized through various modules. Moreover, files of a certain category can be quickly deleted as a whole through category identification, thereby improving deletion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.
[0012] Figure 1 A schematic diagram showing a file storage system according to an embodiment of the present disclosure;
[0013] Figure 2 A flowchart showing a method for storing a file according to an embodiment of the present disclosure is shown;
[0014] Figure 3 A schematic diagram showing a storage device for files according to an embodiment of the present disclosure is shown;
[0015] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;
[0016] Figure 5 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0017] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0019] Figure 1The storage system of the file of the present disclosure is shown. In the embodiment of the present disclosure, the storage system includes a storage driver module, a storage management module and an access gateway module. The storage system stores a folder, and the folder has a category identifier.
[0020] The storage driver module is used to provide an operation function for files in the storage system.
[0021] The storage management module is used to call the storage driver module to perform the operation function and return the file identifier of the stored file to the target electronic device requesting to implement the operation function;
[0022] The access gateway module is used to provide the target electronic device with the operation function for the files in the storage system.
[0023] The target electronic device here generally refers to other electronic devices other than the present device, and in some cases may also refer to the present device. The present device here is the electronic device where the storage system is located.
[0024] In some optional implementations of any embodiment of the present disclosure, there is a secondary index for the files in the storage system, and the secondary index includes a first-level index corresponding to the first-level folder, and a second-level index corresponding to the second-level folder in the first-level folder, the first-level index indicates a first-level identifier, and the second-level index indicates a second-level identifier, the first-level identifier and the second-level identifier correspond to different category identifiers, and the category identifier can be a business category identifier or a time period identifier.
[0025] The primary identifier is one of a service category identifier and a time period identifier, and the secondary identifier is the other of the service category identifier and the time period identifier. The time period here can be any time period such as one day.
[0026] Optionally, each second-level folder includes a plurality of files, and the files in the second-level folder are stored in the form of a file header and file content, wherein the file header includes a file size and a checksum of the file.
[0027] Optionally, the storage management module is further configured to, in response to determining that a file corresponding to a target business category is expired, search for a first-level folder or a second-level folder corresponding to the target business category as a target folder, and delete the target folder.
[0028] In some optional implementations of any embodiment of the present disclosure, the file identifier includes: a business category identifier, a storage time, an internal index, and a file extension, wherein the internal index refers to an index for searching for files within a second-level folder; the internal index includes: an auto-incrementing serial number of the file in the second-level folder, an address offset of the file in the second-level folder, the size of the file, and a checksum of the file.
[0029] In some optional implementations of any embodiment of the present disclosure, the access gateway is also used to parse the file identifier of the file to obtain the business category, storage time and internal index of the file, read the file content of the file through the storage drive, and return it to the target electronic device.
[0030] The present application also provides a file storage system, which can quickly delete a large number of small files that have exceeded the retention period according to business and time, effectively improving the utilization rate of storage. The storage system includes:
[0031] 1) Storage driver module
[0032] The storage driver module is a system module that actually performs file reading and writing, and provides functions such as file reading, writing, and deleting to the outside (target electronic device).
[0033] 2) Storage Management Module
[0034] The storage management module is the core component of this system. It is responsible for calling the write and delete functions of the storage driver, implementing the file write operation, and returning the file identifier of the stored file to the outside.
[0035] For small files such as pictures, merged storage is used for storage.
[0036] The specific methods of merging storage are as follows:
[0037] (1) The specific files to be merged and stored will be indexed at the secondary level according to the business classification and the storage time with a granularity of one day. Usually, two-level folders are used for indexing. If there are multiple merged and stored files on the same day, they will be named according to the self-incrementing sequence number.
[0038] (2) In a merged storage file, small files such as pictures actually stored will be continuously added to the merged storage file in the form of file header plus file content;
[0039] (3) The file header contains the file size (32 bits) and the file's CRC32 checksum (32 bits)
[0040] (4) When writing small files, append is used to maximize the writing performance.
[0041] The composition of the file identifier returned by the storage management module is as follows: business classification identifier / storage time with a granularity of days / internal index.file extension.
[0042] The internal index is the result of BASE64 encoding the following binary stream:
[0043] 0-31 bits Merge the auto-increment sequence number of the stored files 32-63 bits The offset of the file in the merged storage file 64-95 bits File size 96-127 bits CRC32 checksum of the file
[0044] When the storage management module deletes the expired files belonging to each business, it only needs to delete the folder corresponding to the index.
[0045] 3) Access Gateway
[0046] It provides the function of externally accessing files stored in the system. By parsing the file identifier, it obtains information such as the business category to which the file belongs, the storage time, and the internal index of the file. It reads the specific file content through the storage driver module and returns it to the outside.
[0047] This embodiment can adopt the method of the system returning the file identifier, and the access gateway and storage management do not need additional file name-storage location mapping storage, and the access gateway can run independently of the storage management to achieve file access. The use of a secondary index effectively reduces the granularity of storage cycle management, so that files that have exceeded their validity period can be cleared as soon as possible, improving storage utilization. It has file error detection capabilities. By verifying the CRC32 checksum in the file identifier and the CRC32 checksum in the file header, it can be determined whether the file identifier is valid. By performing a CRC32 check on the file content and comparing it with the CRC32 checksum, it can be determined whether the stored file is damaged.
[0048] The present disclosure also provides a method for storing files, such as Figure 2 As shown, the method adopts the storage system described in any of the above embodiments; the storage system includes a storage driver module, a storage management module and an access gateway module, and the storage system stores a folder, and the folder has a category identifier;
[0049] The method comprises:
[0050] The storage driver module provides an operation function for files in the storage system;
[0051] The storage management module calls the storage driver module to perform the operation function, and returns the file identifier of the stored file to the target electronic device requesting to implement the operation function;
[0052] The access gateway module provides the target electronic device with the operation function for the files in the storage system.
[0053] The present disclosure provides a file storage device, which is used to execute the file storage method described in any of the above embodiments, such as Figure 3 As shown, the device includes: a file storage device, the device adopts any of the above storage systems, the storage system includes a storage driver module, a storage management module and an access gateway module, the storage system stores a folder, and the folder has a category identifier;
[0054] The device comprises:
[0055] A first unit is configured to provide the storage driver module with an operation function for files in the storage system;
[0056] The second unit is configured so that the storage management module calls the storage driver module to perform the operation function, and returns the file identifier of the stored file to the target electronic device requesting to implement the operation function;
[0057] The third unit is configured so that the access gateway module provides the target electronic device with the operation function on the files in the storage system.
[0058] The file storage device provided by the above-mentioned embodiment of the present disclosure and the file storage method provided by the embodiment of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.
[0059] The embodiment of the present disclosure also provides an electronic device corresponding to the file storage method provided in the above embodiment, so as to execute the above file storage method. The embodiment of the present disclosure is not limited.
[0060] Please refer to Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 4 As shown, the electronic device 40 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and the processor 400 executes the method provided in any of the aforementioned embodiments of the present disclosure when running the computer program.
[0061] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0062] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs, and the processor 400 executes the programs after receiving the execution instruction. The file storage method disclosed in any implementation of the above-mentioned embodiment of the present disclosure may be applied to the processor 400, or implemented by the processor 400.
[0063] The processor 400 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 400 or the instruction in the form of software. The above processor 400 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.
[0064] The electronic device provided by the embodiment of the present disclosure and the file storage method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.
[0065] The present disclosure also provides a computer-readable storage medium corresponding to the file storage method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 50 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the file storage method provided by any of the aforementioned embodiments will be executed.
[0066] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0067] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the file storage method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0068] It should be noted that:
[0069] In the above text, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0071] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are within the protection of the present disclosure.
Claims
1. A file storage system, the storage system comprising a storage driver module, a storage management module and an access gateway module, the storage system storing a folder, the folder having a category identifier; The storage driver module is used to provide an operation function for files in the storage system; The storage management module is used to call the storage driver module to perform the operation function and return the file identifier of the stored file to the target electronic device requesting to implement the operation function; The access gateway module is used to provide the target electronic device with the operation function for the files in the storage system.
2. The storage system according to claim 1, characterized in that: There is a secondary index for the files in the storage system, and the secondary index includes a first-level index corresponding to the first-level folder, and a second-level index corresponding to the second-level folder in the first-level folder, the first-level index indicates a first-level identifier, and the second-level index indicates a second-level identifier, and the first-level identifier and the second-level identifier correspond to different category identifiers, which can be a business category identifier or a time period identifier.
3. The storage system according to claim 2, characterized in that: Each second-level folder includes a plurality of files. The files in the second-level folder are stored in the form of a file header and file content. The file header includes a file size and a checksum of the file.
4. The storage system according to claim 1, characterized in that: The file identifier includes: Business category identifier, storage time, internal index and file extension, where the internal index refers to the index for searching files in the second-level folder; The internal index includes: the self-incrementing serial number of the file in the second-level folder, the address offset of the file in the second-level folder, the size of the file, and the checksum of the file.
5. The storage system according to claim 1, characterized in that: The access gateway is also used to obtain the business category, storage time and internal index of the file by parsing the file identifier of the file, read the file content of the file through the storage driver, and return it to the target electronic device.
6. The storage system according to claim 2, characterized in that: The storage management module is further configured to, in response to determining that a file corresponding to a target business category is expired, search for a first-level folder or a second-level folder corresponding to the target business category as a target folder, and delete the target folder.
7. A method for storing a file, characterized in that: The method adopts a storage system as described in any one of claims 1 to 6; the storage system comprises a storage driver module, a storage management module and an access gateway module, and a folder is stored in the storage system, and the folder has a category identifier; The method comprises: The storage driver module provides an operation function for files in the storage system; The storage management module calls the storage driver module to perform the operation function, and returns the file identifier of the stored file to the target electronic device requesting to implement the operation function; The access gateway module provides the target electronic device with the operation function on the files in the storage system.
8. A file storage device, characterized in that: The device adopts the storage system according to any one of claims 1 to 6, the storage system comprises a storage driver module, a storage management module and an access gateway module, the storage system stores a folder, and the folder has a category identifier; The device comprises: A first unit is configured to provide the storage driver module with an operation function for files in the storage system; The second unit is configured so that the storage management module calls the storage driver module to perform the operation function, and returns the file identifier of the stored file to the target electronic device requesting to implement the operation function; The third unit is configured so that the access gateway module provides the target electronic device with the operation function on the files in the storage system.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claim 7.