Small file storage method, device and computer equipment

Abstract

The present invention discloses a small file storage method, device and computer equipment, wherein the method comprises the following steps: grouping a plurality of small files to be burned to obtain a plurality of file segments, each file segment comprising at least two small files; merging the small files in each file segment to obtain a plurality of merged files; loading a preset burning script to burn the plurality of merged files into a burning device. The embodiment of the present invention reduces index node metadata, reduces storage operations, improves file storage speed, and also improves file system processing efficiency and operation speed by merging at least two small files into a merged file for burning.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a small file storage method, device and computer equipment. Background Art

[0002] In today's Internet, data is growing explosively. Social networks, mobile communications, online videos, e-commerce and other applications often generate hundreds of millions, even billions or tens of billions of small files (files no larger than 1MB in size). We usually define millions or more as massive, and thus we can quantify the definition of massive small files.

[0003] In traditional file systems, each file needs to create a corresponding index node metadata, and storage and query are performed based on the index node metadata. However, in the scenario of massive small files, traditional file systems can no longer support so many metadata storage operations, resulting in some small file storage operations requiring a long wait time and slow operating system operation. Therefore, it is urgent to propose a small file storage method to achieve the storage of massive small files in order to improve the operating speed of the operating system. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defects in the prior art that the traditional file system can no longer carry the metadata storage operations of massive small files, resulting in some small file storage operations requiring a long waiting time and the operating system running slowly, thereby providing a small file storage method, device and computer equipment.

[0005] According to the first aspect, the present invention discloses a small file storage method, comprising the following steps: grouping multiple small files to be burned to obtain multiple file segments, each file segment including at least 2 small files; merging the small files in each file segment respectively to obtain multiple merged files; loading a preset burning script to burn the multiple merged files into a burning device.

[0006] Optionally, before grouping the multiple small files to be burned, the method also includes: obtaining the current storage capacity of the file system for storing small files; when the current storage capacity of the file system for storing small files is greater than a preset threshold, writing the small files into the file system for storing small files to obtain multiple small files to be burned.

[0007] Optionally, the method further includes: acquiring file attribute information of the plurality of small files to be burned; and storing the file attribute information of the plurality of small files to be burned in a preset database.

[0008] Optionally, the file attribute information includes: the latest modification time of the file; grouping multiple small files to be burned to obtain multiple file segments, including: sorting the multiple small files to be burned according to the latest modification time of the files; grouping the sorted multiple small files to be burned to obtain multiple file segments.

[0009] Optionally, before merging the small files in each file segment to obtain multiple merged files, the method also includes: sorting the small files in each file segment according to the latest modification time of the files in the attribute information; and merging each sorted file segment to obtain multiple merged files.

[0010] Optionally, the merging of small files in each file segment to obtain multiple merged files includes: obtaining file attribute information of the small files in each file segment, filling the file attribute information into a preset file handle, and obtaining the multiple merged files.

[0011] Optionally, the file attribute information includes length information of small files, and the method further includes: obtaining the initial position of each small file in the current merged file based on the position of the previous merged file and the length information of each small file in the current merged file; and associating the initial position of each small file in the current merged file, the file attribute information of the small file and the merged file name of the small file and storing them in the preset database.

[0012] Optionally, the method also includes: when a small file read request is received, obtaining the attribute information of the small file, the initial position and the merged file of the small file from the preset database according to the small file read request, the small file read request including: the file system where the small file is stored and the name of the merged file of the small files; reading the data content of the small file from the merged file where the small file is located according to the length information in the small file attribute information and the initial position.

[0013] Optionally, merging the small files in each file segment separately to obtain multiple merged files includes: when a merge instruction is received, merging the small files in the file segment according to the merge instruction to obtain multiple merged files, and the merge instruction includes: a merge quantity.

[0014] Optionally, the loading of a preset burning script to burn the multiple merged files into a burning device includes: sending a burning request; when a feedback signal is received, burning the merged file into a burning device corresponding to the feedback signal according to the feedback signal, wherein the feedback signal includes a burning quantity.

[0015] Optionally, the method further includes: executing a corresponding small file storage process according to a current state of the finite state machine.

[0016] According to the second aspect, the present invention discloses a small file storage device, including: a grouping module, used to group multiple small files to be burned to obtain multiple file segments, each file segment includes at least 2 small files; a merging module, used to merge the small files in each file segment respectively to obtain multiple merged files; a burning module, used to load a preset burning script to burn the multiple merged files into a burning device.

[0017] According to the third aspect, the present invention also discloses a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the small file storage method as described in the first aspect or any optional embodiment of the first aspect.

[0018] According to a fourth aspect, the present invention further discloses a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the small file storage method as described in the first aspect or any optional implementation of the first aspect are implemented. The technical solution of the present invention has the following advantages:

[0019] The small file storage method and device provided by the present invention obtain multiple file segments by grouping multiple small files to be burned, each file segment includes at least 2 small files, and the small files in each file segment are merged to obtain multiple merged files, and a preset burning script is loaded to burn the multiple merged files into a burning device. The embodiment of the present invention reduces index node metadata and storage operations, improves file storage speed, and also improves file system processing efficiency and operation speed by merging at least 2 small files into one merged file for burning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a specific example of a small file storage method in an embodiment of the present invention;

[0022] Figure 2is a principle block diagram of a specific example of a small file storage device in an embodiment of the present invention;

[0023] Figure 3 FIG. 4 is a specific example diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] The embodiment of the present invention discloses a small file storage method, such as Figure 1 As shown, the following steps are included:

[0029] S11: Grouping a plurality of small files to be recorded to obtain a plurality of file segments, each of which includes at least two small files.

[0030] Exemplarily, the multiple small files to be burned may be stored in a preset database in advance, or may be received in real time. The embodiment of the present invention does not specifically limit the method for obtaining the multiple small files to be burned, and those skilled in the art may determine it according to actual conditions.

[0031] The grouping of the multiple small files to be burned can be performed randomly or according to a preset rule, for example, grouping in a preset number according to the order in which the files are received / stored. The embodiment of the present invention does not specifically limit the grouping method of the multiple small files to be burned, and those skilled in the art can set it according to actual conditions.

[0032] The number of small files in each file segment obtained after grouping can be the same or different. The embodiment of the present invention does not specifically limit the number of small files in each file segment, as long as the number of small files in each file segment is greater than 2. Preferably, in order to simplify the subsequent merging and burning process, the number of small files in each file segment can be the same during grouping.

[0033] S12: Merge the small files in each file segment to obtain multiple merged files.

[0034] Exemplarily, the format of the merged file can be a Tar format file or a Zip format file. The embodiment of the present invention does not specifically limit the format of the merged file, and those skilled in the art can choose according to actual conditions. The method for merging small files in the file segment can be to call a related merging function to merge the small files.

[0035] S13: Load the preset burning script to burn the multiple merged files into the burning device.

[0036] Exemplarily, the preset burning script may be pre-edited and stored in a file system storing small files. The burning device may include: an optical disc, a burning card, etc. The embodiment of the present invention does not specifically limit the burning device, and those skilled in the art may set it according to actual conditions.

[0037] The multiple merged files may be burned into the burning device one by one or multiple files may be burned at the same time. The embodiment of the present invention does not specifically limit the number of files to be burned, and those skilled in the art may set the number according to actual conditions.

[0038] The small file storage method provided by the present invention obtains multiple file segments by grouping multiple small files to be burned, each file segment includes at least 2 small files, and the small files in each file segment are merged to obtain multiple merged files, and a preset burning script is loaded to burn the multiple merged files into a burning device. The embodiment of the present invention reduces index node metadata and storage operations, improves file storage speed, and also improves file system processing efficiency and operation speed by merging at least 2 small files into one merged file for burning.

[0039] As an optional implementation of the embodiment of the present invention, before the above step S11, the small file storage method further includes:

[0040] First, the current storage capacity of the file system for storing small files is obtained.

[0041] Exemplarily, the file system may be an HDFS system or a ZFS system. The present invention does not specifically limit the file system, and those skilled in the art may set it according to actual conditions. The method for obtaining the current storage capacity may be to monitor the status of each file system in real time.

[0042] Secondly, when the current storage capacity of the file system for storing small files is greater than a preset threshold, the small files are written into the file system for storing small files to obtain a plurality of small files to be recorded.

[0043] Exemplarily, the preset threshold value may be 3%. The method of writing small files into a file system for storing small files may be to write in the order of the latest modification of the files, or to write in the order in which the files are sent by the user. When writing small files, one file may be written at a time, or multiple files may be written at a time. The embodiment of the present invention does not specifically limit the setting of the preset threshold value, the method of writing small files, and the number of small files written each time, and those skilled in the art may set them according to actual conditions.

[0044] As an optional implementation of the embodiment of the present invention, the small file storage method further includes:

[0045] First, the file attribute information of a plurality of small files to be burned is obtained.

[0046] Exemplarily, the file attribute information may include: file name, file size, file length, file latest modification time, and custom attributes, etc. The present invention does not specifically limit the file attribute information, and those skilled in the art may set it according to actual conditions. The file attribute information may be sent by the user together with the small file.

[0047] Secondly, the file attribute information of the plurality of small files to be recorded is stored in a preset database.

[0048] The embodiment of the present invention stores the file attribute information of each small file written into the file system into a preset database, which can be directly called in subsequent use. For example, the position of each small file in the merged file is determined according to the file attribute information, and the specific information of the small file is queried according to the file name when reading the small file.

[0049] As an optional implementation of the embodiment of the present invention, the file attribute information includes: the latest modification time of the file; the above step S11 includes:

[0050] First, sort the multiple small files to be burned according to the time when they were most recently modified.

[0051] Exemplarily, the sorting of the multiple small files to be burned according to the latest modification time of the files can be performed in ascending order or in descending order. The embodiment of the present invention does not specifically limit the sorting method of the multiple small files to be burned, and those skilled in the art can determine it according to actual conditions. The embodiment of the present invention uses an ascending sorting method to sort the multiple small files to be burned according to the latest modification time of the files.

[0052] Secondly, the sorted multiple small files to be recorded are grouped to obtain multiple file segments.

[0053] For example, the sorted multiple small files to be recorded can be grouped from front to back according to the sorting order, or randomly selected according to the sorting order. The embodiment of the present invention does not specifically limit the grouping method, and those skilled in the art can set it according to actual conditions. The number of each file segment can be the same or different.

[0054] As an optional implementation of the embodiment of the present invention, before the above step S12, the small file storage method further includes:

[0055] First, the small files in each file segment are sorted according to the latest modification time of the files in the attribute information.

[0056] For example, it can be sorted in ascending order or descending order. The present embodiment does not specifically limit the sorting method of the plurality of small files to be recorded, and those skilled in the art can determine it according to actual conditions. In order to improve the storage and reading speed, the present embodiment uses ascending order to sort the small files in each file segment according to the latest modification time of the file.

[0057] Secondly, each of the sorted file segments is merged to obtain multiple merged files. The specific merging method is described in the above step S12, which will not be repeated here.

[0058] The embodiment of the present invention sorts each file segment according to the latest modification time of the file, so as to facilitate traversal and reading according to the latest modification time of the file when the merge function performs the merge operation, thereby improving the small file merging speed.

[0059] As an optional implementation of the embodiment of the present invention, the above step S12 includes:

[0060] The file attribute information of the small files of each file segment is obtained, and the file attribute information is filled into the preset file handle to obtain multiple merged files.

[0061] Exemplarily, the file attribute information can be directly called from a preset database. The preset file handle can be a variety of file handles set in advance according to a variety of file formats, or can be constructed according to the acquired file attribute information. The embodiment of the present invention does not specifically limit the preset file handle, and those skilled in the art can determine it according to actual conditions. Fill the file attribute information of the small files in each file segment into its corresponding file handle to obtain multiple merged files.

[0062] It should be noted that only when the file handle is closed correctly can the data content in the merged file be read later, that is, close the small file first, and then close the file handle.

[0063] As an optional implementation of the embodiment of the present invention, the file attribute information includes the length information of the small file, and the small file storage method further includes:

[0064] First, the initial position of each small file in the current merged file is obtained according to the position of the previous merged file and the length information of each small file in the current merged file.

[0065] Exemplarily, the initial position of each small file in the current merged file is obtained based on the position of the last merged file and the length information of each small file in the current merged file. Specifically, the initial position of each small file in the current merged file can be determined by summing the position of the last merged file, the handle header length of the preset file handle, and the length of each small file. For the first small file in the current merged file, its initial position is the sum of the position of the last merged file and the handle header length of the preset file handle. The position of the second small file can be determined based on the initial position and file length of the first small file..., and so on, to obtain the initial positions of all small files in the current merged file. In particular, for the first merged file, the position of its last merged file is 0.

[0066] Secondly, the initial positions of the small files in the current merged file, the file attribute information of the small files, and the merged file name of the small files are associated and stored in a preset database.

[0067] The embodiment of the present invention obtains the initial position of each small file by calculation and stores it in a preset database. When reading the file subsequently, the attribute content of the small file can be directly obtained according to the initial position of each small file and the file length.

[0068] As an optional implementation of the embodiment of the present invention, the small file storage method further includes:

[0069] First, when a small file read request is received, the attribute information, initial position and merged file of the small file are obtained from the preset database according to the small file read request. The small file read request includes: the file system where the small file is stored and the name of the merged file of the small file.

[0070] Exemplarily, the small file read request may be sent by a user through a wired network or a wireless network. The embodiment of the present invention does not specifically limit the method for receiving the small file read request, and those skilled in the art may determine it according to the actual sending method. According to the file system where the small files in the small file read request are stored and the name of the merged file of the small files, the attribute information, the initial position and the merged file of the small files are obtained from the preset database.

[0071] Secondly, the data content of the small file is read from the merged file where the small file is located according to the length information and the initial position in the attribute information of the small file.

[0072] Exemplarily, reading the data content of the small file from the merged file where the small file is located according to the length information and the initial position in the small file attribute information can specifically include opening the corresponding merged file, moving the pointer to the initial position, and reading the specific content of the small file according to the length information in the small file attribute information.

[0073] It should be noted that the embodiment of the present invention can execute the reading process of multiple small files at the same time, as long as multiple threads are allocated, thereby improving the file reading efficiency.

[0074] As an optional implementation of the embodiment of the present invention, the above step S12 includes:

[0075] When a merge instruction is received, the small files in the file segment are merged according to the merge instruction to obtain multiple merged files, and the merge instruction includes: a merge quantity. The method for receiving the merge instruction can receive through a wireless network, allocate multiple threads for merging according to the merge quantity in the merge instruction, merge the small files in the multiple file segments within the range allowed by the hardware configuration, improve the efficiency of small file merging processing, and improve the system processing capacity.

[0076] As an optional implementation of the embodiment of the present invention, the above step S13 includes:

[0077] First, send a burn request.

[0078] Exemplarily, the burning request may be sent to one burning device, or to multiple burning devices that are communicatively connected to the file system. The method for sending the burning request may be through a wired network or a wireless network. The embodiment of the present invention does not specifically limit the number of burning requests to be sent, and those skilled in the art may set it according to actual conditions.

[0079] Secondly, when a feedback signal is received, the merged file is burned into a burning device corresponding to the feedback signal according to the feedback signal, and the feedback signal includes the burning quantity.

[0080] Exemplarily, when there is only one burning device, the feedback signal is sent by the burning device; when there are multiple burning devices, the feedback signal may be sent by one or more of the burning devices. The method for receiving the feedback signal corresponds to the method for sending the above-mentioned burning request, which will not be repeated here. Burning the merged file to the burning device corresponding to the feedback signal according to the feedback signal may be burning the merged file of the burning quantity in the feedback signal to the corresponding burning device.

[0081] As an optional implementation of the embodiment of the present invention, the small file storage method further includes:

[0082] Execute the corresponding small file storage process according to the current state of the finite state machine.

[0083] Exemplarily, the states of the finite state machine may include "writable [OPEN]", "waiting for writing to be completed [COMMIT-WAIT]", "merging [PREPARE-COMMITTING]", "merging completed [PREPARE-COMMITTING-DONE]", "burning [COMMITTING]", "burning completed [COMMITTED]", etc. The embodiment of the present invention does not specifically limit the states of the finite state machine, and those skilled in the art may set them according to actual conditions.

[0084] The corresponding small file storage process can be executed according to the current state of the finite state machine:

[0085] (1) When the size of the multiple small files to be burned is smaller than the preset writable threshold of the current ZFS partition (for example, 95% of the remaining space), the ZFS partition is in the OPEN state until it is larger than the preset writable threshold, and then the ZFS partition enters the COMMIT_WAIT state. When the size of the multiple small files to be burned is large, the multiple small files to be burned will be uploaded separately, that is, some of the small files to be burned will be written to the current ZFS partition, and the other part of the small files to be burned will be written to the next ZFS partition.

[0086] (2) A ZFS partition in the COMMIT_WAIT state. When the write count of this ZFS partition is equal to zero, the ZFS partition enters the PREPARE_COMMITTING state.

[0087] (3) For a ZFS partition in the PREPARE_COMMITTING state, a merge operation can be performed only when the ZFS partition receives a merge instruction. The ZFS partition can enter the PREPARE_COMMITTING_DONE state only after the merge operation is completed correctly. If no merge instruction is received, the ZFS partition can only wait.

[0088] (4) After the ZFS partition in the PREPARE_COMMITTING_DONE state successfully sends the burn request, the ZFS partition enters the COMMITTING state.

[0089] (5) For a ZFS partition in the COMMITTING state, the burning operation can be performed only when the ZFS partition receives feedback from the burning device. The COMMITTED state can be entered only after the burning operation is completed correctly. If there is no feedback from the burning device, the ZFS partition can only wait.

[0090] (6) A ZFS partition in the COMMITTED state will release small files only when the ZFS partition receives a release command. Each release operation releases a preset number of ZFS partitions in the COMMITTED state based on the creation time.

[0091] The embodiment of the present invention can relatively simply represent the state of a small file through a finite state machine, and can more simply represent the storage process of a small file according to a state migration mechanism.

[0092] The embodiment of the present invention also discloses a small file storage device, such as Figure 2 As shown, including:

[0093] The grouping module 21 is used to group multiple small files to be burned to obtain multiple file segments, each of which includes at least 2 small files; the specific implementation method is shown in the relevant description of step S11 in the above embodiment, which will not be repeated here.

[0094] The merging module 22 is used to merge the small files in each file segment to obtain multiple merged files; the specific implementation method is shown in the relevant description of step S12 in the above embodiment, which will not be repeated here.

[0095] The burning module 23 is used to load a preset burning script to burn multiple merged files into a burning device. The specific implementation method is described in the relevant description of step S13 in the above embodiment, which will not be repeated here.

[0096] The small file storage device provided by the present invention obtains multiple file segments by grouping multiple small files to be burned, each file segment includes at least 2 small files, merges the small files in each file segment respectively to obtain multiple merged files, and loads a preset burning script to burn the multiple merged files into a burning device. The embodiment of the present invention reduces index node metadata and storage operations, improves file storage speed, and also improves file system processing efficiency and operation speed by merging at least 2 small files into one merged file for burning.

[0097] As an optional implementation of the embodiment of the present invention, the small file storage device also includes:

[0098] The first acquisition module is used to obtain the current storage capacity of the file system for storing small files; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0099] The writing module is used to write the small files into the file system for storing small files when the current storage capacity of the file system for storing small files is greater than a preset threshold value, so as to obtain a plurality of small files to be burned. The specific implementation method is described in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0100] As an optional implementation of the embodiment of the present invention, the small file storage device also includes:

[0101] The second acquisition module is used to obtain file attribute information of multiple small files to be burned; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0102] The storage module is used to store the file attribute information of the multiple small files to be recorded in a preset database. The specific implementation method is shown in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0103] As an optional implementation of the embodiment of the present invention, the file attribute information includes: the latest modification time of the file; the above grouping module 21 includes:

[0104] The first sorting module is used to sort the multiple small files to be burned according to the latest modification time of the files; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0105] The grouping submodule is used to group the sorted multiple small files to be recorded to obtain multiple file segments. The specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0106] As an optional implementation of the embodiment of the present invention, the small file storage device also includes:

[0107] The second sorting module is used to sort the small files in each file segment according to the latest modification time of the files in the attribute information; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0108] The merged file obtaining module is used to merge each of the sorted file segments to obtain multiple merged files. The specific implementation method is described in the corresponding steps in the above embodiment, which will not be repeated here.

[0109] As an optional implementation of the embodiment of the present invention, the merging module 22 includes:

[0110] The filling module is used to obtain the file attribute information of the small files of each file segment, fill the file attribute information into the preset file handle, and obtain multiple merged files. The specific implementation method is shown in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0111] As an optional implementation of the embodiment of the present invention, the file attribute information includes the length information of the small file, and the small file storage device further includes:

[0112] The initial position acquisition module is used to obtain the initial position of each small file in the current merged file based on the position of the previous merged file and the length information of each small file in the current merged file; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0113] The associated storage module is used to associate and store the initial position of each small file in the current merged file, the file attribute information of the small file, and the merged file name of the small file in a preset database. The specific implementation method is described in the corresponding steps in the above embodiment, which will not be repeated here.

[0114] As an optional implementation of the embodiment of the present invention, the small file storage device also includes:

[0115] The third acquisition module is used to obtain the attribute information, initial position and merged file of the small file from the preset database according to the small file read request when a small file read request is received. The small file read request includes: the file system where the small file is stored and the name of the merged file of the small file; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiments, which will not be repeated here.

[0116] The reading module is used to read the data content of the small file from the merged file where the small file is located according to the length information and the initial position in the small file attribute information. The specific implementation method is shown in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0117] As an optional implementation of the embodiment of the present invention, the merging module 22 includes:

[0118] The merging submodule is used to merge the small files in the file segment according to the merging instruction to obtain multiple merged files when receiving the merging instruction. The merging instruction includes: the merging quantity. The specific implementation method is described in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0119] As an optional implementation of the embodiment of the present invention, the above-mentioned burning module 23 includes:

[0120] The sending module is used to send a burning request; the specific implementation method can be found in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0121] The burning submodule is used to burn the merged file to the burning device corresponding to the feedback signal according to the feedback signal when receiving the feedback signal, and the feedback signal includes the burning quantity. The specific implementation method is shown in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0122] As an optional implementation of the embodiment of the present invention, the small file storage device also includes:

[0123] The execution module is used to execute the corresponding small file storage process according to the current state of the finite state machine. The specific implementation method is shown in the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0124] The embodiment of the present invention also provides a computer device, such as Figure 3 As shown, the computer device may include a processor 31 and a memory 32, wherein the processor 31 and the memory 32 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0125] The processor 31 may be a central processing unit (CPU). The processor 31 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0126] The memory 32 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the small file storage method in the embodiment of the present invention (for example, Figure 2 The processor 31 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 32, that is, the small file storage method in the above method embodiment is implemented.

[0127] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 31, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The one or more modules are stored in the memory 32 and when executed by the processor 31, perform the following steps: Figure 1 The small file storage method in the illustrated embodiment.

[0129] For details of the above computer equipment, please refer to Figure 1 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0131] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A small file storage method, characterized in that: The steps include: Grouping multiple small files to be burned to obtain multiple file segments, each file segment includes at least 2 small files; Merge the small files in each file segment separately to obtain multiple merged files; Loading a preset burning script to burn the multiple merged files into a burning device; The method further comprises: Execute the corresponding small file storage process according to the current state of the finite state machine; Among them, executing the corresponding small file storage process according to the current state of the finite state machine includes: When the size of multiple small files to be burned is smaller than the preset writable threshold of the current ZFS partition, the ZFS partition is in a writable state. When the size is larger than the preset writable threshold, the ZFS partition enters a state of waiting for writing to be completed. When the write count of a ZFS partition in the waiting-to-write state is equal to zero, it enters the merging state; When a ZFS partition in the merging state receives a merge instruction, it performs the merge operation and enters the merge completed state after the merge operation is completed correctly. If no merge instruction is received, it waits; After the ZFS partition in the merge completed state successfully sends a burn request, it enters the burning state; When the ZFS partition in the burning state receives a feedback signal from the burning device, it starts the burning operation and enters the burning completion state after the burning operation is completed correctly. If there is no feedback signal from the burning device, it waits. When a ZFS partition in a burn-completed state receives a release instruction, the small file is released; each release operation releases a preset number of ZFS partitions in a burn-completed state according to the creation time.

2. The method according to claim 1, characterized in that Before grouping the plurality of small files to be burned, the method further comprises: Get the current storage capacity of the file system used to store small files; When the current storage capacity of the file system for storing small files is greater than a preset threshold, the small files are written into the file system for storing small files to obtain a plurality of small files to be recorded.

3. The method according to claim 1, characterized in that The method further comprises: Obtain file attribute information of multiple small files to be burned; The file attribute information of the plurality of small files to be recorded is stored in a preset database.

4. The method according to claim 3, characterized in that The file attribute information includes: the latest modification time of the file; multiple small files to be burned are grouped to obtain multiple file segments, including: Sorting the plurality of small files to be burned according to the latest modification time of the files; The sorted multiple small files to be recorded are grouped to obtain multiple file segments.

5. The method according to claim 3 or 4, characterized in that: Before merging the small files in each file segment to obtain a plurality of merged files, the method further includes: sorting the small files in each file segment according to the latest modification time of the files in the attribute information; Each sorted file segment is merged separately to obtain multiple merged files.

6. The method according to claim 3, characterized in that The small files in each file segment are merged to obtain multiple merged files, including: The file attribute information of the small files of each file segment is obtained, and the file attribute information is filled into a preset file handle to obtain the multiple merged files.

7. The method according to claim 6, characterized in that The file attribute information includes the length information of the small file, and the method further includes: According to the position of the last merged file and the length information of each small file in the current merged file, the initial position of each small file in the current merged file is obtained; The initial positions of the small files in the current merged file, the file attribute information of the small files and the merged file name of the small files are associated and stored in the preset database.

8. The method according to claim 7, characterized in that The method further comprises: When a small file read request is received, the attribute information of the small file, the initial position, and the merged file of the small file are obtained from the preset database according to the small file read request, wherein the small file read request includes: the file system where the small file is stored and the name of the merged file of the small file; The data content of the small file is read from the merged file where the small file is located according to the length information in the small file attribute information and the initial position.

9. The method according to claim 1, characterized in that: The small files in each file segment are merged to obtain multiple merged files, including: When a merge instruction is received, the small files in the file segment are merged according to the merge instruction to obtain a plurality of merged files, and the merge instruction includes: a merge quantity.

10. The method according to claim 1, characterized in that The step of loading a preset burning script to burn the multiple merged files into a burning device includes: Send a burn request; When a feedback signal is received, the merged file is burned into a burning device corresponding to the feedback signal according to the feedback signal, and the feedback signal includes a burning quantity.

11. A small file storage device, characterized in that: include: A grouping module, used for grouping a plurality of small files to be recorded to obtain a plurality of file segments, each of which includes at least two small files; A merging module is used to merge the small files in each file segment to obtain multiple merged files; A burning module, used for loading a preset burning script to burn the multiple merged files into a burning device; The device also includes a finite state machine module, which is used to: Execute the corresponding small file storage process according to the current state of the finite state machine; The finite state machine module is also used for: When the size of multiple small files to be burned is smaller than the preset writable threshold of the current ZFS partition, the ZFS partition is in a writable state. When the size is larger than the preset writable threshold, the ZFS partition enters a state of waiting for writing to be completed. When the write count of a ZFS partition in the waiting-to-write state is equal to zero, it enters the merging state; When a ZFS partition in the merging state receives a merge instruction, it performs the merge operation and enters the merge completed state after the merge operation is completed correctly. If no merge instruction is received, it waits; After the ZFS partition in the merge completed state successfully sends a burn request, it enters the burning state; When the ZFS partition in the burning state receives a feedback signal from the burning device, it starts the burning operation and enters the burning completion state after the burning operation is completed correctly. If there is no feedback signal from the burning device, it waits. When a ZFS partition in a burn-completed state receives a release instruction, the small file is released; each release operation releases a preset number of ZFS partitions in a burn-completed state according to the creation time.

12. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the steps of the small file storage method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the small file storage method according to any one of claims 1 to 10 are implemented.

Images