File processing method and device, storage medium and electronic device

Abstract

Embodiments of the present application provide a file processing method and device, a storage medium and an electronic device, wherein the method comprises: scanning first file index information in a first merged file to obtain a first scanning result, wherein the first merged file is a merged file obtained by merging a first group of files, and the first file index information is used to indicate whether the first group of files in the first merged file is deleted; in a case where the first scanning result indicates that a first part of files in the first group of files is deleted and the deleted first part of files satisfies a preset first compaction condition, performing a merging operation according to first remaining files to obtain a second merged file, the first remaining files being files other than the first part of files in the first group of files, and the storage positions of the first remaining files in the second merged file being continuous. Through the embodiments of the present application, the problem of low capacity utilization of the storage system in the related art is solved.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and more specifically, to a method, apparatus, storage medium, and electronic device for processing files. Background Technology

[0002] In an era of exploding media information, there are hundreds of billions or even more media images stored, with an average image size of only 15kb. This is defined as the problem of storing massive amounts of small files. Current storage systems are mainly designed for large files. For example, HDFS and Ceph focus on large files in their implementation strategies such as metadata management, data layout, striping design, and cache management. However, for applications with massive amounts of small files, the performance and storage efficiency will be significantly reduced, or even rendered unusable. One approach is to merge small files into large files. However, when a user deletes a small file, there are still other small files in the merged large file that have not been deleted. The merged large file cannot be deleted, thus creating data gaps where deleted small files become data voids, preventing the release of capacity and resulting in a decrease in the storage system's capacity utilization.

[0003] There is currently no effective solution to the problem of low capacity utilization in storage systems in related technologies. Summary of the Invention

[0004] The present invention provides a method, apparatus, storage medium, and electronic device for processing files, so as to at least solve the problem of low capacity utilization of storage systems in related technologies.

[0005] According to an embodiment of the present invention, a file processing method is provided, comprising: scanning first file index information in a first merged file to obtain a first scan result, wherein the first merged file is a merged file obtained by merging a first group of files, wherein if none of the first group of files in the first merged file are deleted, the storage locations of the first group of files in the first merged file are contiguous, and the first file index information is used to indicate whether the first group of files in the first merged file have been deleted; wherein if the first scan result indicates that a first portion of files in the first group of files have been deleted, and the deleted first portion of files satisfies a preset first compaction condition, performing a merge operation based on a first remaining file to obtain a second merged file, wherein the first remaining file is the file in the first group of files excluding the first portion of files, and the storage locations of the first remaining file in the second merged file are contiguous.

[0006] In an exemplary embodiment, before performing a merging operation based on the first remaining file to obtain a second merged file, the method further includes: determining that the deleted first portion of the file satisfies the first compaction condition if the ratio between the number of files in the first portion of the file and the number of files in the first group of files is greater than or equal to a first ratio threshold, and / or if the ratio between the file size of the first portion of the file and the file size of the first group of files is greater than or equal to a second ratio threshold.

[0007] In one exemplary embodiment, the method further includes: when the first file index information includes the sequence number of the first group of files in the first merged file, determining the number of files in the first partial files to be equal to the number of sequence numbers of the first partial files in the first file index information, and determining the number of files in the first group of files to be equal to the number of sequence numbers of the first group of files in the first file index information; and / or when the first file index information also indicates the storage location of the first group of files in the first merged file, determining the file capacity of the first partial files to be equal to the sum of the capacities corresponding to the storage locations of each file in the first partial files indicated by the first file index information, and determining the file capacity of the first group of files to be equal to the capacity corresponding to the storage locations of the first group of files indicated by the first file index information.

[0008] In an exemplary embodiment, the step of performing a merging operation based on the first remaining file to obtain a second merged file includes: adjusting the storage location of the first remaining file in the first merged file to a contiguous storage location, releasing the storage location of the unstored file in the first merged file, and modifying the first file index information to second file index information to obtain the second merged file, wherein the second file index information is used to indicate the storage location of the first remaining file in the second merged file and to indicate whether the first remaining file in the second merged file has been deleted; or merging the first remaining file into the second merged file and deleting the first merged file, wherein the second merged file includes the second file index information.

[0009] In an exemplary embodiment, the method further includes: scanning third file index information in a third merged file to obtain a second scan result, wherein the third merged file is a merged file obtained by merging the second group of files, and if none of the second group of files in the third merged file have been deleted, the storage locations of the second group of files in the third merged file are contiguous, and the third file index information is used to indicate whether the second group of files in the third merged file have been deleted; if the first scan result indicates that a first part of the files in the first group of files has been deleted, and the deleted first part of the files meets a preset first compaction condition, performing a merge operation based on the first remaining files to obtain a second merged file, including: if the first scan result indicates that a first part of the files in the first group of files has been deleted, and the deleted first part of the files meets a third compaction condition, and the second scan result indicates that a second part of the files in the second group of files has been deleted, and the deleted second part of the files meets a preset second compaction condition, and if the file size of the second remaining file is less than the file size of the first part of the files, writing the second remaining file into the first merged file, and in the first merge... The storage locations of the first and second remaining files in the file are adjusted to be consecutive, the storage locations of unstored files in the first merged file are released, and the first file index information is modified to the third file index information to obtain the second merged file. The third file index information indicates the storage location of the first and second remaining files in the second merged file and whether the first and second remaining files in the second merged file have been deleted. The second remaining file is the file in the second group of files excluding the second part of the files. Alternatively, if the file size of the second remaining file is equal to the file size of the first part of the files, the second remaining file is written to the first merged file, the storage locations of the first and second remaining files in the first merged file are adjusted to be consecutive, and the first file index information is modified to the third file index information to obtain the second merged file. Alternatively, the first and second remaining files are merged into the second merged file, and the first and third merged files are deleted. The second merged file includes the third file index information.

[0010] In one exemplary embodiment, the method further includes: determining that the deleted second part of the file satisfies the second compression condition when the ratio between the number of files in the second part of the file and the number of files in the second group of files is greater than or equal to a third ratio threshold, and / or when the ratio between the file size of the second part of the file and the file size of the second group of files is greater than or equal to a fourth ratio threshold.

[0011] In an exemplary embodiment, the step of performing a merging operation based on the first remaining file to obtain a second merged file includes: performing a merging operation based on the first remaining file and N remaining files to obtain the second merged file, wherein N is equal to 1 or is a positive integer greater than or equal to 2, and the N remaining files are used to represent the files remaining in each of the N merged files after deleting some files.

[0012] According to another embodiment of the present invention, a file processing apparatus is also provided, comprising: a first scanning module, configured to scan first file index information in a first merged file to obtain a first scanning result, wherein the first merged file is a merged file obtained by merging a first group of files, and the storage locations of the first group of files in the first merged file are contiguous when none of the first group of files in the first merged file have been deleted, and the first file index information is used to indicate whether the first group of files in the first merged file has been deleted; and a processing module, configured to perform a merging operation based on a first remaining file to obtain a second merged file when the first scanning result indicates that a first portion of files in the first group of files has been deleted and the deleted first portion of files satisfies a preset first compression condition, wherein the first remaining file is the files in the first group of files other than the first portion of files, and the storage locations of the first remaining files in the second merged file are contiguous.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0014] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This invention obtains a first scan result by scanning the first file index information in a first merged file. The first merged file is a merged file obtained by merging a first group of files. When none of the first group of files in the first merged file have been deleted, the storage locations of the first group of files in the first merged file are contiguous. The first file index information indicates whether the first group of files in the first merged file has been deleted. When the first scan result indicates that a first portion of files in the first group of files has been deleted, and the deleted first portion of files meets a preset first compaction condition, a merge operation is performed based on the first remaining files to obtain a second merged file. The first remaining files in the second merged file are stored contiguously. That is, after deleting a first portion of files in the first merged file, the first remaining files are merged into the second merged file and stored contiguously in the second merged file. This achieves the purpose of releasing the space occupied by the data holes created by deleting the first portion of files, avoiding the problem in related technologies where deleting small files in a large merged file creates data holes, preventing capacity release and causing a decrease in storage system capacity utilization. Therefore, this invention solves the problem of low storage system capacity utilization in related technologies and achieves the effect of improving storage system capacity utilization. Attached Figure Description

[0016] Figure 1 This is a block diagram of the mobile terminal hardware structure of the file processing method according to an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of a document processing method according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of the merged file according to an embodiment of the present invention;

[0019] Figure 4 This is an example diagram of file merging according to an embodiment of the present invention;

[0020] Figure 5 This is an overall flowchart of the document compression method according to an embodiment of the present invention;

[0021] Figure 6 This is an example diagram of an index bitmap according to an embodiment of the present invention;

[0022] Figure 7 This is an example of file compression according to an embodiment of the present invention. Figure 1

[0023] Figure 8 This is an example of file compression according to an embodiment of the present invention. Figure 2 ;

[0024] Figure 9 This is a structural block diagram of a document processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0027] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a block diagram of the mobile terminal hardware structure of the file processing method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the file processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0030] This embodiment provides a file processing method. Figure 2 This is a flowchart of a file processing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0031] Step S202: Scan the first file index information in the first merged file to obtain the first scan result. The first merged file is a merged file obtained by merging the first group of files. If none of the first group of files in the first merged file are deleted, the storage locations of the first group of files in the first merged file are continuous. The first file index information is used to indicate whether the first group of files in the first merged file have been deleted.

[0032] Step S204: If the first scan result indicates that the first part of the files in the first group of files has been deleted and the deleted first part of the files meets the preset first compression condition, a merging operation is performed based on the first remaining files to obtain a second merged file. The first remaining files are the files in the first group of files other than the first part of the files, and the storage locations of the first remaining files in the second merged file are contiguous.

[0033] Through the above steps, a first scan result is obtained by scanning the first file index information in the first merged file. The first merged file is the merged file obtained by merging the first group of files. When none of the first group of files in the first merged file have been deleted, the storage location of the first group of files in the first merged file is contiguous. The first file index information is used to indicate whether the first group of files in the first merged file has been deleted. When the first scan result indicates that the first part of the files in the first group of files has been deleted, and the deleted first part of the files meets the preset first compaction condition, a merge operation is performed based on the first remaining files to obtain a second merged file. The storage location of the first remaining files in the second merged file is contiguous. That is, after deleting the first part of the files in the first merged file, the first remaining files are merged into the second merged file and stored contiguously in the second merged file. This achieves the purpose of releasing the space capacity occupied by the data holes created by deleting the first part of the files, avoiding the problem in related technologies where, when deleting small files in a merged large file, the deleted small files in the merged large file become data holes, resulting in the inability to release capacity and thus causing a decrease in the capacity utilization of the storage system. Therefore, this solves the problem of low capacity utilization of the storage system in related technologies and achieves the effect of improving the capacity utilization of the storage system.

[0034] The entity performing the above steps may be a storage system management device, or a controller or processor in the storage management device, or an application control program in the device, or a processor with human-computer interaction capabilities configured on the storage device, or a processing device or processing unit with similar processing capabilities, but is not limited to these.

[0035] In the above embodiments, scanning the first file index information in the first merged file yields a first scan result. The first merged file is the merged file obtained by merging the first group of files. If none of the first group of files in the first merged file has been deleted, the storage locations of the first group of files in the first merged file are contiguous. The first file index information is used to indicate whether the first group of files in the first merged file has been deleted. (This is in conjunction with the appendix...) Figure 3 To explain, Figure 3This is a schematic diagram of the structure of a merged file according to an embodiment of the present invention. The merged file (such as the first merged file mentioned above) includes an index bitmap and a data area. The data area is used to store multiple small files. For example, the first group of files mentioned above consists of multiple small files. When the first group of files is not deleted in the first merged file, the first group of files are stored consecutively in the first merged file. The index bitmap (or index bitmap area, or bitmap index area) records the status of each file in the first group of files, such as whether each file has been deleted. In practical applications, the index bitmap can also record the sequence number of each file (such as a small file) included in the first merged file, and / or the storage position of each file in the first merged file, such as the start and end positions of each file in the merged large file. When the first scan result indicates that the first part of the files in the first group of files has been deleted, and the deleted first part of the files meets the preset first compaction condition, a merge operation is performed according to the first remaining files to obtain a second merged file. The first remaining files mentioned above are the files in the first group of files other than the first part of the files. The storage position of the first remaining files in the second merged file is consecutive. Figure 4 To explain, Figure 4 This is an example diagram of file merging according to an embodiment of the present invention. Figure 4 Data hole portion (corresponding) Figure 4 The blank portions in the first merged file are generated by deleting parts of the file (such as the first part of the file mentioned above). In practical applications, this can be achieved by scanning the index information of the first file (corresponding to...). Figure 4 The system first indexes the first part of the file to determine which files (or small files) in the first merged file are in a deleted state. Then, it checks whether the deleted first part of the file meets a preset first compaction condition. If the first compaction condition is met, a merge operation is performed on the first remaining files. For example, the first compaction condition is when the ratio of the number of files in the first part of the file to the number of files in the first group of files exceeds a certain threshold (e.g., 15%, 20%, or other values); or, the first compaction condition is when the ratio of the file size of the first part of the file to the file size of the first group of files (or the total size of the first merged file) exceeds a certain threshold (e.g., 20%, 30%, or other values). In this case, a merge operation is performed on the first remaining files (corresponding to...). Figure 4 The files in the first merged file are compressed to obtain a second merged file, and the storage locations of the first remaining files in the second merged file are contiguous, such as... Figure 4 As shown, the second merged file obtained after merging the files is from... Figure 4As can be seen, the data in the second merged file is continuous. That is, after deleting the first part of the file in the first merged file, the remaining first part is merged into the second merged file and stored continuously within it. This achieves the goal of releasing the space occupied by the data gaps created by deleting the first part of the file, avoiding the problem in related technologies where deleting small files from a large merged file creates data gaps, preventing capacity release and thus reducing the storage system's capacity utilization. Therefore, this solves the problem of low storage system capacity utilization in related technologies, achieving the effect of improving the storage system's capacity utilization.

[0036] In an optional embodiment, before performing a merging operation on the first remaining files to obtain a second merged file, the method further includes: determining that the deleted first portion of files satisfies the first compaction condition if the ratio of the number of files in the first portion of files to the number of files in the first group of files is greater than or equal to a first ratio threshold, and / or if the ratio of the file size of the first portion of files to the file size of the first group of files is greater than or equal to a second ratio threshold. In this embodiment, when the ratio of the number of files in the first portion of files to the number of files in the first group of files is greater than or equal to the first ratio threshold (e.g., 15%, 20%, or other values), and / or the ratio of the file size of the first portion of files to the file size of the first group of files is greater than or equal to the second ratio threshold (e.g., 20%, 30%, or other values), it can be determined that the deleted first portion of files satisfies the first compaction condition. At this time, a merging operation can be performed on the first remaining files to obtain a second merged file. Through this embodiment, the purpose of determining whether the deleted first portion of files in the first merged file satisfies the first compaction condition is achieved.

[0037] In an optional embodiment, the method further includes: when the first file index information includes the sequence number of the first group of files in the first merged file, determining the number of files in the first partial files to be equal to the number of sequence numbers of the first partial files in the first file index information, and determining the number of files in the first group of files to be equal to the number of sequence numbers of the first group of files in the first file index information; and / or when the first file index information also indicates the storage location of the first group of files in the first merged file, determining the file capacity of the first partial files to be equal to the sum of the capacities corresponding to the storage locations of each file in the first partial files indicated by the first file index information, and determining the file capacity of the first group of files to be equal to the capacity corresponding to the storage locations of the first group of files indicated by the first file index information. In this embodiment, when the first file index information includes the sequence numbers of the first group of files in the first merged file, the number of files in the first part of the file can be determined as the number of sequence numbers of the first part of the file in the first file index information. Each file (or small file) in the first part of the file corresponds to a different sequence number. For example, if the small files included in the first part of the file correspond to a total of X different sequence numbers, then the number of files in the first part of the file can be determined as X. If the total number of sequence numbers of the first group of files included in the first file index information is Y, then the number of files in the first group of files can be determined as Y. And / or, when the first file index information includes the storage location of the first group of files in the first merged file, the file capacity of the first part of the file can be determined as equal to the sum of the capacities corresponding to the storage locations of each file in the first part of the file represented by the first file index information. For example, if the first part of the file includes X small files, the capacity of each small file can be determined according to the storage location of each small file in the first merged file, for example, M1, M2...M X In practical applications, the storage location can include a start storage location and an end storage location. This allows us to determine the file size of the first part of the file, which is equal to the sum of the sizes of the X smaller files. Similarly, the size of the first group of files can be determined based on its storage location. This embodiment achieves the purpose of determining the number of files in the first part of the file based on the first file index information, and / or determining the file size of the first part of the file based on the first file index information.

[0038] In an optional embodiment, the step of performing a merging operation based on the first remaining files to obtain a second merged file includes: adjusting the storage location of the first remaining files in the first merged file to a continuous storage location, releasing the storage location of unstored files in the first merged file, and modifying the first file index information to second file index information to obtain the second merged file. The second file index information indicates the storage location of the first remaining files in the second merged file and indicates whether the first remaining files in the second merged file have been deleted. Alternatively, the first remaining files are merged into the second merged file, and the first merged file is deleted. The second merged file includes the second file index information. In this embodiment, performing a merging operation based on the first remaining files to obtain a second merged file includes: adjusting the storage location of the first remaining files in the first merged file to a continuous storage location, releasing the storage location of unstored files in the first merged file, and simultaneously modifying the first file index information to second file index information. That is, according to the adjusted storage location of the first remaining files in the merged file, the index information (such as sequence number, storage start position, storage end position, etc.) corresponding to each file (or small file) in the first remaining files is modified accordingly to obtain the second merged file. Figure 3 As shown, the first remaining files in the first merge file are merged. For example, the first remaining files in the first merge file are moved to the locations where there is gap data. For instance, small, undeleted files whose file numbers are adjacent to and follow the gap data can be moved to the location of the gap data. This process is repeated until the storage locations of files not stored in the original first merge file are released, resulting in the second merge file. Figure 3 The second merged file differs from the first merged file; alternatively, the first remaining files can be merged into the second merged file while deleting the original first merged file. For example, a new large file can be created, and the data of the first remaining files in the original first merged file can be read sequentially and written into the new large file to obtain the second merged file. Similarly, the second merged file also includes the corresponding second file index information. Through this embodiment, the purpose of merging the first remaining files in the first merged file in different ways is achieved, while solving the problem of unreleased capacity caused by gaps in the original data.

[0039] In an optional embodiment, the method further includes: scanning the third file index information in the third merged file to obtain a second scan result, wherein the third merged file is a merged file obtained by merging the second group of files, and if none of the second group of files in the third merged file have been deleted, the storage locations of the second group of files in the third merged file are contiguous, and the third file index information is used to indicate whether the second group of files in the third merged file have been deleted; if the first scan result indicates that the first part of the files in the first group of files has been deleted, and the deleted first part of the files meets a preset first compaction condition, performing a merge operation based on the first remaining files to obtain a second merged file, including: if the first scan result indicates that the first part of the files in the first group of files has been deleted, and the deleted first part of the files meets a third compaction condition, and the second scan result indicates that the second part of the files in the second group of files has been deleted, and the deleted second part of the files meets a preset second compaction condition, and if the file size of the second remaining file is less than the file size of the first part of the files, writing the second remaining file into the first merged file, and in the first merge... The storage locations of the first and second remaining files in the file are adjusted to be consecutive, the storage locations of unstored files in the first merged file are released, and the first file index information is modified to the third file index information to obtain the second merged file. The third file index information indicates the storage location of the first and second remaining files in the second merged file and whether the first and second remaining files in the second merged file have been deleted. The second remaining file is the file in the second group of files excluding the second part of the files. Alternatively, if the file size of the second remaining file is equal to the file size of the first part of the files, the second remaining file is written to the first merged file, the storage locations of the first and second remaining files in the first merged file are adjusted to be consecutive, and the first file index information is modified to the third file index information to obtain the second merged file. Alternatively, the first and second remaining files are merged into the second merged file, and the first and third merged files are deleted. The second merged file includes the third file index information.In practical applications, a second scan result can be obtained by scanning the third file index information of the third merged file. The third file index information indicates which files in the second group of files have been deleted. Then, based on the first and second scan results, it is determined whether the first part of the deleted files in the first group of files meets the third compaction condition, and whether the second part of the deleted files in the second group of files meets the second compaction condition. When it is determined that the first part of the deleted files in the first group of files meets the third compaction condition, and the second part of the deleted files in the second group of files meets the second compaction condition, optionally, a merge operation can be performed on the first remaining files in different ways to obtain the second merged file. For example, when the file size of the second remaining file is less than the file size of the first part of the file, that is, when the file size of the third merged file after deleting the second part of the file is less than the file size of the first part of the deleted files in the first merged file, the second remaining file can be written into the first merged file. At this time, the first merged file includes the first remaining file and the second remaining file. At the same time, the storage positions of the first remaining file and the second remaining file in the first merged file are adjusted to consecutive storage positions, and the unstored files in the first merged file are released. The storage location of the file is adjusted accordingly, and the first file index information is modified to the third file index information. This avoids the large workload caused by creating a new large file (or a new merged file) and writing the first remaining file into a new large file. Optionally, when the file size of the second remaining file is equal to the file size of the first part of the file, the second remaining file can be directly written into the first merged file, and the storage locations of the first and second remaining files are adjusted to be contiguous. Similarly, the first file index information is modified to the third file index information, resulting in the second merged file. The second merged file obtained in this way does not have the problem of unreleased capacity due to data gaps. Optionally, the first and second remaining files can be merged into a second merged file, and the first and third merged files can be deleted. For example, a new large file can be created, and the data of the first remaining file in the first merged file can be read and written into the new large file. Similarly, the data of the second remaining file in the third merged file can be read and written into the new large file to obtain the second merged file. At the same time, the first and third merged files are deleted. Similarly, the second merged file also includes the corresponding third file index information. This embodiment achieves the goal of merging the remaining files in multiple merged files by combining the scan results of multiple merged files. It is applicable to scenarios where there are gaps caused by deleting a large number of small files. It can combine multiple merged files and compress them into a larger file, thereby solving the problem of unreleased capacity caused by data gaps in multiple merged files and improving the capacity utilization of the storage system.

[0040] In an optional embodiment, the method further includes: determining that the deleted second part of the file satisfies the second compaction condition when the ratio between the number of files in the second part of the file and the number of files in the second group of files is greater than or equal to a third ratio threshold, and / or when the ratio between the file size of the second part of the file and the file size of the second group of files is greater than or equal to a fourth ratio threshold. In this embodiment, when the ratio of the number of files in the second part of the file to the number of files in the second group of files is greater than or equal to a third ratio threshold (e.g., 50%, 60%, or other values), and / or the ratio of the file size of the second part of the file to the file size of the second group of files is greater than or equal to a fourth ratio threshold (e.g., 55%, 60%, or other values), it can be determined that the deleted second part of the file meets the second compaction condition. The third compaction condition can also be: when the ratio of the number of files in the first part of the file to the number of files in the first group of files is greater than or equal to a fifth ratio threshold (e.g., 50%, 60%, or other values), and / or the ratio of the file size of the first part of the file to the file size of the first group of files is greater than or equal to a sixth ratio threshold (e.g., 55%, 60%, or other values), when it is determined that the second and third compaction conditions are met, a merging operation can be performed on the first remaining file and the second remaining file, i.e., the purpose of joint compaction processing can be achieved. This embodiment achieves the purpose of determining whether the second part of the file deleted in the third merged file meets the second compaction condition, and the purpose of determining whether the first part of the file deleted in the first merged file meets the third compaction condition.

[0041] In an optional embodiment, the step of performing a merging operation based on the first remaining file to obtain a second merged file includes: performing a merging operation based on the first remaining file and N remaining files to obtain the second merged file, where N is equal to 1 or a positive integer greater than or equal to 2, and the N remaining files represent the files remaining in each of the N merged files after deleting some files. Optionally, in practical applications, the merging operation can also be performed on the remaining files in more merged files. This is applicable to application scenarios where there are gaps caused by deleting a large number of small files. For example, if multiple merged files have data gaps, the multiple merged files can be combined and compressed into a larger file to release the capacity occupied by the data gaps in the multiple merged files, thereby improving the capacity utilization of the storage system.

[0042] In an optional embodiment, the step of performing a merge operation on the first remaining file to obtain a second merged file includes: performing a merge operation on the first remaining file and a fourth merged file to obtain the second merged file, wherein the fourth merged file is a merged file obtained by merging a third group of files. In practical applications, a merge operation can also be performed on the first remaining file and the fourth merged file, where the fourth merged file is a merged file obtained by merging a third group of files. For example, if the fourth merged file is a large file that has not been deleted and the data in the fourth merged file is continuous, the first remaining file can also be merged with the fourth merged file.

[0043] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. This application's embodiments solve the problem of data gaps and wasted capacity caused by deleting small files in merged files through a file compression technique. The present invention will be specifically described below with reference to embodiments.

[0044] Figure 5 This is an overall flowchart of a file compression method according to an embodiment of the present invention, the process including:

[0045] S502, small files are written to the merged large file (corresponding to the aforementioned first merged file), and a small file index bitmap is created in the metadata area of ​​the header of the merged large file;

[0046] S504, determine whether the contraction condition is met (corresponding to the aforementioned first contraction condition);

[0047] S506 If the compaction condition is met, perform compaction processing on the original merged large file to obtain a compacted large file, and at the same time update the small file index bitmap in the metadata area of ​​the compacted large file.

[0048] If the judgment result of step S504 is not met, the compaction process will not be performed, and the process will return to step S502 to continue waiting for the index bitmap of the original merged large file to be scanned, and then the compaction condition will be judged based on the scan result.

[0049] The following is a detailed explanation of the process for the above file compression method:

[0050] S1. Merge the header metadata area of ​​the large file (corresponding to the first merged file mentioned above) to create a small file index bitmap;

[0051] S2. Small files are written to the merged large file. The index bitmap records the position index of the small file in the large file and sets its status to "used".

[0052] S3, after deleting a small file, update the corresponding index bitmap to the deleted state;

[0053] S4, periodically scan the index bitmap of large files to determine whether the file compaction condition is met (corresponding to the first compaction condition mentioned above);

[0054] S5, if the file compression condition is met, generate a new large file according to step S1, traverse the index bitmap to read valid data from the large file to be compressed and write it into the new large file (as in the aforementioned second merged file) according to step S2. The data in the new large file is continuous and there are no gaps. Delete the old large file to release capacity. The large file to be compressed is equivalent to the large file after deleting the first part of the aforementioned first merged file.

[0055] S6 periodically scans the index bitmap of large files to determine whether the conditions for multi-file joint compaction are met.

[0056] S7. If the multi-file compression condition is met, generate a new large file according to step S1, traverse the index bitmap to read valid data from multiple files to be compressed, and write the data into the new large file according to step S2. The data in the new large file is continuous and there are no gaps. Delete the old file to release capacity.

[0057] For the layout design of large file data space, such as Figure 3 As shown, the large file data space is divided into a data area and an index area; the index bitmap records the logical positional relationship of small files within the large file and their deletion status.

[0058] The design of the index bitmap is as follows Figure 6 As shown, the contents of the index bitmap are explained below:

[0059] 1) In business operations, the average size of small files is N, and the size of large files is M, then the bitmap length is M / N;

[0060] 2) Each bitmap item includes a sequence number (seq), whether it has been deleted (delete), and the start (bpos) and end (epos) positions of small files within the larger file;

[0061] 3) bit Figure 4 If the data type of an item is determined, then the space size for the data type of each item is S;

[0062] 4) The calculated index area size is (M / N)*S. When creating and merging large files, a corresponding bitmap index area is established.

[0063] 5) Deleting a small file will set the "delete" status under the corresponding bitmap item to True. Specific Implementation Example 1

[0065] Figure 7 This is an example of file compression according to an embodiment of the present invention. Figure 1 , combined Figure 7The following is a description of this embodiment:

[0066] 1.1 Assuming the bitmap length is L1, a large file (corresponding to...) Figure 7 The size of the first merged large file is F1;

[0067] 1.2, Scanning bitmap (corresponding) Figure 7 In the first index area, the number of deleted small files is C, and the sum of the sizes of the deleted files is D;

[0068] 1.3 When C / L1>=R1 (corresponding to the aforementioned first proportion threshold), that is, when the number of small files deleted reaches a certain proportion, the compression condition is met (corresponding to the aforementioned first compression condition);

[0069] 1.4 When D / F1>=R1 (or R1', corresponding to the aforementioned second ratio threshold), that is, the compression condition is met after deleting small files to a certain ratio.

[0070] 1.5. If either of the conditions in 1.3 or 1.4 above is met, the contraction condition is met and contraction is initiated.

[0071] The above-mentioned compaction is equivalent to the aforementioned merging operation. In practical applications, R1 can be a small percentage (such as 15%, 20%, or other values) to solve the problem of empty spaces after deleting a small number of small files. The size of the large file after compaction is still a relatively large file. Specific Implementation Example 2

[0073] Figure 8 This is an example of file compression according to an embodiment of the present invention. Figure 2 , combined Figure 8 The following is a description of this embodiment:

[0074] 2.1 Assume the bitmap length is L2 and the large file size is F2;

[0075] 2.2 Scan the bitmap of large file A (corresponding to the aforementioned first merged file) (corresponding to...) Figure 8 In the first index area, the number of deleted small files N1 is counted, and the sum of the sizes of the deleted files is D1;

[0076] 2.3 Scan the bitmap of large file B (corresponding to the aforementioned third merged file). Figure 8 In the third index area, the number of deleted small files N2 is calculated, and the sum of the sizes of the deleted files is D2;

[0077] 2.4 When N1 / L2>=R2 (corresponding to the fifth proportional threshold mentioned above), that is, after the number of small files deleted reaches a certain proportion, the joint compression condition is met (corresponding to the third compression condition mentioned above);

[0078] Or D1 / F2>=R2 (or R2' corresponds to the aforementioned sixth proportional threshold), that is, after deleting small files to a certain proportion, the joint compression condition is met (corresponding to the aforementioned third compression condition);

[0079] 2.5 When N2 / L2>=R2 (corresponding to the aforementioned third proportional threshold), that is, after deleting a certain proportion of small files, the joint compression condition is met (corresponding to the aforementioned second compression condition);

[0080] Or D2 / F2>=R2 (or R2', corresponding to the aforementioned fourth proportional threshold), that is, after deleting small files to a certain proportion, the joint compression condition is met (corresponding to the aforementioned second compression condition);

[0081] 2.6 If conditions 2.4 and 2.5 above are met, then large files A and B will initiate joint compaction to obtain a merged large file C (corresponding to the aforementioned second merged file).

[0082] The R2 mentioned above can be a relatively large proportion, and it addresses the problem of empty spaces after deleting a large number of small files, requiring multiple files to be combined to compress into a larger file.

[0083] Through the above embodiments, logical disk groups are automatically set according to the hard disk file system type, and storage disk groups are automatically selected according to the bitstream type. This realizes a method for quickly setting up logical disk groups. A logical disk group consists of multiple physical disks with the same file system type. According to different business needs, disk group members are no longer manually or randomly selected, but are automatically added to different logical disk groups according to the file system type of different physical disks, so as to efficiently meet business needs.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0085] This embodiment also provides a file processing device. Figure 9 This is a structural block diagram of a file processing apparatus according to an embodiment of the present invention, such as... Figure 9 As shown, the device includes:

[0086] The first scanning module 902 is used to scan the first file index information in the first merged file to obtain the first scan result. The first merged file is a merged file obtained by merging the first group of files. If none of the first group of files in the first merged file are deleted, the storage locations of the first group of files in the first merged file are continuous. The first file index information is used to indicate whether the first group of files in the first merged file have been deleted.

[0087] The processing module 904 is configured to perform a merging operation based on the first remaining files to obtain a second merged file when the first scan result indicates that the first part of the files in the first group of files has been deleted and the deleted first part of the files meets the preset first compression condition. The first remaining files are the files in the first group of files other than the first part of the files, and the storage locations of the first remaining files in the second merged file are contiguous.

[0088] In an optional embodiment, the above apparatus further includes: a first determining module, configured to determine that the deleted first part of the file satisfies the first compaction condition if, before performing a merging operation based on the first remaining file to obtain a second merged file, the ratio between the number of files in the first part of the file and the number of files in the first group of files is greater than or equal to a first ratio threshold, and / or the ratio between the file size of the first part of the file and the file size of the first group of files is greater than or equal to a second ratio threshold.

[0089] In an optional embodiment, the apparatus further includes: a second determining module, configured to determine the number of files in the first partial files to be equal to the number of the first partial file sequence numbers in the first file index information, and to determine the number of files in the first group files to be equal to the number of the first group file sequence numbers in the first file index information, when the first file index information includes the sequence numbers of the first group files in the first merged file; and / or a third determining module, configured to determine the file capacity of the first partial files to be equal to the sum of the capacities corresponding to the storage locations of the first partial files indicated by the first file index information, and to determine the file capacity of the first group files to be equal to the capacity corresponding to the storage locations of the first group files indicated by the first file index information, when the first file index information further indicates the storage locations of the first group files in the first merged file.

[0090] In an optional embodiment, the processing module 904 includes: a first processing unit, configured to adjust the storage location of the first remaining file in the first merged file to a continuous storage location, release the storage location of the unstored file in the first merged file, and modify the first file index information to second file index information to obtain the second merged file, wherein the second file index information is used to indicate the storage location of the first remaining file in the second merged file and to indicate whether the first remaining file in the second merged file has been deleted; or a second processing unit, configured to merge the first remaining file into the second merged file and delete the first merged file, wherein the second merged file includes the second file index information.

[0091] In an optional embodiment, the above apparatus further includes: a second scanning module, configured to scan the third file index information in the third merged file to obtain a second scanning result, wherein the third merged file is a merged file obtained by merging the second group of files, and when none of the second group of files in the third merged file have been deleted, the storage locations of the second group of files in the third merged file are contiguous, and the third file index information is used to indicate whether the second group of files in the third merged file have been deleted; the above processing module 904 includes: a third processing unit, configured to, when the first scanning result indicates that the first part of the files in the first group of files has been deleted and the deleted first part of the files satisfies a third compression condition, and the second scanning result indicates that the second part of the files in the second group of files has been deleted and the deleted second part of the files satisfies a preset second compression condition, and when the file size of the second remaining file is less than the file size of the first part of the files, write the second remaining file into the first merged file, and adjust the storage locations of the first remaining file and the second remaining file in the first merged file to be contiguous, release The system can be configured to: 1) store the unstored files in the first merged file and modify the first file index information to the third file index information to obtain the second merged file; 2) place the unstored files in the first merged file and modify the first file index information to the third file index information to obtain the second merged file; 3) merge the first and second remaining files in the first merged file to the storage location of the unstored files in the first merged file and modify the first file index information to the third file index information to obtain the second merged file; or 4) merge the first and second remaining files into the second merged file and delete the first and third merged files, wherein the second merged file includes the third file index information.

[0092] In an optional embodiment, the above apparatus further includes: a third determining module, configured to determine that the deleted second part of the file satisfies the second compression condition when the ratio between the number of files in the second part of the file and the number of files in the second group of files is greater than or equal to a third ratio threshold, and / or when the ratio between the file size of the second part of the file and the file size of the second group of files is greater than or equal to a fourth ratio threshold.

[0093] In an optional embodiment, the processing module 904 includes: a sixth processing unit, configured to perform a merging operation based on the first remaining file and N remaining files to obtain the second merged file, wherein N is equal to 1 or is a positive integer greater than or equal to 2, and the N remaining files represent the files remaining in each of the N merged files after deleting some files.

[0094] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0095] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0096] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0097] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0098] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0099] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0100] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing files, characterized in that, include: Scan the first file index information in the first merged file to obtain the first scan result. The first merged file is a merged file obtained by merging the first group of files. If none of the first group of files in the first merged file are deleted, the storage locations of the first group of files in the first merged file are continuous. The first file index information is used to indicate whether the first group of files in the first merged file has been deleted. The first file index information is also used to indicate the sequence number of the first group of files in the first merged file and / or the storage location of the first group of files. If the first scan result indicates that the first part of the files in the first group of files has been deleted and the deleted first part of the files meets the preset first compression condition, a merging operation is performed based on the first remaining files to obtain a second merged file. The first remaining files are the files in the first group of files other than the first part of the files, and the storage locations of the first remaining files in the second merged file are contiguous. If the first scan result indicates that a first portion of files in the first group of files has been deleted and the deleted first portion of files meets a preset first compaction condition, a merging operation is performed based on the first remaining files to obtain a second merged file. This includes: if the first scan result indicates that a first portion of files in the first group of files has been deleted and the deleted first portion of files meets a third compaction condition, and the second scan result indicates that a second portion of files in the second group of files has been deleted and the deleted second portion of files meets a preset second compaction condition, a merging operation is performed based on the first remaining files to obtain a second merged file. The second scan result is obtained by scanning the third file index information in the third merged file. The third merged file is a merged file obtained by merging the second group of files. If none of the second group of files in the third merged file have been deleted, the storage locations of the second group of files in the third merged file are contiguous. The third file index information is used to indicate whether the second group of files in the third merged file have been deleted.

2. The method according to claim 1, characterized in that, Before performing a merge operation based on the first remaining file to obtain a second merged file, the method further includes: If the ratio of the number of files in the first part of the files to the number of files in the first group of files is greater than or equal to a first ratio threshold, and / or if the ratio of the file size of the first part of the files to the file size of the first group of files is greater than or equal to a second ratio threshold, it is determined that the deleted first part of the files meets the first compaction condition.

3. The method according to claim 2, characterized in that, The method further includes: When the first file index information includes the sequence number of the first group of files in the first merged file, the number of files in the first partial file is determined to be equal to the number of sequence numbers of the first partial files in the first file index information, and the number of files in the first group file is determined to be equal to the number of sequence numbers of the first group of files in the first file index information; and / or When the first file index information also indicates the storage location of the first group of files in the first merged file, the file capacity of the first part of the files is determined to be equal to the sum of the capacities corresponding to the storage locations of each file in the first part of the files as indicated by the first file index information, and the file capacity of the first group of files is determined to be equal to the capacity corresponding to the storage locations of the first group of files as indicated by the first file index information.

4. The method according to claim 1, characterized in that, The step of performing a merge operation based on the first remaining file to obtain a second merged file includes: The storage locations of the first remaining files in the first merged file are adjusted to consecutive storage locations. Storage locations where no files are stored in the first merged file are released. The first file index information is modified to second file index information to obtain the second merged file. The second file index information indicates the storage location of the first remaining files in the second merged file and whether the first remaining files in the second merged file have been deleted. Or... The first remaining file is merged into the second merged file, and the first merged file is deleted, wherein the second merged file includes the index information of the second file.

5. The method according to claim 1, characterized in that, The method further includes: If the first scan result indicates that the first portion of files in the first group of files has been deleted, and the deleted first portion of files satisfies the third compaction condition, and the second scan result indicates that the second portion of files in the second group of files has been deleted, and the deleted second portion of files satisfies the preset second compaction condition, then a merging operation is performed based on the first remaining files to obtain the second merged file, including: If the file size of the second remaining file is less than the file size of the first part of the file, the second remaining file is written into the first merged file. The storage locations of the first and second remaining files in the first merged file are adjusted to be contiguous. The storage locations of unstored files in the first merged file are released, and the first file index information is modified to third file index information to obtain the second merged file. The third file index information indicates the storage locations of the first and second remaining files in the second merged file and indicates whether the first and second remaining files in the second merged file have been deleted. The second remaining file is the file in the second group of files excluding the second part of the file. Or... If the file size of the second remaining file is equal to the file size of the first part of the file, the second remaining file is written into the first merged file. The storage locations of the first remaining file and the second remaining file in the first merged file are adjusted to be contiguous, and the first file index information is modified to the third file index information to obtain the second merged file; or The first remaining file and the second remaining file are merged into the second merged file, and the first merged file and the third merged file are deleted, wherein the second merged file includes the index information of the third file.

6. The method according to claim 5, characterized in that, The method further includes: If the ratio of the number of files in the second part of the file to the number of files in the second group of the file is greater than or equal to a third ratio threshold, and / or if the ratio of the file size of the second part of the file to the file size of the second group of the file is greater than or equal to a fourth ratio threshold, it is determined that the deleted second part of the file satisfies the second compaction condition.

7. The method according to claim 1, characterized in that, The step of performing a merge operation based on the first remaining file to obtain a second merged file includes: A merge operation is performed based on the first remaining file and N remaining files to obtain the second merged file, where N is equal to 1 or a positive integer greater than or equal to 2, and the N remaining files are used to represent the files remaining in each of the N merged files after deleting some files.

8. A document processing apparatus, characterized in that, include: The first scanning module is used to scan the first file index information in the first merged file to obtain the first scan result. The first merged file is a merged file obtained by merging the first group of files. If none of the first group of files in the first merged file are deleted, the storage locations of the first group of files in the first merged file are continuous. The first file index information is used to indicate whether the first group of files in the first merged file has been deleted. The first file index information is also used to indicate the sequence number of the first group of files in the first merged file and / or the storage location of the first group of files. The processing module is configured to perform a merging operation on the first remaining files to obtain a second merged file when the first scan result indicates that the first part of the files in the first group of files has been deleted and the deleted first part of the files meets the preset first compression condition. The first remaining files are the files in the first group of files other than the first part of the files, and the storage locations of the first remaining files in the second merged file are contiguous. The processing module performs a merging operation on the first remaining files to obtain a second merged file when the first scan result indicates that a first portion of the files in the first group of files has been deleted and the deleted first portion of files meets a preset first compaction condition. Conversely, when the first scan result indicates that a first portion of the files in the first group of files has been deleted and the deleted first portion of files meets a third compaction condition, and the second scan result indicates that a second portion of the files in the second group of files has been deleted and the deleted second portion of files meets a preset second compaction condition, a merging operation is performed on the first remaining files to obtain a second merged file. The second scan result is obtained by scanning the third file index information in the third merged file. The third merged file is a merged file obtained by merging the second group of files. If none of the second group of files in the third merged file has been deleted, the storage locations of the second group of files in the third merged file are contiguous. The third file index information is used to indicate whether the second group of files in the third merged file has been deleted.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

Images