File storage method, device and equipment and readable storage medium

Abstract

The application provides a file storage method, device and equipment and a readable storage medium. The method provides a dynamic storage optimization mode based on directory granularity and file size perception. The method takes a directory as a unit, analyzes the file size of each incremental file under the directory within a specified time window, extracts a quantitative index reflecting the change characteristics of large / small files to automatically determine the scene type corresponding to each incremental file under each directory, and dynamically matches and applies a target optimization strategy adapted to the scene type corresponding to each directory when the scene type corresponding to each directory to be scanned meets the preset directory-level optimization requirement, thereby realizing fine-grained automatic storage optimization at the directory level and improving the overall performance and storage efficiency of the system.

Description

Technical Field

[0001] This application relates to the field of distributed file storage technology, and in particular to a file storage method, apparatus, device and readable storage medium. Background Technology

[0002] To address the challenges of diverse business needs, distributed file storage requires the ability to read and write massive amounts of files of varying sizes. However, the scenarios involving large numbers of small files and large files differ significantly in terms of file size distribution, I / O behavior, and metadata load. Consequently, their data layout and optimization requirements for the storage system are drastically different, necessitating entirely different storage tuning strategies.

[0003] However, related technologies typically pre-configure globally unified tuning strategies for each directory under the same distributed file system. For example, the entire system may adopt a tuning strategy suitable for small file scenarios or a tuning strategy suitable for large file scenarios. This one-size-fits-all tuning strategy configuration method cannot take into account the actual needs of heterogeneous business loads within the distributed file system, and is very likely to lead to storage performance degradation in scenarios where the globally unified tuning strategy is not applicable. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides a file storage method, apparatus, device and readable storage medium, which can flexibly meet the actual needs of heterogeneous business loads in distributed file systems and provide adaptive tuning strategies for different business loads.

[0005] Specifically, this application is implemented through the following technical solution: According to a first aspect of the embodiments of this application, a file storage method is provided, the method comprising: For each directory to be scanned, the file change characteristics of each incremental file in the directory are determined based on the current file size of each incremental file in the directory within a specified time period. The file change characteristics are used to represent the distribution of large and small files in each incremental file. The specified time period represents the time window from the end time of the previous scan to the current time. Based on the file change characteristics of each incremental file in the directory, the scenario type corresponding to the directory is determined; the scenario type is a small file scenario, a large file scenario, or one of the other scenarios besides small file scenario and large file scenario. If the scenario type corresponding to each directory to be scanned meets the preset directory-level optimization requirements, then based on the scenario type corresponding to each directory to be scanned, the target optimization strategy corresponding to that directory is determined, so as to use the target optimization strategy corresponding to each directory to perform storage operations on each incremental file in each directory to adapt to the scenario type corresponding to each directory.

[0006] In some embodiments, the file change characteristics corresponding to each incremental file in the directory include at least the total number of each incremental file, and at least one of the following: the proportion of small files and the proportion of large files. The proportion of small files is determined based on the number of small files in each incremental file and the total number of incremental files; the proportion of large files is determined based on the number of large files in each incremental file and the total number of incremental files.

[0007] In some embodiments, the scenario type corresponding to the directory is determined based on the file change characteristics of each incremental file in the directory, including: If the file change characteristics corresponding to each incremental file in the directory match the first scenario determination condition, then the scenario type corresponding to the directory is determined to be the small file scenario; the first scenario determination condition includes the condition for determining that the file change characteristics corresponding to each incremental file have small file dominant characteristics; If the file change characteristics of each incremental file in the directory match the second scenario determination condition, then the scenario type corresponding to the directory is determined to be the large file scenario; the second scenario determination condition includes the condition for determining that the file change characteristics of each incremental file have the dominant characteristics of large files; Otherwise, determine that the scene type corresponding to this directory is one of the other scenes.

[0008] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; determining the scenario type corresponding to the directory as other scenarios includes: If the total number of incremental files in the file change characteristics of each incremental file in the directory is greater than or equal to the preset number threshold, then the scenario type corresponding to the directory is determined to be a mixed large and small file scenario. Otherwise, the scenario type corresponding to this directory is determined as a pending scenario.

[0009] In some embodiments, based on the scenario type corresponding to each directory to be scanned, the target optimization strategy for that directory is determined, including: If the scenario type corresponding to this directory is a small file scenario, then the target optimization strategy corresponding to this directory is determined to be the first optimization strategy suitable for small file storage; If the scenario type corresponding to this directory is a large file scenario, then the target tuning strategy for this directory is determined to be the second tuning strategy suitable for large file storage; Otherwise, the target tuning strategy corresponding to this directory is determined to be the default tuning strategy.

[0010] In some embodiments, the default tuning strategy is one of the first tuning strategy and the second tuning strategy; the default tuning strategy is determined in any of the following ways: The default tuning strategy is pre-specified from the first tuning strategy and the second tuning strategy; Alternatively, the default optimization strategy is determined based on the relative relationship between the proportion of small files and the proportion of large files in the file change characteristics of each incremental file in the directory; wherein, if the proportion of small files is greater than the proportion of large files, the default optimization strategy is the first optimization strategy; if the proportion of small files is less than the proportion of large files, the default optimization strategy is the second optimization strategy.

[0011] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; determining the target optimization strategy corresponding to the directory as the default optimization strategy includes: If the scenario type corresponding to this directory is a mixed scenario of large and small files, then configure the default tuning strategy for each incremental file in this directory; If the scenario type corresponding to this directory is a pending scenario, then no default tuning strategy will be configured for this directory, and each incremental file in this directory will continue to be used as part of the incremental files in this directory in the next round of scanning.

[0012] In some embodiments, the preset directory-level optimization requirements include: the scene type corresponding to each directory to be scanned does not meet the first global optimization requirement and does not meet the second global optimization requirement; The first global optimization requirement includes: there are directories with a proportion of at least one preset ratio that correspond to small file scenarios; The second global optimization requirement includes: there must be at least a second preset proportion of directories whose corresponding scenario type is a large file scenario.

[0013] In some embodiments, the method further includes: If the scenario type corresponding to each directory to be scanned meets the first global optimization requirement, then the first optimization strategy suitable for small file storage will be uniformly configured for each directory. If the scenario type corresponding to each directory to be scanned meets the second global optimization requirements, then the second optimization strategy suitable for large file storage will be uniformly configured for each directory.

[0014] According to a second aspect of the embodiments of this application, a file storage device is provided, the device comprising: The file change feature acquisition module is configured to determine the file change features of each incremental file in each directory to be scanned based on the current file size of each incremental file in that directory within a specified time period. The file change features are used to represent the distribution of large and small files in each incremental file. The specified time period represents the time window from the end time of the previous scan to the current time. The scenario type determination module is configured to determine the scenario type of the directory based on the file change characteristics of each incremental file in the directory; the scenario type is a small file scenario, a large file scenario, or one of the scenarios other than small file scenario and large file scenario. The optimization strategy decision module is configured to determine the target optimization strategy for each directory based on the scenario type of each directory to be scanned if the scenario type of each directory meets the preset directory-level optimization requirements. This allows the target optimization strategy to be used to perform storage operations on each incremental file in each directory that are adapted to the scenario type of each directory.

[0015] In some embodiments, the file change characteristics corresponding to each incremental file in the directory include at least the total number of each incremental file, and at least one of the following: the proportion of small files and the proportion of large files; wherein, the proportion of small files is determined based on the number of small files in each incremental file and the total number of each incremental file; and the proportion of large files is determined based on the number of large files in each incremental file and the total number of each incremental file.

[0016] In some embodiments, when the scene type determination module is configured to determine the scene type corresponding to the directory based on the file change characteristics of each incremental file in the directory, it includes: The first determining module is configured to determine the scenario type corresponding to the directory as the small file scenario if the file change characteristics corresponding to each incremental file in the directory match the first scenario determination condition; the first scenario determination condition includes a condition for determining that the file change characteristics corresponding to each incremental file have small file dominant characteristics. The second determining module is configured to determine the scenario type corresponding to the directory as the large file scenario if the file change characteristics corresponding to each incremental file in the directory match the second scenario determination condition; the second scenario determination condition includes a condition for determining that the file change characteristics corresponding to each incremental file have the dominant characteristics of large files. The third determining module is configured to determine the scenario type corresponding to the directory as the other scenario if the file change characteristics corresponding to each incremental file in the directory do not match the first scenario determination condition and the second scenario determination condition.

[0017] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; when the third determining module is configured to determine that the scenario type corresponding to the directory is another scenario, it includes: If the total number of incremental files in the file change characteristics of each incremental file in the directory is greater than or equal to the preset number threshold, then the scenario type corresponding to the directory is determined to be a mixed large and small file scenario. Otherwise, the scenario type corresponding to this directory is determined as a pending scenario.

[0018] In some embodiments, when the optimization strategy decision module is configured to determine the target optimization strategy corresponding to each directory to be scanned based on the scenario type, it includes: The first decision module is configured to determine the target optimization strategy for the directory as the first optimization strategy suitable for small file storage if the scenario type corresponding to the directory is a small file scenario. The second decision module is configured to determine the target optimization strategy for the directory as the second optimization strategy suitable for large file storage if the scenario type corresponding to the directory is a large file scenario. The third decision module is configured to determine the target optimization strategy for the directory as the default optimization strategy if the scenario type corresponding to the directory is neither a large file scenario nor a small file scenario.

[0019] In some embodiments, the default tuning strategy is one of the first tuning strategy and the second tuning strategy; the default tuning strategy is determined in any of the following ways: The default tuning strategy is pre-specified from the first tuning strategy and the second tuning strategy; Alternatively, the default optimization strategy is determined based on the relative relationship between the proportion of small files and the proportion of large files in the file change characteristics of each incremental file in the directory; wherein, if the proportion of small files is greater than the proportion of large files, the default optimization strategy is the first optimization strategy; if the proportion of small files is less than the proportion of large files, the default optimization strategy is the second optimization strategy.

[0020] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; when the third decision module is configured to determine that the target tuning strategy corresponding to the directory is the default tuning strategy, it includes: If the scenario type corresponding to this directory is a mixed scenario of large and small files, then configure the default tuning strategy for each incremental file in this directory; If the scenario type corresponding to this directory is a pending scenario, then no default tuning strategy will be configured for this directory, and each incremental file in this directory will continue to be used as part of the incremental files in this directory in the next round of scanning.

[0021] In some embodiments, the preset directory-level optimization requirements include: the scene type corresponding to each directory to be scanned does not meet the first global optimization requirement and does not meet the second global optimization requirement; The first global optimization requirement includes: there are directories with a scenario type of small file that are at least a first preset proportion; the second global optimization requirement includes: there are directories with a scenario type of large file that are at least a second preset proportion.

[0022] In some embodiments, the apparatus further includes: The first global optimization decision module is configured to uniformly configure the first optimization strategy applicable to small file storage for each directory if the scenario type corresponding to each directory to be scanned meets the first global optimization requirements. The second global optimization decision module is configured to uniformly configure a second optimization strategy suitable for large file storage for each directory if the scenario type corresponding to each directory to be scanned meets the second global optimization requirements.

[0023] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a memory and a processor; the memory being used to store a computer program; and the processor being used to execute the above-described file storage method by invoking the computer program.

[0024] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the above-described file storage method.

[0025] The technical solutions provided in this application embodiment may include the following beneficial effects: In the technical solution provided in this application, by taking directories as units, analyzing the file size of each incremental file within a specified time window, extracting quantitative indicators reflecting the characteristics of large / small file changes, and automatically determining the scenario type corresponding to each incremental file in each directory, and when the scenario type corresponding to each directory to be scanned meets the preset directory-level optimization requirements, dynamically matching and applying the target optimization strategy adapted to the scenario type corresponding to that directory for each directory, fine-grained automatic storage optimization at the directory level is achieved, improving the overall system performance and storage efficiency.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Furthermore, no embodiment in this application needs to achieve all the effects described above. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] Figure 1A This is a schematic flowchart illustrating a file storage method according to an exemplary embodiment of this application; Figure 1B This is an exemplary embodiment of the present application illustrating how to determine the scenario type of a directory based on the file change characteristics of each incremental file in that directory; Figure 1C This is a schematic diagram illustrating a directory-level fine-grained tuning strategy configuration according to an exemplary embodiment of this application; Figure 1D This is a schematic diagram illustrating an optimization strategy decision-making process according to an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the structure of a file storage device according to an exemplary embodiment of this application; Figure 3 This is a hardware schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0030] Before explaining the data storage management method provided in this application, a brief explanation of some of the terms or concepts involved in this application will be provided: Distributed file system: A file system architecture that distributes file data and metadata across multiple physical nodes, supporting unified namespaces, data redundancy, load balancing, and high-availability access across nodes. In this application, the distributed file system provides globally unified directory and file management capabilities, allowing clients to read and write data through standard interfaces.

[0031] Metadata: Data that describes the attributes of a file or directory, excluding the file content itself. In this application, the metadata includes, but is not limited to, information such as file size, creation time, modification time, access permissions, directory of ownership, and data block location, used to support file system management, querying, and performance optimization.

[0032] Directory: A logical container in a file system used to organize and manage files, supporting a hierarchical naming structure. In this application, the directory serves as the basic scope unit for performance tuning. The system independently implements tuning strategies based on the access characteristics of files in each directory, achieving fine-grained resource optimization.

[0033] File: A file is a basic data unit stored in a file system, possessing a unique pathname and attribute information. In a distributed file system, stored files can be categorized as small files and large files. Small files are those whose individual file size does not exceed a set threshold, such as 1MB, while large files are those whose individual file size exceeds a set threshold, such as 1MB.

[0034] Massive small files: Massive small files refer to a large number of files stored in the same directory or namespace in a distributed file system, with the size of each file significantly smaller than a set threshold.

[0035] SSD (Solid State Drive): A non-volatile storage device based on flash memory technology, whose data storage units are NAND flash memory chips with no moving mechanical parts. Compared with traditional hard drives, SSDs have extremely low random access latency (typically in the microsecond range), high IOPS (Input / Output Operations Per Second) performance, and good shock resistance, making them suitable for applications sensitive to read / write latency. In this application, the SSD storage pool is used to host directories with metadata-intensive or frequently accessed small files to improve system response speed and concurrent processing capabilities.

[0036] HDD (Hard Disk Drive): A traditional storage device based on magnetic platters and mechanical read / write heads, accessing data by rotating the disk and moving the read / write head. HDDs have high single-disk capacity and low unit storage cost, but high random access latency (typically in the millisecond range), making them suitable for large file sequential read / write scenarios. In this application, an HDD storage pool is used to store large file-dominated data to achieve the storage optimization goals of high throughput and low cost.

[0037] Prefetching: A storage system optimization technique that predicts data that may be accessed later based on the current access pattern and loads it from the storage medium into the memory cache in advance. Prefetching is suitable for sequential read scenarios and can significantly reduce the latency of subsequent read operations. The prefetch window size can be dynamically adjusted according to file access characteristics: a larger prefetch window can be configured for large file scenarios to improve throughput, while prefetching should be reduced or turned off for small file scenarios to avoid loading invalid data. In this application, the prefetching strategy, as one of the storage optimization methods, can be automatically enabled or adjusted according to the file size characteristics of the directory.

[0038] Small file aggregation refers to the technique of merging multiple small files into a larger file or data block for storage. This reduces metadata overhead in the storage system, improves storage efficiency, and enhances I / O performance. It is suitable for scenarios involving a large number of small files, such as log files, image libraries, and backup data.

[0039] Based on the above, it is clear that with the deepening of digital transformation, distributed file systems are widely used in many fields such as smart cities, artificial intelligence, finance and insurance, medical imaging, and video surveillance. To address the challenges of diverse business needs, distributed file storage requires the ability to read and write massive amounts of files of varying sizes. Specifically, scenarios involving massive amounts of small files (such as AI training data, financial and insurance images, and medical PACS systems) differ significantly from scenarios involving large files (such as video surveillance and archive backups) in terms of file size distribution, I / O behavior, and metadata load. These two scenarios have drastically different requirements for the data layout and optimization of the storage system, thus necessitating entirely different storage tuning strategies.

[0040] In scenarios with massive amounts of small files (such as a facial recognition image library), the system typically needs to manage hundreds of thousands or even millions of small objects (each file ranging from a few KB to hundreds of KB in size). The performance bottleneck mainly lies in the high-frequency metadata operations. Every file creation, opening, reading, or deletion requires interaction with the metadata server, easily leading to metadata server CPU overload, increased file access latency, and low storage efficiency. Therefore, storage optimization is necessary for this type of scenario. For example, storage optimization for massive amounts of small files can be achieved using at least one of the following tuning strategies suitable for small file storage: enable small file aggregation to merge multiple small images into one large object to reduce the number of inodes; bind the directory containing massive amounts of small files to an SSD storage pool to improve the response speed of metadata and small I / O; and disable read-ahead to avoid loading invalid data.

[0041] In contrast, large file scenarios (such as video surveillance systems) exhibit entirely different access characteristics. In these scenarios, individual files are large (typically reaching GB levels), requiring high throughput, sequential writes, and sequential reads. The performance bottlenecks primarily lie in data throughput bandwidth and storage costs. Using a multi-copy redundant storage mechanism can lead to storage overhead of up to three times. Therefore, storage optimization for this type of scenario mainly focuses on improving space utilization and sequential read performance. For example, at least one of the following tuning strategies suitable for large file storage can be used to optimize storage in large file scenarios: employ erasure coding strategies to significantly save storage costs; enable large window pre-reading to improve sequential read throughput during playback; and bind directories to a high-density HDD storage pool to leverage the advantages of sequential access for large files.

[0042] However, related technologies typically pre-configure globally unified tuning strategies for each directory under the same distributed file system. For example, the entire system may adopt a tuning strategy suitable for small file scenarios or a tuning strategy suitable for large file scenarios. This one-size-fits-all tuning strategy configuration method cannot take into account the actual needs of heterogeneous business loads within the distributed file system, and is very likely to lead to storage performance degradation in scenarios where the globally unified tuning strategy is not applicable.

[0043] In view of this, this application provides a file storage method applied to a distributed file system. This method performs statistical analysis on the current file size of each incremental file added or modified since the last scan in each directory of the distributed file system when a scan event is triggered, dynamically identifies its file change characteristics, and automatically determines the scenario type of each directory based on preset scenario judgment conditions. Then, based on the overall scenario distribution of the directory, it decides whether to adopt a globally unified tuning strategy or implement a directory-level fine-grained differentiated strategy configuration. This realizes the on-demand adaptation of tuning strategies for different business storage needs in the distributed file system, and solves the problem of performance degradation or resource waste under heterogeneous business loads caused by the use of a single, fixed global tuning strategy in the prior art.

[0044] See Figure 1A The illustrated flowchart represents a file storage method, which may include at least the following steps: S101, for each directory to be scanned, determine the file change characteristics of each incremental file in the directory based on the current file size of each incremental file in the directory within a specified time period; the file change characteristics are used to represent the distribution of large and small files in each incremental file; the specified time period represents the time window from the end time of the previous scan to the current time. Each directory to be scanned refers to a directory where files have been added or modified since the last scan. For example, if a facial recognition image library directory D1 has had 5,000 new photos written to it in the last hour, then that directory will be used as a directory to be scanned.

[0045] The incremental files in this directory within a specified time period include files newly added to the directory during that time period, as well as existing files in the directory whose content has been modified. In this embodiment, the specified time period can represent a time window from the last scan to the start of the current scan.

[0046] In a distributed file system, the metadata of each file typically includes, but is not limited to, attributes such as file size, permissions, timestamp information, and data block location information. Based on this, for the directory to be scanned, the timestamp information in the metadata of each file in the directory can be used to identify files that have been added or modified since the last scan as incremental files; alternatively, when any file in the directory is modified or a file is added to the directory, the unique identifier of the modified or added file can be recorded synchronously, so as to quickly identify the incremental files in the directory within a specified time period when scanning the directory, and thus perform analysis based on the files that have changed since the end time of the last scan (i.e., incremental files).

[0047] Furthermore, the current file size of each incremental file in the directory within a specified time period can also be obtained based on the metadata of the incremental files. Specifically, a batch metadata query request can be initiated to the metadata server based on the unique identifier (such as file path or global file ID) of each incremental file in the directory. This batch metadata query request carries the unique identifier of each incremental file in the directory. Subsequently, the metadata of each incremental file returned by the metadata server in response to the batch metadata query request is obtained, and the current file size of each incremental file is extracted from it as the basis for subsequent scenario type determination.

[0048] The metadata of files stored in the distributed file system can be recorded in a KV database in the form of key-value pairs; the key field represents the unique identifier of the file, and the metadata of the file indicated by the key field is recorded in the corresponding value field. For example, the following table illustrates a KV storage format for metadata: When metadata is recorded in a KV database in the form of key-value pairs, the metadata server can quickly query matching metadata from the KV database based on the file's unique identifier. By parsing the values ​​of the mtime and size fields, it can quickly determine the size of each incremental file in the directory and its current size.

[0049] The file change characteristics of each incremental file in this directory are used to quantitatively describe the data load characteristics of each incremental file in this directory, reflecting the distribution of large and small files among the incremental files in this directory. Specifically, the file size of the small files does not exceed a set threshold; the file size of the large files exceeds the set threshold. Based on the file change characteristics, the composition of large and small files in each incremental file in this directory can be determined, facilitating the analysis of the file size distribution characteristics of each incremental file in this directory, thereby determining whether the incremental write behavior of this directory is small file-dominated, large file-dominated, or a hybrid type.

[0050] The file change characteristics corresponding to each incremental file in this directory can include at least the total number of incremental files, and at least one of the following: the proportion of small files and the proportion of large files. The proportion of small files is determined based on the number of small files in each incremental file and the total number of incremental files, i.e., small file proportion = (number of files in each incremental file whose size is less than or equal to a set threshold) / (total number of incremental files). The proportion of large files is determined based on the number of large files in each incremental file and the total number of incremental files, i.e., large file proportion = (number of files in each incremental file whose size is greater than a set threshold) / (total number of incremental files). Since the sum of the proportion of small files and the proportion of large files is 1, the proportion of large files can be determined based on the proportion of small files when only the proportion of small files is included, and correspondingly, the proportion of small files can be determined based on the proportion of large files when only the proportion of large files is included.

[0051] Furthermore, the file variation characteristics may also include average file size and frequency distribution of file counts within preset file size ranges (e.g., 0-64KB, 64KB-512KB, 512KB-1MB, >1MB). This fine-grained statistical approach not only supports visual analysis but can also be used to identify the distribution characteristics of files with mixed sizes (e.g., scenarios containing a large number of small images and a small number of large log files), providing support for accurate subsequent determination of the scenario type.

[0052] To achieve dynamic awareness and adaptive tuning of distributed file system storage behavior, this step can be driven by a scan event. When a scan event is detected, it indicates that the distributed file system needs to configure tuning strategies for each incremental file in the distributed file system, and this step begins. This scan event can be triggered periodically (e.g., every 12 hours), and the triggering period can be flexibly configured based on the current business scenario and file storage status of the distributed file system; alternatively, to improve real-time response and resource efficiency, it can also be triggered automatically when set triggering conditions are met. For example, the triggering conditions could be that the total number of incremental files in the distributed file system since the last scan reaches a set total number (e.g., 10,000 files), or the total file size of the incremental files reaches a set total capacity (e.g., 1TB), in which case this scan event will be automatically triggered.

[0053] If this step is being executed for the first time (i.e., no scans have been performed since the distributed file system was initialized), then each incremental file in the specified time period refers to all existing files in the directory, i.e., the full set of files is used as the initial analysis object.

[0054] S102, Based on the file change characteristics of each incremental file in the directory, determine the scenario type corresponding to the directory; the scenario type is a small file scenario, a large file scenario, or one of the other scenarios besides small file scenario and large file scenario; The scenario type is used to characterize the overall access and storage characteristics of incremental files in the directory. Different scenario types reflect different performance bottlenecks and optimization goals.

[0055] This scenario type can be pre-divided into at least three categories: small file scenario, large file scenario, and other scenarios. The small file scenario indicates that the incremental files in this directory are dominated by small files in terms of quantity or access frequency; a small file is defined as a single file whose size does not exceed a set threshold. Correspondingly, the large file scenario indicates that the incremental files in this directory are dominated by large files in terms of quantity or IO bandwidth requirements; a large file is defined as a single file whose size exceeds the set threshold. Other scenarios refer to situations other than the small file and large file scenarios, such as situations where the total number of incremental files is too small to form stable file change characteristics, or where small and large files exhibit a multi-peak / uniform distribution.

[0056] To accurately determine the scenario type, a scenario determination condition can be set for each scenario type. The scenario determination condition for any scenario type is used to determine whether each incremental file in the directory meets the composition of large and small files under that scenario type. Specifically, the first scenario determination condition for the small file scenario includes the condition that the file change characteristics of each incremental file have the characteristics of small files dominating, that is, each incremental file is mostly small files with high-frequency metadata operations and low single-pass IO bandwidth requirements. The second scenario determination condition for the large file scenario includes the condition that the file change characteristics of each incremental file have the characteristics of large files dominating, that is, each incremental file is mostly large files, exhibiting high throughput and sequential read / write access methods. The third scenario determination condition for other scenarios includes the condition that the distribution of incremental files does not meet the characteristics of small files dominating and does not meet the characteristics of large files dominating.

[0057] Therefore, scenario determination conditions for each scenario type can be defined based on quantitative indicators such as the total number of incremental files, the proportion of small files, and the proportion of large files. For example, the first scenario determination condition for the small file scenario may include that the total number of incremental files is greater than or equal to a preset quantity threshold and the proportion of small files is greater than a first proportion threshold; the second scenario determination condition for the large file scenario includes that the total number of incremental files is greater than the preset quantity threshold and the proportion of large files is greater than a second proportion threshold; and the third scenario determination condition for other scenarios includes that neither the first scenario determination condition nor the second scenario determination condition is met.

[0058] Therefore, when determining the scenario type corresponding to a directory, the scenario type can be determined based on the scenario judgment conditions matched by the file change characteristics of each incremental file in that directory. See also Figure 1B An exemplary diagram illustrating how to determine the scenario type of a directory based on the file change characteristics of each incremental file within that directory may include the following steps: S1021, If ​​the file change characteristics of each incremental file in the directory match the first scenario determination condition, then the scenario type corresponding to the directory is determined to be a small file scenario. The small file scenario corresponds to the first scenario determination condition. When the file change characteristics of each incremental file in the directory match the first scenario determination condition, it means that the file change characteristics of each incremental file have the small file dominant characteristics, thus determining that the scenario type corresponding to the directory is the small file scenario.

[0059] For example, the first scenario determination condition for the small file scenario is: the total number of all incremental files is greater than or equal to a preset quantity threshold and the proportion of small files is greater than a first proportion threshold. The process involves obtaining the total number of incremental files and the proportion of small files included in the file change features of each incremental file in the directory. If the extracted total number is greater than or equal to the preset quantity threshold and the extracted proportion of small files is greater than the first proportion threshold, the scenario type corresponding to the directory is determined to be the small file scenario. If the file change features include the proportion of large files, the proportion of small files can be obtained based on the sum of the proportion of large files and the proportion of small files being 1.

[0060] For example, the first scenario determination condition for the small file scenario is: the total number of incremental files is ≥100 and the proportion of small files is >90%. In the file change characteristics of each incremental file under directory D2, the total number of incremental files is 530 and the proportion of small files is 98%. Then the file change characteristics match the first scenario determination condition, and the scenario type corresponding to directory D2 is determined to be the small file scenario.

[0061] S1022, If the file change characteristics of each incremental file in the directory match the second scenario determination condition, then the scenario type corresponding to the directory is determined to be a large file scenario. The large file scenario corresponds to the second scenario determination condition. When the file change characteristics of each incremental file in the directory match the second scenario determination condition, it indicates that the file change characteristics of each incremental file have the dominant characteristics of large files. Thus, the scenario type corresponding to the directory is determined to be a large file scenario.

[0062] For example, the second scenario determination condition for the large file scenario is: the total number of each incremental file is greater than the preset number threshold and the proportion of large files is greater than the second proportion threshold. The total number of each incremental file and the proportion of large files included in the file change characteristics of each incremental file in the directory are obtained. When the total number is greater than or equal to the preset number threshold and the proportion of large files is greater than the second proportion threshold, the scenario type corresponding to the directory is determined to be a large file scenario.

[0063] For example, the second scenario determination condition for the large file scenario is: the total number of incremental files is ≥100 and the proportion of large files is >85%. In the file change characteristics of each incremental file under directory D3, the total number of incremental files is 145 and the proportion of large files is 95%. Then the file change characteristics match the second scenario determination condition, and the scenario type corresponding to directory D3 is determined to be the large file scenario.

[0064] S1023, otherwise, determine that the scene type corresponding to this directory is another scene.

[0065] If the file change characteristics corresponding to each incremental file in the directory do not match either the first scenario determination condition or the second scenario determination condition, it indicates that the scenario type corresponding to the directory does not belong to the small file scenario or the large file scenario. As described in the previous embodiments, the third scenario determination condition for other scenarios includes determining that the distribution of incremental files does not meet the conditions of small file dominance and large file dominance. If the file change characteristics corresponding to each incremental file in the directory match the third scenario determination condition, it can be determined that the scenario type corresponding to the directory is a scenario other than the small file scenario and the large file scenario.

[0066] For example, if the total number of incremental files in the file change characteristics of each incremental file under directory D4 is 625, and the proportion of large files is 57% and the proportion of small files is 43%, then the file change characteristics do not match the first scenario determination condition and the second scenario determination condition, and the scenario type corresponding to directory D4 is determined to be other scenarios.

[0067] For example, if the total number of incremental files in the file change characteristics of each incremental file under directory D5 is 4 and the proportion of large files is 100%, then the file change characteristics do not match the first scenario determination condition and the second scenario determination condition (the total number of incremental files is lower than the preset number threshold), and the scenario type corresponding to directory D5 is determined to be other scenarios.

[0068] S103, if the scene type corresponding to each directory to be scanned meets the preset directory-level optimization requirements, then based on the scene type corresponding to each directory to be scanned, determine the target optimization strategy corresponding to that directory, so as to use the target optimization strategy corresponding to each directory to perform storage operations on each incremental file in each directory to adapt to the scene type corresponding to each directory.

[0069] Directory-level optimization requires that the overall distribution of scenario types corresponding to the directories to be scanned is complex, and there is no significant trend of small file scenarios or large file scenarios dominating. In order to take into account the differentiated I / O characteristics and storage needs of different directories and ensure the storage performance of the storage system, the file change behavior of each directory is modeled in a targeted manner, and each directory to be scanned is independently identified and configured with a strategy to achieve fine-grained storage optimization at the directory level.

[0070] Based on this, the preset directory-level optimization requirements can be set as follows: the scene types corresponding to each directory to be scanned do not meet the first global optimization requirements and do not meet the second global optimization requirements.

[0071] The first global optimization requirement refers to the condition used to determine whether the overall distribution of incremental files in the directories scanned in this round by the distributed file system is dominated by small file scenarios. For example, this first global optimization requirement may include the existence of directories with a scenario type of small file scenarios that is not less than a first preset proportion. For instance, if the first preset proportion is 80%, the total number of directories to be scanned is 1000, and the number of directories with the scenario type of small file scenarios is 952, then the proportion of directories with the scenario type of small file scenarios is 95.2%, which is greater than the first preset proportion. This indicates that the scenario type corresponding to the currently scanned directories meets the first global optimization requirement.

[0072] The second global optimization requirement indicates a condition used to determine whether the overall distribution of incremental files in the directories scanned in this round by the distributed file system is dominated by large file scenarios. For example, this second global optimization requirement may include the existence of directories corresponding to large file scenarios at a rate not less than a second preset proportion. For instance, if the second preset proportion is 70%, the total number of directories to be scanned is 1000, and the number of directories with large file scenarios is 811, then the proportion of directories with large file scenarios is 81.1%, which is greater than the second preset proportion. This indicates that the scenario type corresponding to the currently scanned directories meets the second global optimization requirement.

[0073] If the scenario type corresponding to each directory to be scanned does not meet the first global optimization requirement and does not meet the second global optimization requirement, it means that the directories to be scanned in the distributed file system exhibit diverse file size characteristics. At this time, the fine-grained adaptive optimization mechanism at the directory level can be enabled to accurately match the actual storage characteristics of incremental files in each directory. That is, for each directory, an optimization strategy matching the scenario type corresponding to that directory can be configured for each incremental file in that directory. Different scenario types have preset optimization strategies adapted to that scenario type.

[0074] When determining the target optimization strategy for each directory based on its scenario type, see [reference needed]. Figure 1C The illustration shows a directory-level fine-grained tuning strategy configuration, which configures a tuning strategy that matches the scenario type corresponding to the directory, in order to adjust the storage of each incremental file under the directory: S1031, If ​​the scenario type corresponding to the directory is a small file scenario, then the target optimization strategy corresponding to the directory is determined to be the first optimization strategy suitable for small file storage. If the scenario type corresponding to this directory is a small file scenario, it means that the files added or modified in this directory are mainly small files. Its performance bottleneck lies in high-frequency metadata operations. It is suitable for strategies optimized for small files. Therefore, the target tuning strategy for this directory is determined to be the first tuning strategy suitable for small file storage. This first tuning strategy suitable for small file storage is used to perform storage operations on each incremental file in this directory.

[0075] S1032, If the scenario type corresponding to the directory is a large file scenario, then the target tuning strategy corresponding to the directory is determined to be the second tuning strategy suitable for large file storage. If the scenario type corresponding to this directory is a large file scenario, it means that the newly added or modified files in this directory are mainly large files. Its performance bottleneck lies in high throughput requirements and intensive sequential read and write operations. It is suitable for strategies optimized for large files. Therefore, the target tuning strategy for this directory is determined to be the second tuning strategy suitable for large file storage. This second tuning strategy suitable for large file storage is used to perform storage operations on each incremental file in this directory.

[0076] S1033, otherwise, determine the target tuning strategy corresponding to this directory as the default tuning strategy.

[0077] If the scenario type corresponding to the directory is neither a large file scenario nor a small file scenario, it means that large and small files coexist in each incremental file under the directory, and there is no dominant feature of small files or large files. In this case, the default tuning strategy can be used as a fallback. Configure the default tuning strategy for the directory, and perform storage operations on each incremental file under the directory based on the default tuning strategy.

[0078] The default optimization strategy can be determined from a first optimization strategy suitable for small file storage and a second optimization strategy suitable for large file storage. For example, the default optimization strategy can be a pre-specified fixed strategy, that is, manually or configured to select one of the first optimization strategy suitable for small file storage and the second optimization strategy suitable for large file storage as the system default. Alternatively, the default optimization strategy can also be dynamically determined based on the relative relationship between the proportion of small files and the proportion of large files in the file change characteristics corresponding to each incremental file in the directory: if the proportion of small files is greater than the proportion of large files, the default optimization strategy is the first optimization strategy suitable for small file storage; if the proportion of small files is less than the proportion of large files, the default optimization strategy is the second optimization strategy suitable for large file storage; if the proportion of small files is equal to the proportion of large files, the default optimization strategy can be any one of the two strategies, the first optimization strategy suitable for small file storage and the second optimization strategy suitable for large file storage.

[0079] Alternatively, it could be a general-purpose optimization strategy, different from the first and second optimization strategies mentioned above, with parameter configurations between small file optimization and large file optimization, specifically designed for mixed load scenarios. This application does not limit the specific implementation form of the default optimization strategy; it can be flexibly defined according to actual system requirements.

[0080] Based on the directory-level optimization requirements described in the foregoing embodiments, when the scenario type corresponding to each directory to be scanned does not meet the directory-level optimization requirements, the optimization method for determining the directory is also different from the directory-level optimization method. See [link to documentation]. Figure 1D An exemplary diagram illustrating the optimization strategy decision-making process is shown below: If the scenario type corresponding to each directory to be scanned meets the first global optimization requirement, then the first optimization strategy suitable for small file storage is uniformly configured for each directory; if the scenario type corresponding to each directory to be scanned meets the second global optimization requirement, then the second optimization strategy suitable for large file storage is uniformly configured for each directory; otherwise (i.e., neither global optimization requirement is met), then the directory-level optimization mode is entered, and the target optimization strategy corresponding to each directory is determined based on the scenario type corresponding to each directory to be scanned.

[0081] If the number of directories in the small file scenario meets the first global optimization requirement, it means that the overall distribution of incremental files in the distributed file system is concentrated in the small file scenario, presenting a relatively unified high-frequency metadata operation access pattern. At this time, the storage performance requirements of each directory tend to be low-latency metadata processing, high-concurrency small IO response capability, and efficient inode management. Therefore, optimization strategies suitable for small file storage can be uniformly configured for each incremental file in all directories, such as enabling small file aggregation, binding to SSD storage pool, and disabling pre-read, so as to reduce the complexity of strategy management and system overhead while ensuring system performance.

[0082] If the number of directories in the large file scenario meets the first global optimization requirement, it means that the overall distribution of incremental files in the distributed file system is concentrated in the large file scenario, exhibiting a relatively unified high-throughput, sequential read-write-dominated access mode. At this time, the storage performance requirements of each directory tend to be high-bandwidth data transmission capability, low storage redundancy overhead, and efficient sequential IO processing efficiency. Therefore, optimization strategies suitable for large file storage can be uniformly configured for each incremental file in all directories, such as adopting erasure coding redundancy mechanism, enabling large window pre-read, and binding data to high-density HDD storage pool, so as to reduce the complexity of strategy management and system scheduling overhead while ensuring system performance.

[0083] This embodiment provides a dynamic storage optimization method based on directory granularity and file size awareness. This method operates on a directory-by-directory basis, analyzing the file size of incremental files within a specified time window to extract quantitative indicators reflecting changes in file size. This automatically determines the scene type corresponding to each incremental file in each directory. When the scene type of each directory to be scanned meets preset directory-level optimization requirements, a target optimization strategy adapted to the scene type of that directory is dynamically matched and applied to each directory. This achieves fine-grained automatic storage optimization at the directory level, improving overall system performance and storage efficiency. Furthermore, when the data composition of a directory undergoes a fundamental change, such as a small file directory being converted to a large video directory, the system can automatically re-determine the scene type and switch strategies in subsequent scan cycles, achieving directory-level adaptive adjustment of the optimization strategy. Compared to existing technologies, this application achieves precise adaptation of directory-level optimization strategies through directory-level awareness and scene type recognition, ensuring high-performance operation of various services while improving system storage performance and maintainability.

[0084] In some embodiments, to improve the granularity of scene recognition and achieve more accurate policy matching, other scenarios described in the foregoing embodiments can be further subdivided. In this embodiment, the other scenarios may also include mixed large and small file scenarios and pending scenarios. A mixed large and small file scenario indicates that the number of incremental files in the directory is sufficient (i.e., statistically significant), but the file change characteristics corresponding to each incremental file do not show a small file-dominated feature or a large file-dominated feature; large and small files coexist. A pending scenario indicates that the total number of incremental files in the directory is too small to form stable or reliable file change characteristics, making it impossible to reliably determine the scenario type; further determination is needed after more incremental file data in the directory is obtained.

[0085] Based on this, the determination of the scene type corresponding to the directory as other scenes in step S1023 above can further include the following implementation steps: If the total number of incremental files in the file change characteristics corresponding to each incremental file in the directory is greater than or equal to a preset number threshold, then the scenario type corresponding to the directory is determined to be a mixed file size scenario; otherwise, the scenario type corresponding to the directory is determined to be a pending scenario, that is, when the total number of incremental files in the file change characteristics corresponding to each incremental file in the directory is less than the preset number threshold, the scenario type corresponding to the directory is determined to be a mixed file size scenario.

[0086] For example, taking the scenario type corresponding to directories D4 and D5 as "other scenarios" as an example, the total number of incremental files under directory D4 is 625, which is less than the preset number threshold of 100. Therefore, the scenario type corresponding to directory D4 is determined to be a mixed scenario of large and small files in other scenarios. The total number of incremental files under directory D5 is 4, which is less than the preset number threshold. Therefore, the scenario type corresponding to directory D5 is determined to be an undetermined scenario in other scenarios.

[0087] Based on the above-mentioned detailed scenario classification, the determination of the target optimization strategy corresponding to the directory as the default optimization strategy in step S1033 can be further specified as the following operations: If the scenario type corresponding to this directory is a mixed large and small file scenario, then a default tuning strategy is configured for each incremental file in this directory; if the scenario type corresponding to this directory is a pending scenario, then no default tuning strategy is configured for this directory, and each incremental file in this directory will continue to be used as part of the incremental files in this directory in the next round of scanning.

[0088] In other words, if the scenario type corresponding to the directory is a pending scenario, the default tuning strategy will not be enabled for any incremental files under that directory. The status of these incremental files will be retained as part of the incremental files used to determine the file changes corresponding to that directory in the next round of scanning. For example, for the directory D5 mentioned above, the total number of incremental files is 4. The default tuning strategy will not be enabled for these 4 incremental files for the time being. These 4 incremental files will be used as part of the incremental files for directory D5 in the next round of scanning, thereby avoiding misjudgments and inappropriate strategy configurations due to insufficient samples.

[0089] To enable those skilled in the art to better understand the file storage method provided in this application, this embodiment provides a specific example illustrating the file storage method.

[0090] When a scan event is detected, the directories in which files have been added or modified since the last scan are selected as the directories to be scanned. For example, a distributed file system can mark the directory to which the files involved in an I / O operation belong as an active directory when an I / O operation is received, and then use the directories marked as active directories as the directories to be scanned when the scan begins.

[0091] For each directory to be scanned, a batch query request for metadata of each incremental file in that directory is sent to the metadata server. For example, the metadata of each incremental file in that directory can be obtained by setting the query condition "belongs to this directory and the last modification time of the file is later than the start time of the previous scan". The value of the field representing the file size is extracted from the metadata and used as the current file size of each incremental file in that directory. The file change characteristics corresponding to that directory are then determined, including features that reflect the distribution of large and small files in each incremental file.

[0092] As shown in the table below, the file change characteristics described in this embodiment include the following: The scenario types described in this embodiment include the following types, and the scenario determination conditions for examples are shown in the table below: Therefore, for the file change characteristics corresponding to each incremental file in the several directories shown in the table below, the scenario type corresponding to each directory is determined based on the scenario determination conditions matched by the file change characteristics: The optimization strategy can be flexibly selected based on the overall distribution of scene types corresponding to the directories to be scanned, as exemplified in the table below: The directory-level optimization strategy can be determined as shown in the table below: The first optimization strategy applicable to small files may include, but is not limited to, enabling small file aggregation based on directories, binding directories to SSD storage pools, and using multiple replicas for directories. The second optimization strategy applicable to large files may include, but is not limited to, using erasure coding redundancy levels based on directories and binding directories to HDD storage pools.

[0093] As demonstrated by the above embodiments, this application provides a directory-level file size-aware storage optimization method. Using directories as the basic management unit, it automatically identifies the access and storage patterns corresponding to each directory based on the file size distribution characteristics of incremental files in each directory to be scanned. Furthermore, based on the overall access and storage patterns of each directory to be scanned, it configures optimization strategies suitable for the corresponding scenario type for each directory, achieving adaptive, directory-level storage optimization and improving the storage performance of the distributed file system. Based on this method, when the file composition within a directory changes significantly (e.g., a small file directory originally used for storing logs begins to write large video files), the system can re-evaluate the scenario type in the next scan and switch to a new, suitable strategy, achieving adaptive storage configuration and improving storage space utilization.

[0094] Corresponding to the embodiments of the aforementioned file storage method, see [link to relevant documentation]. Figure 2 As shown, this application also provides an embodiment of a file storage device, the device comprising: The file change feature acquisition module 201 is configured to determine the file change features of each incremental file in the directory based on the current file size of each incremental file in the directory within a specified time period for each directory to be scanned. The file change features are used to represent the distribution of large and small files in each incremental file. The specified time period represents the time window from the end time of the previous scan to the current time. The scenario type determination module 202 is configured to determine the scenario type corresponding to the directory based on the file change characteristics of each incremental file in the directory; the scenario type is a small file scenario, a large file scenario, or one of other scenarios besides small file scenario and large file scenario. The optimization strategy decision module 203 is configured to determine the target optimization strategy corresponding to each directory based on the scenario type of each directory to be scanned if the scenario type of each directory meets the preset directory-level optimization requirements. The target optimization strategy corresponding to each directory is then used to perform storage operations on each incremental file in each directory to adapt to the scenario type of each directory.

[0095] In some embodiments, the file change characteristics corresponding to each incremental file in the directory include at least the total number of each incremental file, and at least one of the following: the proportion of small files and the proportion of large files; wherein, the proportion of small files is determined based on the number of small files in each incremental file and the total number of each incremental file; and the proportion of large files is determined based on the number of large files in each incremental file and the total number of each incremental file.

[0096] In some embodiments, when the scene type determination module is configured to determine the scene type corresponding to the directory based on the file change characteristics of each incremental file in the directory, it includes: The first determining module is configured to determine the scenario type corresponding to the directory as the small file scenario if the file change characteristics corresponding to each incremental file in the directory match the first scenario determination condition; the first scenario determination condition includes a condition for determining that the file change characteristics corresponding to each incremental file have small file dominant characteristics. The second determining module is configured to determine the scenario type corresponding to the directory as the large file scenario if the file change characteristics corresponding to each incremental file in the directory match the second scenario determination condition; the second scenario determination condition includes a condition for determining that the file change characteristics corresponding to each incremental file have the dominant characteristics of large files. The third determining module is configured to determine the scenario type corresponding to the directory as the other scenario if the file change characteristics corresponding to each incremental file in the directory do not match the first scenario determination condition and the second scenario determination condition.

[0097] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; when the third determining module is configured to determine that the scenario type corresponding to the directory is another scenario, it includes: If the total number of incremental files in the file change characteristics of each incremental file in the directory is greater than or equal to the preset number threshold, then the scenario type corresponding to the directory is determined to be a mixed large and small file scenario. Otherwise, the scenario type corresponding to this directory is determined as a pending scenario.

[0098] In some embodiments, when the optimization strategy decision module is configured to determine the target optimization strategy corresponding to each directory to be scanned based on the scenario type, it includes: The first decision module is configured to determine the target optimization strategy for the directory as the first optimization strategy suitable for small file storage if the scenario type corresponding to the directory is a small file scenario. The second decision module is configured to determine the target optimization strategy for the directory as the second optimization strategy suitable for large file storage if the scenario type corresponding to the directory is a large file scenario. The third decision module is configured to determine the target optimization strategy for the directory as the default optimization strategy if the scenario type corresponding to the directory is neither a large file scenario nor a small file scenario.

[0099] In some embodiments, the default tuning strategy is one of the first tuning strategy and the second tuning strategy; the default tuning strategy is determined in any of the following ways: The default tuning strategy is pre-specified from the first tuning strategy and the second tuning strategy; Alternatively, the default optimization strategy is determined based on the relative relationship between the proportion of small files and the proportion of large files in the file change characteristics of each incremental file in the directory; wherein, if the proportion of small files is greater than the proportion of large files, the default optimization strategy is the first optimization strategy; if the proportion of small files is less than the proportion of large files, the default optimization strategy is the second optimization strategy.

[0100] In some embodiments, the other scenarios include mixed file size scenarios and pending scenarios; when the third decision module is configured to determine that the target tuning strategy corresponding to the directory is the default tuning strategy, it includes: If the scenario type corresponding to this directory is a mixed scenario of large and small files, then configure the default tuning strategy for each incremental file in this directory; If the scenario type corresponding to this directory is a pending scenario, then no default tuning strategy will be configured for this directory, and each incremental file in this directory will continue to be used as part of the incremental files in this directory in the next round of scanning.

[0101] In some embodiments, the preset directory-level optimization requirements include: the scene type corresponding to each directory to be scanned does not meet the first global optimization requirement and does not meet the second global optimization requirement; The first global optimization requirement includes: there are directories with a scenario type of small file that are at least a first preset proportion; the second global optimization requirement includes: there are directories with a scenario type of large file that are at least a second preset proportion.

[0102] In some embodiments, the apparatus further includes: The first global optimization decision module is configured to uniformly configure the first optimization strategy applicable to small file storage for each directory if the scenario type corresponding to each directory to be scanned meets the first global optimization requirements. The second global optimization decision module is configured to uniformly configure a second optimization strategy suitable for large file storage for each directory if the scenario type corresponding to each directory to be scanned meets the second global optimization requirements.

[0103] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0105] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 3 As shown, the electronic device 300 includes at least one processor 301, a memory 302, and a bus 303. At least one processor 301 is electrically connected to the memory 302. The memory 302 is configured to store at least one computer-executable instruction, and the processor 301 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any file storage method provided in any embodiment or optional implementation of this application.

[0106] Furthermore, the processor 301 can be an FPGA (Field-Programmable Gate Array) or other devices with logic processing capabilities, such as an MCU (Microcontroller Unit) or a CPU (Central Processing Unit).

[0107] This application also provides another readable storage medium storing a computer program that, when executed by a processor, implements the steps of any file storage method provided in any embodiment or optional implementation of this application.

[0108] The readable storage media provided in this application include, but are not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, readable storage media include any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0109] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0110] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0111] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A file storage method, characterized in that, The method includes: For each directory to be scanned, the file change characteristics of each incremental file in the directory are determined based on the current file size of each incremental file in the directory within a specified time period. The file change characteristics are used to represent the distribution of large and small files in each incremental file. The specified time period represents the time window from the end time of the previous scan to the current time. Based on the file change characteristics of each incremental file in the directory, the scenario type corresponding to the directory is determined; the scenario type is a small file scenario, a large file scenario, or one of the other scenarios besides small file scenario and large file scenario. If the scenario type corresponding to each directory to be scanned meets the preset directory-level optimization requirements, then based on the scenario type corresponding to each directory to be scanned, the target optimization strategy corresponding to that directory is determined, so as to use the target optimization strategy corresponding to each directory to perform storage operations on each incremental file in each directory to adapt to the scenario type corresponding to each directory.

2. The method according to claim 1, characterized in that, The file change characteristics corresponding to each incremental file in the directory include at least the total number of each incremental file, and at least one of the following: the proportion of small files and the proportion of large files. The proportion of small files is determined based on the number of small files in each incremental file and the total number of incremental files; the proportion of large files is determined based on the number of large files in each incremental file and the total number of incremental files.

3. The method according to claim 1 or 2, characterized in that, Based on the file change characteristics of each incremental file in this directory, the scenario type corresponding to this directory is determined, including: If the file change characteristics corresponding to each incremental file in the directory match the first scenario determination condition, then the scenario type corresponding to the directory is determined to be the small file scenario; the first scenario determination condition includes the condition for determining that the file change characteristics corresponding to each incremental file have small file dominant characteristics; If the file change characteristics of each incremental file in the directory match the second scenario determination condition, then the scenario type corresponding to the directory is determined to be the large file scenario; the second scenario determination condition includes the condition for determining that the file change characteristics of each incremental file have the dominant characteristics of large files; Otherwise, determine that the scene type corresponding to this directory is one of the other scenes.

4. The method according to claim 3, characterized in that, The other scenarios include mixed file sizes and undetermined scenarios; The scene type corresponding to this directory is determined to be other scenes, including: If the total number of incremental files in the file change characteristics of each incremental file in the directory is greater than or equal to the preset number threshold, then the scenario type corresponding to the directory is determined to be a mixed large and small file scenario. Otherwise, the scenario type corresponding to this directory is determined as a pending scenario.

5. The method according to claim 1, characterized in that, Based on the scenario type corresponding to each directory to be scanned, determine the target optimization strategy for that directory, including: If the scenario type corresponding to this directory is a small file scenario, then the target optimization strategy corresponding to this directory is determined to be the first optimization strategy suitable for small file storage; If the scenario type corresponding to this directory is a large file scenario, then the target tuning strategy for this directory is determined to be the second tuning strategy suitable for large file storage; Otherwise, the target tuning strategy corresponding to this directory is determined to be the default tuning strategy.

6. The method according to claim 5, characterized in that, The default tuning strategy is one of the first tuning strategy and the second tuning strategy; the default tuning strategy is determined by any of the following methods: The default tuning strategy is pre-specified from the first tuning strategy and the second tuning strategy; Alternatively, the default optimization strategy is determined based on the relative relationship between the proportion of small files and the proportion of large files in the file change characteristics of each incremental file in the directory; wherein, if the proportion of small files is greater than the proportion of large files, the default optimization strategy is the first optimization strategy; if the proportion of small files is less than the proportion of large files, the default optimization strategy is the second optimization strategy.

7. The method according to claim 5, characterized in that, The other scenarios include mixed file sizes and pending scenarios; The target tuning strategy for this directory is determined to be the default tuning strategy, including: If the scenario type corresponding to this directory is a mixed scenario of large and small files, then configure the default tuning strategy for each incremental file in this directory; If the scenario type corresponding to this directory is a pending scenario, then no default tuning strategy will be configured for this directory, and each incremental file in this directory will continue to be used as part of the incremental files in this directory in the next round of scanning.

8. The method according to claim 1, characterized in that, The preset directory-level optimization requirements include: the scene type corresponding to each directory to be scanned does not meet the first global optimization requirement and does not meet the second global optimization requirement; The first global optimization requirement includes: there are directories with a proportion of at least one preset ratio that correspond to small file scenarios; The second global optimization requirement includes: there must be at least a second preset proportion of directories whose corresponding scenario type is a large file scenario.

9. The method according to claim 8, characterized in that, The method further includes: If the scenario type corresponding to each directory to be scanned meets the first global optimization requirement, then the first optimization strategy suitable for small file storage will be uniformly configured for each directory. If the scenario type corresponding to each directory to be scanned meets the second global optimization requirements, then the second optimization strategy suitable for large file storage will be uniformly configured for each directory.

10. A file storage device, characterized in that, The device includes: The file change feature acquisition module is configured to determine the file change features of each incremental file in each directory to be scanned based on the current file size of each incremental file in that directory within a specified time period. The file change features are used to represent the distribution of large and small files in each incremental file. The specified time period refers to the time window from the end time of the previous scan to the current time. The scenario type determination module is configured to determine the scenario type of the directory based on the file change characteristics of each incremental file in the directory; the scenario type is a small file scenario, a large file scenario, or one of the scenarios other than small file scenario and large file scenario. The optimization strategy decision module is configured to determine the target optimization strategy for each directory based on the scenario type of each directory to be scanned if the scenario type of each directory meets the preset directory-level optimization requirements. This allows the target optimization strategy to be used to perform storage operations on each incremental file in each directory that are adapted to the scenario type of each directory.

11. An electronic device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to implement the method as described in any one of claims 1-9.

12. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images