A small file access method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a small file access method and device, electronic equipment and storage medium, and relate to the technical field of computer storage. The method comprises: obtaining storage parameters and defining packing rules according to business requirements, and accurately classifying and aggregating management of small files. The newly uploaded small files are saved to the local read-write storage by using asynchronous tasks, and the response is returned immediately, and the small files are scanned and aggregated to generate index information and small file indexes according to the packing rules, so that efficient storage and rapid retrieval are realized. Based on the access frequency and importance, the aggregated files are divided into hot data and cold data, and are respectively stored in the local high-speed storage and low-cost object storage. When downloading the target file, the file name is used to query the small file index and the index information in the aggregated file to locate the file, and if the file is in the cloud, the file is downloaded to a temporary directory and parsed. The present application effectively solves the problems of low storage efficiency, insufficient universality and high cost in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer storage technology, and in particular to a method, apparatus, electronic device, and storage medium for accessing small files. Background Technology

[0002] With the advent of the digital age, data volume has exploded, and small files, due to their sheer number and complex management, have become a major challenge in the storage field. While traditional small file storage optimization solutions each have their advantages, they are insufficient to fully meet the needs of modern storage environments.

[0003] These traditional methods are often limited to specific scenarios and cannot maintain high efficiency under diverse needs such as storing massive amounts of mixed small and large files, requiring immediate download after uploading, and persistent file storage. Furthermore, the definition of a small file varies significantly across different business scenarios, making traditional solutions lack universality. In addition, traditional methods are significantly insufficient in reducing the number of small files stored, improving query time, throughput, persistence, security, and stability, making it difficult to meet the comprehensive requirements of modern storage systems for efficiency, flexibility, and reliability.

[0004] Therefore, there is an urgent need for a universal and efficient method for storing and retrieving small files that can adapt to different scenarios, significantly improve storage efficiency, ensure data security and reliability, and reduce storage costs and maintenance complexity. Summary of the Invention

[0005] The embodiments of the present invention provide a method for storing and retrieving small files to solve the problems of existing technologies being limited to specific scenarios, unable to simultaneously meet throughput, persistence, security, and stability requirements, and unable to adapt to different scenarios. The technical solution is as follows: According to one aspect of the present invention, a method for storing and retrieving small files includes: obtaining storage parameters according to business needs and defining packaging rules according to the storage parameters; the storage parameters include a small file threshold, upper and lower limits of packaging capacity, and upper limit of packaging quantity; saving newly uploaded small files to local read / write storage and immediately returning a success response; scanning the small files according to the packaging rules through an asynchronous task and storing the small files in an aggregate file in a structured manner, and generating a small file index; the aggregate file includes index information of the small files; dividing the aggregate file into hot data and cold data based on access frequency and importance, retaining the hot data in local high-speed storage, and migrating the cold data to low-cost object storage; the hot data and cold data respectively represent data accessed frequently and accessed infrequently; when downloading a target file, querying the file storage address in the small file index according to a unique identifier; if the query fails, locating the aggregate file where the target file is located, locating the target file according to the index information in the aggregate file; if the aggregate file is not local, downloading it to a temporary directory in the cloud and then parsing it.

[0006] In one embodiment, obtaining storage parameters according to business needs is achieved through the following steps: dynamically adjusting the small file threshold according to the business scenario; the dynamic adjustment includes setting the small file threshold to 500KB in the image storage scenario to distinguish between thumbnails and original images, and setting the small file threshold to 10KB in the log file scenario to optimize the storage of massive small logs.

[0007] In one embodiment, storage parameters are obtained according to business requirements, and packaging rules are defined based on the storage parameters through the following steps: the business scenario is analyzed to determine the storage parameters, and the storage parameters are converted into packaging rules through the configuration management module; the packaging rules include the conditions for file aggregation, the capacity range of the aggregated files, and the file number limit; the file number limit is used to avoid a single aggregated file from being too large.

[0008] In one embodiment, the process of scanning the small files according to the packaging rules using an asynchronous task, storing the small files in a structured format in an aggregate file, and generating a small file index is achieved through the following steps: periodically scanning and reading / writing the small files in storage using an asynchronous task, aggregating the small files into an aggregate file that conforms to a set standard according to the packaging rules, generating corresponding index information in the aggregate file, and generating a small file index; the index information includes a unique identifier for the small file, the offset of the small file, and the length of the small file; the small file index includes a unique identifier, the storage address of the small file, and the storage address of the aggregate file.

[0009] In one embodiment, the structured storage of the small file into the aggregate file is achieved through the following steps: the index information of the small file is structured in the form of key-value pairs; the unique identifier of the small file includes the file name; the offset refers to the starting position of the small file in the aggregate file; the length refers to the data size of the small file; the key-value pair format is used to locate the small file by index.

[0010] In one embodiment, the aggregation files are divided into hot data and cold data based on access frequency and importance through the following steps: the aggregation files are divided into hot data and cold data according to their access frequency and importance; the access frequency includes the number of accesses in the last 7 days; the importance includes business-critical tags; the local high-speed storage includes SSD; and the low-cost object includes cloud disk.

[0011] In one embodiment, when downloading a target file, the unique identifier of the small file is used to query the small file index to locate the aggregate file where the target file is located. The target file is then located and read based on the index information in the aggregate file. If the aggregate file is in the cloud, it is downloaded to a temporary directory and then parsed. This is achieved through the following steps: When downloading a target file, the unique identifier of the target file is used to query the storage address of the target file in the small file index. If the query is successful, the file is downloaded directly and returned. If the query fails, the aggregate file where the target file is located is obtained based on the storage address of the aggregate file in the small file index. The target file is located based on the index information in the aggregate file, downloaded, and returned. If the aggregate file is not stored locally, it is downloaded from the cloud to a temporary directory and then the target file is parsed based on the offset in the index information.

[0012] According to one aspect of the present invention, a small file storage and retrieval device includes: a parameter and rule configuration module, configured to obtain storage parameters according to business needs and define packaging rules according to the storage parameters; the storage parameters include a small file threshold, upper and lower limits of packaging capacity, and upper limit of packaging quantity; an asynchronous aggregation and index generation module, configured to save newly uploaded small files to local read / write storage and immediately return a success response, scan the small files according to the packaging rules through an asynchronous task, structure and store the small files in an aggregate file, and generate a small file index; the aggregate file includes the index information of the small files; and a hot / cold classification and data... The storage module is used to divide the aggregated files into hot data and cold data based on access frequency and importance. The hot data is kept in local high-speed storage, while the cold data is migrated to low-cost object storage. The hot data and cold data represent data accessed frequently and accessed infrequently, respectively. The indexing and file reading module is used to query the file storage address in the small file index based on a unique identifier when downloading the target file. If the query fails, the module locates the aggregated file where the target file is located and locates the target file based on the index information in the aggregated file. If the aggregated file is not local, it is downloaded to a temporary directory in the cloud and then parsed.

[0013] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the small file access method as described above.

[0014] According to one aspect of the present invention, a storage medium has computer-readable instructions stored thereon, which are executed by one or more processors to implement the small file access method described above.

[0015] The beneficial effects of the technical solution provided by this invention are: In the above technical solution, this invention first obtains key storage parameters and defines packaging rules based on business needs to accurately classify and aggregate small files. Based on this, it uses asynchronous tasks to save newly uploaded small files to local read / write storage and immediately returns a response. Simultaneously, it scans and aggregates small files according to the packaging rules, generating index information and a small file index, achieving efficient storage and fast retrieval. Subsequently, based on access frequency and importance, aggregated files are divided into hot data and cold data, stored in local high-speed storage and low-cost object storage respectively, to balance performance and cost. When downloading a target file, the file is located by querying the small file index and aggregate file index information based on the file's unique identifier. If the aggregate file is local, it is read directly; if it is in the cloud, it is downloaded to a temporary directory and then parsed, ensuring fast response and data security. This invention achieves efficient storage and fast access for small files, significantly improving storage efficiency and system performance, while reducing storage costs and maintenance complexity, effectively solving the problems of low efficiency and insufficient versatility of existing technologies in massive small file storage scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a small file access method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating the interaction process of a small file access method in an application scenario. Figure 3 This is a flowchart of file upload in a small file access method within an application scenario; Figure 4 This is a flowchart of a small file download method in a specific application scenario. Figure 5 This is a flowchart of the aggregated file directory cleanup process in a small file access method for an application scenario; Figure 6 This is a block diagram illustrating a small file access device according to an exemplary embodiment; Figure 7 This is a hardware structure diagram of an electronic device according to an exemplary embodiment; Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0020] This invention provides a small file access method that achieves efficient access and management of small files through parameterized configuration and asynchronous aggregation strategies. Combined with a hot / cold data hierarchical storage mechanism, it improves system performance and effectively solves the core problems of traditional solutions, such as low storage efficiency, insufficient versatility, and high cost in scenarios with massive amounts of small files. This small file access method is applicable to small file access devices, which can be electronic devices. The small file access method in this invention can be applied to various scenarios, such as small file access, etc.

[0021] Please see Figure 1 This invention provides a method for accessing small files, which is applicable to electronic devices.

[0022] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0023] like Figure 1 As shown, the method may include the following steps: Step 110: Obtain storage parameters according to business requirements, and define packaging rules based on storage parameters.

[0024] One possible implementation involves analyzing the business scenario to determine storage parameters, and then using a configuration management module to convert these storage parameters into packaging rules.

[0025] The packaging rules include conditions for file aggregation, the size range of the aggregated file, and file quantity limits, etc. The file quantity limit is used to avoid a single aggregated file being too large. Storage parameters include small file thresholds, upper and lower limits of packaged size, and upper limit of packaged quantity, etc., none of which are specified here.

[0026] One possible implementation is to dynamically adjust the small file threshold based on the business scenario. Dynamic adjustment could include setting the small file threshold to 500KB in image storage scenarios to distinguish between thumbnails and original images, or setting it to 10KB in log file scenarios to optimize the storage of massive amounts of small logs, etc., without specifying any limitations here.

[0027] Specifically, the first step is to analyze the specific needs for small file storage based on the actual application scenarios, such as image storage and log file storage. For example, in image storage scenarios, it may be necessary to distinguish between thumbnails and original images, thus requiring the setting of different small file thresholds.

[0028] Furthermore, based on business scenario analysis, key storage parameters are determined, including small file thresholds, upper and lower limits for package size, and upper limit for the number of packages. The small file threshold is used to distinguish between small and large files, while the upper and lower limits for package size and the upper limit for the number of packages are used to control the size and number of aggregated files, avoiding individual aggregated files from being too large or too small.

[0029] Furthermore, the configuration management module translates storage parameters into specific packaging rules, including file aggregation conditions, the aggregated capacity range, and file quantity limits. These rules will guide the subsequent small file aggregation process.

[0030] In the above process, the embodiments of the present invention, through in-depth analysis of business scenarios and flexible configuration of storage parameters, enable the packaging rules to accurately adapt to the needs of different application scenarios, providing a highly customizable storage solution and achieving efficient utilization of storage resources.

[0031] Step 120: Save the newly uploaded small files to local read / write storage and immediately return a success response. Scan the small files according to the packaging rules through an asynchronous task, store the small files in an aggregate file in a structured manner, and generate a small file index.

[0032] In one possible implementation, small files in storage are periodically scanned and read / written by an asynchronous task. Based on the packaging rules, the small files are aggregated into aggregate files that meet the set criteria. Corresponding index information is generated in the aggregate files, and a small file index is generated.

[0033] The aggregate file includes the index information of the small files; the index information includes the unique identifier of the small file, the offset of the small file, the length of the small file, etc., the small file index includes the unique identifier, the storage address of the small file and the storage address of the aggregate file, etc., the unique identifier of the small file includes the file name, etc., none of which are limited here.

[0034] Here, offset refers to the starting position of the small file in the aggregate file; length refers to the size of the small file data; key-value pair format is used to locate small files by index and avoid decompressing the entire aggregate file.

[0035] Specifically, newly uploaded small files are first saved to local read / write storage, and a success response is immediately returned to ensure a smooth user experience. Small files in the read / write storage are periodically scanned via asynchronous tasks, and those meeting certain criteria are aggregated into aggregate files that conform to set standards based on packaging rules. This process occurs in the background and does not affect the upload and download performance of files on the foreground. During aggregation, structured index information is generated and stored in the aggregate file. This index information records the unique identifier, offset, and length of each small file within the aggregate file. This index information will be used for subsequent fast file location and retrieval.

[0036] In the above process, the embodiments of the present invention realize the fast upload and background aggregation of small files through the asynchronous task mechanism, so that the file upload process is almost unaffected by the aggregation operation, providing an efficient and stable file upload service. At the same time, the generation of structured index information also lays the foundation for subsequent fast file reading.

[0037] Step 130: Based on access frequency and importance, the aggregated files are divided into hot data and cold data. Hot data is kept in local high-speed storage, while cold data is migrated to low-cost object storage.

[0038] In one possible implementation, aggregate files are categorized into hot data and cold data based on their access frequency and importance.

[0039] Here, hot data and cold data refer to data accessed frequently and infrequently, respectively; access frequency includes the number of accesses in the last 7 days, importance includes business-critical tags, local high-speed storage includes SSDs, and low-cost objects include cloud disks, etc., none of which are specified here.

[0040] Specifically, aggregate files are evaluated based on their access frequency (e.g., access count in the last 7 days) and importance (e.g., business-criticality tags). High-frequency access and high-importance aggregate files are classified as hot data and retained in local high-speed storage (e.g., SSD); low-frequency access and low-importance aggregate files are classified as cold data and migrated to low-cost object storage (e.g., cloud disk).

[0041] In the above process, the embodiments of the present invention achieve optimal configuration of storage resources through a hierarchical storage strategy for hot and cold data, enabling high-frequency access data to respond quickly, while reducing the storage cost of low-frequency access data, providing an economical and efficient storage solution, and significantly improving the overall performance and economy of the system.

[0042] Step 140: When downloading the target file, the file storage address is queried in the small file index based on the unique identifier. If the query fails, the aggregate file where the target file is located is located. The target file is located based on the index information in the aggregate file. If the aggregate file is not local, it is downloaded to a temporary directory in the cloud and then parsed.

[0043] In one possible implementation, when downloading the target file, the storage address of the target file is queried in the small file index based on its unique identifier. If the query is successful, the file is downloaded and returned directly. If the query fails, the aggregate file containing the target file is obtained based on its storage address in the small file index. The target file is then located using the index information in the aggregate file, downloaded, and returned. If the aggregate file is not stored locally, it is downloaded from the cloud to a temporary directory, and the target file is parsed using the offset in the index information.

[0044] Specifically, when downloading a target file, the system first queries the small file index based on the target file's unique identifier to obtain its storage address. If the query is successful, the target file is downloaded directly and returned. If the query fails, the system retrieves the storage location of the aggregate file containing the target file. If the aggregate file is local, the target file is read directly from high-speed storage based on the index information within the aggregate file. If the aggregate file is not local, it is downloaded from the cloud to a temporary directory, and the target file is then parsed using the index information within the aggregate file. By utilizing the offset and length information in the structured index, the system directly locates the target file within the aggregate file, avoiding the need to decompress the entire aggregate file and improving file reading efficiency.

[0045] In the above process, the embodiments of the present invention achieve efficient downloading of target files through structured index information for rapid location and structured parsing technology. Even when faced with a large number of aggregated files, the target file can be quickly located, providing a fast and stable file download service, and significantly improving user experience and system performance.

[0046] Through the above process, this embodiment of the invention achieves efficient storage and fast access to small files by configuring parameters and defining rules, uploading and asynchronously aggregating small files, tiered storage of hot and cold data, and rapid downloading of target files. This method not only improves storage efficiency and economy but also significantly enhances user experience and system performance, providing an effective solution for the storage and management of massive amounts of small files.

[0047] In one application scenario, the small file access method of the present invention is used for small file access.

[0048] like Figure 2 As shown, the specific steps may include: First, in a multimedia content management platform, users upload a large number of small image files to the system for storage and management. Users perform file uploads through a web interface, and these small image files are initially sent to a file service. Upon receiving the files, the file service triggers an asynchronous packaging task based on the packaging configuration information (pre-set by the system administrator, including parameters such as a small file threshold of 500KB and upper and lower limits for package size).

[0049] Furthermore, the asynchronous packaging task asynchronously packages eligible small files on the pod's local disk and generates index information indicating the small files' relevant information within the packaged file, as well as a small file index indicating the packaged file's address. After packaging is complete, the packaged file is asynchronously uploaded to object storage (COS / OSS). This asynchronous packaging mechanism avoids blocking during file uploads caused by packaging operations, improving upload efficiency and user experience.

[0050] Furthermore, when a user needs to download an image file, they send a file download request to the file service. Based on the request, the file service first looks up the corresponding packaged file address in the small file index according to the filename. Then, it synchronously downloads the corresponding packaged file from object storage based on that address and saves it to the cloud disk mounted on the pod. This process involves capacity management to ensure that the cloud disk has sufficient space. Next, the file service reads the packaged file from the cloud disk and, based on the pre-generated index information, locates and reads the specific small file required by the user. Finally, it returns the small file to the user. Through index-based location and asynchronous download storage strategies, fast and accurate file downloads are achieved, while reasonable management of storage capacity ensures stable system operation.

[0051] Through the above process, this embodiment of the invention, using a multimedia content management platform as an application scenario, elaborates on the implementation process of the small file storage and retrieval method. During the file upload stage, an asynchronous packaging and upload mechanism efficiently processes the upload of small files and stores them in object storage. During the file download stage, synchronous download, capacity management, and index positioning technologies are used to quickly and accurately retrieve the files needed by the user from the cloud drive and return them. This method effectively solves the problems of low efficiency and complex storage management in traditional small file storage methods, improves the overall system performance and user experience, and reduces storage costs. It is suitable for various business scenarios that require processing a large number of small files.

[0052] like Figure 3 As shown, file uploading may include the following steps: Step S1: External request and file upload and saving.

[0053] Specifically, in a scenario where a user posts an attachment on a social media platform, the user publishes a post containing small files such as images or short documents. The user initiates an external request and uploads the file through the platform interface, and the file is transferred to a file service. The file service immediately saves the file to read-write storage, which includes a hotspot file directory and a temporary directory. At this point, the platform returns a success message to the user, as saving the file to read-write storage signifies a successful upload.

[0054] Furthermore, considering that user activity attachments on social media are often trending data and are frequently accessed, the read / write storage can define trending files based on access frequency and other dimensions. This embodiment of the invention improves user experience and ensures efficient access to trending data by quickly saving files to read / write storage and providing immediate feedback.

[0055] Step S2: Asynchronous packaging, index storage, and cleanup.

[0056] Specifically, the file service aggregates small files in the read / write storage temporary directory asynchronously and stores them in the temporary directory, based on the packaging configuration (such as enabling the small file packaging switch, setting small file thresholds, upper and lower limits on packaged file size, and upper limit on the number of files). Then, the aggregated files in the temporary directory are uploaded to backup storage (such as a cloud drive), and the packaged files in the temporary directory are immediately cleaned up after the upload is complete.

[0057] Meanwhile, the file service saves the small file index (containing information such as the unique identifier of the small file and its physical storage address) to the index storage. This embodiment of the invention optimizes storage space utilization through asynchronous packaging and timely cleanup; the index storage facilitates quick file location, meeting the efficient storage and access needs of social media platforms for a large number of small files.

[0058] Through the above process, this embodiment of the invention illustrates a method for storing and retrieving small files, using the scenario of user-uploaded dynamic attachments on a social media platform. From the moment a user uploads a file, the file service quickly saves it to read-write storage and provides feedback. Subsequently, it asynchronously packages and uploads it to backup storage while cleaning up temporary files and saving the index. This method meets the needs of different scenarios through parameterized configuration, utilizes read-write storage to ensure access to hot data, employs backup storage for persistent backup, and uses structured indexes to improve read performance. It effectively solves the problem of small file storage management on social media platforms, improves system efficiency and user experience, and has broad application value.

[0059] like Figure 4 As shown, downloading small files may include the following steps: Step S1: Small file download request and index lookup.

[0060] Specifically, in a user document attachment download scenario on an online document editing platform, a user initiates a request to download a small file (such as an image attachment in a document), i.e., a small file download operation. This request first reaches the file service, which, based on the file's unique identifier, immediately initiates a query operation to the index storage. The index storage stores small file index information, including the physical address of the small file in read / write storage, as well as the physical address of the aggregated small file in backup storage, and other key information.

[0061] Furthermore, through index lookup, the file service can accurately locate the storage location of small files and their aggregated files, providing a basis for subsequent reading operations. This embodiment of the invention, through a fast and accurate index lookup mechanism, lays the foundation for small file downloads, ensures the efficient initiation of the download process, and effectively avoids the time wasted by blind searching.

[0062] Step S2: Read small files.

[0063] Specifically, the file service begins reading the small file based on the retrieved index information. First, it searches for the small file in the read-write storage based on its physical address. If the small file exists in the read-write storage, since read-write storage is usually shared and can respond quickly in a distributed system, the file service directly reads it and returns it to the user.

[0064] If the small file does not exist in the read / write storage, the file service searches for the aggregate file in the aggregate file directory based on its physical address. If the aggregate file exists, the small file is read directly from it and returned. If the aggregate file does not exist in the aggregate file directory, the file service downloads the aggregate file from backup storage to the aggregate file directory, using file locks during this process to prevent duplicate downloads when multiple threads request the same file simultaneously.

[0065] After downloading, small files are read from the aggregated file directory and returned. A dedicated daemon thread manages the capacity of the aggregated file directory, cleaning up aggregated files according to capacity thresholds and expiration dates. This embodiment of the invention, through multi-layered file reading strategies and effective concurrency control and capacity management mechanisms, ensures the efficiency and stability of small file downloads, meeting the rapid attachment download needs of online document editing platform users.

[0066] Through the above process, this embodiment of the invention, using the download of document attachments on an online document editing platform as an example, details the small file download process. From the user initiating a download request, the file service queries the index to locate the file position, and then, depending on the situation, reads the small file from read / write storage, aggregated file directory, or backup storage. File locks and daemon threads ensure efficient and stable downloading. This method effectively solves the problem of small file downloading in distributed systems, improves download efficiency, ensures stable system operation, and provides strong support for online document editing services, possessing high practical value and promotional significance.

[0067] like Figure 5 As shown, cleaning up the aggregated file directory can include the following steps: Step S1: Capacity calculation and judgment.

[0068] Specifically, in a cloud storage service system, the aggregated file directory is used to store aggregated files downloaded from backup storage. After the system initiates the aggregated file directory cleanup process, it first calculates the directory capacity. Then, it checks if the directory capacity exceeds a limit. If not, it further searches for aggregated files whose creation time exceeds the validity period. If such files exist, these expired files are directly cleaned up.

[0069] In the above process, the embodiments of the present invention first calculate the directory capacity and make a preliminary judgment to quickly filter out cases that do not require complex processing, and directly clean up overdue files, thereby improving the cleaning efficiency and reducing unnecessary resource consumption.

[0070] Step S2: File search and renaming cleanup.

[0071] Specifically, if the directory size exceeds the limit, the system begins searching for the oldest created file. Considering that cloud storage services may operate on a distributed system, to reduce competition between instances, the oldest created file is renamed (search for oldest created file -> rename file). After renaming, the file is cleaned up. Then, it is determined whether the directory size after cleanup reaches the lower limit. If not, the next cleanup process begins; if it does, the current cleanup ends.

[0072] In the above process, the embodiments of the present invention effectively avoid instance contention in a distributed environment through renaming operations, ensuring the stability and reliability of the cleanup process. At the same time, the cyclic cleanup mechanism ensures that the directory capacity is always maintained within a reasonable range.

[0073] Through the above process, this embodiment of the invention uses the cleanup of aggregated file directories in a cloud storage service system as a scenario to elaborate on the cleanup process in detail. From capacity calculation and judgment, to file searching, renaming, and cleanup based on different situations, and then to cyclical cleanup until the capacity target is met, this method fully considers the distributed system environment. It reduces contention through renaming, effectively manages directory capacity using a cyclical cleanup mechanism, ensures the stable operation of the cloud storage service system, improves the utilization efficiency of storage resources, and provides strong support for the efficient development of cloud storage services. It has broad applicability and practical value.

[0074] The following are embodiments of the apparatus of the present invention, which can be used to execute the small file access method involved in the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the method embodiments of the small file access method involved in the present invention.

[0075] Please see Figure 6 This invention provides a small file access device 800.

[0076] The small file access device 800 includes, but is not limited to: a parameter and rule configuration module 810, an asynchronous aggregation and index generation module 830, a cold and hot classification and data storage module 850, and an index positioning and file reading module 870.

[0077] The parameter and rule configuration module 810 is used to obtain storage parameters according to business needs and define packaging rules based on the storage parameters; the storage parameters include small file threshold, upper and lower limits of packaging capacity and upper limit of packaging quantity.

[0078] The asynchronous aggregation and index generation module 830 is used to save newly uploaded small files to local read / write storage and immediately return a success response. After scanning the small files according to the packaging rules through asynchronous tasks, the small files are structured and stored in an aggregation file, and a small file index is generated. The aggregation file includes the index information of the small files.

[0079] The hot and cold data classification and data storage module 850 is used to divide aggregated files into hot data and cold data based on access frequency and importance. Hot data is kept in local high-speed storage, while cold data is migrated to low-cost object storage. Hot data and cold data represent data accessed frequently and accessed infrequently, respectively.

[0080] The index location and file reading module 870 is used to query the file storage address in the small file index based on the unique identifier when downloading the target file. If the query fails, it locates the aggregate file where the target file is located and locates the target file based on the index information in the aggregate file. If the aggregate file is not local, it is downloaded to a temporary directory in the cloud and then parsed.

[0081] It should be noted that the small file access provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the small file access device will be divided into different functional modules to complete all or part of the functions described above.

[0082] Furthermore, the embodiments of the small file access device and the small file access method provided in the above embodiments belong to the same concept, and the specific way in which each module performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0083] Figure 7 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0084] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on having... Figure 7 One or more components of the exemplary electronic device 2000 shown.

[0085] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 7 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0086] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0087] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 7 As shown, this does not constitute a specific limitation.

[0088] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0089] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0090] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 7 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, a small file access device may be considered as an application program 253 deployed on the electronic device 2000.

[0091] Data 255 may be signal information, etc., and is stored in memory 250.

[0092] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, a small file access method can be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0093] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0094] Please see Figure 8 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0095] exist Figure 8 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0096] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0097] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0098] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0099] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0100] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0101] The computer-readable instructions are executed by one or more processors 4001 to implement the small file access methods in the above embodiments.

[0102] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the small file access method described above.

[0103] This invention provides a computer program product including computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the small file access method described above.

[0104] Compared with related technologies, the beneficial effects of the present invention are: 1. This invention can efficiently manage the storage and access of small files; by asynchronously packaging small files into aggregate files and storing them in backup storage, while using read-write storage to save hot data, the number of small files is reduced, storage efficiency is improved, and small files can be quickly located and read based on index information, meeting the needs for efficient access to small files in different scenarios.

[0105] 2. This invention features flexible parameter configuration; by parameterizing key factors such as small files and file aggregation, such as enabling small file packaging switches, setting small file thresholds, upper and lower limits of packaged file capacity and quantity, it can meet the small file storage needs of different business scenarios, adapt to scenarios with different bandwidth and disk concurrency performance, and improve the versatility and applicability of the solution.

[0106] 3. This invention ensures the stability of file operations in a distributed system. In a distributed system, file locks are used to lock the aggregated files when downloading them from backup storage, avoiding duplicate downloads when multiple threads request them simultaneously. This ensures the consistency and stability of file operations and reduces system conflicts and resource waste.

[0107] 4. This invention achieves reasonable capacity management of aggregated file directories; by setting a capacity threshold for aggregated file directories and an expiration date for aggregated files, and by having a dedicated daemon thread monitor and clean them up, when the directory capacity exceeds the threshold, aggregated files are cleaned up according to the rules, and expired files are also cleaned up, ensuring that the capacity of the aggregated file directory is maintained at a reasonable level and improving the utilization efficiency of storage resources.

[0108] 5. This invention reduces competition between distributed system instances; during the aggregated file cleanup process, considering the distributed system environment, the files to be cleaned are renamed, reducing competition between instances, ensuring the smooth progress of the cleanup process, and improving the overall reliability and performance of the system.

[0109] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0110] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for storing and retrieving small files, characterized in that, The method includes: Obtain storage parameters according to business needs, and define packaging rules based on the storage parameters; the storage parameters include small file threshold, upper and lower limits of packaging capacity, and upper limit of packaging quantity; The newly uploaded small files are saved to local read / write storage and a success response is returned immediately. After scanning the small files according to the packaging rules through an asynchronous task, the small files are structured and stored in an aggregate file, and a small file index is generated. The aggregate file includes the index information of the small files. The aggregated files are divided into hot data and cold data based on access frequency and importance. The hot data is kept in local high-speed storage, while the cold data is migrated to low-cost object storage. The hot data and cold data represent data accessed frequently and accessed infrequently, respectively. When downloading a target file, the file storage address is queried in the small file index based on the unique identifier. If the query fails, the aggregate file where the target file is located is located. The target file is located based on the index information in the aggregate file. If the aggregate file is not local, it is downloaded to a temporary directory in the cloud and then parsed.

2. The small file access method as described in claim 1, characterized in that, The step of obtaining storage parameters according to business needs includes: The small file threshold is dynamically adjusted according to the business scenario. The dynamic adjustment includes setting the small file threshold to 500KB in the image storage scenario to distinguish between thumbnails and original images, and setting the small file threshold to 10KB in the log file scenario to optimize the storage of massive small logs.

3. The small file access method as described in claim 1, characterized in that, The step of obtaining storage parameters according to business requirements and defining packaging rules based on the storage parameters includes: The storage parameters are determined by analyzing the business scenario, and then converted into packaging rules through the configuration management module. The packaging rules include the conditions for file aggregation, the capacity range of the aggregated files, and the file number limit. The file number limit is used to avoid a single aggregated file from becoming too large.

4. The small file access method as described in claim 1, characterized in that, The step of scanning the small files according to the packaging rules via an asynchronous task, storing the small files in a structured manner into an aggregate file, and generating a small file index includes: The small files in storage are periodically scanned and read / written by an asynchronous task. The small files are aggregated into aggregate files that meet the set criteria according to the packaging rules. Corresponding index information is generated in the aggregate files, and a small file index is generated. The index information includes the unique identifier of the small file, the offset of the small file, and the length of the small file. The small file index includes the unique identifier, the storage address of the small file, and the storage address of the aggregate file.

5. The small file access method as described in claim 4, characterized in that, The step of structuring and storing the small files into an aggregate file includes: The index information of the small files is stored in a structured form of key-value pairs; the unique identifier of the small file includes the filename; the offset refers to the starting position of the small file in the aggregate file; the length refers to the size of the small file data; the key-value pair format is used to locate the small file by index.

6. The small file access method as described in claim 1, characterized in that, The aggregation files are divided into hot data and cold data based on access frequency and importance, including: The aggregated files are categorized into hot data and cold data based on their access frequency and importance; the access frequency includes the number of accesses in the last 7 days; the importance includes business-critical tags; the local high-speed storage includes SSDs; and the low-cost objects include cloud disks.

7. The small file access method as described in claim 1, characterized in that, When downloading the target file, the file storage address is queried in the small file index based on the unique identifier. If the query fails, the aggregate file containing the target file is located, and the target file is located based on the index information in the aggregate file. If the aggregate file is not local, it is downloaded to a temporary directory in the cloud and then parsed, including: When downloading a target file, the storage address of the target file is queried in the small file index based on the unique identifier of the target file. If the query is successful, the file is downloaded directly and returned. If the query fails, the aggregate file where the target file is located is obtained based on the aggregate file storage address in the small file index. After locating the target file based on the index information in the aggregate file, the file is downloaded and returned. If the aggregate file is not stored locally, it is downloaded from the cloud to a temporary directory and the target file is parsed based on the offset in the index information.

8. A small file storage and retrieval device, characterized in that, The device includes: The parameter and rule configuration module is used to obtain storage parameters according to business needs and define packaging rules according to the storage parameters; the storage parameters include small file threshold, upper and lower limits of packaging capacity, and upper limit of packaging quantity; the asynchronous aggregation and index generation module is used to save newly uploaded small files to local read and write storage and immediately return a success response, scan the small files according to the packaging rules through asynchronous tasks, store the small files in an aggregate file in a structured manner, and generate a small file index; the aggregate file includes the index information of the small files; The hot / cold data classification and data storage module is used to divide the aggregated file into hot data and cold data based on access frequency and importance, retain the hot data in local high-speed storage, and migrate the cold data to low-cost object storage; the hot data and cold data represent data accessed frequently and accessed infrequently, respectively. The index location and file reading module is used to query the file storage address in the small file index based on the unique identifier when downloading the target file. If the query fails, the module locates the aggregate file where the target file is located and locates the target file based on the index information in the aggregate file. If the aggregate file is not local, it is downloaded to a temporary directory in the cloud and then parsed.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the small file access method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the small file access method as described in any one of claims 1 to 7.

Images