File processing method and apparatus, computer storage medium, and program product

Abstract

Embodiments of the present application provide a file processing method and device, a computer storage medium and a program product. The method is applied to a Manila component of OpenStack. The Manila component interfaces with multiple storage clusters. The method comprises the following steps: when a first request for creating a target shared file system is received, obtaining capacity information of the multiple storage clusters; the first request comprises a first capacity required to be applied in a first storage pool and a second capacity required to be applied in a second storage pool, the first storage pool has a read-write speed greater than that of the second storage pool, and each storage cluster comprises a first storage pool and a second storage pool; determining a target storage cluster meeting a creation condition from the multiple storage clusters according to the first capacity, the second capacity and the capacity information; and creating the target shared file system in the target storage cluster according to the first capacity and the second capacity.

Description

Technical Field

[0001] This application relates to the field of cloud storage technology, and in particular to a file processing method, apparatus, computer storage medium, and program product. Background Technology

[0002] OpenStack is an open-source cloud computing management platform. Manila is the open-source component within OpenStack responsible for file storage, managing the entire lifecycle of file storage resources. However, currently, shared file systems created using the Manila component lack cold / hot pool tiering capabilities, making it impossible to utilize both cold and hot pool resources simultaneously, resulting in low resource utilization. Summary of the Invention

[0003] This application provides a file processing method, apparatus, computer storage medium, and program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a file processing method applied to the Manila component of OpenStack, wherein the Manila component interfaces with multiple storage clusters, and the method includes: When a first request to create a target shared file system is received, the capacity information of the multiple storage clusters is obtained; the first request includes the first capacity to be applied for in the first storage pool and the second capacity to be applied for in the second storage pool. The read and write speed of the first storage pool is greater than that of the second storage pool. Each storage cluster includes a first storage pool and a second storage pool. Based on the first capacity, the second capacity, and the capacity information, a target storage cluster that meets the creation conditions is determined from the multiple storage clusters. The target shared file system is created in the target storage cluster according to the first capacity and the second capacity.

[0005] This application provides a file processing device applied to the Manila component of OpenStack. The Manila component interfaces with multiple storage clusters. The device includes: The acquisition module is used to acquire the capacity information of the multiple storage clusters when it receives the first request to create a target shared file system; the first request includes the first capacity to be applied for in the first storage pool and the second capacity to be applied for in the second storage pool, the read and write speed of the first storage pool is greater than the read and write speed of the second storage pool, and each storage cluster includes a first storage pool and a second storage pool. The determination module is used to determine the target storage cluster that meets the creation conditions from the multiple storage clusters based on the first capacity, the second capacity and the capacity information; A creation module is used to create the target shared file system in the target storage cluster according to the first capacity and the second capacity.

[0006] This application provides a file processing device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the file processing method provided by one or more of the aforementioned technical solutions.

[0007] This application provides a computer storage medium storing a computer program; when the computer program is executed, it can implement the file processing method provided by one or more of the aforementioned technical solutions.

[0008] This application provides a computer program product, including a computer program that, when executed by a processor, implements the file processing method provided by one or more of the aforementioned technical solutions.

[0009] The file processing method provided in this application, upon receiving a first request to create a target shared file system, can obtain capacity information from multiple storage clusters. By using the requested capacity for the first and second storage pools specified in the request, along with the obtained capacity information for each storage cluster, a target storage cluster that simultaneously meets the capacity requirements of both storage pools can be determined. Because the read / write speed of the first storage pool is greater than that of the second storage pool, the first storage pool can be defined as a hot pool, and the second storage pool as a cold pool. This means that users do not need to create separate shared file systems for the hot and cold pools; a single creation operation can complete the creation of a shared file system spanning both storage pools. The created shared file system possesses tiered hot and cold storage capabilities, fully utilizing the resources of both storage pools and improving resource utilization. Furthermore, the single creation process simplifies the operation and improves creation efficiency. Additionally, since the first storage pool has a faster read / write speed, it is suitable for storing frequently accessed data, while the second storage pool has a slower read / write speed, making it suitable for storing low-frequency archived data. This ensures that the created shared file system can meet the diverse needs of various business scenarios. Attached Figure Description

[0010] Figure 1 A schematic diagram of a process for creating a shared file system; Figure 2 A flowchart illustrating a document processing method provided in an embodiment of this application; Figure 3A flowchart illustrating another document processing method provided in this application embodiment; Figure 4 A timing flowchart for file processing is provided for embodiments of this application; Figure 5 A timing flowchart of yet another file processing method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the composition structure of a document processing device according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a file processing device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions in this application will now be clearly and completely described with reference to the accompanying drawings.

[0012] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the present application can be implemented in any combination.

[0013] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or system that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or system. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the system, such as a portion of a processor, a portion of a program, or software, etc.) in the method or system that includes that element.

[0014] For example, the document processing method provided in the embodiments of this application includes a series of steps, but the document processing method provided in the embodiments of this application is not limited to the steps described. Similarly, the document processing apparatus provided in the embodiments of this application includes a series of modules, but the document processing apparatus provided in the embodiments of this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0015] In the following description, the terms first / second are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that first / second may be interchanged in a particular order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0017] In related technologies, the Manila component, based on the Network File System (NFS) protocol or the Common Internet File System (CIFS) protocol, calls the manila-driver to send instructions to the cold pool or hot pool to complete operations such as the creation, authorization, and deletion of the shared file system (share). The share is provided to users as a shared file system.

[0018] See Figure 1 The existing Manila component's process for creating a shared file system includes: manila-api receiving a request to create a shared file system; manila-scheduler selecting a storage cluster. Specifically, manila-scheduler can use the CapacityFilter filtering algorithm and the CapacityWeigher weighing algorithm to select a suitable storage cluster for creating the shared file system; and manila-share creating a shared file system (share) instance by calling the manila-driver to send the creation command to either the cold pool or the hot pool.

[0019] As can be seen, the shared file system created through the above process will be directly assigned to either the cold pool or the hot pool, lacking the ability to classify hot and cold pools, and cannot utilize the resources of both pools simultaneously, resulting in low resource utilization.

[0020] To address the above problems, the following embodiments are proposed.

[0021] In some embodiments of this application, the file processing method can be implemented by a processor in a file processing device. The processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor.

[0022] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. Where applicable, user personal information has been subjected to de-identification and / or anonymization and / or encryption technical processing.

[0023] Figure 2 This is a flowchart illustrating a file processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, the process may include: Step 100: Upon receiving a request to create a target shared file system, obtain the capacity information of multiple storage clusters.

[0024] In this embodiment, the file processing method can be applied to the OpenStack Manila component; the Manila component is an OpenStack file sharing service component, which may include manila-api, manila-scheduler, and manila-share. In addition, the Manila component can connect to multiple storage clusters to provide file storage services by deploying the manila-driver; here, the type of storage cluster is not specifically limited, for example, it can be a glusterfs storage cluster.

[0025] For example, the target shared file system refers to the shared file system that the user wants to create. When the user wants to create the target shared file system, he / she can initiate a request to create the target shared file system through the terminal. The terminal sends the request to the application programming interface manila-api of the Manila component. At this time, the manila-api of Manila will receive the request initiated by the user to create the target shared file system.

[0026] For example, manila-api can receive user requests via REST API (Representational State Transfer API); where REST API is a web application programming interface based on the HTTP protocol.

[0027] In this embodiment of the application, the request may include a first capacity that the user needs to apply for in the first storage pool and a second capacity that the user needs to apply for in the second storage pool; wherein, the read / write speed of the first storage pool is greater than the read / write speed of the second storage pool.

[0028] For example, the first storage pool, also known as the hot pool, refers to a storage cluster that uses high-performance storage media such as solid-state drives (SSDs) to store frequently accessed data. It is suitable for high-performance, high-frequency access scenarios and has a higher storage cost. The second storage cluster, also known as the cold pool, refers to a storage cluster that uses large-capacity, low-performance storage media such as hard disk drives (HDDs) to store infrequently accessed data. It is suitable for large-capacity, low-frequency access scenarios and has a relatively lower storage cost.

[0029] In practical applications, users can specify a first capacity and a second capacity according to their own needs when making a request; the first capacity can be represented as hot_data_size, and the second capacity can be represented as cold_data_size; the first capacity and the second capacity reflect the user's capacity allocation for hot and cold tiered storage.

[0030] For example, when making a request, users can also specify parameters such as file storage location and file migration time according to their own needs, without specific limitations here; file storage location indicates the storage pool type (hot pool or cold pool) where the file is initially stored, and file migration time indicates the time rule for automatically triggering file migration. For example, if a file has not been accessed for a certain period of time (e.g., 100 days), it will be automatically migrated to the cold pool.

[0031] It should be noted that users can also omit specifying the file storage location and file migration time; if the user does not specify these two parameters, the system default values ​​will be used; the default file storage location can be represented as tier_place and the default file migration time can be represented as tier_migrate_expiration; this will be used as an example for explanation later.

[0032] For example, see Figure 3 and Figure 4 When manila-api receives a request to create a target shared file system, it can also check the hot / cold classification parameters carried in the request, such as the first capacity and second capacity. For example, it checks whether the parameter format is correct, whether the values ​​are within a reasonable range, and whether they meet the capacity limits of the storage cluster, to avoid subsequent scheduling or creation failures due to incorrect parameters. If the parameter check passes, the request, along with the related hot / cold classification parameters such as the first capacity, second capacity, default file storage location, and default file migration time, can be sent to manila-scheduler, providing a complete decision-making basis for subsequent storage cluster scheduling.

[0033] In this embodiment, when the manila-scheduler receives a request, it can obtain capacity information for multiple storage clusters. Each storage cluster may include a first storage pool and a second storage pool; that is, it includes a hot pool and a cold pool.

[0034] For example, a storage cluster is also called a hot / cold tiered storage cluster; see [link to relevant documentation]. Figure 1 Currently, the cold and hot pools that Manila components interface with are two separate storage clusters. See [link / reference]. Figure 3 In this embodiment of the application, the Manila component can manage separate cold pools and separate hot pools through configuration information such as Internet Protocol (IP) addresses, forming multiple sets of cold and hot graded storage clusters, that is, multiple sets of storage clusters, each of which includes a cold pool and a hot pool.

[0035] In this embodiment of the application, the capacity information of each storage cluster may include the capacity information of the first storage pool and the capacity information of the second storage pool in the storage cluster; wherein, the capacity information of the first storage pool and the capacity information of the second storage pool may include the total capacity, the allocated capacity, and the remaining available capacity, respectively.

[0036] In some embodiments, obtaining capacity information for multiple storage clusters may include: for each storage cluster, the manila-scheduler queries the memory cache to see if capacity information for the first and second storage pools in the storage cluster exists; if it exists, the capacity information for the first and second storage pools in the storage cluster is obtained from the memory cache; if it does not exist, the storage cluster is locked, and the capacity information for the first and second storage pools is obtained from the locked storage cluster.

[0037] Here, the memory cache can be a Memcached cache or other types of memory cache; no specific limitation is made here.

[0038] For example, see Figure 4 The manila-share initiates a periodic query to the manila-driver to retrieve storage cluster capacity information. Upon receiving this query, the manila-driver sends a query command to the storage cluster to obtain capacity information and reports the retrieved capacity information to the manila-scheduler. After receiving the capacity information reported by the manila-driver, the manila-scheduler stores it in its memory cache.

[0039] Understandably, because the above query process is performed periodically, the capacity information of each storage cluster stored in the manila-scheduler's memory cache is updated periodically; that is, the memory cache stores the latest capacity information of each storage cluster.

[0040] In this embodiment of the application, when the manila-scheduler receives a request to create a target shared file system, for each storage cluster, it can first query the memory cache to see if the capacity information of the first storage pool and the second storage pool in the storage cluster exists.

[0041] For example, if capacity information exists in the memory cache, the capacity information of the first and second storage pools in the storage cluster is directly retrieved from the memory cache; this improves information retrieval efficiency. If capacity information does not exist in the memory cache, the storage cluster is locked, and the capacity information of the first and second storage pools is retrieved from the locked storage cluster.

[0042] For example, a coordination.synchronized distributed lock can be used to lock the storage cluster, or other types of locks can be used, without being specifically limited here; the purpose of locking is to ensure the consistency and accuracy of capacity information acquisition in scenarios with multiple concurrent requests.

[0043] It should be noted that after obtaining the capacity information of the first and second storage pools from the locked storage cluster, the storage cluster can be unlocked to avoid occupying cluster resources for a long time.

[0044] Step 101: Based on the first capacity, the second capacity, and the capacity information, determine the target storage cluster that meets the creation conditions from multiple storage clusters.

[0045] In this embodiment of the application, after obtaining the capacity information of multiple storage clusters, the manila-scheduler can determine the target storage cluster that meets the creation conditions from the multiple storage clusters based on the first capacity, the second capacity, and the capacity information.

[0046] In some embodiments, determining the target storage cluster that meets the creation conditions from multiple storage clusters based on the first capacity, the second capacity, and capacity information may include: for each storage cluster, the manila-scheduler determines a first super-splitting ratio based on the first capacity, the total capacity of the first storage pool in the storage cluster, and the allocated capacity, and determines a second super-splitting ratio based on the second capacity, the total capacity of the second storage pool in the storage cluster, and the allocated capacity; using the first super-splitting ratio and the second super-splitting ratio, filtering the multiple storage clusters to obtain filtered storage clusters; and determining the target storage cluster that meets the creation conditions based on the filtered storage clusters.

[0047] For example, as can be seen from the above, capacity information can include total capacity and allocated capacity. For each storage cluster, the manila-scheduler can determine the first super-provisioned ratio (hot_provisioned_ratio) based on the first capacity, the total capacity of the first storage pool in the storage cluster, and the allocated capacity. The formula for determining this ratio is as follows: hot_provisioned_ratio=((host_state.hot_provisioned_capacity_gb+hot_share_size) / hot_total) Here, host_state.hot_provisioned_capacity_gb represents the allocated capacity of the first storage pool in the storage cluster, hot_total represents the total capacity of the first storage pool in the storage cluster, and hot_share_size represents the first capacity.

[0048] Similarly, for each storage cluster, manila-scheduler can determine the second super-provisioned ratio (cold_provisioned_ratio) based on the second capacity, the total capacity of the second storage pool in the storage cluster, and the allocated capacity. The formula for determining this ratio is as follows: cold_provisioned_ratio=((host_state.cold_provisioned_capacity_gb+cold_share_size) / cold_total) Here, host_state.cold_provisioned_capacity_gb represents the allocated capacity of the second storage pool in the storage cluster, cold_total represents the total capacity of the second storage pool in the storage cluster, and hot_share_size represents the second capacity.

[0049] For example, after obtaining the first super-resolution ratio corresponding to the first storage pool in the storage cluster, it can be compared with the first preset super-resolution ratio. If the comparison result shows that the first super-resolution ratio is greater than the first preset super-resolution ratio, it is determined that it has been over-resolution, indicating that the first storage pool of the storage cluster cannot meet the capacity requirements. At this time, the storage cluster needs to be filtered out.

[0050] Similarly, after obtaining the second super-resolution ratio corresponding to the second storage pool in the storage cluster, it can be compared with the second preset super-resolution ratio. If the comparison result shows that the second super-resolution ratio is greater than the second preset super-resolution ratio, it is determined that it has been over-resolution, indicating that the second storage pool of the storage cluster cannot meet the capacity requirements. At this time, the storage cluster needs to be filtered out.

[0051] It should be noted that once it is determined that the first or second storage pool of a storage cluster does not meet the capacity requirements, the storage cluster is directly filtered out without needing to determine whether the other storage pool meets the capacity requirements, thus improving processing efficiency.

[0052] Here, the values ​​of the first preset super-resolution ratio and the second preset super-resolution ratio can be the same or different. They can be set according to the actual situation. There is no specific limitation here. For example, the values ​​of both can be 1.

[0053] In this embodiment of the application, by performing the above-mentioned filtering process on multiple storage clusters, a filtered storage cluster can be obtained. It should be noted that the number of filtered storage clusters can be one or more. If the number of filtered storage clusters is determined to be one, then the filtered storage cluster is directly determined as the target storage cluster that meets the creation conditions. If the number of filtered storage clusters is determined to be multiple, then it is necessary to determine the total score corresponding to each storage cluster, and then determine the target storage cluster based on the total score corresponding to each filtered storage cluster. The following is an exemplary description of this process.

[0054] For example, when there are multiple filtered storage clusters, determining the target storage cluster that meets the creation conditions based on the filtered storage clusters may include: for each filtered storage cluster, the manila-scheduler determines a first score based on a first preset super-resolution ratio, the total capacity of the first storage pool in the storage cluster, the allocated capacity, and the first reservation percentage, and determines a second score based on a second preset super-resolution ratio, the total capacity of the second storage pool in the storage cluster, the allocated capacity, and the second reservation percentage; the first score and the second score are weighted and summed to obtain the total score corresponding to each filtered storage cluster; and the target storage cluster is determined based on the total score corresponding to each filtered storage cluster.

[0055] Here, the first reservation percentage refers to the proportion of capacity in the first storage pool that needs to be reserved and not used externally; the second reservation percentage refers to the proportion of capacity in the second storage pool that needs to be reserved and not used externally. The values ​​of the two can be the same or different, and can be set according to the actual situation. There is no specific limitation here. For example, the values ​​of the two can both be 20%.

[0056] In this embodiment of the application, for each storage cluster, the manila-scheduler can determine the first score hot_free based on the first preset super-score ratio, the total capacity of the first storage pool in the storage cluster, the allocated capacity, and the first reserved percentage. The determination formula is as follows: hot_free=math.floor(hot_total*host_state.hot_max_over_subscription_ratio-host_state.hot_provisioned_capacity_gb-hot_total*reserved) Where hot_total*host_state.hot_max_over_subscription_ratio represents the first preset oversubscription ratio, and hot_total*reserved represents the first reserved percentage.

[0057] Similarly, for each storage cluster, manila-scheduler can determine the second score cold_free based on the second preset super-score ratio, the total capacity of the second storage pool in the storage cluster, the allocated capacity, and the second reserved percentage. The determination formula is as follows: cold_free=math.floor(cold_total*host_state.cold_max_over_subscription_ratio-host_state.cold_provisioned_capacity_gb-cold_total*reserved) Where host_state.cold_max_over_subscription_ratio represents the second preset oversubscription ratio, and cold_total*reserved represents the second reserved percentage.

[0058] For example, after obtaining the first score corresponding to the first storage pool and the second score corresponding to the second storage pool in each storage cluster, the first score and the second score can be weighted and summed using weighting parameters to obtain the total score total_free for each storage cluster after filtering.

[0059] Here, the weighting parameters can include the weighting parameters corresponding to the first score and the second score; their values ​​can be set according to the actual situation, and there is no specific limitation here. For example, the weighting ratio of the two can be set to 7:3, that is, the weighting parameter corresponding to the first score is 0.7, and the weighting parameter corresponding to the second score is 0.3; at this time, the formula for determining the total score total_free is as follows: total_free = hot_free * 0.7 + cold_free * 0.3 In this embodiment of the application, after obtaining the total score corresponding to each filtered storage cluster according to the above steps, the target storage cluster can be determined based on the total score.

[0060] In some embodiments, determining the target storage cluster based on the total score corresponding to each filtered storage cluster may include: sorting the filtered multiple storage clusters in descending order according to the total score to obtain a sorting result; and determining the storage cluster ranked first in the sorting result as the target storage cluster.

[0061] For example, after obtaining the total score for each filtered storage cluster, the filtered storage clusters can be sorted in descending order according to the total score to obtain the sorting result; then, the storage cluster ranked first in the sorting result is determined as the target storage cluster; here, the storage cluster ranked first is the storage cluster with the highest total score.

[0062] It should be noted that if there are multiple storage clusters with the highest total scores, one of them will be randomly selected as the target storage cluster.

[0063] Step 102: Create a target shared file system in the target storage cluster according to the first and second capacities.

[0064] In this embodiment, after determining the target storage cluster according to the above steps, the manila-scheduler can create a target shared file system in the target storage cluster according to the first capacity and the second capacity; for ease of understanding, the following is combined with Figure 4 An example is provided.

[0065] For example, see Figure 4 The manila-scheduler can send instructions to create a target shared file system to the corresponding manila-share via a Remote Procedure Call (RPC) message mechanism. Upon receiving the instruction, the manila-share can invoke the manila-driver. The manila-driver, based on the IP addresses of the cold and hot pools in the target storage cluster, sends the instructions via SSH to the cold and hot pools in the target storage cluster, completing the actual creation of the target shared file system. This creation operation includes creating a shared file system of a first capacity in the hot pool and a shared file system of a second capacity in the cold pool.

[0066] For example, after the target shared file system is created, return information for the creation operation can be generated accordingly. If the return information indicates an error, the status of the target shared file system is set to error, and the mount address is not returned. If the return information indicates a normal status, the mount address is returned, and the status of the target shared file system is set to available.

[0067] Correspondingly, if the returned information indicates that everything is normal, the target storage cluster will return the mount address to the corresponding manila-driver. The manila-driver will then synchronize the mount address to manila-share. manila-share will then return the mount address to manila-scheduler via RPC. manila-scheduler will then synchronize the mount address to manila-api, which will then return it to the user who initiated the creation request.

[0068] As can be seen from this embodiment, users can create a shared file system across two storage pools with a single creation operation. The created shared file system has tiered hot and cold storage capabilities, which can fully utilize the resources of both storage pools and improve resource utilization. In addition, the single creation simplifies the operation process and improves creation efficiency. Furthermore, since the first storage pool has a faster read and write speed, it is suitable for storing frequently accessed data, while the second storage pool has a slower read and write speed, making it suitable for storing infrequently accessed archived data. In this way, it can be ensured that the created shared file system can meet the needs of diverse business scenarios.

[0069] In some embodiments, the above method may further include: after creating the target shared file system, periodically detecting the operation frequency of each file located in the first storage pool in the target storage cluster; comparing the operation frequency with a frequency threshold to obtain a first comparison result; determining the first file whose operation frequency is less than the frequency threshold in the first comparison result, and migrating the first file from the first storage pool in the target storage cluster to the second storage pool in the target storage cluster.

[0070] For example, the manila-driver can manage the first storage pool (hot pool) and the second storage pool (cold pool) in the target storage cluster based on cluster configuration information, such as the IP address, storage type, capacity specifications, etc. of each storage pool.

[0071] In this embodiment of the application, after creating the target shared file system, the Manila component can also create a scheduled task to periodically detect the operation frequency of each file located in the first storage pool in the target storage cluster; here, the time interval for periodic detection can be set according to the actual situation, and this embodiment of the application does not make a specific limitation on this, for example, it can be 100 days, etc.

[0072] For example, after detecting the operation frequency of a file in the first storage pool, the operation frequency of the file can be compared with a frequency threshold. If it is determined that the operation frequency of the file is less than the frequency threshold, a data migration tool, such as the JuiceFS data migration tool, can be used to automatically migrate the file from the first storage pool to the second storage pool; thus realizing the dynamic migration of low-frequency access data and balancing storage performance and cost.

[0073] Here, the frequency threshold value can be customized according to the actual situation, and no specific limitation is made here; for example, the operation frequency in the past 100 days is less than 3 times.

[0074] Similarly, the Manila component can also create another scheduled task to periodically detect the operation frequency of each file located in the second storage pool in the target storage cluster; compare the operation frequency with a preset frequency threshold to obtain the comparison result; determine the second file whose operation frequency is greater than the preset frequency threshold in the comparison result, and migrate the second file from the second storage pool in the target storage cluster to the first storage pool in the target storage cluster.

[0075] Here, the time interval for the periodic detection can be set according to the actual situation. This application embodiment does not make a specific limitation on this. For example, it can be every hour.

[0076] For example, after detecting the operation frequency of a file in the second storage pool, the operation frequency of the file can be compared with a preset frequency threshold. If it is determined that the operation frequency of the file is greater than the preset frequency threshold, a data migration tool, such as the JuiceFS data migration tool, can be used to automatically migrate the file from the second storage pool to the first storage pool, thus ensuring the read and write performance of high-frequency access data.

[0077] Here, the preset frequency threshold value can be customized according to the actual situation, and no specific limitation is made here; for example, the operation frequency per hour is greater than 10 times.

[0078] It should be noted that both the first and second files mentioned above are stored under the same directory. Through automatic migration, the first file can be moved from the hot pool directory to the cold pool directory, and the second file can be moved from the cold pool directory to the hot pool directory. Figure 5 As shown.

[0079] As can be seen from the embodiments of this application, the Manila component can automatically migrate data between the hot pool and the cold pool based on the comparison results of file operation frequency and frequency threshold. It migrates low-frequency access data stored in the hot pool to the cold pool, reducing the occupation of high-performance storage resources and reducing storage costs; it migrates high-frequency access data stored in the cold pool to the hot pool, so that the hot pool always carries core data, maximizing the read and write performance of high-frequency data and effectively balancing storage performance and cost.

[0080] In this embodiment of the application, in addition to automatically migrating files in the hot pool and cold pool at regular intervals, active migration can also be performed based on user requests; the active migration process is illustrated below.

[0081] In some embodiments, the method further includes: when receiving a user's migration request for the second file, migrating the second file according to the migration parameters in the migration request; the migration parameters include one or more of the following: the storage cluster identifier corresponding to the second file, the file path, and the type of storage pool to be migrated to.

[0082] For example, the Manila component also includes a migration interface, which can be an API interface added on top of the Manila component for receiving external migration requests.

[0083] In one alternative embodiment, the user can send a migration request for the second file through the terminal. The terminal will then send the request to the migration interface of the Manila component, at which point the Manila migration interface will receive the migration request initiated by the user.

[0084] For example, when the migration interface receives a user's migration request for a second file, it can migrate the second file according to the migration parameters in the migration request; wherein, the migration parameters include one or more of the following: the storage cluster identifier corresponding to the second file, the file path, and the storage pool type to be migrated to.

[0085] Here, the storage cluster identifier can be represented as share_id, which is used to identify the unique storage cluster to which the file belongs, ensuring that the migration operation is accurately applied to the specified storage cluster; the file path can be represented as file_path, which is used to determine the specific file to be migrated; the storage pool type to be migrated can be represented as pre_type, which is used to specify the storage location (cold pool or hot pool) of the file after migration.

[0086] In this embodiment, the Manila component allows users to actively specify the files to be migrated through migration parameters, and then use a data migration tool, such as JuiceFS, to migrate them between hot and cold pools. It should be noted that if the user-specified file does not exist, an error message is returned to the user through the migration interface to avoid invalid operations.

[0087] For example, a new API interface for querying file migration status is added to the Manila component. In this way, users can use this interface to query the migration status of any file in a specified shared file system, such as whether the migration is complete or failed. It should be noted that users can associate the corresponding shared file system with the storage cluster identifier share_id and use this as a range to query the migration status of the target file.

[0088] For example, an API interface for modifying system default parameters can be added to the Manila component. In this way, users can use this interface to customize and adjust the system default parameters, such as modifying the default storage location, default file migration time, and other parameters according to their own needs, so as to determine the data migration strategy that is more in line with their own business needs.

[0089] It should be noted that, in addition to creating the target shared file system according to the above embodiments, the Manila component involved in this application can also perform full lifecycle management operations such as authorization and deletion of the target shared file system, which will not be elaborated here.

[0090] Figure 6 This is a schematic diagram of the composition structure of a document processing device according to an embodiment of this application, as shown below. Figure 6 As shown, the file processing device 60 may include: an acquisition module 600, a determination module 601, and a creation module 602, wherein: The acquisition module 600 is used to acquire the capacity information of the multiple storage clusters when it receives the first request to create a target shared file system; the first request includes the first capacity to be applied for in the first storage pool and the second capacity to be applied for in the second storage pool, the read and write speed of the first storage pool is greater than the read and write speed of the second storage pool, and each storage cluster includes a first storage pool and a second storage pool. The determining module 601 is used to determine, based on the first capacity, the second capacity and the capacity information, a target storage cluster that meets the creation conditions from the multiple storage clusters; A creation module 602 is used to create the target shared file system in the target storage cluster according to the first capacity and the second capacity.

[0091] In some embodiments, the above-described apparatus further includes a migration module, the migration module being configured to: After creating the target shared file system, the operation frequency of each file located in the first storage pool in the target storage cluster is periodically detected; The operating frequency is compared with a frequency threshold to obtain a first comparison result; The first file whose operation frequency is less than the frequency threshold in the first comparison result is identified, and the first file is migrated from the first storage pool in the target storage cluster to the second storage pool in the target storage cluster.

[0092] In some embodiments, the migration module is further configured to: Upon receiving a second request to migrate a second file, the second file is migrated according to the migration parameters in the second request; the migration parameters include one or more of the following: the storage cluster identifier corresponding to the second file, the file path, and the type of storage pool to be migrated to.

[0093] In some embodiments, the Manila component includes a manila-scheduler, the capacity information includes total capacity and allocated capacity, and the determination module 601 is further configured to: For each storage cluster, the manila-scheduler determines a first super-distribution ratio based on the first capacity, the total capacity of the first storage pool in the storage cluster, and the allocated capacity, and determines a second super-distribution ratio based on the second capacity, the total capacity of the second storage pool in the storage cluster, and the allocated capacity. Using the first super-resolution ratio and the second super-resolution ratio, the multiple storage clusters are filtered to obtain the filtered storage clusters; Based on the filtered storage clusters, a target storage cluster that meets the creation conditions is determined.

[0094] In some embodiments, when the number of filtered storage clusters is multiple, the determining module 601 is further configured to: For each filtered storage cluster, the manila-scheduler determines a first score based on a first preset super-resolution ratio, the total capacity of the first storage pool in the storage cluster, the allocated capacity, and the first reserved percentage, and determines a second score based on a second preset super-resolution ratio, the total capacity of the second storage pool in the storage cluster, the allocated capacity, and the second reserved percentage. The first score and the second score are weighted and summed to obtain the total score for each filtered storage cluster. The target storage cluster is determined based on the total score of each filtered storage cluster.

[0095] In some embodiments, the determining module 601 is further configured to: Based on the total score, the filtered storage clusters are sorted in descending order to obtain the sorting result. The storage cluster that ranks first in the sorting results is determined as the target storage cluster.

[0096] In some embodiments, the Manila component includes a manila-scheduler, an acquisition module 600, and is further configured to: For each storage cluster, the manila-scheduler queries the memory cache to see if the capacity information of the first storage pool and the second storage pool in the storage cluster exists. If it exists, then obtain the capacity information of the first storage pool and the second storage pool in the storage cluster from the memory cache; If it does not exist, the storage cluster is locked, and the capacity information of the first storage pool and the second storage pool is obtained from the locked storage cluster.

[0097] In practical applications, the above-mentioned acquisition module 600, determination module 601, creation module 602 and migration module can all be implemented by a processor located in the file processing device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller and microprocessor.

[0098] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0099] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] Specifically, the computer program instructions corresponding to a file processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a file processing method in the storage media are read or executed by a file processing device, any of the file processing methods in the aforementioned embodiments are implemented.

[0101] Based on the same technical concept as the foregoing embodiments, see Figure 7 It illustrates a file processing device 700 provided in an embodiment of this application, which may include: a memory 701 and a processor 702; wherein, Memory 701 is used to store computer programs and data; The processor 702 is configured to execute a computer program stored in the memory to implement any of the file processing methods described in the foregoing embodiments.

[0102] In practical applications, the memory 701 mentioned above can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 702.

[0103] The processor 702 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that for different file processing devices, the electronic device used to implement the above processor function can also be other types, and this application embodiment does not specifically limit the specific types.

[0104] In some embodiments, this application also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the file processing methods described in the foregoing embodiments.

[0105] In some embodiments, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the file processing methods described in the foregoing embodiments.

[0106] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0107] The descriptions of the various embodiments above tend to emphasize the differences between them. Similarities or commonalities can be referred to interchangeably, and for the sake of brevity, they will not be repeated here. The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the various product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined to obtain new method or device embodiments without conflict.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable file processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable file processing device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable document processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A file processing method, characterized in that, The Manila component, applied to OpenStack, interfaces with multiple storage clusters. The method includes: When a first request to create a target shared file system is received, the capacity information of the multiple storage clusters is obtained; the first request includes the first capacity to be applied for in the first storage pool and the second capacity to be applied for in the second storage pool. The read and write speed of the first storage pool is greater than that of the second storage pool. Each storage cluster includes a first storage pool and a second storage pool. Based on the first capacity, the second capacity, and the capacity information, a target storage cluster that meets the creation conditions is determined from the multiple storage clusters. The target shared file system is created in the target storage cluster according to the first capacity and the second capacity.

2. The method according to claim 1, characterized in that, The method further includes: After creating the target shared file system, the operation frequency of each file located in the first storage pool in the target storage cluster is periodically detected; The operating frequency is compared with a frequency threshold to obtain a first comparison result; The first file whose operation frequency is less than the frequency threshold in the first comparison result is identified, and the first file is migrated from the first storage pool in the target storage cluster to the second storage pool in the target storage cluster.

3. The method according to claim 1, characterized in that, The method further includes: Upon receiving a second request to migrate a second file, the second file is migrated according to the migration parameters in the second request; the migration parameters include one or more of the following: the storage cluster identifier corresponding to the second file, the file path, and the type of storage pool to be migrated to.

4. The method according to claim 1, characterized in that, The Manila component includes a manila-scheduler, and the capacity information includes total capacity and allocated capacity. The step of determining the target storage cluster that meets the creation conditions from the multiple storage clusters based on the first capacity, the second capacity, and the capacity information includes: For each storage cluster, the manila-scheduler determines a first super-distribution ratio based on the first capacity, the total capacity of the first storage pool in the storage cluster, and the allocated capacity, and determines a second super-distribution ratio based on the second capacity, the total capacity of the second storage pool in the storage cluster, and the allocated capacity. Using the first super-resolution ratio and the second super-resolution ratio, the multiple storage clusters are filtered to obtain the filtered storage clusters; Based on the filtered storage clusters, a target storage cluster that meets the creation conditions is determined.

5. The method according to claim 4, characterized in that, When there are multiple filtered storage clusters, the step of determining the target storage cluster that meets the creation conditions based on the filtered storage clusters includes... For each filtered storage cluster, the manila-scheduler determines a first score based on a first preset super-resolution ratio, the total capacity of the first storage pool in the storage cluster, the allocated capacity, and the first reserved percentage, and determines a second score based on a second preset super-resolution ratio, the total capacity of the second storage pool in the storage cluster, the allocated capacity, and the second reserved percentage. The first score and the second score are weighted and summed to obtain the total score for each filtered storage cluster. The target storage cluster is determined based on the total score of each filtered storage cluster.

6. The method according to claim 5, characterized in that, The step of determining the target storage cluster based on the total score corresponding to each filtered storage cluster includes: Based on the total score, the filtered storage clusters are sorted in descending order to obtain the sorting result. The storage cluster that ranks first in the sorting results is determined as the target storage cluster.

7. The method according to claim 1, characterized in that, The Manila component includes manila-scheduler, and the step of obtaining the capacity information of the multiple storage clusters includes: For each storage cluster, the manila-scheduler queries the memory cache to see if the capacity information of the first storage pool and the second storage pool in the storage cluster exists. If it exists, then obtain the capacity information of the first storage pool and the second storage pool in the storage cluster from the memory cache; If it does not exist, the storage cluster is locked, and the capacity information of the first storage pool and the second storage pool is obtained from the locked storage cluster.

8. A document processing device, characterized in that, The Manila component, used in OpenStack, interfaces with multiple storage clusters. The device includes: The acquisition module is used to acquire the capacity information of the multiple storage clusters when it receives the first request to create a target shared file system; the first request includes the first capacity to be applied for in the first storage pool and the second capacity to be applied for in the second storage pool, the read and write speed of the first storage pool is greater than the read and write speed of the second storage pool, and each storage cluster includes a first storage pool and a second storage pool. The determination module is used to determine the target storage cluster that meets the creation conditions from the multiple storage clusters based on the first capacity, the second capacity and the capacity information; A creation module is used to create the target shared file system in the target storage cluster according to the first capacity and the second capacity.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images