File data migration method and device, equipment, storage medium and program

Abstract

The invention provides a file data migration method and device, equipment, a storage medium and a program. The method comprises the following steps: receiving object storage access information, wherein the object storage access information is used for requesting migration of target file data; according to the object storage access information, constructing mapping information between a file local directory of the file storage system and an object storage bucket of an object storage cluster; according to the file path under the local directory of the file, generating a target object identifier in the object storage cluster, the target object identifier being generated by converting a hierarchical structure of the file path; and migrating the target file data from the file storage system to an object storage cluster based on the mapping information and the target object identifier. According to the method, the file storage flexibility can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program for migrating file data. Background Technology

[0002] In large-scale distributed storage systems, enterprises or organizations need to manage massive amounts of data efficiently. In particular, when faced with the dual challenges of exponential growth in data volume and control of storage costs, tiered storage strategies become a core requirement.

[0003] In existing technologies, cloud computing platforms, big data analytics systems, or enterprise-level file-sharing platforms typically require users to store frequently accessed hot data in high-performance storage media (such as SSDs) and migrate cold data to low-cost storage media (such as disks or object storage). However, reliance on physical storage media or dedicated storage pools results in limited flexibility in file storage. Summary of the Invention

[0004] This application provides a file data migration method, apparatus, device, storage medium, and program to solve the technical problem of low flexibility in file storage.

[0005] Firstly, this application provides a file data migration method, the method comprising:

[0006] Receive object storage access information, which is used to request the migration of target file data;

[0007] Based on object storage access information, construct the mapping information between the local file directory of the file storage system and the object storage bucket of the object storage cluster;

[0008] Based on the file path in the local directory, generate the target object identifier in the object storage cluster. The target object identifier is generated by transforming the hierarchical structure of the file path.

[0009] Based on mapping information and target object identifiers, target file data is migrated from the file storage system to the object storage cluster.

[0010] In this embodiment, by using the object storage cluster as a cold data storage layer, efficient migration of cold data can be achieved without the need for additional dedicated hardware or storage pools, thereby improving the flexibility of file storage.

[0011] Optionally, based on mapping information and target object identifiers, the target file data is migrated from the file storage system to the object storage cluster, including:

[0012] The mapping information reveals the local directory of the target file in the file storage system.

[0013] Based on the preset migration strategy, target file data is filtered in the local file directory. The preset migration strategy includes file attribute filtering and time parameter filtering.

[0014] The target object identifier is associated with the target file data to obtain associated data, and the associated data is stored in the object storage cluster.

[0015] In this embodiment, the overhead of invalid filtering can be reduced, the probability of hot data being mistakenly migrated and cold data being missed can be lowered, and the accuracy of data filtering can be improved.

[0016] Optionally, the target file data includes multiple data items to be migrated; according to a preset migration strategy, the target file data is filtered in the local file directory, including:

[0017] Based on the preset migration strategy, multiple candidate data are filtered in the local directory of the file;

[0018] Obtain the data features corresponding to each candidate data, including access behavior features, file lifecycle features, and association features;

[0019] The cold data prediction model determines the probability of cold data for each candidate data based on the data characteristics corresponding to each candidate data.

[0020] Candidate data whose cold data probability is greater than or equal to the cold data threshold are identified as data to be migrated, thus obtaining multiple data to be migrated.

[0021] In this embodiment of the application, the cold data prediction model can be used to predict the cold or hot status of files, thereby reducing the false judgment rate, avoiding invalid migration, and improving migration accuracy.

[0022] Optionally, based on a preset migration strategy, multiple candidate data are filtered in the local directory of the file, including:

[0023] Filtering by file attributes allows you to select multiple primary data items in the local directory of a file. File attribute filtering includes data size filtering, file type filtering, and file status filtering.

[0024] By filtering by time parameters, multiple candidate data can be selected from multiple primary data. Time parameter filtering includes access time filtering and modification time filtering.

[0025] In this embodiment, invalid data can be filtered out first by file attributes, and cold data can be accurately located by filtering by time parameters, ensuring that the screening results meet the characteristics of cold data, thereby improving the efficiency and accuracy of screening migration data.

[0026] Optionally, based on object storage access information, mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster is constructed, including:

[0027] Verify the validity of the object's stored access information;

[0028] After successful verification, the mapping relationship between object storage access information and file local directory is generated, thus generating mapping information.

[0029] In this embodiment of the application, by verifying the validity, it can be determined in advance whether data can be migrated to the object storage cluster, which can improve the reliability of the migration.

[0030] Optionally, based on the file path in the local directory of the file, a target object identifier in the object storage cluster is generated, including:

[0031] Extract the relative path from the file path in the local directory of the file. The relative path is the hierarchical structure between the file path and the local directory of the file.

[0032] The relative path is converted into the target object identifier in the object storage cluster, and the hierarchical depth of the target object identifier is less than or equal to a preset hierarchical threshold.

[0033] In this embodiment of the application, converting the relative path into the target object identifier in the object storage cluster can improve the compatibility between the object identifier and the object storage protocol.

[0034] Optionally, migrating target file data from the file storage system to the object storage cluster based on mapping information and target object identifiers also includes:

[0035] During the migration process, a distributed lock is applied to the target file data to prevent multiple nodes from migrating the same file simultaneously.

[0036] In this embodiment, concurrent operation conflicts in multi-node deployment scenarios can be avoided, and multiple nodes migrating the same file at the same time can be prevented from causing data inconsistency.

[0037] Optionally, after migrating the target file data from the file storage system to the object storage cluster, the process may also include:

[0038] In the file storage system, update the status flag of the file metadata corresponding to the target file data from the local storage completion flag to the object storage status flag;

[0039] The local storage completion flag indicates that the actual data blocks and metadata corresponding to the target file data are completely stored in the local storage medium of the file storage system; the object storage status flag indicates that the actual data blocks corresponding to the target file data have been migrated to the object storage cluster, and the file storage system only retains simplified metadata locally.

[0040] In this embodiment of the application, by adding a status flag bit to the file metadata, the accuracy of the storage location routing during file access is ensured.

[0041] Optionally, the method further includes:

[0042] Dynamically forward read and write requests to the file storage system or object storage cluster based on the status flags in the file metadata.

[0043] Dynamically forward deletion requests to the file storage system or object storage cluster based on the status flags in the file metadata.

[0044] In this embodiment, the dynamic forwarding mechanism ensures that the file read / write performance is not affected after migration, and the compatibility is greatly improved.

[0045] Secondly, this application provides a file data migration apparatus, which includes a receiving module, a building module, a generating module, and a migration module:

[0046] The receiving module is used to receive object storage access information, which is used to request the migration of target file data.

[0047] The build module is used to construct the mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster, based on the object storage access information;

[0048] The generation module is used to generate the target object identifier in the object storage cluster based on the file path in the local directory of the file. The target object identifier is generated by transforming the hierarchical structure of the file path.

[0049] The migration module is used to migrate target file data from a file storage system to an object storage cluster based on mapping information and target object identifiers.

[0050] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0051] The memory stores instructions that the computer executes;

[0052] The processor executes computer-executable instructions stored in memory to implement the method as described in any of the first aspects.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the first aspect.

[0054] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0055] The file data migration method, apparatus, device, storage medium, and program provided in this application can use an object storage cluster as a cold data storage layer, achieving efficient migration and management of cold data without the need for additional dedicated hardware or storage pools. This enables hierarchical storage of local hot data and object storage cold data, preserving the efficient response capability of local storage for frequently accessed files while allowing object storage to handle archived data with low access frequency, thus improving the flexibility of file storage. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0058] Figure 2 A flowchart illustrating a file data migration method provided in an embodiment of this application;

[0059] Figure 3 A flowchart illustrating another file data migration method provided in this application embodiment;

[0060] Figure 4 A schematic diagram of the architecture of a data migration method provided in an embodiment of this application;

[0061] Figure 5 This application provides an interactive schematic diagram of file data migration as an embodiment of the present application;

[0062] Figure 6 This is an interactive schematic diagram of a file read operation provided in an embodiment of this application;

[0063] Figure 7 This application provides an interactive schematic diagram of a file write operation as an embodiment of the present application;

[0064] Figure 8 An interactive diagram illustrating another file write operation provided in an embodiment of this application;

[0065] Figure 9 An interactive schematic diagram of a file deletion operation provided in an embodiment of this application;

[0066] Figure 10 This is a schematic diagram of the structure of a file data migration device provided in an embodiment of this application;

[0067] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0069] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 This application scenario includes storage server 101 and object storage cluster 102. Storage server 101 has a file storage system with local storage.

[0070] After users upload massive amounts of log files or image resources to storage server 101, storage server 101 can migrate infrequently accessed files from the file storage system to object storage cluster 102, while retaining the file metadata in the file storage system. In terms of network architecture, the file storage system and object storage cluster 102 interact through standardized APIs (such as the S3 protocol). When users perform read / write operations through the file storage interface, the system determines whether to forward the request to object storage cluster 102 based on the file status flags.

[0071] In related technologies, such as cloud computing platforms, big data analytics systems, or enterprise-level file-sharing platforms, users typically need to store frequently accessed hot data in high-performance storage media (such as SSDs) and migrate cold data to low-cost storage media (such as disks or object storage). However, relying on physical storage media or dedicated storage pools results in high resource investment and poor flexibility.

[0072] The file data migration method provided in this application can receive object storage access information, which is used to request the migration of target file data; construct mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster based on the object storage access information; generate a target object identifier in the object storage cluster based on the file path under the file local directory, which is generated by transforming the hierarchical structure of the file path; and migrate the target file data from the file storage system to the object storage cluster based on the mapping information and the target object identifier.

[0073] In the above execution process, by using the object storage cluster as a cold data storage layer, efficient migration and management of cold data can be achieved without the need to deploy additional dedicated hardware or storage pools. Infrequently accessed historical files (i.e., target file data) are automatically migrated to the object storage cluster, while file metadata is still retained in the file storage system. This reduces storage costs while ensuring that users' file read and write operations are not affected, thus improving the flexibility of file storage.

[0074] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0075] Figure 2 This is a flowchart illustrating a file data migration method provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0076] S201, Receive object storage access information.

[0077] The execution entity of this application embodiment can be a storage server or a file data migration device installed on the storage server. The file data migration device can be implemented by software or by a combination of software and hardware.

[0078] Object storage access information can be used to request the migration of data from a target file.

[0079] File storage systems can receive object storage access information through user-configured settings.

[0080] When users manually input information, cloud storage platform administrators and business operations personnel receive object storage access information through a visual configuration interface, command line interface (CLI), or open application programming interface (API).

[0081] Object storage access information may include the access address (Uniform Resource Locator, URL), access key (AccessKey, AK), secret key (SecretKey, SK), target storage bucket name (bucket_name), etc.

[0082] S202. Based on the object storage access information, construct the mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster.

[0083] The local directory of a file refers to the absolute local directory path (such as / data / logs, / data / backup / images) used in the file storage system to store the target file data. It is the data source directory for file migration.

[0084] The local directory must have valid access permissions, and the files in the directory must meet the migration policy's filtering criteria; it supports binding a single directory to a single storage bucket, or binding multiple subdirectories to the same storage bucket (the mapping relationship is distinguished by the association identifier).

[0085] An object storage bucket refers to a logical storage container in an object storage cluster used to centrally store migrated files (e.g., S3's bucket_logs, OSS's company-backup-2024), and is the target storage container for file migration.

[0086] Object storage buckets have independent access control and data isolation features. The same storage bucket can store multiple files migrated from local directories (distinguished by object key), and must belong to the same cluster as the URL and AK / SK in the object storage access information.

[0087] Mapping information refers to structured data in a file storage system that records the association between a file's local directory and an object storage bucket. Mapping information must be unique (each mapping relationship corresponds to a unique identifier), complete (containing all information required for access and association), and persistently stored to ensure it is not lost after a system restart.

[0088] Mapping information may include association identifiers, access configurations, association relationships, status management, and other auxiliary fields.

[0089] The association identifier is a unique identifier for the mapping relationship, and the field name corresponding to the association identifier can be object_bind_id.

[0090] Access configuration can include the access key (AK) for encrypted storage, the key (SK) for encrypted storage, and the access address (URL) for the object storage cluster.

[0091] The association can include the local directory of the file in the file storage system (absolute path, corresponding to the dir_path field), the name of the target object storage bucket (corresponding to the bucket_name field), etc.

[0092] The association can indicate the status of the mapping relationship (corresponding to the status field), which can include enabled, disabled, or abnormal.

[0093] Other auxiliary fields may include the adapted object storage API protocol version, the mapping creation time, and the last update time of the mapping.

[0094] S203. Generate the target object identifier in the object storage cluster based on the file path in the local directory of the file.

[0095] The target object identifier is generated by transforming the file path hierarchy and serves as a unique index for locating file data in object storage, ensuring accurate retrieval and access of data after migration.

[0096] An object key is a string used to uniquely identify a single data object in an object storage cluster and is the core index field of object storage. It is equivalent to "file path + filename" in a file storage system, but without an actual directory hierarchy. Object storage uses a flat storage structure, and the hierarchy is only for logical display. It must meet the naming conventions of object storage, such as a length ≤ 1024 characters and no illegal characters.

[0097] The file path in the local directory refers to the absolute path of the target file data in the file storage system (e.g., / data / logs / A / B / file1). It consists of "local directory (e.g., / data / logs) + subdirectory level (e.g., A / B) + file name (e.g., file1)" and is the original data source for generating the target object identifier.

[0098] Hierarchical structure conversion refers to a logical conversion method that removes the local directory prefix from a file path, retains the remaining "subdirectory hierarchy + filename" structure, and directly maps it to the target object identifier, ensuring that the original file organization is still reflected after conversion. For example, / data / logs / A / B / file1 becomes A / B / file1 after removing the prefix.

[0099] S204. Based on the mapping information and the target object identifier, migrate the target file data from the file storage system to the object storage cluster.

[0100] In some embodiments, the target file data in the local directory of the file storage system can be obtained from the mapping information; the target file data can be filtered in the local directory according to a preset migration strategy; the target object identifier can be associated with the target file data to obtain associated data, and the associated data can be stored in the object storage cluster based on the access configuration in the mapping information.

[0101] The local file directory limits the scope of the preset migration strategy's filtering. The local file directory can be explicitly defined by the dir_path field corresponding to the associated identifier, ensuring that filtering only applies to files under the bound directory and avoiding accidental migration across directories.

[0102] At the same time, before filtering, the access permissions of the local directory of the file (the file storage system process must have read permissions) and the availability of the directory (not deleted, not mounted abnormal state) will be checked. If the directory is unavailable, the filtering will be terminated and an alarm will be triggered.

[0103] The associated data includes the complete binary content of the target file, the target object identifier, and migration metadata. Specifically, the migration metadata includes information such as file size, modification time, creation time, ETag checksum (a hash value based on the file content), and association identifier.

[0104] The preset migration strategy is a cold data filtering rule configured in the file storage system. It is used to accurately identify files that need to be migrated from the local file directory, avoiding the mistaken migration of hot data or the omission of cold data.

[0105] Preset migration strategies can include file attribute filtering and time parameter filtering.

[0106] File attribute filtering can include data size filtering, type filtering, and file status filtering; time parameter filtering can include access time filtering and modification time filtering.

[0107] In this application, target file data is selectively filtered within the local file directory using a preset migration strategy. This reduces ineffective filtering overhead, lowers the probability of mismigrating hot data and missing cold data, and improves the accuracy of data filtering. Simultaneously, the filtering process is executed asynchronously in the background of the file storage system, without occupying user operation threads, thus avoiding interference with normal local file reading and writing, balancing filtering efficiency and business availability.

[0108] The file data migration method provided in this application can use an object storage cluster as a cold data storage layer, achieving efficient migration and management of cold data without the need for additional dedicated hardware or storage pools. It enables hierarchical storage of local hot data and object storage cold data, preserving the efficient response capability of local storage for frequently accessed files while allowing object storage to handle archived data with low access frequency, thus improving the flexibility of file storage.

[0109] Figure 3 This is a flowchart illustrating another file data migration method provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0110] S301, Receive object storage access information.

[0111] The execution process of S301 can be found in the execution process of S201, and will not be repeated here.

[0112] S302. Verify the validity of object storage access information.

[0113] It can verify the validity of the access address, access key, and key in the object storage access information.

[0114] Specifically, a verification request for object storage access information can be sent to the object storage cluster. The verification request is used to verify the validity of the access address, access key, and key in the object storage access information. The verification result sent by the object storage cluster can be received. The verification result can include verification success or verification failure.

[0115] When an object storage cluster receives a verification request, it can verify the validity of the access address, access key, and key separately. If the access address, access key, and key all pass the validity verification, the verification result can be determined as successful; if any one of the validity verifications fails, the verification result is verified as unsuccessful.

[0116] Object storage clusters can perform endpoint matching verification, server status verification, and protocol adaptation verification on access addresses.

[0117] The endpoint matching verification process involves parsing the access address (URL) in the verification request, extracting the domain name / IP address and port, and comparing it with the cluster's own list of service endpoints. If the address does not belong to a valid endpoint configured in the cluster, the access address is directly determined to be invalid.

[0118] Server status verification checks whether the service node corresponding to the access address is in a normal operating state (not offline, under maintenance, or experiencing a fault). If the node is unavailable (e.g., due to cluster expansion causing some endpoints to be temporarily shut down), the service corresponding to the access address will be returned as temporarily unavailable.

[0119] Protocol compatibility verification checks whether the protocol header (http: / / / https: / / ) of the access address matches the protocols supported by the cluster. If the cluster only supports HTTPS, but the request uses HTTP, the access address is deemed unsupported, and verification fails.

[0120] Object storage clusters can perform AK existence verification, AK status verification, and AK attribution verification on access keys.

[0121] Object storage clusters can perform key signature generation logic verification, signature consistency comparison, and request timeliness verification.

[0122] S303. After successful verification, the mapping relationship between object storage access information and file local directory is generated, and mapping information is generated.

[0123] A distributed ID generation algorithm can be used to generate globally unique association identifiers, ensuring that each file directory and storage bucket combination corresponds to a unique identifier and avoiding conflicts.

[0124] For example, the distributed ID generation algorithm can be the Snowflake algorithm, and the associated identifier object_bind_id can be 12345.

[0125] Mapping information can be stored in the form of structured data tables, such as relational databases like MySQL and PostgreSQL, or cached in Redis to improve query efficiency.

[0126] The core fields and examples of the mapping information can be found in Table 1.

[0127] Table 1

[0128]

[0129] In this application, the validity of object storage access information can be verified. After successful verification, the mapping relationship between object storage access information and local file directories can be generated to determine in advance whether data can be migrated to the object storage cluster, thereby improving the reliability of the migration.

[0130] S304. Extract the relative path from the file path in the local directory of the file.

[0131] A relative path is a hierarchical structure between a file path and the file's local directory.

[0132] Specifically, the mapping information can be queried by association identifier to obtain the local directory of the target file data, extract the absolute path of the target file, and strip the prefix part of the absolute path that completely matches the local directory of the file to obtain the relative path fragment; if there are illegal characters in the relative path fragment, the illegal characters corresponding to the relative path fragment are replaced with alternative characters to obtain the relative path; if there are no illegal characters, the relative path fragment is determined as the relative path.

[0133] For example, the local directory of the file is " / data / logs", and the absolute path is " / data / logs / A / B / file1". After removing " / data / logs" from the absolute path, the relative path fragment is "A / B / file1".

[0134] By filtering for illegal characters, you can remove prohibited characters from relative path fragments that are restricted by object storage and replace them with valid alternative characters. For example, illegal characters can be "\r", "\n", "?", "*", etc., though these may vary slightly depending on the object storage specification and can be automatically adapted. Alternative characters can include "_", etc.

[0135] In this application, the hierarchical relationship of the original file can be preserved through relative path level conversion, making it convenient for users to inspect data according to the target structure using native object storage tools.

[0136] S305. Convert the relative path to the target object identifier in the object storage cluster.

[0137] In some embodiments, after the relative path is converted into an initial object identifier, a lightweight query request can be sent to the object storage cluster to verify whether the initial object identifier already exists in the target bucket; if it exists, a version number is added to the end of the object identifier until a unique target object identifier is generated.

[0138] For example, if the relative path is "A / B / file_1" and the initial object identifier after conversion is "A / B / file_1", and the query finds that the identifier already exists in the target bucket, then a version number is added to the end to get "A / B / file_1_V2"; if "A / B / file_1_V2" still exists, it continues to increment to "A / B / file_1_V3" until the verification passes.

[0139] In this application, a uniqueness verification mechanism can be used to determine the unique object identifier corresponding to each file after migration, thereby avoiding data overwriting or loss caused by duplicate identifiers in object storage and ensuring the consistency of data migration.

[0140] Meanwhile, to adapt to the limitations of object storage protocols on the level depth (such as some protocols / products directly limiting the level threshold, or indirectly limiting the level through the identifier length), the level depth of the object identifier must be less than or equal to the preset level threshold.

[0141] The hierarchy depth of a relative path is determined by a separator (" / "). For example, the relative path "a / b / c / file.txt" has a separator of 3 and a hierarchy of 4.

[0142] If the current level depth corresponding to the relative path is less than or equal to the preset level threshold, the relative path is determined as the initial object identifier; if the current level corresponding to the relative path is greater than the preset level threshold, the core level at the front end of the relative path is retained, and the file identifier suffix is ​​added to obtain the initial object identifier.

[0143] Delete the level nodes that exceed the preset level threshold at the end. At the same time, add the original file name (including the suffix) as the last node to the retained path to ensure that the object identifier can still be associated with the original file.

[0144] For example, the pre-screening level threshold is 8, the relative path is "a / b / c / d / e / f / g / h / i / file.txt", and the current level depth of the relative path is 10, which exceeds the threshold by 2 levels. At this time, the core path "a / b / c / d / e / f / g / h" of the front-end 8 nodes is retained, and the original file name "file.txt" is added to the end, resulting in the object identifier "a / b / c / d / e / f / g / h / file.txt" (level depth 8, which meets the threshold requirement).

[0145] In this application, the hierarchical depth of the object identifier is controlled by a preset hierarchical threshold, which can adapt to the hierarchical restrictions of different object storage protocols and indirectly control the character length of the object identifier, reducing the resource consumption of the identifier during storage and transmission. At the same time, combined with the uniqueness verification mechanism, it not only ensures protocol compatibility but also avoids data conflicts and improves the reliability of data migration.

[0146] By extracting relative paths from file paths in the local directory and converting these relative paths into target object identifiers in the object storage cluster, the compatibility between object identifiers and object storage protocols can be improved.

[0147] S306. In the mapping information, obtain the local directory of the target file data in the file storage system.

[0148] S307. According to the preset migration strategy, filter the target file data in the local file directory.

[0149] The target file data may include multiple pieces of data to be migrated.

[0150] In some embodiments, multiple candidate data can be filtered in the local directory of the file according to a preset migration strategy; data features corresponding to each candidate data can be obtained; cold data prediction model can be used to determine the cold data probability corresponding to each candidate data according to the data features corresponding to each candidate data; and candidate data whose cold data probability corresponding to each candidate data is greater than or equal to the cold data threshold can be determined as data to be migrated, so as to obtain multiple data to be migrated.

[0151] Data characteristics can include access behavior characteristics, file lifecycle characteristics, and association characteristics.

[0152] Access behavior characteristics can include the number of visits in the past 7 days / 30 days / 90 days, the access interval, the number of users accessing the service, and the distribution of access times (such as whether they are concentrated during peak business periods).

[0153] File lifecycle characteristics can include creation time, modification frequency (number of modifications in the last 90 days), and overall access popularity of the directory where the file is located (such as the average access frequency of other files in the directory).

[0154] Association features can include the access relationship between files and other files (e.g., the probability of accessing file B when file A is accessed) and the periodicity of the business line to which the file belongs (e.g., the access pattern of e-commerce log files during promotional periods).

[0155] For any candidate data, its data features can be input into the cold data prediction model to obtain the corresponding cold data probability.

[0156] For example, the probability of cold data is between 0 and 1. The closer the value is to 1, the higher the confidence that the file is cold data. The cold data threshold can be configured according to business needs, and the cold data threshold can be 0.8.

[0157] If the probability of cold data is greater than or equal to the cold data threshold, then the candidate data is determined as data to be migrated.

[0158] If the probability of cold data is less than the cold data threshold, then manual review or secondary rule verification will be performed.

[0159] The cold data prediction model is a model trained on a lightweight classification model using a training set. Lightweight classification models can include logistic regression, random forest, LightGBM, etc.

[0160] Training set construction: Extract massive file samples from historical access logs and generate labeled training samples by combining manual or automatic annotation rules.

[0161] In this application, a cold data prediction model can be used to predict the cold or hot status of files, thereby reducing the false judgment rate, avoiding invalid migration, and improving migration accuracy.

[0162] Preset migration strategies can include file attribute filtering and time parameter filtering.

[0163] Specifically, you can filter by file attributes to select multiple primary data points in the local directory of a file; or filter by time parameters to select multiple candidate data points from multiple primary data points.

[0164] File attribute filtering can include data size filtering, file type filtering, and file status filtering.

[0165] Data size filtering can be achieved by setting two thresholds to filter data whose file size is less than or equal to the first threshold (e.g., 1KB) or greater than or equal to the second threshold (e.g., 50GB).

[0166] Among them, the first threshold is less than the second threshold: filtering extremely small files that are less than or equal to the first threshold can avoid problems such as excessive network bandwidth consumption and low migration efficiency caused by migration; filtering extremely large files that are greater than or equal to the second threshold can avoid affecting the overall migration task progress due to excessive migration time of a single file, or exceeding the single upload size limit of object storage.

[0167] The file type filter only selects files with preset formats. These preset formats include low-priority archive files such as log files, backup files, historical images, and configuration files. It explicitly excludes critical files that require real-time local access, such as core configuration files (.conf, .yml), executable files (.exe, .sh), and real-time database files (.db, .ibd).

[0168] File status filtering can include excluding encrypted files, files that are currently being used by a process, etc.

[0169] Time parameter filtering can include access time filtering and modification time filtering.

[0170] Access time filtering can determine the interval between the most recent access time of a file and the current time, filter data whose interval is less than or equal to a first preset time. These data are high-frequency access data and need to be retained in the file storage system. Data whose interval is greater than the first preset time is determined as candidate data.

[0171] The modified time filter supports two filtering logics, and the filtering logic can be configured:

[0172] It can determine the creation time between the file creation time and the current time, and filter data whose creation time is greater than or equal to a second preset time (e.g., 90 days), that is, to identify data with a longer creation time as candidate data;

[0173] Alternatively, determine the modification duration between the most recent modification time and the current time, and filter data whose modification duration is greater than or equal to a third preset duration (e.g., 60 days), thus identifying data that has not been updated for a long time as candidate data.

[0174] It's worth noting that users can configure preset migration strategies through the migration strategy configuration interface. All thresholds (first / second threshold, first / second / third preset duration) and filtering types can be flexibly configured to adapt to the tiered storage needs of different business scenarios.

[0175] In this application, invalid data can be filtered out first by file attributes, and cold data can be accurately located by filtering by time parameters, ensuring that the screening results meet the characteristics of cold data, thus improving the efficiency and accuracy of screening migration data.

[0176] S308. Associate the target object identifier with the target file data to obtain associated data, and store the associated data in the object storage cluster.

[0177] It can extract migration metadata and binary content of files, combine them with target object identifiers, and generate associated data.

[0178] To ensure data transmission security, the binary content of files in the associated data can be temporarily encrypted (e.g., using the AES-128 symmetric encryption algorithm). The encryption key is only temporarily stored locally in the file storage system and is automatically decrypted and stored after the data is uploaded to the object storage cluster (or the encryption state is retained according to user configuration), avoiding data leakage caused by network transmission or interception by intermediate nodes during the migration process.

[0179] Based on the file size corresponding to the associated data, a scenario-based upload strategy can be adopted to adapt to the migration needs of different file sizes, ensuring upload efficiency and compatibility.

[0180] When the file size corresponding to the associated data is less than or equal to the preset size, the binary data in the associated data is directly bound to the object identifier through the PutObject API of the object storage and uploaded to the target storage bucket. Streaming transmission does not require local caching.

[0181] When the file size corresponding to the associated data is larger than the preset size, the chunked upload is automatically triggered, splitting the binary content of the file into standard chunks of 5MB to 1GB. First, the CreateMultipartUpload API is called to initialize the chunked upload task to the object storage cluster. Then, the UploadPart API is called in sequence to upload each chunk (each chunk carries a unique chunk number). After all chunks are uploaded, the CompleteMultipartUpload API is called to submit a chunk merging request and generate a complete object.

[0182] The preset size can be configured according to the single upload limit supported by the object storage cluster, and it is compatible with mainstream standards such as the S3 protocol, and can support 5G.

[0183] In some embodiments, a distributed lock is applied to the target file data during the migration process, and the migration task and user read / write operations are mutually exclusive by relying on the status flags of the file metadata. This can avoid concurrent operation conflicts in multi-node deployment scenarios and prevent data inconsistency caused by multiple nodes migrating the same file at the same time.

[0184] Before uploading: Request a distributed lock from the distributed coordination service (such as etcd, ZooKeeper). The unique identifier (LockKey) of the distributed lock is in the format of object_bind_id:object_key:file_path (e.g., 12345:A / B / file_1: / data / logs / A / B / file_1) to ensure exclusivity across nodes. At the same time, check the hiera_flag status of the file metadata. If it is DATAFILE (local complete storage), allow migration; if it is UPLOADING (already occupied by migration tasks on other nodes), terminate the current migration request to avoid conflicts.

[0185] During the upload process: Start an independent lock renewal thread, and send a renewal request to the distributed coordination service every 10 minutes (less than the default timeout of 30 minutes for distributed locks, which is configurable) to reset the lock timeout and prevent the lock from being automatically released due to the long upload time of large file chunks;

[0186] After uploading: Regardless of success or failure, the locks are unlocked in the order that the distributed locks were released first, ensuring that lock resources are reclaimed in a timely manner.

[0187] The file data migration method of this application embodiment uses an object storage cluster as a cold data storage layer. Without the need to deploy additional dedicated hardware or storage pools, it can achieve efficient migration and management of cold data. It automatically migrates infrequently accessed historical files (i.e., target file data) to the object storage cluster while retaining the file metadata in the file storage system. This reduces storage costs while ensuring that users' file read and write operations are not affected, thus improving the flexibility of file storage.

[0188] Figure 4 This is a schematic diagram illustrating the architecture of a data migration method provided in an embodiment of this application. Please refer to [link / reference]. Figure 4 The file storage system first requires the administrator to specify the object storage resource information to be bound, including the object storage access URL (address), access key AK, and key SK. The file storage system then sends a verification request to the object storage cluster to verify the connectivity of the URL, the validity of the AK / SK, and the permissions. After completing the verification, the object storage cluster returns a "verification passed" confirmation message to the file storage system.

[0189] The file storage system persistently stores the verified object storage access information (URL, AK, SK) in local metadata. The file storage system requests information about all the buckets it has created from the object storage cluster; the object storage cluster returns the bucket list as a response message to the file storage system.

[0190] The file storage system selects the local file directory and associates it with the specified bucket on the object storage side. The mapping relationship between the local file directory path (local_path) and the object storage bucket (bucket_hiera) is saved locally, thus generating the mapping information.

[0191] Table 2 shows the mapping information. The cluster binding relationship table on the left contains the basic access information for the object storage cluster. cluster_bind_id (PK) is the unique primary key of the cluster binding relationship (PK is the primary key), which identifies an independent object storage cluster access configuration; URL is the access address of the object storage cluster (e.g., https: / / s3.xxxx.com); AK is the object storage access key for encrypted storage; SK is the object storage key for encrypted storage.

[0192] The directory / bucket binding table on the right shows the mapping between the local directory of the stored file and the object storage bucket. `object_bind_id(PK)` is the unique primary key (i.e., the association identifier) ​​of the directory-bucket binding relationship; `dir_path` is the absolute path of the local directory of the file storage system; and `bucket_name` is the object storage bucket (e.g., `bucket_logs`). The `cluster_bind_id` on the left and the `object_bind_id` on the right are associated.

[0193] Table 2

[0194]

[0195] After completing the above configuration, cold data that conforms to the migration strategy in the subsequent file storage system will be automatically uploaded to the object storage bucket (bucket_hiera) corresponding to the mapping relationship.

[0196] Figure 5 This is an interactive schematic diagram illustrating a file data migration process provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 This process is the specific interactive process of migrating cold data from the file storage system to the object storage cluster. After the background migration strategy of the file storage system is triggered, the corresponding object storage access information (URL, AK, target bucket name) is retrieved by using the field name (object_bind_id) associated with the target file.

[0197] Update the file metadata's file flag (hiera_flag) to UPLOADING|DATAFILE to indicate that the file is in the process of migrating to object storage, while retaining the complete data locally (to avoid data loss due to migration interruption).

[0198] The complete binary content of the file to be migrated is read from the local storage medium of the file storage; based on the acquired object storage access information, the read file content data is uploaded to the object storage cluster; the object storage cluster receives the file content and persists it with the target object identifier as an index.

[0199] After object storage completes data saving, it returns a "successful response" confirmation message to the file storage system. The file storage system updates the file flag (hiera_flag) of the file metadata to DATAOBJ (indicating that the file data has been migrated to object storage). Simultaneously, if the file metadata is not associated with an associated identifier (object_bind_id), this identifier is added to ensure that subsequent accesses can be associated with the corresponding object storage information. The file binary data on the local file storage system is deleted, retaining only the simplified metadata containing hiera_flag, object_bind_id, and object identifier, thus releasing local storage resources. The simplified metadata includes the object identifier and object_bind_id.

[0200] In some embodiments, after the target file data is migrated from the file storage system to the object storage cluster, the status flag of the file metadata corresponding to the target file data in the file storage system can be updated from the local storage completion flag to the object storage status flag.

[0201] The local storage completion flag (i.e., DATAFILE) indicates that the actual data blocks and metadata corresponding to the target file data are completely stored in the local storage medium of the file storage system; the object storage status flag (i.e., DATAOBJ) indicates that the actual data blocks corresponding to the target file data have been migrated to the object storage cluster, and the file storage system only retains simplified metadata locally.

[0202] A flag is added to the file metadata to distinguish different file states. The specific state descriptions are shown in Table 3.

[0203] Table 3

[0204]

[0205] The default value of the file flag hiera_flag is DATAFILE, meaning that the file is initially stored entirely locally by default.

[0206] Meanwhile, the metadata of directories with established mapping relationships (i.e., associated directories) will record the corresponding object_bind_id. Newly created files under this associated directory will automatically inherit this object_bind_id in their metadata, ensuring that when new files meet the migration conditions, they can be directly associated with the corresponding object storage bucket without having to repeatedly configure the mapping relationship.

[0207] In this application, by adding a status flag to the file metadata and maintaining its dynamic updates, combined with the object_bind_id inheritance mechanism of the associated directory, the accurate identification of the file storage status and the automatic association of migration relationships can be achieved, ensuring the accuracy of storage location routing when accessing files.

[0208] After migrating a file storage system to an object storage cluster, read and write requests can be dynamically forwarded to the file storage system or the object storage cluster based on the status flags in the file metadata; similarly, deletion requests can be dynamically forwarded to the file storage system or the object storage cluster based on the status flags in the file metadata, without affecting file read and write operations. Below, we will combine... Figures 6 to 9 Please provide a detailed explanation.

[0209] Figure 6 This is an interactive diagram illustrating a file read operation provided in an embodiment of this application. Please refer to [link / reference]. Figure 6 The user sends a file read request to the file storage system, including the target file path (file_path), read offset (offset), and read data size (size). The file storage system queries the file's metadata to confirm that its file flag (hiera_flag) status is DATAOBJ (indicating that the file data has been migrated to object storage). It extracts the associated identifier (object_bind_id) from the file metadata and uses this identifier to query the corresponding object storage cluster access information (URL, AK, SK, etc.). Based on the file path (file_path) in the user's request, it generates the target object identifier (Object Key) in object storage according to preset rules (such as truncating relative paths and transforming hierarchical structures). Based on the obtained access information and object identifier, the file storage system initiates a data read request to the object storage cluster, specifying the read offset (offset) and data size (size). The object storage cluster reads the corresponding data segment and returns it to the file storage system, which then forwards the data segment to the user, completing the read operation.

[0210] Figure 7 This is an interactive diagram illustrating a file write operation as provided in an embodiment of this application. Please refer to [link / reference]. Figure 7When a user sends a write request to the file storage system, requesting a write operation to be performed on the target file, the file storage system queries the file's metadata and checks its file flag (hiera_flag) status. At this time, the file status may be DATAFILE (local complete storage) or UPLOADING (being migrated to object storage). If the file is currently in the UPLOADING state (i.e., being migrated), the file storage system immediately interrupts the migration task corresponding to the file to avoid conflicts between the migration process and the write operation. The hiera_flag of the file metadata is then changed to DATAFILE, indicating that the file data must be completely preserved in the local storage medium of the file storage system, ensuring that the write operation can directly affect the local file.

[0211] Figure 8 This is an interactive diagram illustrating another file write operation provided in an embodiment of this application. Please refer to... Figure 8 The user sends a write request to the file storage system, carrying the target file path (file_path) in the request. The file storage system queries the file's metadata and confirms that its hiera_flag status is DATAOBJ (indicating that the file data is currently only stored in object storage). Since the file data is not local, the file storage system first returns a retry error code to the user, indicating that the file is temporarily unwritable and needs to wait for the data to be retrieved. Then, it updates the hiera_flag of the file metadata to DATAOBJ|PULLING, indicating that the file is being retrieved from object storage to the local machine.

[0212] Extract the association identifier (object_bind_id) from the file metadata, and use this identifier to query the corresponding object storage cluster access information (URL, AK, etc.); generate the target object identifier (Object Key) in the object storage based on the file path (file_path), and start the data retrieval task.

[0213] The file storage system initiates a data read request to the object storage cluster based on access information and object identifier to obtain complete file data; the object storage cluster returns the corresponding file data to the file storage system; the file storage system writes the retrieved file data to the local storage medium, restoring the file's complete local storage; and updates the hiera_flag of the file metadata to DATAFILE, indicating that the file has been completely stored locally, at which point the user can re-initiate a write operation.

[0214] Figure 9 This is an interactive diagram illustrating a file deletion operation provided in an embodiment of this application. Please refer to [link / reference]. Figure 9When a user sends a deletion request to the file storage system, the request includes the target file path (file_path). The file storage system first checks the file's hiera_flag status. If the file is currently in the PULLING (data retrieval in progress) or UPLOADING (data migration in progress) state, it immediately interrupts the corresponding retrieval or migration task to avoid conflicts between the task and the deletion operation.

[0215] If the file's hiera_flag status is DATAOBJ (data is stored in object storage), extract the association identifier (object_bind_id) from the file metadata, and use this identifier to query the corresponding object storage cluster access information (URL, AK, etc.); generate the corresponding target object identifier (Object Key) in object storage based on the file path (file_path);

[0216] The file storage system initiates a deletion request to the object storage cluster based on access information and object identifier, deleting the object data corresponding to the file. After the object storage cluster completes the data deletion, it returns a successful operation response to the file storage system. The file storage system clears the local file metadata and returns a successful deletion operation response to the user.

[0217] Transparent access to migrated files is achieved through status flags in metadata. When a user reads a file, the system can determine whether the file is stored in object storage based on the status flag (such as DATAOBJ) and automatically forward the request to object storage. When a user writes to a file, if the file is in the DATAOBJ state, the system triggers a fetch task to synchronize the data back to the local machine.

[0218] In this application, file status is managed through flags, decoupling user operations from the underlying storage and significantly reducing complexity. Simultaneously, a dynamic forwarding mechanism ensures that file read / write performance remains unaffected after migration, greatly improving compatibility. Furthermore, fine-grained management of status flags (such as UPLOADING and PULLING) effectively avoids data conflicts during the migration process.

[0219] Figure 10 This is a schematic diagram of a file data migration device provided in an embodiment of this application. Please refer to... Figure 10 The file data migration device 1000 includes a receiving module 1001, a building module 1002, a generating module 1003, and a migration module 1004.

[0220] The receiving module 1001 is used to receive object storage access information, which is used to request the migration of target file data.

[0221] Module 1002 is used to construct the mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster based on the object storage access information.

[0222] The generation module 1003 is used to generate a target object identifier in the object storage cluster based on the file path in the local directory of the file. The target object identifier is generated by converting the hierarchical structure of the file path.

[0223] The migration module 1004 is used to migrate target file data from the file storage system to the object storage cluster based on mapping information and target object identifier.

[0224] Optionally, the migration module 1004 is specifically used for:

[0225] The mapping information reveals the local directory of the target file in the file storage system.

[0226] Based on the preset migration strategy, target file data is filtered in the local file directory. The preset migration strategy includes file attribute filtering and time parameter filtering.

[0227] The target object identifier is associated with the target file data to obtain associated data, and the associated data is stored in the object storage cluster.

[0228] Optionally, the target file data includes multiple data items to be migrated; the migration module 1004 is specifically used for:

[0229] Based on the preset migration strategy, multiple candidate data are filtered in the local directory of the file;

[0230] Obtain the data features corresponding to each candidate data, including access behavior features, file lifecycle features, and association features;

[0231] The cold data prediction model determines the probability of cold data for each candidate data based on the data characteristics corresponding to each candidate data.

[0232] Candidate data whose cold data probability is greater than or equal to the cold data threshold are identified as data to be migrated, thus obtaining multiple data to be migrated.

[0233] Optionally, the migration module 1004 is specifically used for:

[0234] Filtering by file attributes allows you to select multiple primary data items in the local directory of a file. File attribute filtering includes data size filtering, file type filtering, and file status filtering.

[0235] By filtering by time parameters, multiple candidate data can be selected from multiple primary data. Time parameter filtering includes access time filtering and modification time filtering.

[0236] Optionally, module 1002 is specifically used for:

[0237] Verify the validity of the object's stored access information;

[0238] After successful verification, the mapping relationship between object storage access information and file local directory is generated, thus generating mapping information.

[0239] Optionally, the generation module 1003 is specifically used for:

[0240] Extract the relative path from the file path in the local directory of the file. The relative path is the hierarchical structure between the file path and the local directory of the file.

[0241] The relative path is converted into the target object identifier in the object storage cluster, and the hierarchical depth of the target object identifier is less than or equal to a preset hierarchical threshold.

[0242] Optionally, the migration module 1004 is also used for:

[0243] During the migration process, a distributed lock is applied to the target file data to prevent multiple nodes from migrating the same file simultaneously.

[0244] Optionally, the migration module 1004 is also used for:

[0245] In the file storage system, update the status flag of the file metadata corresponding to the target file data from the local storage completion flag to the object storage status flag;

[0246] The local storage completion flag indicates that the actual data blocks and metadata corresponding to the target file data are completely stored in the local storage medium of the file storage system; the object storage status flag indicates that the actual data blocks corresponding to the target file data have been migrated to the object storage cluster, and the file storage system only retains simplified metadata locally.

[0247] Optionally, the migration module 1004 is also used for:

[0248] Dynamically forward read and write requests to the file storage system or object storage cluster based on the status flags in the file metadata;

[0249] Dynamically forward deletion requests to the file storage system or object storage cluster based on the status flags in the file metadata.

[0250] The file data migration device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0251] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 11The electronic device 1100 may include a processor 1101 and a memory 1102 communicatively connected to the processor 1101. Exemplarily, the processor 1101 and the memory 1102 are interconnected via a bus 1103.

[0252] Memory 1102 stores computer-executed instructions;

[0253] The processor 1101 executes computer execution instructions stored in the memory 1102, causing the processor 1101 to perform the file data migration method as described in the above method embodiment.

[0254] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the file data migration method of the above-described method embodiments.

[0255] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the file data migration method shown in the above method embodiments.

[0256] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0257] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0258] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0259] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0260] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0261] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0262] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0263] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for migrating file data, characterized in that, The method includes: Receive object storage access information, the object storage access information being used to request the migration of target file data; Based on the object storage access information, construct the mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster; Based on the file path in the local directory of the file, a target object identifier in the object storage cluster is generated, and the target object identifier is generated by transforming the hierarchical structure of the file path; Based on the mapping information and the target object identifier, the target file data is migrated from the file storage system to the object storage cluster.

2. The method according to claim 1, characterized in that, Based on the mapping information and the target object identifier, the target file data is migrated from the file storage system to the object storage cluster, including: The mapping information is used to obtain the local file directory of the target file data in the file storage system. According to a preset migration strategy, the target file data is filtered in the local directory of the file. The preset migration strategy includes file attribute filtering and time parameter filtering. The target object identifier is associated with the target file data to obtain associated data, and the associated data is stored in the object storage cluster.

3. The method according to claim 2, characterized in that, The target file data includes multiple data items to be migrated; According to a preset migration strategy, the target file data is filtered in the local file directory, including: According to the preset migration strategy, multiple candidate data are filtered in the local directory of the file; Obtain the data features corresponding to each candidate data, including access behavior features, file lifecycle features, and association features; The cold data prediction model determines the cold data probability corresponding to each candidate data based on the data characteristics corresponding to each candidate data. Candidate data whose cold data probability is greater than or equal to the cold data threshold are identified as the data to be migrated, so as to obtain the plurality of data to be migrated.

4. The method according to claim 3, characterized in that, According to the preset migration strategy, multiple candidate data are filtered in the local directory of the file, including: The file attribute filtering filters multiple first data items in the local directory of the file, including data size filtering, file type filtering, and file status filtering. The multiple candidate data are filtered from the multiple first data by the time parameter filtering, which includes access time filtering and modification time filtering.

5. The method according to claim 1, characterized in that, Based on the object storage access information, construct mapping information between the file local directory of the file storage system and the object storage bucket of the object storage cluster, including: Verify the validity of the object's stored access information; After successful verification, the mapping relationship between the object storage access information and the local directory of the file is generated to form mapping information.

6. The method according to claim 1, characterized in that, Based on the file path in the local directory of the file, generate the target object identifier in the object storage cluster, including: Extract a relative path from the file path in the local directory of the file, where the relative path is the hierarchical structure between the file path and the local directory of the file; The relative path is converted into a target object identifier in the object storage cluster, wherein the hierarchical depth of the target object identifier is less than or equal to a preset hierarchical threshold.

7. The method according to claim 1, characterized in that, Based on the mapping information and the target object identifier, migrating the target file data from the file storage system to the object storage cluster further includes: A distributed lock is applied to the target file data during the migration process to prevent multiple nodes from migrating the same file simultaneously.

8. The method according to claim 1, characterized in that, After migrating the target file data from the file storage system to the object storage cluster, the process further includes: In the file storage system, update the status flag of the file metadata corresponding to the target file data from the local storage completion flag to the object storage status flag; The local storage completion flag indicates that the actual data block and metadata corresponding to the target file data are completely stored in the local storage medium of the file storage system; the object storage status flag indicates that the actual data block corresponding to the target file data has been migrated to the object storage cluster, and the file storage system only retains simplified metadata locally.

9. The method according to claim 8, characterized in that, Also includes: Dynamically forward read and write requests to the file storage system or object storage cluster based on the status flags in the file metadata; Dynamically forward deletion requests to the file storage system or object storage cluster based on the status flags in the file metadata.

10. A file data migration device, characterized in that, The device includes a receiving module, a building module, a generating module, and a migration module, wherein, The receiving module is used to receive object storage access information, which is used to request the migration of target file data. The construction module is used to construct mapping information between the local file directory of the file storage system and the object storage bucket of the object storage cluster based on the object storage access information. The generation module is used to generate a target object identifier in the object storage cluster based on the file path in the local directory of the file. The target object identifier is generated by converting the hierarchical structure of the file path. The migration module is used to migrate the target file data from the file storage system to the object storage cluster based on the mapping information and the target object identifier.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images