File storage adjustment method and device, equipment, storage medium and program product

Abstract

The invention relates to a file storage adjustment method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: when a storage adjustment request is received, determining a to-be-adjusted file and a target storage level corresponding to the storage adjustment request; determining an original storage pool corresponding to the to-be-adjusted file, and determining a target storage pool corresponding to the target storage level; after the state of the to-be-adjusted file is modified from an accessible state to a frozen state, data blocks, in the original storage pool, of the to-be-adjusted file are migrated to the target storage pool; and after migration is completed, modifying the state of the file to be adjusted from the frozen state to the accessible state. By adopting the method, the storage adjustment efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a file storage adjustment method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] In modern data centers and enterprise-level storage systems, file tiered storage is a common data management strategy that aims to optimize the use of storage resources while balancing cost and performance. File tiered storage distributes data to storage media with different performance and cost, ensuring that frequently accessed data can be accessed quickly, while infrequently used data is stored on lower-cost media.

[0003] Existing file storage adjustment methods only allow storage level adjustments when a file is newly created or when the file is empty (does not contain data). This means that once a file begins to be used and contains data, its storage level cannot be adjusted unless the user manually creates a new file and migrates the data to it. This process is not only cumbersome, but may also cause data inconsistency and access interruptions, resulting in low efficiency and accuracy in file storage adjustment. Summary of the invention

[0004] Based on this, it is necessary to provide a file storage adjustment method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of file storage adjustment in response to the above technical problems.

[0005] In a first aspect, the present application provides a file storage adjustment method, the method comprising:

[0006] When a storage adjustment request is received, determining a file to be adjusted and a target storage level corresponding to the storage adjustment request;

[0007] Determine an original storage pool corresponding to the file to be adjusted, and determine a target storage pool corresponding to the target storage level;

[0008] After changing the state of the file to be adjusted from an accessible state to a frozen state, migrating the data blocks of the file to be adjusted in the original storage pool to the target storage pool;

[0009] After the migration is completed, the state of the file to be adjusted is changed from the frozen state to the accessible state.

[0010] In one embodiment, the step of changing the state of the file to be adjusted from an accessible state to a frozen state includes:

[0011] generating a freezing notice for the file to be adjusted;

[0012] Sending the freezing notification to the sending end of the storage adjustment request, so that the sending end changes the state of the to-be-adjusted file from an accessible state to a frozen state based on the freezing notification;

[0013] The step of changing the state of the file to be adjusted from the frozen state to the accessible state includes:

[0014] generating a thawing notification for the file to be adjusted;

[0015] The thawing notification is sent to the sending end of the storage adjustment request, so that the sending end changes the state of the to-be-adjusted file from the frozen state to the accessible state based on the thawing notification.

[0016] In one embodiment, migrating the data blocks of the file to be adjusted in the original storage pool to the target storage pool includes:

[0017] After establishing a control command channel between the original storage pool and the target storage pool, packing the data blocks of the file to be adjusted in the original storage pool into a source file block object;

[0018] The source file segment object is sent to a target storage pool so that data blocks in the source file segment object are written into a target file segment object pre-created in the target storage pool.

[0019] In one embodiment, sending the source file block object to the target storage pool includes:

[0020] Storing the source file block object read in the original storage pool to the storage protocol stack of the kernel;

[0021] Adopting zero copy technology, copying the source file block object stored in the storage protocol stack of the kernel to the network protocol stack of the kernel;

[0022] The source file block objects stored in the kernel network protocol stack are sent to a target storage pool through the kernel network protocol stack.

[0023] In one embodiment, the number of the source file block objects is at least two, and sending the source file block objects to the target storage pool includes:

[0024] At least two of the original file block objects are sent to the target storage pool in a concurrent manner.

[0025] In one embodiment, the method further comprises:

[0026] After completing the data block migration, the metadata corresponding to the file to be adjusted is updated based on the storage information of the target file in the target storage pool, and the reference relationship between the metadata and the storage pool is updated; the metadata includes the file storage location.

[0027] In a second aspect, the present application further provides a file storage adjustment device, the device comprising:

[0028] A storage level determination module, configured to, when receiving a storage adjustment request, determine a file to be adjusted and a target storage level corresponding to the storage adjustment request;

[0029] A storage pool determination module, used to determine an original storage pool corresponding to the file to be adjusted, and to determine a target storage pool corresponding to the target storage level;

[0030] A data migration module, configured to migrate data blocks of the file to be adjusted in the original storage pool to the target storage pool after changing the state of the file to be adjusted from an accessible state to a frozen state;

[0031] The state modification module is used to modify the state of the file to be adjusted from the frozen state to the accessible state after the migration is completed.

[0032] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0033] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0034] A computer program product comprises a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0035] The above-mentioned file storage adjustment method, device, computer equipment, computer-readable storage medium and computer program product, when receiving a storage adjustment request, determine the file to be adjusted and the target storage level corresponding to the storage adjustment request; determine the original storage pool corresponding to the file to be adjusted, and determine the target storage pool corresponding to the target storage level; after changing the state of the file to be adjusted from an accessible state to a frozen state, migrate the data blocks of the file to be adjusted in the original storage pool to the target storage pool; after completing the migration, change the state of the file to be adjusted from a frozen state to an accessible state, by first changing the state of the file to a frozen state, it is avoided that the data blocks of the file are modified or read during the migration process, thereby ensuring the consistency and integrity of the data, the method allows the dynamic adjustment of the storage level of files containing data without the need for users to manually create new files or migrate data, by setting the file state to a frozen state, the migration of data can be completed in the background without suspending the operation of the system or affecting the normal operation of the user, which reduces manual operations and system downtime, and significantly improves the efficiency of storage adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A diagram of an application environment of a file storage adjustment method in an embodiment;

[0038] Figure 2 A schematic diagram of a flow chart of a file storage adjustment method in one embodiment;

[0039] Figure 3 A schematic diagram of a flow chart of a data migration step in an embodiment;

[0040] Figure 4 A schematic diagram of a flow chart of a file storage adjustment method in another embodiment;

[0041] Figure 5 A structural block diagram of a file storage adjustment device in one embodiment;

[0042] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] The file storage adjustment method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The above-mentioned file storage adjustment method can be executed separately by the server. Taking the execution on the server 104 as an example, the terminal 102 can send a storage adjustment request to the server. When the server 104 receives the storage adjustment request, it determines the file to be adjusted and the target storage level corresponding to the storage adjustment request; determines the original storage pool corresponding to the file to be adjusted, and determines the target storage pool corresponding to the target storage level; after changing the state of the file to be adjusted from the accessible state to the frozen state, migrate the data blocks of the file to be adjusted in the original storage pool to the target storage pool; after completing the migration, change the state of the file to be adjusted from the frozen state to the accessible state.

[0045] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0046] In an exemplary embodiment, Figure 2 As shown, a file storage adjustment method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S208. Among them:

[0047] S202: When a storage adjustment request is received, determine the file to be adjusted and the target storage level corresponding to the storage adjustment request.

[0048] The storage adjustment request is a request for adjusting the storage level of one or more files, and may carry information such as the identification of the file to be adjusted and the target storage level. The identification of the file to be adjusted may be the file path or file name of the file to be adjusted.

[0049] The files to be adjusted refer to files that need to have their storage levels adjusted. Specifically, they may be frequently used files or files that are infrequently accessed but need to be retained for a long time.

[0050] The target storage level refers to the storage level specified by the storage adjustment request, which can be a standard level or a low-frequency level. It is understandable that different storage levels correspond to different storage pools, where the storage pool corresponding to the standard level can be used to store frequently used files, and the storage pool corresponding to the low-frequency level can be used to store files that are not frequently accessed but need to be retained for a long time.

[0051] Specifically, when receiving a storage adjustment request, the server can parse the storage adjustment request to obtain a parsing result, and extract a file identifier and a storage level identifier from the parsing result, determine the file to be adjusted based on the file identifier, and determine the target storage level based on the storage level identifier.

[0052] In one embodiment, the server is a metadata server, a file management client runs on the terminal, and a file management page of the file management client is displayed. The storage administrator triggers a file storage adjustment operation for the file to be adjusted through the file management page. The terminal calls the "Set Metadata Extended Attribute Interface" in response to the file storage adjustment operation, thereby sending a storage adjustment request to the metadata server. After the metadata server receives the storage adjustment request through the "Set Metadata Extended Attribute Interface", it determines the file to be adjusted and the target storage level corresponding to the storage adjustment request through the tier deployed on the metadata server. Among them, the metadata server is a central server responsible for managing all file storage-related metadata, including file location, access rights, storage level, status information, etc. The tier is a component deployed on the metadata server, which is responsible for automatically identifying and implementing the storage level adjustment of files according to preset rules and policies.

[0053] In one embodiment, after receiving a file storage adjustment request, the server may first save the file storage adjustment request to an asynchronous queue through an asynchronous task manager, and then retrieve the storage adjustment request from the asynchronous queue through a tier deployed on a metadata server, and the tier determines the file to be adjusted and the target storage level corresponding to the storage adjustment request. The asynchronous task manager is responsible for monitoring the asynchronous queue and scheduling the execution of these storage adjustment tasks according to the system's resource status and priority. This allows the system to utilize resources more efficiently and process multiple storage adjustment requests at the same time.

[0054] S204, determining an original storage pool corresponding to the file to be adjusted, and determining a target storage pool corresponding to the target storage level.

[0055] The original storage pool is a storage pool that currently stores the data blocks of the file to be adjusted, and the target storage pool is a new storage pool where the data blocks of the file to be adjusted will be stored after the file migration.

[0056] Specifically, after determining the file to be adjusted, the server can obtain the metadata of the file to be adjusted, and determine the original storage pool of the file to be adjusted based on the metadata of the file to be adjusted, and search for the storage pool identifier from the parsing result of the storage adjustment request. If the storage pool identifier is found, the storage pool corresponding to the storage pool identifier is determined as the target storage pool. If the storage pool identifier is not found, multiple candidate storage pools matching the target storage level can be obtained, and the storage status information of each candidate storage pool can be obtained, and the target storage pool is selected from the candidate storage pools based on the storage status information.

[0057] Among them, the storage status information may specifically include the storage capacity, performance and other information of the storage pool. The storage capacity is used to determine whether there is enough space in the storage pool to store the data blocks of the files to be adjusted. The performance is used to determine whether the I / O performance and read / write speed of the storage pool meet the usage frequency and performance requirements of the files to be adjusted.

[0058] In one embodiment, the server is a metadata server. After a tierer is deployed on the metadata server and determines the file to be adjusted and the target storage level corresponding to the storage adjustment request, the metadata corresponding to the file to be adjusted can be obtained from the metadata pool, and the original storage pool of the file to be adjusted can be determined based on the metadata of the file to be adjusted.

[0059] S206: After the state of the file to be adjusted is changed from the accessible state to the frozen state, the data blocks of the file to be adjusted in the original storage pool are migrated to the target storage pool.

[0060] The state of the file to be adjusted is used to characterize the operability and accessibility of the file to be adjusted; the accessible state is the state of the file under normal circumstances, indicating that the file can be normally accessed by users or applications without any restrictions. In this state, the file can be read, written or executed; the frozen state is a special file state, which is used to indicate that the file cannot be modified or accessed temporarily. In the frozen state, the file cannot be modified, read, etc.

[0061] A data block is the smallest storage unit allocated to a file on a storage medium. It represents a fragment of the file content. By dividing files into blocks, the file system can manage, read, and write files on storage media (such as disks, SSDs, etc.) more efficiently.

[0062] In one embodiment, before migrating the file to be adjusted, the server may also adjust the status of the file to be adjusted, specifically including freezing the file to be adjusted, wherein freezing the file to be adjusted may specifically include modifying the permission setting of the file to be adjusted, thereby changing the status of the file to be adjusted from an accessible state to a frozen state.

[0063] In one embodiment, after the server changes the state of the file to be adjusted from the accessible state to the frozen state, it can use a data migration tool or script to migrate the data blocks of the file to be adjusted from the original storage pool to the target storage pool, and delete the data blocks of the file to be adjusted in the original storage pool.

[0064] S208, after the migration is completed, the state of the file to be adjusted is changed from a frozen state to an accessible state.

[0065] Specifically, after migrating the data blocks of the file to be adjusted to the target storage pool, data verification can be performed on the data blocks of the file to be adjusted in the target storage pool. After the verification is passed, the status of the file to be adjusted is adjusted, specifically including unfreezing the file to be adjusted, wherein unfreezing the file to be adjusted can specifically be restoring the permission settings of the file to be adjusted, thereby changing the status of the file to be adjusted from a frozen state to an accessible state.

[0066] Specifically, data verification may use methods such as checksum or hashing to compare file data in the original storage pool and the target storage pool to ensure that the data is not damaged or lost during the migration process.

[0067] In the above-mentioned file storage adjustment method, when the server receives a storage adjustment request, it determines the file to be adjusted and the target storage level corresponding to the storage adjustment request; determines the original storage pool corresponding to the file to be adjusted, and determines the target storage pool corresponding to the target storage level; after changing the state of the file to be adjusted from an accessible state to a frozen state, migrates the data blocks of the file to be adjusted in the original storage pool to the target storage pool; after completing the migration, changes the state of the file to be adjusted from a frozen state to an accessible state. By first changing the state of the file to a frozen state, it is avoided that the data blocks of the file are modified or read during the migration process, thereby ensuring the consistency and integrity of the data. This method allows dynamic adjustment of the storage level of files containing data without the need for users to manually create new files or migrate data. By setting the file state to a frozen state, data migration can be completed in the background without suspending the system operation or affecting the normal operation of the user, which reduces manual operations and system downtime, and significantly improves the efficiency of storage adjustment.

[0068] In one embodiment, the process by which the server changes the state of the file to be adjusted from an accessible state to a frozen state includes the following steps: generating a freeze notification for the file to be adjusted; sending the freeze notification to the sending end that stores the adjustment request, so that the sending end changes the state of the file to be adjusted from an accessible state to a frozen state based on the freeze notification; the process by which the server changes the state of the file to be adjusted from a frozen state to an accessible state includes the following steps: generating a thaw notification for the file to be adjusted; sending the thaw notification to the sending end that stores the adjustment request, so that the sending end changes the state of the file to be adjusted from a frozen state to an accessible state based on the thaw notification.

[0069] The freeze notification is a specific message or command used to notify a system or person to freeze specific files or data.

[0070] Specifically, before migrating the data blocks of the file to be adjusted, the server may also generate a freeze notification for the file to be adjusted, and send the freeze notification to the sending terminal that stores the adjustment request. After receiving the freeze notification, the sending terminal may parse the content of the freeze notification, determine that the file to be adjusted needs to be frozen, and freeze the file to be adjusted. After completing the freezing of the file to be adjusted, the result of successful freezing may be fed back to the server. After receiving the result of successful freezing, the server may start to migrate the data blocks of the file to be adjusted. After completing the migration of the data blocks of the file to be adjusted, the server generates a thaw notification for the file to be adjusted that has been migrated, and sends the thaw notification to the sending terminal that stores the adjustment request. After receiving the thaw notification, the sending terminal may parse the content of the thaw notification, determine that the file to be adjusted needs to be thawed, and thaw the file to be adjusted. After completing the thaw of the file to be adjusted, the result of successful thaw may be fed back to the server.

[0071] In one embodiment, the server is a metadata server, and a tier is deployed on the metadata server. Before migrating the data blocks of the file to be adjusted, the tier may generate a freeze notification for the file to be adjusted, and send the freeze notification to a terminal that sends a storage adjustment request. A freezer is deployed on the terminal, and the file to be adjusted is frozen by the freezer to prevent applications and operating systems from accessing the file to be adjusted. After completing the migration of the data blocks of the file to be adjusted, the tier generates a thaw notification for the file to be adjusted, and sends the thaw notification to a terminal that sends the storage adjustment request. A thawer is deployed on the terminal, and the file to be adjusted is thawed by the thawer to allow applications and operating systems to access the file to be adjusted.

[0072] In the above embodiment, the server avoids the modification or reading of the data blocks of the file during the migration process by first modifying the state of the file to a frozen state, thereby ensuring the consistency and integrity of the data, and restores the file to an accessible state after the migration is completed, so that the user can immediately access the latest data version, thereby enhancing the reliability of the system.

[0073] In one embodiment, the process of migrating the data blocks of the file to be adjusted in the original storage pool to the target storage pool by the server includes the following steps: after establishing a control command channel between the original storage pool and the target storage pool, packaging the data blocks of the file to be adjusted in the original storage pool into a source file block object; sending the source file block object to the target storage pool so that the data blocks in the source file block object are written into the target file block object pre-established in the target storage pool.

[0074] Among them, the Control Command Channel is a dedicated communication path used to transmit control commands and management information between two or more systems or components.

[0075] The source file block object refers to the data structure formed by encapsulating the data blocks of the files to be migrated in the original storage pool. It contains all or part of the data blocks of the files. These data blocks are the units of file segmentation on physical storage and are used as the basic units of transmission during the migration process.

[0076] The target file block object is a data structure pre-established in the target storage pool for receiving the migrated data blocks. It is the receiver of the source file block object data and ensures that the data blocks are correctly written into the target storage pool.

[0077] In one embodiment, the server is a metadata server, and the original storage pool and the target storage pool are respectively located in different data servers. For example, the original storage pool corresponds to the original data server, and the target storage pool corresponds to the target data server. The metadata server sends a file migration instruction to the original data server and the target data server. The original data server and the target data server establish a control command channel and a file data transmission channel between the original data server and the target data server based on the file migration instruction. The original data server packages the data blocks of the file to be adjusted in the original storage pool into the source file block object, and transmits the source file block object to the target data server through the established file data transmission channel. The target data server creates a target file block object. After receiving the source file block object, the target data server writes the data blocks of the file to be adjusted in the source file block object into the pre-established target file block object.

[0078] In one embodiment, the server is a metadata server, the original storage pool and the target storage pool are located in different data servers, for example, the original storage pool corresponds to the original data server, the target storage pool corresponds to the target data server, and the original data server and the target data server are respectively deployed with migrators, the metadata server sends a file migration instruction to the migrator of the original data server and the migrator of the target data server, and after the migrator of the original data server and the migrator of the target data server establish a control command channel based on the file migration instruction, the data blocks of the file to be adjusted in the original storage pool are packaged into the source file segmentation object, and the source file segmentation object is sent to the target storage pool, so as to write the data blocks in the source file segmentation object into the target file segmentation object pre-established in the target storage pool. Among them, the migrator is a specially designed software component responsible for processing and executing the data migration task from one storage system to another storage system, and the main responsibilities of the migrator include data reading, packaging, transmission, writing, and error handling and status reporting during the migration process.

[0079] In the above embodiment, the server ensures that the transmission of migration instructions is safe and controlled by establishing a control command channel between the original storage pool and the target storage pool, thereby reducing the risk of misoperation or data loss during the migration process, and packing the data blocks of the file to be adjusted in the original storage pool into the source file block object; the source file block object is sent to the target storage pool so that the data blocks in the source file block object are written into the target file block object pre-established in the target storage pool. Encapsulating data into the source file block object can maintain the integrity of the data block, ensuring that the data will not be damaged or tampered with during the migration process.

[0080] In one embodiment, the process of the server sending the source file block object to the target storage pool includes the following steps: storing the source file block object read in the original storage pool to the kernel's storage protocol stack; using zero copy technology, copying the source file block object stored in the kernel's storage protocol stack to the kernel's network protocol stack; sending the source file block object stored in the kernel's network protocol stack to the target storage pool through the kernel's network protocol stack.

[0081] Among them, zero copy technology is an optimization method for data transmission in computer systems. Its purpose is to reduce the number of times data is copied during transmission between applications and systems, thereby increasing data processing speed and reducing CPU load. In the traditional data transmission process, data often needs to be copied from kernel space (memory area managed by the operating system) to user space (memory area where user programs run), or vice versa. This copy operation consumes a lot of CPU resources and memory bandwidth, and increases response time. Zero copy technology avoids these unnecessary data copies and directly performs data reading and transmission operations in kernel space, thereby reducing system overhead.

[0082] Specifically, the server sends a migration instruction to the original storage pool. The original storage pool reads the source file block object in response to the migration instruction, and stores the read source file block object in the buffer of the kernel storage protocol stack corresponding to the original storage pool. Then, the zero-copy technology is used to copy the source file block object stored in the kernel storage protocol stack to the kernel network protocol stack without going through the user space. The kernel network protocol stack performs network encapsulation on the source file block object, adds necessary network transmission header information (such as TCP / IP header), and prepares to send it through the network. The encapsulated data is sent out through the network interface, and the destination is the target storage pool. The target storage pool receives the source file block object through its corresponding kernel network protocol stack, and uses the zero-copy technology to copy the source file block object in the kernel network protocol stack to the kernel storage protocol stack of the target storage pool, and writes the data block in the source file block object in the kernel storage protocol stack into the pre-established target file block object, thereby realizing the migration of the file to be adjusted.

[0083] In one embodiment, the server is a metadata server, and the original storage pool and the target storage pool are located in different data servers. For example, the original storage pool corresponds to the original data server, and the target storage pool corresponds to the target data server. Figure 3As shown, migrators are deployed on the original data server and the target data server respectively, wherein the migrator deployed on the original data server can be called a source migrator, and the migrator deployed on the target data server can be called a target migrator. The metadata server sends a migration instruction to the source migrator, and the source migrator reads the source file block object in response to the migration instruction, and stores the read source file block object in the buffer of the storage protocol stack of the kernel corresponding to the source migrator, and then uses the zero-copy technology to copy the source file block object stored in the storage protocol stack of the kernel to the network protocol stack of the kernel without going through the user space. The kernel network protocol stack performs network encapsulation on the source file block object, adds necessary network transmission header information (such as TCP / IP header), and prepares to send it through the network. The encapsulated data is sent out through the network interface, and the destination is the target migrator. The target migrator receives the source file block object through its corresponding kernel network protocol stack, and uses zero-copy technology to copy the source file block object in the kernel network protocol stack to the kernel storage protocol stack of the target migrator, and writes the data block in the source file block object in the kernel storage protocol stack into the pre-established target file block object, thereby realizing the migration of the files to be adjusted.

[0084] In the above embodiment, the server stores the source file block object read in the original storage pool to the kernel's storage protocol stack, and uses zero-copy technology to copy the source file block object stored in the kernel's storage protocol stack to the kernel's network protocol stack, and sends the source file block object stored in the kernel's network protocol stack to the target storage pool through the kernel network protocol stack. The zero-copy technology allows data to be transmitted directly within the kernel space, reducing the CPU's processing of data copying, thereby freeing up CPU resources to process other system tasks or process more network requests. By avoiding unnecessary memory copying, the data processing speed is improved, especially when transmitting large amounts of data.

[0085] In one embodiment, the number of source file segment objects is at least two, and the process of the server sending the source file segment objects to the target storage pool includes the following steps: sending at least two original file segment objects to the target storage pool in a concurrent manner.

[0086] Specifically, when the number of source file block objects is two or more, the server can establish multiple file data transmission channels between the original storage pool and the target storage pool, and start multiple transmission tasks at the same time according to a predetermined concurrency strategy, each task corresponding to a source file block object, to ensure the synchronous execution of concurrent transmissions.

[0087] In the above embodiment, the server sends at least two original file block objects to the target storage pool in a concurrent manner. Such concurrent operation allows multiple data streams to be performed simultaneously, reducing the overall transmission time. Processing and sending multiple file block objects at the same time can significantly improve the overall speed of data migration.

[0088] In one embodiment, the above-mentioned file storage adjustment method also includes the following steps: after completing the data block migration, updating the metadata corresponding to the file to be adjusted based on the storage information of the file to be adjusted in the target storage pool, and updating the reference relationship between the metadata and the storage pool; the metadata includes the file storage location.

[0089] Specifically, after completing the data migration, the data blocks of the file to be adjusted in the original storage pool are first deleted. After the deletion is completed, the metadata of the file to be adjusted is first updated to cancel the reference relationship of the metadata of the file to be adjusted to the original storage pool. Then, based on the storage information of the file to be adjusted in the target storage pool, the metadata of the file to be adjusted is updated for a second time to add a reference relationship of the metadata of the file to be adjusted to the target storage pool.

[0090] Among them, the first update can specifically be to delete the storage location information in the metadata of the file to be adjusted, so as to release the reference relationship of the metadata of the file to be adjusted to the original storage pool; the second update can specifically be to add new storage location information in the metadata of the file to be adjusted, so as to realize the reference relationship of the metadata of the newly added file to be adjusted to the target storage pool.

[0091] In the above embodiment, after completing the data block migration, the server updates the metadata corresponding to the file to be adjusted based on the storage information of the file to be adjusted in the target storage pool, and updates the reference relationship between the metadata and the storage pool, thereby preventing file access errors and data loss caused by outdated metadata. The metadata can directly access the stored data by accurately pointing to the new location of the file without searching or redirecting, thereby improving the efficiency and response speed of data access.

[0092] The present application also provides an application scenario, which can be specifically executed by a file hierarchical distributed system when applying the above-mentioned file storage adjustment method, such as Figure 4As shown, the file grading distributed system includes three functional modules: a tier, a freezer and a migrator. The tier is responsible for grading files, executing transactions, and rolling back exceptions. One module is deployed on each active metadata server. The freezer and thaw module is responsible for freezing and thawing file access. During the file grading process, the application or system is blocked from accessing the file. After the grading is completed, the access to the file needs to be released. One module is deployed on each file system access client. The migrator is responsible for high-performance and high-concurrency migration of file data between different data storage servers and different storage data storage pools. One module is deployed on each data storage server. The above-mentioned file storage adjustment method is executed through the system, which specifically includes the following steps:

[0093] 1. Front desk part.

[0094] 1. The storage administrator calls the metadata extended attribute setting interface system to adjust the file storage layout;

[0095] 2. The metadata extended attribute adjustment interface sends a network request to the distributed file system metadata server;

[0096] 3. The file system metadata server receives the storage layout adjustment request, and the asynchronous task manager saves the request persistently to the asynchronous queue, and the file classification foreground link ends;

[0097] 2. File classification background part.

[0098] 4.1. For each file to be classified, notify the freezer to perform the freezing operation;

[0099] 4.2. Freezer freezes files, preventing applications and operating systems from accessing them;

[0100] 5.1、Read the block objects of the file stored in the standard data pool;

[0101] 5.2. Write the block objects into the low-frequency data pool;

[0102] 6.1. Update file metadata and remove the reference relationship to the standard data pool;

[0103] 6.2. Update file metadata and add a reference relationship pointing to the data storage pool;

[0104] 7. Batch delete the original block objects of the standard pool;

[0105] 8.1. Notify the freezer to perform thawing operations

[0106] 8.2. The freezer unfreezes the files, allowing applications and the operating system to access the files.

[0107] By executing the above-mentioned file storage adjustment method through the above-mentioned file grading distributed system, rapid file grading can be achieved, reducing the impact on business access; and high-performance cross-data storage server data migration can be achieved to achieve rapid cross-node data migration.

[0108] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides a file storage adjustment device for implementing the file storage adjustment method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file storage adjustment device embodiments provided below can refer to the limitations of the file storage adjustment method above, and will not be repeated here.

[0110] In an exemplary embodiment, Figure 5 As shown, a file storage adjustment device is provided, including: a storage level determination module 502, a storage pool determination module 504, a data migration module 506 and a state modification module 508, wherein:

[0111] The storage level determination module 502 is used to determine the file to be adjusted and the target storage level corresponding to the storage adjustment request when receiving the storage adjustment request;

[0112] The storage pool determination module 504 is used to determine the original storage pool corresponding to the file to be adjusted, and determine the target storage pool corresponding to the target storage level;

[0113] A data migration module 506, configured to migrate data blocks of the file to be adjusted in the original storage pool to the target storage pool after changing the state of the file to be adjusted from the accessible state to the frozen state;

[0114] The state modification module 508 is used to modify the state of the file to be adjusted from the frozen state to the accessible state after the migration is completed.

[0115] In the above embodiment, when a storage adjustment request is received, the file to be adjusted and the target storage level corresponding to the storage adjustment request are determined; the original storage pool corresponding to the file to be adjusted, and the target storage pool corresponding to the target storage level are determined; after the state of the file to be adjusted is changed from an accessible state to a frozen state, the data blocks of the file to be adjusted in the original storage pool are migrated to the target storage pool; after the migration is completed, the state of the file to be adjusted is changed from a frozen state to an accessible state. By first changing the state of the file to a frozen state, the data blocks of the file are avoided from being modified or read during the migration process, thereby ensuring the consistency and integrity of the data. This method allows dynamic adjustment of the storage level of files containing data without the need for users to manually create new files or migrate data. By setting the file state to a frozen state, data migration can be completed in the background without suspending the system operation or affecting the normal operation of the user. This reduces manual operations and system downtime, and significantly improves the efficiency of storage adjustment.

[0116] In one embodiment, the state modification module 508 is further used to: generate a freeze notification for the file to be adjusted; send the freeze notification to the sending end of the storage adjustment request, so that the sending end changes the state of the file to be adjusted from an accessible state to a frozen state based on the freeze notification;

[0117] The state modification module 508 is further used to: generate a thawing notification for the file to be adjusted; and send the thawing notification to the sending end of the storage adjustment request, so that the sending end changes the state of the file to be adjusted from the frozen state to the accessible state based on the thawing notification.

[0118] In one embodiment, the data migration module 506 is further used to: after establishing a control command channel between the original storage pool and the target storage pool, pack the data blocks of the file to be adjusted in the original storage pool into a source file block object; send the source file block object to the target storage pool so that the data blocks in the source file block object are written into the target file block object pre-established in the target storage pool.

[0119] In one embodiment, the data migration module 506 is also used to: store the source file block object read in the original storage pool to the kernel's storage protocol stack; use zero copy technology to copy the source file block object stored in the kernel's storage protocol stack to the kernel's network protocol stack; and send the source file block object stored in the kernel's network protocol stack to the target storage pool through the kernel's network protocol stack.

[0120] In one embodiment, the number of the source file block objects is at least two, and the data migration module 506 is further used to send at least two of the original file block objects to the target storage pool in a concurrent manner.

[0121] In one embodiment, the device also includes a metadata update module, which is used to: after completing the data block migration, update the metadata corresponding to the file to be adjusted based on the storage information of the target file in the target storage pool, and update the reference relationship between the metadata and the storage pool; the metadata includes the file storage location.

[0122] Each module in the above-mentioned file storage adjustment device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file storage adjustment method is implemented.

[0124] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file storage adjustment method, characterized in that: The method comprises: When a storage adjustment request is received, determining a file to be adjusted and a target storage level corresponding to the storage adjustment request; Determine an original storage pool corresponding to the file to be adjusted, and determine a target storage pool corresponding to the target storage level; After changing the state of the file to be adjusted from an accessible state to a frozen state, migrating the data blocks of the file to be adjusted in the original storage pool to the target storage pool; After the migration is completed, the state of the file to be adjusted is changed from the frozen state to the accessible state.

2. The method according to claim 1, characterized in that The step of changing the state of the file to be adjusted from an accessible state to a frozen state includes: generating a freezing notice for the file to be adjusted; Sending the freezing notification to the sending end of the storage adjustment request, so that the sending end changes the state of the to-be-adjusted file from an accessible state to a frozen state based on the freezing notification; The step of changing the state of the file to be adjusted from the frozen state to the accessible state includes: generating a thawing notification for the file to be adjusted; The thawing notification is sent to the sending end of the storage adjustment request, so that the sending end changes the state of the to-be-adjusted file from the frozen state to the accessible state based on the thawing notification.

3. The method according to claim 1, characterized in that The step of migrating the data blocks of the file to be adjusted in the original storage pool to the target storage pool includes: After establishing a control command channel between the original storage pool and the target storage pool, packing the data blocks of the file to be adjusted in the original storage pool into a source file block object; The source file segment object is sent to a target storage pool so that data blocks in the source file segment object are written into a target file segment object pre-created in the target storage pool.

4. The method according to claim 3, characterized in that The step of sending the source file block object to the target storage pool includes: Storing the source file block object read in the original storage pool to the storage protocol stack of the kernel; Adopting zero copy technology, copying the source file block object stored in the storage protocol stack of the kernel to the network protocol stack of the kernel; The source file block objects stored in the kernel network protocol stack are sent to a target storage pool through the kernel network protocol stack.

5. The method according to claim 3, characterized in that: The number of the source file block objects is at least two, and sending the source file block objects to the target storage pool includes: At least two of the original file block objects are sent to the target storage pool in a concurrent manner.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: After completing the data block migration, the metadata corresponding to the file to be adjusted is updated based on the storage information of the target file in the target storage pool, and the reference relationship between the metadata and the storage pool is updated; the metadata includes the file storage location.

7. A file storage adjustment device, characterized in that: The device comprises: A storage level determination module, configured to, when receiving a storage adjustment request, determine a file to be adjusted and a target storage level corresponding to the storage adjustment request; A storage pool determination module, used to determine an original storage pool corresponding to the file to be adjusted, and to determine a target storage pool corresponding to the target storage level; A data migration module, configured to migrate data blocks of the file to be adjusted in the original storage pool to the target storage pool after changing the state of the file to be adjusted from an accessible state to a frozen state; The state modification module is used to modify the state of the file to be adjusted from the frozen state to the accessible state after the migration is completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images