File migration method, device, equipment, storage medium and program product

Abstract

The application provides a file migration method, device, equipment, storage medium and program product, and relates to the technical field of financial technology or the technical field of file migration. The method comprises the following steps: mounting a storage volume of a distributed storage system to a target server and a batch processing node; configuring a dynamic routing rule in the target server, and mapping a file access path requested by a user to a storage path of the storage volume, wherein a bottom path is set in the dynamic routing rule, and when the storage path of the storage volume is unavailable, the file access path requested by the user is mapped to the bottom path; and migrating bulk files and incremental files from an original server to the storage volume. The method provided in the application is suitable for mass file migration, and can improve the file migration efficiency.

Description

Technical Field

[0001] This application relates to the field of financial technology or the field of file migration technology, and in particular to a file migration method, apparatus, device, storage medium and program product. Background Technology

[0002] In the context of digital transformation and information technology application innovation (IT innovation) in the financial industry, the decommissioning and replacement of traditional Windows servers has become an important task. In this scenario, Windows servers have accumulated a large number of existing files over a long period of time, and they also need to receive incremental files from upstream systems through the Electronic Trading RAN (ETRAN) system every day.

[0003] In related technologies, channel applications use batch processing programs to decompress, rename, and export data from files, storing the results in an application node library, and finally providing file download functionality to users through online services. This approach struggles to handle the efficient migration of massive amounts of files. Summary of the Invention

[0004] This application provides a file migration method, apparatus, device, storage medium, and program product, which can be applied to the migration of massive files and improve file migration efficiency.

[0005] In a first aspect, this application provides a file migration method, comprising: mounting a storage volume of a distributed storage system to a target server and a batch processing node; configuring dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume, wherein the dynamic routing rules are configured with a fallback path, and when the storage path of the storage volume is unavailable, the file access path requested by the user is mapped to the fallback path; and migrating existing files and incremental files from the original server to the storage volume.

[0006] In one possible embodiment, mounting a storage volume of a distributed storage system to a target server and a batch processing node includes: mounting the storage volume to a local directory on the target server via a distributed file system protocol; and mounting the same storage volume to a container directory on the batch processing node.

[0007] In one possible embodiment, configuring dynamic routing rules in the target server includes: generating a configuration file for dynamic routing rules based on migration path identifiers; deploying the configuration file for dynamic routing rules to a reverse proxy server, so that the reverse proxy server matches the dynamic routing rules according to the file access path requested by the user.

[0008] In one possible embodiment, before migrating existing files from the original server to the storage volume, the file migration method further includes: compressing the existing files to generate a compressed file package; migrating existing files from the original server to the storage volume includes: decompressing the compressed file package to the local storage space of the target server.

[0009] In one possible embodiment, migrating incremental files from the original server to the storage volume includes: processing the incremental files in batches according to their file type and size; allocating the batched incremental files to batch processing nodes, which are used to perform preset operations on the incremental files to obtain incremental files after the preset operations; and migrating the preset processed incremental files from the original server to the storage volume.

[0010] In one possible embodiment, before mounting the storage volume of the distributed storage system to the target server and batch processing nodes, the file migration method further includes: collecting the resource status of the migration task in real time through a status monitoring interface, and adjusting the number of batch processing nodes based on the resource status of the migration task.

[0011] In one possible embodiment, after migrating existing and incremental files from the original server to the storage volume, the file migration method further includes: after the migration is completed, performing consistency verification on the existing and incremental files through a distributed verification mechanism; and triggering an automatic repair process when the consistency verification fails.

[0012] In one possible embodiment, the consistency verification of the migrated files through the distributed verification mechanism includes: calculating the verification value of the source file and the target file through a hash verification algorithm, and the automatic repair process includes at least one of the following: re-migrating the files that failed the verification, marking the files that failed the migration verification and recording the log.

[0013] Secondly, this application provides a file migration device, comprising: a mounting module for mounting a storage volume of a distributed storage system to a target server and a batch processing node; a path mapping module for configuring dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume, wherein the dynamic routing rules include a fallback path, which maps the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable; and a migration module for migrating existing files and incremental files from the original server to the storage volume.

[0014] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in any of the first aspects.

[0015] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0017] In this embodiment, the electronic device mounts a storage volume of a distributed storage system to the target server and batch processing nodes. This allows the target server and batch processing nodes to use the storage volume as a unified storage interface, and the local programs of the target server and batch processing nodes access the distributed storage system through this unified storage interface. The distributed storage system is compatible with both the source and target servers, eliminating heterogeneous differences between them and reducing the complexity of file migration. Furthermore, the distributed storage system has a larger storage capacity and supports expansion, making it suitable for massive file migrations as described in this embodiment, thus improving file migration efficiency.

[0018] Electronic devices can map user-requested file access paths to storage paths on storage volumes by configuring dynamic routing rules in the target server, thus enabling dynamic routing of user requests and ensuring seamless switching between different storage paths during file migration.

[0019] By setting a fallback path in dynamic routing rules, the risk of service interruption can be avoided by mapping the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable.

[0020] Efficient migration of existing and incremental files from the original server to the storage volume can be achieved. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of the file migration method according to an embodiment of this application;

[0023] Figure 2 This is a flowchart of a file migration method according to an embodiment of this application;

[0024] Figure 3 This is a flowchart of a file migration method according to another embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a file migration apparatus according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0030] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0031] It should be noted that the file migration method, apparatus, equipment, storage medium and storage product provided in this application can be used in the field of financial technology or the field of file migration technology, or in any field other than the field of financial technology or the field of file migration technology. This application does not limit the application field of the file migration method, apparatus, equipment, storage medium and storage product.

[0032] In related technologies, file migration mainly relies on traditional batch processing frameworks and local storage solutions. The specific process includes: file storage and processing, and service access.

[0033] File storage and processing: The Windows server acts as the core storage node, receiving upstream files through the Electronic Transaction System (ETRAN). The files are stored in compressed packages on the local file system. Batch processing programs (such as those based on Windows Task Scheduler or traditional batch scripts) are responsible for decompressing the files, renaming them to conform to business specifications, and exporting the file data to the application node library.

[0034] Service Access: Users access the file storage path on the Windows server through the online service and download files directly.

[0035] The relevant technologies have at least one of the following technical problems:

[0036] Efficiency bottleneck: Operations such as decompressing and renaming massive amounts of files rely on the processing power of a single machine, which is difficult to scale, and performance is prone to degradation when the number of files is large.

[0037] Poor storage scalability: Windows servers have limited local storage capacity, which cannot meet the storage needs of massive files accumulated over a long period of time.

[0038] Service continuity risk: If the storage path is switched directly during the migration process, it may result in interruption of user access or loss of files.

[0039] Compatibility issues: Traditional batch processing frameworks are not compatible with target servers (such as SUSE servers), requiring the refactoring of processing logic and increasing development costs.

[0040] Based on this, embodiments of this application provide a file migration method, apparatus, device, storage medium, and program product, aiming to solve at least one of the above-mentioned technical problems in the related art.

[0041] Figure 1 This is a schematic diagram illustrating an application scenario of the file migration method according to an embodiment of this application. For example... Figure 1 As shown, user 1 initiates a file download request through the financial application 3 installed on terminal device 2. Terminal device 2 then sends the file download request to target server 3. Target server 3 (SUSE server) and batch processing node 4 are mounted with storage volumes of distributed storage system 5. Target server 3 can distribute file download requests to one of the following: distributed storage system 5, its own local storage space, or Windows server 6. Batch processing node 4 can perform batch processing tasks for incremental files, which are then migrated to distributed storage system 5 after batch processing. Existing files can be manually migrated from Windows server 4 to the local storage space of target server 6 and then to distributed storage system 5.

[0042] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0043] Figure 2 This is a flowchart illustrating a file migration method according to an embodiment of this application. The file migration method described in this application can be executed by an electronic device.

[0044] like Figure 2 As shown, the file migration method of this application embodiment includes steps S110 to S130.

[0045] S110. Mount the storage volume of the distributed storage system to the target server and batch processing node.

[0046] A distributed storage system refers to a shared storage architecture composed of multiple storage devices, supporting cross-node data access. An example of a distributed storage system is a distributed file storage service (FSS). Storage volumes are, for example, FSS volumes, which are the logical units within a distributed file storage system.

[0047] Mounting refers to mapping a remote storage space to a local file system path to enable file read and write operations.

[0048] The target server is a SUSE server. Batch processing nodes are used to process batch files. Batch processing nodes may include multiple Platform as a Service (PaaS) containers, each of which is used to process at least a portion of the batch files.

[0049] S120. Configure dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume.

[0050] Dynamic routing rules refer to routing configurations that dynamically match stored paths based on user requests. Dynamic routing rules include fallback paths.

[0051] A fallback path refers to a pre-defined alternative path in a dynamic routing rule. For example, the fallback path could be the original storage system's storage path, such as a Windows server. When the storage path of a storage volume becomes unavailable, the file access path requested by the user is mapped to the fallback path.

[0052] S130. Migrate existing and incremental files from the original server to the storage volume.

[0053] The original server was, for example, a Windows server.

[0054] Existing files refer to files that were previously stored on the original server.

[0055] Incremental files refer to files downloaded from upstream.

[0056] In this embodiment, the electronic device mounts a storage volume of a distributed storage system to the target server and batch processing nodes. This allows the target server and batch processing nodes to use the storage volume as a unified storage interface, and the local programs of the target server and batch processing nodes access the distributed storage system through this unified storage interface. The distributed storage system is compatible with both the source and target servers, eliminating heterogeneous differences between them and reducing the complexity of file migration. Furthermore, the distributed storage system has a larger storage capacity and supports expansion, making it suitable for massive file migrations as described in this embodiment, thus improving file migration efficiency.

[0057] Electronic devices can map the file access path requested by the user to the storage path of the storage volume by configuring dynamic routing rules in the target server. This enables dynamic routing of user requests, ensuring seamless switching between different storage paths during file migration without the user's awareness.

[0058] By setting a fallback path in dynamic routing rules, the risk of service interruption can be avoided by mapping the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable.

[0059] Efficient migration of existing and incremental files from the original server to the storage volume can be achieved.

[0060] Figure 3 This is a flowchart of a file migration method according to another embodiment of this application.

[0061] like Figure 3 As shown in the embodiment of this application, step S110, which involves mounting the storage volume of the distributed storage system to the target server and the batch processing node, includes steps S111 to S112.

[0062] S111. Mount the storage volume to the local directory of the target server using a distributed file system protocol.

[0063] Distributed file system protocols refer to file system communication protocols that support mounting on multiple nodes. Examples of distributed file system protocols include the Network File System (NFS) protocol and the GlusterFile System (GlusterFS) protocol.

[0064] S112. Mount the same storage volume to the container directory of the batch processing node.

[0065] The container directory refers to the internal path of the container used to mount storage volumes in the batch processing node.

[0066] Through steps S111 and S112, the target server and the batch processing node can form a shared storage space, which is the same storage volume.

[0067] For example, during the mounting process described above, the distributed file system protocol can ensure consistency of access across multiple nodes through heartbeat detection and data synchronization mechanisms. This ensures that batch task processing on batch processing nodes is synchronized with file operations on the target server, thereby improving overall migration efficiency and reducing the risk of data conflicts.

[0068] In this embodiment, mounting the storage volume to the local directory of the target server via a distributed file system protocol ensures file read / write consistency. Simultaneously, mounting the same storage volume to the container directory of the batch processing node allows batch processing tasks to directly access the shared storage space of the same storage volume. Therefore, mounting the same storage volume to both the target server and the batch processing node supports parallel file read / write operations, avoiding the reduced migration efficiency and performance bottlenecks caused by input / output blocking during single-node file access.

[0069] like Figure 3 As shown, in one possible embodiment, the configuration of dynamic routing rules in the target server in step S120 includes steps S121 to S122.

[0070] S121. Generate a configuration file for dynamic routing rules based on the migration path identifier.

[0071] A dynamic routing configuration file is a configuration file that contains mappings between migration path identifiers and target storage paths, such as nginx.conf in Nginx. For example, in a dynamic routing configuration file, location / old points to the Windows server, and location / new points to the storage volume.

[0072] S122. Deploy the configuration file of the dynamic routing rules to the reverse proxy server so that the reverse proxy server matches the dynamic routing rules according to the file access path requested by the user.

[0073] A reverse proxy server is used to provide network proxy services that support dynamic routing. An example of a reverse proxy server is Nginx.

[0074] For example, an electronic device may pre-build a migration state management library. The migration state management library stores the mapping relationship between migration path identifiers and migration states, which includes the following fields: migration path identifier, migration state, target storage path, and fallback path switch. The electronic device's development script periodically reads the migration state management library and automatically generates a configuration file for dynamic routing rules.

[0075] For example, if the migration task is complete, a configuration file for generating dynamic routing rules based on the migration path identifier is used to configure the fallback path switch to be turned off and to indicate that the target storage path is the storage volume. If the migration task is not complete, a configuration file for generating dynamic routing rules based on the migration path identifier is used to configure the fallback path switch to be turned on and to indicate that the target storage path is the storage volume. If the storage volume search fails, the target storage path is switched to the fallback path.

[0076] In this embodiment, by generating a configuration file for dynamic routing rules based on the migration path identifier and deploying the configuration file to the reverse proxy server, the file access path requested by the user can be automatically and dynamically matched, solving the problem that static routing configuration cannot be dynamically adjusted during file migration. Specifically, the configuration file for dynamic routing rules defines the mapping relationship between the migration path identifier and the target storage path. This allows the reverse proxy server to route the file access path requested by the user to the corresponding storage path through the dynamic routing rules when it receives a user request, avoiding user request failures due to the inability to adjust the storage path.

[0077] Furthermore, dynamic routing rules can be flexibly configured according to business needs. Through the preset fallback paths in the dynamic routing rules, electronic devices can automatically identify the availability status of storage volumes and switch to the fallback path, reducing manual intervention. The reverse proxy server does not depend on a specific storage architecture and can be adapted to various distributed storage systems.

[0078] like Figure 2 As shown, in one possible embodiment, before migrating the existing files from the original server to the storage volume in step S130, the file migration method further includes step S141.

[0079] S141. Compress the existing files to generate a compressed file package.

[0080] For example, electronic devices use compression tools to compress existing files, generating compressed file packages. A compressed file refers to a software tool used for file compression. A compressed file package is a compressed archive containing multiple files.

[0081] like Figure 3 As shown, step S130, which involves migrating existing files from the original server to the storage volume, includes step S131.

[0082] S131. Extract the compressed file package to the local storage space of the target server.

[0083] For example, the electronic device supports manual migration of existing files via a File Transfer Protocol (FTP) visualization tool. Specifically, the electronic device allows business personnel to select existing files to be migrated using the FTP visualization tool and complete the migration of the existing files by executing steps S131 to S132.

[0084] In this embodiment, the electronic device compresses existing files to generate compressed file packages, thereby reducing the file size of existing files and lowering transmission bandwidth usage. After the electronic device transmits the compressed file package to the local storage space of the target server, the target server can restore the compressed file package to the storage volume mounted on the target server through a decompression operation. This shortens the migration cycle of existing files and reduces resource consumption during the migration process.

[0085] like Figure 3 As shown, in one possible embodiment, step S130, which involves migrating incremental files from the original server to the storage volume, includes steps S132 to S134.

[0086] S132. Process incremental files in batches according to file type and file size.

[0087] File type refers to the format or content classification of a file, such as text file or binary file.

[0088] File size refers to the storage capacity of a file, such as less than 1GB or greater than 1GB.

[0089] Batch processing refers to splitting an incremental file into multiple batches based on its file type and size.

[0090] S133. The batched incremental files are allocated to the batch processing nodes. The batch processing nodes are used to perform preset operations on the incremental files to obtain the incremental files after the preset operations.

[0091] For example, electronic devices can set preset operation processing priorities for different file types according to actual business needs.

[0092] For example, regarding file size, after incremental files are split into multiple batches, batches of files smaller than a first threshold can be processed in parallel by multiple PaaS containers on the batch processing node using preset operations. Batches of files larger than or equal to the first threshold can be sharded to obtain multiple shards, which are then processed in parallel by multiple PaaS containers using preset operations. The first threshold is, for example, 1GB.

[0093] For example, an electronic device may add computing resources to a PaaS container that performs preset operations on fragments to improve the preset operation processing speed of file batches whose file size is greater than or equal to a first threshold.

[0094] For example, the preset operation processes include decompression and renaming. The incremental file is also a compressed file package generated through compression processing. The batch processing node performs operations such as decompression and renaming on the compressed file package of the incremental file to ensure that the file naming of the incremental file meets business requirements.

[0095] S134. Migrate the pre-processed incremental files from the original server to the storage volume.

[0096] In this embodiment, the electronic device can improve file migration efficiency by processing incremental files in batches according to file type and file size, allocating the batched incremental files to batch processing nodes, and migrating the pre-processed incremental files from the original server to the storage volume. This is applicable to the massive file migration scenario in this embodiment.

[0097] For example, parallel processing of batches of files smaller than a first threshold avoids frequent switching of the task scheduler, reduces system overhead, and reduces resource contention.

[0098] Fragmenting files whose size is greater than or equal to the first threshold can fully utilize the bandwidth advantage of distributed storage and improve file migration efficiency.

[0099] like Figure 2 As shown, in one possible embodiment, before mounting the storage volume of the distributed storage system to the target server and batch processing node in step S110, the file migration method further includes step S101.

[0100] S101. Collect the resource status of the migration task in real time through the status monitoring interface, and adjust the number of batch processing nodes based on the resource status of the migration task.

[0101] A status monitoring interface is an interface used to collect real-time data on resource status (such as CPU utilization and storage bandwidth utilization). Computing resources include, for example, central processing unit (CPU) utilization and storage bandwidth utilization.

[0102] In this embodiment, the electronic device collects the resource status of the migration task in real time through a status monitoring interface, and adjusts the number of batch processing nodes based on the resource status of the migration task, thereby optimizing resource utilization and reducing operating costs. Specifically, the dynamic scaling mechanism that adjusts the number of batch processing nodes based on the resource status of the migration task can avoid resource idleness or overload, ensuring the efficient execution of the migration task. Allocating resources on demand can reduce unnecessary computation and storage overhead, and is especially suitable for long-running migration tasks (such as daily incremental file migration).

[0103] For example, when an electronic device detects insufficient storage bandwidth, it can increase the number of PaaS containers to automatically scale up. Conversely, when the completion rate of migration tasks exceeds a preset threshold, the electronic device can reduce the number of PaaS containers to automatically scale down, thereby saving computing resources. The preset threshold is, for example, set to 0.75.

[0104] like Figure 2 As shown, in one possible embodiment, after migrating existing and incremental files from the original server to the storage volume in step S130, the file migration method further includes steps S142 to S143.

[0105] S142. After the migration is completed, the consistency of existing files and incremental files is checked through a distributed verification mechanism.

[0106] Distributed verification mechanisms are used to perform parallel consistency checks on existing and incremental files through a distributed computing framework.

[0107] Consistency checks are used to compare the check values ​​of existing and incremental files before migration with those of existing and incremental files before and after migration, to ensure the data integrity of the migrated existing and incremental files.

[0108] For example, step S142, which performs consistency verification on the migration file through a distributed verification mechanism, includes: calculating the verification values ​​of the source file and the target file using a hash verification algorithm.

[0109] The source files are the existing and incremental files before the file migration, and the target files are the existing and incremental files after the file migration.

[0110] Hash verification algorithms are algorithms used to generate unique identifiers for files. Hash verification algorithms include Message-Digest Algorithm 5 (MD5) and Secure Hash Algorithm 256-bit (SHA-256).

[0111] S143. Trigger the automatic repair process when the consistency check fails.

[0112] The automatic repair process refers to the handling strategy after a verification failure.

[0113] For example, the automatic repair process includes at least one of the following: re-migrating the files that failed to be verified, marking the files that failed to be migrated and logging them.

[0114] In this embodiment, after migration is complete, the electronic device performs consistency checks on existing and incremental files using a distributed verification mechanism, and triggers an automatic repair process when consistency checks fail, ensuring data consistency and reliability during file migration. Specifically, the distributed verification mechanism covers massive file integrity checks on both existing and incremental files, preventing data loss due to transmission errors or storage anomalies. When consistency checks fail, the automatic repair process reduces manual troubleshooting costs, making it particularly suitable for scenarios with high file integrity requirements, such as financial scenarios.

[0115] Figure 4 This is a schematic diagram of the file migration device according to an embodiment of this application. Figure 4 As shown, the file migration apparatus provided in this application embodiment includes:

[0116] Mounting module 210 is used to mount the storage volume of the distributed storage system to the target server and batch processing node.

[0117] The path mapping module 220 is used to configure dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume. The dynamic routing rules include a fallback path, which maps the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable.

[0118] Migration module 230 is used to migrate existing and incremental files from the original server to the storage volume.

[0119] In one possible embodiment, the mounting module includes: a first mounting submodule for mounting the storage volume to a local directory of the target server via a distributed file system protocol; and a second mounting submodule for mounting the same storage volume to a container directory of a batch processing node.

[0120] In one possible embodiment, the path mapping module includes: a configuration file generation submodule, used to generate a configuration file for dynamic routing rules based on the migration path identifier; and a deployment submodule, used to deploy the configuration file for dynamic routing rules to the reverse proxy server, so that the reverse proxy server matches the dynamic routing rules according to the file access path requested by the user.

[0121] In one possible embodiment, the file migration apparatus further includes: a compression processing module for compressing existing files to generate a compressed file package. The migration module includes: a first migration submodule for decompressing the compressed file package to the local storage space of the target server.

[0122] In one possible embodiment, the migration module includes: a batch processing submodule for batch processing incremental files according to file type and file size; an allocation submodule for allocating the batched incremental files to batch processing nodes, whereby the batch processing nodes perform preset operation processing on the incremental files to obtain incremental files after preset operation processing; and a second migration submodule for migrating the preset processed incremental files from the original server to the storage volume.

[0123] In one possible embodiment, the file migration device further includes an adjustment module, configured to collect the resource status of the migration task in real time through a status monitoring interface, and adjust the number of batch processing nodes based on the resource status of the migration task.

[0124] In one possible embodiment, the file migration device further includes: a verification module, used to perform consistency verification on the existing files and incremental files through a distributed verification mechanism after the migration is completed; and a repair module, used to trigger an automatic repair process when the consistency verification fails.

[0125] In one possible embodiment, the verification module includes a verification submodule for calculating the verification value between the source file and the target file using a hash verification algorithm. The automatic repair process includes at least one of the following: re-migrating files that failed verification, marking files that failed verification during migration, and logging them.

[0126] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, this application embodiment provides an electronic device including a processor 301 and a memory 302. Optionally, the device further includes a communication component 303. The processor 301, memory 302, and communication component 303 are connected via a bus 304.

[0127] In the specific implementation process, the memory 302 stores code, and the processor 301 runs the code stored in the memory 302 to execute the method of the above method embodiment.

[0128] The specific implementation process of processor 301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0129] In the above Figure 5In the illustrated embodiments, it should be understood that the processor 301 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0130] The memory 302 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage.

[0131] Bus 304 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 304 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0132] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the above-described method embodiments.

[0133] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0134] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0135] This application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the embodiments described above.

[0136] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0137] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0138] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0139] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0140] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0141] If the integrated unit / module is implemented as a software program module and sold or used as an independent financial product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software financial product. This computer software financial product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0142] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0143] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file migration method, characterized in that, include: Mount the storage volume of the distributed storage system to the target server and batch processing node; Configure dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume. The dynamic routing rules include a fallback path, which maps the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable. Migrate existing and incremental files from the original server to the storage volume.

2. The method according to claim 1, characterized in that, The step of mounting the storage volume of the distributed storage system to the target server and batch processing node includes: The storage volume is mounted to the local directory of the target server using a distributed file system protocol; Mount the same storage volume to the container directory of the batch processing node.

3. The method according to claim 1, characterized in that, Configuring dynamic routing rules in the target server includes: A configuration file for generating dynamic routing rules is generated based on the migration path identifier; The configuration file of the dynamic routing rules is deployed to the reverse proxy server, so that the reverse proxy server matches the dynamic routing rules according to the file access path requested by the user.

4. The method according to claim 1, characterized in that, Before migrating existing files from the original server to the storage volume, the method further includes: The existing files are compressed to generate a compressed file package; The migration of existing files from the original server to the storage volume includes: The compressed file package is decompressed to the local storage space of the target server.

5. The method according to claim 1, characterized in that, The migration of incremental files from the original server to the storage volume includes: The incremental files are processed in batches according to their file type and size; The batched incremental files are allocated to the batch processing node, and the batch processing node is used to perform preset operation processing on the incremental files to obtain incremental files after preset operation processing. The pre-processed incremental files are migrated from the original server to the storage volume.

6. The method according to any one of claims 1-5, characterized in that, Before mounting the storage volume of the distributed storage system to the target server and batch processing nodes, the method further includes: The resource status of the migration task is collected in real time through the status monitoring interface, and the number of batch processing nodes is adjusted based on the resource status of the migration task.

7. The method according to any one of claims 1-5, characterized in that, After migrating existing and incremental files from the original server to the storage volume, the method further includes: After the migration is completed, a distributed verification mechanism is used to verify the consistency between the existing files and the incremental files. An automatic repair process is triggered when consistency checks fail.

8. The method according to claim 7, characterized in that, The consistency verification of the migration files through a distributed verification mechanism includes: The automatic repair process calculates the verification value between the source file and the target file using a hash verification algorithm, and includes at least one of the following: re-migrating files that failed verification, marking files that failed verification during migration, and logging them.

9. A file migration device, characterized in that, include: The mounting module is used to mount the storage volume of the distributed storage system to the target server and batch processing node; The path mapping module is used to configure dynamic routing rules in the target server to map the file access path requested by the user to the storage path of the storage volume. The dynamic routing rules include a fallback path, which maps the file access path requested by the user to the fallback path when the storage path of the storage volume is unavailable. The migration module is used to migrate existing and incremental files from the original server to the storage volume.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images