A lightweight cross-platform storage migration method, device and medium based on NETCore

Abstract

The present invention discloses a lightweight cross-platform storage migration method, device and medium based on NETCore, including: after receiving the start signal, the start end triggers the verification signal and performs the connection test; wherein, the trigger verification signal is used for the start end to perform the verification process according to the information filled in by the user in the preset file; the connection end performs the synchronization preparation operation and the progress feedback operation according to the user's selected reading mode information; after receiving the notification signal of the connection end completing the file loading, the synchronization end generates a file synchronization result signal. Its beneficial effect is that it is no longer limited to the types and manufacturers of both the file source and the file target storage, even if it is a binary file stored in the database, the local network address provided by Nfs can be read and uploaded.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud service storage, and specifically relates to a lightweight cross-platform storage migration method, device and medium based on NETCore. Background Art

[0002] With the popularity of enterprise software SaaS (Software as a Service, hereinafter referred to as SaaS) in recent years, more and more enterprises are choosing to implement SaaS because it can effectively reduce the cost of enterprise deployment and operation and maintenance, can iterate and update faster and adapt to market demand, and provides a second choice for traditional enterprises to try information technology for the first time with low cost trial and error.

[0003] SaaS implementation is generally implemented by third-party service providers using the various capabilities provided by the public cloud (Public Cloud) to implement and provide SaaS software services. In addition to using the ECS (Elastic Compute Service, Elastic Cloud Server, hereinafter referred to as ECS) provided by public cloud vendors to deploy applications, OSS (Object Storage Service, Object Storage Service, hereinafter referred to as OSS) is also used to manage the necessary components required for the operation of application software and the documents generated by production operations.

[0004] Enterprises that are moving to the cloud for the first time sometimes switch to another larger public cloud service group that is more suitable for the enterprise due to growth in business volume and version iterations, or the third-party service provider proactively changes the public cloud manufacturer. At this time, the enterprise has also been leasing the service for a long time. Since the third-party service provider belongs to the after-sales system (CRM), the main work content is work order reporting. In addition to facing huge database migrations, they also have to deal with the synchronization of dozens of GB, and some even TB of repair photos, videos and other files.

[0005] When an enterprise's local system is converted to SaaS or switched from another third-party supplier to our company, in addition to synchronizing historical data, all files carried by the historical data must also be synchronized to SaaS. Some manufacturers can upload and migrate through the tools they provide, and some local implementations will store the files in the database in binary, or store them in the network device Nas (Network Attached Storage) and provide a network access address. In this way, even the tools provided by public cloud vendors are helpless.

[0006] In response to the problems arising from the above implementation, service providers generally consider the following solutions:

[0007] 1. Use tools or services provided by existing public cloud vendors for migration and synchronization, such as Microsoft Azure Storage Explorer provided by Azure or obs-browser-plus provided by Huawei Cloud.

[0008] 2. Use file synchronization tools provided by third parties for migration, such as S3 Browser and WinSCP, which are specially designed for S3 (Amazon Simple Storage Service, Amazon file storage protocol, hereinafter referred to as S3).

[0009] In the above two solutions, the official tool mentioned in Solution 1 only has the function of manually uploading ordinary files to the cloud storage service in one direction. From the perspective of realizing the synchronization strategy of cloud storage to cloud storage, the tool currently cannot meet the relevant requirements and has certain limitations in achieving complete and efficient migration and synchronization tasks.

[0010] The third-party tools involved in Solution 2 also have some aspects that need to be considered. For example, S3Browser is not very versatile due to its targeted design, WinSCP's FTP capability has shortcomings when dealing with the S3 protocol, and its support for some cloud storage is weak, and it cannot adapt well to complex and diverse storage environments and task requirements. Therefore, these two solutions currently face challenges to varying degrees when dealing with data migration and file synchronization issues during the implementation of enterprise SaaS, and further exploration of more optimized and complete solutions is needed. Summary of the invention

[0011] In order to solve the problems raised in the above background technology, the present invention provides a lightweight cross-platform storage migration method based on NETCore.

[0012] To achieve the above object, the present invention provides the following technical solution: A lightweight cross-platform storage migration method based on NETCore includes:

[0013] After receiving the start signal, the start end triggers the verification signal and performs a connection test; wherein, the triggering verification signal is used for the start end to perform a verification process according to the information filled in by the user in the preset file;

[0014] After the connection end receives the signal that the initiator has completed the execution, the front end provides synchronization mode information for the user to select; the connection end reads the mode information according to the user's selection and performs synchronization preparation operations and progress feedback operations;

[0015] After receiving the notification signal from the connection end that the file loading is completed, the synchronization end generates a file synchronization result signal.

[0016] After receiving the start signal, the start end performs verification and connection test to ensure that the system performs necessary preparations before starting storage migration, improving the stability and reliability of storage migration. The orderly response of the connection end and the synchronization end realizes the automation and efficiency of the storage migration process.

[0017] Further: the verification process includes basic configuration information verification and configuration information to be executed verification;

[0018] The basic configuration information verification includes: judging the current software operating environment, the software log system status, the existence of the file relationship record to be synchronized, and the maximum number of files to be processed in parallel;

[0019] The configuration information verification to be performed includes: reading the configuration files of the synchronization source and synchronization target, confirming the file system types of the two parties to be synchronized, and verifying whether the configuration files are complete and correct.

[0020] The verification process includes basic configuration information verification and pending configuration information verification to ensure that basic configurations such as the software operating environment and log system status are correct. At the same time, the configuration files of the synchronization source and target are verified to ensure that the prerequisites for storage migration are accurate and correct, reducing the risk of errors during the migration process.

[0021] Further: when the basic configuration information verification and the configuration information to be executed verification are completed; the startup end performs a connection test;

[0022] The connection test includes: connection test and permission test on public cloud storage services, databases, and local file systems;

[0023] When the connection test is completed, the initiator determines the number of files to be synchronized and the address of the storage service file relationship according to the configuration information to be executed;

[0024] If the file source is a storage service source, read the file relationship record file and confirm that it is not empty and has the correct format.

[0025] Perform connection tests and permission tests on public cloud storage services, databases, and local file systems to ensure the accessibility and legitimacy of each data source during storage migration. Determine the number of files to be synchronized and the address of the storage service file relationship, and read the file relationship record file in the case of storage service sources, which helps to accurately plan storage migration tasks and improve migration efficiency.

[0026] Further: after the connection end receives the signal that the initiator has completed the execution, the front end provides synchronization mode information for the user to select, and the connection end reads the mode information according to the user's selection, and performs synchronization preparation operations and progress feedback operations;

[0027] Among them, the types of synchronization modes include: basic mode and performance mode;

[0028] The synchronization preparation operation includes: when receiving the user confirmation of the synchronization mode, the connection end reads the synchronization mode information to determine whether to enable the multi-file parallel transmission function;

[0029] Based on the detection result of the configuration information of the source and the target that has been completed by the initiator, the scheduling programs of the source and the target are started to enter the ready state;

[0030] When the user selects the basic mode, a database connection is opened in advance, and the database is used to record new file relationship information, and a temporary data table is created to record temporary data at the beginning of the synchronization operation;

[0031] According to the preset batch size, the list of files to be synchronized is retrieved from the file relationship record file and loaded into the memory batch by batch, wherein the file relationship record file is the metadata describing the file storage location and synchronization status;

[0032] The progress feedback operation includes: for each batch, recording the number of files successfully loaded into the memory, and generating a unique batch number to track the current loading progress;

[0033] The number of files loaded in each batch and the batch sequence number information are output to the command line interface so that users can monitor the progress of the synchronization operation in real time.

[0034] Provides synchronization mode selection to meet the needs of different users. The synchronization preparation operation and progress feedback operation in the basic mode enable users to monitor the synchronization progress in real time. At the same time, pre-opening the database connection and creating a temporary data table provide guarantees for implementing the retransmission function and accurately recording synchronization information. In the performance mode, the multi-file parallel transmission function is enabled to increase the speed of storage migration.

[0035] Further: after receiving the notification signal of the connection end completing the file loading, the synchronization end generates a file synchronization result signal; wherein generating the file synchronization result signal includes: the synchronization end obtains the file information loaded by the connection end to prepare for the subsequent file upload operation;

[0036] Start a loop process for the first batch of files loaded by the connection end;

[0037] When the user selects the basic mode for synchronization, before executing the file upload action, the synchronization end will query the database table specifically used to record the new file relationship to confirm whether the file has completed the synchronization process in order to realize the retransmission function; if there is no synchronization record of the file in the query database table, the scheduling end of the target storage will be called again to detect whether the current file already exists in the target storage, so as to avoid the problem of repeated file upload and overwriting. If it does exist, a corresponding prompt message will be given;

[0038] When the current batch of files is uploaded, the synchronization end calls the batch acquisition function provided by the connection end to obtain the next batch of files to be processed;

[0039] When obtaining the next batch of files to be processed, the synchronization end continuously executes the file upload process in a loop until the number of batch files obtained is less than the preset number standard of a batch of files, and determines that the last batch has been reached;

[0040] When the last batch of files is uploaded, the synchronization end performs statistical summary work, prints the file synchronization related data to the log system, and feeds back to the user in the command line interface to inform the user of the relevant information that the file synchronization strategy has been executed; at the same time, the synchronization end completes the resource cleanup and other finishing work, releases the relevant resources and restores the system to the initial state, waiting for the next synchronization task instruction.

[0041] The synchronization end generates a file synchronization result signal to ensure the integrity and traceability of storage migration. In the basic mode, query and detection are performed before file synchronization to avoid the problem of repeated file upload and overwriting, and improve data security. The file upload process is continuously executed in a loop until the last batch is completed, ensuring that all files to be synchronized can be processed. Statistical summary work and resource cleanup and finishing work make the system more standardized and efficient.

[0042] Further: starting a loop processing process for the first batch of files loaded by the connection end; wherein the loop processing process includes: automatically adapting and selecting a suitable upload strategy to transfer the files to the target storage according to the storage type of the source files;

[0043] For database binary storage files, they are first extracted from the database to the memory and then uploaded to the target in binary format. If it is a file storage system, the cloud vendor's established standard copy method is used to directly read the disk file and copy it to the target storage area.

[0044] A loop processing process is started for the first batch of files, and the upload strategy is automatically adapted according to the storage type of the source files, which improves the adaptability and flexibility of storage migration. Different processing methods are used for different types of files to ensure that the files can be correctly transferred from the source storage to the target storage.

[0045] Further: In basic mode, the connection end performs source verification to confirm whether the file has been synchronized, and queries the existence of the file in the target storage to avoid overwriting.

[0046] The connection end performs source verification and queries the existence of files in the target storage in basic mode, which further enhances the accuracy and reliability of storage migration in basic mode and avoids unnecessary synchronization operations and data overwriting problems.

[0047] Further: In performance mode, the connection end processes file transfers in parallel through multi-threading technology, and does not provide a retransmission mechanism, directly overwriting existing files at the target location.

[0048] In performance mode, the connection end uses multi-threading technology to parallelize file transfers, and does not provide a retransmission mechanism to directly overwrite existing files at the target location, meeting the needs of users who have high requirements for storage migration speed and improving migration efficiency in performance mode.

[0049] A lightweight cross-platform storage migration device based on NETCore, including:

[0050] The startup module is used to initialize the number of files to be pushed and the integrity of the target address;

[0051] The connection module is used to initialize the source and target of synchronization. If the source is a local file, there is no need to initialize the source. If it is a database, it will try to connect to the database in advance to retrieve the number of files to be synchronized and the binary integrity.

[0052] The synchronization module is used for replication synchronization. The synchronization end will automatically determine the acquisition and push scheme to be adopted based on the storage types of both parties.

[0053] The startup module, connection module and synchronization module are responsible for different functions, realizing the modularization and specialization of the storage migration process. It can initialize the number of retrieved files and the integrity of the target address, synchronize the source and target, perform replication synchronization and automatically determine the acquisition and push schemes, thus improving the practicality and efficiency of the storage migration device.

[0054] A computer-readable storage medium stores program instructions therein; when the program instructions are executed by a processor, a lightweight cross-platform storage migration method based on NETCore is implemented.

[0055] The startup module, connection module and synchronization module are responsible for different functions, realizing the modularization and specialization of the storage migration process. It can initialize the number of retrieved files and the integrity of the target address, synchronize the source and target, perform replication synchronization and automatically determine the acquisition and push schemes, thus improving the practicality and efficiency of the storage migration device.

[0056] The present invention solves the defects existing in the background technology and has the following beneficial effects:

[0057] 1. This lightweight cross-platform storage migration method based on NETCore has strong storage heterogeneity compatibility. Regardless of the type and manufacturer of the source and target storage, including binary files in the database, even local network address-related data provided by NFS (Network File System), it can achieve accurate reading and efficient upload. Its architecture can adapt to the needs of complex business scenarios, and it also performs well in lightweight file synchronization tasks, showing excellent flexibility and convenience.

[0058] 2. This lightweight cross-platform storage migration method based on NETCore breaks through the limitations of execution location and relies on the .NETCore cross-platform development framework to perfectly support the operation of the three major operating systems: Windows, MacOS, and Linux. It also has an open application programming interface (API) that can be called by other applications and can be started and executed through third-party programs, with good scalability and interactivity.

[0059] 3. This lightweight cross-platform storage migration method based on NETCore is based on a controllable file relationship recording mechanism, and ordinary users can independently set the files to be synchronized. Compared with conventional manufacturers that only support folder or keyword synchronization, the present invention can effectively reduce data traffic consumption and significantly improve synchronization speed, achieving the dual goals of traffic optimization and efficient synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0061] Figure 1 A schematic diagram of the overall execution flow chart of the present invention;

[0062] Figure 2 A schematic diagram of a startup end verification flow chart;

[0063] Figure 3 A schematic diagram of a connection end workflow diagram in the present invention;

[0064] Figure 4 A schematic diagram of a synchronization end workflow diagram in the present invention; DETAILED DESCRIPTION

[0065] The present invention will now be further described in detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams that only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0066] Application Overview:

[0067] Although Huawei Cloud Object Migration Service has rich source support, including Amazon Cloud (China), Alibaba Cloud and other cloud service providers, and has high transmission efficiency, its limitation is that it can only realize cloud-to-cloud operation mode, and is only applicable to cloud vendors that can open external network access to cloud storage. It cannot synchronize local files, nor can it interact with cloud vendors that only provide intranet file storage and non-supported cloud vendors such as Tianyi Cloud and Ziguang Cloud.

[0068] DragonDisk is a file synchronization tool for the S3 protocol developed by a foreign developer Tony in 2012. Although the tool supports local files as the source, can be used across platforms such as Windows, MacOS, and Linux, and can interact with some overseas cloud service providers including GoogleCloud and AwsS3, it does not support the file strategies used by domestic cloud providers, and is not even compatible with Microsoft's Azure, let alone uploading FTP files to the target storage.

[0069] Both of the above-mentioned two existing technologies have a common problem, that is, neither supports uploading files stored in binary form in the database to the storage service, and has a single target and can only be used in one direction, and cannot achieve the diversity of sources and targets, nor can it generate files from the storage service to the local file system.

[0070] This application uses a comprehensive verification process, including basic configuration information verification and pending configuration information verification, to check the software operating environment, log system, file relationship records, configuration file integrity and other aspects to ensure that the basic conditions for compatibility with various sources and target storage are ready. Regardless of the manufacturer or type of storage, from database binary files to storage from different cloud vendors, it can be processed. For example, in the connection test, the public cloud storage service, database, and local file system are connected and permission tested, so that storage from different sources can be identified and connected, and when determining information such as the number and address of files to be synchronized, it lays the foundation for subsequent operations and is no longer limited to specific cloud vendors or storage types.

[0071] This application uses the NETCore cross-platform development framework, which enables the program to run on three platforms: Windows, MacOS, and Linux, without relying on a specific operating system. For local file access methods, whether it is FTP or NAS, etc., they can be processed regardless of their differences. This is because in a series of operations at the startup end, the connection end, and the synchronization end, through reasonable module design and process arrangement, from the startup verification and connection test, to the synchronization preparation at the connection end, and then to the file processing at the synchronization end, each link works together to adapt to the differences between different platforms and access methods, thereby breaking the barriers to local file migration with cloud vendors.

[0072] In the synchronization process, this application has a special processing method for database binary storage files, which are first extracted to the memory and then uploaded in binary format, ensuring that the binary files in the database can be smoothly migrated to the target storage. At the same time, two execution modes, basic mode and performance mode, are provided to meet the needs of different scenarios. The source verification, target position detection and retransmission functions in the basic mode ensure the accuracy and reliability of data synchronization, which is suitable for scenarios with high requirements for data accuracy and small file size; the multi-threaded parallel processing and file overwrite mechanism in the performance mode greatly improve the synchronization speed while satisfying the premise that users can change the parallel processing parameters. It is suitable for scenarios with high speed requirements and relatively low requirements for data accuracy. And through controllable file relationship records, users can set synchronization files, reduce unnecessary traffic consumption and increase synchronization speed, solving the problems of high traffic consumption and slow speed in traditional synchronization solutions.

[0073] Exemplary methods:

[0074] like Figure 1 As shown, it is a schematic diagram of the overall execution process of a lightweight cross-platform storage migration method based on NETCore in the present invention.

[0075] The program will use NETCore (.NetCore, Microsoft's cross-platform development framework) as the low-level layer. Since it will serve cross-platform (supporting Windows, MacOS, and Linux), the execution program will not be developed in an interface. It will run with configuration files and CMD (Command Prompt, hereinafter referred to as CMD). The program will provide two execution modes, namely: basic mode and performance mode.

[0076] Basic mode: synchronize one file at a time. Before synchronizing each file, the source side will verify whether it has been synchronized before and whether the target location already exists. After successful synchronization, the post-synchronization relationship and synchronization log will be recorded. Retransmission is supported, and the speed is slow.

[0077] Performance mode: According to the maximum number of files to be processed in parallel set in the configuration file, the user can change it, and the maximum supported is 200. Multi-threaded mode is automatically turned on to upload multiple files at the same time. No source verification synchronization record is performed, and no judgment is made as to whether the target location already exists. After synchronization, only failure / success logs are recorded, and new relationships are not recorded. Retransmission is not supported, and files are overwritten by default. The speed is the fastest.

[0078] The method specifically comprises the following steps:

[0079] A lightweight cross-platform storage migration method based on NETCore includes:

[0080] The startup end verification flow chart is as follows Figure 2 As shown, after the program is run, the startup end verification process will be automatically called up, and verification will be initiated actively according to the information filled in by the user in the program preset file.

[0081] S11. Basic configuration information verification: The startup end will determine the current software operating environment, whether the software log system is normal, whether the relationship records of the files to be synchronized exist, and the maximum number of files to be processed in parallel according to the content of the configuration file preset for the user.

[0082] S12. Verification of configuration information to be executed: The initiator starts to read two configuration files, namely OSSSource (synchronization source configuration) and OSSTarget (synchronization target configuration), to confirm the types of file systems of the two parties to be synchronized and whether there is any missing or mismatched configuration.

[0083] S13, Connection Test: Based on the information confirmed above, if one party is a cloud storage service provided by a public cloud vendor, try to call basic acquisition and upload operations through the identity information provided in the configuration file. The source only tests acquisition, while the target tests both upload and acquisition.

[0084] If the source in the above information is a database, the RemoteConnectionTest will be tried using the database connection information to confirm whether the database is accessed normally;

[0085] If the source or target in the above information is a common file system (FTP, NFS), a file system folder permission access test will be performed, and file acquisition or creation operations will be performed to confirm whether the user executing the program meets the local file system permissions.

[0086] S14. Confirm the number of files to be synchronized and the storage service file relationship address: According to the configuration information retrieved in the first step, confirm the number of files to be synchronized. If it is a storage service source, it is necessary to read the file relationship record file to confirm that the record is not empty and the storage format is correct.

[0087] Glossary:

[0088] File relationship record: In software development, when it comes to file storage operations, the most common solution is to record the relative location of the file on the operating system disk (filefullname) in the database after the upload is completed. The same is true for storage services (generally referring to OSS services provided by public cloud vendors, and also locally deployed minio, FTP and other services that provide online addresses). Each file will have a relative address stored in the software's persistent storage (static files are generally Excel or CSV, database) as a record.

[0089] Further: The connection end workflow is as follows Figure 3 As shown, after the execution of the program startup end is completed, it will stay on the mode selection interface for the user to select the relevant mode (basic mode, performance mode). After the user selects the mode to confirm the synchronization, the connection end starts the job, actively activates the source and target callers, loads files or file relationship records in batches from the list to be synchronized, and records the current number of successful loads and the current batch number of all batches, which are displayed in CMD.

[0090] S21. Read the execution mode selected by the user to confirm whether to enable simultaneous transmission of multiple files.

[0091] S22. Since the initiator has already tested the configuration information of the source and target parties, the schedulers of both parties are directly opened here and are ready.

[0092] S23. If the user selects the basic mode, the connection end will pre-open a database connection for recording the new relationship, create a temporary table for recording, and store a temporary data to indicate the start of the current synchronization.

[0093] S24. The connection end starts to load the first batch of files to be processed from the file relationship record, loads them into the memory, prints them into the log file, and feeds back to the user on the CMD: currently there are xx files to be synchronized, the current batch is the xth batch, and this batch loads xx files. If the source is a binary file stored in the database, it is loaded from the database into the memory.

[0094] Further: The synchronization end workflow is as follows Figure 4 As shown, the connection end will notify the synchronization end after loading is completed. The synchronization end will start to upload files with all the loaded information, and perform different detailed operations according to the selected mode. After the current batch ends, the loading method provided by the connection end will continue to load the next batch until the last batch is executed. The statistical summary is printed in the log system and fed back to the user on the CMD: The file synchronization policy has ended, the total number of files: xxx, success: xxx, failure: xxx.

[0095] S31. The first batch of files loaded by the loop connection end are automatically uploaded to the target storage according to the storage type of the source files (local file system, cloud storage file system, files stored in database binary). If the files are stored in database binary, they are read from the library and loaded into the memory, and then uploaded to the target in binary. If it is a file storage system, the standard copy method provided by the cloud vendor is used to directly read the disk file and copy it to the target storage.

[0096] S32. If the basic mode is selected, the database table that records the new relationship created in advance will be queried before uploading to confirm whether the file has been synchronized for the retransmission function. If there is no record, the scheduler of the target storage will be called again to determine whether the current file exists in the target storage to avoid overwriting. If it exists, a prompt will be given.

[0097] S33. After the current batch is executed, the batch acquisition provided by the connection end will be called again to obtain the next batch. If the number of files in the acquired batch is less than the number of a batch, it will be considered as the last batch and ready to end and count.

[0098] A lightweight cross-platform storage migration device based on NETCore, including:

[0099] The startup module is used to initialize the number of files to be pushed and the integrity of the target address;

[0100] The connection module is used to initialize the source and target of synchronization. If the source is a local file, there is no need to initialize the source. If it is a database, it will try to connect to the database in advance to retrieve the number of files to be synchronized and the binary integrity.

[0101] The synchronization module is used for replication synchronization. The synchronization end will automatically determine the acquisition and push scheme to be adopted based on the storage types of both parties.

[0102] A computer-readable storage medium stores program instructions therein; when the program instructions are executed by a processor, a lightweight cross-platform storage migration method based on NETCore is implemented.

[0103] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the lightweight cross-platform storage migration method based on NETCore according to various embodiments of the present application described in the above method section of this specification.

[0104] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0105] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the lightweight cross-platform storage migration method based on NETCore according to various embodiments of the present application described in the above method section of this specification.

[0106] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0108] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example.

[0109] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A lightweight cross-platform storage migration method based on NETCore, characterized in that include: After receiving the start signal, the start end triggers the verification signal and performs a connection test; wherein, the triggering verification signal is used for the start end to perform a verification process according to the information filled in by the user in the preset file; After the connection end receives the signal that the initiator has completed the execution, the front end provides synchronization mode information for the user to select; the connection end reads the mode information according to the user's selection and performs synchronization preparation operations and progress feedback operations; Among them, the types of synchronization modes include: basic mode and performance mode; The synchronization preparation operation includes: when receiving the user confirmation of the synchronization mode, the connection end reads the synchronization mode information to determine whether to enable the multi-file parallel transmission function; Based on the detection result of the configuration information of the source and the target that has been completed by the initiator, the scheduling programs of the source and the target are started to enter the ready state; When the user selects the basic mode, a database connection is opened in advance, and the database is used to record new file relationship information, and a temporary data table is created to record temporary data at the beginning of the synchronization operation; According to the preset batch size, the list of files to be synchronized is retrieved from the file relationship record file and loaded into the memory batch by batch, wherein the file relationship record file is the metadata describing the file storage location and synchronization status; The progress feedback operation includes: for each batch, recording the number of files successfully loaded into the memory, and generating a unique batch number to track the current loading progress; Output the number of files loaded in each batch and the batch number information to the command line interface so that users can monitor the progress of the synchronization operation in real time; After receiving the notification signal from the connection end that the file loading is completed, the synchronization end generates a file synchronization result signal.

2. According to a NETCore-based lightweight cross-platform storage migration method according to claim 1, it is characterized in that: The verification process includes basic configuration information verification and configuration information verification to be executed; The basic configuration information verification includes: judging the current software operating environment, the software log system status, the existence of the file relationship record to be synchronized, and the maximum number of files to be processed in parallel; The configuration information verification to be performed includes: reading the configuration files of the synchronization source and synchronization target, confirming the file system types of the two parties to be synchronized, and verifying whether the configuration files are complete and correct.

3. According to a NETCore-based lightweight cross-platform storage migration method according to claim 1, it is characterized in that: When the basic configuration information verification and the configuration information to be executed verification are completed; The initiator performs a connection test; The connection test includes: connection test and permission test on public cloud storage services, databases, and local file systems; When the connection test is completed, the initiator determines the number of files to be synchronized and the address of the storage service file relationship according to the configuration information to be executed; If the file source is a storage service source, read the file relationship record file and confirm that it is not empty and has the correct format.

4. According to a NETCore-based lightweight cross-platform storage migration method according to claim 1, it is characterized in that: After receiving the notification signal of the connection end completing the file loading, the synchronization end generates a file synchronization result signal; wherein generating the file synchronization result signal includes: the synchronization end obtains the file information loaded by the connection end to prepare for the subsequent file upload operation; Start a loop process for the first batch of files loaded by the connection end; When the user selects the basic mode for synchronization, before executing the file upload action, the synchronization end will query the database table specifically used to record the new file relationship to confirm whether the file has completed the synchronization process in order to realize the retransmission function; if there is no synchronization record of the file in the query database table, the scheduling end of the target storage will be called again to detect whether the current file already exists in the target storage, so as to avoid the problem of repeated file upload and overwriting. If it does exist, a corresponding prompt message will be given; When the current batch of files is uploaded, the synchronization end calls the batch acquisition function provided by the connection end to obtain the next batch of files to be processed; When obtaining the next batch of files to be processed, the synchronization end continuously executes the file upload process in a loop until the number of batch files obtained is less than the preset number standard of a batch of files, and determines that the last batch has been reached; When the last batch of files is uploaded, the synchronization end performs statistical summary work, prints the file synchronization related data to the log system, and feeds back to the user in the command line interface to inform the user of the completion of the file synchronization strategy execution; at the same time, the synchronization end completes resource cleanup, releases related resources and restores the system to its initial state, waiting for the next synchronization task instruction.

5. According to a NETCore-based lightweight cross-platform storage migration method according to claim 4, it is characterized in that: A loop processing process is started for the first batch of files loaded by the connection end; wherein the loop processing process includes: automatically adapting and selecting an appropriate upload strategy to transfer the files to the target storage according to the storage type of the source files; For database binary storage files, they are first extracted from the database to the memory and then uploaded to the target in binary format. If it is a file storage system, the cloud vendor's established standard copy method is used to directly read the disk file and copy it to the target storage area.

6. According to a NETCore-based lightweight cross-platform storage migration method according to claim 4, it is characterized in that: In basic mode, the connection end performs source verification to confirm whether the file has been synchronized, and queries the existence of the file in the target storage to avoid overwriting.

7. According to a NETCore-based lightweight cross-platform storage migration method according to claim 4, it is characterized in that: In performance mode, the connection end processes file transfers in parallel through multi-threading technology, does not provide a retransmission mechanism, and directly overwrites existing files at the target location.

8. A lightweight cross-platform storage migration device based on NETCore, used to implement a lightweight cross-platform storage migration method based on NETCore according to any one of claims 1 to 7, characterized in that: include: The startup module is used to initialize the retrieval of the number of files to be pushed and the integrity of the target address of the files to be synchronized; The connection module is used to initialize the source and target of synchronization. If the source is a local file, there is no need to initialize the source. If it is a database, it will try to connect to the database in advance to retrieve the number of files to be synchronized and the binary integrity. The synchronization module is used for replication synchronization. The synchronization end will automatically determine the acquisition and push schemes to be adopted according to the storage types of both parties. The synchronization modes include: basic mode and performance mode. The synchronization preparation operation includes: when receiving the user confirmation of the synchronization mode, the connection end reads the synchronization mode information to determine whether to enable the multi-file parallel transmission function; Based on the detection result of the configuration information of the source and the target that has been completed by the initiator, the scheduling programs of the source and the target are started to enter the ready state; When the user selects the basic mode, a database connection is opened in advance, and the database is used to record new file relationship information, and a temporary data table is created to record temporary data at the beginning of the synchronization operation; According to the preset batch size, the list of files to be synchronized is retrieved from the file relationship record file and loaded into the memory batch by batch, wherein the file relationship record file is the metadata describing the file storage location and synchronization status; The progress feedback operation includes: for each batch, recording the number of files successfully loaded into the memory, and generating a unique batch number to track the current loading progress; The number of files loaded in each batch and the batch sequence number information are output to the command line interface so that users can monitor the progress of the synchronization operation in real time.

9. A computer-readable storage medium having program instructions stored therein, characterized in that: When the program instructions are executed by the processor, they implement a lightweight cross-platform storage migration method based on NETCore as described in any one of claims 1-7.

Images