File checking and repairing method based on proxy migration mechanism
Through the file verification and repair method based on the proxy migration mechanism, the problems of existing tools in retrieval efficiency, computing power utilization and resource consumption are solved, and an efficient and stable file migration process is achieved to ensure data integrity and rapid repair.
Patent Information
- Application Number
- CN202411048969.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Existing file migration tools have shortcomings in data retrieval efficiency, host computing power utilization rate and resource consumption, resulting in inefficiency in the migration process and waste of resources.
The file verification and repair method based on the proxy migration mechanism is adopted, and through full traversal and incremental transmission, real-time differentiated information verification and slicing repair, resource utilization and retrieval efficiency are optimized, and collaborative computing and rapid repair are achieved.
It improves the efficiency and stability of the file migration process, ensures the accuracy and integrity of data transmission, and reduces the risk of resource consumption and data loss.
Smart Images

Figure CN119071285A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of file migration, and in particular to a file checking and repairing method based on an agent migration mechanism. Background Art
[0002] In the current technological context, with the rapid development of technologies such as cloud computing and big data, as well as the growth of data volume and storage demand, the widespread application of file migration technology promotes digital transformation and information technology innovation, and becomes one of the keys to promoting innovation and development in various industries. Enterprises may need to migrate a large amount of file data from local storage to cloud platforms or new data centers, which makes the demand for file migration increasingly urgent.
[0003] File migration is the process of transferring files from one location to another. In this process, factors such as data integrity, migration speed, and resource usage must also be considered, which makes "file data verification" and "file data repair" important steps to ensure data quality.
[0004] File data verification is used to verify the accuracy, completeness and consistency of data to avoid errors, loss or inconsistencies. File data repair is a key measure to prevent data loss and damage, and helps maintain data integrity and reliability. In an actual production environment, users can use file data verification to promptly discover and resolve potential problems to ensure the quality of data after migration. Afterwards, file data repair can ensure the accuracy and completeness of data during the migration process and reduce the risk of data loss and damage.
[0005] Traditional file copying methods can only simply transfer data, but cannot verify and repair data in real time, resulting in a time-consuming and laborious migration process. Therefore, modern file migration tools are usually used. Although some migration tools support verification and repair functions, there are still some problems that need to be solved. (1) First, the retrieval efficiency is too low: Although some tools provide verification and repair functions, they still have room for improvement in data retrieval efficiency. Due to the large amount of data or high complexity, traditional tools often cannot achieve efficient data verification and rapid error location, resulting in less than ideal execution efficiency. (2) Secondly, collaborative computing is difficult: Existing file migration tools have limitations in utilizing host computing power. Generally, traditional tools can only utilize host computing power in one direction, and it is difficult to perform collaborative computing through the source data machine and the destination data machine, resulting in low resource utilization when the tool is performing verification and repair, limiting the efficiency and speed of the file migration process. (3) In addition, too many resources are occupied: The file verification and repair process often occupies a large amount of resources, including computing resources, storage resources, etc., resulting in large resource consumption, affecting the stability and efficiency of the overall file migration process. Summary of the invention
[0006] The purpose of the present invention is to provide a file verification and repair method based on a proxy migration mechanism, aiming to overcome the challenges of current file migration technology such as low retrieval efficiency, insufficient host computing power utilization and excessive resource consumption, and to achieve an efficient and stable file migration process.
[0007] The technical solution adopted by the present invention is:
[0008] A file checking and repairing method based on an agent migration mechanism comprises the following steps:
[0009] Step 1: The source data machine traverses all files and then transfers all files.
[0010] Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so as to continuously synchronize the incremental data recorded in real time after the full transfer is completed;
[0011] Step 3, determine whether the user chooses to stop incremental transmission during the file transfer process; if so, the host that stops the incremental transmission enters the "waiting for migration and cutover" state; otherwise, execute step 5;
[0012] Step 4, determining whether the user chooses to perform file transfer verification and repair; if yes, executing step 5; otherwise, executing step 6;
[0013] In step 5, the host performs real-time differential information verification through fast file comparison, and performs a slice repair process based on the differential information to retransmit the lost or damaged files; this step can ensure the consistency of the files at the target location with the files at the source location, including confirming that all files have been successfully transferred and performing necessary corrective operations, such as retransmitting lost or damaged files.
[0014] Step 6: Wait for the full file transfer to be completed and synchronize the incremental data recorded in real time to complete the file migration.
[0015] Furthermore, in step 1, metadata information of the file is recorded during the full traversal, and the metadata information includes the file size and path.
[0016] Furthermore, when the host starts to check and repair in step 5, a comparison instruction is issued through the migration client, and the source data machine and the target data machine will work together to perform a quick file comparison and generate a detailed difference list through file comparison; when there are difference values in this file comparison, the corresponding difference list will be repaired.
[0017] Furthermore, step 5 specifically includes the following steps:
[0018] Step 5-1: The host starts to check and repair, and issues a comparison command through the migration client;
[0019] Step 5-2, the source data machine and the target data machine will work together to quickly compare files and generate a detailed difference list;
[0020] Step 5-3, judging whether there is differential information based on the difference list; if yes, performing a repair operation on the corresponding difference list and then executing step 5-4; otherwise, executing step 5-4;
[0021] Step 5-4, manually recheck whether the repair is successful; if yes, complete the repair; otherwise, execute step 5-1.
[0022] Furthermore, the rapid file comparison in step 5 specifically includes the following steps:
[0023] Step 5-11, when the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules;
[0024] Step 5-12, the source end data machine sends a command with a comparison order list to the target end data machine for synchronization operation;
[0025] Step 5-13, based on the performance of the source and target data machines, the optimal data extraction step value is calculated and selected; the purpose is to balance the speed and accuracy of the comparison. The smaller the step value, the denser the information extracted during the comparison process, and the corresponding comparison speed may be reduced; the larger the step value, the faster the comparison, but some subtle differences may be missed.
[0026] Step 5-14, based on the optimal data extraction step value, traverse the files to be compared one by one in the order in the comparison order list, and perform a hash operation on the file contents to convert the file contents into a unique hash value of a fixed length;
[0027] Step 5-15, compare the hash values of the corresponding files of the source data machine and the target data machine; if the hash values are different, it is determined that there are differences in the contents of the corresponding files, and the system records the corresponding difference information.
[0028] Furthermore, the selection of the optimal data extraction step value in step 5-13 specifically includes the following steps:
[0029] Step 5-13-1, initialize the step reference value, set to the total amount of data divided by 1024;
[0030] Step 5-13-2, the current step value is set to 2 / 3 of the base value;
[0031] Step 5-13-3, simulate and generate hash values based on the current step value, and calculate the host CPU and memory usage;
[0032] Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.
[0033] Furthermore, the steps of extracting the hash calculation in step 5-14 are as follows:
[0034] Step 5-14-1, construct an all-zero sequence, that is, set all data to zero value;
[0035] Step 5-14-2, obtaining data content extracted from a source file or database based on an optimal data extraction step value;
[0036] Step 5-14-3, extracting data samples from the extracted data content to obtain sample extraction results;
[0037] Specifically, data sampling is the process of randomly or selectively selecting a portion of data records or data rows from a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sampling, large-scale data can be effectively processed, data processing efficiency can be improved, and the accuracy and reliability of analysis results can be ensured. The specific steps of data sampling are:
[0038] S01, setting the character spacing for data sample extraction, i.e. setting the optimal step;
[0039] S02, traverse the extracted data content and extract one bit (1 byte) every character spacing;
[0040] S03, determining whether the traversal has reached the end of the file; if so, merging the extracted data values to obtain the extraction result.
[0041] In data sampling, data merging is needed to achieve the ultimate goal of simplifying calculations. Merging data values is the process of combining the same data items in two or more data sets into one data set, which can simplify the data analysis process, reduce duplicate data, and obtain a more comprehensive and accurate data set. Common methods for merging data values include merging rows, columns, or using association keys to merge data from different data sources.
[0042] Step 5-14-4, interpolating the data content of the sample extraction result with the all-zero sequence;
[0043] Step 5-14-5, perform hash calculation on the extracted values after interpolation to generate a random hash value for data verification or comparison to ensure the integrity and accuracy of the data.
[0044] Furthermore, the slice repair process in step 5 specifically includes the following steps:
[0045] Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22;
[0046] Step 5-22, start data repair, determine whether the size of the file to be repaired is larger than the slice size; if so, cut the file according to the slice size to obtain a number of slice data and execute step 5-23; otherwise, delete the corresponding file and retransmit the corresponding file from the source host and then execute step 5-25;
[0047] Step 5-23, sequentially compare the hash of a single slice data with the data on the source host to determine whether there is a difference; if so, re-upload all slice data subsequent to the current slice data, merge them into a repair file, and then execute step 5-24; otherwise, select the next slice data set and execute step 5-23;
[0048] Step 5-24, calculate the resource occupancy status of the repair process, and determine whether the resource occupancy is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25;
[0049] Specifically, as a feasible implementation, the set value of resource occupancy is 80%.
[0050] Step 5-25, output the verification and repair results to complete the repair process.
[0051] The present invention adopts the above technical solution and adopts an agent-based migration method. The system can flexibly and dynamically utilize the computing resources of the source data machine and the destination data machine to ensure the efficiency and speed of file migration in different links. The present invention optimizes the resource consumption in the file repair process by designing a special file retrieval method, quickly locates file differences, and improves the stability and efficiency of the overall file migration process. The present invention uses a slicing recheck and a slicing post-merging method to quickly respond to repair operations when an abnormality occurs, achieve efficient file repair, and improve data recovery speed.
[0052] The beneficial effects of the present invention are as follows: (1) Comprehensive analysis of file migration status: In order to ensure the smooth progress of file migration operations, this solution will divide the file migration status in detail, including full migration, incremental migration, and audit repair status. (2) Source-destination host collaborative verification and transmission mechanism: Utilizing the proxy function of the source data machine and the destination data machine, the file verification and repair collaboration between the two is realized, ensuring the accuracy and efficiency of the data transmission process. (3) Implementation of the source-destination host rapid verification function: By adopting the client proxy mode, a detection algorithm is constructed, and its complex algorithm flow includes multiple key steps, which realizes an efficient and fast file data verification function. (4) Implementation of target file rapid repair technology: Accurately locate the data layer differences of each large file, combined with intelligent technologies such as segmented transmission and data reorganization, effectively replace the traditional file retransmission method, and realize fast and efficient repair of the target file. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments;
[0054] Figure 1 A schematic diagram of the platform architecture of a file checking and repairing method based on an agent migration mechanism of the present invention;
[0055] Figure 2 It is a virtualization nesting diagram;
[0056] Figure 3 This is a diagram of the MySQL semi-synchronous replication architecture;
[0057] Figure 4 This is the Redis cluster architecture diagram;
[0058] Figure 5 Deployment architecture diagram for Prometheus;
[0059] Figure 6 Migrate the control relationship diagram for the source and destination hosts
[0060] Figure 7 A schematic diagram of a process of a file checking and repairing method based on an agent migration mechanism of the present invention;
[0061] Figure 8 The real-time difference checking and repairing flow chart of the present invention;
[0062] Fig. 9 A flowchart of the fast file comparison action of the present invention;
[0063] Fig.10 is a flow chart of the best step calculation of the present invention;
[0064] Fig.11 It is a flow chart of the extraction hash calculation of the present invention;
[0065] Fig.12 It is a flow chart of the data sample extraction algorithm of the present invention;
[0066] Fig.13 A logical schematic diagram for data merging of the present invention;
[0067] Fig.14 This is a flow chart of efficient difference repair of the present invention;
[0068] Fig.15 This is a flow chart of the slice repair operation of the present invention;
[0069] Fig.16 It is a flow chart of the dynamic shredded file reorganization mechanism of the present invention;
[0070] Fig.17 This is a flow chart of the file repair compression transmission mechanism of the present invention;
[0071] Fig.18 This is a flow chart of host interaction and concurrent task scheduling of the present invention;
[0072] Fig.19 Schematic diagram of the interaction between hosts and concurrent task scheduling of the present invention. Implementation
[0074] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0075] like Figures 1 to 19 As shown in one, the present invention discloses a file verification and repair method based on a proxy migration mechanism. First, all files need to be traversed in full to ensure that each file is scanned and identified. In this process, the system will record the metadata information of the file, including the file size and path, etc. Then it will enter the full file transfer stage to transfer the file from the source location to the target location. During the transmission process, the system will record the incremental information of the file in real time. After the full transfer is completed, the system will continue to synchronize the incremental data recorded in real time. During the file transfer process, the user can choose to stop the incremental transmission at any time. The host that stops the increment can enter the "pending migration and cutover" state, at which time the user can choose to verify and repair the file transfer. This step can ensure the consistency of the files at the target location with the files at the source location, including confirming that all files have been successfully transferred and performing necessary corrective operations, such as retransmitting lost or damaged files.
[0076] A file checking and repairing method based on an agent migration mechanism comprises the following steps:
[0077] Step 1: The source data machine traverses all files and then transfers all files.
[0078] Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so as to continuously synchronize the incremental data recorded in real time after the full transfer is completed;
[0079] Step 3, determine whether the user chooses to stop incremental transmission during the file transfer process; if so, the host that stops the incremental transmission enters the "waiting for migration and cutover" state; otherwise, execute step 5;
[0080] Step 4, determining whether the user chooses to perform file transfer verification and repair; if yes, executing step 5; otherwise, executing step 6;
[0081] In step 5, the host performs real-time differential information verification through fast file comparison, and performs a slice repair process based on the differential information to retransmit the lost or damaged files; this step can ensure the consistency of the files at the target location with the files at the source location, including confirming that all files have been successfully transferred and performing necessary corrective operations, such as retransmitting lost or damaged files.
[0082] Step 6: Wait for the full file transfer to be completed and synchronize the incremental data recorded in real time to complete the file migration.
[0083] Furthermore, in step 1, metadata information of the file is recorded during the full traversal, and the metadata information includes the file size and path.
[0084] Furthermore, when the host starts to check and repair in step 5, a comparison instruction is issued through the migration client, and the source data machine and the target data machine will work together to perform a quick file comparison and generate a detailed difference list through file comparison; when there are difference values in this file comparison, the corresponding difference list will be repaired.
[0085] Furthermore, step 5 specifically includes the following steps:
[0086] Step 5-1: The host starts to check and repair, and issues a comparison command through the migration client;
[0087] Step 5-2, the source data machine and the target data machine will work together to quickly compare files and generate a detailed difference list;
[0088] Step 5-3, judging whether there is differential information based on the difference list; if yes, performing a repair operation on the corresponding difference list and then executing step 5-4; otherwise, executing step 5-4;
[0089] Step 5-4, manually recheck whether the repair is successful; if yes, complete the repair; otherwise, execute step 5-1.
[0090] Furthermore, the rapid file comparison in step 5 specifically includes the following steps:
[0091] Step 5-11, when the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules;
[0092] Step 5-12, the source end data machine sends a command with a comparison order list to the target end data machine for synchronization operation;
[0093] Step 5-13, based on the performance of the source and target data machines, the optimal data extraction step value is calculated and selected; the purpose is to balance the speed and accuracy of the comparison. The smaller the step value, the denser the information extracted during the comparison process, and the corresponding comparison speed may be reduced; the larger the step value, the faster the comparison, but some subtle differences may be missed.
[0094] Step 5-14, based on the optimal data extraction step value, traverse the files to be compared one by one in the order in the comparison order list, and perform a hash operation on the file contents to convert the file contents into a unique hash value of a fixed length;
[0095] Step 5-15, compare the hash values of the corresponding files of the source data machine and the target data machine; if the hash values are different, it is determined that there are differences in the contents of the corresponding files, and the system records the corresponding difference information.
[0096] Furthermore, the selection of the optimal data extraction step value in step 5-13 specifically includes the following steps:
[0097] Step 5-13-1, initialize the step reference value, set to the total amount of data divided by 1024;
[0098] Step 5-13-2, the current step value is set to 2 / 3 of the base value;
[0099] Step 5-13-3, simulate and generate hash values based on the current step value, and calculate the host CPU and memory usage;
[0100] Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.
[0101] Furthermore, the steps of extracting the hash calculation in step 5-14 are as follows:
[0102] Step 5-14-1, construct an all-zero sequence, that is, set all data to zero value;
[0103] Step 5-14-2, obtaining data content extracted from a source file or database based on an optimal data extraction step value;
[0104] Step 5-14-3, extracting data samples from the extracted data content to obtain sample extraction results;
[0105] Specifically, data sampling is the process of randomly or selectively selecting a portion of data records or data rows from a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sampling, large-scale data can be effectively processed, data processing efficiency can be improved, and the accuracy and reliability of analysis results can be ensured. The specific steps of data sampling are:
[0106] S01, setting the character spacing for data sample extraction, i.e. setting the optimal step;
[0107] S02, traverse the extracted data content and extract one bit (1 byte) every character spacing;
[0108] S03, determining whether the traversal has reached the end of the file; if so, merging the extracted data values to obtain the extraction result.
[0109] In data sampling, data merging is needed to achieve the ultimate goal of simplifying calculations. Merging data values is the process of combining the same data items in two or more data sets into one data set, which can simplify the data analysis process, reduce duplicate data, and obtain a more comprehensive and accurate data set. Common methods for merging data values include merging rows, columns, or using association keys to merge data from different data sources.
[0110] Step 5-14-4, interpolating the data content of the sample extraction result with the all-zero sequence;
[0111] Step 5-14-5, perform hash calculation on the extracted values after interpolation to generate a random hash value for data verification or comparison to ensure the integrity and accuracy of the data.
[0112] Furthermore, the slice repair process in step 5 specifically includes the following steps:
[0113] Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22;
[0114] Step 5-22, start data repair, determine whether the size of the file to be repaired is larger than the slice size; if so, cut the file according to the slice size to obtain a number of slice data and execute step 5-23; otherwise, delete the corresponding file and retransmit the corresponding file from the source host and then execute step 5-25;
[0115] Step 5-23, sequentially compare the hash of a single slice data with the data on the source host to determine whether there is a difference; if so, re-upload all slice data subsequent to the current slice data, merge them into a repair file, and then execute step 5-24; otherwise, select the next slice data set and execute step 5-23;
[0116] Step 5-24, calculate the resource occupancy status of the repair process, and determine whether the resource occupancy is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25;
[0117] Specifically, as a feasible implementation, the set value of resource occupancy is 80%.
[0118] Step 5-25, output the verification and repair results to complete the repair process.
[0119] The specific principle of the present invention is described in detail below:
[0120] (I) Method implementation environment
[0121] (1) The platform architecture design of the implementation environment of the present invention is as follows Figure 1 As shown, it specifically includes the following parts:
[0122] 1. Virtualization nested deployment: Figure 2 As shown, the implementation environment of the present invention is deployed in a Kubernetes containerized management mode, in which the Kubernetes part will adopt a virtualized nested mode, using containerization technology based on the virtualization platform. The platform uses VPC resources with a configuration of more than 16c|64g, builds a Kubernetes environment on 9 hosts as the basic base of the migration control platform, and uses application containerized automatic orchestration to achieve high availability of the system.
[0123] 2. Software cluster deployment: The platform achieves high-concurrency processing of platform instructions through cluster deployment of MySQL and Redis.
[0124] 2.1) MySQL: Master database (1 host): Install MySQL database and configure it as the master database, responsible for data writing, updating and deleting operations. Slave database (1 host): Install MySQL database and configure it as the slave database, responsible for data reading and backup.
[0125] Master-slave replication architecture: Use the master-slave replication function that comes with MySQL to achieve high availability and fault tolerance of the database. One host is used as the master database, responsible for data writing, updating, and deleting operations; the other host is used as the slave database, responsible for data backup and reading operations. When the master database fails, the slave database can be switched to the new master database to ensure service continuity and data integrity. Master-slave replication can be implemented using the built-in function of MySQL. Figure 3 shown.
[0126] MySQL's master-slave replication uses a semi-synchronous replication method. In traditional master-slave replication, the replication of the slave database is asynchronous. After the master database executes a transaction, it immediately returns a response to the client without waiting for the slave database to complete the transaction. This asynchronous replication may cause data delays between the master and slave databases, and cause the slave database to be unable to update in time. After using semi-synchronous replication, when completing a transaction, it waits for at least one slave database to successfully replicate the transaction before returning a response to the client. Although the performance of data operations may be affected to a certain extent, it can effectively solve the problem of master-slave data consistency.
[0127] 2.2) Redis: Use the built-in cluster function of Redis to achieve high availability and fault tolerance of the Redis database. In this Redis cluster, 6 nodes need to be created, 3 of which are master nodes and the other 3 are slave nodes. Each node stores a part of the data and can process the read and write requests of the client. When the master node fails, the cluster will force the slave node to upgrade itself to the new master node to ensure the continuity of service and the integrity of data.
[0128] like Figure 4 As shown, each node has a corresponding master-slave relationship, and the specific examples are as follows: Node 1 (master) port number is 18000. Node 2 (master) port number is 18000. Node 3 (master) port number is 18000. Node 4 (slave) port number is 18001. Node 5 (slave) port number is 18001. Node 6 (slave) port number is 18001.
[0129] 3. Monitoring environment deployment: The platform uses Prometheus combined with Grafana to promptly detect and handle fault problems.
[0130] 3.1) Grafana: Use Grafana to visualize data. Point the data source configured in Grafana to Prometheus to ensure that Grafana can receive all collected data. Create and generate a dashboard to visualize data, such as displaying CPU utilization percentage, host memory usage, etc.
[0131] 3.2) Prometheus: Figure 5 As shown, the node collection client is used to monitor the operating status of the three main functional nodes, and the Redis collection client and the MySQL collection client are used to monitor the status of two types of database clusters respectively, and the above monitoring information is reported to the monitoring platform for processing.
[0132] (2) The source and destination hosts to be migrated: Figure 6 As shown in the figure, a dedicated migration agent is designed for the source and target hosts to be migrated, and it is ensured that the agent only needs a 1-core processor and 1GB of memory to run smoothly. During the migration process, the migration agent will make full use of the resources of the source and target machines for reasonable scheduling.
[0133] (II) Method logic design:
[0134] (1) Overall design of file migration: Figure 7 As shown in the figure, the file migration process includes several key steps, namely, full file data traversal, full file transfer, incremental file recording, incremental file transfer, stop incremental transfer, and start verification and repair. When the system starts migration, it first needs to traverse all files in full to ensure that each file is scanned and identified. During this process, the system will record the metadata information of the file, including file size and path. Then it will enter the full file transfer stage to transfer the file from the source location to the target location. During the transfer process, the system will record the incremental information of the file in real time. After the full transfer is completed, the system will continue to synchronize the incremental data recorded in real time.
[0135] During the file transfer process, users can choose to stop incremental transfer at any time. The host that stops the incremental transfer will enter the "pending migration and cutover" state, at which time users can choose to verify and repair the file transfer. This step can ensure the consistency of the files at the target location with the files at the source location, including confirming that all files have been successfully transferred and taking necessary corrective actions, such as retransmitting lost or damaged files.
[0136] (2) Real-time difference checking and repair: Real-time difference checking and repair refers to the process of real-time monitoring and comparison of system data during system operation, discovering differences between data, and immediately repairing and adjusting them. This process can effectively detect data anomalies and errors, ensure data accuracy and integrity, and improve system stability and reliability.
[0137] like Figure 8 As shown, the process of real-time difference checking and repairing of the present invention is as follows:
[0138] 1. After the host stops incrementing, it enters the "waiting for migration and cutover" state. At this time, the user can selectively perform file transfer verification and repair operations. If the user attempts to use verification and repair, the host will enter the "real-time difference verification and repair" process.
[0139] 2. After the host starts checking and repairing, it issues a comparison command through the agent client. At this time, the source and target data machines will work together to quickly compare files and generate a detailed difference list through file comparison. If this comparison contains a difference value, the corresponding difference list can be repaired.
[0140] (2.1) Fast file comparison: Fast file comparison is a technology used to detect differences between two or more files. It is usually used in backup, version control, and system migration scenarios. Fig. 9 As shown, the following are the detailed steps for quick file comparison:
[0141] Generate a comparison order list: When the fast file comparison operation is started, the system will first generate a comparison order list based on the preset sorting rules (such as file name, size, modification time, etc.). This list determines which files need to be compared first. Then, the system sends this list to the agent on the target machine to process the comparison task in parallel.
[0142] Select the best data extraction step value: Before starting the actual file content comparison, the system will perform a step to determine the most appropriate data extraction step value. This step value is calculated based on the performance of the host, in order to balance the speed and accuracy of the comparison. The smaller the step value, the denser the information extracted during the comparison process, and the corresponding comparison speed may be reduced; the larger the step value, the faster the comparison will be, but some subtle differences may be missed.
[0143] Execute file comparison: Once the step value is determined, the system will enter the specific file comparison stage. In this stage, the system will traverse the files to be compared one by one in the order of the comparison order list and perform hash operations on the file contents. The hash operation can convert the file content into a unique hash value of a fixed length, which can be used to quickly compare whether the file contents are the same.
[0144] Compare hash values: After completing the hash calculation of all files, the system will compare the hash values of the corresponding files on the source host and the target host. If the hash values are different, it means that there are differences in the file content, and the system will record these differences.
[0145] (2.1.1) Optimal data extraction step selection algorithm: Optimal data extraction step refers to the process of selecting data samples through certain step rules during the data sample extraction process. The optimal data extraction step can help select data samples more scientifically and effectively during the data sampling process to ensure that the sampling results are representative and reliable for the entire data set.
[0146] like Fig.10 As shown, the algorithm for selecting the best data extraction step in the present invention will be performed according to the following steps:
[0147] 1. Initialize the stepping reference value and set it to the total amount of data divided by 1024.
[0148] 2. The current step value is set to 2 / 3 of the base value.
[0149] 3. Simulate the generation of hash values and calculate the host CPU and memory usage.
[0150] 4. If both CPU and memory usage are less than or equal to 5% of the estimated allocation, set the step value to 2 / 3 of the current value and recalculate the resource usage.
[0151] 5. If both CPU and memory usage are greater than 5% of the estimated allocation, set the step value to 3 / 2 of the current value and output the optimal step value.
[0152] (2.1.2) Extractive hash calculation: Extractive hash calculation refers to extracting part of the content (usually a fixed-length string or number) from a piece of text or data, and then performing hash calculation on the extracted content to generate a unique hash value. This method can be used to quickly compare the similarity or uniqueness of a large amount of data, and is often used in application scenarios such as data deduplication, similarity matching, and data indexing.
[0153] like Fig.11 As shown, the extraction hash calculation in the present invention will be performed according to the following steps:
[0154] 1. Construct an all-zero sequence, that is, set all data to zero values.
[0155] 2. Perform data extraction to extract data from source files or databases.
[0156] 3. Interpolate the result of the decimation algorithm with the all-zero sequence.
[0157] 4. Perform hash calculation on the extracted value after interpolation.
[0158] 5. Generate random hash values for data verification or comparison to ensure data integrity and accuracy.
[0159] like Fig.12As shown in the figure, data sampling is the process of randomly or selectively selecting a part of data records or data rows from a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sampling, large-scale data can be effectively processed, data processing efficiency can be improved, and the accuracy and reliability of analysis results can be ensured.
[0160] In data sampling, data merging is needed to achieve the ultimate goal of simplifying calculations. Merging data values is the process of merging the same data items in two or more data sets into one data set, which can simplify the data analysis process, reduce duplicate data, and obtain a more comprehensive and accurate data set. Common methods for merging data values include merging rows, columns, or using association keys to merge data from different data sources. Fig.13 As shown, the data sample extraction algorithm logic diagram of the merged data value in the method of the present invention.
[0161] (2.2) Efficient difference repair: Fig.14 As shown, efficient difference repair is based on a slice-based difference repair method. Slice-based difference repair refers to a method of dividing a piece of data into multiple different segments (or slices), and then performing difference repair on these segments. This method is suitable for relatively large data or files. By slicing the data, differences in the data can be found more efficiently and repaired.
[0162] Slice-based file difference repair is a method of repairing damaged or lost files, in which the file is divided into multiple small pieces or slices, each slice containing a portion of the file's data. When a file is damaged or lost, slice-based file difference repair can automatically detect and repair these errors without re-downloading the entire file. The present invention adopts a method of automatically adjusting the slice size, using an intelligent algorithm to dynamically adjust the slice size to meet the performance requirements of different hardware devices and files of different sizes. Fig.15 As shown, the slice repair process specifically includes the following steps:
[0163] Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22;
[0164] Step 5-22, start data repair, determine whether the size of the file to be repaired is larger than the slice size; if so, cut the file according to the slice size to obtain a number of slice data and execute step 5-23; otherwise, delete the corresponding file and retransmit the corresponding file from the source host and then execute step 5-25;
[0165] Step 5-23, sequentially compare the hash of a single slice data with the data on the source host to determine whether there is a difference; if so, re-upload all slice data subsequent to the current slice data, merge them into a repair file, and then execute step 5-24; otherwise, select the next slice data set and execute step 5-23;
[0166] Step 5-24, calculate the resource occupancy status of the repair process, and determine whether the resource occupancy is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25;
[0167] Specifically, as a feasible implementation, the set value of resource occupancy is 80%.
[0168] Step 5-25, output the verification and repair results to complete the repair process.
[0169] (III) Client Function Design: The client of the source and target data machines has the following functions:
[0170] (1) Fragmented file reorganization mechanism: Fragmented file optimization mechanism is a technology used to deal with fragmentation problems in computer file systems. Fragmentation refers to the discontinuous fragments of files scattered on the disk when they are stored, which will slow down the file reading and writing speed and reduce system performance. The fragmented file optimization mechanism reorganizes the file storage location and organizes the file fragments into continuous blocks to improve the file reading and writing speed. Fig.16 As shown, the present invention adopts a dynamic fragmented file reorganization mechanism, and the specific steps are as follows:
[0171] S11, resource evaluation: analyzing the current stage idle resource information to obtain resource evaluation results, the idle resource information includes CPU usage, memory usage and disk I / O capacity;
[0172] S12, determine whether the current stage is idle; if so, reassemble the complete file from the fragments; otherwise, execute S12.
[0173] That is, before the file is transferred, try to reassemble the file segments (slices) into a complete file locally. This can be done by creating a temporary file and writing each file segment (slice) to the file in sequence. If system resources allow, this process can be performed in parallel.
[0174] (2) File repair compression transmission mechanism: The compression transmission mechanism is a technology that reduces the amount of network transmission data by compressing data. During the data transmission process, the data becomes more compact after compression, thereby reducing the bandwidth and transmission time required for transmission. Through the compression transmission mechanism, network resources can be saved, transmission speed can be increased, and the possibility of network congestion can be reduced during the data transmission process. Fig.17 As shown in the figure, the specific process of file repair compression transmission is as follows:
[0175] S21, read the characters and the number of repetitions of the corresponding characters from the compressed file, and determine whether the continuous number is written; if so, write the characters read from the compressed file into the recovery file and execute S22; otherwise, execute S21;
[0176] S22, determine whether the file compression is complete; if so, close the compressed file and end; otherwise, execute S21.
[0177] (3) Host interaction and concurrency: Host interaction and concurrency refer to the way different hosts communicate and interact in a computer network. Concurrency refers to the ability to process multiple tasks or connections at the same time, while host interaction refers to the process of communication and data transmission between hosts. In the file verification and repair method of the present invention, the host uses an interactive and concurrent method to perform comparison and repair operations on each task. Specifically, when performing file verification, the host compares data with the user in an interactive manner to ensure the integrity and correctness of the file. During the repair operation, the host uses concurrent technology to process multiple tasks simultaneously, improving operational efficiency and speed. In this way, the host can perform file verification and repair more effectively, improving the security and stability of the system. At the same time, this interactive and concurrent method also reflects the intelligence and efficiency of the host in processing tasks, providing users with more convenient and reliable services. Fig.18 As shown, the host interaction and concurrent task scheduling process steps are as follows:
[0178] When the interactive and concurrent tasks start, the source machine first sends instructions to the target host to start the interactive task. Then, both the source host and the target host will create a thread pool. Once a task request related to verification or repair is received, the source host will send the task request to the target host, including a list of files to be migrated and related parameters. After receiving the task request, the target host will assign the task to a thread in the thread pool for processing based on factors such as the size and type of the file. After the task is executed, the thread pool will automatically close and record the migration log related to verification and repair. Through this process, the interactive and concurrent tasks are carried out in an orderly manner, ensuring the smooth progress of file verification and repair operations. Inter-host interactive and concurrent task scheduling is as follows: Fig.19 shown.
[0179] The present invention adopts the above technical solution and adopts an agent-based migration method. The system can flexibly and dynamically utilize the computing resources of the source data machine and the destination data machine to ensure the efficiency and speed of file migration in different links. The present invention optimizes the resource consumption in the file repair process by designing a special file retrieval method, quickly locates file differences, and improves the stability and efficiency of the overall file migration process. The present invention uses a slicing recheck and a slicing post-merging method to quickly respond to repair operations when an abnormality occurs, achieve efficient file repair, and improve data recovery speed.
[0180] The beneficial effects of the present invention are as follows: (1) Comprehensive analysis of file migration status: In order to ensure the smooth progress of file migration operations, this solution will divide the file migration status in detail, including full migration, incremental migration, and audit repair status. (2) Source-destination host collaborative verification and transmission mechanism: Utilizing the proxy function of the source data machine and the destination data machine, the file verification and repair collaboration between the two is realized, ensuring the accuracy and efficiency of the data transmission process. (3) Implementation of the source-destination host rapid verification function: By adopting the client proxy mode, a detection algorithm is constructed, and its complex algorithm flow includes multiple key steps, which realizes an efficient and fast file data verification function. (4) Implementation of target file rapid repair technology: Accurately locate the data layer differences of each large file, combined with intelligent technologies such as segmented transmission and data reorganization, effectively replace the traditional file retransmission method, and realize fast and efficient repair of the target file.
[0181] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.
Claims
1. A file checking and repairing method based on an agent migration mechanism, characterized in that: It includes the following steps: Step 1: The source data machine traverses all files and then transfers all files. Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so as to continuously synchronize the incremental data recorded in real time after the full transfer is completed; Step 3: determine whether the user chooses to stop incremental transmission during the file transfer process; if so, the host that stops the incremental transmission enters the "waiting for migration and cutover" state; Otherwise, go to step 5; Step 4, determining whether the user chooses to perform file transfer verification and repair; if yes, proceed to step 5; Otherwise, go to step 6; Step 5: The host performs real-time differential information verification through fast file comparison, and performs a slice repair process based on the differential information to retransmit the lost or damaged files; Step 6: Wait for the full file transfer to be completed and synchronize the incremental data recorded in real time to complete the file migration.
2. The file checking and repairing method based on the proxy migration mechanism according to claim 1 is characterized by: In step 1, the metadata information of the file is recorded during the full traversal, and the metadata information includes the file size and path.
3. The file checking and repairing method based on the proxy migration mechanism according to claim 1 is characterized by: When the host starts to check and repair in step 5, a comparison instruction is issued through the migration client, and the source data machine and the target data machine will work together to perform a quick file comparison and generate a detailed difference list through file comparison; when there are difference values in this file comparison, the corresponding difference list will be repaired.
4. The file checking and repairing method based on the proxy migration mechanism according to claim 1 is characterized by: Step 5 specifically includes the following steps: Step 5-1: The host starts to check and repair, and issues a comparison command through the migration client; Step 5-2, the source data machine and the target data machine will work together to quickly compare files and generate a detailed difference list; Step 5-3, judging whether there is differential information based on the difference list; if yes, performing a repair operation on the corresponding difference list and then executing step 5-4; otherwise, executing step 5-4; Step 5-4, manually recheck whether the repair is successful; if yes, complete the repair; otherwise, execute step 5-1.
5. The file checking and repairing method based on the proxy migration mechanism according to claim 1 is characterized by: The rapid file comparison in step 5 specifically includes the following steps: Step 5-11, when the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules; Step 5-12, the source end data machine sends a command with a comparison order list to the target end data machine for synchronization operation; Step 5-13, calculating and selecting the best data extraction step value based on the performance of the source end data machine and the target end data machine; Step 5-14, based on the optimal data extraction step value, traverse the files to be compared one by one in the order in the comparison order list, and perform a hash operation on the file contents to convert the file contents into a unique hash value of a fixed length; Step 5-15, compare the hash values of the corresponding files of the source data machine and the target data machine; if the hash values are different, it is determined that there are differences in the contents of the corresponding files, and the system records the corresponding difference information.
6. The file checking and repairing method based on the proxy migration mechanism according to claim 5 is characterized by: The selection of the optimal data extraction step value in step 5-13 specifically includes the following steps: Step 5-13-1, initialize the step reference value, set to the total amount of data divided by 1024; Step 5-13-2, the current step value is set to 2 / 3 of the base value; Step 5-13-3, simulate and generate hash values based on the current step value, and calculate the host CPU and memory usage; Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.
7. The file checking and repairing method based on the proxy migration mechanism according to claim 5 is characterized by: The steps to extract the hash calculation in steps 5-14 are as follows: Step 5-14-1, construct an all-zero sequence, that is, set all data to zero value; Step 5-14-2, obtaining data content extracted from a source file or database based on an optimal data extraction step value; Step 5-14-3, extracting data samples from the extracted data content to obtain sample extraction results; Step 5-14-4, interpolating the data content of the sample extraction result with the all-zero sequence; Step 5-14-5, perform hash calculation on the extracted values after interpolation to generate a random hash value for data verification or comparison to ensure the integrity and accuracy of the data.
8. The file checking and repairing method based on the proxy migration mechanism according to claim 7 is characterized by: The specific steps for data sample extraction in step 5-14-3 are: S01, setting the character spacing for data sample extraction, i.e. setting the optimal step; S02, traverse the extracted data content and extract one bit (1 byte) every character spacing; S03, after traversing to the end of the file, merge the extracted data values to obtain the extraction result.
9. The file checking and repairing method based on the proxy migration mechanism according to claim 1 is characterized by: The slice repair process in step 5 specifically includes the following steps: Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22; Step 5-22, start data repair, determine whether the size of the file to be repaired is larger than the slice size; if so, cut the file according to the slice size to obtain a number of slice data and execute step 5-23; otherwise, delete the corresponding file and retransmit the corresponding file from the source host and then execute step 5-25; Step 5-23, sequentially compare the hash of a single slice data with the data on the source host to determine whether there is a difference; if so, re-upload all slice data subsequent to the current slice data, merge them into a repair file, and then execute step 5-24; otherwise, select the next slice data set and execute step 5-23; Step 5-24, calculate the resource occupancy status of the repair process, and determine whether the resource occupancy is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25; Step 5-25, output the verification and repair results to complete the repair process.
Citation Information
Patent Citations
Database data migration method, device and equipment
CN109933632A
Database migration method and device, storage medium and electronic equipment
CN114546992A
Version upgrading method and device for database
CN114969054A
Data migration method and device, electronic equipment and storage medium
CN117312234A
