A file verification and repair method based on agent migration mechanism

Abstract

The application discloses a file checking and repairing method based on an agent migration mechanism, and comprises the following steps: step 1, full file transmission is performed after full traversal of all files by a source data machine; step 2, files are transmitted from a source position to a target position, and incremental information of the files is recorded in real time during the transmission process; step 3, when a user selects to stop incremental transmission during the file transmission process, the host machine that stops the incremental transmission enters a 'to-be-migrated cut-off' state; step 4, it is judged whether the user selects to perform file transmission checking and repairing; if yes, step 5 is executed; otherwise, step 6 is executed; step 5, the host machine performs real-time differential information checking through fast file comparison, and performs a slice repairing process based on the differential information, so that lost or damaged files are retransmitted; and step 6, file migration is completed after waiting for full file transmission to end and synchronizing incremental data recorded in real time. Through the slice type rechecking and slice merging method, the application can quickly respond and carry out repairing operation when an exception occurs, realizes efficient file repairing, and improves the data recovery speed.

Description

Technical Field

[0001] The present invention relates to the technical field of file migration, and in particular to a file verification and repair method based on an agent migration mechanism. Background Art

[0002] In today's technological landscape, with the rapid development of technologies like cloud computing and big data, and the growing demand for data volumes and storage, the widespread application of file migration technology is driving digital transformation and IT innovation, becoming a key driver of innovation across various industries. Enterprises may need to migrate large amounts of file data from local storage to cloud platforms or new data centers, making the need for file migration increasingly urgent.

[0003] File migration is the process of transferring files from one location to another. During this process, factors such as data integrity, migration speed, and resource usage must also be considered. This makes "file data verification" and "file data repair" crucial steps in ensuring data quality.

[0004] File data verification verifies data accuracy, completeness, and consistency to avoid errors, loss, or inconsistencies. File data repair is a key measure to prevent data loss and corruption, helping to maintain data integrity and reliability. In actual production environments, users can use file data verification to promptly identify and resolve potential issues, ensuring post-migration data quality. File data repair can then ensure data accuracy and integrity during the migration process, reducing the risk of data loss and corruption.

[0005] Traditional file copying methods can only simply transfer data and cannot verify and repair data in real time, resulting in a time-consuming and labor-intensive migration process. Therefore, modern file migration tools are usually used. Although some migration tools support verification and repair functions, there are still some problems that need to be solved. (1) First, the retrieval efficiency is too low: Although some tools provide verification and repair functions, they still have room for improvement in data retrieval efficiency. Due to the large amount of data or high complexity, traditional tools are often unable to achieve efficient data verification and rapid error location, resulting in less than ideal execution efficiency. (2) Secondly, collaborative computing is difficult: Existing file migration tools have limitations in utilizing host computing power. Generally, traditional tools can only utilize host computing power in one direction, and it is difficult to perform collaborative computing through the source data machine and the destination data machine, resulting in low resource utilization when the tool is performing verification and repair, limiting the efficiency and speed of the file migration process. (3) In addition, excessive resource consumption: The file verification and repair process often occupies a large amount of resources, including computing resources, storage resources, etc., resulting in large resource consumption, affecting the stability and efficiency of the overall file migration process. Summary of the Invention

[0006] The purpose of the present invention is to provide a file verification and repair method based on a proxy migration mechanism, aiming to overcome the challenges of current file migration technology such as low retrieval efficiency, insufficient host computing power utilization and excessive resource consumption, and to achieve an efficient and stable file migration process.

[0007] The technical solution adopted in the present invention is:

[0008] A file verification and repair method based on an agent migration mechanism comprises the following steps:

[0009] Step 1: The source data machine traverses all files and then transfers all files.

[0010] Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so as to continuously synchronize the real-time recorded incremental data after the full transfer is completed;

[0011] Step 3: Determine whether the user chooses to stop incremental transfer during the file transfer process; if so, the host that stops incremental transfer enters the "waiting for migration and cutover" state; otherwise, proceed to step 5;

[0012] Step 4: Determine whether the user chooses to perform file transfer verification and repair; if yes, proceed to step 5; otherwise, proceed to step 6;

[0013] In step 5, the host performs real-time differential information verification through fast file comparison and performs a slice repair process based on the differential information to retransmit lost or damaged files. This step ensures the consistency of files at the target location with those at the source location, including confirming that all files have been successfully transferred and performing necessary corrective actions, such as retransmitting lost or damaged files.

[0014] Step 6: Wait for the full file transfer to complete and synchronize the incremental data recorded in real time to complete the file migration.

[0015] Furthermore, in step 1, metadata information of the file is recorded during the full traversal, and the metadata information includes the file size and path.

[0016] Furthermore, when the host starts to check and repair in step 5, a comparison instruction is issued through the migration client, and the source data machine and the target data machine will work together to perform a rapid file comparison and generate a detailed difference list through the file comparison; when there are difference values ​​in this file comparison, the corresponding difference list will be repaired.

[0017] Furthermore, step 5 specifically includes the following steps:

[0018] Step 5-1: The host starts checking and repairing, and issues comparison instructions through the migration client;

[0019] In step 5-2, the source and target data machines will work together to quickly compare files and generate a detailed list of differences.

[0020] Step 5-3: Determine whether there is differential information based on the difference list; if so, perform a repair operation on the corresponding difference list and then execute step 5-4; otherwise, execute step 5-4;

[0021] In step 5-4, manually recheck whether the repair is successful; if so, complete the repair; otherwise, proceed to step 5-1.

[0022] Furthermore, the rapid file comparison in step 5 specifically includes the following steps:

[0023] Step 5-11: When the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules;

[0024] Step 5-12, the source modem sends a command with a comparison order list to the target modem for synchronization operation;

[0025] In steps 5-13, the optimal data extraction step size is calculated and selected based on the performance of the source and target data machines. This is done to balance comparison speed and accuracy. A smaller step size extracts more information during the comparison, potentially reducing comparison speed. A larger step size increases comparison speed but may miss subtle differences.

[0026] Step 5-14: Based on the optimal data extraction step value, the files to be compared are traversed one by one in the order in the comparison order list, and the file contents are hashed to convert the file contents into a unique hash value of a fixed length;

[0027] Step 5-15, compare the hash values ​​of the corresponding files of the source data machine and the target data machine; if the hash values ​​are different, it is determined that there is a difference in the corresponding file content, and the system records the corresponding difference information.

[0028] Furthermore, the selection of the optimal data extraction step value in steps 5-13 specifically includes the following steps:

[0029] Step 5-13-1, initialize the step reference value and set it to the total amount of data divided by 1024;

[0030] Step 5-13-2, the current step value is set to 2 / 3 of the base value;

[0031] Step 5-13-3, simulate and generate hash values ​​based on the current step value, and calculate the host CPU and memory usage;

[0032] Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.

[0033] Further, the step of extracting the hash calculation in step 5-14 is as follows:

[0034] Step 5-14-1, construct a sequence of all zeros, i.e. set all data to zero value;

[0035] Step 5-14-2, obtain the data content extracted from the source file or database based on the optimal data extraction step value;

[0036] Step 5-14-3, perform data sample extraction on the extracted data content to obtain a sample extraction result;

[0037] Specifically, data sample extraction is a process of randomly or selectively selecting a portion of data records or data rows in a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sample extraction, large-scale data can be effectively processed, improving data processing efficiency while ensuring the accuracy and reliability of the analysis results. The specific steps of data sample extraction are as follows:

[0038] S01, set the character interval of data sample extraction, i.e. set the optimal step;

[0039] S02, traverse the extracted data content, and extract a bit (1 byte) every character interval;

[0040] S03, determine whether the end of the file has been reached; if so, merge the extracted data values to obtain the extraction result.

[0041] In data sample extraction, data needs to be merged to achieve the purpose of final simplified calculation. Merging data is a process of merging the same data items in two or more data sets into one data set, which can simplify the data analysis process, reduce redundant data, and obtain a more comprehensive and accurate data set. Common methods of merging data values include merging rows, columns, or using association keys to merge data from different data sources.

[0042] Step 5-14-4, interpolate the data content of the sample extraction result with the all-zero sequence;

[0043] Step 5-14-5, perform hash calculation on the interpolated extraction value to generate a hash value for data verification or comparison to ensure data integrity and accuracy.

[0044] Further, the slice repair process in step 5 specifically includes the following steps:

[0045] Step 5-21, obtain the differentiated information and determine whether the slice size has been allocated; if yes, execute step 5-22; otherwise, initialize the slice size and execute step 5-22;

[0046] Step 5-22, start data repair, determine whether the size of the file to be repaired is greater than the slice size; if yes, cut the file into a plurality of slice data in the slice size and execute step 5-23; otherwise, delete the corresponding file and retransmit the corresponding file from the source host and execute step 5-25;

[0047] Step 5-23, sequentially compare a single slice data with the data on the source host by hash, and determine whether there is a difference; if yes, reupload all slice data subsequent to the current slice data, and combine them into a repaired file and execute step 5-24; otherwise, select the next slice data set and execute step 5-23;

[0048] Step 5-24, calculate the resource occupation state of the repair process, and determine whether the resource occupation is less than a set value; if yes, re-calculate the allocated slice size and update the value of the slice size, and execute step 5-25; otherwise, execute step 5-25;

[0049] Specifically, as a feasible implementation manner, the set value of the resource occupation is 80%.

[0050] Step 5-25, output the repair result for checking to complete the repair process.

[0051] The present application adopts the above technical solution, adopts the proxy-based migration mode, and the system can dynamically utilize the computing resource of the source data machine and the destination data machine, and ensure the efficiency and speed of file migration at different links. The present application optimizes the resource consumption in the file repair process by designing a special file retrieval method, quickly locates the file difference, and improves the stability and efficiency of the overall file migration process. The present application quickly reacts to carry out repair operation when an exception occurs through the slice re-verification and slice merging method, realizes efficient file repair, and improves the data recovery speed.

[0052] The beneficial effects of the present invention are as follows: (1) Comprehensive analysis of file migration status: In order to ensure the smooth progress of file migration operations, this solution will divide the file migration status in detail, including full migration, incremental migration, and audit repair status. (2) Source-destination host collaborative verification and transmission mechanism: Utilizing the proxy function of the source-end data machine and the destination-end data machine, the file verification and repair collaboration between the two is realized, ensuring the accuracy and efficiency of the data transmission process. (3) Implementation of the source-destination host rapid verification function: By adopting the client-side proxy mode, a detection algorithm is constructed, and its complex algorithm process includes multiple key steps, which realizes an efficient and fast file data verification function. (4) Implementation of target file rapid repair technology: Accurately locate the data layer differences of each large file, combine intelligent technologies such as fragmented transmission and data reorganization, effectively replace the traditional file retransmission method, and realize fast and efficient repair of the target file. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0054] Figure 1 A schematic diagram of the platform architecture of a file verification and repair method based on an agent migration mechanism of the present invention;

[0055] Figure 2 This is a schematic diagram of virtualization nesting;

[0056] Figure 3 This is a diagram of the MySQL semi-synchronous replication architecture;

[0057] Figure 4 This is the Redis cluster architecture diagram;

[0058] Figure 5 Deployment architecture diagram for Prometheus;

[0059] Figure 6 Migrate the control relationship diagram for the source and destination hosts

[0060] Figure 7 A flowchart of a file checking and repairing method based on an agent migration mechanism according to the present invention;

[0061] Figure 8 This is a flowchart of the real-time difference checking and repairing of the present invention;

[0062] Figure 9 A flowchart of the fast file comparison action of the present invention;

[0063] Figure 10 A flow chart of the optimal step calculation of the present invention;

[0064] Figure 11 This is a flowchart of the hash calculation process of the present invention;

[0065] Figure 12 Schematic diagram of the flow of the data sample extraction algorithm of the present invention;

[0066] Figure 13 A logical diagram of the data merging of the present invention;

[0067] Figure 14 This is a flowchart of the efficient difference repair process of the present invention;

[0068] Figure 15 This is a flow chart of the slice repair operation of the present invention;

[0069] Figure 16 This is a flow chart of the dynamic shredded file reorganization mechanism of the present invention;

[0070] Figure 17 This is a flow chart of the file repair compression transmission mechanism of the present invention;

[0071] Figure 18 This is a flow chart of host interaction and concurrent task scheduling in the present invention;

[0072] Figure 19 Schematic diagram of the host-to-host interaction and concurrent task scheduling of the present invention. Implementation Method

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0074] like Figures 1 to 19 As shown in one of them, the present invention discloses a file verification and repair method based on the proxy migration mechanism, which first requires a full traversal of all files to ensure that each file is scanned and identified. During this process, the system will record the metadata information of the file, including the file size and path, etc. It will then enter the full file transfer phase to transfer the file from the source location to the target location. During the transmission process, the system will record the incremental information of the file in real time. After the full transfer is completed, the system will continue to synchronize the incremental data recorded in real time. During the file transfer process, the user can choose to stop the incremental transfer at any time. The host that stops the increment will enter the "pending migration and cutover" state, at which time the user can choose to verify and repair the file transfer. This step can ensure the consistency of the files at the target location with the files at the source location, including confirming that all files have been successfully transferred and performing necessary corrective operations, such as retransmitting lost or damaged files.

[0075] A file verification and repair method based on an agent migration mechanism comprises the following steps:

[0076] Step 1: The source data machine traverses all files and then transfers all files.

[0077] Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so as to continuously synchronize the real-time recorded incremental data after the full transfer is completed;

[0078] Step 3: Determine whether the user chooses to stop incremental transfer during the file transfer process; if so, the host that stops incremental transfer enters the "waiting for migration and cutover" state; otherwise, proceed to step 5;

[0079] Step 4: Determine whether the user chooses to perform file transfer verification and repair; if yes, proceed to step 5; otherwise, proceed to step 6;

[0080] In step 5, the host performs real-time differential information verification through fast file comparison and performs a slice repair process based on the differential information to retransmit lost or damaged files. This step ensures the consistency of files at the target location with those at the source location, including confirming that all files have been successfully transferred and performing necessary corrective actions, such as retransmitting lost or damaged files.

[0081] Step 6: Wait for the full file transfer to complete and synchronize the incremental data recorded in real time to complete the file migration.

[0082] Furthermore, in step 1, metadata information of the file is recorded during the full traversal, and the metadata information includes the file size and path.

[0083] Furthermore, when the host starts to check and repair in step 5, a comparison instruction is issued through the migration client, and the source data machine and the target data machine will work together to perform a rapid file comparison and generate a detailed difference list through the file comparison; when there are difference values ​​in this file comparison, the corresponding difference list will be repaired.

[0084] Furthermore, step 5 specifically includes the following steps:

[0085] Step 5-1: The host starts checking and repairing, and issues comparison instructions through the migration client;

[0086] In step 5-2, the source and target data machines will work together to quickly compare files and generate a detailed list of differences.

[0087] Step 5-3: Determine whether there is differential information based on the difference list; if so, perform a repair operation on the corresponding difference list and then execute step 5-4; otherwise, execute step 5-4;

[0088] In step 5-4, manually recheck whether the repair is successful; if so, complete the repair; otherwise, proceed to step 5-1.

[0089] Furthermore, the rapid file comparison in step 5 specifically includes the following steps:

[0090] Step 5-11: When the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules;

[0091] Step 5-12, the source modem sends a command with a comparison order list to the target modem for synchronization operation;

[0092] In steps 5-13, the optimal data extraction step size is calculated and selected based on the performance of the source and target data machines. This is done to balance comparison speed and accuracy. A smaller step size extracts more information during the comparison, potentially reducing comparison speed. A larger step size increases comparison speed but may miss subtle differences.

[0093] Step 5-14: Based on the optimal data extraction step value, the files to be compared are traversed one by one in the order in the comparison order list, and the file contents are hashed to convert the file contents into a unique hash value of a fixed length;

[0094] Step 5-15, compare the hash values ​​of the corresponding files of the source data machine and the target data machine; if the hash values ​​are different, it is determined that there is a difference in the corresponding file content, and the system records the corresponding difference information.

[0095] Furthermore, the selection of the optimal data extraction step value in steps 5-13 specifically includes the following steps:

[0096] Step 5-13-1, initialize the step reference value and set it to the total amount of data divided by 1024;

[0097] Step 5-13-2, the current step value is set to 2 / 3 of the base value;

[0098] Step 5-13-3, simulate and generate hash values ​​based on the current step value, and calculate the host CPU and memory usage;

[0099] Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.

[0100] Furthermore, the steps for extracting the hash calculation in steps 5-14 are as follows:

[0101] Step 5-14-1, construct an all-zero sequence, that is, set all data to zero value;

[0102] Step 5-14-2, obtaining data content extracted from the source file or database based on the optimal data extraction step value;

[0103] Step 5-14-3, performing data sample extraction on the extracted data content to obtain sample extraction results;

[0104] Specifically, data sample extraction is a process of randomly or selectively selecting a portion of data records or data rows in a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sample extraction, large-scale data can be effectively processed, improving data processing efficiency while ensuring the accuracy and reliability of the analysis results. The specific steps of data sample extraction are as follows:

[0105] S01, setting the character interval of data sample extraction, i.e., setting the optimal step;

[0106] S02, traversing the extracted data content, extracting a bit (1 byte) every character interval;

[0107] S03, determining whether the end of the file has been reached; if so, merging the extracted data values to obtain the extraction results.

[0108] In data sample extraction, merging data is needed to achieve the purpose of final simplified calculation. Merging data is a process of merging the same data items in two or more data sets into one data set, which can simplify the data analysis process, reduce redundant data, and obtain a more comprehensive and accurate data set. Common methods of merging data values include merging rows, columns, or using association keys to merge data from different data sources.

[0109] Step 5-14-4, interpolating the data content of the sample extraction results with the all-zero sequence;

[0110] Step 5-14-5, performing hash calculation on the interpolated extraction values to generate a hash value for data verification or comparison to ensure data integrity and accuracy.

[0111] Further, the slice repair process in step 5 includes the following steps:

[0112] Step 5-21, obtaining differential information and determining whether the slice size has been allocated; if so, executing step 5-22; otherwise, initializing the slice size and executing step 5-22;

[0113] Step 5-22, starting data repair, determining whether the file size to be repaired is greater than the slice size; if so, cutting the file into several slice data with the slice size and executing step 5-23; otherwise, deleting the corresponding file and retransmitting the corresponding file from the source host before executing step 5-25;

[0114] In step 5-23, hash and compare the individual slice data with the data on the source server to determine if there are any differences. If so, re-upload all subsequent slice data after the current slice data and merge them into a repaired file before executing step 5-24. Otherwise, select the next slice data set and execute step 5-23.

[0115] Step 5-24, calculate the resource usage status of the repair process and determine whether the resource usage is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25;

[0116] Specifically, as a feasible implementation, the set value of resource occupancy is 80%.

[0117] Step 5-25: Output the verification and repair results to complete the repair process.

[0118] The specific principle of the present invention is described in detail below:

[0119] (1) Method implementation environment

[0120] (1) The platform architecture design of the implementation environment of the present invention is as follows Figure 1 As shown, it specifically includes the following parts:

[0121] 1. Virtualization nested deployment: such as Figure 2 As shown, the implementation environment of this invention is deployed using Kubernetes containerized management. The Kubernetes component uses a nested virtualization approach, leveraging containerization technology within a virtualization platform. The platform utilizes VPC resources configured with greater than 16c / 64g capacity, building a Kubernetes environment on nine hosts as the foundation for the migration control platform. High system availability is achieved through automated orchestration of application containerization.

[0122] 2. Software cluster deployment: The platform achieves high-concurrency processing of platform commands through cluster deployment of MySQL and Redis.

[0123] 2.1) MySQL: Master (1 host): Install the MySQL database and configure it as the master, responsible for data writing, updating, and deleting operations. Slave (1 host): Install the MySQL database and configure it as the slave, responsible for data reading and backup.

[0124] Master-slave replication architecture: Use the master-slave replication function that comes with MySQL to achieve high availability and fault tolerance of the database. One host acts as the master database, responsible for data writing, updating, and deleting operations; the other host acts as the slave database, responsible for data backup and reading operations. When the master database fails, the slave database can be switched to the new master database to ensure service continuity and data integrity. Master-slave replication can be implemented using the functions that come with MySQL. Figure 3 shown.

[0125] MySQL's master-slave replication uses a semisynchronous replication method. In traditional master-slave replication, replication between slaves is asynchronous. After the master executes a transaction, it immediately returns a response to the client without waiting for the slave to complete the transaction. This asynchronous replication can cause data delays between the master and slave databases, preventing slaves from being updated in a timely manner. With semisynchronous replication, a transaction completes only after waiting for at least one slave to successfully replicate the transaction before returning a response to the client. While data operation performance may be affected, it effectively resolves the issue of master-slave data consistency.

[0126] 2.2) Redis: Use Redis's built-in clustering functionality to achieve high availability and fault tolerance for the Redis database. In this Redis cluster, create six nodes: three master nodes and three slave nodes. Each node stores a portion of the data and handles client read and write requests. If a master node fails, the cluster forces a slave node to become the new master, ensuring service continuity and data integrity.

[0127] like Figure 4 As shown, each node has a corresponding master-slave relationship. The specific examples are as follows: Node 1 (master) port number is 18000. Node 2 (master) port number is 18000. Node 3 (master) port number is 18000. Node 4 (slave) port number is 18001. Node 5 (slave) port number is 18001. Node 6 (slave) port number is 18001.

[0128] 3. Monitoring environment deployment: The platform uses Prometheus combined with Grafana to promptly detect and handle faults.

[0129] 3.1) Grafana: Use Grafana to visualize data. Configure the data source in Grafana to point to Prometheus to ensure that Grafana can receive all collected data. Create a dashboard to visualize the data, such as displaying CPU utilization percentage and host memory usage.

[0130] 3.2) Prometheus: Figure 5 As shown, the node collection client is used to monitor the operating status of the three main functional nodes, and the Redis collection client and MySQL collection client are used to monitor the status of two types of database clusters respectively, and the above monitoring information is reported to the monitoring platform for processing.

[0131] (2) The source and destination hosts to be migrated: Figure 6 As shown, a dedicated migration agent is designed for the source and target hosts to be migrated, ensuring that the agent only requires a single-core processor and 1GB of memory to run smoothly. During the migration process, the migration agent fully utilizes the resources of the source and target hosts for optimal scheduling.

[0132] (2) Method logic design:

[0133] (1) Overall design of file migration: Figure 7 As shown in the figure, the file migration process includes several key steps: full file data traversal, full file transfer, incremental file recording, incremental file transfer, stopping incremental transfer, and starting verification and repair. When the system begins migration, it first needs to traverse all files in full to ensure that each file is scanned and identified. During this process, the system records the file's metadata information, including file size and path. Then, it enters the full file transfer phase, transferring the file from the source location to the target location. During the transfer, the system will record the file's incremental information in real time. After the full transfer is complete, the system will continue to synchronize the real-time recorded incremental data.

[0134] During the file transfer process, users can choose to stop the incremental transfer at any time. The host where the incremental transfer is stopped enters the "Pending Migration and Cutover" state, at which point users can choose to perform a file transfer verification and repair. This step ensures file consistency at the target location and at the source location, including confirming that all files have been successfully transferred and taking any necessary corrective actions, such as retransmitting missing or damaged files.

[0135] (2) Real-time discrepancy checking and repair: Real-time discrepancy checking and repair refers to the process of real-time monitoring and comparison of system data during system operation, identifying discrepancies between data, and immediately repairing and adjusting them. This process can effectively detect data anomalies and errors, ensure data accuracy and integrity, and improve system stability and reliability.

[0136] like Figure 8 As shown, the process of real-time difference verification and repair of the present invention is as follows:

[0137] 1. After the host stops incremental migration, it enters the "Waiting for Migration and Cutover" state. At this time, users can selectively perform file transfer verification and repair operations. If the user attempts to use verification and repair, the host will enter the "Real-time Difference Verification and Repair" process.

[0138] 2. After the host begins verification and repair, it issues a comparison command through the agent client. The source and target data machines will then collaborate to quickly compare files, generating a detailed list of differences. If this comparison contains any differences, repair operations can be performed on the corresponding differences.

[0139] (2.1) Fast file comparison: Fast file comparison is a technology used to detect differences between two or more files. It is usually used in backup, version control, and system migration scenarios. Figure 9 As shown, the following are the detailed steps for quick file comparison:

[0140] Generating a comparison order list: When a fast file comparison operation is initiated, the system first generates a comparison order list based on pre-defined sorting rules (such as file name, size, and modification time). This list determines which files are prioritized for comparison. The system then sends this list to the agent on the target machine to facilitate parallel processing of the comparison tasks.

[0141] Selecting the optimal data extraction step size: Before actually comparing file content, the system performs a step to determine the most appropriate data extraction step size. This step size is calculated based on the host's performance, aiming to balance comparison speed and accuracy. A smaller step size extracts more information during the comparison process, potentially reducing comparison speed. A larger step size increases comparison speed but may miss subtle differences.

[0142] Performing File Comparison: Once the step value is determined, the system enters the specific file comparison phase. During this phase, the system will iterate through the files to be compared, one by one, in the order specified in the comparison order list, and perform a hash operation on the file contents. This hash operation converts the file contents into a unique, fixed-length hash value, which can be used to quickly verify file content identity.

[0143] Compare hash values: After hashing all files, the system compares the hash values ​​of the corresponding files on the source and target hosts. If the hash values ​​differ, this indicates differences in the file contents, and the system records these differences.

[0144] (2.1.1) Optimal Data Extraction Step Selection Algorithm: Optimal data extraction step selection refers to the process of selecting data samples using specific step rules during the data sampling process. Optimal data extraction step selection can help select data samples more scientifically and efficiently during the data sampling process, ensuring that the sampling results are representative and reliable for the entire dataset.

[0145] like Figure 10 As shown, the algorithm for optimal data extraction step selection in the present invention will be performed according to the following steps:

[0146] 1. Initialize the step reference value and set it to the total amount of data divided by 1024.

[0147] 2. The current step value is set to 2 / 3 of the base value.

[0148] 3. Simulate the generation of hash values ​​and calculate the host CPU and memory usage.

[0149] 4. If both CPU and memory usage are less than or equal to 5% of the projected allocation, set the step value to 2 / 3 of the current value and recalculate the resource usage.

[0150] 5. If both CPU and memory usage are greater than 5% of the estimated allocation, set the step value to 3 / 2 of the current value and output the optimal step value.

[0151] (2.1.2) Extractive Hash Calculation: Extractive hash calculation involves extracting a portion of content (usually a fixed-length string or number) from a piece of text or data, then performing a hash calculation on the extracted content to generate a unique hash value. This method can be used to quickly compare the similarity or uniqueness of large amounts of data and is commonly used in applications such as data deduplication, similarity matching, and data indexing.

[0152] like Figure 11 As shown, the extraction hash calculation in the present invention will be performed according to the following steps:

[0153] 1. Construct an all-zero sequence, that is, set all data to zero values.

[0154] 2. Perform data extraction to extract data from source files or databases.

[0155] 3. Interpolate the result of the decimation algorithm with the all-zero sequence.

[0156] 4. Perform hash calculation on the extracted value after interpolation.

[0157] 5. Generate random inspection hash values ​​for data verification or comparison to ensure data integrity and accuracy.

[0158] like Figure 12As shown in Figure 2, data sampling is the process of randomly or selectively selecting a subset of data records or rows from a computer file to represent the entire file. This helps analysts quickly understand the characteristics of a dataset. Data sampling can effectively process large amounts of data, improve data processing efficiency, and ensure the accuracy and reliability of analytical results.

[0159] In data sampling, data merging is required to achieve the ultimate goal of simplifying calculations. Merging data values ​​is the process of combining the same data items in two or more data sets into one data set. This can simplify the data analysis process, reduce duplicate data, and obtain a more comprehensive and accurate data set. Common methods for merging data values ​​include merging rows, columns, or using association keys to merge data from different data sources. Figure 13 As shown, the data sample extraction algorithm logic diagram of the merged data value in the method of the present invention.

[0160] (2.2) Efficient difference repair: Figure 14 As shown, efficient difference repair is based on a slice-based difference repair method. Sliced ​​difference repair involves splitting a piece of data into multiple segments (or slices) and then performing difference repair on these segments. This method is suitable for relatively large data or files. By slicing the data, differences can be more efficiently identified and repaired.

[0161] Sliced ​​file differential repair is a method for repairing damaged or lost files, in which the file is divided into multiple small pieces or slices, each slice containing a portion of the file's data. When a file is damaged or lost, sliced ​​file differential repair can automatically detect and repair these errors without having to re-download the entire file. The present invention adopts a method for automatically adjusting the slice size, using an intelligent algorithm to dynamically adjust the slice size to meet the performance requirements of different hardware devices and files of different sizes. Figure 15 As shown in FIG, the slice repair process specifically includes the following steps:

[0162] Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22;

[0163] Step 5-22: Start data repair and determine whether the file size to be repaired is larger than the slice size. If so, cut the file into slices according to the slice size to obtain a number of slice data and execute step 5-23. Otherwise, delete the corresponding file and retransmit the corresponding file from the source server and execute step 5-25.

[0164] In step 5-23, hash and compare the individual slice data with the data on the source server to determine if there are any differences. If so, re-upload all subsequent slice data after the current slice data and merge them into a repaired file before executing step 5-24. Otherwise, select the next slice data set and execute step 5-23.

[0165] Step 5-24, calculate the resource usage status of the repair process and determine whether the resource usage is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25;

[0166] Specifically, as a feasible implementation, the set value of resource occupancy is 80%.

[0167] Step 5-25: Output the verification and repair results to complete the repair process.

[0168] (3) Client Function Design: The client of the source and target modems has the following functions:

[0169] (1) Fragmented file reorganization mechanism: Fragmented file optimization mechanism is a technology used to deal with fragmentation problems in computer file systems. Fragmentation refers to the discontinuous fragments of files scattered on the disk when they are stored, which will cause the file reading and writing speed to slow down and reduce system performance. Fragmented file optimization mechanism reorganizes the file storage location and organizes the file fragments into continuous blocks to improve the file reading and writing speed. Figure 16 As shown, the present invention adopts a dynamic fragmented file reorganization mechanism, and the specific steps are as follows:

[0170] S11, resource evaluation: Analyze the current stage idle resource information to obtain resource evaluation results. The idle resource information includes CPU usage, memory usage, and disk I / O capacity;

[0171] S12, determine whether the current stage is idle; if so, reconstruct the complete file from the fragments; otherwise, execute S12.

[0172] This means that before transferring the file, we attempt to reassemble the file segments (slices) into a complete file locally. This can be done by creating a temporary file and writing each file segment (slice) to that file in sequence. This process can be performed in parallel if system resources permit.

[0173] (2) File repair compression transmission mechanism: The compression transmission mechanism is a technology that reduces the amount of network transmission data by compressing data. During the data transmission process, the data becomes more compact after compression processing, thereby reducing the bandwidth and transmission time required for transmission. Through the compression transmission mechanism, network resources can be saved, transmission speed can be increased, and the possibility of network congestion can be reduced during the data transmission process. Figure 17 As shown in the figure, the specific process of file repair, compression and transmission is as follows:

[0174] S21, read the characters and the number of repetitions of the corresponding characters from the compressed file, and determine whether the continuous number is completed; if so, write the characters read from the compressed file into the restored file and execute S22; otherwise, execute S21;

[0175] S22, determine whether the file compression is complete; if so, close the compressed file and end; otherwise, execute S21.

[0176] (3) Host interaction and concurrency: Host interaction and concurrency refer to the ways in which different hosts communicate and interact in a computer network. Concurrency refers to the ability to process multiple tasks or connections at the same time, while host interaction refers to the process of communication and data transmission between hosts. In the file verification and repair method of the present invention, the host uses an interactive and concurrent method to perform comparison and repair operations on each task. Specifically, when performing file verification, the host compares data with the user in an interactive manner to ensure the integrity and correctness of the file. In the repair operation, the host uses concurrent technology to process multiple tasks at the same time, which improves operational efficiency and speed. In this way, the host can perform file verification and repair more effectively, improving the security and stability of the system. At the same time, this interactive and concurrent method also reflects the intelligence and efficiency of the host in processing tasks, providing users with more convenient and reliable services. As Figure 18 As shown, the host interaction and concurrent task scheduling process steps are as follows:

[0177] When the interactive and concurrent tasks begin, the source machine first sends an instruction to the target host to start the interactive task. Then, both the source host and the target host will create a thread pool. Once a task request related to verification or repair is received, the source host will send the task request to the target host, including a list of files to be migrated and related parameters. After receiving the task request, the target host will assign the task to a thread in the thread pool for processing based on factors such as the size and type of the file. After the task is executed, the thread pool will automatically close and record the migration log related to verification and repair. Through this process, the interactive and concurrent tasks are carried out in an orderly manner, ensuring the smooth progress of file verification and repair operations. Inter-host interactive and concurrent task scheduling is as follows: Figure 19 shown.

[0178] The present application adopts the above technical scheme, adopts the migration mode based on the agent, and the system can dynamically utilize the computing resource of the source data machine and the destination data machine, guarantees the efficiency and speed of file migration in different links.

[0179] The present application has the following advantages: (1) comprehensive analysis of file migration state: in order to ensure the smooth operation of file migration, the present application divides the state of file migration in detail, including full migration, incremental migration and audit repair state. (2) source and destination host cooperative verification and transmission mechanism: using the agent function of the source data machine and the destination data machine, the cooperative work of file verification and repair between them is realized, and the accuracy and efficiency in the data transmission process are ensured. (3) realization of source and destination host rapid verification function: by adopting the client agent mode, the detection algorithm is constructed, the complex algorithm process includes multiple key steps, and the efficient and rapid file data verification function is realized. (4) target file rapid repair technology landing: accurately positioning the data layer difference of each large file, combining intelligent technologies such as slice transmission and data reorganization, effectively replacing the traditional file retransmission mode, realizing the rapid and efficient repair of target file.

[0180] Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

Claims

1. A file verification and repair method based on an agent migration mechanism, characterized by: It includes the following steps: Step 1: The source data machine traverses all files and then transfers all files. Step 2: Transfer the file from the source location to the target location, and record the incremental information of the file in real time during the transfer process, so that the incremental data recorded in real time can be continuously synchronized after the full transfer is completed; Step 3: Determine whether the user has chosen to stop incremental transfer during the file transfer process. If so, the source and target modems where the incremental transfer was stopped enter the "pending migration and cutover" state. Otherwise, go to step 5; Step 4: Determine whether the user chooses to perform file transfer verification and repair; if yes, proceed to step 5; Otherwise, go to step 6; In step 5, the source and target data machines perform real-time differential information verification through rapid file comparison and perform a slice repair process based on the differential information to retransmit lost or damaged files. The rapid file comparison in step 5 specifically includes the following steps: Step 5-11: When the fast file comparison operation is started, the source data machine first generates a comparison order list according to the preset sorting rules; Step 5-12, the source modem sends a command with a comparison order list to the target modem for synchronization operation; Step 5-13, calculating and selecting the optimal data extraction step value based on the performance of the source end data machine and the target end data machine; Step 5-14: Based on the optimal data extraction step value, the files to be compared are traversed one by one in the order in the comparison order list, and the file contents are hashed to convert the file contents into a unique hash value of a fixed length; Step 5-15, comparing the hash values ​​of the corresponding files of the source data machine and the target data machine; If the hash values ​​are different, it is determined that there are differences in the corresponding file contents, and the system records the corresponding difference information; Step 6: Wait for the full file transfer to complete and synchronize the incremental data recorded in real time to complete the file migration.

2. The file verification and repair method based on the proxy migration mechanism according to claim 1, characterized in that: In step 1, the metadata information of the file is recorded during the full traversal. The metadata information includes the file size and path.

3. The file verification and repair method based on the proxy migration mechanism according to claim 1, characterized in that: In step 5, when the source and target data machines begin to verify and repair, a comparison instruction is issued through the migration client. The source and target data machines will work together to perform a rapid file comparison and generate a detailed difference list through the file comparison. If there are difference values ​​in this file comparison, the corresponding difference list will be repaired.

4. The file verification and repair method based on the proxy migration mechanism according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5-1: The source and target data machines begin verification and repair, and comparison instructions are issued through the migration client. In step 5-2, the source and target data machines will work together to quickly compare files and generate a detailed list of differences. Step 5-3: Determine whether there is differential information based on the difference list; if so, perform a repair operation on the corresponding difference list and then execute step 5-4; otherwise, execute step 5-4; In step 5-4, manually recheck whether the repair is successful; if so, complete the repair; otherwise, proceed to step 5-1.

5. The file verification and repair method based on the proxy migration mechanism according to claim 4, characterized in that: The selection of the optimal data extraction step value in steps 5-13 specifically includes the following steps: Step 5-13-1, initialize the step reference value and set it to the total amount of data divided by 1024; Step 5-13-2, the current step value is set to 2 / 3 of the base value; Step 5-13-3: Generate a hash value based on the current step value and calculate the CPU and memory usage of the source and target data machines. Step 5-13-4, determine whether the CPU and memory usage is less than or equal to 5% of the expected allocation; if so, set the step value to 2 / 3 of the current value and recalculate the resource usage; otherwise, set the step value to 3 / 2 of the current value and output the optimal step value.

6. The file verification and repair method based on the proxy migration mechanism according to claim 4, characterized in that: The steps for extracting the hash calculation in steps 5-14 are as follows: Step 5-14-1, construct an all-zero sequence, that is, set all data to zero value; Step 5-14-2, obtaining data content extracted from the source file or database based on the optimal data extraction step value; Step 5-14-3, extracting data samples from the extracted data content to obtain sample extraction results; Step 5-14-4, interpolate the data content of the sample extraction result with the all-zero sequence; Step 5-14-5, perform hash calculation on the extracted values ​​after interpolation to generate a randomized hash value for data verification or comparison to ensure the integrity and accuracy of the data.

7. The file verification and repair method based on the proxy migration mechanism according to claim 6, characterized in that: The specific steps for data sample extraction in step 5-14-3 are: S01, set the character spacing for data sample extraction, that is, set the optimal step; S02, traverse the extracted data content and extract a bit every character spacing; S03, after traversing to the end of the file, the extracted data values ​​are merged to obtain the extraction result.

8. The file verification and repair method based on the proxy migration mechanism according to claim 1, characterized in that: The slice repair process in step 5 specifically includes the following steps: Step 5-21, obtain the difference information and determine whether the slice size has been allocated; if so, execute step 5-22; otherwise, initialize the slice size and execute step 5-22; Step 5-22, data repair begins, and it is determined whether the size of the file to be repaired is larger than the slice size. If so, the file is cut into slices according to the slice size to obtain a number of slice data and step 5-23 is executed. Otherwise, the corresponding file is deleted and retransmitted from the source data machine, and then step 5-25 is executed. In step 5-23, hash and compare the individual slice data with the data on the source data machine to determine if there are any differences. If so, re-upload all slice data subsequent to the current slice data and merge them into a repair file before executing step 5-24. Otherwise, select the next slice data set and execute step 5-23. Step 5-24, calculate the resource usage status of the repair process and determine whether the resource usage is less than the set value; if so, recalculate the allocated slice size and update the slice size value, and execute step 5-25; otherwise, execute step 5-25; Step 5-25: Output the verification and repair results to complete the repair process.

Images