Data migration verification method and device, electronic equipment and storage medium

Abstract

This application provides a data migration verification method, apparatus, electronic device, and storage medium, relating to the field of data processing technology. It dynamically determines the subsequent sampling verification ratio by combining the quality status of the current batch of migrated data with the system resource status. It adaptively selects the matching verification level based on the importance of the files to be verified and historical error records. Furthermore, when an erroneous file is discovered, it extracts relevant features for root cause analysis, and then executes a backtracking verification strategy associated with already verified files and an enhanced verification strategy for subsequent migrated data. Therefore, it can solve the problems in existing technologies where full-volume verification is time-consuming and resource-intensive, fixed sampling verification lacks flexibility, and it cannot adaptively adjust the verification method according to file characteristics and system status, making it difficult to accurately trace and prevent the spread of errors after they occur. This achieves a technical effect that balances verification efficiency and data consistency, and optimizes system resource allocation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration verification method and apparatus, electronic device and storage medium. Background Technology

[0002] Data migration is a key technology for optimizing storage resource allocation and ensuring business continuity in scenarios such as enterprise data center upgrades, cloud storage integration, and big data platform construction. Data verification is the core link to ensure the integrity and consistency of data after migration, and the existing mainstream solutions are full verification and fixed sampling verification.

[0003] While full verification can guarantee the highest accuracy, when dealing with massive amounts of data, it requires traversing all data blocks for complete comparison, which is not only time-consuming but also consumes a large amount of server computing, memory, and network bandwidth resources, severely slowing down the migration progress and affecting the operation of related businesses. Fixed sampling verification lacks flexibility, which can easily lead to resource waste when the migration quality is stable, and when the quality fluctuates, it may lead to inaccurate error rate assessment and the risk of missed detection due to insufficient samples, affecting the data reliability of subsequent business decisions. Summary of the Invention

[0004] This application provides a data migration verification method, apparatus, electronic device, and storage medium to at least partially solve one of the technical problems in the related art.

[0005] This application provides a data migration verification method, including:

[0006] Based on the quality of the current batch of migration data and the status of system resources, the sampling verification ratio of subsequent batches of migration data is dynamically determined. For the file to be verified, the verification level is adaptively selected and verified based on its importance and historical error records. The verification levels include at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content. When a file with errors is found during verification, the relevant features of the file with errors are extracted for root cause analysis. Based on the analysis results, backtracking verification is performed on the associated verified files, and enhanced verification strategies are implemented on the subsequent data to be migrated.

[0007] This application also provides a data migration verification device, including: The determination unit is used to dynamically determine the sampling verification ratio of subsequent batches of migration data based on the quality status of the current batch of migration data and the system resource status. The verification unit is used to adaptively select the matching verification level for the file to be verified based on its importance and historical error records, and to perform verification. The verification levels include at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content. The analysis unit is used to extract relevant features of the erroneous file when the verification finds an error, to perform root cause analysis of the error, to perform backtracking verification on the associated verified files based on the analysis results, and to implement enhanced verification strategies for subsequent data to be migrated.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0011] This application provides a data migration verification method, apparatus, electronic device, and storage medium. It dynamically determines the subsequent sampling verification ratio by combining the quality status of the current batch of migrated data with the system resource status. It adaptively selects the matching verification level based on the importance of the files to be verified and historical error records. Furthermore, when an erroneous file is discovered, it extracts relevant features for root cause analysis, and then executes a backtracking verification strategy associated with already verified files and an enhanced verification strategy for subsequent data to be migrated. Therefore, it can solve the problems in existing technologies where full-volume verification is time-consuming and resource-intensive, fixed sampling verification lacks flexibility, and it cannot adaptively adjust the verification method according to file characteristics and system status, making it difficult to accurately trace and prevent error propagation after errors occur. It achieves the technical effects of balancing verification efficiency and data consistency, optimizing system resource allocation, reducing resource waste, lowering the risk of missed detections, effectively blocking error propagation, and ensuring data migration reliability and business continuity.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1A flowchart illustrating a data migration verification method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a data migration verification device provided in an embodiment of this application. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The specific application environment architecture or specific hardware architecture on which the execution of the data migration verification method depends is described here.

[0017] The embodiments of this application provide a data migration verification method. Figure 1 This is a flowchart illustrating a data migration verification method provided in an embodiment of this application.

[0018] like Figure 1 As shown, the method includes the following steps: Step 101: Dynamically determine the sampling verification ratio of subsequent batches of migration data based on the quality status of the current batch of migration data and the system resource status.

[0019] In the embodiments of this application, the quality status of migration data refers to relevant indicators that reflect the accuracy and reliability of the current data migration, and the system resource status refers to the occupancy level of hardware and network resources that carry out data migration and verification tasks. By collecting and comprehensively analyzing these two types of information in real time, the sampling verification ratio of subsequent batches of migration data is dynamically adapted, so that the sampling ratio can be intelligently adjusted according to the fluctuation of migration quality and changes in resource load. This avoids verification deviations caused by insufficient samples when the quality is unstable, and reduces unnecessary resource consumption when the quality is stable or resources are scarce. As one implementation method, the verification pass rate can be selected as the core indicator for quality status, and CPU utilization, memory occupancy, and bandwidth utilization can be collected for system resource status. The sampling ratio can be calculated based on a preset algorithm combined with the above indicators, and priority is given to sampling resources for data with high business weight and high historical error rate.

[0020] By dynamically adjusting the sampling ratio driven by both quality and resources, a dynamic balance between verification accuracy and resource efficiency is achieved. This effectively solves the problems of lack of flexibility in traditional fixed sampling verification and excessive resource consumption in full verification, and significantly improves the adaptability and economy of data migration verification.

[0021] Step 102: For the file to be verified, based on its importance and historical error records, adaptively select the matching verification level and perform verification. The verification level includes at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content.

[0022] In the embodiments of this application, the importance of the file to be verified refers to the weight of the file's impact on business operations and decisions. Historical error records are error-related information generated during the past migration or verification process of the file. By comprehensively evaluating these two core factors, an adaptive matching verification level is made to suit the file's risk characteristics. Different verification levels correspond to different comparison ranges and depths. Resource consumption is positively correlated with verification accuracy. The core logic is to allocate more sufficient verification resources to high-risk files and simplify the verification process for low-risk files. The verification levels at least cover metadata-based consistency comparison that only compares the basic identification information of the file, partial content-based consistency comparison that selects key segments of the file for verification, and full content-based consistency comparison that verifies the complete data of the file. As one implementation method, the importance of the file can be divided into multiple levels according to business priority, and historical error records can be quantified as historical error rates. When the importance of the file reaches a preset high priority or the historical error rate exceeds a set threshold, a full content-based consistency comparison is triggered. When the file is a normal business file and has no historical errors, metadata-based consistency comparison can be used.

[0023] By precisely matching file risk characteristics with verification levels, the verification accuracy of high-importance, high-error-risk files is ensured, while unnecessary resource consumption of low-risk files is avoided. This effectively solves the problem of unreasonable resource allocation in traditional verification methods and achieves an optimized balance between verification accuracy and resource efficiency.

[0024] Step 103: When an error is found in a file during verification, the relevant features of the error file are extracted for error root cause analysis. Based on the analysis results, backtracking verification is performed on the associated verified files, and an enhanced verification strategy is implemented on the subsequent data to be migrated.

[0025] In the embodiments of this application, the relevant features of the erroneous file refer to key information that characterizes the background, attributes, and relationships of the error. After the erroneous file is discovered during verification, these features are extracted to conduct root cause analysis, accurately locating the core cause of the error. Then, based on the analysis results, the verification process is optimized bidirectionally: on the one hand, backtracking verification is performed on verified files that are potentially related to the error to avoid undiscovered errors hidden in historical data; on the other hand, enhanced verification strategies are implemented on subsequent data to be migrated to prevent the recurrence of similar errors in advance. As one implementation method, the relevant features of the erroneous file may include time features, storage location features, and type features. Root cause analysis can infer causes such as network transmission fluctuations, storage system failures, or encoding parsing anomalies based on the feature distribution patterns. Backtracking verification can be performed on verified files related to batches, storage nodes, or file formats. Enhanced verification strategies may include increasing the sampling rate and upgrading the verification strength.

[0026] By combining precise root cause identification with optimized two-way verification, the propagation of errors is effectively blocked, overcoming the shortcomings of traditional verification methods that make it difficult to trace and control errors after they occur, and significantly improving the integrity and reliability of data migration verification.

[0027] This application provides a data migration verification method that dynamically determines the subsequent sampling verification ratio by combining the quality status of the current batch of migrated data with the system resource status. It adaptively selects the matching verification level based on the importance of the files to be verified and historical error records. Furthermore, when erroneous files are discovered, it extracts relevant features for root cause analysis, and then executes backtracking verification of associated verified files and enhanced verification strategies for subsequent data to be migrated. Therefore, it can solve the problems of existing technologies where full-volume verification is time-consuming and resource-intensive, fixed sampling verification lacks flexibility, and it cannot adaptively adjust the verification method according to file characteristics and system status, making it difficult to accurately trace and prevent error propagation after errors occur. This method achieves the technical effects of balancing verification efficiency and data consistency, optimizing system resource allocation, reducing resource waste, lowering the risk of missed detections, effectively blocking error propagation, and ensuring data migration reliability and business continuity.

[0028] In the embodiments involved in this application, there are various feasible specific implementation methods. To clearly and completely illustrate the technical solutions of this disclosure, the implementation methods listed below are merely exemplary and do not constitute a limitation on the scope of protection of this disclosure. That is, in addition to the implementation methods described below, other implementation methods that can be obtained by those skilled in the art based on the technical content disclosed in this disclosure through reasonable logical analysis, reasoning, or limited experimentation should also be covered within the scope of protection of this disclosure. The following specifically describes some exemplary implementation methods: As a specific implementation of this application, based on the basic scheme, it further defines the sampling verification ratio of subsequent batches of migration data dynamically according to the quality status of the current batch of migration data and the system resource status. This includes: obtaining the verification pass rate, error classification statistics, and resource consumption indicators during the verification process of the current batch of data; obtaining the current CPU utilization, memory usage, and network bandwidth utilization of the system; and calculating the sampling verification ratio based on the verification pass rate, error classification statistics, resource consumption indicators, and system resource utilization through predetermined adjustment rules.

[0029] Specifically, the quality of the current batch of migration data is quantified through the pass rate and error classification statistics. The pass rate is calculated as (number of files that passed verification ÷ total number of sampled files) × 100% and rounded to two decimal places. The error classification statistics include the percentage of various errors such as missing data, content tampering, and formatting errors. Resource consumption indicators during the verification process include the average CPU usage, memory usage, and bandwidth consumption of the migration agent during the verification period. System resource status is obtained through real-time monitoring, specifically the CPU usage, memory usage, and network bandwidth usage refreshed every minute, with the average of the current five consecutive monitoring data points used as valid data. The predetermined adjustment rule adopts a preset algorithm formula, namely, the sampling rate of the next batch = max(10%, min(100%, baseline sampling rate × (1 + α × (target pass rate - current pass rate) + β × error severity coefficient + γ × (1 - system load ratio) + δ × business weight coefficient + ε × hot data weighting coefficient))), where α, β, γ, δ, and ε are preset weight coefficients. The baseline sampling rate is pre-calculated based on the initial full-volume verification results and resource load baseline. The error severity coefficient is determined based on error classification statistics. The system load ratio is calculated from the current system resource utilization rate and resource load baseline value.

[0030] By comprehensively collecting key quantitative indicators in terms of quality and resources, and using preset rules with multi-factor weighting to calculate the sampling ratio, the sampling rate adjustment becomes more objective and accurate. This ensures the verification coverage when quality fluctuates and avoids ineffective consumption when resources are scarce, further improving the intelligence level of data migration verification and the efficiency of resource utilization.

[0031] As a specific implementation of this application, based on the basic scheme, the sampling verification ratio is further defined by calculating it through predetermined adjustment rules, including: the sampling verification ratio is calculated comprehensively based on the benchmark sampling rate, the deviation between the current batch verification pass rate and the preset target pass rate, the weighting coefficient reflecting the severity of the error, the system resource load rate, the weighting coefficient reflecting the importance of the file business, and the weighting coefficient reflecting the data activity.

[0032] Specifically, the baseline sampling rate is pre-calculated using the full verification results and resource load baseline during the initialization phase. The calculation formula is: max(benchmark sampling rate lower limit, min(benchmark sampling rate upper limit, (1)) The initial average error rate is calculated as (initial average error rate / resource consumption coefficient × default sampling rate), where the initial average error rate is based on the full verification results of the 100 batches of data before migration, and the resource consumption coefficient is determined by combining the average CPU usage, memory usage, and peak bandwidth during the initial verification. The current batch verification pass rate is calculated as (number of files that passed verification ÷ total number of sampled files) × 100% and retained to two decimal places. Its deviation from the preset target pass rate is directly included as an adjustment factor in the calculation. The weighting coefficient (β) reflecting the severity of errors is set according to the error classification statistics, with the highest coefficient for content tampering errors, followed by data missing errors, and the lowest for format errors. The system resource load rate is the ratio of the average of the current CPU usage, memory usage, and network bandwidth usage to the resource load baseline. The weighting coefficient (δ) reflecting the importance of file business corresponds to the predefined 1-5 level business weight labels of the file, with the 5 level core business data corresponding to the largest coefficient. The weighting coefficient (ε) reflecting data activity focuses on hot data, and files with modification records in the past 30 days are given an additional weighting value of 0.5. The sampling verification ratio is calculated using a preset algorithm formula, namely, the sampling rate of the next batch = max(10%, min(100%, baseline sampling rate × (1 + α × (target pass rate - current pass rate) + β × error severity coefficient + γ × (1 - system load ratio) + δ × business weight coefficient + ε × hot data weighting coefficient))), where α and γ are preset fixed weight coefficients to ensure the balanced effect of each adjustment factor.

[0033] By integrating six core factors—benchmark sampling rate, quality deviation, error severity, system load, business importance, and data activity—to calculate the sampling ratio, the sampling rate adjustment is made more in line with the multi-dimensional needs of actual migration scenarios. This ensures the verification coverage of high-risk data and core business data, while dynamically adapting to system resource status and data dynamic changes, significantly improving the accuracy of sampling verification and scenario adaptability.

[0034] As a specific implementation of this application, based on the basic solution, it further limits the selection of a matching verification level and performs verification based on the importance of the file and historical error records. This includes: predefining multiple verification levels, each corresponding to a different scope of verification content and resource consumption level; matching the importance attribute, historical error rate, and current system resource load status of the file to be verified with the applicable conditions of each verification level, thereby selecting the verification level.

[0035] Specifically, the predefined verification levels (i.e., "verification strength") are divided into three levels: lenient, standard, and strict. Each level corresponds to a specific scope of verification content and resource consumption level: the lenient level only performs consistency comparison on file metadata (including filename, size, and modification timestamp), with the lowest resource consumption; the standard level, based on metadata comparison, selects core segments of the file to perform partial content consistency comparison, with moderate resource consumption; and the strict level performs a full consistency comparison on the complete file data, with the highest resource consumption. The importance attribute of the file to be verified uses predefined business weight labels of 1-5 levels (level 5 being core business data). The historical error rate is calculated as (number of historical error files ÷ total number of historical verification files) × 100%. The current system resource load status is obtained through real-time monitoring, specifically CPU utilization, memory usage, and network bandwidth utilization refreshed every minute, with the average of 5 consecutive monitoring readings taken as valid data. The applicable conditions for each verification level are preset as follows: core business files (levels 4-5) or files with a historical error rate > 5% are matched with the strict level by default; ordinary business files (levels 1-3) with a historical error rate ≤ 5% are matched with the standard level if the system load is < 50% for 10 minutes, and with the system load > 70% for 5 consecutive minutes; other cases are matched with the standard level by default. File types with a historical error rate > 5% are forcibly upgraded by one level of verification strength, and the default level is restored after 3 consecutive batches without errors.

[0036] By clearly defining the criteria and adaptation rules for verification levels, precise matching between file risk characteristics and system resource status is achieved. This ensures the verification accuracy of core business files and files with high error risk, while dynamically adapting verification resource consumption according to system load. This effectively avoids the resource waste of a "one-size-fits-all" verification approach and further improves the targeting and resource utilization efficiency of data migration verification.

[0037] As a specific implementation of this application, based on the basic solution, the relevant features of the erroneous file are further extracted for error root cause analysis, including: collecting the timestamp information, storage location information and file attribute information of the erroneous file; and determining the error root cause category based on the distribution characteristics of the erroneous file in the dimensions of time, space and type, according to the preset inference rules. The root cause category includes network transmission abnormality, storage system abnormality or data encoding processing abnormality.

[0038] Specifically, the collection of relevant features for error files needs to cover three core dimensions: time, space, and type. Timestamp information must be collected precisely to the millisecond level, while also recording the migration batch to which the error file belongs. Storage location information specifically includes the IP address of the storage node where the file resides and its detailed directory path. File attribute information focuses on file format (e.g., .docx, .csv, .jpg) and size range (e.g., 100MB-500MB). The preset inference rules use decision tree logic to determine the root cause category by analyzing the distribution patterns of features: if the standard deviation of the time distribution of error files is >2 hours and they are scattered across ≥5 storage nodes, and there is no specific format concentration (≤20%), then it is determined to be a network transmission anomaly; if more than 70% of error files are concentrated on the same storage node, and 60% of the error times fall within the system maintenance window (e.g., 2:00-4:00 AM), then it is determined to be a storage system anomaly; if the error rate of a specific format file is ≥5 times that of other formats, and the error type is labeled "content tampering" (actually an encoding conversion error), then it is determined to be a data encoding processing anomaly.

[0039] By defining the dimensions of feature collection and quantifying the root cause inference rules, the system achieves accurate and rapid location of error root causes, avoiding the subjectivity and ambiguity of traditional error analysis. This provides a reliable basis for the targeted execution of subsequent backtracking verification and enhanced verification strategies, significantly improving the efficiency and accuracy of error handling.

[0040] As a specific implementation of this application, based on the basic scheme, it further limits the backtracking verification of the related verified files according to the analysis results, including: if the root cause is a network transmission anomaly, then perform content sampling verification on all files in the migration batch to which the erroneous file belongs; if the root cause is a storage system anomaly, then perform full content verification on all files migrated within a preset time period on the storage node where the erroneous file is located; if the root cause is a data encoding processing anomaly, then perform full content verification on recently migrated files of the same type as the erroneous file.

[0041] Specifically, the execution strategy for backtracking verification strictly matches the root cause analysis results: If the root cause is network transmission anomaly, the backtracking object is limited to all files within the migration batch to which the erroneous file belongs. Content sampling verification (i.e., standard-level verification) is performed, comparing the core segments of each file for consistency to ensure that the batch data is free of hidden errors caused by network fluctuations. If the root cause is storage system anomaly, the preset time period is set to the past 24 hours. The backtracking object is all files migrated from the storage node containing the erroneous file within this time period. Full content verification (i.e., strict-level verification) is performed, comparing the complete file data using hash algorithms such as SHA-256 to comprehensively investigate batch errors caused by storage node failures. If the root cause is abnormal data encoding processing, recently migrated files are specifically defined as all files in the last three batches of migrated data with the same format as the erroneous file. Full content verification is performed to accurately cover similar files affected by encoding conversion issues. During the backtracking process, if another erroneous file is found within the current verification range, the backtracking range continues to expand forward using the same strategy until no errors are found after continuously verifying a complete batch, at which point the backtracking terminates.

[0042] By adopting a root cause-oriented differentiated backtracking strategy, we achieved precise positioning and comprehensive coverage of the scope of error impact. This not only avoided the waste of resources caused by indiscriminate backtracking, but also effectively prevented the omission of historical error files, significantly improved the pertinence and thoroughness of error correction, and further ensured the overall consistency of data migration.

[0043] As a specific implementation of this application, based on the basic solution, the embodiment of this application further includes: performing full content verification on a preset number of initial batch migration data to obtain an initial error rate and system resource consumption benchmark; calculating the benchmark sampling rate based on the initial error rate and system resource consumption benchmark when dynamically adjusting the sampling verification ratio; constructing and storing the feature information of the files to be migrated, the feature information including at least business importance identifier, historical error records and time attribute information.

[0044] Specifically, the initial batch migration data is set to 100 batches, each containing 10,000 files. A full content verification is enforced on these initial batches, using hash algorithms such as SHA-256 to calculate the content consistency between the source and target files. Metadata (filename, size, modification time) matching rate and content hash consistency rate are recorded. The initial error rate (i.e., the initial average error rate) is calculated based on the verification results. System resource consumption baselines are obtained through real-time monitoring, recording the migration agent's CPU usage, memory usage, and network bandwidth utilization every minute. The average of all monitored data is used as the resource load baseline value. The baseline sampling rate is calculated using a preset formula: Baseline Sampling Rate = max(Lower Baseline Sampling Rate, min(Upper Baseline Sampling Rate, (1)) The initial average error rate is calculated as (initial average error rate / resource consumption coefficient × default sampling rate), where the resource consumption coefficient is determined based on the system resource consumption baseline during the initial full verification, and the upper and lower limits of the baseline sampling rate are preset fixed thresholds. The feature information of the files to be migrated is constructed by scanning source data and stored in the file feature library. The business importance identifier is a predefined 1-5 level business weight label (level 5 being core business data). Historical error records include the number and type of errors in past file migrations or verifications. Time attribute information includes the file creation timestamp and the most recent modification timestamp. It also includes basic feature information such as file size (accurate to bytes) and format type (e.g., .docx, .csv, .jpg), providing data support for subsequent dynamic sampling and verification level selection.

[0045] The initial batch full verification established a reliable quality and resource benchmark. Combined with the construction of the file feature library, it provided accurate data basis for subsequent dynamic adjustment of sampling ratio and adaptive matching of verification level, effectively ensuring the stability and accuracy of the entire verification scheme and laying a solid foundation for the efficient implementation of subsequent core technical links.

[0046] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0047] Corresponding to the data migration verification method described above, this disclosure also proposes a data migration verification device. Since the device embodiments of this disclosure correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0048] Figure 2 This is a schematic diagram of the structure of a data migration verification device provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, it includes: The determination unit 21 is used to dynamically determine the sampling verification ratio of the migration data of the subsequent batches based on the quality status of the current batch of migration data and the system resource status. The verification unit 22 is used to adaptively select a matching verification level for the file to be verified based on its importance and historical error records, and to perform verification. The verification levels include at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content. The analysis unit 23 is used to extract relevant features of the erroneous file for error root cause analysis when the verification finds an erroneous file, and to perform backtracking verification on the associated verified files based on the analysis results, as well as to implement enhanced verification strategies for subsequent data to be migrated.

[0049] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0050] For a description of the features in the embodiment corresponding to the data migration verification device, please refer to the relevant description of the embodiment corresponding to the data migration verification method, which will not be repeated here.

[0051] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data migration verification method embodiments.

[0052] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data migration verification method embodiments at runtime.

[0053] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0054] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data migration verification method embodiments.

[0055] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data migration verification method embodiments.

[0056] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0057] The data migration verification method, apparatus, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data migration verification method, characterized in that, include: Based on the quality of the current batch of migration data and the status of system resources, the sampling verification ratio of subsequent batches of migration data is dynamically determined. For the file to be verified, based on its importance and historical error records, an appropriate verification level is adaptively selected and verified. The verification level includes at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content. When a file with errors is found during verification, the relevant features of the file with errors are extracted for root cause analysis. Based on the analysis results, backtracking verification is performed on the associated verified files, and enhanced verification strategies are implemented on the subsequent data to be migrated.

2. The method according to claim 1, characterized in that, The step of dynamically determining the sampling verification ratio for subsequent batches of migration data based on the quality status of the current batch of migration data and the system resource status includes: Obtain the pass rate, error classification statistics, and resource consumption indicators during the verification process for the current batch of data; Obtain the current CPU usage, memory usage, and network bandwidth usage of the system; Based on the verification pass rate, error classification statistics, resource consumption indicators, and system resource utilization, the sampling verification ratio is calculated using predetermined adjustment rules.

3. The method according to claim 2, characterized in that, The step of calculating the sampling verification ratio using predetermined adjustment rules includes: The sampling verification ratio is calculated by comprehensively considering the baseline sampling rate, the deviation between the current batch verification pass rate and the preset target pass rate, the weighting coefficient reflecting the severity of errors, the system resource load rate, the weighting coefficient reflecting the importance of file business, and the weighting coefficient reflecting data activity.

4. The method according to claim 1, characterized in that, The process of adaptively selecting and performing verification based on the importance of the error and historical error records includes: Multiple verification levels are predefined, each corresponding to a different scope of verification content and resource consumption level; The importance attributes and historical error rate of the file to be verified are matched with the current system resource load status and the applicable conditions of each verification level to select the verification level.

5. The method according to claim 1, characterized in that, The extraction of relevant features from the erroneous file for root cause analysis includes: Collect timestamp information, storage location information, and file attribute information of the erroneous files; Based on the distribution characteristics of erroneous files in terms of time, space, and type, the root cause category of the error is determined according to preset inference rules. The root cause category includes network transmission anomaly, storage system anomaly, or data encoding and processing anomaly.

6. The method according to claim 5, characterized in that, The backtracking verification of the associated verified files based on the analysis results includes: If the root cause is a network transmission anomaly, then perform content sampling verification on all files in the migration batch to which the erroneous file belongs; If the root cause is a storage system anomaly, then a full content verification will be performed on all files migrated from the storage node where the erroneous file is located within a preset time period. If the root cause is an abnormal data encoding process, then perform a full content check on recently migrated files of the same type as the erroneous file.

7. The method according to claim 1, characterized in that, Also includes: Perform full content verification on a preset number of initial batches of migration data to obtain an initial error rate and system resource consumption baseline. Based on the initial error rate and system resource consumption benchmark, the benchmark sampling rate used for subsequent dynamic adjustment of the sampling verification ratio is calculated. The feature information of the files to be migrated is constructed and stored. The feature information includes at least business importance identifiers, historical error records, and time attribute information.

8. A data migration verification device, characterized in that, include: The determination unit is used to dynamically determine the sampling verification ratio of subsequent batches of migration data based on the quality status of the current batch of migration data and the system resource status. The verification unit is used to adaptively select a matching verification level for the file to be verified based on its importance and historical error records, and to perform verification. The verification level includes at least consistency comparison based on metadata, consistency comparison based on partial content, and consistency comparison based on all content. The analysis unit is used to extract relevant features of the erroneous file when the verification finds an error, to perform root cause analysis of the error, to perform backtracking verification on the associated verified files based on the analysis results, and to implement enhanced verification strategies for subsequent data to be migrated.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data migration verification method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the data migration verification method according to any one of claims 1-7.

Images