An abnormality detection method, device, system and storage medium for data migration

Abstract

Embodiments of the present application provide a data migration detection method, device, system and storage medium, the method comprising: obtaining information digests of next layer sub-files of a source file and a target file respectively; the source file represents a file package of migrated data; the target file represents a file package obtained after migration of the source file; the information digest of the sub-file comprises a size of the sub-file, a storage location of the sub-file and a number of next layer files of the sub-file; generating an information digest of the source file based on the information digests of the next layer sub-files of the source file; generating an information digest of the target file based on the information digests of the next layer sub-files of the target file; and determining a data migration detection result according to the information digest of the source file and the information digest of the target file.

Description

Technical Field

[0001] This application relates to data migration technology, and more particularly to a method, apparatus, system, and storage medium for detecting anomalies in data migration. Background Technology

[0002] In related technologies, when applying anomaly detection of file migration to big data platforms, the data volume involved may reach terabytes (TB) or petabytes (PB). Calculating information summaries of this data based on its content often requires significant computing resources, making large-scale detection difficult. Summary of the Invention

[0003] This application aims to provide an anomaly detection method, apparatus, system, and storage medium for data migration.

[0004] In a first aspect, embodiments of this application provide a method for detecting data migration, including:

[0005] Information digests of the next-level sub-files of the source file and the target file are obtained respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0006] Generate an information digest of the source file based on the information digest of the next-level sub-file of the source file;

[0007] Generate an information digest of the target file based on the information digest of the next-level sub-file of the target file;

[0008] The detection results of data migration are determined based on the information digests of the source file and the target file.

[0009] Secondly, embodiments of this application provide a data migration detection device, comprising:

[0010] The acquisition module is used to acquire the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0011] The first generation module is used to generate an information digest of the source file based on the information digest of the next-level sub-file of the source file;

[0012] The second generation module is used to generate an information digest of the target file based on the information digest of the next-level sub-file of the target file;

[0013] The first determining module is used to determine the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0014] Thirdly, the embodiments disclosed in this application also provide a data migration detection system, including: a memory and a processor;

[0015] The memory stores a computer program that can run on a processor, which, when executing the program, implements the steps of the data migration detection method according to any one of the preceding claims.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing executable instructions for implementing the steps of the data migration detection method described above when executed by a processor.

[0017] In this embodiment, information digests of the next-level sub-files of the source and target files are obtained. Information digests of the source file are generated based on these sub-files, and information digests of the target file are generated based on these sub-files. Finally, the detection result of the data migration is determined based on the information digests of the source and target files. Since the information digests of the source and target files are generated from the information digests of the next-level sub-files, rather than from the entire content of the source and target files, the computational resource consumption problem of information digest generation can be effectively solved, making it suitable for large-scale data detection.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 A schematic diagram illustrating the implementation process of a data migration detection method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram illustrating the implementation process of the batch detection method for abnormal migration files in a big data platform provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram of the deep-dimensional drilling process provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram illustrating the transmission flow of the task start summary generation instruction provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the digest generation instruction transmission process provided in an embodiment of this application when an anomaly is detected;

[0025] Figure 6 A schematic diagram illustrating the process of the information transmission module transmitting information between the summary generation module, the anomaly detection module, and the deep-dimensional drilling module in an embodiment of this application.

[0026] Figure 7 The anomaly detection module provided in this application embodiment determines the source file information digest I(X) i ) and target document information digest I(Y) i Is the implementation process consistent? (Diagram showing the process flow)

[0027] Figure 8 A schematic diagram of the algorithm flow for the interaction between the deep-dimensional drilling module and other modules provided in the embodiments of this application;

[0028] Figure 9 A schematic diagram illustrating the implementation flow of the result output algorithm provided in the embodiments of this application;

[0029] Figure 10 A schematic diagram illustrating the structural composition of a data migration detection device provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of a data migration detection system provided in an embodiment of this application. Detailed Implementation

[0031] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the embodiments of the present application can be implemented in any combination.

[0032] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0033] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, U and / or W can represent: U existing alone, U and W existing simultaneously, and W existing alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of U, W, and V can mean including any one or more elements selected from the set consisting of U, W, and V.

[0034] During the migration of the big data platform, data files in the two resource pools frequently experience transmission errors due to network connection problems, task processing thread anomalies, access timeouts, and cluster resource constraints. The necessity of comprehensive data file anomaly detection is primarily reflected in the following aspects:

[0035] ① Numerous files: The number of files migrated daily reaches millions;

[0036] ② Huge data volume: The daily data migration volume reaches tens of billions of bytes (TB), or even petabytes (PB);

[0037] ③ Problems occur frequently: especially in the early stages of migrating services, the probability of network and cluster instability is higher;

[0038] ④ Complex anomaly types: There are problems such as missing data files or verification files, inconsistent sizes between target and source files, migration tool renaming failure, and duplicate file pushes.

[0039] The relevant data migration technologies have the following drawbacks:

[0040] 1. To obtain test results by comparing data with the same key, each data item needs to be encapsulated into a key-value pair. Therefore, it is more suitable for relational databases with primary keys. When promoting it in anomaly detection scenarios of databases without primary keys and non-data type files, the cost of generating key-value pairs is relatively high.

[0041] 2. Splitting large files into multiple smaller files can solve the problem of excessively large single file sizes requiring retransmission when a large file fails. However, this solution does not consider the computational resource consumption of generating message digests using the Message-Digest Algorithm 5 (MD5) algorithm in scenarios where the number of files migrated on a big data platform is too large.

[0042] 3. By dividing the message digest calculation work into multiple threads, the problem of computational efficiency is alleviated to some extent. However, when the number of files and the file size of a single thread are too large, the scalability of this method is limited by the number of underlying processor cores.

[0043] To address the shortcomings of existing technical solutions in detecting anomalies during big data migration, the following technical issues need to be resolved:

[0044] 1. The detection method and apparatus are applicable not only to anomaly detection of data type file migration containing primary keys, but also to data type files without primary keys and non-data type files.

[0045] 2. Extending migration file anomaly detection to petabyte-scale big data platform migration operations requires addressing the computational resource consumption issue of generating information digests.

[0046] 3. In order to detect millions of migration files, it is necessary to solve the problems of how to perform batch detection and how to output the detection results efficiently.

[0047] 4. In order to cope with complex migration anomaly scenarios, it is necessary to improve the flexibility and scalability of detection methods.

[0048] To address the aforementioned technical problems, this application provides a data migration detection method. This method can be used to detect the consistency between data before and after migration. Although it does not detect the migration process itself, by comparing the data before and after migration, it can infer which faults or anomalies occurred during the migration process. This data migration detection method can be applied to a data migration detection system, which includes at least an information digest generation module and an anomaly detection module, such as... Figure 1 As shown, the method includes:

[0049] Step S101: The digest generation module obtains the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0050] Here, the source file can contain multiple subfiles, and each subfile can contain multiple types of data. For example, source file A can contain subfiles A1, A2, A3, and A4, and A1 can contain different types of data such as word1, excel1, and Microsoft Office PowerPoint (PPT)1.

[0051] It is understandable that the message digest of a subfile includes the metadata of the subfile, which can be obtained through the message digest of the next level of grandchildren of the subfile.

[0052] In one possible implementation, the summary generation module obtains the information summary of the next-level sub-files of the source file and the target file respectively. Alternatively, the summary generation module may directly obtain the information summary of the next-level sub-files of the source file and the target file from the blocks used to store metadata information in the next-level sub-files of the source file and the target file respectively.

[0053] Step S102: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-file of the source file;

[0054] It's understandable that the information digest of a source file does not refer to the information obtained from the actual content of the source file. The information digest includes the source file's metadata, namely, the size of the source file, its storage location, and the number of its sub-files.

[0055] In one possible implementation, the summary generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file. This can be achieved by the summary generation module obtaining the number of sub-files, the storage location of each sub-file, and the size of each sub-file from the information digest of the next-level sub-files of the source file, and determining the size of the source file, the storage location of the source file, and the number of sub-files as the information digest of the source file based on the number of sub-files, the storage location of each sub-file, and the size of each sub-file.

[0056] Step S103: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0057] It's understandable that the information digest of an object file does not refer to the information obtained from the actual content of the object file. The information digest includes the object file's metadata, namely, the size of the object file, its storage location, and the number of its sub-files.

[0058] In one possible implementation, the summary generation module generates an information summary of the target file based on the information summary of the next-level sub-files of the target file. This can be achieved by the summary generation module obtaining the number of sub-files, the storage location of each sub-file, and the size of each sub-file from the information summary of the next-level sub-files of the target file, and determining the size of the source file, the storage location of the target file, and the number of sub-files as the information summary of the target file based on the number of sub-files, the storage location of each sub-file, and the size of each sub-file.

[0059] It is understandable that after the digest generation module obtains the information digests corresponding to the source file and the target file, the information transmission module can transmit the obtained information digests corresponding to the source file and the target file to the target server.

[0060] Step S104: The anomaly detection module determines the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0061] Here, the anomaly detection module can be located on the target server. In this way, after the target server receives the information digests of the source file and the target file from the digest generation module, the anomaly detection module can determine the detection result of the data migration by comparing whether the information digests of the source file and the target file are the same.

[0062] In one implementation, the detection result of data migration is determined based on the comparison results. This can be done by determining that the detection result of data migration is normal if the information digests of the source file and the target file are exactly the same, and by determining that there is a certain type of detection anomaly if the information digests of the source file and the target file are different.

[0063] In practical applications, steps S101 to S104 can be implemented using a processor in the data migration anomaly detection system. The processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor.

[0064] In this embodiment, information digests of the next-level sub-files of the source and target files are obtained. Information digests of the source file are generated based on these sub-files, and information digests of the target file are generated based on these sub-files. Finally, the detection result of the data migration is determined based on the information digests of the source and target files. Since the information digests of the source and target files are generated from the information digests of the next-level sub-files, rather than from the entire content of the source and target files, the computational resource consumption problem of information digest generation can be effectively solved, making it suitable for large-scale data detection.

[0065] Another data migration detection method provided in this application embodiment includes:

[0066] Step 201: The digest generation module obtains the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0067] Step 203: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0068] Step 205: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0069] Step 207: The anomaly detection module determines the source file information corresponding to the anomaly type to be detected from the information digest of the source file and the target file information corresponding to the anomaly type to be detected from the information digest of the target file;

[0070] Understandably, the types of anomalies to be detected can include at least: missing files, duplicate files, inconsistent file sizes, and failed file renaming.

[0071] Here, the source file information corresponding to the anomaly type to be detected can be metadata related to the anomaly type to be detected in the information digest of the source file, which is part of the information digest of the source file; the target file information corresponding to the anomaly type to be detected can be metadata related to the anomaly type to be detected in the information digest of the target file, which is part of the information digest of the target file.

[0072] Step 209: If the source file information and the target file information are consistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is normal.

[0073] Step 211: If the source file information and the target file information are inconsistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is an anomaly.

[0074] Step 213: The anomaly detection module identifies both the source file and the target file as vertex file packages in the data cube model;

[0075] Here, the data cube model can be a multidimensional space constructed from dimensions, containing all the basic data to be analyzed, and all data aggregation operations are performed on the cube. A dimension can be understood as a perspective from which to observe the data, or as an axis of the cube.

[0076] Understandably, a vertex file package can be a file package viewed from the top level in a data cube model. For example, the top-level file package of the source files can be source file A, which can contain subfiles A1, A2, A3, and A4. A1 can contain different types of data such as word1, excel1, and PPT1. In the case that source file A is an abnormal file, source file A can be used as the vertex file package of the data cube model.

[0077] It is understandable that if the information digest of the source file and the information digest of the target file are inconsistent, the anomaly detection module can pass the data stream to the next layer module located on the target server.

[0078] Step 215: The deep-dimensional drill-down module performs at least one drill-down analysis on each vertex file package to obtain the target drill-down analysis results;

[0079] Here, the deep-dimensional drill-down module can be located on the target server to obtain data streams from the anomaly detection module.

[0080] Understandably, at least one drill-down analysis can be a subdivision of at least one specific dimension of the vertex file package, such as the first dimension and the second dimension; the target drill-down result can refer to the information summary of the target drill-down file obtained after performing at least one drill-down analysis on the vertex file package.

[0081] In one possible implementation, step S215 may involve the deep-dimensional drill-down module generating a drill-down command when it determines that the information digest of the source file and the information digest of the target file are inconsistent. The drill-down command is then transmitted to the digest generation modules of the source server and the target server through the information transmission module. The digest generation modules parse the drill-down command, generate and execute the digest command, and obtain the information digest of the first drill-down file of the source file and the information digest of the first drill-down file of the target file.

[0082] Step 217: Based on the target drill-down analysis results, the anomaly detection module determines the anomaly sub-file of the anomaly type to be detected.

[0083] Here, the anomaly detection module can obtain the information digest of the target drill-down file of the source file and the information digest of the target drill-down file of the target file through the summary generation module based on the target drill-down analysis results. Then, it can determine whether the information digest of the target drill-down file of the source file and the information digest of the target drill-down file of the target file are consistent, and determine the anomaly sub-file of the anomaly type to be detected based on the judgment result.

[0084] In this embodiment, to determine the detection result of data migration, drill-down analysis is only performed on source and target files with inconsistent information digests. For source and target files with consistent information digests, the detection result of data migration can be determined directly based on the information digests of the source and target files, without the need to analyze the next-level sub-files of the source and target files. In this way, the detection of migration file anomalies can be extended to the migration business of big data platforms at the petabyte level, realizing batch detection of migration data with high efficiency.

[0085] This application provides another method for detecting data migration, the method comprising:

[0086] Step 301: The digest generation module determines the file attribute value corresponding to the file attribute of the next-level sub-file of the source file as the information digest of the next-level sub-file of the source file based on the inode of each disk sector directory entry and the block information corresponding to the directory entry;

[0087] The source file represents the file package of the data to be migrated;

[0088] The target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file.

[0089] Here, file attributes must include at least: file size, number of files, and file name.

[0090] Step 303: The digest generation module determines the file attribute value corresponding to the file attribute of the next-level sub-file of the target file as the information digest of the next-level sub-file of the target file based on the inode of each disk sector directory entry and the block information corresponding to the directory entry.

[0091] Step 305: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0092] Step 307: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0093] Step 309: The anomaly detection module determines the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0094] In this embodiment, based on the inode of each disk sector directory entry in the next-level sub-file of the source file and the block information corresponding to the directory entry, the file attribute value corresponding to the file attribute of the next-level sub-file of the source file is determined to be the information digest of the next-level sub-file of the source file; similarly, based on the inode of each disk sector directory entry in the next-level sub-file of the target file and the block information corresponding to the directory entry, the file attribute value corresponding to the file attribute of the next-level sub-file of the target file is determined to be the information digest of the next-level sub-file of the target file. Thus, obtaining the information digests of the next-level sub-files of the source and target files is simpler and more accurate, saving resources.

[0095] This application provides yet another method for detecting data migration, the method comprising:

[0096] Step 401: The summary generation module obtains the file directory of the next-level sub-file of the source file;

[0097] Step 403: The digest generation module determines the block information corresponding to the file directory of the next-level sub-file of the source file from the inode of each disk sector directory entry based on the file directory of the next-level sub-file of the source file;

[0098] Step 405: The digest generation module determines the names of the grandchildren of the next-level sub-files of the source file and the inode corresponding to each grandchildren name from the block information corresponding to the file directory of the next-level sub-file of the source file;

[0099] Step 407: The digest generation module determines the number of grandchild files based on the inode corresponding to each grandchild file name, which is the information digest of the next-level sub-file of the source file;

[0100] Step 409: The digest generation module determines the file attribute value corresponding to the file attribute of the next-level sub-file of the target file as the information digest of the next-level sub-file of the target file based on the inode of each disk sector directory entry and the block information corresponding to the directory entry.

[0101] Step 411: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0102] Step 413: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0103] Step 415: The anomaly detection module determines the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0104] In this embodiment, by obtaining the file directory of the next-level sub-file of the source file, and based on the file directory of the next-level sub-file of the source file, determining the block information corresponding to the file directory of the next-level sub-file of the source file from the inode of each disk sector directory entry, and then determining the grandchild filenames and the inode corresponding to each grandchild filename of the next-level grandchild file of the source file from the block information corresponding to the file directory of the next-level sub-file of the source file; and based on the inode corresponding to each grandchild filename, determining the number of grandchild files as the information digest of the next-level sub-file of the source file. Thus, the method for obtaining the information digest of the next-level sub-file of the source file does not consume excessive resources and is more concise.

[0105] This application provides another method for detecting data migration, the method comprising:

[0106] Step 501: The digest generation module determines the block information of the inode corresponding to each grandchild filename of the next-level sub-file of the source file;

[0107] Step 503: The digest generation module finds the size of each grandchild file from the block information of the inode corresponding to each grandchild file name, and uses it as the information digest of the next level sub-file of the source file;

[0108] Here, the sum of the sizes of each of the aforementioned grandchild files can be used as the information digest of the next-level sub-file of the source file.

[0109] Step 505: The digest generation module determines the file attribute value corresponding to the file attribute of the next-level sub-file of the target file as the information digest of the next-level sub-file of the target file based on the inode of each disk sector directory entry and the block information corresponding to the directory entry.

[0110] Step 507: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0111] Step 509: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0112] Step 511: The anomaly detection module determines the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0113] In this embodiment, the size of each grandchild file is found from the block information of the inode corresponding to each grandchild file name of the determined source file's subfile, and used as the information digest of the next-level subfile of the source file. In this way, the method of obtaining the size of the next-level subfile of the source file does not consume too many resources and is more concise.

[0114] This application provides another method for detecting data migration, the method comprising:

[0115] Step 601: The digest generation module obtains the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0116] Step 603: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0117] Step 605: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0118] Step 607: The summary generation module determines the source file information corresponding to the anomaly type to be detected from the information summary of the source file and the target file information corresponding to the anomaly type to be detected from the information summary of the target file;

[0119] Step 609: If the source file information and the target file information are consistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is normal.

[0120] Step 611: If the source file information and the target file information are inconsistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is an anomaly.

[0121] Step 613: The anomaly detection module determines whether the source file and the target file are the lowest-level files, and obtains the determination result;

[0122] Here, the lowest-level file refers to a file that cannot be further divided. For example, the lowest-level file could be a Word document.

[0123] In one possible implementation, the anomaly detection module determines whether the source file and the target file are the lowest level files, and obtains a judgment result. Alternatively, the anomaly detection module may determine whether the source file information and the target file information are consistent. If the source file information and the target file information are inconsistent, the module may determine whether the source file and the target file can be drilled down for analysis, and obtain a judgment result.

[0124] Step 615: The anomaly detection module determines whether to identify both the source file and the target file as vertex file packages in the data cube model based on the judgment result;

[0125] It is understandable that the anomaly detection module determines whether to classify both the source file and the target file as vertex file packages in the data cube model based on the judgment result. This can be because the anomaly detection module classifies both the source file and the target file as vertex file packages in the data cube model if it determines that neither the source file nor the target file is a bottom-level file; or it determines that neither the source file nor the target file can be further divided if it determines that neither the source file nor the target file is a bottom-level file, and therefore does not classify the source file and the target file as vertex file packages in the data cube model.

[0126] Step 617: If both the source file and the target file are determined to be vertex file packages in the data cube model, perform at least one drill-down analysis on each vertex file package to obtain the target drill-down analysis results;

[0127] Step 619: Based on the target drilling analysis results, determine the abnormal sub-file of the abnormal type to be detected.

[0128] In this embodiment, the anomaly detection module determines whether the source file and the target file are the lowest-level files, and determines whether to identify both the source file and the target file as vertex file packages in the data cube model based on the determination results. In this way, it is not necessary to blindly drill down into the vertex file packages, thus avoiding the resource waste caused by the segmentation of the lowest-level files.

[0129] This application provides another method for detecting data migration, the method comprising:

[0130] Step 701: The digest generation module obtains the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0131] Step 703: The digest generation module generates an information digest of the source file based on the information digest of the next-level sub-files of the source file;

[0132] Step 705: The digest generation module generates an information digest of the target file based on the information digest of the next-level sub-files of the target file;

[0133] Step 707: The anomaly detection module determines the source file information corresponding to the anomaly type to be detected from the information digest of the source file and the target file information corresponding to the anomaly type to be detected from the information digest of the target file;

[0134] Step 709: If the source file information and the target file information are consistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is normal.

[0135] Step 711: If the source file information and the target file information are inconsistent, the anomaly detection module determines that the detection result of the anomaly type to be detected in the data migration is an anomaly.

[0136] Step 713: The anomaly detection module identifies both the source file and the target file as vertex file packages in the data cube model;

[0137] Step 715: The deep-dimensional drill-down module performs the first drill-down analysis on the first dimension of the vertex file package corresponding to the source file and the vertex file package corresponding to the target file, respectively, to obtain the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file;

[0138] It is understandable that the vertex file packages corresponding to the source file and the target file can contain multiple identical dimensions; for example, they can contain dimension 1, dimension 2, and dimension 3 simultaneously.

[0139] In one implementation, the deep-dimensional drill-down module performs a first drill-down analysis on the first dimension of the vertex file package corresponding to the source file and the vertex file package corresponding to the target file, obtaining the information digest of the first drill-down file corresponding to the source file and the first drill-down file digest corresponding to the target file. This can be achieved by the deep-dimensional drill-down module generating a drill-down command to perform drill-down analysis on dimension 1, and transmitting the drill-down command to the digest generation modules of the source server and the target server through an information transmission module. The digest generation module responds to the drill-down command to generate and execute the digest command, thereby obtaining the information digest of the first drill-down file of the source file and the information digest of the first drill-down file of the target file.

[0140] Step 717: The deep-dimensional drilling module determines the information digest of the first drilling file corresponding to the source file and the information digest of the first drilling file corresponding to the target file as the target drilling analysis result;

[0141] Step 719: If the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file are inconsistent, update the first drill-down file corresponding to the source file and the first drill-down file corresponding to the target file to the vertex file package in the data cube model;

[0142] Step 721: Perform a second drill-down analysis on the second dimension of the vertex file package corresponding to the first drill-down file of the source file and the vertex file package corresponding to the second drill-down file of the target file, respectively, to obtain the information digest of the second drill-down file corresponding to the source file and the information digest of the second drill-down file corresponding to the target file;

[0143] In one possible implementation, the first dimension and the second dimension can be different dimensions; that is, the dimension of the second drill-down analysis is different from the dimension of the first drill-down analysis.

[0144] Step 723: Determine the information digest of the second drilling file corresponding to the source file and the information digest of the second drilling file corresponding to the target file as the target drilling analysis result;

[0145] Step 725: Based on the target drill-down analysis results, the anomaly detection module determines the anomaly sub-file of the anomaly type to be detected.

[0146] In this embodiment of the application, by performing drill-down analysis on the first dimension of the vertex file package corresponding to the source file and the vertex file package corresponding to the target file, the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file can be obtained. Therefore, the detection result of data migration can be accurately determined based on the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file.

[0147] In this embodiment, if the information digest of the first drill-down file of the source file and the information digest of the first drill-down file of the target file are consistent, the detection result of the data migration is output as normal. If the information digest of the first drill-down file of the source file and the information digest of the first drill-down file of the target file are inconsistent, the first drill-down files of the source file and the first drill-down files of the target file are identified as new vertex file packages in the data cube model. A second drill-down analysis is performed on each new vertex file package to obtain a second drill-down analysis result. Based on the second drill-down analysis result, the detection result of the data migration is confirmed. In this way, only abnormal files can be subjected to a second drill-down analysis, which can save resources.

[0148] This application proposes a batch detection method and system for migration file anomalies in a big data platform. It innovatively generalizes the generation of information digests to the file package level, extracting information digests from the corresponding blocks of inodes and directory entries in the Linux file system, starting from the vertex file package. The information digests of the source and target files are transmitted to the same server for comparison. When an anomaly is detected, deep drilling is performed only on the file package with the detected anomaly. Regarding anomaly detection and result output, the anomaly detection algorithm can be extended based on anomaly findings in migration operations, outputting the cause of the anomaly and a list of anomaly files.

[0149] The batch detection method for migration file anomalies in a big data platform provided in this application embodiment is divided into 5 modules: summary generation module, information transmission module, anomaly detection module, deep drilling-down module, and result output module.

[0150] The digest generation module is used to generate information digests of file packets from the source server and the target server respectively, based on the information digest extraction algorithm of file system inodes and blocks;

[0151] The information transmission module is used to transmit information between the summary generation module, the anomaly detection module, and the deep-dimensional drill-down module. On the one hand, it is used to transmit the source file package information summary to the target server. On the other hand, if an anomaly is detected in the anomaly detection module, the drill-down source file package instruction is transmitted through this module.

[0152] The anomaly detection module is used to verify the source file package information digest and the destination file package information digest;

[0153] The deep-dimensional drill-down module is designed to generate a drill-down command when the anomaly detection module detects a problem. This command is then transmitted to the source server and the target server to generate a message digest of the drill-down file package.

[0154] The result output module is used to output whether the migration file detection passed or failed; when a file abnormality is detected, it outputs the reason for the abnormality and a list of abnormal files.

[0155] Figure 2 This application provides a schematic diagram illustrating the implementation process of a batch detection method for abnormal migration files on a big data platform, as shown in the embodiments below. Figure 2 As shown, they can be divided into two types:

[0156] Scenario 1: No file anomaly exists. The detected information flow is: summary generation module 20 --> information transmission module 21 --> anomaly detection module 22 --> result output module 24.

[0157] Scenario 2: A file anomaly exists. The information flow for its detection circulates through the summary generation module 20, information transmission module 21, anomaly detection module 22, and deep-dimensional drill-down module 23, following the direction ①-->②-->③-->④-->⑤. During the loop, the result of normal file detection is transmitted to the result output module 24 along the direction of arrow ⑥; after the loop is completed, the result of anomaly file detection is transmitted to the result output module 24 along the direction of arrow ⑥.

[0158] In this embodiment, the deep-dimensional drill-down module 23 uses the data cube concept, which mainly drills down the file package from multiple dimensions based on the file storage structure. Figure 3 This is a schematic diagram of the deep-dimensional drilling process provided in the embodiments of this application, such as... Figure 3 As shown, after the first drill-down of the vertex file package 30, a first-level file package 31 is obtained. After the second drill-down of the first file package 31, a second file package 32 is obtained. Here, the first-level file package 31 can be a file package obtained by subdividing the vertex file package in a specific dimension, and the second file package 32 can be a file package obtained by subdividing the first file package 31 in other dimensions.

[0159] In this embodiment, the digest generation module 20 differs from traditional data-level, file-level, and MD5 algorithm-based proofreading methods. This solution generalizes the generation of information digests to the file-level and packet-level, and extracts information digests based on the inode number and the block corresponding to the directory entry in the Linux file system.

[0160] The algorithm for generating information summaries specifically includes the following steps:

[0161] Step 1: Input file directory (vertex file package collection);

[0162] Step 2: Locate the block information corresponding to the file directory from the inode of each disk sector directory entry;

[0163] Step 3: Search for the filename NAME(X) in the block information corresponding to the file directory. i ) and filename NAME(X i The corresponding inode number;

[0164] Step 4: Calculate the number of files N(X) based on the inode number of the filename. i ), and search for NAME(X) in the corresponding inode block. i File size SIZE(X) i Information such as )

[0165] Step 5: Based on the query results, generate information summaries as needed according to the types of anomalies to be detected in batches.

[0166] The summary generation module 20 triggering mechanism in this application embodiment includes two aspects:

[0167] (1) After the current batch of files on the big data platform has been migrated from the source server to the target server, the digest generation module 20 parses the digest instruction C (abstract) and executes the digest instruction C (abstract) to generate information digests of vertex file packages from the source server and the target server respectively.

[0168] Figure 4 A schematic diagram illustrating the transmission flow of the task start summary generation instruction provided in this application embodiment is shown below. Figure 4 As shown, after the task starts at 41, the summary generation module 20 parses and executes the summary instruction C 42 to generate a message summary 43.

[0169] (2) After the anomaly detection module 22 detects an abnormal file package, it passes the instruction C(abnormal) to the deep dimension drill-down module; the deep dimension drill-down module parses the file package to be drilled down and generates the drill-down instruction C(drilling); the drill-down instruction C(drilling) is passed to the source server and the target server respectively through the information transmission module 21; the source server and the target server parse the instruction C(drilling) through the digest generation module 20, generate the digest instruction C(abstract), and then execute the digest instruction C(abstract) to generate the information digest of the drill-down file package.

[0170] Figure 5 This is a schematic diagram of the digest generation instruction transmission process when an anomaly is detected, provided in an embodiment of this application. Figure 5 As shown, after detecting an anomaly 51, the summary generation module 20 generates an instruction C(abnormal) 52 and passes it to the deep-dimensional drill-down module 23. The deep-dimensional drill-down module 23 parses the drill-down file package and generates a drill-down instruction C(drilling) 53. The drill-down instruction C(drilling) 53 is transmitted to the source server and the target server summary generation module 20 respectively through the information transmission module 21. The summary generation module 20 parses the drill-down instruction C(drilling) 53, generates a summary instruction C(abstract) 54, and executes the summary instruction C(abstract) 54 to generate an information summary 55 of the drill-down file package.

[0171] In this embodiment, at the start of the task, information digests are generated for both vertex source file packages and vertex target file packages. After batch detection of vertex file packages is completed, deep drilling is performed only on abnormal file packages, and corresponding information digests are generated. Therefore, the computational cost of this scheme can be reduced from O(n×m) to at most O(n). Here, m is the number of underlying files, and n is the number of top-level file packages, where m is often much larger than n.

[0172] Information transmission module 21 is used to transmit information between the summary generation module 20, the anomaly detection module 22, and the deep-dimensional drilling module 23. This mainly includes the transmission of information summaries and drilling commands. The specific implementation process is as follows: Figure 6 As shown, it includes:

[0173] 1. In the summary generation module 20 of the source server and the target server, information summaries of the source file package and the target file package are generated respectively; then, they are transmitted to the target server through the information transmission module 21 to provide the data required for verification by the subsequent anomaly detection module 22.

[0174] 2. After the anomaly detection module 22 detects an anomaly in the migration file, the deep-dimensional drill-down module 23 generates a drill-down instruction for the specified file package; then, the information transmission module 21 transmits the drill-down instruction to the summary generation modules 20 of the source server and the target server respectively.

[0175] The anomaly detection module 22 can not only detect diverse anomaly types, but also be expanded according to business needs, exhibiting strong scalability. The expansion of anomaly type detection requires anomaly discovery in migration business scenarios, with data sourced from the information digests of the digest generation module 20.

[0176] The detection mechanism of the anomaly detection module 22 is to judge the source file information digest I(X) i ) and target document information digest I(Y) i Whether they are consistent, and the implementation process is as follows: Figure 7 As shown, the process includes:

[0177] Step S701: Anomaly detection module 22 determines I(X) i ) = I(Y i If yes, proceed to step S702; otherwise, proceed to step S703.

[0178] Step S702: Result output module 24;

[0179] Here, after the data stream is passed to the result output module 24, the result output module 24 outputs that the detection is normal.

[0180] Step S703: Deep-dimensional drilling module 23;

[0181] Understandably, after the data stream is passed to the deep-dimensional drill-down module 23, it is further processed in the deep-dimensional drill-down module 23.

[0182] In this embodiment, the deep-dimensional drill-down module 23 introduces the concept of a data cube, automatically drilling down only for abnormal files, which can improve the efficiency of anomaly detection. If the anomaly detection module 22 detects a problem, the deep-dimensional drill-down module generates a drill-down command. The algorithm flow implemented by this module in interaction with other modules is as follows: Figure 8 As shown, the specific steps are as follows:

[0183] Step S800: Start

[0184] Step S801: Obtain the information digest I(X) of the vertex source file package. i ), obtain the information digest I(Y) of the vertex target file package. i );

[0185] Here, the information digest I(X) of the vertex source file package can be obtained through the digest generation module 20. i) and the information digest I(Y) of the vertex target file package i ).

[0186] Step S802: Determine I(X) i ) = I(Y i If yes, proceed to step S807; if no, proceed to step S803.

[0187] Here, the anomaly detection module 22 can be used to determine I(X) i Is it equal to I(Y)? i ).

[0188] Step S803: Determine if the vertex source file package and vertex target file package can be drilled down. If not, proceed to step S808; if yes, proceed to step S804.

[0189] Here, the deep-dimensional drill-down module 23 can be used to determine whether the vertex source file package and vertex target file package can be drilled down.

[0190] Understandably, determining whether vertex source and vertex target file packages can be drilled down could involve determining whether the vertex source and vertex target file packages are already at the lowest level of the directory tree.

[0191] Step S804: Obtain the information digest I(X) of the drilling anomaly source file. j ), obtain the information digest I(Y) of the drilling anomaly target file. j );

[0192] Here, in step S804, the drilling command can be transmitted to the summary generation module 20 through the information transmission module 21, and the information summary I(X) of the drilling anomaly source file can be obtained through the summary generation module 20. j Information digest I(Y) of drilling anomaly target file j ).

[0193] Step S805: Determine I(X) j ) = I(Y j If yes, proceed to step S807; if no, proceed to step S806.

[0194] Here, I(X) can be compared via the anomaly detection module 22. j Is it equal to I(Y)? j ).

[0195] Step S806: Determine whether the drilling error source file and the drilling error target file can be drilled. If yes, proceed to step S804; otherwise, proceed to step S808.

[0196] Here, the deep-dimensional drill-down module 23 can be used to determine whether the abnormal source file and the abnormal target file can be drilled down.

[0197] Step S807: File is normal;

[0198] Here, "file normal" indicates that the data migration detection result is normal; step S807 can be to pass the data stream to the result output module 24, and the result output module 24 outputs "detection normal".

[0199] Step S808: File error;

[0200] Here, "file error" indicates that the data migration detection result is abnormal; step S808 may be to pass the data stream to the result output module 24, and the result output module 24 outputs "file error".

[0201] In this embodiment, the result output module 24 is used to output the results of batch detection of migration files. When an anomaly is detected in a migration file, the cause and a list of abnormal files can be automatically output. Based on the migration business scenario, the anomaly detection module 22 expands the types of anomalies detected, and accordingly expands the anomaly causes and abnormal file lists generated by the result output module 24. Taking the detection of file loss, file duplication, inconsistent file size, and file renaming failure as examples, the flow of the result output algorithm is as follows: Figure 9 As shown, it includes the following steps:

[0202] Step S900: Begin;

[0203] Step S901: Determine N(X) i )=N(Y i If not, proceed to step S902; if yes, proceed to step S905.

[0204] Here, step S901 can be the anomaly detection module 22 determining the number N(X) of vertex source file packages. i The number of vertex target file packages N(Y) i Whether they are consistent.

[0205] Step S902: Determine whether the abnormal file can be drilled down. If yes, proceed to step S903; otherwise, proceed to step S904.

[0206] Here, step S902 can determine whether the abnormal file package can be drilled down using the deep-dimensional drill-down module 23.

[0207] Step S903: Determine N(X) j )=N(Y j If yes, proceed to step S901; otherwise, proceed to step S902.

[0208] Here, step S903 can be the anomaly detection module 22 comparing the number N(X) of the drill-down source file packages. j The number of vertex target file packages N(Y) j Whether they are consistent.

[0209] Step S904: Determine N(X) l )>N(Y l If yes, output a list of missing files; otherwise, output a list of duplicate files.

[0210] Here, step S904 can be the anomaly detection module 22 determining the number N(X) of the underlying source files. l ) and the number of underlying object files N(Y) l The difference between N(X) and N(X) is as follows. l )>N(Y l If N(X) is not found, then output the file loss exception and the list of missing files, and end the detection; l ) <N(Y l If the output is "file duplicate exception" and "file duplicate list", then the detection will end.

[0211] Step S905: SIZE(X) i ) = SIZE(Y i If not, proceed to step S906; if yes, the output file is normal.

[0212] Here, step S905 can be the anomaly detection module 22 determining the size SIZE(X) of the source file. i ) and the size of the target file SIZE (Y i Check if they match. If they do not match, proceed to step S906; otherwise, the output file is normal and the detection ends.

[0213] Step S906: Determine if the abnormal file can be drilled down. If yes, proceed to step S907; otherwise, proceed to step S908.

[0214] Here, step S906 can be the deep-dimensional drill-down module 23 determining whether the abnormal file package can be drilled down. If yes, proceed to step S907; if no, proceed to step S908.

[0215] Step S907: Determine SIZE(X) j ) = SIZE(Y j If not, proceed to step S906; if yes, the output file is normal.

[0216] Here, step S907 can be the anomaly detection module 22 comparing the size SIZE(X) of the drill source file package. j ) and the size of the vertex target file package SIZE (Y jAre they consistent? If not, repeat step S906; otherwise, the output file is normal, and the test ends.

[0217] Step S908: Determine NAME(X) i ) = NAME(Y i If yes, output a list of files with inconsistent sizes; otherwise, output a list of files that failed to be renamed.

[0218] Here, step S908 can be the anomaly detection module 22 determining the name of the source file NAME(X). i ) and the name of the target file NAME(Y i Are they consistent? If they are consistent, output the file size inconsistency exception and the file size inconsistency list, and end the detection; otherwise, output the file renaming failure exception and the file renaming failure list, and end the detection.

[0219] Based on the foregoing embodiments, this application provides a data migration detection device, which includes various units and modules included in each unit, and can be implemented by a processor in a data migration detection system; in the implementation process, the processor can be a central processing unit (CPU), microprocessor unit (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0220] Figure 10 This is a schematic diagram of the composition of the data migration detection device provided in the embodiments of this application, as shown below. Figure 10 As shown, the device 1000 includes:

[0221] The acquisition module 1001 is used to acquire the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data; the target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file;

[0222] The first generation module 1002 is used to generate an information digest of the source file based on the information digest of the next-level sub-file of the source file;

[0223] The second generation module 1003 is used to generate an information digest of the target file based on the information digest of the next-level sub-file of the target file;

[0224] The first determining module 1004 is used to determine the detection result of data migration based on the information digest of the source file and the information digest of the target file.

[0225] In one embodiment, the first determining module 1004 is configured to determine source file information corresponding to the anomaly type to be detected from the information digest of the source file and target file information corresponding to the anomaly type to be detected from the information digest of the target file; if the source file information and the target file information are consistent, the detection result of the anomaly type to be detected in the data migration is determined to be normal; if the source file information and the target file information are inconsistent, the detection result of the anomaly type to be detected in the data migration is determined to be abnormal.

[0226] In one embodiment, the apparatus further includes: a second determining module, configured to determine both the source file and the target file as vertex file packages in a data cube model when the source file information and the target file information are inconsistent; perform at least one drill-down analysis on each vertex file package to obtain a target drill-down analysis result; and determine the abnormal sub-file of the anomaly type to be detected based on the target drill-down analysis result.

[0227] In one embodiment, the acquisition module 1001 is configured to determine, based on the inode of each disk sector directory entry of the next-level sub-file of the source file and the block information corresponding to the directory entry, the file attribute value corresponding to the file attribute of the next-level sub-file of the source file is a message digest of the next-level sub-file of the source file; and based on the inode of each disk sector directory entry of the next-level sub-file of the target file and the block information corresponding to the directory entry, determine the file attribute value corresponding to the file attribute of the next-level sub-file of the target file is a message digest of the next-level sub-file of the target file.

[0228] In one implementation, the file attributes of the next-level sub-files of the source file include the number of grandchild files; then, the acquisition module 1001 is used to acquire the file directory of the next-level sub-files of the source file; determine the block information corresponding to the file directory of the next-level sub-files of the source file from the inodes of the disk sector directory entries based on the file directory of the next-level sub-files of the source file; determine the grandchild file names and the inode corresponding to each grandchild file name from the block information corresponding to the file directory of the next-level sub-files of the source file; and determine the number of grandchild files as the information digest of the next-level sub-files of the source file based on the inode corresponding to each grandchild file name.

[0229] In one embodiment, the acquisition module 1001 is used to determine the block information of the inode corresponding to each grandchild file name of the next-level sub-file of the source file; and to find the size of each grandchild file from the block information of the inode corresponding to each grandchild file name, as the information digest of the next-level sub-file of the source file.

[0230] In one embodiment, the apparatus further includes: a bottom-level file determination module, configured to determine whether the source file and the target file are bottom-level files respectively when the source file information and the target file information are inconsistent, and obtain a determination result; and determine whether to determine both the source file and the target file as vertex file packages in the data cube model based on the determination result.

[0231] In one embodiment, the second determining module is configured to perform a first drill-down analysis on the first dimension of the vertex file package corresponding to the source file and the vertex file package corresponding to the target file, respectively, to obtain the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file; determine the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file as the target drill-down analysis result; if the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file are inconsistent, update the first drill-down file corresponding to the source file and the first drill-down file corresponding to the target file to the vertex file package in the data cube model; perform a second drill-down analysis on the second dimension of the vertex file package corresponding to the first drill-down file corresponding to the source file and the vertex file package corresponding to the second drill-down file corresponding to the target file, respectively, to obtain the information digest of the second drill-down file corresponding to the source file and the information digest of the second drill-down file corresponding to the target file; and determine the information digest of the second drill-down file corresponding to the source file and the information digest of the second drill-down file corresponding to the target file as the target drill-down analysis result.

[0232] Furthermore, in this embodiment, the functional modules can be integrated into a single processor, or each processor can exist as a separate physical entity, or two or more processors can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional module.

[0233] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0234] Generally speaking, the computer program instructions corresponding to a migration data detection method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a migration data detection method in the storage media are read or executed by a data migration system, any one of the migration data detection methods in the aforementioned embodiments is implemented.

[0235] Based on the same technical concept as the foregoing embodiments, see Figure 11 This illustration shows a structural diagram of a migration data detection system provided in an embodiment of this application, which may include: a memory 1101 and a processor 1102; wherein,

[0236] The memory 1101 is used to store computer programs and data;

[0237] The processor 1102 is used to execute the computer program stored in the memory to implement any of the migration data detection methods in the foregoing embodiments.

[0238] In practical applications, the aforementioned memory 1101 may be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 1102.

[0239] The processor 1102 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. Understandably, for different augmented reality cloud platforms, the electronic device used to implement the above processor functions can also be other types, and this application embodiment does not limit the specific type.

[0240] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0241] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0242] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0243] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0244] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0246] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the above-described embodiments. The above-described embodiments are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of this application without departing from the spirit and scope of the claims. All of these modifications are within the scope of protection of this application.

Claims

1. A method for detecting data migration, characterized in that, include: Obtain the information digests of the next-level sub-files of the source file and the target file respectively; the source file represents the file package of the migrated data. The target file represents the file package obtained after migrating the source file; the information digest of the sub-file includes the size of the sub-file, the storage location of the sub-file, and the number of next-level files of the sub-file; Generate an information digest of the source file based on the information digest of the next-level sub-file of the source file; Generate an information digest of the target file based on the information digest of the next-level sub-file of the target file; Determine the source file information corresponding to the anomaly type to be detected from the information digest of the source file, and determine the target file information corresponding to the anomaly type to be detected from the information digest of the target file; In the case where the source file information and the target file information are inconsistent, the detection result of the anomaly type to be detected in the data migration is determined as an anomaly. Both the source file and the target file are identified as vertex file packages in the data cube model; Perform at least one drill-down analysis on each vertex file package to obtain the target drill-down analysis results; Based on the target drilling analysis results, anomaly subfiles of the anomaly type to be detected are determined.

2. The method according to claim 1, characterized in that, The method further includes: If the source file information and the target file information are consistent, the detection result of the anomaly type to be detected in the data migration is determined to be normal.

3. The method according to claim 1, characterized in that, The step of obtaining the information digests of the next-level sub-files of the source file and the target file respectively includes: Based on the inode of each disk sector directory entry in the next-level sub-file of the source file and the block information corresponding to the directory entry, the file attribute value corresponding to the file attribute of the next-level sub-file of the source file is determined to be the information digest of the next-level sub-file of the source file; based on the inode of each disk sector directory entry in the next-level sub-file of the target file and the block information corresponding to the directory entry, the file attribute value corresponding to the file attribute of the next-level sub-file of the target file is determined to be the information digest of the next-level sub-file of the target file.

4. The method according to claim 3, characterized in that, The file attributes of the next-level sub-files of the source file include the number of grandchild files; therefore, Based on the inode and block information corresponding to the directory entries of each disk sector in the next-level sub-file of the source file, the file attribute value corresponding to the file attribute of the next-level sub-file of the source file is determined to be an information digest of the next-level sub-file of the source file, including: Obtain the file directory of the next level sub-file of the source file; Based on the file directory of the next-level sub-file of the source file, the block information corresponding to the file directory of the next-level sub-file of the source file is determined from the inode of each disk sector directory entry; Determine the names of the grandchild files included in the grandchild files of the next-level sub-files of the source file and the inode corresponding to each grandchild file name from the block information corresponding to the file directory of the next-level sub-files of the source file; Based on the inode corresponding to each of the grandchild filenames, the number of grandchild files is determined as the information digest of the next-level sub-file of the source file.

5. The method according to claim 4, characterized in that, The file attributes of the next-level sub-file of the source file also include the sub-file size. The determination of the file attribute value corresponding to the file attributes of the next-level sub-file of the source file based on the inode and block information corresponding to each disk sector directory entry of the next-level sub-file of the source file, including: Determine the block information of the inode corresponding to each grandchild filename of the next level subfile of the source file; The size of each grandchild file is determined from the block information of the inode corresponding to each grandchild file name, and used as the information digest of the next-level sub-file of the source file.

6. The method according to claim 1, characterized in that, The method further includes: If the source file information and the target file information are inconsistent, determine whether the source file and the target file are the lowest level files respectively, and obtain the determination result; Based on the judgment result, determine whether to identify both the source file and the target file as vertex file packages in the data cube model.

7. The method according to claim 1, characterized in that, The step of performing at least one drill-down analysis on each vertex file package to obtain the target drill-down analysis results includes: A first drill-down analysis is performed on the first dimension of the vertex file package corresponding to the source file and the vertex file package corresponding to the target file, respectively, to obtain the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file; The information digest of the first drilling file corresponding to the source file and the information digest of the first drilling file corresponding to the target file are determined as the target drilling analysis result.

8. The method according to claim 7, characterized in that, The method further includes: If the information digest of the first drill-down file corresponding to the source file and the information digest of the first drill-down file corresponding to the target file are inconsistent, update the first drill-down file corresponding to the source file and the first drill-down file corresponding to the target file to the vertex file package in the data cube model; A second drill-down analysis is performed on the second dimension of the vertex file package corresponding to the first drill-down file of the source file and the vertex file package corresponding to the second drill-down file of the target file, respectively, to obtain the information digest of the second drill-down file corresponding to the source file and the information digest of the second drill-down file corresponding to the target file; The information digest of the second drilling file corresponding to the source file and the information digest of the second drilling file corresponding to the target file are determined as the target drilling analysis result.

9. A data migration detection device, characterized in that, The data migration detection device is used to perform the steps in the data migration detection method according to any one of claims 1 to 8.

10. A data migration detection system, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the steps of the data migration detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the data migration detection method according to any one of claims 1 to 8.

Images