Data verification methods, devices, computer equipment and storage media
By extracting and fusing features from the target source data, a matching verification strategy is selected, which solves the problem of low data verification efficiency after big data migration and achieves efficient data verification and differentiated verification.
Patent Information
- Application Number
- CN202611544949.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-30
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies have low data verification efficiency after big data migration and cannot effectively improve the efficiency of data verification.
By acquiring target source data and target migration data, feature extraction and feature fusion are performed to obtain fused data features. Based on the fused data features, a verification strategy scheme matching the target source data is selected for differential verification.
It improves data verification efficiency in large-scale data migration scenarios, reduces unnecessary verification operations, and enhances the integrity and reliability of data verification.
Smart Images

Figure CN123068408A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, and particularly to a data verification method, apparatus, computer equipment, and storage medium. Background Technology
[0002] In big data migration scenarios, it is often necessary to validate data before and after the migration to ensure that no errors occur during the process. For example, in the financial sector, transaction records and settlement data require rigorous validation after migration to prevent financial errors; similarly, in the medical field, electronic medical records and imaging data need to be validated after system upgrades or data centralization to prevent data errors from affecting clinical decisions and medical safety. Currently, traditional full-data validation algorithms are often used when processing large-scale data, resulting in low data validation efficiency. Therefore, improving data validation efficiency after big data migration has become a pressing technical problem. Summary of the Invention
[0003] This application provides a data verification method, apparatus, computer equipment, and storage medium to improve data verification efficiency after big data migration is completed.
[0004] Firstly, a data verification method is provided, including: Obtain target source data and target migration data; Feature extraction is performed on the target source data to obtain metadata features and topology graph features; The metadata features and the topology graph features are fused to obtain fused data features; Based on the characteristics of the fused data, the preset verification strategy schemes are filtered to obtain the target verification scheme; Data verification is performed on the target source data and the target migration data according to the target verification scheme and the fused data features.
[0005] Secondly, a data verification device is provided, comprising: The data acquisition module is used to acquire target source data and target migration data; The feature extraction module is used to extract features from the target source data to obtain metadata features and topology graph features; The feature fusion module is used to fuse the metadata features and the topology map features to obtain fused data features. The scheme filtering module is used to filter preset verification strategy schemes based on the characteristics of the fused data to obtain the target verification scheme; The data verification module is used to perform data verification on the target source data and the target migration data according to the target verification scheme and the fused data features.
[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data verification method.
[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described data verification method.
[0008] In the aforementioned data verification method, apparatus, computer equipment, and storage medium, the target source data and target migration data are acquired, and features are extracted from the target source data to obtain metadata features and topology graph features. This allows the target source data to be characterized from two dimensions: data attributes and data relationships. Furthermore, by fusing the metadata features and topology graph features, fused data features are obtained, enabling the fused data features to comprehensively reflect the data characteristics and topological relationship characteristics of the target source data. Subsequently, based on the fused data features, a preset verification strategy is selected to obtain a target verification scheme that matches the data characteristics of the target source data, reducing unnecessary verification operations caused by using fixed verification schemes or full verification methods. Finally, data verification is performed on the target source data and target migration data based on the target verification scheme and fused data features, achieving differentiated verification based on data characteristics. This reduces the data verification overhead in large-scale data migration scenarios and improves data verification efficiency. Attached Figure Description
[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is a schematic diagram of an application environment for a data verification method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a data verification method in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S102; Figure 4 yes Figure 2A schematic diagram of a specific implementation method for step S103; Figure 5 yes Figure 4 A schematic diagram of a specific implementation of step S104; Figure 6 This is a schematic diagram of the structure of a data verification device in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device according to one embodiment of this application. Detailed Implementation
[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0012] Based on this, embodiments of this application provide a data verification method and apparatus, a computer device and a storage medium, which aim to improve data verification efficiency after big data migration is completed.
[0013] The data verification method, apparatus, computer equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the data verification method in this application is described.
[0014] The data verification method provided in this application embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain target source data and target migration data from the client; extract features from the target source data to obtain metadata features and topology graph features; fuse the metadata features and topology graph features to obtain fused data features; filter a preset verification strategy based on the fused data features to obtain a target verification scheme; perform data verification on the target source data and target migration data based on the target verification scheme and fused data features, and feed the verification results back to the client. In this application, feature extraction characterizes the target source data from two dimensions: data attributes and data relationships; furthermore, feature fusion of metadata features and topology graph features comprehensively reflects the data characteristics and topological relationship characteristics of the target source data; thus, a target verification scheme matching the data characteristics of the target source data is obtained, reducing unnecessary verification operations caused by using a fixed verification scheme or full verification method; finally, data verification is performed to achieve differentiated verification based on data characteristics, thereby reducing data verification overhead and improving data verification efficiency in large-scale data migration scenarios.
[0015] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description uses specific embodiments to illustrate this application.
[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the data verification method provided in this application embodiment includes the following steps: Step S101: Obtain target source data and target migration data; Step S102: Extract features from the target source data to obtain metadata features and topology graph features; Step S103: Perform feature fusion on metadata features and topology graph features to obtain fused data features; Step S104: Based on the characteristics of the fused data, the preset verification strategy scheme is filtered to obtain the target verification scheme; Step S105: Perform data verification on the target source data and target migration data according to the target verification scheme and the characteristics of the fused data.
[0017] This application embodiment acquires target source data and target migration data, and extracts features from the target source data to obtain metadata features and topology graph features, thereby representing the target source data from two dimensions: data attributes and data relationships. Furthermore, by fusing the metadata features and topology graph features, fused data features are obtained, enabling the fused data features to comprehensively reflect the data characteristics and topological relationship characteristics of the target source data. Then, based on the fused data features, a preset verification strategy is selected to obtain a target verification scheme that matches the data characteristics of the target source data, reducing unnecessary verification operations caused by using fixed verification schemes or full-scale verification methods. Finally, data verification is performed on the target source data and target migration data based on the target verification scheme and fused data features, achieving differentiated verification based on data characteristics, thereby reducing data verification overhead in large-scale data migration scenarios and improving data verification efficiency.
[0018] For step S101, the target source data refers to the data to be migrated before the data migration operation is executed; the target migration data refers to the data stored in the target data system after the data migration operation is performed on the target source data. It should be noted that the target source data is a big data category, such as TB or PB level data.
[0019] For example, in a fintech scenario, when a bank migrates transaction data from its original database to a new database due to an upgrade of its core business system, the transaction log data stored in the original database before the migration is used as the target source data, such as transaction account, transaction amount, transaction time, transaction type, and transaction status; correspondingly, the transaction log data stored in the new database after the execution data migration is used as the target migration data.
[0020] For example, in a healthcare scenario, when a medical institution migrates electronic medical record data from its original medical information system to a new medical data platform, the electronic medical record data stored in the original medical information system before the migration is used as the target source data, such as patient basic information, diagnostic information, test results, medical orders, and treatment records; correspondingly, the electronic medical record data stored in the new medical data platform after the data migration is used as the target migration data.
[0021] In some embodiments, step S102 may include, but is not limited to: Step S201: Obtain metadata information of the target source data to obtain metadata features; Step S202: Obtain the nesting relationship of each sub-data in the target source data to obtain the logical structure pattern; Step S203: Map each sub-data to a graph node, and extract topological graph features based on the logical structure pattern and graph nodes to obtain topological graph features.
[0022] This application embodiment obtains metadata information of the target source data to acquire metadata features, thereby characterizing the target source data from the data attribute dimension. Furthermore, by acquiring the nesting relationships between sub-data in the target source data, a logical structure pattern is obtained, and each sub-data is mapped to a graph node. A topology graph is constructed based on the logical structure pattern and graph nodes to obtain topology graph features, thus transforming the nesting relationships between sub-data into a topology structure that reflects the logical relationships between data. Therefore, by characterizing the target source data from the two dimensions of data attributes and logical relationships between data through metadata features and topology graph features respectively, the data characteristics of the target source data can be reflected more comprehensively and accurately, thereby improving data verification efficiency in large-scale data migration scenarios.
[0023] For step S201, metadata information describes the basic attributes of the target source data, such as data size, file organization, and data volume. Metadata features are characteristics obtained by extracting and statistically analyzing the above metadata information, used to characterize the overall data size and file attributes of the target source data. Specifically, metadata features include the number of files in the target source data, the file size of each file, the total file size, file type, the number of records in each file, the total number of records, the data block size, and the number of data blocks.
[0024] For example, in a financial transaction data migration scenario, the target source data includes multiple files used to store transaction logs. Assuming the target source data comprises 100 files, of which 80 are CSV files and 20 are JSON files, with each file ranging in size from 500MB to 2GB, the total size of the 100 files is 120GB, the total number of record rows is 800 million, and each file is stored in 128MB data blocks, then the following information is extracted: number of files (100), number of CSV files (80), number of JSON files (20), file size range (500MB to 2GB), total file size (120GB), total number of record rows (800 million), and data block size (128MB). This extracted information is then used as metadata features.
[0025] For step S202, the logical structure pattern is a structural pattern used to characterize the internal data organization hierarchy of the target source data, determined based on the nesting relationships between the sub-data. For example, if the target source data includes multiple transaction data files, each transaction data file includes multiple user data, each user data includes multiple account data, and each account data includes multiple transaction records, then the nesting relationship of "transaction data file → user data → account data → transaction record" is determined, and the corresponding logical structure pattern is obtained based on this nesting relationship. As another example, for a JSON-type transaction file, if the root object includes multiple user objects (User), the user objects nest account objects (Account), and the Account objects further nest multiple transaction objects (Transaction), then the logical structure pattern of "root object → user → account → transaction" is obtained.
[0026] For step S203, based on the logical structure pattern of the target source data, the sub-data at different levels are mapped to corresponding graph nodes, and edges are established between the corresponding graph nodes according to the nesting relationship between different sub-data, thereby converting the original hierarchical nesting structure of the target source data into a topological graph structure.
[0027] For example, in the aforementioned financial transaction data, a transaction data file File_A includes a user User_A, which in turn includes an account Account_A. Account_A contains transaction records Transaction_A and Transaction_B. File_A, User_A, Account_A, Transaction_A, and Transaction_B are mapped to nodes N1, N2, N3, N4, and N5, respectively. Based on the nesting relationship of "File_A contains User_A," a connection is established between N1 and N2. Based on the nesting relationship of "User_A contains Account_A," a connection is established between N2 and N3. Based on the nesting relationship of "Account_A contains Transaction_A and Transaction_B," connections are established between N3 and N4, and between N3 and N5, respectively. This yields the corresponding topology graph, which is then stored in an array as a topology graph feature.
[0028] In some embodiments, step S103 may include, but is not limited to: Step S301: Extract latent vectors from metadata features to obtain metadata latent vectors; Step S302: Extract latent vectors from the features of the topological graph to obtain the latent vectors of the topological graph; Step S303: Align and fuse the metadata latent vector and the topology graph latent vector based on the preset vector fusion network to obtain fused data features.
[0029] This application embodiment extracts latent vectors from metadata features and topology graph features respectively, obtaining metadata latent vectors and topology graph latent vectors. This converts metadata features representing data scale and file attributes and topology graph features representing data logical structure into vector representations that can be fused. Furthermore, based on a preset vector fusion network, the metadata latent vectors and topology graph latent vectors are aligned and fused, which can coordinate the representation differences between different types of features and establish feature associations between data scale, file attributes and data logical structure, thereby avoiding the problem that a single type of feature is insufficient to represent the target source data.
[0030] In step S301, latent vector extraction refers to inputting metadata features into a multilayer perceptron and mapping the original metadata features to a latent vector space of a preset dimension. For example, metadata features include the number of files, the size of each file, the total file size, file type, the number of rows, and the number of data blocks. The number of files or file size alone cannot fully reflect the data scale characteristics of the target source data. Therefore, the multilayer perceptron is used to extract the combined relationships between these metadata features to obtain hidden features that characterize the data scale, data distribution, and file composition. For example, when the total file size of the target source data is the same, but the number of files and the size of individual files differ, the hidden features characterize the data organization characteristics of the target source data as either "a few large files" or "a large number of small files."
[0031] Specifically, a multilayer perceptron includes an input layer, at least one hidden layer, and an output layer. The number of nodes in the input layer is determined by the dimension of the metadata features; the hidden layer includes a fully connected layer and an activation function, such as the ReLU activation function, to extract the nonlinear correlation between different metadata features; the output layer maps the output of the hidden layers to a vector space of a preset dimension, thereby obtaining the metadata latent vector.
[0032] In step S302, latent vector extraction of topological graph features is performed using a pre-trained graph convolutional neural network (GCN). The latent vectors of the topological graph represent hidden features such as local connectivity, hierarchical relationships, and overall structural complexity among the sub-data within the target source data. For example, for the topological structure "file → user → account → transaction record," GCN aggregates information from adjacent graph nodes, ensuring that the hidden representation of an account node includes not only information about the account node itself but also information about the user nodes and transaction record nodes connected to it. After multi-layer graph convolution, the structural relationships between nodes at different levels are further extracted, thereby representing the structural characteristics of the target source data, such as nesting depth, branching degree, and relationship complexity.
[0033] Specifically, GCN comprises an input layer, one or more graph convolutional layers, and an output layer. The input layer receives the initial node features of each graph node and an adjacency matrix representing the connectivity relationships between the graph nodes. The graph convolutional layers determine the neighboring nodes of each graph node based on the adjacency matrix, aggregate the features of the current node and its neighbors, and perform feature transformation using a weight matrix and activation function to update the representation of each graph node. After passing through one or more graph convolutional layers, graph-level pooling operations such as average pooling, summative pooling, or max pooling are used to aggregate the hidden representations of multiple nodes into a vector representation of the entire topological graph, resulting in the topological graph hidden vector.
[0034] Step S303, alignment and fusion, refers to mapping the metadata latent vector and the topology graph latent vector to the same dimension and the same feature space, and then fusing them in that feature space. Since they have different vector dimensions and feature distributions, a linear mapping layer is first used to map the metadata latent vector and the topology graph latent vector separately, so that they are converted to the same preset dimension, thereby achieving dimension alignment.
[0035] Furthermore, the vector fusion network includes two feature mapping branches and a fusion layer. Each feature mapping branch includes a fully connected layer to map the metadata latent vectors and the topology graph latent vectors to the same latent vector space. The fusion layer concatenates the aligned latent vectors and uses the fully connected layer to perform feature interaction and dimensionality reduction on the concatenated result, obtaining fused data features. For example, the aligned metadata latent vectors and the topology graph latent vectors are concatenated, and the concatenated vector is then input into the fully connected layer. The fully connected layer learns the combination relationship between data scale characteristics and data structure characteristics, ultimately outputting fused data features of a preset dimension.
[0036] In some embodiments, step S104 may include, but is not limited to: Step S401: Input the fused data features into the preset strategy classification network for calculation to obtain the probability distribution vector; Step S402: Determine the matching probability of each verification strategy scheme based on the probability distribution vector; Step S403: Obtain the maximum matching probability from the matching probabilities, and determine the target verification scheme from the verification strategy scheme based on the maximum matching probability.
[0037] This application embodiment calculates a probability distribution vector by inputting the fused data features into a preset strategy classification network. This vector is then used to convert the comprehensive features of the target source data, such as data size, file composition, and logical structure, into the matching degree corresponding to each verification strategy scheme. Furthermore, the matching probability of each verification strategy scheme is determined based on the probability distribution vector, and the verification strategy scheme with the highest matching probability is selected as the target verification scheme. This allows for adaptive selection of a matching verification method based on the data characteristics of the target source data, avoiding unnecessary verification overhead caused by using a fixed verification scheme, thereby improving data verification efficiency.
[0038] In step S401, the policy classification network is a pre-trained neural network used to classify the data type of the target source data based on its data characteristics. The policy classification network employs a multilayer perceptron classification network, comprising an input layer, at least one hidden layer, and an output layer. The input layer receives fused data features; the hidden layer includes fully connected layers and activation functions to learn the nonlinear mapping relationship between features such as data size, file composition, and topology in the fused data features and different data types; the number of nodes in the output layer is consistent with the preset number of data types, for example, set to 3 output nodes, and the output results of each output node are normalized using the Softmax function to obtain a probability distribution vector. The probability distribution vector includes the probability that the target source data belongs to one of three data types: a single massive file, a deeply nested file with many branching nodes, and a small-to-medium-sized regular file. For example, a probability distribution vector of [0.75, 0.15, 0.10] indicates a high degree of matching between the target source data and the data type of a single massive file.
[0039] For steps S402 to S403, in this embodiment, a correspondence between data types and verification strategy schemes is pre-established. Specifically, a single massive file corresponds to the sampling Bloom filter verification scheme, data with deep nesting levels and many node branches corresponds to the Merkle tree verification scheme, and small-to-medium-sized regular files correspond to the full parallel hash verification scheme. For example, when the probability distribution vector is [0.75, 0.15, 0.10], the matching probabilities of the sampling Bloom filter verification scheme, the Merkle tree verification scheme, and the full parallel hash verification scheme are determined to be 0.75, 0.15, and 0.10, respectively.
[0040] In some embodiments, step S105 may include, but is not limited to: Step S501: If the target verification scheme is a Merkle tree verification scheme, perform data verification on the target source data and target migration data according to the Merkle tree verification scheme to obtain the first verification result; Step S502: If the target verification scheme is a sampling Bloom filter verification scheme, perform data verification on the target source data and target migration data according to the sampling Bloom filter verification scheme to obtain the first verification result; Step S503: If the target verification scheme is a full parallel hash verification scheme, perform data verification on the target source data and the target migration data according to the full parallel hash verification scheme to obtain the first verification result; Step S504: Perform semantic data verification on the target source data and target migration data based on the first verification result and the fused data features.
[0041] This application embodiment selects a Merkle tree verification scheme, a sampling Bloom filter verification scheme, or a full parallel hash verification scheme to verify the target source data and target migration data based on the determined target verification scheme, and obtains a first verification result. This allows for the adoption of a verification method that is suitable for different data scales and logical structures, reducing unnecessary data reading and computation overhead caused by using a fixed full verification method. Furthermore, semantic data verification is performed on the target source data and target migration data based on the first verification result and fused data features. This enables the identification of semantic anomalies at the data content level by combining the data characteristics of the target source data with the basic verification, compensating for the insufficient ability of the basic verification to identify semantic anomalies. This improves both the efficiency and integrity of big data migration verification.
[0042] For steps S501 to S504, for a single massive file, the data volume is large and the data organization structure is relatively simple. If hash verification is performed directly on all data line by line, it is necessary to read and calculate massive amounts of data, resulting in significant computational overhead. The sampling Bloom filter verification scheme selects a portion of the data for sampling and uses a Bloom filter to determine the set membership relationship. It can quickly verify the migrated data without comparing all data line by line, thereby reducing the amount of data read and computation. Therefore, it is more suitable for massive and structurally simple data.
[0043] For data with deep nesting levels and numerous node branches, such as data with multi-table cascading relationships, the hierarchical and relational relationships between different sub-data are quite complex. Merkle trees calculate the hash values of data blocks or sub-data layer by layer and aggregate them upwards to form a tree-like hash structure, so that the hierarchical organization of the data corresponds to the verification structure. When there are differences between the source data and the migrated data, the root node compares the hash values layer by layer to locate the branch where the difference occurs, without having to repeatedly compare all related data. This reduces the data comparison scope under complex relational structures and improves the efficiency of verification and difference location.
[0044] For small to medium-sized regular files, since the amount of data to be verified is relatively limited, the computational overhead of hashing all the data is within a controllable range. Therefore, a full-scale parallel hash verification scheme is adopted. The data is divided into multiple data blocks, and multiple computing tasks are used to calculate the hash value of each data block in parallel. Then, the hash values corresponding to the source data and the migrated data are compared. In this way, while ensuring that all data participates in the verification, parallel computing is used to shorten the overall verification time, making it suitable for small to medium-sized regular files.
[0045] In some embodiments, step S504 may include, but is not limited to: Step S601: Extract fingerprints from the features of the fused data to obtain the semantic fingerprint of the source data; Step S602: Extract fingerprints from the target migration data to obtain semantic fingerprints of the migration data; Step S603: Calculate the similarity between the semantic fingerprint of the source data and the semantic fingerprint of the migration data, and determine the second verification result based on the similarity. Step S604: If both the first verification result and the second verification result indicate that the verification has passed, then the target verification result of the target source data and the target migration data is determined to be successful. Step S605: If the first verification result or the second verification result indicates that the verification failed, then the target verification result of the target source data and the target migration data is determined to be a verification failure.
[0046] This application embodiment extracts fingerprints from the fused data features and the target migrated data to obtain semantic fingerprints of the source data and the migrated data, thereby converting the data before and after migration into fingerprint representations that can be used to characterize the semantic properties of the data. Further, a second verification result is determined by calculating the similarity between the semantic fingerprints of the source data and the migrated data, realizing quantitative verification of the semantic consistency of the data before and after migration. Then, the final target verification result is determined by combining the first verification result and the second verification result. Verification is considered successful only when both the basic data verification and the semantic data verification pass, and verification is considered failed when either verification fails. This realizes the joint judgment of basic data verification and semantic data verification, reduces the possibility of missing semantic anomalies due to using only the basic verification method, and improves the completeness and reliability of migration data verification.
[0047] For steps S601 to S602, fingerprint extraction involves converting the fused data features into a fixed-length vector characterizing the overall characteristics of the target source data. The source data semantic fingerprint is a vector representation generated based on the fused data features, characterizing the comprehensive features of the target source data in terms of data scale, file composition, and logical structure. Specifically, a pre-defined fingerprint extraction network is used to extract fingerprints from the fused data features. This fingerprint extraction network includes one or more fully connected layers and an output layer. The fused data features are input into the fully connected layer, where feature mapping is performed using a weight matrix. Activation functions such as ReLU are used to extract the combination relationships between different features. The output layer then maps the processed features into a fixed-dimensional fingerprint vector. Finally, the fingerprint vector is normalized to obtain the source data semantic fingerprint.
[0048] The steps for extracting fingerprints from target migration data are the same as those for obtaining semantic fingerprints from source data through a series of steps such as feature extraction and feature fusion, and will not be elaborated here.
[0049] Step S603: Calculate the similarity between the source data semantic fingerprint and the migration data semantic fingerprint using cosine similarity. Further, a similarity threshold is preset, for example, 0.95, and the calculated similarity is compared with the threshold. When the similarity is greater than or equal to the threshold, the second verification result is determined to be successful; when the similarity is less than the threshold, the second verification result is determined to be unsuccessful.
[0050] For steps S604 and S605, the first verification result is a basic verification result obtained through a Merkle tree verification scheme, a sampled Bloom filter verification scheme, or a full parallel hash verification scheme. Since different basic verification schemes focus on data integrity or data content consistency—for example, the sampled Bloom filter verification scheme only verifies a portion of the sampled data—even if the first verification result is passed, there may still be cases where unsampled data exhibits anomalies, leading to missed detections. Other basic verification methods may also fail to reflect the differences in overall features between the data before and after migration due to limitations in verification granularity or the verification object. Therefore, semantic fingerprint similarity verification is further introduced based on the first verification, using the second verification result to supplement the detection of any potential missed detections.
[0051] Specifically, if both the first and second verification results indicate that the verification passed, it means that the target source data and the target migration data not only passed the corresponding basic data verification, but also that their semantic fingerprints have a sufficiently high degree of similarity. In this case, the target verification result is determined to be successful. If the first verification result is successful but the second verification result is unsuccessful, it indicates that there may be omissions in the basic data verification, and there are still differences between the target source data and the target migration data that require further verification. Therefore, the target verification result is determined to be unsuccessful. If the first verification result itself is unsuccessful, then the target verification result is also determined to be unsuccessful. Thus, by jointly determining the basic data verification and semantic data verification, the possibility of omissions caused by a single verification method is reduced.
[0052] In some embodiments, data verification may also include, but is not limited to: Step S701: If the target verification result indicates that the verification failed, determine the data difference location of the target migration data based on the first verification result; Step S702: Correct the target migration data based on the data difference location and the target source data.
[0053] In this embodiment of the application, when the target verification result indicates a verification failure, the location of the data difference in the target migration data is determined based on the first verification result, thereby locating the data anomaly discovered during verification to the corresponding data area; furthermore, the target migration data is corrected based on the data difference location and the target source data, so that only the data with differences is corrected, without the need to re-migrate or process the entire target migration data, thereby reducing the scope of data correction and processing overhead after verification failure, and improving the correction efficiency of abnormal migration data.
[0054] For step S701, the data difference location is the position of data in the target migration data that is inconsistent with the target source data. Since the first verification result is obtained by the Merkle tree verification scheme, the sampling Bloom filter verification scheme, or the full parallel hash verification scheme, the data difference location is determined according to the verification result corresponding to the actual verification scheme used.
[0055] Specifically, when using the Merkle tree verification scheme, based on the Merkle tree nodes with inconsistent hash values in the target source data and target migration data, the root node is used to search down layer by layer for child nodes with different hash values until the leaf node where the difference occurs is located. The location of the data block with the difference is then determined based on the mapping relationship between the leaf node and the data block.
[0056] When using a sampling Bloom filter validation scheme, the corresponding positions of the sampled data that fail the Bloom filter member validation are determined as the data difference positions.
[0057] When using a full-parallel hash verification scheme, the hash values of corresponding data blocks in the target source data and the target migration data are compared, and the positions corresponding to the data blocks with inconsistent hash values are determined as the data difference positions. For example, if a file is divided into 100 data blocks, and the hash value of the source data corresponding to the 35th data block is inconsistent with the hash value of the migration data, then the position of the 35th data block is determined as the data difference position.
[0058] It should be noted that when the first verification result indicates that the verification passed, while the second verification result indicates that the verification failed, it means that the basic data verification may not have directly located the specific anomaly. In this case, based on the data segmentation granularity used in the first verification, the target source data and target migration data are further segmented and compared to narrow down the range of data with semantic differences, and the location of the abnormal data block or abnormal record obtained from the further comparison is determined as the data difference location.
[0059] For step S702, after determining the location of the data difference, the source data corresponding to the location of the data difference is obtained from the target source data, and the data at the corresponding location in the target migration data is corrected using the source data. Data correction includes operations such as data replacement and data supplementation.
[0060] For example, when a difference is found between the 35th data block of the target migration data and the target source data, the corresponding 35th data block is read from the target source data and migrated back to the target data system to replace the abnormal data block in the target migration data. When the data difference manifests as missing records, the missing records are retrieved from the target source data and added to the corresponding positions. When the data difference manifests as duplicate records, the duplicate records in the target migration data are deleted. This limits the scope of data correction to the detected data difference locations without requiring the complete migration of the target source data, reducing data reading, network transmission, and write operations, and improving the efficiency of correction after anomalies occur in the migration data.
[0061] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0062] In one embodiment, a data verification device is provided, which corresponds one-to-one with the data verification methods described in the above embodiments. For example... Figure 6 As shown, the data verification device includes: Data acquisition module 801 is used to acquire target source data and target migration data; The feature extraction module 802 is used to extract features from the target source data to obtain metadata features and topology graph features; The feature fusion module 803 is used to fuse metadata features and topology map features to obtain fused data features. The scheme selection module 804 is used to select preset verification strategy schemes based on the characteristics of the fused data to obtain the target verification scheme; The data verification module 805 is used to verify the target source data and target migration data according to the target verification scheme and the characteristics of the fused data.
[0063] Detailed descriptions of each functional module are as follows: In one embodiment, the feature extraction module 802 is specifically used for: Obtain metadata information from the target source data to obtain metadata characteristics; Obtain the nesting relationship of each sub-data in the target source data to get the logical structure pattern; Each sub-data is mapped to a graph node, and topological graph features are extracted based on the logical structure pattern and graph nodes to obtain topological graph features.
[0064] Specifically, the feature extraction module 802 can be used to implement the above steps S201 to S203, which will not be described in detail here.
[0065] In one embodiment, the feature fusion module 803 is specifically used for: Latent vector extraction is performed on metadata features to obtain metadata latent vectors; Latent vectors are extracted from the features of the topological graph to obtain the latent vectors of the topological graph; Based on a pre-defined vector fusion network, the latent vectors of metadata and the latent vectors of the topology graph are aligned and fused to obtain fused data features.
[0066] Specifically, the feature fusion module 803 can be used to implement the above steps S301 to S303, which will not be described in detail here.
[0067] In one embodiment, the scheme selection module 804 is specifically used for: The fused data features are input into a pre-defined strategy classification network for calculation to obtain a probability distribution vector; Based on the probability distribution vector, determine the matching probability of each verification strategy scheme; Obtain the maximum matching probability from the matching probabilities, and determine the target verification scheme from the verification strategy scheme based on the maximum matching probability.
[0068] Specifically, the scheme selection module 804 can be used to implement the above steps S401 to S303, which will not be described in detail here.
[0069] In one embodiment, the data verification module 805 is specifically used for: If the target verification scheme is the Merkle tree verification scheme, the target source data and target migration data are verified according to the Merkle tree verification scheme to obtain the first verification result; If the target verification scheme is a sampling Bloom filter verification scheme, the target source data and target migration data are verified according to the sampling Bloom filter verification scheme to obtain the first verification result; If the target verification scheme is a full parallel hash verification scheme, the target source data and target migration data are verified according to the full parallel hash verification scheme to obtain the first verification result; Semantic data verification is performed on the target source data and target migration data based on the first verification result and the characteristics of the fused data.
[0070] Specifically, the data verification module 805 can be used to implement the above steps S501 to S504, which will not be described in detail here.
[0071] In one embodiment, the data verification module 805 is specifically used for: Fingerprint extraction is performed on the features of the fused data to obtain the semantic fingerprint of the source data; Fingerprint extraction is performed on the target migration data to obtain the semantic fingerprint of the migration data; Calculate the similarity between the semantic fingerprint of the source data and the semantic fingerprint of the migrated data, and determine the second verification result based on the similarity. If both the first and second verification results indicate that the verification passed, then the target verification result for the target source data and the target migration data is determined to be successful. If the first or second verification result indicates a verification failure, then the target verification result for both the target source data and the target migration data is determined to be a verification failure.
[0072] Specifically, the data verification module 805 can be used to implement the above steps S601 to S605, which will not be described in detail here.
[0073] In one embodiment, the data verification module 805 is specifically used for: If the target verification result indicates that the verification failed, the location of the data difference in the target migration data is determined based on the first verification result. The target migration data is corrected based on the location of the data differences and the target source data.
[0074] Specifically, the data verification module 805 can be used to implement the above steps S701 to S702, which will not be described in detail here.
[0075] This application provides a data verification device that acquires target source data and target migration data, extracts features from the target source data to obtain metadata features and topology graph features, thereby characterizing the target source data from two dimensions: data attributes and data relationships. Furthermore, by fusing the metadata features and topology graph features, a fused data feature is obtained, which comprehensively reflects the data characteristics and topological relationship characteristics of the target source data. Then, based on the fused data feature, a preset verification strategy is selected to obtain a target verification scheme that matches the data characteristics of the target source data, reducing unnecessary verification operations caused by using a fixed verification scheme or full verification method. Finally, data verification is performed on the target source data and target migration data according to the target verification scheme and the fused data feature, achieving differentiated verification based on data characteristics, thereby reducing data verification overhead in large-scale data migration scenarios and improving data verification efficiency.
[0076] For specific limitations regarding the data verification device, please refer to the limitations on the data verification method above, which will not be repeated here. Each module in the aforementioned data verification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0077] Please see Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the data verification method of the embodiments of this application, including: Obtain target source data and target migration data; Feature extraction is performed on the target source data to obtain metadata features and topology graph features; Metadata features and topology graph features are fused to obtain fused data features; Based on the characteristics of the fused data, the preset verification strategy schemes are filtered to obtain the target verification scheme; Data verification is performed on the target source data and target migration data based on the target verification scheme and the characteristics of the fused data.
[0078] The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0079] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain target source data and target migration data; Feature extraction is performed on the target source data to obtain metadata features and topology graph features; Metadata features and topology graph features are fused to obtain fused data features; Based on the characteristics of the fused data, the preset verification strategy schemes are filtered to obtain the target verification scheme; Data verification is performed on the target source data and target migration data based on the target verification scheme and the characteristics of the fused data.
[0080] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0081] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0083] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0084] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A data verification method, characterized in that, The method includes: Obtain target source data and target migration data; Feature extraction is performed on the target source data to obtain metadata features and topology graph features; The metadata features and the topology graph features are fused to obtain fused data features; Based on the characteristics of the fused data, the preset verification strategy schemes are filtered to obtain the target verification scheme; Data verification is performed on the target source data and the target migration data according to the target verification scheme and the fused data features.
2. The method according to claim 1, characterized in that, The step of extracting features from the target source data to obtain metadata features and topology graph features includes: Obtain the metadata information of the target source data to obtain the metadata features; Obtain the nesting relationship of each sub-data in the target source data to obtain the logical structure pattern; Each of the sub-data is mapped to a graph node, and topological graph features are extracted based on the logical structure pattern and the graph nodes to obtain the topological graph features.
3. The method according to claim 1, characterized in that, The feature fusion of the metadata features and the topology graph features to obtain fused data features includes: Latent vector extraction is performed on the metadata features to obtain the metadata latent vector; Latent vector extraction is performed on the features of the topological graph to obtain the latent vectors of the topological graph; The metadata latent vectors and the topology graph latent vectors are aligned and fused based on a preset vector fusion network to obtain the fused data features.
4. The method according to claim 1, characterized in that, The step of filtering the preset verification strategy scheme based on the fused data features to obtain the target verification scheme includes: The fused data features are input into a preset strategy classification network for calculation to obtain a probability distribution vector; Based on the probability distribution vector, determine the matching probability of each of the verification strategy schemes; Obtain the maximum matching probability from the matching probabilities, and determine the target verification scheme from the verification strategy scheme based on the maximum matching probability.
5. The method according to any one of claims 1-4, characterized in that, The step of performing data verification on the target source data and the target migration data according to the target verification scheme and the fused data features includes: If the target verification scheme is a Merkle tree verification scheme, the target source data and the target migration data are verified according to the Merkle tree verification scheme to obtain a first verification result; If the target verification scheme is a sampling Bloom filter verification scheme, the target source data and the target migration data are verified according to the sampling Bloom filter verification scheme to obtain a first verification result; If the target verification scheme is a full parallel hash verification scheme, the target source data and the target migration data are verified according to the full parallel hash verification scheme to obtain the first verification result; Semantic data verification is performed on the target source data and the target migration data based on the first verification result and the fused data features.
6. The method according to claim 5, characterized in that, The step of performing semantic data verification on the target source data and the target migration data based on the first verification result and the fused data features includes: Fingerprint extraction is performed on the features of the fused data to obtain the semantic fingerprint of the source data; Fingerprint extraction is performed on the target migration data to obtain the semantic fingerprint of the migration data; Calculate the similarity between the source data semantic fingerprint and the migration data semantic fingerprint, and determine the second verification result based on the similarity. If both the first verification result and the second verification result indicate that the verification passed, then the target verification result of the target source data and the target migration data is determined to be successful. If the first verification result or the second verification result indicates a verification failure, then the target verification result for the target source data and the target migration data is determined to be a verification failure.
7. The method according to claim 6, characterized in that, The method further includes: If the target verification result indicates a verification failure, the data difference location of the target migration data is determined based on the first verification result; The target migration data is corrected based on the location of the data difference and the target source data.
8. A data verification device, characterized in that, The device includes: The data acquisition module is used to acquire target source data and target migration data; The feature extraction module is used to extract features from the target source data to obtain metadata features and topology graph features; The feature fusion module is used to fuse the metadata features and the topology map features to obtain fused data features. The scheme filtering module is used to filter preset verification strategy schemes based on the characteristics of the fused data to obtain the target verification scheme; The data verification module is used to perform data verification on the target source data and the target migration data according to the target verification scheme and the fused data features.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data verification method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data verification method as described in any one of claims 1 to 7.
