A heterogeneous data standardization conversion method, system, medium and program product

Abstract

A heterogeneous data standardization conversion method, system, medium and program product, in the method, the version identifier is extracted according to the mark information of the source data to be converted, and the source data to be converted is stored to the conversion buffer; the version mapping chain is established according to the version identifier of the source data to be converted; the source data to be converted is restored step by step based on the version mapping chain, and the historical version data sequence is obtained; the reference version data with the highest structural similarity with the source data to be converted is screened out from the historical version data sequence, and the historical conversion rule corresponding to the reference version data is determined; the difference items between the source data to be converted and the reference version data are extracted, and the historical conversion rule is incrementally updated according to the difference items, and the target conversion rule is obtained; the target conversion rule is used for conversion processing of the source data to be converted, and the target data is obtained. The application improves the efficiency and success rate of heterogeneous data conversion.

Description

Technical Field

[0001] This application belongs to the field of data conversion, and in particular relates to a method, system, medium and program product for the standardization conversion of heterogeneous data. Background Technology

[0002] With the advent of the big data era, heterogeneous data generated and accumulated in various information systems has experienced explosive growth. This heterogeneous data includes various forms such as structured data, semi-structured data, and unstructured data. Due to significant differences in data formats, data structures, and semantic expressions from different data sources, it is difficult to effectively integrate and process the data in a unified manner, which severely restricts the full utilization and value mining of data resources.

[0003] In related technologies, a common approach is to predefine mapping rules from source data format to target data format, and then convert the source data into a standard format according to these rules during the data conversion process. This method enables basic conversion and integration of heterogeneous data, making unified data management and application possible.

[0004] However, since the format and structure of heterogeneous data sources often change, mapping rules need to be updated and maintained frequently. This not only increases the workload of system maintenance, but also easily leads to data conversion errors or failures due to untimely rule updates, reducing the efficiency and success rate of heterogeneous data conversion. Summary of the Invention

[0005] This application provides a heterogeneous data standardization conversion method, system, medium, and program product to improve the efficiency and success rate of heterogeneous data conversion.

[0006] In the first aspect, this application provides a heterogeneous data standardization conversion method, which extracts version identifiers based on the tagging information of the source data to be converted and stores the source data to be converted in a conversion buffer; Establish a version mapping chain based on the version identifier of the source data to be converted. The version mapping chain includes all version change records from the historical version to the current version of the source data to be converted. Based on the version mapping chain, the source data to be transformed is restored step by step to obtain the historical version data sequence. Each historical version data in the historical version data sequence corresponds to a transformation rule. Select the reference version data with the highest similarity to the source data structure to be converted from the historical version data sequence, and determine the historical conversion rules corresponding to the reference version data; Extract the differences between the source data to be converted and the reference version data, and incrementally update the historical conversion rules based on the differences to obtain the target conversion rules; The source data to be transformed is processed using target transformation rules to obtain the target data.

[0007] By employing the above technical solution, and extracting the version identifier of the source data to be converted and establishing a version mapping chain, all changes to the data from historical versions to the current version are recorded. Based on the complete version mapping chain, the source data to be converted can be restored level by level, generating a sequence containing data from each historical version, ensuring that each historical version accurately matches the corresponding conversion rule. By selecting the reference version data with the highest structural similarity to the source data to be converted from the historical version data sequence, the most suitable historical conversion rule for the current conversion task can be found. The historical conversion rules are updated by analyzing the differences between the source data to be converted and the reference version data, allowing the conversion rules to adapt to changes in the data structure. Using the updated target conversion rule for conversion processing can achieve accurate data conversion while maintaining data structure consistency, improving the accuracy and reliability of heterogeneous data standardization conversion.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, a version mapping chain is established based on the version identifier of the source data to be converted, specifically including: Obtain the version inheritance attribute information of the source data to be converted. The version inheritance attribute information includes version identifier, data structure, and version dependency relationship. A multi-branch version inheritance tree is constructed based on version inheritance attribute information, and version nodes with the same parent version are constructed as sibling nodes at the same level; Calculate the structural and data differences between adjacent nodes in the version inheritance tree, and generate version difference feature vectors; Clustering analysis is performed on version nodes based on version difference feature vectors, and version nodes with similarity exceeding a preset threshold are merged. Based on the dependencies between the merged version nodes, a version mapping chain in the form of a directed acyclic graph is constructed.

[0009] By employing the above technical solution, and by acquiring version inheritance attribute information and constructing a multi-branch version inheritance tree, version nodes with the same parent version are grouped into sibling nodes at the same level, thus achieving a hierarchical expression of version relationships. The structural and data differences between adjacent nodes in the version inheritance tree are calculated to generate version difference feature vectors, which quantify the differences between versions. Based on these feature vectors, version nodes are clustered, and highly similar version nodes are merged, reducing redundancy. The merged version nodes are then constructed into a directed acyclic graph (DAG) version mapping chain according to their dependencies, making the version mapping relationship clearer and more concise, reducing the complexity of version management, and improving the efficiency of version mapping.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the source data to be converted is restored step by step based on the version mapping chain, specifically including: Starting from the current version node in the version mapping chain, traverse backwards along the version mapping chain to the initial version node; Extract data structure change records and data content conversion rules between adjacent version nodes; Based on the data structure change records, the current version of the data structure is restored to obtain the data structure of the previous version. The data content in the restored data structure is transformed using the applied data content transformation rules; The transformed data is used as the new current version data, and the step of restoring the structure of the current version data based on the data structure change record is repeated until the initial version is reached. By employing the aforementioned technical solution, and traversing the version mapping chain backwards from the current version node to the initial version node, the data structure change records and data content conversion rules between adjacent version nodes are accurately extracted. Based on the data structure change records, the current version data is structurally restored, and the data content in the restored data structure is converted using the data content conversion rules, ensuring the integrity and consistency of the data during the restoration process. By executing structure restoration and content conversion step-by-step until the initial version is reached, data version traceability is achieved, making the data conversion process verifiable and enhancing its reliability.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, after transforming the source data to be transformed using the target transformation rules to obtain the target data, the method further includes: A two-way verification channel is constructed based on the target data. The two-way verification channel includes a forward conversion link and a reverse restoration link. In the two-way verification channel, the target data is iterated and verified in multiple rounds. The data loss in each round of iteration is calculated and compared with the preset loss. When the data loss exceeds the preset loss level, the key features causing the data loss are extracted, and the target transformation rules are adjusted based on the key features. The adjusted target transformation rules are reapplied to the source data to be transformed to obtain the updated target data. Perform semantic integrity analysis on the updated target data, establish a semantic relationship graph between fields, and complete the data based on the semantic relationship graph.

[0012] By adopting the above technical solution and constructing a bidirectional verification channel containing both forward conversion and reverse restoration links, the target data is iteratively verified and the data loss is calculated in multiple rounds within the channel, thus achieving quality monitoring of the conversion results. When the data loss exceeds the preset loss level, the key features causing the data loss are extracted, and the target conversion rules are adjusted accordingly, improving the adaptability of the conversion rules. Semantic integrity analysis is performed on the updated target data, and a semantic relationship graph between fields is established. Data completion is performed based on the semantic relationship graph, ensuring the integrity and semantic consistency of the converted data and improving the accuracy and usability of the data conversion.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, key features causing data loss are extracted, and the target transformation rules are adjusted based on these key features, specifically including: Analyze the fields where statistical loss occurs and the type of loss to generate a list of loss features; Classify and summarize the feature items in the loss feature list to determine the feature type that appears most frequently. Based on the feature type, retrieve the corresponding conversion rule from the preset rule base; Analyze the parameter configuration in the transformation rules to identify the parameter items that cause data loss; Modify the value range or calculation method of parameter items according to the preset parameter adjustment strategy; Update the modified conversion rules to the preset rule library and mark the rule version number.

[0014] By adopting the above technical solution, and through statistical analysis of the location and type of data loss to generate a list of loss features, combined with the classification and summarization of feature items to determine high-frequency feature types, the system can accurately locate the specific link leading to data loss. Based on the feature type, the system retrieves corresponding transformation rules from a preset rule base, analyzes the rule parameter configuration, and identifies the parameter items causing the loss, thus establishing a clear correspondence between data loss and specific rule parameters. By modifying the parameter value range or calculation method through preset parameter adjustment strategies, and updating the modified rules to the rule base while marking the version number, the system can optimize transformation rules in a targeted manner, avoiding the recurrence of the same type of data loss. Data-level feedback guides rule improvement, enhancing the accuracy and robustness of transformation rules, achieving adaptive optimization of transformation rules, and reducing the information loss rate during data transformation.

[0015] In conjunction with some embodiments of the first aspect, in some embodiments, after data completion based on the semantic association graph, the method further includes: The completed target data is subjected to quality checks based on a preset set of data quality rules, and a verification report containing information on the location of quality problems is generated. The quality checks include data format consistency, reasonable value range, and compliance with business rules. Based on the quality problem location information in the verification report, extract the corresponding conversion rule identifier and completion operation record, and establish a link between the quality problem and the data processing process; Based on the analysis of the relationship chain, the link in which quality problems occur is analyzed, and the data items with quality problems are corrected in a targeted manner in combination with business rules; Synchronously record the triggering conditions, correction methods, and correction results of the correction to obtain the correction record; Optimize the corresponding conversion rules and completion strategies based on the correction records.

[0016] By adopting the above technical solution, quality checks are performed on the completed target data, including format consistency, value rationality, and compliance with business rules. A verification report containing problem location information is generated, enabling the system to comprehensively assess data quality. By extracting the conversion rule identifiers and completion operation records corresponding to quality problems, a link between quality problems and the processing process is established, allowing the system to trace and locate the specific stage where the problem occurred. In conjunction with business rules, targeted corrections are performed on data with quality problems, and the triggering conditions, methods, and results of the corrections are recorded, making the correction process traceable. Based on the correction records, conversion rules and completion strategies are optimized, establishing a closed-loop processing mechanism for quality problem discovery, location, correction, and optimization. Through experience accumulation, data processing capabilities are continuously improved, enhancing the accuracy and reliability of data conversion results.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, the corresponding conversion rules and completion strategies are optimized based on the correction records, specifically including: Analyze the execution frequency of various correction operations in the correction records to identify high-frequency correction scenarios; Extract data features and correction methods in high-frequency correction scenarios to obtain correction patterns; Adjust the parameter configuration and execution conditions of the corresponding conversion rules according to the correction rules; The data mapping relationships in the completion strategy are supplemented and improved based on the correction rules.

[0018] By employing the aforementioned technical solution, high-frequency correction scenarios are identified through statistical analysis of the execution frequency of various correction operations in the correction records. Data characteristics and correction methods within these scenarios are extracted to reveal correction patterns, allowing the system to grasp the most common problem patterns in data processing. Based on these patterns, the parameter configuration and execution conditions of the transformation rules are adjusted, and the data mapping relationships in the completion strategy are supplemented and improved. This enables the system to transform past processing experience into specific rule optimization solutions. This statistical analysis-based rule optimization method guides rule improvement by mining the regularities in historical correction records, avoiding the uncertainty caused by blind adjustments, improving the accuracy and efficiency of rule optimization, achieving continuous evolution of transformation rules and completion strategies, and enhancing the overall intelligence level of the data processing system.

[0019] Secondly, embodiments of this application provide a heterogeneous data standardization conversion system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and one or more processors call the computer instructions to cause the system to perform the method described in the first aspect and any possible implementation thereof.

[0020] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a system, cause the system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Fourthly, embodiments of this application provide a computer program product, characterized in that, when the computer program product is run on a system, it causes the system to execute the method described in any possible implementation of the first aspect.

[0022] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. This application provides a method for heterogeneous data standardization transformation. By extracting version identifiers from the source data to be transformed and establishing a version mapping chain, it records all changes to the data from historical versions to the current version. Based on the complete version mapping chain, the source data to be transformed can be restored step by step, generating a sequence containing data from each historical version, ensuring that each historical version accurately matches the corresponding transformation rule. By selecting the reference version data with the highest structural similarity to the source data to be transformed from the historical version data sequence, the most suitable historical transformation rule for the current transformation task can be found. By analyzing the differences between the source data to be transformed and the reference version data, the historical transformation rules are updated, allowing the transformation rules to adapt to changes in the data structure. Using the updated target transformation rule for transformation processing can achieve accurate data transformation while maintaining data structure consistency, improving the accuracy and reliability of heterogeneous data standardization transformation.

[0023] 2. This application provides a heterogeneous data standardization transformation method. By constructing a bidirectional verification channel containing a forward transformation link and a reverse restoration link, the target data is iteratively verified and the data loss degree is calculated in the channel, thus achieving quality monitoring of the transformation results. When the data loss degree exceeds the preset loss degree, the key features causing the data loss are extracted and the target transformation rules are adjusted accordingly, improving the adaptability of the transformation rules. Semantic integrity analysis is performed on the updated target data and a semantic relationship graph between fields is established. Data completion is performed based on the semantic relationship graph, ensuring the integrity and semantic consistency of the transformed data and improving the accuracy and usability of the data transformation.

[0024] 3. This application provides a heterogeneous data standardization transformation method. It performs quality checks on the completed target data, including format consistency, value rationality, and business rule compliance, generating a verification report containing problem location information, enabling the system to comprehensively assess data quality. By extracting the transformation rule identifiers and completion operation records corresponding to quality problems, a link between quality problems and the processing process is established, allowing the system to trace and locate the specific steps in which the problem occurred. Combined with business rules, targeted corrections are performed on data with quality problems, and the triggering conditions, methods, and results of the corrections are recorded, making the correction process traceable. Based on the correction records, transformation rules and completion strategies are optimized, establishing a closed-loop processing mechanism for quality problem discovery, location, correction, and optimization. Through experience accumulation, data processing capabilities are continuously improved, enhancing the accuracy and reliability of data transformation results. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a heterogeneous data standardization conversion method in an embodiment of this application.

[0026] Figure 2This is a flowchart illustrating a data quality assurance method based on two-way verification in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the physical device structure of a heterogeneous data standardization conversion system provided in an embodiment of this application. Detailed Implementation

[0028] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0030] The following example is used in conjunction with Figure 1 The following describes a heterogeneous data standardization conversion method in an embodiment of this application: Please see Figure 1 This is a flowchart illustrating a heterogeneous data standardization conversion method in an embodiment of this application.

[0031] S101. Extract the version identifier based on the tagging information of the source data to be converted, and store the source data to be converted in the conversion buffer; In this step, the system first needs to extract version identification information from the source data to be converted. Typically, different versions of data will contain version numbers, release dates, and other information that distinguishes them in their metadata or tagging information. The system can obtain the corresponding version identifier by parsing the source data's metadata or tagging information. After extracting the version identifier, the system temporarily stores the source data to be converted in a conversion buffer so that subsequent conversion operations can directly access and use this data.

[0032] In practical implementation, the system can use methods such as regular expression matching and keyword extraction to quickly locate and extract version identification information from the metadata or tagging information of the source data. For the transformation buffer, the system can choose a space in memory or use external storage such as temporary files or database tables. The purpose of the transformation buffer is to provide an efficient and easily accessible temporary storage location for the data to be transformed during the data transformation process, avoiding the impact of frequent I / O operations on transformation performance.

[0033] It's worth noting that if the source data is not formatted correctly or the metadata is incomplete, the system may be unable to accurately extract the version identifier. To address this issue, the system can design a default value for the version identifier or use heuristic rules for inference. For example, the creation time or last modification time of the data can be used as a substitute for the version identifier. Additionally, the system can introduce a manual intervention mechanism, allowing users to manually specify or correct the version identifier of the source data to ensure the accuracy of subsequent conversions.

[0034] S102. Establish a version mapping chain based on the version identifier of the source data to be converted; The system establishes a version mapping chain based on the version identifier of the source data to be converted. The version mapping chain includes all version change records from the historical version to the current version of the source data to be converted. Specifically, it obtains the version inheritance attribute information of the source data to be converted, which includes the version identifier, data structure, and version dependency relationship. A multi-branch version inheritance tree is constructed based on version inheritance attribute information, and version nodes with the same parent version are constructed as sibling nodes at the same level; Calculate the structural and data differences between adjacent nodes in the version inheritance tree, and generate version difference feature vectors; Clustering analysis is performed on version nodes based on version difference feature vectors, and version nodes with similarity exceeding a preset threshold are merged. Based on the dependencies between the merged version nodes, a version mapping chain in the form of a directed acyclic graph is constructed.

[0035] In this step, the system needs to establish a complete version mapping chain based on the version identifier of the source data. The version mapping chain records the evolution of the source data from its initial version to the current version, including the differences and changes between each version. By analyzing and extracting the version mapping chain, the system can clearly understand the version evolution of the source data, providing an important basis for subsequent data restoration and conversion.

[0036] To construct a version mapping chain, the system first needs to obtain the version inheritance attribute information of the source data, including version identifiers, data structure definitions, and dependencies between versions. This information can typically be obtained from a version management system, data dictionary, or configuration file. After obtaining complete version inheritance attribute information, the system can construct a multi-branch version inheritance tree based on the dependencies between versions. In the inheritance tree, each node represents a data version, and the connections between parent and child nodes represent the inheritance relationship between versions. Nodes at the same level are identified as sibling versions, indicating that they share a common parent version.

[0037] After constructing the version inheritance tree, the system needs to further calculate the differences between different versions. By comparing the data structure definitions and data content of adjacent version nodes, the system can obtain two metrics: structural difference and data difference. Structural difference measures the differences between the data models of two versions, such as the addition, deletion, and renaming of fields. Data difference measures the differences between data records in two versions, such as the insertion, updating, and deletion of records. The system can choose an appropriate algorithm (such as the Diff algorithm) to calculate the difference and generate a feature vector of differences between each pair of adjacent versions.

[0038] After obtaining the difference feature vectors, the system can use clustering algorithms (such as K-means) to perform cluster analysis on the version nodes. The goal of clustering is to group version nodes with similar difference features into the same class and merge them. A similarity threshold can be set during clustering; when the similarity between two version nodes exceeds this threshold, their differences are considered negligible, and they are merged into the same version. Cluster analysis can simplify the structure of the version inheritance tree and remove redundant version nodes.

[0039] Finally, based on the merged version nodes and their dependencies, the system constructs a version mapping chain in the form of a directed acyclic graph (DAG). In this DAG, each node represents a data version, and the directed edges between nodes represent the evolution direction between versions. The version mapping chain clearly depicts the evolution trajectory of the source data from the initial version to the current version, providing an important basis for data reconstruction in subsequent steps.

[0040] It's important to note that if the source data has too many versions or the differences between versions are too large, building the version mapping chain can be time-consuming. To address this, the system can introduce an incremental update mechanism, focusing only on the differences between the most recent few versions, rather than starting from the initial version. Additionally, the system can employ a distributed computing framework to split the version mapping chain building task across multiple nodes for parallel processing, thereby improving building efficiency.

[0041] S103. Based on the version mapping chain, restore the source data to be converted level by level to obtain the historical version data sequence; The system reconstructs the source data to be transformed step by step based on the version mapping chain, obtaining a sequence of historical version data. Each historical version data in the sequence corresponds to a transformation rule. Specifically, starting from the current version node in the version mapping chain, the system traverses backwards along the version mapping chain to the initial version node. Extract data structure change records and data content conversion rules between adjacent version nodes; Based on the data structure change records, the current version of the data structure is restored to obtain the data structure of the previous version. The data content in the restored data structure is transformed using the applied data content transformation rules; The transformed data is used as the new current version data, and the step of restoring the structure of the current version data based on the data structure change record is repeated until the initial version is reached.

[0042] In this step, the system needs to restore the source data to be transformed step by step based on the constructed version mapping chain, ultimately obtaining a historical version data sequence. This historical version data sequence contains all intermediate version data tracing back from the current version to the initial version, with each historical version corresponding to an independent transformation rule. Through this restoration process, the system can obtain complete snapshots of the source data at different historical periods, preparing for subsequent difference comparisons and transformation rule generation.

[0043] In practice, the system can start from the current version node in the version mapping chain and traverse the chain backwards along the directed edges until it reaches the initial version node. During the traversal, the system needs to extract the data structure change records and data content conversion rules between each pair of adjacent version nodes. The data structure change records describe the differences in the data model between the two versions, such as the addition, deletion, and renaming of fields. The data content conversion rules describe how to convert the data content of the new version into the form of the old version, such as data type conversion and the filling of default values.

[0044] After obtaining the data structure change records, the system can reverse-engineer the data structure of the current version based on this information to obtain the data structure of the previous version. The restoration process requires undoing all structural changes made by the current version relative to the previous version, such as removing newly added fields, restoring deleted fields, and reverting renamed field names. After the structure restoration, the system then uses the corresponding data content conversion rules to convert the data records of the current version back to the form of the previous version, for example, removing the content of newly added fields in the new version, or restoring the field content to the values ​​of the old version.

[0045] After the conversion is complete, the system obtains a complete snapshot of the previous version's data. This snapshot serves as the starting point for the next round of restoration. The system treats it as the new current version data and continues to traverse the version mapping chain for step-by-step restoration. Finally, when the system reaches the initial version node, the historical version data sequence is completed. The first element of this sequence is the initial version of the source data, the last element is the current source data version to be converted, and the elements in between are the various historical versions restored in sequence.

[0046] During data restoration, the system may face the risk of data loss or corruption, especially when the transformation rules are incorrect or the data structure definition is flawed. To improve the robustness of restoration, the system can back up the data before each transformation and set a restore point. This way, even if an error occurs during restoration, it can quickly roll back to the most recent restore point without having to start from scratch. Simultaneously, the system can also introduce a data verification mechanism to compare the data before and after each transformation, ensuring that the restored data is consistent with historical snapshots, thereby guaranteeing the correctness of the restoration.

[0047] S104. Select the reference version data with the highest similarity to the source data structure to be converted from the historical version data sequence, and determine the historical conversion rules corresponding to the reference version data. In this step, the system needs to select a version from the historical version data sequence that is most similar to the source data structure to be converted, as the reference version data. This reference version data will play an important role in the subsequent conversion process, and its corresponding conversion rules can be used as the initial conversion rules for the source data to be converted, greatly reducing the workload of generating conversion rules.

[0048] To select the most similar reference version data, the system can employ a structural similarity comparison method. First, the system needs to extract the structural features of each version data in both the source data to be converted and the historical version data sequence. Structural features typically include elements such as entities, attributes, and relationships in the data model, as well as their organization. After extracting the structural features, the system can select an appropriate similarity metric to calculate the structural similarity between the source data to be converted and each historical version data. Common similarity metrics include edit distance, Jaccard coefficient, and cosine similarity.

[0049] After calculating the structural similarity of all versions, the system selects the historical version with the highest similarity as the reference version. Generally, the historical version with the highest similarity is structurally closest to the source data to be converted, resulting in the lowest conversion cost. After determining the reference version, the system also needs to obtain the corresponding historical conversion rules. These historical conversion rules record how to convert the reference version data to its predecessor, providing detailed descriptions of both data structure and content conversion. These historical conversion rules can serve as the basis for converting the source data; by making local adjustments and optimizations, a target conversion rule that conforms to the characteristics of the source data can be obtained.

[0050] It's important to note that simple structural similarity comparisons can have limitations, failing to account for semantic differences in the data. For example, two versions of data may appear to have identical structures, but the meanings and value ranges of their fields could be completely different. To improve the accuracy of reference version selection, the system can incorporate data content distribution features in addition to structural similarity. By comparing the value range distribution of various fields across different versions, the system can identify the version that is most semantically similar. Furthermore, if the historical version data sequence is very large, exhaustively comparing the structural similarity of all versions can be time-consuming. In such cases, the system can introduce hierarchical indexes or hashing techniques based on data structure features to narrow down the comparison range through rapid location and filtering, thereby improving the efficiency of reference version data selection.

[0051] S105. Extract the differences between the source data to be converted and the reference version data, and incrementally update the historical conversion rules based on the differences to obtain the target conversion rules. In this step, the system needs to generate a target transformation rule that perfectly matches the characteristics of the source data to be transformed, based on the selected reference version data and its corresponding historical transformation rules. Since there are often some differences between the reference version data and the source data to be transformed, some modifications and improvements need to be made to the historical transformation rules.

[0052] First, the system needs to compare the structure and content of the source data to be transformed with the reference version data to extract the differences between them. These differences can be structural, such as the addition or deletion of fields in the source data, or changes in the data type of existing fields. They can also be content-level, such as changes in the value range of certain fields or differences in the distribution characteristics of the data records. Extracting these differences can be done using a method similar to the Diff algorithm, which involves comparing the structural definitions and data content of the two versions of data one by one to identify all the differences.

[0053] After obtaining the differences, the system needs to analyze the potential impact of each difference on the data transformation and adjust the historical transformation rules accordingly. For structural differences, the system needs to add or remove corresponding transformation steps in the transformation rules to accommodate the new fields or data types. For example, if a new field is added to the source data to be transformed, the system needs to add corresponding assignment statements to the transformation rules to generate appropriate values ​​for this new field. Conversely, if a field is deleted from the source data to be transformed, the system needs to remove all steps related to that field from the transformation rules.

[0054] For differences at the data content level, the system mainly needs to adjust the data mapping and transformation functions in the transformation rules. For example, if the value range of a field changes, the system needs to modify the corresponding data mapping in the transformation rules to map the new value range to the old value range. Furthermore, if the data distribution characteristics change (such as the emergence of new outliers), the system may need to introduce new data cleaning and normalization functions to ensure the quality of the transformed data.

[0055] Based on the results of the comprehensive difference analysis, the system incrementally updates the historical transformation rules, ultimately generating a target transformation rule that matches the characteristics of the source data to be transformed. Incremental updates refer to making partial modifications, deletions, and additions based on the original rules, rather than starting from scratch. Using incremental updates maximizes the reuse of existing transformation logic and reduces redundant development work. Furthermore, since most of the transformation logic is inherited from historical rules, its correctness and stability are also guaranteed to a certain extent.

[0056] However, in practical applications, there may be situations where the source data to be converted differs significantly from the reference version data, necessitating extensive modifications to historical conversion rules. To address this, the system can introduce a modular design for conversion rules, dividing them into relatively independent sub-rules or functions. This allows updates to the conversion rules to modify only the sub-rules with the most significant differences, without affecting other parts of the logic. Furthermore, the system can incorporate version management for conversion rules, generating a new rule version for each modification. Version management prevents rule updates from interfering with each other and enables quick rollback to previous stable versions.

[0057] S106. The source data to be transformed is processed using the target transformation rules to obtain the target data.

[0058] In this step, the system uses the previously generated target transformation rules to transform the source data, ultimately outputting result data that meets the target data format and quality requirements. This is the final and most crucial step in the entire data transformation process, as the quality of the transformation result directly determines the effectiveness of the data in subsequent use.

[0059] In practice, the system first needs to load the source data to be transformed and the target transformation rules. The source data is usually stored in a transformation buffer, while the target transformation rules exist in the form of configuration files or code logic. After loading, the system transforms the source data according to the steps and order defined in the transformation rules. The transformation process typically includes multiple stages such as data cleaning, data format conversion, data value mapping, and data completion, with each stage potentially applying one or more transformation functions.

[0060] During the transformation process, the system must strictly adhere to the constraints and semantics of the transformation rules to ensure data consistency before and after the transformation. Simultaneously, the system also needs to perform necessary data validation and quality checks. On one hand, it must verify whether the transformed data conforms to the structure and type requirements of the target data model; if any discrepancies are found, they must be reported or corrected promptly. On the other hand, the system needs to check key quality indicators of the transformation results, such as completeness, uniqueness, and value validity.

[0061] After the conversion is complete, the system outputs the converted result data as the target data. The output method of the target data can be determined according to the specific application scenario and requirements. For example, the target data can be directly written to the target database or file system, or it can be passed to downstream data consumers through API interfaces or message queues. Regardless of the output method used, the system must ensure the integrity and consistency of the target data to ensure that subsequent users can successfully read and process this data.

[0062] During data transformation, data quality or performance issues may arise, affecting the usability of the transformation results. To address these issues, the system can employ several optimization techniques. For example, data sharding and parallel processing can improve the throughput and efficiency of the transformation. The source data can be divided into multiple independent data shards, each of which can be transformed independently, and the results can then be merged. Another example is the introduction of a data caching mechanism. Intermediate results that are frequently used or have high transformation costs can be cached for direct use in subsequent steps. Furthermore, incremental transformation techniques can be introduced to fully utilize existing transformation results. If only a portion of the source data has changed, incremental transformation can transform only the changed portion, avoiding the need to transform the entire dataset again, thus saving transformation time and computational resources.

[0063] In the above embodiments, by extracting the version identifier of the source data to be converted and establishing a version mapping chain, all changes to the data from historical versions to the current version are recorded. Based on the complete version mapping chain, the source data to be converted can be restored step by step, generating a sequence containing data from each historical version, ensuring that each historical version data can be accurately matched with the corresponding conversion rule. By selecting the reference version data with the highest structural similarity to the source data to be converted from the historical version data sequence, the most suitable historical conversion rule for the current conversion task can be found. The historical conversion rule is updated by analyzing the differences between the source data to be converted and the reference version data, allowing the conversion rule to adapt to changes in the data structure. Using the updated target conversion rule for conversion processing can achieve accurate data conversion while maintaining data structure consistency, improving the accuracy and reliability of heterogeneous data standardization conversion.

[0064] To further ensure the accuracy and completeness of data transformation, this application also provides a data quality assurance method based on two-way verification. By constructing a two-way verification channel to verify the transformation results, and performing rule optimization and data completion when data loss is detected, the reliability of data transformation is improved. The following describes a method in conjunction with... Figure 2 The present application describes a data quality assurance method based on two-way verification in its embodiments: Please see Figure 2 This is a flowchart illustrating a data quality assurance method based on two-way verification in an embodiment of this application.

[0065] S201. Construct a two-way verification channel based on target data; The system constructs a two-way verification channel based on the target data. The two-way verification channel includes a forward conversion link and a reverse restoration link.

[0066] In this step, the system needs to build a two-way verification channel based on the transformed target data. The two-way verification channel consists of two parts: a forward transformation link and a reverse restoration link, used to verify and confirm the results of the data transformation in both directions. The forward transformation link refers to the process of converting the source data into the target data according to the transformation rules; the reverse restoration link is the process of restoring the target data back to the source data according to the reverse rules. By comparing the data obtained from the forward transformation and the reverse restoration, potential data omissions, distortions, and other problems that may occur during the transformation process can be identified.

[0067] In practical implementation, the system can construct the reverse restoration link by automatically generating reverse rules. The system first analyzes each conversion step in the forward conversion rule and infers its corresponding reverse operation. For example, for data type conversion in the forward conversion, a corresponding reverse type conversion is needed during reverse restoration; for field splitting in the forward conversion, field merging is needed during reverse restoration. The system organizes the inferred reverse operations in the reverse order of the forward conversion to form a complete reverse rule. When actually building the verification channel, the system can reuse the original conversion engine; it only needs to configure the reverse rules as input to the conversion engine. This approach reduces redundant development work and improves system reusability.

[0068] During the construction of the verification channel, some forward transformation rules may be difficult to reverse, resulting in the inability to generate complete reverse rules. For example, there may be irreversible data cleaning operations during the forward transformation (such as null value filtering), or multiple fields may be merged into one field and cannot be split again. To address these situations, the system can assist in reverse reconstruction by introducing additional mapping tables or metadata. For instance, for null value filtering, the filtered fields and their positions can be recorded during the forward transformation, and null values ​​can be filled back during reverse reconstruction based on this information. Similarly, for field merging, delimiters can be added to the merged fields during the forward transformation, and the fields can be split based on these delimiters during reverse reconstruction. Through these auxiliary methods, the system can ensure the integrity and reversibility of the verification channel as much as possible.

[0069] S202. Perform multiple rounds of iterative verification on the target data in the two-way verification channel, calculate the data loss degree in each round of iteration, and compare the data loss degree with the preset loss degree. In this step, the system needs to utilize the previously established bidirectional verification channel to verify the transformed target data. The purpose of verification is to check for any data omissions or distortions during the transformation process, so as to promptly identify and correct errors in the transformation rules. Verification is conducted iteratively, with each iteration performing a complete forward transformation and reverse restoration of the target data. By comparing the target data obtained from the forward transformation and the source data obtained from the reverse restoration, the system can calculate the data loss degree of that transformation round, i.e., the proportion of inconsistencies between the data before and after the transformation. The data loss degree can serve as a quantitative indicator of the quality of the transformation rule; the lower the loss degree, the better the accuracy and reversibility of the transformation rule.

[0070] In practical implementation, the system can adopt a batch verification approach to improve verification efficiency. The target data is divided into multiple batches, and only one batch is verified at a time. During verification, the system first restores the target data of that batch to the source data via a reverse restoration link, and then converts the restored source data back to the target data via a forward transformation link. By comparing the target data obtained from the two transformations, the system can detect whether the batch of data has changed during the transformation process. If inconsistencies are found, the system records these differences and includes them in the data loss statistics. After all data batches have been verified, the system combines the data loss of each batch to obtain the global data loss. The system compares the global loss with a preset tolerance threshold; if the loss exceeds the threshold, the subsequent transformation rule optimization process is triggered.

[0071] During iterative verification, issues may arise where changes to transformation rules cause the restore link to fail. Since the restore link is generated based on a specific version of the transformation rules, if the transformation rules change, the original restore link may fail to correctly reverse-transform the data. To address this, the system can introduce a versioning management mechanism, generating a new version for each change to the transformation rules. During data verification, the system always uses the same versions of transformation and restore rules as those used when transforming the target data, ensuring consistency in the verification process. Furthermore, once the transformation rules are optimized, the system automatically updates the restore link to keep it synchronized with the latest version of the transformation rules. This ensures the verification channel operates continuously and effectively, providing reliable quality assurance for data transformation.

[0072] S203. When the data loss exceeds the preset loss level, extract the key features that cause the data loss and adjust the target transformation rules based on the key features. When the data loss exceeds the preset loss level, the system extracts the key features that cause the data loss and adjusts the target transformation rules based on the key features. Specifically, it counts the field location and loss type where the data loss occurs and generates a list of loss features. Classify and summarize the feature items in the loss feature list to determine the feature type that appears most frequently. Based on the feature type, retrieve the corresponding conversion rule from the preset rule base; Analyze the parameter configuration in the transformation rules to identify the parameter items that cause data loss; Modify the value range or calculation method of parameter items according to the preset parameter adjustment strategy; Update the modified conversion rules to the preset rule library and mark the rule version number.

[0073] In this step, the system dynamically optimizes and adjusts the transformation rules based on the data validation results. When the data loss exceeds a preset tolerance threshold, the system automatically triggers the optimization process for the transformation rules. The core idea of ​​optimization is to identify the key factors causing data loss and modify the transformation rules accordingly, thereby improving the accuracy and reversibility of the transformation.

[0074] In practice, the system first needs to extract the key features causing data loss from the validation results. By analyzing the location and cause of each inconsistency before and after the transformation, the system summarizes typical scenarios and patterns that lead to data loss. For example, data loss may be concentrated on certain specific fields or related to certain special data types and values. Extracting key features helps the system quickly locate the root cause of the problem and reduce ineffective optimization attempts. The system summarizes the extracted key features according to category and severity, generating a list of loss features as a key reference for subsequent optimization.

[0075] After obtaining the list of loss features, the system needs to further determine the optimization strategy corresponding to each feature type. By statistically analyzing historical optimization records, the system identifies the most frequent optimization methods for each feature type, forming a set of effective optimization experience bases. For example, for the feature type of floating-point precision loss, common optimization methods might be to improve numerical precision or switch to fixed-point numbers. The system summarizes these optimization experiences into a series of standardized transformation rule templates and stores them in a pre-set rule base. When a new data loss feature is identified, the system can directly retrieve the corresponding rule template from the rule base and generate targeted transformation rules based on it.

[0076] When generating optimized conversion rules, the system also needs to appropriately adjust and optimize key parameters in the rule template. Parameters affecting conversion results typically include data type definitions, format patterns, and boundary conditions. By analyzing the statistical characteristics of data samples, the system automatically recommends the optimal parameter configuration. For example, the system can optimize the precision parameter of floating-point numbers based on the value distribution of numerical samples; or optimize the maximum length parameter of strings based on the length distribution of text samples. Automatic parameter optimization can further improve the matching degree and adaptability of conversion rules.

[0077] Finally, the system updates the optimized transformation rules to the transformation rule base and assigns version numbers to the new rules. In subsequent data transformation and verification, the system will automatically use the latest version of the transformation rules to ensure continuous improvement in transformation results. Through key feature extraction and transformation rule optimization, the system can achieve a closed-loop self-optimization process, continuously learning and improving transformation strategies, and enhancing the automation level of data transformation and verification.

[0078] It's important to note that automatically optimizing transformation rules may introduce new side effects or overfitting risks. To control these risks, the system can set constraints during rule optimization. For example, it can limit the magnitude of changes made to the rules in a single optimization to prevent excessive changes from affecting rule stability; or it can require optimized rules to undergo regression testing to ensure they function correctly in all known scenarios. The system can also incorporate a manual review mechanism to randomly check and confirm automatically generated rules, especially for core rules with a wide impact, which can be subject to mandatory manual review before release. Through these risk control measures, the system can improve optimization efficiency while ensuring the reliability and maintainability of transformation rules.

[0079] S204. Reapply the adjusted target transformation rules to the source data to be transformed to obtain the updated target data; In this step, the system reapplies the optimized transformation rules to the source data, generating a new round of transformation results. The purpose of this step is to ensure that the optimization results are effectively reflected in the target data, and also to provide input for subsequent data validation. Through iterative optimization of transformation rules and re-transformation of data, the system can continuously improve the quality and effectiveness of data transformation.

[0080] In practice, the system first needs to load the optimized transformation rules and the source data to be transformed. Unlike the initial transformation, the system now selects the latest optimized version of the transformation rules, rather than the original ones. Similarly, the source data to be transformed may also change; for example, after fixing upstream data quality issues, updated source data may be used as input. After loading, the system transforms the source data step by step according to the new transformation rules. Because the rules have been optimized, the transformation effect should be improved, especially at critical points that could lead to data loss.

[0081] After the transformation is complete, the system will output updated target data. Compared to the initial transformation result, the new target data should be superior in terms of data completeness and accuracy. However, since rule optimization itself may introduce new problems, the updated target data still needs to be verified. The system inputs the new target data into the verification channel, repeating the data transformation and restoration process. On the one hand, by observing the changes in data loss, the optimization effect can be evaluated; on the other hand, the verification results can also guide further rule optimization. Through the iterative process of transformation optimization and data verification, the system can continuously approach the optimal transformation effect.

[0082] When applying new conversion rules, the system needs to consider several additional factors. First, since rule optimization may involve adjustments to data types and formats, the system needs to adapt the data reading and writing methods accordingly during conversion. For example, if the rule adjusts the date format, the system needs to modify the date parsing and output code accordingly. Similarly, if the rule adjusts numerical precision, the system needs to check intermediate calculation steps to avoid cumulative precision loss. Second, the system needs to remain sensitive to changes in the source data. If the structure or content of the source data changes, the original conversion rules may no longer be applicable. In this case, in addition to re-optimizing the conversion rules, the system needs to analyze the changes in the source data and identify new conversion requirements. The system can use techniques such as incremental data probing and data distribution comparison to promptly detect changes in the source data and trigger the corresponding rule optimization process.

[0083] S205. Perform semantic integrity analysis on the updated target data, establish a semantic relationship graph between fields, and complete the data based on the semantic relationship graph.

[0084] In this step, the system needs to perform semantic-level checks and repairs on the transformed target data. Unlike the previous steps, which focused on structural integrity and value accuracy, this step prioritizes the coherence and consistency of the data's semantics. By analyzing the semantic relationships between different fields, the system can identify and fill in semantic gaps or anomalies in the target data, further improving the data's usability and reliability.

[0085] In practical implementation, the system first needs to perform semantic parsing and association analysis on the target data. The purpose of semantic parsing is to identify the real-world semantics represented by each field, such as customer name, order amount, and transaction date. Semantic association analysis reveals the semantic dependencies and constraints between different fields, such as the chronological relationship between order amount and transaction date, or the correspondence between customer name and customer level. Semantic parsing can utilize field names, annotations, metadata, and other information to match predefined semantic dictionaries; semantic association analysis can utilize data quality rules, business specifications, domain knowledge, and other information to uncover implicit dependencies between fields.

[0086] After completing semantic parsing and association analysis, the system can build a semantic association graph between fields. A semantic association graph is a graphical data model where nodes represent the semantic concepts of fields, and edges represent the semantic relationships between fields. Common semantic relationships include functional dependencies, primary and foreign key relationships, and value range constraints. The semantic association graph intuitively depicts the semantic structure and constraints of the target data, facilitating the system's discovery and location of semantic anomalies. For example, by analyzing the semantic association graph, the system can identify data records that violate functional dependencies (such as multiple order numbers corresponding to the same transaction serial number) or field values ​​that violate value range constraints (such as age field values ​​exceeding the normal range).

[0087] After identifying semantic anomalies in the target data based on the semantic association graph, the system needs to further attempt to complete and repair these anomalies. The basic idea for completion and repair is to use the constraint information contained in the semantic association to infer the correct values ​​of missing or incorrect fields. For example, if a functional dependency is found between certain fields, the value of one field can be calculated based on the value of another field; if the value of a field is found to be outside the normal range, it can be replaced with a similar boundary value, and so on.

[0088] In the above embodiments, a bidirectional verification channel containing a forward conversion link and a reverse restoration link is constructed. Multiple rounds of iterative verification and data loss calculation are performed on the target data within this channel, achieving quality monitoring of the conversion results. When the data loss exceeds a preset loss level, key features causing the data loss are extracted, and the target conversion rules are adjusted accordingly, improving the adaptability of the conversion rules. Semantic integrity analysis is performed on the updated target data, and a semantic relationship graph between fields is established. Data completion is performed based on the semantic relationship graph, ensuring the integrity and semantic consistency of the converted data and improving the accuracy and usability of the data conversion.

[0089] Furthermore, in another embodiment, the system performs a quality check on the completed target data according to a preset data quality rule set, and generates a verification report containing quality problem location information. The quality check includes data format consistency, reasonable value range, and compliance with business rules. Based on the quality problem location information in the verification report, extract the corresponding conversion rule identifier and completion operation record, and establish a link between the quality problem and the data processing process; Based on the analysis of the relationship chain, the link in which quality problems occur is analyzed, and the data items with quality problems are corrected in a targeted manner in combination with business rules; Synchronously record the triggering conditions, correction methods, and correction results of the correction to obtain the correction record; Based on the correction records, optimize the corresponding conversion rules and completion strategies. Specifically, count the execution frequency of various correction operations in the correction records to determine high-frequency correction scenarios. Extract data features and correction methods in high-frequency correction scenarios to obtain correction patterns; Adjust the parameter configuration and execution conditions of the corresponding conversion rules according to the correction rules; The data mapping relationships in the completion strategy are supplemented and improved based on the correction rules.

[0090] In this embodiment, the system introduces a data quality check and feedback optimization mechanism to further improve the effectiveness of data transformation and completion. By performing a quality check on the completed target data, data quality issues arising during the transformation and completion process can be promptly identified and located. The system-generated verification report not only includes specific information about the problematic data but also the stage and cause analysis of the problem. This fine-grained location information helps the system quickly identify weaknesses in the transformation rules and completion strategies, providing precise direction for subsequent optimization.

[0091] In practice, the system first needs to perform a comprehensive check on the target data according to a pre-defined set of data quality rules. This set typically includes three check items: data format consistency, reasonable value range, and compliance with business rules. Specifically, the data format consistency check ensures that the data format and type of the same field are consistent across all records; the reasonable value range check ensures that the value of each field is within a legal range; and the business rule compliance check ensures that the data meets the constraints of specific business scenarios (such as a monetary field must be greater than zero, or a date field must be within a reasonable time range). The system outputs the check results in the form of a verification report, which not only lists all detected quality issues but also marks the location coordinates of each problematic data point (such as the row and column numbers of the data file).

[0092] After receiving the verification report, the system needs to further analyze the causes of the quality issues. By parsing the location coordinates in the verification report, the system pinpoints the complete process of transformation and completion of the problematic data. Based on this, the system extracts relevant transformation rule identifiers (i.e., the specific transformation rules that caused the problem) and completion operation records (i.e., the specific completion strategies and parameters that caused the problem), and establishes a link between the problematic data and the transformation and completion process. This link clearly depicts the generation context of the problematic data, helping the system trace and locate the root cause of the problem. For example, if a large number of date format errors are found, the link can quickly pinpoint the error in the transformation rules of the date parsing stage; if a large number of out-of-range values ​​are found, the link can reveal an error in the boundary handling of the completion strategy.

[0093] After locating the problematic link through the correlation chain, the system needs to correct the problematic data. Unlike conventional data cleaning, this correction is a targeted process designed for specific quality issues and business rules. The system first determines feasible correction solutions based on the problem type and business constraints. For example, data with incorrect formatting can be corrected using regular expression matching and replacement; data with values ​​outside the range can be corrected through boundary truncation or mean padding. Then, based on the transformation and completion links located through the correlation chain, the system performs targeted correction on the problematic data. This targeted correction minimizes the impact on the data while ensuring that quality requirements are met.

[0094] While correcting problematic data, the system also simultaneously records the context information for each correction, forming a complete correction log. The correction log typically includes the type of problem that triggered the correction, the applied correction method, the data values ​​before and after the correction, and relevant statistical information. On one hand, the correction log serves as a basis for system self-auditing and evaluation, helping the system quantify the robustness of the transformation and completion processes; on the other hand, the correction log is also an important input for optimizing transformation rules and completion strategies, containing rich information for improvement. The system periodically analyzes the accumulated correction logs to identify patterns and rules worthy of optimization.

[0095] For recurring problem patterns in the correction logs, the system automatically extracts key features to form high-frequency correction scenarios. These high-frequency correction scenarios typically represent weaknesses in the system's conversion and completion process and are key areas for optimization. The system further analyzes the causes of these high-frequency scenarios, summarizes the underlying data characteristics and effective correction methods, and extracts reusable correction rules. These rules reveal the inherent connections between problems and can guide targeted improvements to the system's conversion rules and completion strategies.

[0096] For conversion rules, the system adjusts the rule's parameter configuration (such as format mode, boundary conditions, etc.) and execution conditions (such as filter expressions, branch logic, etc.) based on the data characteristics in the correction rules, enabling the rules to adapt to more abnormal situations. For example, if date parsing errors are found to mostly occur under certain specific formats, the system can add a dedicated format branch to the date conversion rule to improve compatibility with these abnormal formats. As another example, if value range issues are found to mostly occur in certain specific data sources, the system can add filter conditions at the data conversion entry point to intercept suspicious data at the source.

[0097] For the completion strategy, the system supplements and expands the data mapping relationships in the strategy based on effective correction methods within the correction rules. Data mapping is the core of the completion strategy, determining how to infer missing fields based on existing information. The system abstracts the inference logic of various correction methods into mapping rules and sets corresponding applicable conditions. When a specific missing pattern is encountered during the completion process, the system can automatically apply the matching mapping rule to dynamically correct and fill in field values. For example, by analyzing correction records, the system discovers that missing values ​​in the customer level field can be inferred based on the customer's total spending. Therefore, the system adds a new mapping rule to the completion strategy, describing the correspondence between total spending and customer level. In this way, when encountering a missing customer level in the future, the system can automatically apply this rule for completion without manual correction.

[0098] In the above embodiments, quality checks are performed on the completed target data, including format consistency, value rationality, and compliance with business rules. A verification report containing problem location information is generated, enabling the system to comprehensively assess the data quality status. By extracting the conversion rule identifiers and completion operation records corresponding to quality problems, a link between quality problems and the processing process is established, allowing the system to trace and locate the specific stage where the problem occurred. In conjunction with business rules, targeted corrections are performed on data with quality problems, and the triggering conditions, methods, and results of the corrections are recorded, making the correction process traceable. Based on the correction records, conversion rules and completion strategies are optimized, establishing a closed-loop processing mechanism for quality problem discovery, location, correction, and optimization. Through experience accumulation, data processing capabilities are continuously improved, enhancing the accuracy and reliability of data conversion results.

[0099] The system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of a heterogeneous data standardization conversion system provided in an embodiment of this application.

[0100] It should be noted that, Figure 3 The structure of the system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0101] like Figure 3As shown, the system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage portion 308 into Random Access Memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0102] The following components are connected to I / O interface 305: input section 306 including a camera, infrared sensor, etc.; output section 307 including a liquid crystal display (LCD) and speakers, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card and a modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0103] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0104] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. The transmitted data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the system described in the above embodiments; or it may exist independently and not assembled into the system. The storage medium carries one or more computer programs that, when executed by a processor of a system, cause the system to implement the methods provided in the above embodiments.

[0107] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0108] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0109] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for standardizing and transforming heterogeneous data, characterized in that, include: Extract the version identifier based on the tagging information of the source data to be converted, and store the source data to be converted in the conversion buffer; A version mapping chain is established based on the version identifier of the source data to be converted. The version mapping chain includes all version change records from the historical version to the current version of the source data to be converted. Based on the version mapping chain, the source data to be converted is restored step by step to obtain a historical version data sequence, and each historical version data in the historical version data sequence corresponds to a conversion rule. From the historical version data sequence, select the reference version data with the highest similarity to the source data structure to be converted, and determine the historical conversion rules corresponding to the reference version data; Extract the differences between the source data to be converted and the reference version data, and incrementally update the historical conversion rules based on the differences to obtain the target conversion rules; The target data is obtained by transforming the source data using the target transformation rules.

2. The method according to claim 1, characterized in that, The step of establishing a version mapping chain based on the version identifier of the source data to be converted specifically includes: Obtain the version inheritance attribute information of the source data to be converted, the version inheritance attribute information including version identifier, data structure and version dependency relationship; A multi-branch version inheritance tree is constructed based on the version inheritance attribute information, and version nodes with the same parent version are constructed as sibling nodes at the same level. Calculate the structural and data differences between adjacent nodes in the version inheritance tree, and generate a version difference feature vector; Based on the version difference feature vector, cluster analysis is performed on the version nodes, and version nodes with similarity exceeding a preset threshold are merged. Based on the dependencies between the merged version nodes, a version mapping chain in the form of a directed acyclic graph is constructed.

3. The method according to claim 1, characterized in that, The step-by-step restoration of the source data to be converted based on the version mapping chain specifically includes: Starting from the current version node of the version mapping chain, traverse the version mapping chain backwards to the initial version node; Extract data structure change records and data content conversion rules between adjacent version nodes; Based on the data structure change record, the current version of the data is restored to obtain the data structure of the previous version. The data content in the restored data structure is transformed using the aforementioned data content transformation rules; The converted data is used as the new current version data, and the step of restoring the structure of the current version data based on the data structure change record is repeated until the initial version is reached.

4. The method according to claim 1, characterized in that, After the source data to be transformed is processed using the target transformation rule to obtain the target data, the method further includes: A two-way verification channel is constructed based on the target data, and the two-way verification channel includes a forward conversion link and a reverse restoration link; In the bidirectional verification channel, the target data is iterated and verified in multiple rounds. The data loss degree in each round of iteration is calculated and compared with the preset loss degree. When the data loss exceeds the preset loss, key features causing the data loss are extracted, and the target transformation rule is adjusted based on the key features. The adjusted target transformation rules are reapplied to the source data to be transformed to obtain the updated target data. Semantic integrity analysis is performed on the updated target data to establish a semantic relationship graph between fields, and data completion is performed based on the semantic relationship graph.

5. The method according to claim 4, characterized in that, The extraction of key features that cause data loss, and the adjustment of the target transformation rules based on these key features, specifically includes: Analyze the fields where statistical loss occurs and the type of loss to generate a list of loss features; The feature items in the loss feature list are classified and summarized to determine the feature type that appears most frequently. Based on the aforementioned feature type, the corresponding conversion rule is retrieved from the preset rule base; Analyze the parameter configuration in the transformation rules to identify the parameter items that cause data loss; The value range or calculation method of the parameter item is modified according to the preset parameter adjustment strategy; Update the modified conversion rules to the preset rule base and mark the rule version number.

6. The method according to claim 4, characterized in that, After performing data completion based on the semantic association graph, the method further includes: The completed target data is subjected to quality checks based on a preset set of data quality rules, and a verification report containing information on the location of quality problems is generated. The quality checks include data format consistency, reasonable value range, and compliance with business rules. Based on the quality problem location information in the verification report, extract the corresponding conversion rule identifier and completion operation record, and establish a link between the quality problem and the data processing process; Based on the aforementioned association chain analysis, the generation process of the quality problem is determined, and the data items with the quality problem are corrected in a targeted manner in accordance with business rules. Synchronously record the triggering conditions, correction methods, and correction results of the correction to obtain the correction record; The corresponding conversion rules and completion strategies are optimized based on the correction records.

7. The method according to claim 6, characterized in that, The optimization of the corresponding conversion rules and completion strategies based on the corrected records specifically includes: Statistically analyze the execution frequency of various correction operations in the correction records to identify high-frequency correction scenarios; Extract the data features and correction methods under the high-frequency correction scenario to obtain the correction rules; Adjust the parameter configuration and execution conditions of the corresponding conversion rules according to the aforementioned correction rules; The data mapping relationships in the completion strategy are supplemented and improved based on the aforementioned correction rules.

8. A heterogeneous data standardization conversion system, characterized in that, The system includes: One or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the system, the system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the system, the system performs the method as described in any one of claims 1-7.

Images