A multi-source public data integration system supporting dynamic schema mapping

Abstract

The application discloses a multi-source public data integration system supporting dynamic mode mapping, which comprises a source data access unit, a mode and mapping record unit and a mode change identification unit; the mode and mapping record unit saves source mode versions, unified target modes, effective mappings and downstream dependencies; the mode change identification unit compares new and old source modes of the same source to obtain change elements and a new source mode version; the system further comprises an affected range determination unit, a candidate mapping formation unit, a temporary verification unit and a version selection and switching unit. The multi-source public data integration system supporting dynamic mode mapping determines the actual affected range after the field or interface structure of the public data changes, adjusts and verifies only the related rules, uses new rules after the verification, and isolates the related records when the verification fails, so that other data can continue to be processed normally.

Description

Technical Field

[0001] This invention relates to the field of public data processing technology, specifically to a multi-source public data integration system that supports dynamic pattern mapping. Background Technology

[0002] Public data platforms typically need to access data provided by different departments or business systems through database tables, data interfaces, or batch files. Through methods such as field mapping, type conversion, and record association, data from different sources and with varying structures are transformed into a unified data structure for querying, statistics, and interface calls. In this process, the table or interface paths, field names, data types, field lengths, hierarchical relationships, associations, and constraints used by each data source together constitute the corresponding source schema. The platform writes the data in the source schema into a unified target schema based on the pre-configured mapping relationship.

[0003] For example, application number 202111579722.3 describes a heterogeneous data integration and query system and method based on RDF. This system sets up a raw data layer, a data encapsulation layer, a semantic layer, a data query layer, and an application layer. It describes heterogeneous data sources through RDF Schema and integrates and queries data from different sources by utilizing schema mapping, query transformation, and result encapsulation between local and global schemas. This provides an implementation method for the unified description and query of heterogeneous data.

[0004] However, the data structure received by the public data platform will not always remain unchanged. Data providers may add, delete, split, or merge fields during system operation, and may also change field types, interface levels, inter-table relationships, or constraints. Mapping rules that were originally working properly may become invalid only in some places. If the platform cannot identify in time which fields, statistical items, and output interfaces are truly affected, it may adjust a large number of still working mappings. If the modified mappings are processed for official data without confirmation, errors may continue to be transmitted to statistical results and service interfaces. Especially when the old interface has been deactivated or multiple data sources are running at the same time, the platform also needs to prevent new format data from continuing to use old mappings and prevent local changes in one source from causing other sources to become unusable. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-source public data integration system that supports dynamic pattern mapping, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-source public data integration system supporting dynamic pattern mapping, the system comprising a source data access unit, a pattern and mapping recording unit, and a pattern change identification unit;

[0007] The source data access unit acquires data and source patterns from various public data sources;

[0008] The pattern and mapping record unit stores the source pattern version, unified target pattern, effective mapping, and downstream dependencies;

[0009] The pattern change identification unit compares the old and new source patterns from the same source to obtain the changed elements and the new source pattern version.

[0010] The system also includes an affected scope determination unit, a candidate mapping formation unit, a temporary verification unit, and a version selection and switching unit;

[0011] The affected scope determination unit starts with the changed element and searches step by step along the effective mapping and downstream dependencies to obtain the affected scope.

[0012] The candidate mapping forming unit, based on the effective mapping, forms ineffective candidate mappings corresponding to the new source mode version only for the affected range;

[0013] The temporary verification unit operates the candidate mapping and obtains the verification result based on the verification data with confirmation results in a temporary verification area isolated from the formal data area and the formal output interface;

[0014] When the verification is successful, the version selection and switching unit establishes the effective relationship between the new source mode version and the candidate mapping. When the verification fails, the candidate mapping remains ineffective, allowing new format records that depend on the candidate mapping to enter the waiting area, and allowing mappings that have not entered the affected range to continue to be effective.

[0015] Furthermore, the source schema includes at least table or interface path, field name, data type, field length, hierarchical relationship, association relationship, and constraints. The schema change identification unit identifies changes in fields (addition, deletion, splitting, merging, data type or field length changes, hierarchical path changes, primary / foreign key or inter-table association changes, and constraints) by comparing the new and old source schemas. Candidate mapping formation is not triggered when the interface version changes but the structural elements of the source schema remain unchanged.

[0016] As a further step, the pattern and mapping recording unit records the source field, source relation, target field, target relation, transformation rule, association processing, statistical items, and output interface as dependency nodes, and records field mapping, field combination or splitting, type conversion, record connection, statistical usage, and interface output as directed dependency edges; the affected scope determination unit starts from the dependency node corresponding to the changed element, expands only along the directed dependency edges that match the change type and satisfy the propagation conditions, and stops expanding when there are no subsequent dependencies or the subsequent dependencies are unrelated to the change type, thus obtaining the least affected closure.

[0017] As a further step, the affected range determination unit writes access markers to the visited dependent nodes and directed dependent edges, stops repeated expansion when the same dependent node is reached again; takes the union of the minimum affected closures obtained from multiple changed elements, records the source of influence and marks rule conflicts for cases where the same target object is repeatedly affected by different paths, and expands fields with common association keys, primary and foreign keys or integrity constraints as a whole association constraint group.

[0018] As a further step, the candidate mapping is based on the effective mapping and records the new source mode version, the rule identifier that was replaced, the added or modified rules, the applicable objects of each rule, the coverage priority, the conflict check result, the manual confirmation status and the verification status. When reading the mapping, the coverage rules in the candidate mapping are matched first, and then the unaffected rules in the base version are read.

[0019] As a further step, when the same target field hits multiple candidate rules at the same time, the candidate mapping forming unit selects candidate rules according to the applicable source mode version, the applicable object scope, and the confirmed coverage priority; if the candidate rules still cannot be uniquely determined based on the applicable source mode version, the applicable object scope, and the coverage priority, the conflicting candidate rules are kept in an inactive state and transferred to manual confirmation.

[0020] As a further step, when the old interface and the new interface are used in parallel, the verification data includes old format records and new format records of the same data object and the same time range. The temporary verification unit processes the corresponding records using the old mapping and the candidate mapping respectively and compares the results. When the old interface is no longer used, the verification data includes new format samples and their confirmed target results, or new format records and target results confirmed by humans.

[0021] As a further step, the temporary verification unit performs structural constraint checks, record correspondence checks, cross-source association checks, and output interface checks on the candidate mappings. It takes the target fields, target types, uniqueness, association integrity, and fields required for the formal output interface as hard constraints, and takes the missing optional fields and abnormal source values ​​as data quality checks. The unit then summarizes the check results in the order of record, candidate rule, association constraint group, and candidate mapping version.

[0022] As a further step, the temporary verification unit determines that if a field in the source pattern is empty in a single record, it is considered that the source data is missing; if the source value does not conform to the data type declared by the data provider, it is considered that the source data is abnormal; and if the source field has a value that meets the requirements but the mapping result is missing, the type is incorrect, or the write position is incorrect, it is considered that the mapping error is incorrect. The source data abnormality only affects the corresponding record, and the mapping error causes the candidate rule involving the mapping error to fail the verification.

[0023] As a further step, the candidate mappings are managed according to the following states: draft, pending confirmation, under verification, verified, effective, verified failed, and deactivated. Candidate rules with common association keys, primary-foreign keys, or integrity constraints are identified as association constraint groups. The version selection and switching unit only makes the candidate rules take effect as a whole through the same switching transaction or the same effective identifier when all candidate rules in the association constraint group have passed verification, and makes the same batch of data use the corresponding source schema version and mapping version. When verification fails, the corresponding new format record is written to the pending area. After the candidate mapping is corrected and re-verified, the records that have not yet formed formal results are reprocessed using an idempotent key composed of source identifier, record identifier, source schema version, and receiving batch.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: This multi-source public data integration system that supports dynamic pattern mapping first determines the actual affected scope after the field or interface structure of public data changes, only adjusts and verifies the relevant rules, uses the new rules after the verification is passed, and isolates the relevant records when the verification fails, so that other data can continue to be processed normally, as shown in the following details.

[0025] 1. The system starts from the field or relationship that has changed, searches step by step along the effective mapping and downstream usage relationships to form the least affected closure, and only updates the rules and output objects that truly depend on the change. In simpler terms, it first identifies the places that need to be modified and then makes adjustments, allowing unrelated mappings and other data sources to continue operating, reducing unnecessary rebuilds and shutdowns;

[0026] 2. Candidate mappings are first processed in a temporary verification area isolated from the formal data area, where data with confirmed results is processed, and the structure, records, relationships, and output results are checked separately. In this way, even if there are problems with the new rules, the errors will remain in the verification stage, and it can distinguish between anomalies in the source data itself and errors in the mapping rules, reducing the possibility of erroneous results entering the formal data flow;

[0027] 3. The system associates the source mode version, mapping version, and receiving batch. Upon successful verification, the relevant rules are switched as a whole. If verification fails, only the affected records are placed in the pending processing area and reprocessed according to the idempotent key after the rules are corrected. In simpler terms, rules are changed only after verification, and the corresponding records are isolated if verification fails. This reduces version mixing and duplicate writes without affecting the continued processing of other normal data. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the system composition and main data relationships of the present invention;

[0029] Figure 2 This is a schematic diagram of the overall processing flow after the source mode of the present invention has changed;

[0030] Figure 3 This is a schematic diagram illustrating the dependency traversal and the least affected closure in this invention.

[0031] Figure 4 This is a schematic diagram illustrating the affected range when the location field is split in this invention;

[0032] Figure 5 This is a schematic diagram illustrating the candidate mapping coverage relationship and rule selection of the present invention;

[0033] Figure 6 This is a schematic diagram illustrating the verification classification and version switching of candidate mappings in this invention;

[0034] Figure 7 This is a schematic diagram illustrating the idempotent reprocessing of candidate mapping version status and records to be processed in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figures 1-7 The present invention provides the following technical solution:

[0037] This embodiment provides a multi-source public data integration system that supports dynamic pattern mapping, such as... Figure 2 As shown, the system includes at least a source data access unit, a pattern and mapping record unit, a pattern change identification unit, an affected scope determination unit, a candidate mapping formation unit, a temporary verification unit, a version selection and switching unit, a pending processing unit, and an integration output unit. Each unit can be deployed on the same server or separately according to data access, temporary verification, and formal processing. The system's data entry includes data provided by various public data sources, interface definitions, or table structure descriptions. The pattern and mapping record unit stores the source pattern, unified target pattern, currently effective mapping relationship, downstream dependency relationship, and corresponding version status.

[0038] The following combination Figures 1 to 7 Tables 1 to 4 further illustrate the system. The number of sources, access methods, and deployment locations in this implementation can be configured according to the actual situation of the public data platform. The source mode, mapping, dependency relationship, and verification results processed by each unit retain the corresponding version identifier so as to verify the complete process from discovery to effectiveness of a structural change.

[0039] The dynamic pattern mapping referred to in this embodiment refers to the identification, local adjustment, isolation verification, and version switching of the correspondence between a data source and a unified target pattern when the structural changes occur in the field composition, data type, field length, hierarchical path, association relationship, or constraint conditions of a certain data source after the system is put into operation. Ordinary correspondence processing between standard fields whose meanings have been determined only because of different names is not considered as structural changes that this embodiment focuses on solving.

[0040] like Figure 1 As shown, the system operates in the following order: source data access, pattern change identification, affected scope determination, candidate mapping formation, temporary verification, and version selection and switching. Only candidate mappings that pass verification can process the corresponding version of the official data. If verification fails, only new format records that depend on the candidate mapping are isolated, while mappings from other sources and those not affected by structural changes can continue to operate.

[0041] like Figure 2 As shown, the system includes a source data access unit, a pattern and mapping recording unit, and a pattern change identification unit; the system also includes an affected scope determination unit, a candidate mapping formation unit, a temporary verification unit, and a version selection and switching unit; in order to save the new format records that cannot be safely entered into the formal process and to receive the integration results after verification, this embodiment also sets up a pending processing unit and an integration output unit. The units transmit traceable source pattern version, mapping version, changed content, affected scope, verification results, and record status. The received content, main processing, and output status are shown in Table 1.

[0042] Components Received content Main processing Output or status Source data access unit Data, interface definitions, or table structure descriptions from various sources Data is received based on source and interface version; data extraction or reading patterns are defined. Source records and their mode versions Pattern and Mapping Record Unit Source pattern, unified target pattern, currently active mapping, and downstream usage relationship Save version compatibility and current active status. Traceable patterns and mapping records Pattern change recognition unit New and old models from the same source Compare fields, types, levels, associations, and constraints. Specific changes Affected Scope Determination Unit Changes, currently active mappings, and downstream usage relationships Search for direct and indirect dependencies level by level from the location of the change. Scope that needs to be checked or updated Candidate mapping forming unit Scope of impact, rule templates, and new patterns Mapping is generated only for the affected parts or after manual verification. Candidate mapping versions Temporary verification unit Candidate mapping, validation data, and confirmation results Isolate and run the system to inspect the structure, records, associations, and outputs. Classification of Pass, Fail, and Abnormal Version selection and switching unit Verification results, source mode version, and mapped version Control the activation of specific versions to prevent the same batch of data from using different versions. Effective relationships and switch records Unit to be processed New format records that cannot be safely entered into the formal process Save separately and reprocess after mapping correction. Pending status and reprocessing results Integrated output unit Validated mapping results Write to the formal data area (unified target data area) and provide query or interface output to downstream. Unified data and service results

[0043] Table 1 System composition and processing relationships between its components

[0044] The above units can be set in different program components on the same server, or data access, temporary verification and formal processing can be deployed separately and version records and processing results can be passed through controlled interfaces. The formal data area is an area that stores verified integrated data according to a unified target model. The formal query and output interface reads or provides data from the formal data area. The temporary results formed by the candidate mapping cannot enter the formal data area before the candidate mapping takes effect.

[0045] S1, obtain the data and source patterns of each public data source, and read the unified target pattern, currently effective mapping and downstream dependency records required for this processing. The source data access unit obtains the data and source patterns of each public data source; the pattern and mapping record unit saves the source pattern version, unified target pattern, effective mapping and downstream dependency relationship.

[0046] In practice, source data can be accessed through data interfaces, database tables, or batch files. The source data access unit associates the source records with their source schemas according to the source identifier, interface or table identifier, interface version, and receiving batch. The source schema can be read from database metadata, interface description files, or structure descriptions published by the data provider. When the source cannot directly provide complete metadata, the field paths and data types can be extracted based on the actual returned records, and then confirmed by the administrator.

[0047] The unified target schema is used to specify the target fields, target relationships, and constraints adopted by the formal data area. The effective mapping records at least the source field or source relationship, the target field or target relationship, the transformation rule, the applicable source schema version, and the effective status. The downstream dependency records which transformation rules, association processing, statistical items, and output interfaces continue to use the corresponding target field, so that the system can find the objects that actually use the location from a source structure location.

[0048] When each batch of data enters the processing flow, the system saves the source identifier, reception time, source mode version, and mapping version. It first fixes the source mode version used for the batch of data, and then reads the effective mapping that matches the version. This avoids using the old mapping for one part of the same batch of data and the new mapping for another part, and at the same time, it preserves the basis for determining the range of records to be reprocessed later.

[0049] S2, compare the old and new source patterns from the same source, identify the structural elements that have actually changed, and form a new source pattern version. The pattern change identification unit compares the old and new source patterns from the same source to obtain the changed elements and the new source pattern version. During the comparison, the same data source is located by the source identifier and the interface or table identifier. The previously accepted source pattern of that source is used as the old source pattern, and the structural description obtained this time is used as the new source pattern. The source pattern includes at least the table or interface path, field name, data type, field length, hierarchical relationship, association relationship, and constraint conditions. The pattern change identification unit identifies changes in field addition, deletion, splitting, merging, data type or field length, hierarchical path, primary and foreign keys or inter-table association relationships, and constraint conditions by comparing the old and new source patterns. Candidate mapping is not triggered when the interface version changes but the structural elements of the source pattern do not change.

[0050] The comparison results should at least record the data source of the change, the old source schema version, the new source schema version, the changed elements, the definitions before and after the change. For example, record which interface path added fields, which fields a field was split into, what the field type or length changed from structure to structure, and what adjustments were made to primary and foreign keys, required conditions, unique conditions, or code ranges. When the interface version number changes but the fields, types, lengths, levels, associations, and constraints remain unchanged, the system only records the correspondence between the interface version and the original source schema. Changing a normal data value from one value to another is considered a data content update. If a field existing in the source schema is empty only in a single record, it is considered a source data quality issue. Neither of these situations is considered a change in the source schema structure.

[0051] S3, starting with the changed element, the affected scope is determined by following the currently used mapping relationship and subsequent usage relationship. The affected scope determination unit starts with the changed element and searches step by step along the effective mapping and downstream dependency relationship to obtain the affected scope. Here, the step-by-step search first determines the target object directly corresponding to the changed field or change relationship, and then continues to search for the transformation, connection, statistics and output objects that use the target object until the subsequent objects are irrelevant to this change or there is no subsequent dependency. The pattern and mapping recording unit records the source field, source relationship, target field, target relationship, transformation rule, association processing, statistics item and output interface as dependency nodes, and records field mapping, field combination or split, type conversion, record connection, statistics usage and interface output as directed dependency edges. The affected scope determination unit starts from the dependency node corresponding to the changed element and only expands along the directed dependency edges that match the change type and satisfy the propagation condition, and stops expanding when there is no subsequent dependency or the subsequent dependency is irrelevant to the change type, to obtain the minimum affected closure.

[0052] like Figure 3 As shown, dependency nodes represent source fields, target fields, processing rules, or output objects that can be inspected individually. Directed dependency edges indicate that the result of the previous object will continue to be used by the next object. Propagation conditions are used to determine whether a certain type of change will continue to have an impact along the usage relationship. For example, field deletion will propagate to the required target fields and interfaces that depend on that field, field type changes will propagate to type conversion and length validation, and association key changes will propagate to the query of record joins and dependent join results. Other sources that have no dependency on the corresponding field and relationship will not continue to expand. The direct objects and subsequent objects that need to be inspected for different structural changes are shown in Table 2. The scope of impact can be saved according to fields, tables, interfaces, or association relationships.

[0053] Source pattern change Directly inspect the object Objects that may continue to be affected Objects that typically do not need to be updated Add a field Does a target field exist that needs to receive this field? Output interface using the target field Current effective mapping when there is no usage relationship Field deletion Target field and required constraints corresponding to the original field Statistical items and interfaces that depend on this target field Do not use other source mappings for deleted fields Field splitting or merging Field combination, splitting, and null value handling rules Combinations of address, name, etc., and their downstream outputs Fields unrelated to the combined result Data type or length change Type conversion, truncation, and validation rules Requires output of a specific type or length Processing that is independent of both content and type Changes in hierarchy or relationships Field paths, primary and foreign keys, and record join rules The target relationship and query depend on the result of this join. Mapping of independent tables or independent sources Constraints or code scope changes Required fields, uniqueness, and code conversion rules Statistics and interfaces using corresponding code or constraints Other fields that do not use this constraint

[0054] Table 2. Scope of impact corresponding to changes in different modes

[0055] The affected range determination unit writes access markers to visited dependent nodes and directed dependent edges, and stops repeated expansion when the same dependent node is reached again. It takes the union of the minimum affected closures obtained from multiple changed elements, records the source of influence and marks rule conflicts in cases where the same target object is repeatedly affected by different paths, and expands fields with common association keys, primary and foreign keys or integrity constraints as a group of association constraints. The access markers can prevent repeated lookups when primary and foreign keys refer to each other or when multiple statistical items refer to the same field. After the range formed by multiple changed elements is merged, their respective sources of influence are still retained. Therefore, the system can not only identify the situation where the same target object is affected by multiple paths at the same time, but also point out the rule conflicts that need to be resolved when candidate mappings are formed in the future.

[0056] For newly added fields, if there is no target field in the unified target schema that needs to receive that field, and the statistics, association, and output interfaces do not use that field, the system only saves the changed element and its new source schema version. It does not include existing mappings in the affected scope, nor does it create candidate mappings related to the newly added field. Figure 4 As shown, when Department 1 splits the original location description field into structured fields such as city, district / county, street, and detailed address, the system starts from the changed location field and finds the location combination rules, unified location description, regional code, and statistical items and location output interfaces that depend on these results. Departments 2 and 3 are not included in this scope if their data structures do not depend on the location relationship. Location fields with common regional association keys or integrity requirements are included in the inspection as a whole as an association constraint group.

[0057] The resulting hierarchical record of the affected scope records the source of change, propagation path, associated constraint group, and rule conflict status of each object, enabling subsequent processing to form candidate mappings only for that scope, while leaving a complete relational basis for this local adjustment.

[0058] S4. Based on the currently effective mapping, a local candidate mapping corresponding to the new source mode version is formed. The candidate mapping forming unit forms an ineffective candidate mapping corresponding to the new source mode version only for the affected range, based on the effective mapping. This allows the rules that need to be adjusted to be extracted separately. Field correspondence, type conversion, record connection, and interface output rules that are not included in the affected range still use the original effective version. The candidate mapping is based on the effective mapping and records the new source mode version, the identifier of the replaced rule, the added or modified rule, the applicable objects of each rule, the coverage priority, the conflict check result, the manual confirmation status, and the verification status. When reading the mapping, the coverage rules in the candidate mapping are matched first, and then the unaffected rules in the base version are read.

[0059] like Figure 5 As shown, candidate mappings can be saved as difference records relative to the base version, or as complete version snapshots that can express the same coverage relationship. Regardless of the saving format, it should be clear which original rules the new rule replaces, which source mode version it applies to, and which base rules remain valid. When the same target field hits multiple candidate rules at the same time, the candidate mapping forming unit selects candidate rules according to the applicable source mode version, the scope of applicable objects, and the confirmed coverage priority. If the candidate rules cannot be uniquely determined based on the applicable source mode version, the scope of applicable objects, and the coverage priority, the conflicting candidate rules are kept in an inactive state and transferred to manual confirmation.

[0060] Candidate rules can be formed based on pre-saved type conversion rules, field splitting or merging rules, interface path adjustment instructions, and change instructions issued by the data provider. For example, when the original single location description field is split into city, district, street and detailed address fields, location combination rules can be formed according to the hierarchical order confirmed by the data provider, and the corresponding area identifier can be written into the area code field of the unified target pattern.

[0061] When a source field is deleted and there is no clear replacement in the data, the system does not automatically fill in the meaning of the original field based on similar field names or numerical forms. Only when the platform has configured the alternative data source, alternative field, and applicable conditions will a candidate mapping for using the alternative source be formed. In other cases, manual confirmation will be required, and the new format records that depend on the field will be listed as pending objects.

[0062] S5. In a temporary verification area isolated from the formal processing flow, the candidate mapping is checked. The temporary verification unit runs the candidate mapping based on the verification data with confirmation results and obtains the verification results in the temporary verification area, which is isolated from the formal data area and the formal output interface. The temporary verification area can be an independent database, an independent schema space, or a temporary table with access control. Its write permissions are isolated from the formal data write flow. The temporary results formed cannot be called by the formal query or output interface before the candidate mapping takes effect.

[0063] When the old and new interfaces are used in parallel, the verification data includes old-format records and new-format records of the same data object and within the same time range. The temporary verification unit processes the corresponding records using the old mapping and the candidate mapping respectively and compares the results. When the old interface is no longer used, the verification data includes new-format samples and their confirmed target results, or new-format records and target results confirmed by humans. Having confirmed results means that it is possible to predetermine which target fields and field values ​​should be formed after the new-format records are correctly processed. If there is only historical raw data of the old format without corresponding new-format records, or only new-format records without target results confirmed by the data provider or humans, the candidate mapping cannot be automatically passed because these data cannot prove that the candidate mapping can correctly handle the changed field composition, hierarchical path and association relationship.

[0064] The temporary verification unit performs structural constraint checks, record correspondence checks, cross-source association checks, and output interface checks on the candidate mappings. It takes the target fields, target types, uniqueness, association integrity, and fields required for the formal output interface as hard constraints, and takes the missing optional fields and abnormal source values ​​as data quality checks. The check results are summarized in the order of record, candidate rule, association constraint group, and candidate mapping version.

[0065] The verification conditions are set separately for different target fields. The key object identifier field can be required to be non-empty and unique. The location description field can be required to have the correct field type and combination order. The region code field can be required to conform to the predetermined code range. If any hard constraint is not met, the corresponding candidate rule or associated constraint group cannot pass the verification. If the optional field is missing or individual source values ​​are abnormal, the corresponding record will be recorded separately and isolated. The main inspection contents and handling methods are shown in Table 3.

[0066] Validate Project Main inspection contents Post-processing Handling of failures or exceptions Structural constraints Target field type, length, required, uniqueness, and code range This item is retained through records. Locate the corresponding conversion or verification rule Record corresponding Key fields of the same data object and the results of splitting and combining them Confirm that the candidate mapping can handle the new format Candidate mappings are not working and rules need to be corrected. Cross-source association Association keys such as object identifier fields and association results Allow continued generation of multi-source association results Isolate affected related records; other sources continue processing. Output interface The interface requires specific fields, levels, and record status. Allow access to subsequent version switching Stopping the corresponding candidate output does not affect the dependency-free interface. Source data quality Cases such as fields being empty or values ​​not matching the declared type. Qualified records will continue to be verified. Exceptions are recorded separately and not confused with mapping errors. Mapping error The source value exists, but the target value is missing, incorrect, or misplaced. The corresponding conditions can be met when this type of error is not present. Candidate mapping is not working; record the error location and input sample.

[0067] Table 3. Key Validation Items and Handling Procedures for Candidate Mappings

[0068] The temporary verification unit determines that if a field in the source pattern is empty in a single record, it is considered that the source data is missing; if the source value does not conform to the data type declared by the data provider, it is considered that the source data is abnormal; and if the source field has a value that meets the requirements but the mapping result is missing, the type is incorrect, or the write position is incorrect, it is considered that the mapping error is incorrect. The source data abnormality only affects the corresponding record, and the mapping error causes the candidate rule involving the mapping error to fail the verification.

[0069] like Figure 6 As shown, each check first generates a check result for a single record, then aggregates them into specific candidate rules. Candidate rules with common association keys, primary and foreign keys, or integrity constraints are further summarized by association constraint groups, and finally, an overall result of the candidate mapping version is formed. If the source data is missing or abnormal, only the corresponding record can be marked. However, if the target field that must be formed is missing, the record cannot enter the formal data area. If the mapping error is incorrect, the candidate rule involving the error and its association constraint group will be prevented from passing the verification.

[0070] S6. Establish version effectiveness relationships according to the verification results, or isolate new format records that fail verification. When verification passes, the version selection and switching unit establishes the effectiveness relationship between the new source mode version and the candidate mapping. When verification fails, the candidate mapping remains ineffective, allowing new format records that depend on the candidate mapping to enter the waiting area, and allowing mappings that do not enter the affected scope to continue to be effective. The candidate mapping is managed according to the status of draft, pending confirmation, verification in progress, verification passed, effective, verification failed, and deactivated. Candidate rules with common association keys, primary and foreign keys, or integrity constraints are determined as association constraint groups. The version selection and switching unit only makes the candidate rules as a whole effective through the same switching transaction or the same effectiveness identifier when all candidate rules in the association constraint group have passed verification, and makes the same batch of data use the corresponding source mode version and mapping version. When verification fails, the corresponding new format record is written to the waiting area. After the candidate mapping is corrected and re-verified, the records that have not yet formed formal results are reprocessed using an idempotent key composed of source identifier, record identifier, source mode version, and receiving batch.

[0071] When switching, record the effective time, verification result, and applicable scope of the candidate mapping. The new batch of received data will only be visible after all candidate rules belonging to the same associated constraint group have been successfully written. If any rule fails to be written, the original effective version will be retained and the switching result will be revoked. If the old interface can still provide old format data, the old format data can continue to use the old mapping corresponding to its source mode version. If the old interface has been decommissioned, the system will stop using the old mapping to process the new format data and isolate the new format records that have not yet passed verification. Historical correct data that has entered the formal data area can continue to be used, but its data update time or current status should be recorded.

[0072] The pending processing area must at least store the record identifier, source identifier, reception time, source mode version, mapping version, failure reason, number of reprocessing attempts, and processing status. The system also stores the discovery time, change content, affected scope, candidate mapping, verification data source, various check results, effective time, and reprocessing results for each change, enabling subsequent rule corrections to locate the original input, failure location, and the version used at the time. Figure 7 As shown, a candidate mapping can only be switched to effective after it has entered the states of pending confirmation, verification, and verification passed from the draft stage. Candidate mappings that fail verification remain ineffective, and the corresponding new format records enter the processing area. After the candidate mapping is corrected and re-verified, the system first checks whether the record has formed a formal result based on the idempotent key, and only re-executes the mapping for records that have not yet formed a formal result.

[0073] Running Case

[0074] The following operational data is set to illustrate the system processing procedure. The platform accesses three independent sources provided by Department 1, Department 2 and Department 3, and associates records belonging to the same data object with the object identifier field.

[0075] Within the same verification time range, in the old format record with object identifier D20260001, the status field 1 provided by Department 1 is "normal" and the location description is "No. 88 Ankang Road, Huimin Street, Qingyun District, Haizhou City". The status field 2 provided by Department 2 is "verified" and the status field 3 provided by Department 3 is "valid". The above fields are written into the unified target pattern according to their respective effective mappings.

[0076] After upgrading the interface, Department 1 split the original single location description field into city "Haizhou City", district "Qingyun District", street "Huimin Street", and detailed address "No. 88 Ankang Road". The confirmed target result is the unified location description "No. 88 Ankang Road, Huimin Street, Qingyun District, Haizhou City" and the example area code "QY001". The data structure of Departments 2 and 3 remains unchanged.

[0077] After comparing the old and new source patterns of Department 1, the system confirmed that the change was a change in field splitting and interface level. Then, it searched along the current effective mapping of Department 1 to find the location combination rules, unified location description field, regional code field, basic information output interface and statistical items formed by regional code. The status fields of Department 2 and Department 3 do not use the location structure of Department 1, so they are not included in the scope of this change.

[0078] Candidate mappings only cover the location combination rules and area code rules of Department 1, and process new format records with confirmation results in the temporary verification area; the record with object identifier D20260002 has a street field in the source schema, but the street value of this record is empty, so the system determines it as missing source data; the city, district, street and detailed address of object identifier D20260003 all have the required values, but the candidate mapping does not form an area code, so the system determines it as a mapping error. The above processing is shown in Table 4.

[0079] Processing stage Department 1 Source Department Two Sources Department Three Sources Platform processing results Change recognition The location description has been split from a single field into multiple structured fields. The structure remained unchanged. The structure remained unchanged. Only the location field structure of department one is changed. Impact confirmed Location target fields, related interfaces, and statistical items are affected. No location structure dependency No location structure dependency Forming a localized affected area Candidate Mapping Forming location combinations and region code mapping Continue the currently active mapping Continue the currently active mapping Candidate mappings do not access formal data. verify Use the new format sample and corresponding confirmation results Continue normal processing Continue normal processing Distinguishing between missing source data and mapping errors Effective or Failed If successful, the process is partially switched; otherwise, the new record awaits processing. Unaffected Unaffected Avoid complete shutdown or complete reconstruction

[0080] Table 4. Processing of changes in multi-source data structure

[0081] When both the location combination rule and the region code rule pass verification, the system makes both effective as a whole through the same switching transaction or the same effective identifier, and starts using the new source mode version and candidate mapping version of Department 1 from the new receiving batch; when either rule fails or the switching write fails, both rules remain ineffective, the new format record corresponding to Department 1 enters the waiting area, and Department 2, Department 3 and other unaffected mappings continue to be processed.

[0082] After the candidate mapping is corrected and re-verified, the system generates an idempotent key based on the source identifier, record identifier, source mode version, and receiving batch. For records with missing source data, such as D20260002, the data quality status is retained. For records with mapping errors, such as D20260003, the corrected region code rules are used for reprocessing. Records that have already formed formal results are not written again. Records that have not yet formed formal results are updated with their processing status, number of reprocessing times, and result time after processing is completed.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-source public data integration system supporting dynamic pattern mapping, characterized in that, The system includes a source data access unit, a pattern and mapping recording unit, and a pattern change recognition unit; The source data access unit acquires data and source patterns from various public data sources; The pattern and mapping record unit stores the source pattern version, unified target pattern, effective mapping, and downstream dependencies; The pattern change identification unit compares the old and new source patterns from the same source to obtain the changed elements and the new source pattern version. The system also includes an affected scope determination unit, a candidate mapping formation unit, a temporary verification unit, and a version selection and switching unit; The affected scope determination unit starts with the changed element and searches step by step along the effective mapping and downstream dependencies to obtain the affected scope. The candidate mapping forming unit, based on the effective mapping, forms ineffective candidate mappings corresponding to the new source mode version only for the affected range; The temporary verification unit operates the candidate mapping and obtains the verification result based on the verification data with confirmation results in a temporary verification area isolated from the formal data area and the formal output interface; When the verification is successful, the version selection and switching unit establishes the effective relationship between the new source mode version and the candidate mapping. When the verification fails, the candidate mapping remains ineffective, allowing new format records that depend on the candidate mapping to enter the waiting area, and allowing mappings that have not entered the affected range to continue to be effective.

2. The multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, The source schema includes at least table or interface path, field name, data type, field length, hierarchical relationship, association relationship, and constraints. The schema change identification unit identifies changes in fields (addition, deletion, splitting, merging, data type or field length, hierarchical path, primary / foreign key or inter-table association relationship, and constraints) by comparing the new and old source schemas. Candidate mapping is not triggered when the interface version changes but the structural elements of the source schema remain unchanged.

3. A multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, The pattern and mapping recording unit records source fields, source relations, target fields, target relations, transformation rules, association processing, statistical items, and output interfaces as dependency nodes, and records field mapping, field combination or splitting, type conversion, record connection, statistical usage, and interface output as directed dependency edges. The affected scope determination unit starts from the dependency node corresponding to the changed element, expands only along the directed dependency edges that match the change type and satisfy the propagation conditions, and stops expanding when there are no subsequent dependencies or the subsequent dependencies are unrelated to the change type, thus obtaining the least affected closure.

4. A multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, The affected scope determination unit writes access markers to visited dependent nodes and directed dependent edges, and stops repeated expansion when the same dependent node is reached again; it takes the union of the minimum affected closures obtained from multiple changed elements, records the source of influence and marks rule conflicts for cases where the same target object is repeatedly affected by different paths, and expands fields with common association keys, primary and foreign keys or integrity constraints as a whole association constraint group.

5. A multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, The candidate mapping is based on the effective mapping and records the new source mode version, the rule identifier that was replaced, the added or modified rules, the applicable objects of each rule, the coverage priority, the conflict check result, the manual confirmation status and the verification status. When reading the mapping, the coverage rules in the candidate mapping are matched first, and then the unaffected rules in the base version are read.

6. A multi-source public data integration system supporting dynamic pattern mapping according to claim 5, characterized in that, When multiple candidate rules are matched for the same target field, the candidate mapping forming unit selects candidate rules according to the applicable source mode version, applicable object scope, and confirmed coverage priority; if the candidate rules cannot be uniquely determined according to the applicable source mode version, applicable object scope, and coverage priority, the conflicting candidate rules are kept in an inactive state and transferred to manual confirmation.

7. A multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, When the old interface and the new interface are used in parallel, the verification data includes old format records and new format records of the same data object and the same time range. The temporary verification unit processes the corresponding records using the old mapping and the candidate mapping respectively and compares the results. When the old interface is no longer in use, the verification data includes new format samples and their confirmed target results, or new format records and target results that have been manually confirmed.

8. A multi-source public data integration system supporting dynamic pattern mapping according to claim 7, characterized in that, The temporary verification unit performs structural constraint checks, record correspondence checks, cross-source association checks, and output interface checks on the candidate mappings. It takes the target fields, target types, uniqueness, association integrity, and fields required for the formal output interface as hard constraints, and takes the missing optional fields and abnormal source values ​​as data quality checks. The check results are summarized in the order of record, candidate rule, association constraint group, and candidate mapping version.

9. A multi-source public data integration system supporting dynamic pattern mapping according to claim 8, characterized in that, The temporary verification unit determines that if a field in the source pattern is empty in a single record, it is considered that the source data is missing; if the source value does not conform to the data type declared by the data provider, it is considered that the source data is abnormal; and if the source field has a value that meets the requirements but the mapping result is missing, the type is incorrect, or the writing position is incorrect, it is considered that the mapping is incorrect. The source data anomaly only affects the corresponding record, and the mapping error causes the candidate rule involving the mapping error to fail verification.

10. A multi-source public data integration system supporting dynamic pattern mapping according to claim 1, characterized in that, The candidate mappings are managed according to the following states: draft, pending confirmation, under verification, verified, effective, verified failed, and deactivated. Candidate rules with common association keys, primary-foreign keys, or integrity constraints are identified as association constraint groups. The version selection and switching unit only makes the candidate rules take effect as a whole through the same switching transaction or the same effective identifier when all candidate rules in the association constraint group have passed verification, and makes the same batch of data use the corresponding source schema version and mapping version. When verification fails, the corresponding new format record is written to the pending area. After the candidate mapping is corrected and re-verified, the records that have not yet formed a formal result are reprocessed using an idempotent key composed of source identifier, record identifier, source schema version, and receiving batch.

Images