A field-level rule-driven multi-mode government affair data fusion method and system
By using a field-level rule-driven multi-mode fusion method, we have solved the problems of coarse fusion granularity and hard-coded strategies in existing government data fusion, and achieved efficient and flexible data fusion and traceability capabilities to meet the complex needs of government scenarios.
Patent Information
- Application Number
- CN202610935680.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-26
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for integrating government data suffer from problems such as coarse granularity of integration, hard-coded strategies, the need for redeployment when rules change, and insufficient field-level traceability capabilities, which cannot meet the accuracy, flexibility, continuity, and performance requirements of government scenarios.
A multi-mode fusion method driven by field-level rules is adopted. By externalizing the fusion strategy into configurable and versionable rules, and combining the fusion execution engine with mechanisms such as rule caching and function caching, field-level conflict resolution and synchronous generation of traceability information are achieved.
It improves the accuracy and usability of the fusion results, reduces development and maintenance costs, ensures business continuity, enhances traceability, and improves processing efficiency and solution reusability.
Smart Images

Figure CN123069243A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing and data fusion technology, specifically providing a field-level rule-driven multi-mode government data fusion method and system. Background Technology
[0002] Cross-departmental integration of government data has become a core foundation for collaborative government services. The core objective of government data integration is to align, integrate, and disambiguate heterogeneous data from different departments to form an authoritative and unified data view, thereby providing data support for government services.
[0003] The current mainstream solution in the field of government data fusion is entity-level multi-source government data fusion technology. Its core features are: achieving the aggregation of multi-source government data through entity alignment, resolving conflicts based on data source weights and timestamps, and constructing entity-level data fusion results. However, this type of solution suffers from the following unresolved core defects in government application scenarios, which are also the technical problems specifically addressed by this invention: (1) Coarse granularity of fusion and insufficient data accuracy: Focusing on the fusion of datasets or entities, adopting a globally unified conflict resolution strategy, it is unable to adapt to the differences in business semantics of different fields, which can easily lead to data distortion and fail to meet the rigid requirements of government data for accuracy.
[0004] (2) The integration mode is simple and the flexibility is extremely poor: it usually only supports 1-2 simple conflict resolution strategies based on timestamps and data source weights. The integration logic is hard-coded in the program. Rule changes require redevelopment, testing and deployment. The response cycle is long and the maintenance cost is high. It cannot adapt to the complex and ever-changing business needs of government scenarios.
[0005] (3) Lack of rule-based hot update capability and poor business adaptability: rule changes require service restart, which can easily lead to business interruption and cannot meet the requirements of 24 / 7 uninterrupted service in government scenarios.
[0006] (4) Lack of traceability and insufficient compliance: After fusion, only the final result is retained, and the relationship between the result value and the original data source and original record is lost. Field-level reverse tracing cannot be achieved, and the statutory requirements for government data security compliance audit and problem backtracking cannot be met.
[0007] (5) Insufficient performance support and difficulty in engineering implementation: If existing technologies attempt to be integrated down to the field level, performance problems such as long processing time, high resource consumption, and high database I / O pressure will occur. Furthermore, there is a lack of standardized and lightweight engineering implementation solutions, making cross-platform migration and rapid replication and promotion difficult. It is unable to meet the needs of government scenarios for processing tens of millions and hundreds of millions of massive data and rapid implementation in multiple regions.
[0008] In summary, how to solve the problems of coarse granularity of existing solutions, hard-coded strategies, and insufficient traceability is an urgent issue for those skilled in the art. Summary of the Invention
[0009] This invention aims to address the problems in existing government data fusion methods, such as coarse granularity of fusion, hard-coded fusion strategies, the need for redeployment when rules are changed, insufficient field-level traceability, low efficiency of field-level fusion processing, high cost of manual rule configuration, and insufficient field semantic recognition. It provides a field-level rule-driven multi-mode government data fusion method and system.
[0010] The core design of this invention lies in using the target field as the smallest fusion control unit, externalizing the field fusion strategy into configurable and versionable fusion rules; encapsulating the specific fusion logic into a fusion processing unit that can be registered and dynamically invoked; the fusion execution engine resolves field-level conflicts of multiple source candidate values under the same proxy key based on the fusion rules, and simultaneously generates traceability information bound to the result value of the target field while generating the fusion result value; at the same time, it supports large-scale data processing through mechanisms such as rule caching, function caching, proxy key pre-sharding, parallel processing, and batch transaction writing.
[0011] The technical solution adopted by this invention to solve its technical problem is: A field-level rule-driven method for multi-modal government data fusion includes the following steps: S1. Data Acquisition and Preprocessing: Acquire heterogeneous data from multiple sources from at least two government business systems, clean and transform the data, standardize fields and determine valid values, generate preprocessed aligned data, and generate a unified proxy key across systems for the same government service object; S2. Rule and Mode Configuration: Construct field-level fusion rule configuration information and fusion mode configuration information, supporting field semantic metadata construction, field semantic tag generation, automatic candidate rule recommendation, confidence scoring, and manual confirmation for release. The field-level fusion rule configuration information uses a single target field as the smallest configuration granularity. Each fusion rule includes at least information for associating with the target field, source field mapping information, associated fusion mode identifier, fusion execution parameters, and rule version information. The fusion mode configuration information is used to store the mapping relationship between fusion modes and fusion processing units, achieving decoupling between fusion logic and engine code.
[0012] S3. Data Fusion Execution: The fusion execution engine loads the field-level fusion rule configuration information and fusion mode configuration information, parses, verifies, and pre-compiles the fusion rules, builds a field-level rule cache, and loads the fusion processing unit to build a fusion function cache; based on the proxy key, the preprocessed aligned data is grouped, and for each target field in each group of data, the corresponding fusion rule is matched from the field-level rule cache, and the corresponding fusion processing unit is dynamically called according to the fusion mode identifier to perform conflict resolution on multi-source candidate values and generate the fusion result value of the target field; at the same time as generating the fusion result value, the field-level traceability information bound to the fusion result value is generated synchronously to build an auditable lineage link of "fusion result value - source system - original record - rule version"; S4. Results and Traceability Output: Write the fusion result value into the fusion result storage area, synchronously write the field-level traceability information into the fusion traceability storage area, and simultaneously write the fusion full-process execution log into the execution log storage area.
[0013] In a preferred embodiment, the determination of valid values includes pre-setting filtering rules for different target fields. The filtering rules are used to remove field values that are null, empty strings, illegal encoding, non-compliant format, exceed the business value range, duplicate values, or do not meet business constraints, and only retain valid candidate values to participate in field-level fusion.
[0014] In a preferred embodiment, the proxy key is generated by an irreversible hash algorithm or other irreversible identifier conversion algorithm from the resident ID number, unified social credit code, business object number or combination thereof of the government service object, and a mapping relationship between the original identity identifier and the proxy key is established.
[0015] In a preferred embodiment, the fusion execution parameters are stored in JSON, key-value pairs, or other extensible data structures. The parameter content includes, but is not limited to, a data source priority list, a set of authoritative sources, a timestamp field, a string separator, deduplication rules, sorting rules, calculation expressions, primary and foreign key association conditions, anomaly degradation strategies, and default value strategies.
[0016] In a preferred embodiment, the field-level semantic reasoning and automated rule generation mechanism is used to construct a field semantic profile based on the target field name, field annotation, field data type, field value distribution characteristics, source system attributes, field update time characteristics, historical rule configuration, and historical fusion results; identify the semantic category of the target field based on the field semantic profile, and generate corresponding candidate fusion modes, candidate fusion parameters, and candidate rule versions; after the candidate rules undergo rule integrity verification, parameter legality verification, source system matching verification, and manual confirmation, they are written into the field-level fusion rule configuration information and enter the rule versioning management process.
[0017] In a preferred embodiment, the field semantic profile includes at least field name features, field annotation features, field data type features, field value format features, null value rate features, duplication rate features, source system credibility features, update time distribution features, and historical value consistency features; the field semantic categories include one or more of the following: identifier fields, identity attribute fields, contact information fields, address fields, status fields, time fields, amount fields, enumeration dictionary fields, calculation-derived fields, and primary / foreign key identifier fields.
[0018] In a preferred embodiment, the fusion processing unit has a standardized interface. The input includes a target field identifier, a set of candidate values, a source system identifier corresponding to the candidate value, original record location information, update time information, and fusion execution parameters. The output includes a fusion result value and field-level traceability information corresponding to the fusion result value.
[0019] In a preferred embodiment, the fusion mode supports registration-based extension; when adding a new fusion mode, a new fusion processing unit is developed according to the standardized interface, and the fusion mode identifier, mode name, fusion processing unit call path and activation status are registered in the fusion mode configuration information, and the fusion mode is bound to the target field through field-level fusion rules, without modifying the core code of the fusion execution engine.
[0020] In a preferred embodiment, the field-level fusion rules support versioned hot updates. When a fusion rule changes, a new rule version is generated for the changed fusion rule; the fusion execution engine performs version verification before the fusion task starts or at a preset check time, identifies the changed rules, asynchronously loads and builds a new rule cache copy; after the new rule cache copy passes integrity verification, fusion pattern matching verification, parameter validity verification, and function call validity verification, it replaces the old rule cache in an atomic switching manner, so that started tasks continue to use the old version cache, and newly started tasks use the new version cache.
[0021] In a preferred embodiment, the data fusion execution process employs a performance optimization mechanism combining pre-sharding, dual caching, parallel execution, and batch writing. Pre-sharding refers to generating shard indexes based on surrogate key hash values during the preprocessing stage; dual caching refers to constructing field-level rule caches and fusion function static caches; parallel execution refers to processing data in parallel by shard and performing parallel fusion on independent target fields within the same shard; batch writing refers to aggregating the fusion results, traceability information, and execution logs in a memory buffer and then committing them in batch transactions.
[0022] This invention also provides a field-level rule-driven multi-mode government data fusion system for implementing the method, comprising: The data preprocessing module is used to perform the data acquisition and preprocessing step S1; The field semantic reasoning and rule generation module is used to obtain one or more of the following information about the target field: field name, field annotation, field data type, field value sample, field value format features, null value rate, duplication rate, source system attributes, source system credibility, field update time features, historical rule configuration, and historical fusion results. It constructs a field semantic profile, identifies the field semantic category of the target field, and generates or recommends candidate fusion modes, candidate fusion execution parameters, and candidate field-level fusion rules. The rules and patterns management module is used to maintain the fusion rule configuration information and fusion pattern configuration information, execute the rules and patterns configuration step S2, and verify, confirm, manage the version, and publish the candidate field-level fusion rules. The fusion execution engine module is used to execute the data fusion execution step S3, and has built-in rule cache management unit, hot update control unit, parallel scheduling unit, fusion mode function library and traceability information management unit; The result and traceability output module is used to execute the result and traceability output step S4.
[0023] Compared with existing technologies, the field-level rule-driven multi-modal government data fusion method and system of the present invention have the following outstanding advantages: (1) Improve the matching degree between field fusion results and business semantics. This invention uses a single target field as the smallest fusion control unit and configures fusion modes and fusion parameters for different business semantic fields respectively, avoiding the problem of unreasonable field values caused by the unified fusion strategy at the entity level, which helps to improve the accuracy and usability of government data fusion results.
[0024] (2) Improve the efficiency of fusion rule adjustment and business adaptation. This invention externalizes the configuration of fusion rules, fusion parameters and fusion modes. The fusion execution engine dynamically calls the fusion processing unit according to the rules. When adding or adjusting the fusion mode, there is no need to modify the core code of the engine, which helps to reduce development and maintenance costs and improve business response speed.
[0025] (3) Improve business continuity during rule changes. This invention enables running fusion tasks and newly started fusion tasks to use different versions of the rule cache through rule versioning management, incremental loading, asynchronous cache copy construction and atomic switching, thereby reducing the impact of rule adjustments on running tasks.
[0026] (4) Improve the field-level traceability of the fusion results. This invention generates field-level traceability information simultaneously while generating the fusion result value of each target field, forming a link between the fusion result value, the source system, the original record, and the rule version, which facilitates data quality verification, audit tracking, and problem backtracking.
[0027] (5) Improve the efficiency of engineering processing for field-level fusion. This invention combines mechanisms such as surrogate key pre-sharding, field-level rule caching, fusion function caching, shard-level parallelism, field-level parallelism, and batch transaction writing to reduce runtime database access and repeated parsing overhead, which helps support large-scale government data fusion processing.
[0028] (6) Improve the ease of reuse and promotion of the solution. This invention can be engineered and encapsulated through standardized table structure, standardized fusion parameters, standardized fusion processing unit interface and visual configuration capabilities, and can be interfaced with mainstream data integration tools, making it easy to replicate and deploy in different regions or different government business scenarios. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a flowchart illustrating a field-level rule-driven multi-modal government data fusion method. Figure 2 This is a schematic diagram of the architecture of a field-level rule-driven multi-modal government data fusion method; Figure 3 This is a data preprocessing flowchart in a field-level rule-driven multi-modal government data fusion method. Figure 4 This is a schematic diagram of the fusion rule configuration process in a field-level rule-driven multi-mode government data fusion method. Figure 5 This is a schematic diagram of the data fusion operation process in a field-level rule-driven multi-mode government data fusion method. Figure 6 This is a schematic diagram of the output process of the fusion result in a field-level rule-driven multi-mode government data fusion method. Figure 7 This is a schematic diagram of field-level refined fusion control in a field-level rule-driven multi-mode government data fusion method. Figure 8 This is a schematic diagram of rule caching and hot updating in a field-level rule-driven multi-mode government data fusion method. Detailed Implementation
[0031] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] The following is a preferred embodiment: Example
[0033] like Figure 1-8 As shown in this embodiment, a field-level rule-driven multi-mode government data fusion method includes the following steps: S1. Data Acquisition and Preprocessing: Acquire heterogeneous data from multiple sources from at least two government business systems, clean and transform the data, standardize fields and determine valid values, generate preprocessed aligned data, and generate a unified proxy key across systems for the same government service object; S2. Rule and Mode Configuration: Construct fusion rule configuration information to configure fusion strategies at the smallest granularity of a single target field. Each fusion rule must include at least the information for associating the target field, the source field mapping information, the associated fusion mode identifier, the fusion execution parameters, and the rule version information; construct fusion mode configuration information to store the mapping relationship between fusion modes and fusion processing units, thereby decoupling the fusion logic from the engine code. S3. Data Fusion Execution: The fusion execution engine loads the fusion rules and fusion processing unit, completes rule pre-compilation and cache construction; based on the proxy key, the preprocessed data is grouped, and for each target field in each group of data, the fusion processing unit is dynamically called according to its fusion rules to perform conflict resolution, generate fusion result value, and at the same time generate field-level traceability information bound to the fusion result value; S4. Results and Traceability Output: Write the fusion result value into the fusion result storage area, synchronously write the field-level traceability information into the fusion traceability storage area, and simultaneously write the fusion full-process execution log into the execution log storage area.
[0034] The determination of valid values is specifically as follows: For each target field, preset filtering rules are set to automatically remove invalid field values such as null values, empty strings, illegal encoding, non-standard format, and values that exceed the business value range, and only valid values are retained for fusion.
[0035] The proxy key is generated from the resident ID number of the government service recipient using an irreversible encryption algorithm, establishing and storing a unique mapping relationship between the ID number and the proxy key.
[0036] The fusion execution parameters are stored using an extensible data structure, including data source priority, authoritative source set, timestamp field, deduplication rules or exception degradation strategy parameters.
[0037] The fusion mode supports registration-based extension: adding a new fusion mode only requires developing a standardized fusion processing unit, registering it in the fusion mode storage, and then binding it to the target field through fusion rule configuration, without modifying the core code of the fusion execution engine; When the fusion mode is priority-based fusion, the first non-empty valid value is extracted as the fusion result value according to the data source priority list defined in the fusion parameters. When the fusion mode is multi-value concatenation fusion, the candidate values from multiple sources are deduplicated and sorted according to the fusion parameters, and then concatenated using a specified delimiter. When the fusion mode is latest time fusion, the latest non-empty valid value is selected as the fusion result value according to the timestamp field specified in the fusion parameters.
[0038] The rules and patterns configuration also includes a rule versioning hot update step, specifically: When the configuration information of the fusion rule changes, a new version number is assigned to the changed rule; the fusion execution engine performs version verification before the task starts, identifies the incrementally changed rule, and asynchronously loads and builds a new rule cache copy; A consistency check is performed on the new cache copy. If the check passes, an atomic switching method is used to switch the rule cache from the old version to the new cache copy. The fusion task that is currently being executed continues to use the old version cache, while the newly started fusion task uses the new version cache, thus realizing hot update of rules.
[0039] The field-level tracing information generated in step S3 includes at least: information for identifying the target field, fusion result value, information for indicating the original source system, information for locating the original record, and information for associating fusion rules and their versions, which are used to construct an auditable lineage link of "fusion result value - source system - original record - rule version" for reverse tracing by field.
[0040] In the fusion execution process in step S3, a two-layer parallel execution mechanism of "shard-level parallelism + field-level parallelism" is adopted. Specifically, the data is distributed to multiple parallel processing channels based on the surrogate key hash pre-sharding to achieve shard-level parallelism. Within a single shard, the fusion logic of each target field is independent of each other, and field-level parallel execution is achieved by using multi-threading.
[0041] (i) The existing entity-level globally unified data fusion strategy cannot adapt to the differences in business semantics of different fields in government data, resulting in insufficient business rationality and data distortion in the fusion results; This application decentralizes the granularity of fusion control down to a single field, designing a full-link fusion architecture with the "target field" as the smallest unit. Through a structured fusion rule table, each target field is independently bound with fusion mode, execution parameters, priority, and version information, realizing differentiated conflict resolution strategies for different business semantic fields. At the same time, it constructs a three-dimensional fusion strategy system of "horizontal model unification, vertical entity convergence, and field-level value fusion" to achieve refined control over the entire process from model to value.
[0042] By decentralizing the granularity of fusion control to the target field, the degree of matching between the fusion results and business semantics can be improved, reducing the risk of data distortion caused by a globally unified fusion strategy, thereby enhancing the accuracy and adaptability of government data fusion results.
[0043] (ii) The existing technology has hard-coded integration logic, which only supports 1-2 simple conflict resolution strategies. When adding integration logic, the code needs to be modified and redeployed, which cannot adapt to the complex and ever-changing integration needs of government scenarios.
[0044] This application adopts a decoupled architecture of "abstract encapsulation of fusion mode + dynamic binding of rule table + registration-based extension". Each fusion logic is encapsulated as an independent, pluggable fusion function. The mapping relationship between mode identifier and fusion function is established through the fusion mode table. Eight general fusion modes for government scenarios are pre-built and can be extended through mode registration and rule configuration. When a new fusion mode needs to be added, there is no need to modify the core engine code. It only needs to be registered in the mode table and configured in the rule table to take effect.
[0045] It can be adapted to various integration scenarios in government data, and the iteration cycle of integration logic is shortened from hours to minutes, enabling rapid response to business needs.
[0046] (iii) The existing technology integration logic and program code are hard coupled, rule changes require redevelopment, testing and deployment, which can easily lead to business interruption and cannot adapt to the needs of frequent adjustments to government data rules.
[0047] When the engine starts, it creates an index based on the target field, completes the parsing and pre-compilation of the fusion rules, and builds a field-level rule cache. It designs a hot update mechanism of "version verification - incremental loading - replica verification - atomic switching". When the rule changes, a new version number is automatically generated, the engine asynchronously builds a new cache replica, and completes the switch after passing the consistency verification. The tasks being executed are not affected, the new tasks automatically use the new rules, and the rule version information is recorded throughout the process.
[0048] It can update and switch data fusion rules without interrupting existing fusion tasks, thereby improving rule iteration efficiency and reducing operational complexity.
[0049] (iv) Regarding the issue that the association between the result value and the original data source is lost after the integration of existing technologies, it is impossible to achieve field-level reverse tracing and meet the rigid requirements of government data security compliance audit and problem backtracking.
[0050] This invention integrates the atomic mechanism for execution and traceability information generation. Each fusion function, while returning the fusion result value, must simultaneously output information such as the original source system, rule ID and version number, and execution time corresponding to the result. The fusion traceability table enables the storage of a single traceability record for each target field, constructing a complete and auditable lineage link of "fusion result value - source system - original record - rule version", supporting multi-dimensional reverse tracing by target field, proxy key, and source system.
[0051] It enables reverse tracing of field-level fusion results, meets the application needs of government data compliance audit and problem backtracking, and improves the convenience of problem location and verification.
[0052] The overall integration process is as follows: S10, Data Acquisition and Preprocessing; The following steps are taken to clean, standardize, and align entities of multi-source heterogeneous government raw data to generate a qualified input dataset: S101. Multi-source data access: Obtain raw data from at least two government business systems, including service recipient business data from departments such as security, human resources and social security, medical insurance, and civil affairs; S102. Unified Proxy Key Generation: Based on the resident ID card number of the service object, a unified proxy key is generated across the system using the national cryptographic algorithm. A unique mapping relationship between the ID card number and the proxy key is established and stored in the proxy key mapping table. S103. Data Cleaning and Standardization: Perform cleaning and transformation operations on the raw data, including verification of the legality of the ID number, standardization of coding rules, marking of missing values, and removal of illegal / duplicate values; S104, Valid Value Determination and Data Entry: Perform valid value determination for each field, remove invalid data, associate the processed standardized data with the proxy key, source system identifier, and original record primary key, and write them in batches to the preprocessing alignment table.
[0053] S20, Rules and Mode Configuration; To achieve complete decoupling of the integration strategy from the code, and to transform business requirements into executable standardized rules, the specific steps are as follows: S201. Fusion Mode Registration and Management: Construct a fusion mode table, encapsulate each fusion logic into an independent pluggable fusion function, register it in the fusion mode table, and establish a mapping relationship between fusion modes and fusion functions; In this embodiment, eight general fusion modes for government scenarios are pre-set, which can support the registration and expansion of new fusion modes without modifying the core code of the fusion execution engine; S202. Field Semantic Profile Construction. The fusion execution engine or rule and pattern management module obtains the target field's field name, field comments, field data type, field length, field value samples, null value rate, duplication rate, format distribution, source system identifier, source system credibility, update time distribution, and historical fusion records to form a field semantic profile.
[0054] S203. Candidate Fusion Pattern Reasoning. Candidate fusion patterns are matched based on the semantic profile of the fields. For identity-related fields, the authoritative source fusion pattern is preferred; for contact information fields, the multi-value concatenation fusion pattern or the latest time fusion pattern is recommended; for address fields, the latest time fusion pattern or the priority-based fusion pattern is recommended; for status fields, the unique source fusion pattern, the authoritative source fusion pattern, or the primary / foreign key identifier fusion pattern is recommended; for calculated derivation fields, the calculated derivation fusion pattern is recommended.
[0055] S204. Candidate Fusion Parameter Generation. Based on source system credibility, field update time, field null value rate, field duplication rate, and historical consistency, candidate fusion parameters are generated. For example, a data source priority list is generated based on source system credibility; a timestamp field parameter is generated based on field update time; a deduplication strategy parameter is generated based on field value duplication rate; valid value judgment rules are generated based on field value format; and anomaly degradation and default value strategies are generated based on historical fusion anomaly records.
[0056] S205. Candidate Rule Verification and Release. Perform integrity verification, fusion pattern matching verification, fusion parameter validity verification, source field existence verification, and conflict rule detection on automatically generated candidate field-level fusion rules, and generate confidence scores for the candidate rules. Candidate rules that pass verification and whose confidence scores reach a preset threshold enter the manual confirmation or pending release state; confirmed candidate rules are written to the fusion rule table, generate rule version numbers, and enter the rule versioning hot update process; candidate rules that fail verification or whose confidence scores are below the preset threshold are written to the candidate rule generation table and marked as pending review.
[0057] S206. Rule Pre-compilation and Caching: When the fusion execution engine starts, it loads all fusion rules in the enabled state in batches, builds primary key indexes according to target fields, completes rule parsing and pre-compilation, and builds field-level rule cache; at the same time, it preloads fusion functions in the fusion mode table and builds static function cache.
[0058] S30, Data fusion execution, the fusion execution engine drives the resolution of field conflicts; The specific steps are as follows: S301, Data Grouping: Using the surrogate key as the aggregation basis, hash-shard and group the multi-source records in the preprocessed alignment table to ensure that all records of the same entity are encapsulated into an independent data packet to be merged; S302, Conflict Identification: Traverse each data packet to be fused, and perform valid value judgment on the multi-source candidate values of each target field one by one. If there are two or more inconsistent remaining valid values, trigger the conflict resolution process. S303, Rule Matching and Dynamic Dispatch: The engine loads the fusion rule corresponding to the enabled status of the target field from the rule cache, specifies the fusion mode and fusion parameters, and dynamically calls the corresponding pre-compiled fusion function; S304, Conflict Resolution Execution: The fusion function executes the corresponding conflict resolution logic according to the configured fusion parameters, and generates a unique fusion result value for the target field; S305. Synchronous generation of traceability information: When the fusion function returns the fusion result value, it simultaneously outputs the original source system identifier, original record primary key, rule ID and version number corresponding to the result, and atomically binds them to the fusion result value.
[0059] S40. Results and traceability output; To achieve persistent storage and end-to-end traceability of the fusion results, the specific steps are as follows: S401, Data Encapsulation: Encapsulate the fusion result values of all target fields under the same surrogate key into a standardized record, and associate it with the fusion batch number and fusion time; S402, Batch Transaction Write: The method of "in-memory aggregated data + batch transaction entry" is adopted to write the fusion results to the fusion result table in batches, write the atomically bound field-level traceability information to the fusion traceability table synchronously, and write the fusion execution log to the fusion log table to ensure the transaction consistency of the three tables. S403. Result Verification: Perform post-consistency verification on the data written to the fusion result table, including verification of required fields not being empty and verification of business value ranges, to ensure the accuracy of the fusion results.
[0060] The fusion rule table (dw_fusion_rule) stores field-level fusion rules. Core fields include fusion rule ID, target table name, target field name, source table name, source field name, fusion mode ID, fusion parameters (JSON), execution priority, enabled status, rule version number, operator, and update time.
[0061] Fusion Mode Table (dw_fusion_mode): Stores the mapping relationship between fusion modes and functions. Core fields include fusion mode ID, mode name, mode description, fusion function name, script path, enabled status, and version number.
[0062] The proxy key mapping table (dw_svc_recips_sk) stores the mapping relationship between the document number of the storage object and the proxy key. The core fields include proxy key, document number, document type, generation time, and activation status.
[0063] The preprocessing alignment table (dw_svc_recips_pre) stores the preprocessed data after cleaning, standardization, and entity alignment. The core fields include the business primary key, surrogate key, source system identifier, standardized field, sharding index, and ingestion time.
[0064] The fusion result table (dw_svc_recips_info) stores the final fusion result. The core fields include surrogate key, normalized field, fusion batch number, fusion time, and result status.
[0065] The fusion traceability table (dw_svc_recips_source) stores field-level lineage information. Core fields include traceability ID, proxy key, target field name, fusion result value, source system identifier, original record primary key, rule ID, rule version number, and record time.
[0066] The fusion log table (dw_fusion_log) records the execution logs of data fusion tasks. The core fields include log ID, task ID, proxy key, processing time, rule version number, exception information, and recording time.
[0067] Define the core terms: (1) Proxy key: A unique key value used to uniformly identify the same government service object across data sources. It is generated by the resident ID number of the service object through the national cryptographic irreversible algorithm. Its core function is to solve the problem of inconsistent identification of "multiple IDs for the same entity" in different government business systems and to provide a unified benchmark for data aggregation and integration. (2) Valid value determination: For each target field, a set of preset filtering rules is used to automatically remove field values that are null, empty strings, illegal encoding, non-standard format, or exceed the business value range, and only retain valid values to participate in the fusion; (3) Field-level fusion rules: Using a single target field as the smallest granularity, structured configuration records that bind fusion mode, fusion parameters, and execution attributes are uniformly stored in the fusion rule table to realize the externalized management of fusion strategies; (4) Fusion mode: An executable function that performs conflict resolution operation for multiple candidate values of the same target field. Each mode corresponds to a standardized conflict resolution logic, which can be flexibly selected according to the business semantics of the field. (5) Fusion parameters: A set of structured parameters used to control the execution details of the fusion mode. It is stored in JSON format and includes a list of data source priorities, a set of authoritative sources, a string separator, a timestamp field, deduplication rules, and anomaly degradation strategies. (6) Field-level tracing: During the fusion execution process, the final value of each target field is recorded synchronously, including its original source system, original record primary key, rule version, execution time, and other information, forming a field-level lineage that can be traced back. (7) Conflict resolution: For the problem of inconsistent candidate values from multiple sources for the same entity and the same target field, the process of executing the corresponding fusion logic according to the preset fusion rules to generate a unique and authoritative fusion result value.
[0068] (8) Engineering encapsulation: The core logic, table structure and configuration rules of the fusion engine are standardized and encapsulated to form a lightweight component that can be deployed across platforms, supports the connection with mainstream data integration tools, and realizes the convenient implementation of the solution.
[0069] To support the above methods, a performance optimization mechanism and engineering implementation plan are designed to form a relatively complete technical support system, including: 1. Using the target field as the smallest unit of fusion control, it breaks the limitations of traditional entity-level global fusion, enabling refined and differentiated configuration of fusion strategies, including: (1) Horizontal model unification layer: Standardize the table structure, field naming and data type of multi-source government data to form a unified target fusion model, solve the problem of cross-departmental data model heterogeneity, and lay the foundation for field-level fusion; (2) Vertical entity aggregation layer: Based on the surrogate key, the multi-source entity records of the same government service object are accurately aggregated to ensure the uniqueness of the merged object and avoid cross-entity data confusion; (3) Field-level value fusion layer: Configure fusion rules independently for each target field, and bind differentiated fusion modes and parameters according to the business semantics of the field (identifier class, status class, time series class, calculation class, etc.) to achieve refined conflict resolution of "one field, one strategy".
[0070] The three-tiered system is progressive, enabling refined control over the entire process from data modeling to entity aggregation and field value retrieval. This helps improve the matching degree between the fusion results and the business semantics of the fields, and reduces the risk of data distortion caused by fusion.
[0071] 2. Adopting a design approach of "decoupling + encapsulation + registration," the fusion logic and core engine code are decoupled, and extensions are supported through standard interfaces and registration configurations, including: (1) Abstraction and encapsulation of fusion logic: Define a standardized fusion function interface. The input parameters are a list of multi-source candidate values and fusion parameters in JSON format. The output parameters are the fusion result value and traceability information. All fusion logic is encapsulated into independent, pluggable fusion functions according to this interface. (2) Pattern and function mapping: Construct a fusion pattern table, establish a one-to-one mapping relationship between fusion patterns and fusion functions, and the engine realizes dynamic calling of fusion functions through pattern identifiers; (3) Registration-based extension: Eight general integration modes for government scenarios are pre-set. When a new integration mode is added, only the integration function needs to be developed according to the standard interface, the integration mode table needs to be registered, and then the integration rule table needs to be bound to the target field to take effect. No need to modify the core code of the integration execution engine.
[0072] The eight pre-defined fusion modes of this mechanism are shown in the table below: Table 1:
[0073] 3. Achieve a smooth implementation of rule changes, primarily addressing the issues of service restarts and business interruptions required for rule adjustments, including: (1) Version management: Each rule in the rule table is assigned a unique version number. When a rule is changed (mode switching, parameter adjustment, etc.), a new version number is generated and the update time, change content, and operator are recorded to achieve traceable management of the rule change process. (2) Incremental version verification: Before each batch of fusion tasks is started, the engine automatically compares the version number of the cache rules with the version number of the rule table, and only performs incremental loading on the rules that have undergone version changes, so as to reduce unnecessary full loading overhead. (3) Asynchronous replica construction: The engine loads, parses and pre-compiles the rules asynchronously through an independent thread to build a new rule cache replica, which is isolated from the main fusion task to avoid affecting the fusion operation being executed; (4) Consistency verification: Perform multi-dimensional verification on the new cache copy, including rule integrity, pattern identifier matching, fusion parameter legality, function call validity, etc. If the verification fails, the switch will be abandoned. Only after the verification passes can it enter the effective stage. (5) Atomic switching takes effect: The atomic switching method is adopted to switch the rule cache from the old version to the new copy; the tasks that are being executed continue to use the old version cache, and newly started tasks automatically enable the new version cache, thereby achieving a smooth transition in the rule update process.
[0074] 4. Atomized binding of fusion execution and traceability generation ensures a one-to-one correspondence between fusion results and traceability information, achieving field-level end-to-end traceability, including: (1) Atomic generation of traceability information: The traceability information generation logic is embedded in each fusion function. When the fusion function returns the fusion result value, it must simultaneously output standardized traceability information, including target field name, fusion result value, original source system, original record primary key, rule ID, rule version number, and execution time, so as to realize "result generation is traceability generation" and avoid traceability information loss or mismatch. (2) Structured storage of traceability information: Construct a fusion traceability table, store an independent traceability record for each fusion result of each target field, realize refined management of traceability information, and support independent traceability of single field; (3) Multi-dimensional reverse tracing: Based on the fusion traceability table, a complete lineage link of "fusion result value - source system - original record - rule version" is constructed. It supports reverse tracing by multiple dimensions such as proxy key, target field name, source system, and fusion batch number, which can quickly locate the original data source of the fusion result and the basis for rule execution. (4) Linking source tracing with logs: Linking source tracing information with the integrated log table allows for simultaneous viewing of the execution logs and exception information of the field during the tracing process, achieving the integration of "source tracing + problem investigation".
[0075] 5. To address the performance bottlenecks of the field-level fusion architecture, a four-layer core optimization mechanism was designed to ensure efficient execution of massive data fusion: (1) Pre-sharding optimization: In the preprocessing stage, data pre-sharding is completed based on the surrogate key hash value, and sharding indexes are established to avoid full table scans during fusion execution, thereby improving data grouping efficiency; (2) Dual caching optimization: Construct a dual caching system of "field-level rule caching + fusion function static caching", complete pre-compilation and pre-loading when the engine starts, reduce repeated access to the database at runtime, and improve the efficiency of rule query and function call; (3) Parallel execution optimization: A two-layer parallel architecture of "shard-level parallelism + field-level parallelism" is adopted. The data is distributed to multiple parallel channels according to the pre-sharding. The field fusion logic within a single shard is executed independently in parallel to improve the fusion processing efficiency. (4) Batch write optimization: The “memory aggregated data + batch transaction entry” mode is adopted. First, the data to be entered into the database is aggregated in the memory buffer. After the preset batch threshold is reached, the database batch transaction is committed to write the data at once, so as to reduce database I / O operations and improve writing efficiency.
[0076] 6. Focusing on the core objectives of lightweight design, standardization, and cross-platform compatibility, construct a comprehensive engineering system covering the entire process from technical implementation to deployment and promotion, enabling rapid replication and large-scale deployment of solutions, including: (1) Lightweight technology implementation: The fusion execution engine can be developed using programming languages such as Python and modularly encapsulated to facilitate deployment and maintenance, and supports running on mainstream operating systems such as Windows and Linux; (2) Standardized table structure and configuration: The seven core data tables, including the fusion rule table, fusion mode table, and proxy key mapping table, are designed in a standardized manner. A unified table structure, field naming, and data type specification are formulated. At the same time, a standardized configuration specification for fusion rules and fusion parameters is formulated to avoid the problem of difficult implementation caused by personalized configuration. (3) Compatibility with mainstream tools: The engine is designed with standardized input and output interfaces, which can be connected with mainstream data integration / ETL tools such as Apache NiFi, DataX, and Kettle, making it easy to integrate into the existing government data processing system; (4) Visual configuration and operation and maintenance: During the implementation process, a lightweight visual configuration platform can be developed to complete operations such as registration of fusion mode, configuration of field-level rules, and management of rule versions through the management interface; at the same time, a visual operation and maintenance interface is provided to support functions such as monitoring of fusion tasks, traceability query, and anomaly alarm, so as to reduce the threshold of use; (5) Reusable engineering components: The fusion engine, standardized table structure and related configuration capabilities can be encapsulated into reusable engineering components. When deployed, the rules can be configured in combination with specific government scenarios to improve deployment efficiency. Example
[0077] A multi-modal government data fusion system, implementing the above methods, is implemented in Python and used in conjunction with Apache NiFi or DataX data integration tools, supporting cross-platform deployment and reuse. It includes: The data preprocessing module is used to perform the data acquisition and preprocessing step S1; The rules and patterns management module is used to maintain the fusion rule configuration information and fusion pattern configuration information, and to execute the rules and patterns configuration step S2; The fusion execution engine module is used to execute the data fusion execution step S3, and has built-in rule cache management unit, hot update control unit, parallel scheduling unit, fusion mode function library and traceability information management unit; The result and traceability output module is used to execute the result and traceability output step S4.
[0078] The core scheduling module (scheduler) serves as the core for task coordination and distribution, managing and controlling the entire execution logic of the fusion tasks. It groups and splits the dataset based on the service object proxy key, achieving distributed parallel execution through multi-channel task distribution and dynamic load balancing. Simultaneously, it handles scheduled execution and manual triggering commands, coordinating task dependency management and resource scheduling to ensure the orderly triggering and efficient execution of fusion tasks.
[0079] Fusion Execution Module (Executor): As the core execution unit of the fusion logic, it is responsible for the tasks distributed by the distributed execution scheduling module and for resolving data conflicts. It calls the corresponding functions for different fusion modes to specifically implement the conflict resolution business logic of the specified mode, and at the same time completes the standardized encapsulation of the fusion results to provide qualified output data for the result output module.
[0080] The rule hot update module is responsible for rule version verification, incremental rule loading, new cache copy construction, consistency verification, and atomic switching operations. Combined with the rule cache construction, eviction and cleanup, and failure fallback mechanism, it achieves smooth switching during the rule update process, avoids interference with the execution of fusion tasks, and ensures the continuity of system business.
[0081] Pattern loading module: Responsible for loading the fusion pattern configuration, associating and binding the corresponding fusion execution function, providing pattern-based logic support for the fusion execution module, and ensuring the scalability and independence of the fusion logic.
[0082] The traceability management module is responsible for the standardized generation, format verification, and temporary storage of traceability information. It ensures the atomic binding of traceability information and fusion results, provides qualified data for writing to the fusion traceability table, and supports the traceability query requirements of the upper layer's fusion results.
[0083] General tools module: Provides general technical support for the entire process of the fusion execution engine, such as encapsulating general functions such as data format verification, parameter format conversion, runtime log generation, exception capture and fault tolerance handling, effectively improving code reusability and reducing the overall system maintenance cost. Example
[0084] This embodiment takes the fusion of resident data from three government departments—security, human resources and social security, and medical insurance—as an example to fully demonstrate the entire execution process of the present invention. Those skilled in the art can replicate this embodiment.
[0085] Implementation Example Basic Information: Integration Goal: To build a unified information view for service recipients, including 5 core fields: name, ID number, mobile phone number, place of residence, and personnel status; Data sources: Security and Household Registration Management System, Human Resources and Social Security Management System, Medical Insurance Benefits Distribution System; Operating environment: 4-node cluster, each node with 16 cores and 32GB of RAM, MySQL 8.0 database, Python 3.9 development engine.
[0086] Example Table Structure Design: (1) Fusion rule table (dw_fusion_rule): Stores field-level fusion rules. Core fields include fusion rule ID, target table name, target field name, source table name, source field name, fusion mode ID, fusion parameters (JSON), execution priority, enabled status, rule version number, operator, and update time.
[0087] (2) Fusion mode table (dw_fusion_mode): Stores the mapping relationship between fusion modes and functions. The core fields include fusion mode ID, mode name, mode description, fusion function name, script path, enabled status, and version number.
[0088] (3) Proxy key mapping table (dw_svc_recips_sk): Stores the mapping relationship between the ID number of the storage object and the proxy key. The core fields include proxy key, ID number, ID type, generation time, and activation status.
[0089] (4) Preprocessing alignment table (dw_svc_recips_pre): Stores preprocessing result data after cleaning, standardization and entity alignment. Core fields include business primary key, proxy key, source system identifier, standardization field, sharding index and entry time.
[0090] (5) Fusion Result Table (dw_svc_recips_info): Stores the final fusion result. The core fields include surrogate key, normalized field, fusion batch number, fusion time, and result status.
[0091] (6) Fusion source table (dw_svc_recips_source): Stores field-level lineage information. Core fields include source ID, proxy key, target field name, fusion result value, source system identifier, original record primary key, rule ID, rule version number, and record time.
[0092] (7) Fusion Log Table (dw_fusion_log): Records the execution log of data fusion tasks. The core fields include log ID, task ID, proxy key, processing time, rule version number, exception information, and recording time.
[0093] (8) Field semantic metadata table (dw_field_semantic_meta): Used to store field semantic profiles. Core fields include metadata ID, target table name, target field name, field name, field comment, field data type, field value format characteristics, null value rate, duplication rate, source system identifier, semantic category, semantic tag, and update time.
[0094] (9) Candidate rule generation table (dw_fusion_rule_candidate): Used to store automatically generated or recommended candidate field-level fusion rules. The core fields include candidate rule ID, target table name, target field name, recommended fusion mode ID, recommended fusion parameters, confidence score, generation basis, verification result, review status, confirmer, and generation time.
[0095] The specific implementation steps are as follows: Step 1: Data Preprocessing Proxy key generation: Based on the resident ID number 110101199001011234, a proxy key is generated using the national cryptographic SM3 algorithm, such as SK202400000001; Data cleaning and standardization: After cleaning and valid value determination of the raw data from the three systems, the data in the preprocessed alignment table is as follows: Table 2:
[0096] Step 2: Generating Fusion Rules Example of field semantic recognition: Table 3:
[0097] The field-level fusion rules configured for the five target fields are as follows (stored in the fusion rule table): Table 4:
[0098] Step 3: Integration of Execution and Conflict Resolution Using the surrogate key SK202400000001 as the grouping benchmark, perform field-by-field fusion on the five fields: (1) Name field: The execution priority is merged step by step. According to the priority of "security → human resources and social security → medical insurance", the first non-empty valid value "Zhang San" is extracted. The source information is: source system (security system), original record primary key (GA001), rule version (V1.0). (2) ID number field: Perform authoritative source fusion, only take the value "110101199001011234" from the security system, traceability information: source system (security system), original record primary key (GA001), rule version (V1.0); (3) Mobile phone number field: Perform multi-value concatenation and fusion, deduplication and sort in reverse order of update time, concatenate with semicolons to form "13900139000; 13800138000", traceability information: source system (human resources and social security system, security system), original record primary key (RS002, GA001), rule version (V1.0); (4) Place of residence field: Perform the latest time fusion and take the latest value of the human resources and social security system "XX Road, Xicheng District, Beijing" with the latest update time. Source information: source system (human resources and social security system), original record primary key (RS002), rule version (V1.0). (5) Personnel status field: Perform unique source fusion, take the value "employed" from the human resources and social security system, trace the source information: source system "human resources and social security system", original record primary key "RS002", rule version (V1.0).
[0099] Step 4: Results and Source Traceability Output The final fusion result table is recorded as follows: Table 5:
[0100] The merged traceability table stores the following records: Table 6:
[0101] As can be seen from the above; We have comprehensively innovated and optimized existing government data fusion technologies to address their core shortcomings, while building a performance assurance and engineering promotion system. This has enabled breakthroughs in multiple dimensions, including data accuracy, business adaptability, compliance auditing, performance efficiency, and engineering implementation, providing a complete and feasible technical solution for cross-departmental data fusion in digital construction.
[0102] This application achieves differentiated fusion with "one policy per field" through a field-level refined fusion management architecture. It breaks free from the limitations of entity-level global unified policies, reduces the limitations brought by entity-level global unified policies, improves the matching degree between fusion results and field business semantics, and reduces the risk of field value distortion caused by global unified policies, thereby providing more reliable data support for government services and government decision-making.
[0103] The plug-in multi-mode fusion mechanism decouples the fusion logic from the code. The eight pre-built fusion modes can cover various government fusion scenarios and support expansion without modifying the core code of the fusion execution engine. The rule versioning hot update mechanism enables smooth updates during rule changes, improving the efficiency of rule iteration and response to business needs, and is well adapted to the application requirements of government scenarios where business changes are rapid and continuous operation is required.
[0104] The field-level end-to-end lineage tracing mechanism enables field-level tracing of fusion results, constructing an auditable lineage link of "fusion result value - source system - original record - rule version", which facilitates problem investigation and compliance verification.
[0105] By employing mechanisms such as pre-sharding, caching, parallel execution, and batch writing, the efficiency of field-level fusion processing can be improved to a certain extent, reducing database access pressure. This approach is suitable for fusion processing scenarios involving large-scale government data.
[0106] The system adopts a cross-platform deployment approach, allowing for standardized design of core table structures and configurations, and enabling integration with mainstream data integration tools. In the preferred implementation, it also includes visual configuration and maintenance capabilities to lower the barrier to entry and improve deployment convenience.
[0107] To achieve automated and refined management of the entire process of government data integration, reduce repetitive work in manual development, rule adjustment, and problem investigation, while improving data integration processing efficiency and data quality, and providing data support for cross-departmental data sharing, "one-stop online service", smart government and other applications.
[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A field-level rule-driven multi-modal government data fusion method, characterized in that, Includes the following steps: S1. Data Acquisition and Preprocessing: Acquire heterogeneous data from multiple sources from at least two government business systems, clean and transform the data, standardize fields and determine valid values, generate preprocessed aligned data, and generate a unified proxy key across systems for the same government service object; S2. Rule and Mode Configuration: Construct fusion rule configuration information to configure fusion strategies at the smallest granularity of a single target field. Each fusion rule must include at least the information for associating the target field, the source field mapping information, the associated fusion mode identifier, the fusion execution parameters, and the rule version information; construct fusion mode configuration information to store the mapping relationship between fusion modes and fusion processing units, thereby decoupling the fusion logic from the engine code. S3. Data Fusion Execution: The fusion execution engine loads the fusion rules and fusion processing units to complete rule pre-compilation and cache construction; Based on the surrogate key, the preprocessed data is grouped. For each target field in each group of data, the fusion processing unit is dynamically invoked according to its fusion rules to perform conflict resolution, generate fusion result value, and generate field-level traceability information bound to the fusion result value. S4. Results and Traceability Output: Write the fusion result value into the fusion result storage area, synchronously write the field-level traceability information into the fusion traceability storage area, and simultaneously write the fusion full-process execution log into the execution log storage area.
2. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, In step S1, the determination of valid values specifically involves: setting up filtering rules for each target field, automatically removing invalid field values such as null values, empty strings, illegal encoding, non-compliant formats, and values exceeding the business value range, and retaining only valid values for fusion.
3. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, In step S1, the proxy key is generated from the resident ID number of the government service recipient using an irreversible encryption algorithm, and a unique mapping relationship between the ID number and the proxy key is established and stored.
4. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, The fusion execution parameters are stored using an extensible data structure, including data source priority, authoritative source set, timestamp field, deduplication rules or exception degradation strategy parameters.
5. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, It also includes field-level semantic reasoning and automated rule generation steps, which include: obtaining one or more of the following for the target field: field name, field annotation, field data type, field length, field value sample, field value format features, null value rate, duplication rate, source system attributes, source system credibility, field update time features, historical rule configuration, and historical fusion results, and constructing a field semantic profile; Based on the semantic profile of the fields, the semantic category of the target fields is identified. The semantic category of the fields includes one or more of the following: identifier fields, identity attribute fields, contact information fields, address fields, status fields, time fields, amount fields, enumeration dictionary fields, calculation-derived fields, and primary / foreign key identifier fields. Based on the semantic category of the field, candidate fusion modes, candidate fusion execution parameters, and candidate field-level fusion rules are generated or recommended. The candidate field-level fusion rules are then subjected to rule integrity verification, parameter validity verification, source field matching verification, and fusion mode matching verification. Once the verification is passed, the fusion rule configuration information is formed.
6. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, When the fusion mode is priority-based fusion, the first non-empty valid value is extracted as the fusion result value according to the data source priority list defined in the fusion parameters; when the fusion mode is multi-value concatenation fusion, the candidate values from multiple sources are deduplicated and sorted according to the fusion parameters, and then concatenated using a specified delimiter. When the fusion mode is latest time fusion, the latest non-empty valid value is selected as the fusion result value based on the timestamp field specified in the fusion parameters.
7. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, It also includes the rule versioning hot update step, specifically: When the configuration information of the fusion rule changes, a new version number is assigned to the changed rule; the fusion execution engine performs version verification before the task starts, identifies the incrementally changed rule, and asynchronously loads and builds a new rule cache copy; A consistency check is performed on the new cache copy. If the check passes, an atomic switching method is used to switch the rule cache from the old version to the new cache copy. The fusion task that is currently being executed continues to use the old version cache, while the newly started fusion task uses the new version cache, thus realizing hot update of rules.
8. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, The field-level traceability information generated in S3 includes at least: information for identifying the target field, fusion result value, information for indicating the original source system, information for locating the original record, and information for associating fusion rules and their versions. This information is used to construct an auditable lineage link of "fusion result value - source system - original record - rule version" for reverse tracing by field.
9. The multi-mode government data fusion method driven by field-level rules according to claim 1, characterized in that, The fusion execution process in S3 adopts a two-layer parallel execution mechanism of "shard-level parallelism + field-level parallelism". Specifically, data is distributed to multiple parallel processing channels based on surrogate key hash pre-sharding to achieve shard-level parallelism; within a single shard, the fusion logic of each target field is independent of each other, and field-level parallel execution is achieved by using multi-threading.
10. A field-level rule-driven multi-mode government data fusion system, used to implement the method described in any one of claims 1-9, characterized in that, include: The data preprocessing module is used to perform the data acquisition and preprocessing step S1; The field semantic reasoning and rule generation module is used to obtain one or more of the following information about the target field: field name, field annotation, field data type, field value sample, field value format features, null value rate, duplication rate, source system attributes, source system credibility, field update time features, historical rule configuration, and historical fusion results. It constructs a field semantic profile, identifies the field semantic category of the target field, and generates or recommends candidate fusion modes, candidate fusion execution parameters, and candidate field-level fusion rules. The rules and patterns management module is used to maintain the fusion rule configuration information and fusion pattern configuration information, execute the rules and patterns configuration step S2, and verify, confirm, manage the version, and publish the candidate field-level fusion rules. The fusion execution engine module is used to execute the data fusion execution step S3, and has built-in rule cache management unit, hot update control unit, parallel scheduling unit, fusion mode function library and traceability information management unit; The result and traceability output module is used to execute the result and traceability output step S4.
