Cross-domain data integration and fusion method, terminal and storage medium based on large model

Abstract

The present invention provides a cross-domain data integration and fusion method, terminal and storage medium based on a large model, the method comprising: performing format conversion and cleaning of cross-domain raw data through a unified data access adaptation layer, and outputting a standardized data set; utilizing a large model for semantic understanding and integration, parsing field names and generating field semantic labels in combination with a business terminology knowledge base, determining business relevance by calculating similarity, and generating inter-domain field mapping relationships; performing technical standard conversion and data fusion processing based on the mapping relationship, and performing intelligent quality governance by combining a quality governance standard library with a large model. The present invention effectively solves technical problems such as insufficient semantic understanding and inconsistent standards in existing cross-domain data integration by introducing a large model to assist in data understanding and governance, and significantly improves the accuracy and automation level of data integration.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-domain data integration and fusion method based on a large model, a terminal, and a storage medium. Background Art

[0002] As digital transformation deepens, enterprises face growing demand for data integration and fusion across departments, systems, and platforms. Cross-domain data integration refers to the unified management and integration of heterogeneous data distributed across different data domains (e.g., different business systems, departments, and organizations). The rapid development of big model technology in recent years has provided new technical means to address the complex challenges of cross-domain data integration, potentially overcoming the limitations of traditional approaches.

[0003] However, existing cross-domain data integration solutions have the following key technical difficulties:

[0004] First, data semantic understanding is insufficient. Existing technologies primarily rely on predefined mapping rules and manual experience for field matching. This approach struggles to accurately understand and address semantic differences between different data domains. In practice, due to the diversity and complexity of business terminology, rule-based mapping methods often require extensive manual intervention, reducing efficiency and making it difficult to guarantee mapping accuracy.

[0005] Second, the lack of unified technical standards is a prominent issue. Different data domains often use different data formats, encoding standards, and technical specifications. This heterogeneity makes it difficult to ensure consistency during data integration. During data conversion and fusion, the lack of unified standards and specifications can easily lead to data distortion, format errors, and other issues, impacting the quality of data integration.

[0006] Third, data quality governance lacks intelligence. Traditional data governance approaches rely primarily on static rule bases for quality checks and remediation, which cannot effectively address emerging data quality issues. As data scale and complexity increase, the types of quality issues are constantly evolving, and fixed governance rules are unable to meet dynamically changing governance needs.

[0007] Based on the above analysis, there is an urgent need for a cross-domain data integration solution that can provide intelligent semantic understanding, unified technical standard conversion, and adaptive quality management. This invention addresses these technical difficulties and proposes an innovative technical solution based on a large model. Summary of the Invention

[0008] The purpose of this invention is to provide a cross-domain data integration and fusion method based on a large model to solve technical problems existing in the existing technology, such as insufficient semantic understanding, inconsistent technical standards, and lack of intelligent quality management, and to improve the accuracy and automation level of cross-domain data integration.

[0009] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] A cross-domain data integration and fusion method based on a large model includes the following steps:

[0011] Access integration: receiving cross-domain raw data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting standardized data sets. The standardized data sets use a unified naming format, data types are mapped to types supported by the system, data content is complete and has no duplicate values, and missing values ​​are uniformly marked.

[0012] Semantic understanding integration: Based on a large model, the characteristics of standardized data sets in each data domain are analyzed, and the field data characteristics in each domain are extracted. By parsing the field names and combining them with the business terminology knowledge base, the business meaning of the fields in each domain is identified and field semantic labels are generated. The business relevance is determined by calculating the field name similarity and combining the field semantic labels, and the field mapping relationship between domains is generated.

[0013] Technical standard integration, based on the field semantic labels, performs unified technical standard conversion on the standardized data set, including unified data format specifications and encoding standards, and outputs standardized data that conforms to the unified technical specifications;

[0014] Data fusion processing, based on the standardized data of the unified technical specifications and the inter-domain field mapping relationship, identifies and matches data records of the same entity across domains, merges the attributes of the matching entities according to the data domain rights and responsibilities, and generates fused entity data;

[0015] Quality governance, based on the quality governance standard library, matches and repairs the problem features of the fused entity data, executes standard repair rules for successfully matched quality problems, uses the large model to analyze unmatched problems, identifies quality problems of entity attribute value consistency and entity relationship correctness, generates and executes repair rules, updates quality problems and processing methods to the quality governance standard library, and outputs governed high-quality data.

[0016] The large model is a language model that has been pre-trained with massive data and has the ability to understand cross-domain knowledge, and can perform intelligent analysis of data types, format specifications and business rules;

[0017] A further technical solution is that the large model processing in the semantic understanding integration step specifically includes:

[0018] Extract the data characteristics of each field in each domain, classify it into numeric type, character type, date and time type, and Boolean type through analysis of the field data, and determine the valid range of each type of field based on statistical distribution;

[0019] Parsing and semantically identifying the field names in each domain, and generating field semantic labels including data domain, business type, and field attributes in combination with the business terminology knowledge base;

[0020] Generate inter-domain field mapping relationships, calculate the similarity of field names between different data domains based on the edit distance algorithm, determine the business association relationship between fields between different data domains based on the field semantic labels, convert the inter-domain field name similarity and business association degree into feature vectors, and determine the inter-domain field mapping relationship by calculating the Euclidean distance between the field mapping feature vector to be determined and the standard mapping sample feature vector in the verified field mapping set.

[0021] A further technical solution is that the following basic configurations must be completed before the method is executed:

[0022] Build a business terminology knowledge base, including standard term definitions, field naming rules, and business mapping relationships;

[0023] Establish a collection of validated field mappings to guide new field mappings;

[0024] Configure the technical specification conversion benchmark library, including numerical specifications, text specifications, time specifications, and classification specifications;

[0025] Set the priority order between data domains for conflict resolution when merging data attributes;

[0026] Initialize the quality governance standard library, which includes a problem feature pattern library, a standard repair rule library and a problem-rule mapping relationship table. The problem feature pattern library is used to identify data consistency, completeness, accuracy and correlation issues. The standard repair rule library contains processing rules corresponding to problem features. The problem-rule mapping relationship table is used to achieve rapid matching of quality problems and processing methods.

[0027] A further technical solution is that the technical standard integration steps specifically include:

[0028] Initializing the conversion environment, and reading the numerical specification, text specification, time specification, and classification specification in the technical specification conversion benchmark library;

[0029] Based on the field semantic tags, the target conversion specification of each field is retrieved and determined from the technical specification conversion benchmark library;

[0030] Perform data format conversion operations according to the specified target specifications.

[0031] A further technical solution is that the data fusion processing steps specifically include:

[0032] Analyzing the business attributes in each data domain based on the field semantic tags to identify fields with unique identification characteristics, wherein the unique identification characteristic refers to a constraint characteristic that field values ​​are not repeated;

[0033] Associating the unique identification field through the inter-domain field mapping relationship to generate an entity identification field mapping set, and performing cross-domain data record matching based on the mapping set;

[0034] Based on the inter-domain field mapping relationship and the data domain authority and responsibility relationship, attribute values ​​of different data records of the same entity are merged. When attribute values ​​in the authority and responsibility data domain are missing, valid values ​​in other domains are adopted according to the data domain priority.

[0035] The merged attribute values ​​are integrated to form fused entity data.

[0036] A further technical solution is that the quality management steps specifically include:

[0037] Perform problem feature matching on the fused entity data to determine whether there are quality issues that match the problem features recorded in the quality management standard library. For quality issues with matching features, execute the repair rules in the standard library.

[0038] For quality issues not covered in the standard library, the large model is used to analyze the fused entity data, and based on the technical specification conversion benchmark library, the consistency of attribute values ​​in terms of numerical precision, character encoding, time format and classification code is identified, the relationship rules between entities are verified, and a list of quality issues is output;

[0039] Using the large model to generate and execute repair rules including problem location conditions and processing operations based on the field semantic labels and the inter-domain field mapping relationship;

[0040] The quality issues and their treatment methods will be updated to the quality management standard library to guide subsequent quality management work.

[0041] The present invention also provides a terminal device for implementing the above method. Specifically, the terminal device includes: a memory for storing a computer program code containing program instructions, wherein the program instructions are used to implement the cross-domain data integration and fusion governance method based on a large model; a processor, wherein the processor is connected to the memory via a system bus and is used to call and execute the program instructions; and a computer-readable storage medium, wherein the computer-readable storage medium is connected to the processor and is used to store the execution results of the method and the intermediate data required during its execution. When the program instructions are executed by the processor, the processor executes each step of the cross-domain data integration and fusion governance method based on a large model.

[0042] The beneficial effects of the present invention are as follows:

[0043] (1) The present invention is based on the semantic understanding integration mechanism of the big model. By intelligently parsing field names and combining them with the business terminology knowledge base to generate field semantic labels, it realizes the intelligent mapping of cross-domain data fields, solves the problem of field mapping relying on manual experience and low efficiency in traditional methods, and significantly improves the accuracy and efficiency of data integration.

[0044] (2) The technical standard integration solution designed by the present invention realizes the standardized processing of cross-domain heterogeneous data through a unified data access adaptation layer and technical specification conversion benchmark library, solves the problem of inconsistent standards between different data domains, and effectively ensures the consistency and standardization of the data integration process.

[0045] (3) This invention innovatively combines the quality governance standard library with the big model to build an adaptive quality governance mechanism that can automatically identify and repair data quality problems and continuously update governance rules. It solves the problems of lack of intelligence and poor scalability of traditional quality governance methods and achieves continuous optimization of data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The flowchart is an implementation example of a cross-domain data integration and fusion governance method based on a large model.

[0047] Figure 2 Integrating flow charts for semantic understanding.

[0048] Figure 3 Integrate flow charts for technical standards.

[0049] Figure 4 This is the data fusion processing flow chart.

[0050] Figure 5 Schematic diagram of the closed-loop process of quality management. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] As a possible embodiment of the present invention, before executing the core steps of the cross-domain data integration and fusion governance method, the following pre-configuration needs to be completed:

[0053] SP: Pre-configuration steps, used to initialize the basic environment required for system operation.

[0054] In the present invention, further, the SP step specifically includes:

[0055] SP01: Build a business terminology knowledge base. Specifically, this base should include: standard business terminology definitions, including standardized terms and definitions for each business domain; field naming rules, which specify the format of field names, including naming methods (such as camel case or underscore), naming elements, and their order; and business mapping relationships, which define association rules and data flow constraints between different business domains.

[0056] SP02: Create a verified field mapping collection. Specifically, this collection is used to store historically verified field mapping relationships, including the correspondence between source and target fields. This serves as a reference sample for field mapping verification and is used to evaluate the accuracy of newly created mappings.

[0057] SP03: Configure a technical specification conversion benchmark library. Specifically, this library includes: numerical specifications, which specify the precision requirements, units of measurement, and valid range of numerical types; text specifications, which define rules such as character set encoding and text length limits; time specifications, which unify date and time formats and time zone processing; and classification specifications, which define standards for categorized data such as status codes and enumeration values.

[0058] SP04: Set data domain priority. Specifically, this configuration is used to clarify the data management responsibilities and permissions of each data domain, define the priority order for data conflicts, and determine the direction and rules for cross-domain data synchronization.

[0059] SP05: Initialize the quality governance standard library. Specifically, a complete quality governance standard system needs to be established before the system is put into operation. This standard library serves as the basic support for identifying and handling quality issues. First, a problem feature pattern library is constructed. The library covers data consistency problem features (used to identify standardization issues of attribute values ​​in terms of format, precision, encoding, etc.), data integrity problem features (used to identify problems such as missing required fields and empty values ​​of key attributes), data accuracy problem features (used to identify data deviation problems such as outliers and erroneous values), and data relevance problem features (used to identify entity relationship conflicts, referential integrity, etc.).

[0060] Secondly, a standard repair rule library is constructed, which contains processing rules corresponding to the problem characteristics, covering data conversion rules (used to deal with problems such as non-standard format and inconsistent coding), data completion rules (used to deal with integrity issues such as missing data and null values), data correction rules (used to deal with accuracy issues such as outliers and erroneous values), and relationship repair rules (used to deal with correlation issues such as entity relationship conflicts).

[0061] Finally, a problem-rule mapping table was established, linking problem characteristics with corresponding remediation rules. This allowed for rapid matching of quality issues with treatment methods, providing rule support for subsequent quality governance work. By establishing a comprehensive quality governance standards library, the standardization and controllability of the quality governance process were ensured.

[0062] In this embodiment, comprehensive pre-configured configurations provide fundamental support for subsequent data integration and fusion governance. These configurations not only standardize data processing standards but also provide a foundation for ensuring data quality. Furthermore, each configuration can be dynamically adjusted based on business needs, ensuring the flexibility and adaptability of system operations.

[0063] As a possible embodiment of the present invention, Figure 1 As shown in the figure, a cross-domain data integration and fusion governance method based on a large model is provided, which includes the following steps:

[0064] S100: Access integration step, used to receive cross-domain raw data, perform format conversion and cleaning through a unified data access adaptation layer, and output a standardized data set.

[0065] In the present invention, further, step S100 specifically includes:

[0066] S101: Receive raw data from different data domains. Specifically, the raw data may come from multiple different business systems or data sources, and their data formats, naming conventions, and encoding standards may all differ. For example, different business systems may use different database types, such as Oracle and MySQL, or different file formats, such as CSV and XML.

[0067] S102: Data format conversion is performed through the unified data access adaptation layer. Specifically, based on the format characteristics of the original data, the corresponding adapter component is called to perform format conversion. For example, data in a relational database is read and converted using a database connection adapter; file-based data is parsed and converted using a file format adapter.

[0068] S103: Clean the converted data. Specifically, the cleaning process includes the following operations: unified naming format, converting field names to a unified format specified by the system, such as camel case; data type mapping, mapping the original data type to a standard type supported by the system; data integrity check, checking and processing duplicate values, and uniformly marking missing values.

[0069] S104: Output a standardized dataset. Specifically, the converted and cleaned data is stored according to a predefined standard format to form a standardized dataset. This dataset has a unified naming format, standard data types, complete and non-duplicate data content, and unified missing value marking.

[0070] In this embodiment, by setting up a unified data access adaptation layer, standardized processing of raw data from different sources and formats is achieved, laying the foundation for subsequent semantic understanding and data fusion. At the same time, through a standardized data cleaning process, data quality and consistency are guaranteed, and data availability is improved.

[0071] As a possible embodiment of the present invention, refer to Figure 2 After receiving the standardized dataset, the semantic understanding integration steps are performed as follows:

[0072] S200: Semantic understanding integration step, used to analyze the characteristics of standardized data sets in each data domain based on the large model, identify the business meaning of fields in the domain and generate mapping relationships between fields.

[0073] In the present invention, further, the large model processing in step S200 specifically includes:

[0074] S201: Extract data features from each field within the domain. Specifically, the data types of each field within the domain are categorized into numeric, character, date and time, and Boolean. Statistical analysis is performed on the field data to determine its validity range: for numeric fields, upper and lower thresholds are set based on data distribution; for character fields, the length range is determined; and for date and time fields, a valid time interval is set. This feature extraction method based on statistical distribution can effectively identify outliers and ensure data validity.

[0075] S202: Perform field name parsing and semantic recognition. Specifically, the field name is first parsed according to the pre-configured naming rules. The parsed results are then matched with the standard terms in the business terminology knowledge base, and field semantic labels are generated through semantic recognition. The label contains three key dimensions: data domain (identifies the business field to which the field belongs), business type (indicates the business purpose of the field), and field attributes (describes the characteristic attributes of the field). This multi-dimensional semantic annotation method provides a reliable semantic foundation for subsequent field mapping.

[0076] S203: Generate inter-domain field mapping relationships. Specifically, this step includes the following processes:

[0077] The edit distance algorithm is used to calculate the similarity of field names between different data domains;

[0078] Determine the business relationship between fields based on the field semantic labels generated above;

[0079] The field name similarity and business relevance are converted into feature points in the vector space to construct a feature vector. This feature vector contains a field name similarity component and a business semantic relevance component, and each component is normalized to a value in the range [0, 1].

[0080] S204: Verify the field mapping relationship. Specifically, based on the standard mapping sample of the verified field mapping set in SP02, calculate the Euclidean distance between the feature vector corresponding to the field mapping to be determined and the feature vector of the standard mapping sample. When the calculated Euclidean distance is less than a preset threshold, it is confirmed that the inter-domain field mapping relationship exists.

[0081] For example, in a data integration scenario, the preset field mapping determination threshold is 0.1. If the calculated Euclidean distance between the feature vector of the field to be determined (<0.85, 0.75>) and the feature vector of the verified mapping sample (<0.82, 0.78>) is 0.058, since 0.058 is less than the preset threshold of 0.1, it can be confirmed that a valid mapping relationship exists between the two fields. Conversely, if the calculated Euclidean distance is 0.15, it exceeds the preset threshold, indicating that there is no reliable mapping relationship between the two fields.

[0082] In this example, by using a large model for intelligent analysis, combined with feature extraction, semantic recognition, and mapping verification, we achieve an accurate understanding of the business meaning of fields and the reliable establishment of mapping relationships. This approach not only improves the accuracy of data integration but also lays a solid foundation for subsequent data fusion processing.

[0083] As a possible embodiment of the present invention, refer to Figure 3,After completing the semantic understanding integration, perform the technical standard integration steps, as follows:

[0084] S300: Technical standard integration step, used to perform unified technical standard conversion on standardized data sets based on field semantic labels to ensure consistency of data format specifications and encoding standards.

[0085] In the present invention, further, step S300 specifically includes:

[0086] S301: Initialize the conversion environment. Specifically, the specification definitions from the technical specification conversion benchmark library in SP03 are loaded through a configuration file. Numerical specifications, text specifications, time specifications, and classification specifications are loaded into memory. A specification index table is established, and conversion parameters are initialized. This step improves the efficiency of subsequent specification retrieval through a caching mechanism and ensures the stability of the conversion process.

[0087] S302: Determine the target conversion specification. Specifically, based on the data domain, business type, and field attribute information in the field semantic tag, determine the target conversion specification in the technical specification conversion benchmark library.

[0088] First, we construct search criteria based on field semantic tags, sequentially matching data domains, business types, and field attributes to accurately locate relevant specification definitions within the specification conversion benchmark library. Data domain matching ensures that the search is within the correct business scope, locating the appropriate specification set based on the business type, and then using field attribute information to narrow the specification search scope.

[0089] Secondly, the applicability of the retrieved candidate specifications is determined. This involves checking data type compatibility to ensure that the source data type can be converted to the target type without loss. The adaptability of the data value range is verified to ensure that the converted data meets the value requirements of the target specification. Furthermore, the compatibility of the data format is verified to confirm that the source data format can be converted according to the specification requirements.

[0090] Finally, the target specification is determined. If a single specification meets the requirements, it is directly selected as the target conversion specification. If multiple candidate specifications are available, the optimal specification is selected based on the data domain priority. The determined field specification mapping is recorded in the conversion configuration table, serving as the basis for subsequent conversion operations. This structured specification determination process ensures that the most appropriate conversion specification is found for each field.

[0091] S303: Perform data format conversion. Specifically, according to the determined target specification, the following conversion processing method is adopted:

[0092] For numeric data, call the numeric processing function to perform precision adjustment, unit conversion and interval normalization; for text data, use the string processing function to perform character set conversion, encoding standardization and format normalization; for time data, use the date and time processing function to unify the format, handle time zone conversion and standardize the representation; for categorical data, perform code conversion and enumeration value standardization based on the mapping table.

[0093] In this example, standardized technical standards integration steps achieve unified data standards at the technical level, providing a reliable foundation for subsequent data integration and quality management. Furthermore, meticulous specification matching and conversion processing ensure the accuracy and consistency of data conversion.

[0094] As a possible embodiment of the present invention, Figure 4 As shown in the figure, after completing the technical standard integration, the data fusion processing steps are performed as follows:

[0095] S400: Data fusion processing step, used to identify and match data records of the same entity across domains based on standardized data and inter-domain field mapping relationships of unified technical specifications, and perform attribute merging processing.

[0096] In the present invention, further, step S400 specifically includes:

[0097] S401: Identify unique identification fields. Specifically, analyze the business attributes of each data domain based on field semantic tags and identify fields with unique identification characteristics. A unique identification characteristic is a constraint that ensures that field values ​​within the data domain are unique. By analyzing the field value distribution characteristics and business rule constraints, the set of unique identification fields in each data domain is determined.

[0098] S402: Generate entity identification field mapping set. Specifically, this step includes the following processing:

[0099] First, based on the inter-domain field mapping relationships generated in step S200, mapping relationships involving unique identification fields are screened. Second, the screened mapping relationships are verified to ensure the consistency of the mapping fields in cross-domain scenarios. The verification process includes: checking the format consistency of field values, verifying the correspondence between field values, and confirming the timeliness of field values. Finally, the verified mapping relationships are integrated into an entity identification field mapping set for subsequent data record matching. This mapping set contains information such as the source domain identification field, the target domain identification field, mapping rules, and validity period.

[0100] S403: Perform cross-domain data record matching. Specifically, based on the entity identifier field mapping set, the following matching strategy is adopted:

[0101] For identification fields with direct mapping relationships, an exact matching method is used, that is, records with exactly the same field values ​​are determined to be the same entity; for identification fields with format differences, the format is first standardized according to the mapping rules, and then matching is performed.

[0102] S404: Execute attribute value merging. Specifically, for the same entity data record that has been matched and confirmed, attribute value merging is performed based on the field mapping relationship between domains and the data domain rights and responsibilities relationship:

[0103] First, according to the data field priority order set in step SP04, the responsible data field of each attribute is determined.

[0104] Secondly, attribute values ​​are processed according to the principle of "taking the primary domain as the priority." If the attribute value in the primary domain exists and is valid, the primary domain value is used. If the attribute value in the primary domain is missing, valid values ​​in other domains are used according to the data domain priority order set in step SP04.

[0105] Finally, the merged attribute values ​​are integrated to form fused entity data, which contains information such as a unified entity identifier, valid attribute values ​​from each domain, and the domain identifier of the attribute value.

[0106] In this example, precise cross-domain data fusion is achieved through rigorous unique identifier field identification, entity record matching, and attribute value merging. Furthermore, by introducing data domain rights and responsibilities and a priority mechanism, the accuracy and authority of the data fusion results are ensured. This method effectively addresses the entity identification and attribute merging issues in cross-domain data integration, providing a reliable data foundation for subsequent quality management.

[0107] As a possible embodiment of the present invention, refer to Figure 5 ,After completing the data fusion process, perform the quality management steps, as follows:

[0108] S500: Quality management step, used to perform intelligent quality inspection and repair on the fused entity data based on a combination of a quality management standard library and a large model.

[0109] In the present invention, further, step S500 specifically includes:

[0110] S501: Perform problem feature matching. Specifically, based on the problem feature patterns in the quality management standard library initialized in step SP05, feature detection is performed on the fused entity data. By matching data features with the problem feature patterns in the standard library, quality issues such as consistency, completeness, accuracy, and relevance in the data are identified. For quality issues that match the feature patterns in the standard library, the system automatically calls the standard remediation rules pre-set in step SP05 to resolve them.

[0111] S502: Big model analysis of unmatched issues. Specifically, for quality issues that are not covered in the quality governance standard library initialized in step SP05, the system uses a big model for intelligent analysis. First, based on the technical specification conversion benchmark library configured in step SP03, the system identifies consistency issues in entity attribute values ​​in terms of numerical precision, character encoding, time format, and classification code. Secondly, verify the relationship rules between entities and check whether business constraints such as subordination and mutual exclusion between entities are met. Through the analysis of the big model, the system will organize all identified quality issues into a problem list, which contains detailed information such as problem type, involved fields, and degree of violation.

[0112] S503: Generate remediation rules. Specifically, the system uses a large model based on field semantic labels and inter-domain field mappings to generate remediation rules for each issue in the quality issue list. During rule generation, the system first analyzes the data characteristics and business scenarios of the issues to establish precise problem location criteria. Next, it designs corresponding problem handling actions based on technical specifications and business rules. Finally, the problem location criteria and handling actions are combined to form a complete remediation rule.

[0113] S504: Execute repair processing. Specifically, the system addresses the quality issue according to the generated repair rules. During the repair process, the system first selects the data records to be processed based on the problem location criteria, and then performs repairs on the problematic data according to the processing operations. The system also records key information during the repair process, including the data status before and after the repair and the rules applied, to ensure traceability of the repair process.

[0114] S505: Update the Quality Governance Standard Library. Specifically, the system updates the problem characteristics and treatment methods discovered during this quality governance process to the quality governance standard library initialized in step SP05. During this update process, the newly discovered quality problem characteristics and their identification methods are first added to the problem characteristic pattern library. Next, the newly generated remediation rules are added to the standard remediation rule library. Finally, the rules are scored and optimized based on the remediation results, achieving continuous improvement in quality governance capabilities.

[0115] In this example, the quality governance standard library initialized in step SP05 is combined with a large model to achieve intelligent quality governance of fused data. This approach not only addresses known quality issues but also discovers and resolves emerging quality issues through large-model analysis. Furthermore, by continuously updating the quality governance standard library, it dynamically optimizes governance capabilities, effectively ensuring the continuous improvement of data quality.

[0116] As a possible embodiment of the present invention, a specific device for implementing the above-mentioned cross-domain data integration and fusion management method is provided, and its technical implementation is as follows:

[0117] The device includes a memory, a processor, and a computer-readable storage medium, wherein the memory and the processor are connected via a system bus to form a data exchange path.

[0118] In the present invention, the memory may include but is not limited to high-speed RAM memory, non-volatile memory (such as solid-state hard disk, mechanical hard disk, etc.). The memory is used to store the computer program of the present invention, which contains program code for implementing all the steps of the above-mentioned cross-domain data integration and fusion management. Specifically, the memory is provided with the following key program modules:

[0119] (1) Data access adapter module: Contains access adapter programs for various data sources, used to implement the data format conversion and cleaning functions in step S100;

[0120] (2) Semantic understanding integration module: contains the program code that calls the large model interface to implement field feature extraction, semantic recognition, and mapping relationship generation in step S200;

[0121] (3) Technical standard conversion module: contains program codes for data normalization processing, and realizes the unified conversion of technical standards in step S300;

[0122] (4) Data fusion processing module: contains program codes for entity recognition matching and attribute merging, and implements the data fusion function of step S400;

[0123] (5) Quality management module: contains the program code for quality inspection and repair processing, and implements the data quality management in step S500.

[0124] The processor can be a general-purpose processor (such as Intel or AMD CPU) or a dedicated data processing chip. The processor implements the specific functions of each of the above steps by executing the program code in the memory. During program execution, the processor can:

[0125] Call the data access adaptation module to complete the format conversion and standardization of cross-domain raw data;

[0126] Load the semantic understanding integration module to achieve field semantic understanding and mapping through large model analysis;

[0127] Run the technical standard conversion module to perform unified conversion of data specifications;

[0128] Start the data fusion processing module to complete entity matching and attribute merging;

[0129] Trigger the quality governance module to implement intelligent inspection and repair of data quality.

[0130] The computer-readable storage medium can be in various forms, such as a magnetic disk, an optical disk, or a solid-state memory, and is used to store the aforementioned computer program. When loaded and executed by a processor, this program implements all steps of the aforementioned cross-domain data integration and fusion governance method. During execution, the program can access various configuration information in memory, including a business terminology knowledge base, a technical specification conversion benchmark library, and a quality governance standard library, to ensure the proper operation of the method.

[0131] Through the organic combination of the above-mentioned hardware environment and software components, the device provided by the present invention can efficiently realize the intelligent integration and fusion governance of cross-domain data, and provide reliable technical support for the unified management of enterprise data assets.

Claims

1. A cross-domain data integration and fusion management method based on a large model, characterized by: The method comprises: Access integration: receiving cross-domain raw data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting standardized data sets. The standardized data sets use a unified naming format, data types are mapped to types supported by the system, data content is complete and has no duplicate values, and missing values ​​are uniformly marked. Semantic understanding integration: Based on a large model, the characteristics of standardized data sets within each data domain are analyzed, and the field data characteristics within each domain are extracted. By parsing the field names and combining them with the business terminology knowledge base, the business meaning of the fields within each domain is identified and field semantic labels are generated. The business relevance is determined by calculating the field name similarity and combining the field semantic labels to generate field mapping relationships between domains. Technical standard integration, based on the field semantic labels, performs unified technical standard conversion on the standardized data set, including unified data format specifications and encoding standards, and outputs standardized data that conforms to the unified technical specifications; Data fusion processing, based on the standardized data of the unified technical specifications and the inter-domain field mapping relationship, identifies and matches data records of the same entity across domains, merges the attributes of the matching entities according to the data domain rights and responsibilities, and generates fused entity data; Quality governance: Matching and repairing problem features of the fused entity data based on the quality governance standard library. Standard repair rules are applied to successfully matched quality issues. Unmatched issues are analyzed using the large model to identify quality issues related to the consistency of entity attribute values ​​and the correctness of entity relationships. Repair rules are generated and executed. Quality issues and treatment methods are updated to the quality governance standard library, and high-quality data that has been governed is output. The large model is a language model that has been pre-trained with massive data and has the ability to understand cross-domain knowledge, and can perform intelligent analysis of data types, format specifications and business rules; The large model processing in the semantic understanding integration step specifically includes: Extract the data characteristics of each field in each domain, classify it into numeric type, character type, date and time type, and Boolean type through analysis of the field data, and determine the valid range of each type of field based on statistical distribution; Parsing and semantically identifying the field names in each domain, and generating field semantic labels including data domain, business type, and field attributes in combination with the business terminology knowledge base; Generate inter-domain field mapping relationships, calculate the similarity of field names between different data domains based on the edit distance algorithm, determine the business association relationship between fields between different data domains based on the field semantic labels, convert the inter-domain field name similarity and business association degree into feature vectors, and determine the inter-domain field mapping relationship by calculating the Euclidean distance between the field mapping feature vector to be determined and the standard mapping sample feature vector in the verified field mapping set.

2. A cross-domain data integration and fusion management method based on a large model according to claim 1, characterized in that: The following basic configurations must be completed before executing this method: Build a business terminology knowledge base, including standard term definitions, field naming rules, and business mapping relationships; Establish a collection of validated field mappings to guide new field mappings; Configure the technical specification conversion benchmark library, including numerical specifications, text specifications, time specifications, and classification specifications; Set the priority order between data domains for conflict resolution when merging data attributes; Initialize the quality governance standard library, which includes a problem feature pattern library, a standard repair rule library and a problem-rule mapping relationship table. The problem feature pattern library is used to identify data consistency, completeness, accuracy and correlation issues. The standard repair rule library contains processing rules corresponding to problem features. The problem-rule mapping relationship table is used to achieve rapid matching of quality problems and processing methods.

3. The cross-domain data integration and fusion management method based on a large model according to claim 1 is characterized in that: The technical standard integration steps specifically include: Initializing the conversion environment, and reading the numerical specification, text specification, time specification, and classification specification in the technical specification conversion benchmark library; Based on the field semantic tags, the target conversion specification of each field is retrieved and determined from the technical specification conversion benchmark library; Perform data format conversion operations according to the specified target specifications.

4. The cross-domain data integration and fusion management method based on a large model according to claim 1 is characterized in that: The data fusion processing step specifically includes: Analyzing the business attributes in each data domain based on the field semantic tags to identify fields with unique identification features, wherein the unique identification feature refers to a constraint feature that field values ​​are not repeated; Associating the unique identification field through the inter-domain field mapping relationship to generate an entity identification field mapping set, and performing cross-domain data record matching based on the mapping set; Based on the inter-domain field mapping relationship and the data domain authority and responsibility relationship, attribute values ​​of different data records of the same entity are merged. When attribute values ​​in the authority and responsibility data domain are missing, valid values ​​in other domains are adopted according to the data domain priority. The merged attribute values ​​are integrated to form fused entity data.

5. The cross-domain data integration and fusion management method based on a large model according to claim 1 is characterized in that: The quality management steps specifically include: Perform problem feature matching on the fused entity data to determine whether there are quality issues that match the problem features recorded in the quality management standard library. For quality issues with matching features, execute the repair rules in the standard library. For quality issues not covered in the standard library, the large model is used to analyze the fused entity data, and based on the technical specification conversion benchmark library, the consistency of attribute values ​​in terms of numerical precision, character encoding, time format and classification code is identified, the relationship rules between entities are verified, and a list of quality issues is output; Using the large model to generate and execute repair rules including problem location conditions and processing operations based on the field semantic labels and the inter-domain field mapping relationship; The quality issues and their treatment methods will be updated to the quality management standard library to guide subsequent quality management work.

6. A cross-domain data integration and fusion management device based on a large model, characterized by: include: A memory, wherein the memory is used to store program instructions containing computer program codes, wherein the program instructions are used to implement the cross-domain data integration and fusion governance method based on a large model according to any one of claims 1 to 5; a processor, the processor being connected to the memory via a system bus and configured to call and execute the program instructions; Computer-readable storage medium, which is connected to the processor and is used to store the execution results of the cross-domain data integration and fusion governance method and the data information required during the execution of the method.

Images