Cross-domain data integration and fusion method based on large model, terminal and storage medium

Abstract

The invention provides a cross-domain data integration and fusion method based on a large model, a terminal and a storage medium, and the method comprises the steps: carrying out the format conversion and cleaning of cross-domain original data through a unified data access adaptation layer, and outputting a standardized data set; carrying out semantic understanding integration by utilizing a large model, analyzing field names, generating field semantic tags in combination with a business term knowledge base, judging business relevance by calculating similarity, and generating an inter-domain field mapping relationship; and performing technical standard conversion and data fusion processing based on the mapping relationship, and performing intelligent quality treatment by adopting a mode of combining a quality treatment standard library and a large model. According to the method, a large model is introduced to assist data understanding and treatment, so that the technical problems of insufficient semantic understanding, non-uniform standards and the like in existing cross-domain data integration are effectively solved, and the accuracy and automation level of data integration are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-domain data integration and fusion method, terminal and storage medium based on a large model. Background Art

[0002] As digital transformation deepens, enterprises are facing an increasing demand for data integration and fusion across departments, systems, and platforms. Cross-domain data integration refers to the technology of unified management and fusion of heterogeneous data distributed in different data domains (such as different business systems, departments, and organizations). In recent years, the rapid development of big model technology has provided new technical means to solve complex problems in cross-domain data integration, which is expected to break through the limitations of traditional methods.

[0003] However, existing cross-domain data integration solutions have the following key technical difficulties:

[0004] First, the ability to understand data semantics is insufficient. Existing technologies mainly rely on predefined mapping rules and manual experience for field matching. This method is difficult to accurately understand and process the semantic differences between different data domains. In practical applications, due to the diversity and complexity of business terms, rule-based mapping methods often require a lot of manual intervention, which not only reduces efficiency but also makes it difficult to ensure the accuracy of mapping.

[0005] Second, the problem of inconsistent technical standards is prominent. Different data domains often use different data formats, coding standards, and technical specifications. This heterogeneity makes it difficult to ensure consistency in the data integration process. In the process of data conversion and fusion, due to the lack of unified standards and specifications, problems such as data distortion and format errors are prone to occur, affecting the quality of data integration.

[0006] Third, data quality governance lacks intelligence. Traditional data governance methods mainly rely on static rule bases for quality inspection and repair, which cannot effectively deal with emerging data quality issues. As data scale and complexity increase, the types of quality issues are also constantly evolving, and fixed governance rules are difficult to meet dynamically changing governance needs.

[0007] Based on the above analysis, there is an urgent need for a cross-domain data integration solution that can provide intelligent semantic understanding, unified technical standard conversion, and adaptive quality management. This invention proposes an innovative technical solution based on a large model to address these technical difficulties. Summary of the invention

[0008] The purpose of the present invention is to provide a cross-domain data integration and fusion method based on a large model to solve the technical problems existing in the prior art, such as insufficient semantic understanding, inconsistent technical standards, and lack of intelligent quality management, so as to improve the accuracy and automation level of cross-domain data integration.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0010] A cross-domain data integration and fusion method based on a large model includes the following steps:

[0011] Access integration, receiving cross-domain raw data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting a standardized data set, wherein the standardized data set adopts a unified naming format, the data type is mapped to a type supported by the system, the data content is complete and has no duplicate values, and the missing value marking is unified;

[0012] Semantic understanding integration: Based on the big model, the features of the standardized data sets in each data domain are analyzed, the field data features in each domain are extracted, the business meaning of the fields in each domain is identified by parsing the field names and combining the business terminology knowledge base, and field semantic labels are generated. The business relevance is determined by calculating the field name similarity and combining the field semantic labels, and the field mapping relationship between domains is generated;

[0013] Technical standard integration, based on the field semantic labels, performs unified technical standard conversion on the standardized data set, including unified data format specifications and encoding standards, and outputs standardized data that conforms to unified technical specifications;

[0014] Data fusion processing, based on the standardized data of the unified technical specification and the field mapping relationship between the domains, identifying and matching data records of the same entity across domains, merging the attributes of the matching entities according to the rights and responsibilities of the data domains, and generating fused entity data;

[0015] Quality governance, based on the quality governance standard library, matches and repairs the problem features of the fused entity data, executes standard repair rules for successfully matched quality problems, uses the large model to analyze unmatched problems, identifies quality problems with the consistency of entity attribute values ​​and the correctness of entity relationships, generates and executes repair rules, updates quality problems and processing methods to the quality governance standard library, and outputs governed high-quality data.

[0016] The large model is a language model that has been pre-trained with massive data and has the ability to understand cross-domain knowledge, and can perform intelligent analysis on data types, format specifications and business rules;

[0017] A further technical solution is that the large model processing in the semantic understanding integration step specifically includes:

[0018] Extract the data features of the fields in each domain, classify them into numeric type, character type, date and time type, and Boolean type by analyzing the field data, and determine the valid range of each type of field based on statistical distribution;

[0019] Parsing and semantically identifying the field names in each domain, and generating field semantic labels including data domain, business type, and field attributes in combination with the business terminology knowledge base;

[0020] Generate inter-domain field mapping relationships, calculate the similarity of field names between different data domains based on the edit distance algorithm, determine the business association relationship between fields between different data domains based on the field semantic labels, convert the inter-domain field name similarity and business association degree into feature vectors, and determine the inter-domain field mapping relationship by calculating the Euclidean distance between the field mapping feature vector to be determined and the standard mapping sample feature vector in the verified field mapping set.

[0021] A further technical solution is that the following basic configurations need to be completed before the method is executed:

[0022] Build a business terminology knowledge base, including standard term definitions, field naming rules, and business mapping relationships;

[0023] Establish a collection of validated field mappings to guide new field mappings;

[0024] Configure the technical specification conversion benchmark library, including numerical specifications, text specifications, time specifications and classification specifications;

[0025] Set the priority order between data domains to handle conflicts when merging data attributes;

[0026] Initialize the quality governance standard library, including the problem feature pattern library, the standard repair rule library and the problem-rule mapping relationship table, wherein the problem feature pattern library is used to identify data consistency, completeness, accuracy and correlation problems, the standard repair rule library contains processing rules corresponding to the problem features, and the problem-rule mapping relationship table is used to achieve rapid matching of quality problems and processing methods.

[0027] A further technical solution is that the technical standard integration steps specifically include:

[0028] Initialize the conversion environment, and read the numerical specification, text specification, time specification and classification specification in the technical specification conversion reference library;

[0029] Based on the field semantic tags, the target conversion specifications of each field are retrieved and determined from the technical specification conversion benchmark library;

[0030] Perform data format conversion operations according to the specified target specification.

[0031] A further technical solution is that the data fusion processing step specifically includes:

[0032] Analyzing the business attributes in each data domain based on the field semantic label, identifying the field with a unique identification feature, wherein the unique identification feature refers to a constraint feature that the field value is not repeated;

[0033] Associating the unique identification field through the inter-domain field mapping relationship, generating an entity identification field mapping set, and performing matching of cross-domain data records based on the mapping set;

[0034] Based on the field mapping relationship between domains and the rights and responsibilities relationship of data domains, attribute values ​​of different data records of the same entity are merged. When attribute values ​​in the rights and responsibilities data domain are missing, valid values ​​of other domains are adopted according to the data domain priority.

[0035] The merged attribute values ​​are integrated to form fused entity data.

[0036] A further technical solution is that the quality management steps specifically include:

[0037] Matching problem features on the fused entity data to determine whether there are quality problems that match the problem features recorded in the quality management standard library, and executing repair rules in the standard library for quality problems with matching features;

[0038] For quality issues not covered in the standard library, the large model is used to analyze the fused entity data, and based on the technical specification conversion benchmark library, the consistency of attribute values ​​in terms of numerical precision, character encoding, time format and classification code is identified, the relationship rules between entities are verified, and a list of quality issues is output;

[0039] The large model is used to generate and execute repair rules including problem location conditions and processing operations based on the field semantic labels and the inter-domain field mapping relationship;

[0040] The quality issues and their treatment methods are updated to the quality management standard library to guide subsequent quality management work.

[0041] The present invention also provides a terminal device for implementing the above method. Specifically, the terminal device includes: a memory for storing a computer program code containing program instructions, wherein the program instructions are used to implement the cross-domain data integration and fusion governance method based on a large model; a processor, wherein the processor is connected to the memory through a system bus and is used to call and execute the program instructions; and a computer-readable storage medium, wherein the computer-readable storage medium is connected to the processor and is used to store the execution results of the method and the intermediate data required during its execution. When the program instructions are executed by the processor, the processor executes each step of the cross-domain data integration and fusion governance method based on a large model.

[0042] The beneficial effects of the present invention are as follows:

[0043] (1) The present invention is based on the semantic understanding integration mechanism of the big model. By intelligently parsing field names and combining them with the business terminology knowledge base to generate field semantic tags, it realizes the intelligent mapping of cross-domain data fields, solves the problem of field mapping relying on manual experience and low efficiency in traditional methods, and significantly improves the accuracy and efficiency of data integration.

[0044] (2) The technical standard integration solution designed by the present invention realizes the standardized processing of cross-domain heterogeneous data through a unified data access adaptation layer and technical specification conversion benchmark library, solves the problem of inconsistent standards between different data domains, and effectively ensures the consistency and standardization of the data integration process.

[0045] (3) The present invention innovatively combines the quality governance standard library with the big model to construct an adaptive quality governance mechanism that can automatically identify and repair data quality problems and continuously update governance rules. This solves the problems of lack of intelligence and poor scalability of traditional quality governance methods and achieves continuous optimization of data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The present invention is a flow chart of an implementation scheme of a cross-domain data integration and fusion governance method based on a big model.

[0047] Figure 2 Integrating flow charts for semantic understanding.

[0048] Figure 3 Integrate flow charts for technical standards.

[0049] Figure 4 This is the data fusion processing flow chart.

[0050] Figure 5 Schematic diagram of the closed-loop process of quality management. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0052] As a possible embodiment of the present invention, before executing the core steps of the cross-domain data integration and fusion governance method, the following preset configurations need to be completed:

[0053] SP: Preset configuration steps, used to initialize the basic environment required for system operation.

[0054] In the present invention, further, the SP step specifically includes:

[0055] SP01: Build a business terminology knowledge base. Specifically, the library needs to include: standard business terminology definitions, including standardized terms and their definitions for each business domain; field naming rules, which specify the format standards for field names, including naming methods (such as camel case or underscore), naming elements and their order; business mapping relationships, which define association rules and data flow constraints between different business domains.

[0056] SP02: Establish a verified field mapping collection. Specifically, this collection is used to store field mapping relationships that have passed historical verification, including the correspondence between source fields and target fields, as a reference sample for field mapping verification and to evaluate the accuracy of newly created mappings.

[0057] SP03: Configure the technical specification conversion benchmark library. Specifically, the library includes: numerical specifications, which specify the precision requirements, measurement units, and valid range of numerical types; text specifications, which define rules such as character set encoding and text length restrictions; time specifications, which unify date and time formats and time zone processing methods; classification specifications, which specify the standards for classification data such as status codes and enumeration values.

[0058] SP04: Set the priority order of data domains. Specifically, this configuration is used to clarify the data management responsibilities and permissions of each data domain, stipulate the priority order of values ​​when data conflicts occur, and determine the flow and rules of cross-domain data synchronization.

[0059] SP05: Initialize the quality governance standard library. Specifically, a complete quality governance standard system needs to be established before the system is put into operation. The standard library serves as the basic support for the identification and handling of quality problems. First, a problem feature pattern library is constructed, which covers data consistency problem features (used to identify normative problems in attribute values ​​in terms of format, precision, encoding, etc.), data integrity problem features (used to identify problems such as missing required fields and empty values ​​of key attributes), data accuracy problem features (used to identify data deviation problems such as outliers and erroneous values), and data relevance problem features (used to identify entity relationship conflicts, reference integrity, etc.).

[0060] Secondly, a standard repair rule library is constructed, which contains processing rules corresponding to the characteristics of the problem, covering data conversion rules (used to deal with problems such as irregular formats and inconsistent encoding), data completion rules (used to deal with integrity issues such as missing data and null values), data correction rules (used to deal with accuracy issues such as outliers and erroneous values), and relationship repair rules (used to deal with association issues such as entity relationship conflicts).

[0061] Finally, a problem-rule mapping relationship table is established, which associates problem features with corresponding repair rules, realizes the rapid matching of quality problems and treatment methods, and provides rule support for subsequent quality governance work. By establishing a complete quality governance standard library, the standardization and controllability of the quality governance process are ensured.

[0062] In this embodiment, through perfect preset configuration, basic support is provided for subsequent data integration and fusion governance. These configurations not only standardize data processing standards, but also provide basic guarantees for ensuring data quality. At the same time, various configurations can be dynamically adjusted according to business needs, ensuring the flexibility and adaptability of system operation.

[0063] As a possible embodiment of the present invention, Figure 1 As shown in the figure, a cross-domain data integration and fusion governance method based on a large model is provided, which includes the following steps:

[0064] S100: Access integration step, used to receive cross-domain raw data, perform format conversion and cleaning through a unified data access adaptation layer, and output a standardized data set.

[0065] In the present invention, further, step S100 specifically includes:

[0066] S101: Receive raw data from different data domains. Specifically, the raw data may come from multiple different business systems or data sources, and their data formats, naming rules and encoding standards may be different. For example, different business systems may use different database types, such as Oracle, MySQL, etc., or use different file formats, such as CSV, XML, etc.

[0067] S102: Perform data format conversion through the unified data access adapter layer. Specifically, according to the format characteristics of the original data, call the corresponding adapter component to perform format conversion. For example, for data in a relational database, the database connection adapter is used to read and convert; for file-type data, the file format adapter is used to parse and convert.

[0068] S103: Clean the converted data. Specifically, the cleaning process includes the following operations: unified naming format, converting the field name to a unified format specified by the system, such as camel case naming; data type mapping, mapping the original data type to a standard type supported by the system; data integrity check, checking and processing duplicate values, and uniformly marking missing values.

[0069] S104: Output standardized data set. Specifically, the data after format conversion and cleaning is stored in a predefined standard format to form a standardized data set. The data set has a unified naming format, standard data types, complete and non-repetitive data content, and a unified missing value mark.

[0070] In this embodiment, by setting up a unified data access adaptation layer, standardized processing of raw data from different sources and formats is achieved, laying the foundation for subsequent semantic understanding and data fusion. At the same time, through a standardized data cleaning process, the quality and consistency of the data are guaranteed, and the availability of the data is improved.

[0071] As a possible embodiment of the present invention, refer to Figure 2 ,After receiving the standardized data set, the semantic understanding integration steps are performed as follows:

[0072] S200: Semantic understanding integration step, used to analyze the characteristics of standardized data sets in each data domain based on the large model, identify the business meaning of the fields in the domain and generate mapping relationships between fields.

[0073] In the present invention, further, the large model processing in step S200 specifically includes:

[0074] S201: Extract the data features of the fields in each domain. Specifically, the data types of the fields in each domain are divided into numeric type, character type, date and time type, and Boolean type. By performing statistical analysis on the field data, its effective range is set: for numeric fields, the upper and lower limit thresholds are set according to the data distribution; for character fields, the length range is determined; for date and time fields, the effective time interval is set. This feature extraction method based on statistical distribution can effectively identify outliers and ensure the validity of the data.

[0075] S202: Perform field name resolution and semantic recognition. Specifically, first resolve the field name according to the preset naming rules. Then match the resolution result with the standard terms in the business terminology knowledge base, and generate field semantic labels through semantic recognition. The label contains three key dimensions: data domain (identifies the business field to which the field belongs), business type (indicates the business purpose of the field), and field attributes (describes the characteristic attributes of the field). This multi-dimensional semantic annotation method provides a reliable semantic basis for subsequent field mapping.

[0076] S203: Generate inter-domain field mapping relationship. Specifically, this step includes the following process:

[0077] The edit distance algorithm is used to calculate the similarity of field names between different data domains;

[0078] Determine the business association relationship between fields based on the field semantic labels generated above;

[0079] The field name similarity and business relevance are converted into feature points in the vector space to construct a feature vector. The feature vector contains the field name similarity component and the business semantic relevance component, and the value range of each component after normalization is [0,1].

[0080] S204: Verify the field mapping relationship. Specifically, based on the standard mapping sample of the verified field mapping set in SP02, calculate the Euclidean distance between the feature vector corresponding to the field mapping to be determined and the feature vector of the standard mapping sample. When the calculated Euclidean distance is less than the preset threshold, it can be confirmed that there is an inter-domain field mapping relationship.

[0081] For example, in a data integration scenario, the preset field mapping judgment threshold is 0.1. When the Euclidean distance between the feature vector of the field mapping to be judged <0.85,0.75> and the feature vector of the verified mapping sample <0.82,0.78> is calculated to be 0.058, since 0.058 is less than the preset threshold of 0.1, it can be confirmed that there is a valid mapping relationship between the two fields. On the contrary, if the calculated Euclidean distance is 0.15, it exceeds the preset threshold, indicating that there is no reliable mapping relationship between the two fields.

[0082] In this embodiment, by using a large model for intelligent analysis, combined with steps such as feature extraction, semantic recognition and mapping verification, an accurate understanding of the business meaning of the field and a reliable establishment of the mapping relationship are achieved. This method not only improves the accuracy of data integration, but also lays a solid foundation for subsequent data fusion processing.

[0083] As a possible embodiment of the present invention, refer to Figure 3,After completing the semantic understanding integration, execute the technical standard integration steps, as follows:

[0084] S300: Technical standard integration step, used to perform unified technical standard conversion on standardized data sets based on field semantic labels to ensure consistency of data format specifications and encoding standards.

[0085] In the present invention, further, step S300 specifically includes:

[0086] S301: Initialize the conversion environment. Specifically, load the specification definition of the technical specification conversion benchmark library in SP03 through the configuration file, load the numerical specification, text specification, time specification and classification specification into the memory, establish the specification index table, and initialize the conversion parameters. This step improves the efficiency of subsequent specification retrieval through the cache mechanism and ensures the stability of the conversion process.

[0087] S302: Determine the target conversion specification. Specifically, based on the data domain, business type and field attribute information in the field semantic tag, determine the target conversion specification in the technical specification conversion benchmark library.

[0088] First, we build search conditions based on field semantic tags, match data domains, business types, and field attributes in turn, and accurately locate relevant specification definitions in the specification conversion benchmark library. We ensure that we search within the correct business scope by matching data domains, locate the corresponding specification set based on business types, and then use field attribute information to narrow the scope of specification searches.

[0089] Secondly, the applicability of the retrieved candidate specifications is judged. By checking the data type matching, ensure that the source data type can be converted to the target type without loss; verify the adaptability of the data value range to ensure that the converted data meets the value requirements of the target specification; and verify the compatibility of the data format to confirm that the source data format can be converted according to the specification requirements.

[0090] Finally, the target specification is determined. When there is only one specification that meets the conditions, it is directly determined as the target conversion specification; when there are multiple candidate specifications, the optimal specification is selected according to the data domain priority order. The determined field specification mapping relationship will be recorded in the conversion configuration table as the basis for subsequent conversion operations. This structured specification determination process ensures that the most appropriate conversion specification can be found for each field.

[0091] S303: Execute data format conversion. Specifically, according to the determined target specification, the following conversion processing method is adopted:

[0092] For numeric data, call the numeric processing function to perform precision adjustment, unit conversion and interval normalization; for text data, use the string processing function to perform character set conversion, encoding standardization and format normalization; for time data, use the date and time processing function to unify the format, handle time zone conversion and standardize the representation; for categorical data, perform code conversion and enumeration value standardization based on the mapping table.

[0093] In this embodiment, through the standardized technical standard integration steps, the standard unification of data at the technical level is achieved, providing a reliable foundation for subsequent data fusion and quality management. At the same time, through meticulous specification matching and conversion processing, the accuracy and consistency of data conversion are guaranteed.

[0094] As a possible embodiment of the present invention, Figure 4 As shown in the figure, after completing the technical standard integration, the data fusion processing steps are performed as follows:

[0095] S400: Data fusion processing step, used to identify and match data records of the same entity across domains based on standardized data and inter-domain field mapping relationships of unified technical specifications, and perform attribute merging processing.

[0096] In the present invention, further, step S400 specifically includes:

[0097] S401: Identify unique identification fields. Specifically, analyze the business attributes in each data domain based on the field semantic tags and identify fields with unique identification features. The unique identification feature refers to the constraint feature that field values ​​are not repeated in the data domain. By analyzing the value distribution characteristics of the field and the business rule constraints, determine the set of unique identification fields in each data domain.

[0098] S402: Generate an entity identification field mapping set. Specifically, this step includes the following processing:

[0099] First, based on the inter-domain field mapping relationship generated in step S200, the mapping relationship involving the unique identification field is screened out; secondly, the screened mapping relationship is verified to ensure the consistency of the mapping field in the cross-domain scenario. The verification process includes: checking the format consistency of the field value, verifying the correspondence of the field value, and confirming the timeliness of the field value; finally, the mapping relationship that has passed the verification is integrated to form an entity identification field mapping set for subsequent data record matching. The mapping set contains information such as the source domain identification field, the target domain identification field, the mapping rules, and the validity period.

[0100] S403: Perform cross-domain data record matching. Specifically, based on the entity identifier field mapping set, the following matching strategy is adopted:

[0101] For identification fields with direct mapping relationships, an exact match method is used, that is, records with exactly the same field values ​​are determined to be the same entity; for identification fields with format differences, the format is first standardized according to the mapping rules, and then matching is performed.

[0102] S404: Execute attribute value merging. Specifically, for the same entity data record that has been matched and confirmed, attribute value merging is performed based on the field mapping relationship between domains and the rights and responsibilities relationship between data domains:

[0103] First, according to the data domain priority order set in step SP04, the responsible data domain of each attribute is determined.

[0104] Secondly, the attribute value is processed according to the principle of "taking the competent domain as the standard". When the attribute value in the competent data domain exists and is valid, the value of the competent domain is used; when the attribute value in the competent domain is missing, the valid value in other domains is used according to the data domain priority order set in step SP04.

[0105] Finally, the merged attribute values ​​are integrated to form fused entity data, which contains information such as unified entity identification, valid attribute values ​​from each domain, and source domain identification of attribute values.

[0106] In this embodiment, accurate fusion of cross-domain data is achieved through strict unique identification field recognition, entity record matching, and attribute value merging processing. At the same time, by introducing data domain rights and responsibilities and priority mechanisms, the accuracy and authority of data fusion results are ensured. This method can effectively handle entity identification and attribute merging problems in cross-domain data integration, and provide a reliable data foundation for subsequent quality management.

[0107] As a possible embodiment of the present invention, refer to Figure 5 ,After completing the data fusion process, perform the quality management steps as follows:

[0108] S500: Quality management step, used to perform intelligent quality inspection and repair on the fused entity data based on a combination of the quality management standard library and the big model.

[0109] In the present invention, further, step S500 specifically includes:

[0110] S501: Perform problem feature matching. Specifically, based on the problem feature pattern in the quality management standard library initialized in step SP05, feature detection is performed on the fused entity data. By matching data features with the problem feature pattern in the standard library, quality issues such as consistency, completeness, accuracy and relevance in the data are identified. For quality issues that match the feature pattern in the standard library, the system automatically calls the standard repair rules preset in step SP05 for processing.

[0111] S502: Big model analysis of unmatched issues. Specifically, for quality issues that are not covered in the quality governance standard library initialized in step SP05, the system uses a big model for intelligent analysis. First, based on the technical specification conversion benchmark library configured in step SP03, the system identifies consistency issues in entity attribute values ​​in terms of numerical precision, character encoding, time format, and classification code. Secondly, verify the relationship rules between entities and check whether business constraints such as subordination and mutual exclusion between entities are met. Through the analysis of the big model, the system organizes all identified quality issues into a problem list, which contains detailed information such as problem type, fields involved, and degree of violation.

[0112] S503: Generate repair rules. Specifically, the system uses a large model based on field semantic labels and field mapping relationships between domains to generate repair rules for various problems in the quality problem list. In the rule generation process, first analyze the data characteristics and business scenarios where the problem occurs, and build accurate problem location conditions. Secondly, design corresponding problem handling operations based on technical specifications and business rules. Finally, assemble the problem location conditions and processing operations to form a complete repair rule.

[0113] S504: Perform repair processing. Specifically, the system processes the quality problem according to the generated repair rules. During the repair process, the data records to be processed are first screened according to the problem location conditions, and then the problem data is repaired according to the processing operation. The system synchronously records the key information in the repair process, including the data status before and after the repair, the applied rules, etc., to ensure the traceability of the repair process.

[0114] S505: Update the quality management standard library. Specifically, the system updates the problem features and processing methods found in this quality management process to the quality management standard library initialized in step SP05. During the update process, first, the newly discovered quality problem features and their identification methods are added to the problem feature pattern library, then the newly generated repair rules are added to the standard repair rule library, and finally, the rules are scored and optimized according to the repair effect to achieve continuous improvement of quality management capabilities.

[0115] In this embodiment, the quality management standard library initialized in step SP05 is combined with the big model to achieve intelligent quality management of fused data. This method can not only handle known quality problems, but also discover and solve new quality problems through big model analysis. At the same time, it realizes dynamic optimization of management capabilities by continuously updating the quality management standard library, providing effective guarantee for the continuous improvement of data quality.

[0116] As a possible embodiment of the present invention, a specific device for implementing the above cross-domain data integration and fusion management method is provided, and its technical implementation is as follows:

[0117] The device includes a memory, a processor and a computer-readable storage medium, wherein the memory and the processor are connected via a system bus to form a data exchange path.

[0118] In the present invention, the memory may include but is not limited to high-speed RAM memory, non-volatile memory (such as solid-state hard disk, mechanical hard disk, etc.). The memory is used to store the computer program of the present invention, which contains program codes for implementing all the steps of the above-mentioned cross-domain data integration and fusion management. Specifically, the following key program modules are set in the memory:

[0119] (1) Data access adapter module: includes access adapter programs for various data sources, used to implement the data format conversion and cleaning functions in step S100;

[0120] (2) Semantic understanding integration module: contains program code for calling the large model interface to implement field feature extraction, semantic recognition, and mapping relationship generation in step S200;

[0121] (3) Technical standard conversion module: contains program codes for data normalization processing, and realizes the unified conversion of technical standards in step S300;

[0122] (4) Data fusion processing module: including program codes for entity recognition matching and attribute merging, realizing the data fusion function of step S400;

[0123] (5) Quality management module: contains the program code for quality inspection and repair processing, and implements the data quality management in step S500.

[0124] The processor can be a general-purpose processor (such as Intel, AMD CPU) or a dedicated data processing chip. The processor implements the specific functions of the above steps by executing the program code in the memory. During the program execution, the processor can:

[0125] Call the data access adaptation module to complete the format conversion and standardization of cross-domain raw data;

[0126] Load the semantic understanding integration module to achieve field semantic understanding and mapping through large model analysis;

[0127] Run the technical standard conversion module to perform unified conversion of data specifications;

[0128] Start the data fusion processing module to complete entity matching and attribute merging;

[0129] Trigger the quality governance module to achieve intelligent inspection and repair of data quality.

[0130] The computer-readable storage medium may be in various forms such as a disk, an optical disk, a solid-state memory, etc., for storing the above-mentioned computer program. When the program is loaded and executed by the processor, all steps of the cross-domain data integration and fusion governance method as described above will be implemented. During the execution of the program, various configuration information in the memory may be accessed, including a business terminology knowledge base, a technical specification conversion benchmark library, a quality governance standard library, etc., to ensure the normal operation of the method.

[0131] Through the organic combination of the above-mentioned hardware environment and software components, the device provided by the present invention can efficiently realize the intelligent integration and fusion management of cross-domain data, and provide reliable technical support for the unified management of enterprise data assets.

Claims

1. A cross-domain data integration and fusion management method based on a large model, characterized in that: The method comprises: Access integration, receiving cross-domain raw data, performing format conversion and cleaning through a unified data access adaptation layer, and outputting a standardized data set, wherein the standardized data set adopts a unified naming format, the data type is mapped to a type supported by the system, the data content is complete and has no duplicate values, and the missing value marking is unified; Semantic understanding integration: Based on the big model, the features of the standardized data sets in each data domain are analyzed, the field data features in each domain are extracted, the business meaning of the fields in each domain is identified by parsing the field names and combining the business terminology knowledge base, and field semantic labels are generated. The business relevance is determined by calculating the field name similarity and combining the field semantic labels, and the field mapping relationship between domains is generated; Technical standard integration, based on the field semantic labels, performs unified technical standard conversion on the standardized data set, including unified data format specifications and encoding standards, and outputs standardized data that conforms to unified technical specifications; Data fusion processing, based on the standardized data of the unified technical specification and the field mapping relationship between the domains, identifying and matching data records of the same entity across domains, merging the attributes of the matching entities according to the rights and responsibilities of the data domains, and generating fused entity data; Quality governance: matching and repairing problem features of the fused entity data based on the quality governance standard library, executing standard repair rules for successfully matched quality problems, using the large model to analyze unmatched problems, identifying quality problems of entity attribute value consistency and entity relationship correctness, generating and executing repair rules, updating quality problems and processing methods to the quality governance standard library, and outputting governed high-quality data; Among them, the large model is a language model that has been pre-trained with massive data and has the ability to understand cross-domain knowledge. It can perform intelligent analysis on data types, format specifications and business rules.

2. According to the large model-based cross-domain data integration and fusion management method of claim 1, it is characterized in that: The large model processing in the semantic understanding integration step specifically includes: Extract the data features of the fields in each domain, classify them into numeric type, character type, date and time type, and Boolean type by analyzing the field data, and determine the valid range of each type of field based on statistical distribution; Parsing and semantically identifying the field names in each domain, and generating field semantic labels including data domain, business type, and field attributes in combination with the business terminology knowledge base; Generate inter-domain field mapping relationships, calculate the similarity of field names between different data domains based on the edit distance algorithm, determine the business association relationship between fields between different data domains based on the field semantic labels, convert the inter-domain field name similarity and business association degree into feature vectors, and determine the inter-domain field mapping relationship by calculating the Euclidean distance between the field mapping feature vector to be determined and the standard mapping sample feature vector in the verified field mapping set.

3. According to a large model-based cross-domain data integration and fusion management method according to claim 1, it is characterized in that: The following basic configurations need to be completed before executing the method: Build a business terminology knowledge base, including standard term definitions, field naming rules, and business mapping relationships; Establish a collection of validated field mappings to guide new field mappings; Configure the technical specification conversion benchmark library, including numerical specifications, text specifications, time specifications, and classification specifications; set the priority order between data domains for conflict resolution when merging data attributes; Initialize the quality governance standard library, including the problem feature pattern library, the standard repair rule library and the problem-rule mapping relationship table, wherein the problem feature pattern library is used to identify data consistency, completeness, accuracy and correlation problems, the standard repair rule library contains processing rules corresponding to the problem features, and the problem-rule mapping relationship table is used to achieve rapid matching of quality problems and processing methods.

4. According to the large model-based cross-domain data integration and fusion management method of claim 1, it is characterized in that: The technical standard integration steps specifically include: Initialize the conversion environment, and read the numerical specification, text specification, time specification and classification specification in the technical specification conversion reference library; Based on the field semantic tags, the target conversion specifications of each field are retrieved and determined from the technical specification conversion benchmark library; Perform data format conversion operations according to the specified target specification.

5. According to the large model-based cross-domain data integration and fusion management method of claim 1, it is characterized in that: The data fusion processing step specifically includes: Analyzing the business attributes in each data domain based on the field semantic label, identifying the field with a unique identification feature, wherein the unique identification feature refers to a constraint feature that the field value is not repeated; Associating the unique identification field through the inter-domain field mapping relationship, generating an entity identification field mapping set, and performing matching of cross-domain data records based on the mapping set; Based on the field mapping relationship between domains and the rights and responsibilities relationship of data domains, attribute values ​​of different data records of the same entity are merged. When attribute values ​​in the rights and responsibilities data domain are missing, valid values ​​of other domains are adopted according to the data domain priority. The merged attribute values ​​are integrated to form fused entity data.

6. According to the large model-based cross-domain data integration and fusion management method of claim 1, it is characterized in that: The quality management steps specifically include: Matching problem features on the fused entity data to determine whether there are quality problems that match the problem features recorded in the quality management standard library, and executing repair rules in the standard library for quality problems with matching features; For quality issues not covered in the standard library, the large model is used to analyze the fused entity data, and based on the technical specification conversion benchmark library, the consistency of attribute values ​​in terms of numerical precision, character encoding, time format and classification code is identified, the relationship rules between entities are verified, and a list of quality issues is output; The large model is used to generate and execute repair rules including problem location conditions and processing operations based on the field semantic labels and the inter-domain field mapping relationship; The quality issues and their treatment methods are updated to the quality management standard library to guide subsequent quality management work.

7. A cross-domain data integration and fusion management device based on a large model, characterized in that: include: A memory, wherein the memory is used to store program instructions including computer program codes, wherein the program instructions are used to implement the cross-domain data integration and fusion management method based on a large model according to any one of claims 1 to 6; A processor, the processor is connected to the memory via a system bus, and is used to call and execute the program instructions; Computer-readable storage medium, the computer-readable storage medium is connected to the processor and is used to store the execution results of the cross-domain data integration and fusion governance method and the data information required during the execution of the method.

Images