Data integration and fusion method and system based on large model

Abstract

The invention discloses a data integration and fusion method and system based on a large model, and the method comprises the steps: taking the metadata information of initial multi-source data and representative sample data as the input context of a pre-trained LLM large language model, and carrying out the training through a Prompt prompt project guide model, thereby obtaining a plurality of enhanced data source mode descriptions; inputting a plurality of enhanced data source mode descriptions into an LLM large-scale language model, enabling the model to be a data integration engine through a Prompt prompt project, guiding the model to recognize semantic equivalence fields among different data source modes, and outputting a dynamic mode mapping scheme; analyzing a conflict reason and reasoning a conflict resolution strategy, and outputting a data cleaning suggestion and a data analysis result; and loading the fused data into the target data storage. The method can adapt to complex and changeable data structures, and the flexibility and efficiency of data integration are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data integration and fusion method and system based on a large model. Background Art

[0002] Traditional data integration methods face numerous challenges when dealing with these heterogeneous data sources. Unstructured or semi-structured data requires manual coding of complex parsing rules and program code to extract key information and perform structured conversions, which is not only time-consuming and labor-intensive but also error-prone. Furthermore, semantic conflicts and inconsistencies between different data sources make data schema alignment and mapping difficult. Furthermore, during data cleansing and conflict resolution, traditional methods struggle to effectively address data conflicts, missing data, and anomalies due to a lack of in-depth understanding of data semantics and business context. This results in low-quality integrated data that cannot meet enterprise requirements for data accuracy and completeness. Summary of the Invention

[0003] The purpose of the present invention is to solve the above problems and to design a data integration and fusion method and system based on a large model.

[0004] To achieve the above object, the technical solution of the present invention is that, further, in the above-mentioned data integration and fusion method based on the large model, the data integration and fusion method includes the following steps: Acquiring multi-source heterogeneous data, performing structural conversion on semi-structured data in the multi-source heterogeneous data, and reading pattern information and sample data from the structured data to obtain initial multi-source data; The metadata information and representative sample data of the initial multi-source data are used as the input context of the pre-trained LLM large language model, and the model is trained through the prompt engineering guidance to obtain multiple enhanced data source pattern descriptions; Input the multiple enhanced data source schema descriptions into the LLM large language model, use the prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; Input missing data records and multiple enhanced data source schema descriptions in data fusion into the LLM large language model, analyze conflict causes and infer conflict resolution strategies, and output data cleaning suggestions and data analysis results; The fused data is loaded into a target data storage based on the dynamic pattern mapping scheme, data cleaning suggestions and data parsing results.

[0005] Furthermore, in the above-mentioned large-model-based data integration and fusion method, the steps of acquiring multi-source heterogeneous data, performing structural conversion on semi-structured data in the multi-source heterogeneous data, and reading pattern information and sample data from the structured data to obtain initial multi-source data include: Taking entity names, attribute values, and event time information from the text data in the multi-source heterogeneous data, and converting them into a structured table format; Parsing the JSON and XML formats in the multi-source heterogeneous data and extracting node information, and converting it into a semi-structured representation of a flat table structure according to preset rules; The metadata information and constraints in the structured data in the multi-source heterogeneous data are read, sample data is extracted, and initial multi-source data is obtained.

[0006] Furthermore, in the above-mentioned large-model-based data integration and fusion method, the metadata information and representative sample data of the initial multi-source data are used as the input context of the pre-trained LLM large-scale language model, and the model is trained through the prompt engineering guidance to obtain multiple enhanced data source pattern descriptions, including: Integrate the metadata information of each data source and the extracted sample data to obtain input context data; Using a pre-trained LLM large language model to deeply process the input context data; Perform semantic analysis on each field to identify the semantic relationship between fields. Primary and foreign key relationships represent the association between entities, and parent-child relationships represent the hierarchical structure of data. Infer the business domain and context to which the data source belongs, and obtain the business scope and related background involved in the data to obtain multiple enhanced data source schema descriptions.

[0007] Furthermore, in the above-mentioned large-model-based data integration and fusion method, the multiple enhanced data source schema descriptions are input into the LLM large language model, and the model is used as a data integration engine through prompt engineering, guiding the model to identify semantically equivalent fields between different data source schemas and output a dynamic schema mapping solution, including: Use the model to compare and analyze multiple enhanced schema descriptions to identify semantically equivalent fields in different data sources; Reason about the conversion relationships between different data structures and determine how to convert nested JSON structures into table structures in relational databases; Determine how to convert relational data into a structure suitable for target storage, including hierarchical JSON and XML, and automatically generate semantic mapping rules between fields, including data type conversion, value range conversion suggestions, and structure conversion logic; LLM large language model is used to detect potential semantic conflicts and output dynamic pattern mapping solutions.

[0008] Furthermore, in the above-mentioned large-model-based data integration and fusion method, the missing data records and multiple enhanced data source schema descriptions in the data fusion are input into the LLM large language model, the conflict causes are analyzed and conflict resolution strategies are inferred, and data cleaning suggestions and data analysis results are output, including: Use a large model to analyze the causes of conflicts and identify conflicts with inconsistent values ​​as data errors or timestamp differences; Conflict resolution strategies based on credibility, timeliness, and data source priority or business logic reasoning; For clear data type conversion and rule-based value filling, generated processed data is output with data cleaning suggestions.

[0009] Furthermore, in the above-mentioned large-model-based data integration and fusion method, the missing data records and multiple enhanced data source schema descriptions in the data fusion are input into the LLM large language model, the conflict causes are analyzed and conflict resolution strategies are inferred, and data cleaning suggestions and data analysis results are output, which also includes: The LLM large language model is used to understand the semantics of each record and analyze the meaning and function of each attribute field in the record; Compare data attributes and semantics, determine whether the records belong to the same entity based on the importance, similarity and correlation factors of the attributes, and parse based on the judgment results to obtain data parsing results.

[0010] Furthermore, in the above-mentioned data integration and fusion method based on the large model, the loading of the fused data into the target data storage based on the dynamic model mapping scheme, the data cleaning suggestion and the data analysis result includes: Based on the dynamic schema mapping solution, data cleaning suggestions and data analysis results, perform data conversion, cleaning and merging operations; According to the resolution strategy, data conflicts, missing data and anomalies are handled, and the same entity data from different data sources are integrated to obtain fused data; The fused data is loaded into a target data storage, which includes at least a data warehouse, a data lake, and a database.

[0011] Furthermore, in the data integration and fusion system based on the large model, the data integration and fusion system includes the following modules: A multi-source data acquisition module is used to acquire multi-source heterogeneous data, perform structural conversion on semi-structured data in the multi-source heterogeneous data, read pattern information and sample data from the structured data, and obtain initial multi-source data; A data enhancement recognition module is used to use the metadata information and representative sample data of the initial multi-source data as the input context of the pre-trained LLM large language model, guide the model training through the prompt engineering, and obtain multiple enhanced data source pattern descriptions; A data dynamic mapping module is used to input the multiple enhanced data source schema descriptions into the LLM large language model, use prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; A data analysis and cleaning module is used to input missing data records and multiple enhanced data source schema descriptions in data fusion into the LLM large language model, analyze conflict causes and infer conflict resolution strategies, and output data cleaning suggestions and data analysis results; The data fusion storage module is used to load the fused data into the target data storage based on the dynamic pattern mapping solution, data cleaning suggestions and data analysis results.

[0012] Furthermore, in the data integration and fusion system based on the large model, the multi-source data acquisition module includes the following submodules: A conversion submodule, configured to extract entity names, attribute values, and event time information from the text data in the multi-source heterogeneous data and convert them into a structured table format; A parsing submodule is used to parse the JSON and XML formats in the multi-source heterogeneous data and extract node information, and convert it into a semi-structured representation of a flat table structure according to preset rules; The reading submodule is used to read metadata information and constraints in the structured data in the multi-source heterogeneous data, extract sample data, and obtain initial multi-source data.

[0013] Furthermore, in the data integration and fusion system based on the large model, the data fusion storage module includes the following submodules: A processing submodule, configured to perform data conversion, cleaning, and merging operations based on the dynamic schema mapping solution, data cleaning suggestions, and data parsing results; The integration submodule is used to handle data conflicts, missing data, and anomaly issues according to the resolution strategy, and integrate the same entity data from different data sources to obtain fused data; The storage submodule is used to load the fused data into a target data storage, which includes at least a data warehouse, a data lake, and a database.

[0014] Its beneficial effects include: 1. Leveraging natural language processing technology and the intelligent parsing capabilities of large models, it automatically completes information extraction and structured conversion without requiring extensive manual programming, significantly improving the efficiency of data access and initial processing and reducing labor and time costs. 2. Compared with traditional manual pattern alignment and mapping methods, it is faster and more accurate, adapting to complex and changing data structures, and improving the flexibility and efficiency of data integration. 3. It can continuously improve the effectiveness and performance of data integration and fusion methods, enabling them to better adapt to changing data environments and business needs, and ensuring the long-term effectiveness and reliability of data integration work. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0016] Figure 1 This is a schematic diagram of a first embodiment of a data integration and fusion method based on a large model in an embodiment of the present invention; Figure 2 Schematic diagram of a second embodiment of a data integration and fusion method based on a large model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the first embodiment of the data integration and fusion system based on a large model in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 As shown, the data integration and fusion method based on the large model includes the following steps: Step 101: Acquire multi-source heterogeneous data, perform structural conversion on semi-structured data in the multi-source heterogeneous data, read pattern information and sample data from the structured data, and obtain initial multi-source data; Specifically, in this embodiment, entity names, attribute values, and event time information are obtained from the text data in the multi-source heterogeneous data, and converted into a structured table format; Parsing the JSON and XML formats in the multi-source heterogeneous data and extracting node information, and converting it into a semi-structured representation of a flat table structure according to preset rules; The metadata information and constraints in the structured data in the multi-source heterogeneous data are read, sample data is extracted, and initial multi-source data is obtained.

[0020] Specifically, Accessing heterogeneous data sources: The system must be compatible with multiple data sources, including but not limited to relational databases (MySQL, Oracle, SQL Server), non-relational databases (MongoDB, Cassandra), file systems (CSV files, Excel spreadsheets), API data, log data, and more. Each data source has its own unique format and access method. We provide corresponding access adapter modules for different types of data sources to ensure stable and efficient data acquisition.

[0021] Unstructured / semi-structured data processing: For text data, we leverage natural language processing (NLP) techniques, such as named entity recognition, keyword extraction, and semantic analysis, to extract key information from the text, such as entity names, attribute values, and event times. This information is then converted into a structured table, where each row represents an entity and each column represents an attribute. For data in JSON and XML formats, we parse their structure, extract node information, and convert it into a flat table or a semi-structured representation that retains a hierarchical structure, based on pre-set rules or through pattern inference, to facilitate subsequent processing and analysis.

[0022] Structured data processing: Directly reads metadata information for structured data, including table names, field names, field types (integer, string, date and time), and constraints (primary key, foreign key, not-null constraint, unique constraint). Simultaneously, extract a certain amount of sample data. The sample data should be representative and reflect the overall characteristics and distribution of the data in the data source. For example, the first 1,000 data items can be used as a sample to enable subsequent large-scale model-based processing to better understand the data's semantics and patterns.

[0023] Step 102: Using the metadata information and representative sample data of the initial multi-source data as input context for a pre-trained LLM large language model, the model is trained through a prompt engineering guide to obtain multiple enhanced data source pattern descriptions; Specifically, in this embodiment, the metadata information of each data source and the extracted sample data are integrated to obtain input context data; Using a pre-trained LLM large language model to deeply process the input context data; Perform semantic analysis on each field to identify the semantic relationship between fields. Primary and foreign key relationships represent the association between entities, and parent-child relationships represent the hierarchical structure of data. Infer the business domain and context to which the data source belongs, and obtain the business scope and related background involved in the data to obtain multiple enhanced data source schema descriptions.

[0024] Specifically, Input context construction: The metadata of each data source and the extracted sample data are integrated to form a rich input context. The metadata details the structure and constraints of the data, while the sample data provides actual data examples. The combination of the two enables the large model to better understand the actual meaning and application scenarios of the data.

[0025] Prompt Engineering: Design a series of carefully constructed prompts to guide the large model in semantic understanding and pattern parsing. For example, for field semantic understanding, a prompt might be, "Please explain in detail the real-world concept represented by the field 'Customer Name', and explain its specific meaning and usage in this data source." For identifying semantic relationships between fields, a prompt might be, "Analyze whether there is a primary-foreign key relationship or other association between the fields 'Order Number' and 'Customer Order Number', and explain the basis for this determination." For inferring the business domain of a data source, a prompt might be, "Based on the provided metadata and sample data, infer which business domain this data source is primarily used for, such as e-commerce, finance and insurance, or healthcare, and describe the main business scenarios in that domain and the core entities and relationships involved in the data."

[0026] Large model processing: Utilizing a large language model (LLM) pre-trained or fine-tuned for data integration tasks, the model deeply processes the input context. The model first performs semantic analysis on each field to understand the real-world concepts it represents—not just the literal meaning, but also its meaning and role in specific business scenarios. It then identifies semantic relationships between fields, such as primary and foreign key relationships representing associations between entities, and parent-child relationships representing data hierarchies. Furthermore, through analysis of the overall data, the model infers the business domain and context of the data source, clarifying the business scope and relevant background of the data.

[0027] Output Enhanced Schema Description: Generates an enhanced, semantically annotated data source schema description. This description includes not only the original structural information (table name, field name, field type, and constraints), but also the deeper semantic information inferred by the large model, such as the business definition of each field ("Customer Number" represents a code that uniquely identifies a customer in the business system), detailed descriptions of the semantic relationships between fields ("The 'Customer Number' field in the Order table forms a foreign key relationship with the 'Customer Number' field in the Customer table, linking order and customer entities"), the business domain label of the data source ("E-commerce - Order Management"), and the position and role of the data source in the business process.

[0028] Step 103: Input the multiple enhanced data source schema descriptions into the LLM large language model, use the prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; Specifically, in this embodiment, a model is used to compare and analyze multiple enhanced schema descriptions to identify semantically equivalent fields in different data sources; Reason about the conversion relationships between different data structures and determine how to convert nested JSON structures into table structures in relational databases; Determine how to convert relational data into a structure suitable for target storage, including hierarchical JSON and XML, and automatically generate semantic mapping rules between fields, including data type conversion, value range conversion suggestions, and structure conversion logic; LLM large language model is used to detect potential semantic conflicts and output dynamic pattern mapping solutions.

[0029] Specifically, Input enhanced schema descriptions: The LLM enhanced schema descriptions of the multiple data sources obtained in step 102 are used as input to the large model for schema alignment and mapping generation. These enhanced schema descriptions contain rich semantic information, providing a foundation for the model to understand the differences and connections between different data sources.

[0030] Prompt Engineering Design: Designing specific prompts requires the model to play the role of a "data integration engine," requiring extensive domain knowledge and data integration experience. For example, a prompt might be, "As a data integration engine, please analyze the schema descriptions of the following two data sources, identify semantically equivalent or related fields, resolve synonyms ('user ID' and 'UID' represent the same concept) and homonyms ('address' may have different meanings in different data sources), and reason about transformations between different structures, such as flattening nested JSON structures into relational tables and vice versa."

[0031] Large Model Processing: The model first compares and analyzes multiple enhanced schema descriptions. Leveraging its powerful semantic understanding capabilities, it identifies semantically equivalent fields across different data sources, even if these fields have different names (synonyms) or the same name but different semantics (homononyms). For example, it determines whether "product code" in data source A and "SKU" in data source B represent the same product identification concept. It then infers the conversion relationships between different data structures. For example, for nested JSON structures, it determines how to convert them into table structures in a relational database, including how to split nested fields and establish associations. For relational data, it determines how to convert them into structures suitable for the target storage, such as hierarchical JSON or XML. Simultaneously, it automatically generates semantic mapping rules between fields, including data type conversion (converting string dates to datetime types), value range conversion suggestions (converting "male / female" in data source A to "1 / 2" in data source B), and structure conversion logic (clarifying the specific mapping relationship between JSONPath and SQL columns, such as "$.order.items"). .product_id" is mapped to the "Product ID" column in the order details table). Furthermore, the model detects potential semantic conflicts or uncertainties, such as when the data types of fields with the same semantics in two data sources differ significantly and cannot be automatically converted, or when there are inconsistencies in the field value ranges, and generates corresponding reports.

[0032] Output dynamic mapping scheme: Generates a cross-data source, semantically driven dynamic schema mapping scheme. This scheme includes a detailed field mapping table that lists the correspondence between each data source field and target field, conversion rules, and considerations; a structure conversion logic document that describes the conversion steps and methods between different data structures; and a report on potential semantic conflicts and uncertainties to provide reference for subsequent data processing.

[0033] Step 104: Input the missing data records and multiple enhanced data source schema descriptions in the data fusion into the LLM large language model, analyze the conflict causes and infer the conflict resolution strategy, and output data cleaning suggestions and data analysis results; Specifically, in this embodiment, a large model is used to analyze the causes of conflicts, and conflicts with inconsistent values ​​are judged to be data errors or timestamp differences; Conflict resolution strategies based on credibility, timeliness, and data source priority or business logic reasoning; For clear data type conversion and rule-based value filling, generated processed data is output with data cleaning suggestions.

[0034] The LLM large language model is used to understand the semantics of each record and analyze the meaning and function of each attribute field in the record; Compare data attributes and semantics, determine whether the records belong to the same entity based on the importance, similarity and correlation factors of the attributes, and parse based on the judgment results to obtain data parsing results.

[0035] Specifically, Data problem detection: During data conversion or fusion, real-time detection of data conflicts (inconsistent attribute values ​​of the same entity in different sources, such as a customer's phone number being different in two data sources), missing values ​​(null values ​​for a key attribute of an entity), or anomalies (negative numbers or obviously unreasonable values ​​in the age field) can be performed.

[0036] Input Information Preparation: When a data issue is detected, the specific conflicting / missing data record (or its contextual summary, such as its associated record information), the relevant schema information output in step 102 (including field semantic descriptions and constraints), and the fused target schema or business rule requirements (format and value range requirements for fields in the target data) are input into the big model. Big Model Processing: Leveraging the big model's contextual reasoning capabilities, the cause of the conflict is first analyzed. For example, for value inconsistencies, the cause is determined to be a data error (typing error), a timestamp discrepancy (different data sources record information at different points in time), or a different perspective (different data sources describe the same entity from different dimensions). Then, based on credibility (data source authority, historical data quality), timeliness (latest updated data is more reliable), data source priority (data from core business systems takes precedence over data from auxiliary systems), or business logic (conflict resolution strategies specified by business rules), the most appropriate conflict resolution strategy is inferred. For example, the value from the most recently updated data source, the value from a highly trusted data source, or a reasonable value derived from other relevant fields (calculating age from birth date and current date to fill in missing age values) can be selected. For missing values, we generate reasonable filling suggestions, such as those based on field semantics (when the "gender" field is missing, we infer it based on common gender characteristics in names) and relevant record context (the distribution of values ​​for this field in other entities of the same category). We also identify and flag high-uncertainty conflicts that require manual intervention, such as those where the cause of the conflict is unclear or where there are multiple possible resolution strategies that are difficult to automatically determine.

[0037] Output of cleaning and resolution results: For specific data issues, output of cleaning / resolution suggestions or automated execution results. For issues that can be automatically addressed, such as explicit data type conversions and rule-based value filling, processed data is directly generated. For issues requiring manual intervention, detailed conflict descriptions, possible cause analysis, and recommended solutions are provided for further manual processing.

[0038] The model first performs a semantic understanding of each record, analyzing the meaning and function of each attribute field within the record. It then comprehensively compares all attributes and their semantics, taking into account factors such as attribute importance, similarity, and relevance, to infer whether the records belong to the same entity. For example, if two records share the same ID number, they are directly considered the same entity. If the ID numbers are different, but multiple key attributes such as name, address, and phone number are highly similar, further analysis is conducted to determine the discrepancy, such as possible aliases or address changes for the same entity. A judgment result (yes / no) is generated, along with key reasoning, detailing which information supports or opposes the match, such as "Basis supporting the match: identical name, same address, same last four digits of phone number; basis opposing the match: different ID numbers, possible data entry error or entity change." If a unified entity representation is required, the model integrates high-quality information from various sources, such as selecting attribute values ​​from highly trustworthy data sources and filling in missing information to create a complete and accurate entity record.

[0039] Output: Outputs the entity matching judgment result and confidence / explanation. The confidence level can be expressed as a percentage, reflecting the model's degree of certainty about the match result; the explanation details the basis for the judgment and the reasoning process. For the fused entity record, a unified representation is provided that integrates the valid information from each data source to ensure the integrity and accuracy of the entity information.

[0040] Step 105: Load the fused data into a target data storage based on the dynamic pattern mapping solution, data cleaning suggestions, and data analysis results.

[0041] Specifically, in this embodiment, based on the dynamic schema mapping solution, data cleaning suggestions and data analysis results, data conversion, cleaning and merging operations are performed; According to the resolution strategy, data conflicts, missing data and anomalies are handled, and the same entity data from different data sources are integrated to obtain fused data; The fused data is loaded into a target data storage, which includes at least a data warehouse, a data lake, and a database.

[0042] Specifically, Data processing execution: Based on the dynamic mapping scheme generated in step 3 and the cleaning / digestion / parsing results of steps 4 / 5, data conversion, cleaning, and merging operations are automatically or semi-automatically performed. Automatic execution is suitable for processing steps with clear rules and low risks, such as converting data types according to preset mapping rules and selecting appropriate values ​​according to the digestion strategy. Semi-automatic execution is suitable for steps that require manual review or intervention, such as manually confirming conflicts marked as high uncertainty before processing. During the data conversion process, mapping rules and conversion logic are strictly followed to ensure data accuracy and consistency. During the cleaning process, data conflicts, missing data, and anomalies are handled according to the digestion strategy. In the merge operation, the same entity data from different data sources is integrated to form a unified record.

[0043] Data loading: Load the fused data into the target data store, which can be a data warehouse, data lake, or database. Choose the appropriate storage method based on your application needs. During the loading process, ensure that the data format meets the requirements of the target storage, establish appropriate data indexes and partitions, and improve data access efficiency.

[0044] Quality feedback loop (optional): Establish a quality feedback mechanism to conduct a quality assessment of the fused data. Evaluation metrics include the residual conflict rate (the percentage of unresolved data conflicts), data completeness (the proportion of missing values), and data accuracy (the degree of consistency with the real data). Collect user feedback to understand user satisfaction with the fused data and any problems encountered during use. Input the quality assessment results and newly discovered pattern issues into the large model as feedback information to optimize prompts in subsequent steps or fine-tune the LLM model. For example, if certain types of semantic conflicts are frequently missed, adjust the prompt to more clearly guide the model's attention to these conflicts. If the model's confidence level is low during entity resolution, fine-tune the model to improve the accuracy of its judgments. Through continuous feedback and optimization, the effectiveness and performance of the entire data integration and fusion method can be improved.

[0045] Its beneficial effects include: 1. Leveraging natural language processing technology and the intelligent parsing capabilities of large models, it automatically completes information extraction and structured conversion without requiring extensive manual programming, significantly improving the efficiency of data access and initial processing and reducing labor and time costs. 2. Compared with traditional manual pattern alignment and mapping methods, it is faster and more accurate, adapting to complex and changing data structures, and improving the flexibility and efficiency of data integration. 3. It can continuously improve the effectiveness and performance of data integration and fusion methods, enabling them to better adapt to changing data environments and business needs, and ensuring the long-term effectiveness and reliability of data integration work.

[0046] See also Figure 2In the data integration and fusion method based on a large model, the multiple enhanced data source schema descriptions are input into the LLM large language model. The model is used as a data integration engine through a prompting project to guide the model to identify semantically equivalent fields between different data source schemas and output a dynamic schema mapping solution, including the following steps: Step 201: Use the model to compare and analyze multiple enhanced schema descriptions to identify semantically equivalent fields in different data sources; Step 202: Infer the conversion relationship between different data structures and determine how to convert the nested JSON structure into a table structure in a relational database; Step 203: Determine how to convert relational data into a structure suitable for target storage, including hierarchical JSON and XML, and automatically generate semantic mapping rules between fields, including data type conversion, value range conversion suggestions, and structure conversion logic; Step 204: Utilize the LLM large language model to detect potential semantic conflicts and output a dynamic mode mapping solution.

[0047] The above is an introduction to the embodiment of the data integration and fusion method based on the large model of the present invention. Figure 3 In the data integration and fusion system based on the big model, the data integration and fusion system includes the following modules: A multi-source data acquisition module is used to acquire multi-source heterogeneous data, perform structural conversion on semi-structured data in the multi-source heterogeneous data, read pattern information and sample data from the structured data, and obtain initial multi-source data; A data enhancement recognition module is used to use the metadata information and representative sample data of the initial multi-source data as the input context of the pre-trained LLM large language model, guide the model training through the prompt engineering, and obtain multiple enhanced data source pattern descriptions; A data dynamic mapping module is used to input the multiple enhanced data source schema descriptions into the LLM large language model, use prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; A data analysis and cleaning module is used to input missing data records and multiple enhanced data source schema descriptions in data fusion into the LLM large language model, analyze conflict causes and infer conflict resolution strategies, and output data cleaning suggestions and data analysis results; The data fusion storage module is used to load the fused data into the target data storage based on the dynamic pattern mapping solution, data cleaning suggestions and data analysis results.

[0048] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. The data integration and fusion method based on a large model is characterized by: The data integration and fusion method comprises the following steps: Acquiring multi-source heterogeneous data, performing structural conversion on semi-structured data in the multi-source heterogeneous data, and reading pattern information and sample data from the structured data to obtain initial multi-source data; The metadata information and representative sample data of the initial multi-source data are used as the input context of the pre-trained LLM large language model, and the model is trained through the prompt engineering guidance to obtain multiple enhanced data source pattern descriptions; Input the multiple enhanced data source schema descriptions into the LLM large language model, use the prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; Input missing data records and multiple enhanced data source schema descriptions in data fusion into the LLM large language model, analyze conflict causes and infer conflict resolution strategies, and output data cleaning suggestions and data analysis results; The fused data is loaded into a target data storage based on the dynamic pattern mapping scheme, data cleaning suggestions and data parsing results.

2. The data integration and fusion method based on a large model according to claim 1, characterized in that: The acquiring of multi-source heterogeneous data, performing structural conversion on semi-structured data in the multi-source heterogeneous data, and reading pattern information and sample data from the structured data to obtain initial multi-source data includes: Taking entity names, attribute values, and event time information from the text data in the multi-source heterogeneous data, and converting them into a structured table format; Parsing the JSON and XML formats in the multi-source heterogeneous data and extracting node information, and converting it into a semi-structured representation of a flat table structure according to preset rules; The metadata information and constraints in the structured data in the multi-source heterogeneous data are read, sample data is extracted, and initial multi-source data is obtained.

3. The data integration and fusion method based on a large model according to claim 1, characterized in that: The metadata information and representative sample data of the initial multi-source data are used as the input context of the pre-trained LLM large language model, and the model is trained through the prompt engineering guidance to obtain multiple enhanced data source mode descriptions, including: Integrate the metadata information of each data source and the extracted sample data to obtain input context data; Using a pre-trained LLM large language model to deeply process the input context data; Perform semantic analysis on each field to identify the semantic relationship between fields. Primary and foreign key relationships represent the association between entities, and parent-child relationships represent the hierarchical structure of data. Infer the business domain and context to which the data source belongs, and obtain the business scope and related background involved in the data to obtain multiple enhanced data source schema descriptions.

4. The data integration and fusion method based on a large model according to claim 1, characterized in that: The multiple enhanced data source schema descriptions are input into the LLM large language model, and the model is used as a data integration engine through prompt engineering, guiding the model to identify semantically equivalent fields between different data source schemas and output a dynamic schema mapping solution, including: Use the model to compare and analyze multiple enhanced schema descriptions to identify semantically equivalent fields in different data sources; Reason about the conversion relationships between different data structures and determine how to convert nested JSON structures into table structures in relational databases; Determine how to convert relational data into a structure suitable for target storage, including hierarchical JSON and XML, and automatically generate semantic mapping rules between fields, including data type conversion, value range conversion suggestions, and structure conversion logic; LLM large language model is used to detect potential semantic conflicts and output dynamic pattern mapping solutions.

5. The data integration and fusion method based on a large model according to claim 1, characterized in that: The missing data records and multiple enhanced data source schema descriptions in the data fusion are input into the LLM large language model, the conflict causes are analyzed and conflict resolution strategies are inferred, and data cleaning suggestions and data analysis results are output, including: Use a large model to analyze the causes of conflicts and identify conflicts with inconsistent values ​​as data errors or timestamp differences; Conflict resolution strategies based on credibility, timeliness, and data source priority or business logic reasoning; For clear data type conversion and rule-based value filling, generated processed data is output with data cleaning suggestions.

6. The data integration and fusion method based on a large model according to claim 1, characterized in that: The method includes inputting missing data records and multiple enhanced data source schema descriptions in the data fusion into the LLM large language model, analyzing conflict causes and inferring conflict resolution strategies, and outputting data cleaning suggestions and data analysis results, and further includes: The LLM large language model is used to understand the semantics of each record and analyze the meaning and function of each attribute field in the record; Compare data attributes and semantics, determine whether the records belong to the same entity based on the importance, similarity and correlation factors of the attributes, and parse based on the judgment results to obtain data parsing results.

7. The data integration and fusion method based on a large model according to claim 1, characterized in that: The step of loading the fused data into a target data storage based on the dynamic pattern mapping solution, the data cleaning suggestion, and the data analysis result includes: Based on the dynamic schema mapping solution, data cleaning suggestions and data analysis results, perform data conversion, cleaning and merging operations; According to the resolution strategy, data conflicts, missing data and anomalies are handled, and the same entity data from different data sources are integrated to obtain fused data; The fused data is loaded into a target data storage, which includes at least a data warehouse, a data lake, and a database.

8. The data integration and fusion system based on the large model is characterized by: The data integration and fusion system includes the following modules: A multi-source data acquisition module is used to acquire multi-source heterogeneous data, perform structural conversion on semi-structured data in the multi-source heterogeneous data, read pattern information and sample data from the structured data, and obtain initial multi-source data; A data enhancement recognition module is used to use the metadata information and representative sample data of the initial multi-source data as the input context of the pre-trained LLM large language model, guide the model training through the prompt engineering, and obtain multiple enhanced data source pattern descriptions; A data dynamic mapping module is used to input the multiple enhanced data source schema descriptions into the LLM large language model, use prompt engineering to make the model a data integration engine, guide the model to identify semantically equivalent fields between different data source schemas, and output a dynamic schema mapping solution; A data analysis and cleaning module is used to input missing data records and multiple enhanced data source schema descriptions in data fusion into the LLM large language model, analyze conflict causes and infer conflict resolution strategies, and output data cleaning suggestions and data analysis results; The data fusion storage module is used to load the fused data into the target data storage based on the dynamic pattern mapping solution, data cleaning suggestions and data analysis results.

9. The data integration and fusion system based on a large model according to claim 8, characterized in that: The multi-source data acquisition module includes the following submodules: A conversion submodule, configured to extract entity names, attribute values, and event time information from the text data in the multi-source heterogeneous data and convert them into a structured table format; A parsing submodule is used to parse the JSON and XML formats in the multi-source heterogeneous data and extract node information, and convert it into a semi-structured representation of a flat table structure according to preset rules; The reading submodule is used to read metadata information and constraints in the structured data in the multi-source heterogeneous data, extract sample data, and obtain initial multi-source data.

10. The data integration and fusion system based on a large model according to claim 8, characterized in that: The data fusion storage module includes the following submodules: A processing submodule, configured to perform data conversion, cleaning, and merging operations based on the dynamic schema mapping solution, data cleaning suggestions, and data parsing results; The integration submodule is used to handle data conflicts, missing data, and anomaly issues according to the resolution strategy, and integrate the same entity data from different data sources to obtain fused data; The storage submodule is used to load the fused data into a target data storage, which includes at least a data warehouse, a data lake, and a database.

Images