Metadata generation method and apparatus

Abstract

This application provides a method and apparatus for generating metadata. The method includes: acquiring a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, and the full basic information includes at least two granularities, with each sub-data structure having a corresponding granularity; encapsulating the full basic information using the meta-model to obtain full data instances; and clustering and merging the full data instances using a retrieval-enhanced generation model based on a preset clustering and merging method to generate target metadata. The retrieval-enhanced generation model is used to cluster the full data instances according to the preset clustering and merging method, which includes grouping according to the descriptive information of the full data instances. This achieves the technical effect of automatically generating metadata and solves the problems of low metadata generation efficiency and difficulty in guaranteeing data fusion quality in related technologies.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for generating metadata. Background Technology

[0002] Data fusion is the logical or physical aggregation of data from different sources, formats, and characteristics to provide enterprises with comprehensive data sharing. Designing a consistent data model to ensure data integrity, accuracy, and consistency is key to data fusion, while metadata serves as the foundation and link for building data models and enabling data sharing. Currently, metadata is manually generated.

[0003] However, manually generating unified metadata has drawbacks such as large workload, difficulty in normalization, high error rate, and poor maintainability, resulting in low metadata generation efficiency and difficulty in ensuring the quality of data fusion. Summary of the Invention

[0004] This application provides a method and apparatus for generating metadata, which at least solves the problems of low metadata generation efficiency and difficulty in guaranteeing data fusion quality in related technologies.

[0005] According to one embodiment of this application, a method for generating metadata is provided, comprising: obtaining a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, the full basic information includes at least two granularities, and each sub-data structure has a corresponding granularity; encapsulating the full basic information using the meta-model to obtain full data instances, wherein the full data instances represent the full basic information stored according to the rules indicated by the meta-model; and clustering and merging the full data instances using a retrieval-enhanced generation model based on a preset clustering and merging method to generate target metadata, wherein the retrieval-enhanced generation model is used to cluster the full data instances according to the preset clustering and merging method, the preset clustering and merging method including grouping according to the descriptive information of the full data instances.

[0006] In an exemplary embodiment, the step of clustering and merging the full data instances based on a preset clustering and merging method using a retrieval-enhanced generation model to generate target metadata includes: clustering and merging the full data instances based on the preset clustering and merging method using the retrieval-enhanced generation model to determine target aggregate instances, wherein the target aggregate instances include instances corresponding to table-level metadata, and the at least two sub-data structures include the table-level metadata; generating target metadata based on the target aggregate instances, wherein the target metadata is configured to be assembled according to a target machine language.

[0007] In an exemplary embodiment, the step of clustering and merging the full data instances based on a preset clustering and merging method using the retrieval enhancement generation model to determine the target aggregated instance includes: setting the preset clustering and merging method; storing the preset clustering and merging method in the knowledge base corresponding to the retrieval enhancement generation model; and automatically clustering and merging the full data instances using the knowledge base referenced by the retrieval enhancement generation model to determine the target aggregated instance.

[0008] In an exemplary embodiment, the step of clustering and merging the full data instances based on a preset clustering and merging method using the retrieval enhancement generation model to determine the target aggregated instance includes: obtaining the description information corresponding to each table-level metadata in the full data instances through the retrieval enhancement model; grouping the full data instances according to the description information to obtain at least two groups of data instances corresponding to the at least two sub-data structures respectively; and merging each group of data instances according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance.

[0009] In an exemplary embodiment, the step of merging the data instances for each group according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregate instance includes: when the sub-data structure corresponds to the table-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is considered as the first group of data instances: adding and filling the table-level data source attribute parameters of each data instance in the first group of data instances to obtain target table-level data source attribute parameters; setting the table-level identifier attribute parameter of each data instance in the first group of data instances to be filled according to the first language business text to obtain target table-level identifier attribute parameters corresponding to the target aggregate instance; setting the table-level description attribute parameter of each data instance in the first group of data instances to be filled according to the second language business text to obtain target table-level description attribute parameters, wherein the first language business text and the second language business text use different languages; generating the target aggregate instance based on the target table-level data source attribute parameters, the target table-level identifier attribute parameters, and the target table-level description attribute parameters.

[0010] In an exemplary embodiment, the step of merging the data instances for each group according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance includes: when the sub-data structure corresponds to the field-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is considered as the second group of data instances: performing semantic deduplication on the field-level description attribute parameters and field-level data item attribute parameters of each data instance in the second group of data instances, setting the deduplicated field-level description attribute parameters to be filled according to the second language business text to obtain the target field-level description attribute parameters; and setting the field-level label of each data instance in the second group of data instances to... The identification attribute parameter is set to be filled according to the first language business text to obtain the target field-level identification attribute parameter, wherein the first language business text and the second language business text use different languages; the field-level data type attribute parameter of each data instance in the second group of data instances is filled with the original value to obtain the target field-level data type attribute parameter; the field-level size attribute parameter of each data instance in the second group of data instances is filled with the maximum value to obtain the target field-level size attribute parameter; the target aggregation instance is generated based on the target field-level description attribute parameter, the target field-level identification attribute parameter, the target field-level data type attribute parameter, and the target field-level size attribute parameter.

[0011] In an exemplary embodiment, the step of merging the data instances for each group according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance includes: when the sub-data structure corresponds to dictionary-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is considered a third group of data instances: semantically deduplicating the dictionary-level value parameters of each data instance in the third group of data instances to obtain target dictionary-level value attribute parameters; filling the dictionary-level description attribute parameters and dictionary-level data type attribute parameters of each data instance in the third group of data instances with their original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; and generating the target aggregated instance based on the target dictionary-level value attribute parameters, the target dictionary-level description attribute parameters, and the target dictionary-level data type attribute parameters.

[0012] According to another embodiment of this application, a metadata generation apparatus is provided, comprising: an acquisition module, configured to acquire a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, the full basic information includes at least two granularities, and each sub-data structure has a corresponding granularity; an encapsulation module, configured to encapsulate the full basic information using the meta-model to obtain full data instances, wherein the full data instances represent the full basic information stored according to the rules indicated by the meta-model; and a clustering module, configured to cluster and merge the full data instances using a retrieval-enhanced generation model based on a preset clustering and merging method to generate target metadata, wherein the retrieval-enhanced generation model is used to cluster the full data instances according to the preset clustering and merging method, the preset clustering and merging method including grouping according to the description information of the full data instances.

[0013] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0014] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This application, firstly, establishes a meta-model design that includes at least two sub-data structures, such as table-level metadata and field-level metadata, while the full-scale foundational information includes at least two granularities, such as the database and table levels. This design ensures that each sub-data structure corresponds to a specific granularity of foundational information, thereby achieving precise data encapsulation. Next, the full-scale foundational information is encapsulated through the meta-model to generate full-scale data instances. In this step, the full-scale foundational information is organized and stored according to the rules of the meta-model, forming structured data instances and providing a standardized foundation for subsequent data processing and analysis. Finally, based on a preset clustering and merging method, a Retrieval Enhancement Generative Model (RAG model) is used to cluster and merge the full-scale data instances. During this process, the RAG model groups the full-scale data instances according to their descriptive information, achieving logical centralization and integration of the data, and ultimately automatically generating target metadata. This achieves the goal of unified management and fusion of data from different sources and formats, thus realizing the technical effect of automated metadata generation and solving the problems of low metadata generation efficiency and difficulty in guaranteeing data fusion quality in related technologies. Attached Figure Description

[0016] Figure 1This is a hardware structure block diagram of a server device for a metadata generation method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a method for generating metadata according to an embodiment of this application;

[0018] Figure 3 This is a flowchart illustrating a method for generating metadata according to an embodiment of this application;

[0019] Figure 4 This is a structural block diagram of a metadata generation apparatus according to an embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a metadata generation method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown. A processor 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0023] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the metadata generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] This embodiment provides a method for generating metadata. Figure 2 This is a flowchart of a metadata generation method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0026] Step S202: Obtain a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures and the full basic information includes at least two granularities, and each sub-data structure has a corresponding granularity.

[0027] Step S204: Use the meta-model to encapsulate the full set of basic information to obtain a full set of data instances, where the full set of data instances represents the full set of basic information stored according to the rules indicated by the meta-model.

[0028] Step S206: Based on a preset clustering and merging method, cluster and merge all data instances using a retrieval-enhanced generation model to generate target metadata. The retrieval-enhanced generation model is used to cluster all data instances according to the preset clustering and merging method, which includes grouping according to the description information of all data instances.

[0029] This application, firstly, establishes a meta-model design that includes at least two sub-data structures, such as table-level metadata and field-level metadata, while the full-scale foundational information includes at least two granularities, such as the database and table levels. This design ensures that each sub-data structure corresponds to a specific granularity of foundational information, thereby achieving precise data encapsulation. Next, the full-scale foundational information is encapsulated through the meta-model to generate full-scale data instances. In this step, the full-scale foundational information is organized and stored according to the rules of the meta-model, forming structured data instances and providing a standardized foundation for subsequent data processing and analysis. Finally, based on a preset clustering and merging method, a Retrieval Enhancement Generative Model (RAG model) is used to cluster and merge the full-scale data instances. During this process, the RAG model groups the full-scale data instances according to their descriptive information, achieving logical centralization and integration of the data, and ultimately automatically generating target metadata. This achieves the goal of unified management and fusion of data from different sources and formats, thus realizing the technical effect of automated metadata generation and solving the problems of low metadata generation efficiency and difficulty in guaranteeing data fusion quality in related technologies.

[0030] The entities that perform the above steps can be servers, terminals, etc., but are not limited to these.

[0031] The execution order of steps S202 and S204 can be interchanged; that is, step S204 can be executed first, and then step S202 can be executed.

[0032] Optionally, in this embodiment, step S202 refers to the process of obtaining a pre-created meta-model and full-scale basic information extracted from at least one data source. The meta-model includes at least two sub-data structures, which refer to data descriptions of different levels and types that constitute the meta-model, such as table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. These sub-data structures correspond to different data granularities; for example, table-level metadata corresponds to tables in a database, and field-level metadata corresponds to fields in a table. Full-scale basic information includes at least two granularities, which refer to the level of detail of the information extracted from the data source, such as database name, table name, field name, and their attributes such as field length and field precision. Each sub-data structure has a corresponding granularity, ensuring that the meta-model can comprehensively describe and encapsulate the basic information of the data source.

[0033] Optionally, in this embodiment, step S204 refers to the process of encapsulating all basic information using a meta-model to obtain full data instances. A full data instance represents all basic information stored according to the rules indicated by the meta-model. This process involves organizing and storing the basic information extracted from the data source according to the structure and requirements of the meta-model, forming data instances in a unified format. These data instances reflect the complete picture of the data source, including table structure, field attributes, data content, etc., and are the foundation for subsequent data fusion and automatic metadata generation.

[0034] Optionally, in this embodiment, step S206 refers to the process of clustering and merging all data instances using a retrieval-enhanced generation model based on a preset clustering and merging method, thereby generating target metadata. The retrieval-enhanced generation model is used to cluster all data instances according to the preset clustering and merging method. This model combines retrieval and generation technologies, enabling logical or physical centralized processing of data instances based on rules in the knowledge base. The preset clustering and merging method includes grouping according to the descriptive information of all data instances. This means that based on descriptive information such as the business semantics and attribute characteristics of the data instances, data instances with the same or similar characteristics are grouped together for further merging processing to generate target metadata. This metadata will be used to guide subsequent data fusion and application development.

[0035] It should be noted that during the process of acquiring the pre-created metamodel and the full set of basic information extracted from at least one data source, the sub-data structures of the metamodel and the granularity of the full set of basic information can have various combinations and forms. For example, the sub-data structures of the metamodel may include table-level metadata and field-level metadata, while the granularity of the full set of basic information may be database-level database names and table names, or field-level field names and field types. Each sub-data structure corresponds to a granularity, such as table-level metadata corresponding to database and table information, and field-level metadata corresponding to detailed field information. In addition, the metamodel may also contain element-level metadata and dictionary-level metadata, corresponding to the granularity of data item content and dictionary content, respectively. This application does not limit this.

[0036] In the process of encapsulating the full set of basic information using the meta-model to obtain a full data instance, the storage rules and format of the full data instance can vary according to different needs and scenarios. For example, the full data instance can be stored in JSON format, containing detailed information on table-level, field-level, element-level, and dictionary-level metadata. This information may include the name of the data source, the description of the table, the type and size of the fields, the content and type of the data items, the content and type of the dictionary, etc. The full data instance may also include additional information such as data version information, creation time, and modification time to meet the needs of data management and tracking. This application does not limit this.

[0037] The full set of data instances is clustered and merged using a retrieval-enhanced generative model based on a preset clustering and merging method to generate target metadata. During this process, the preset clustering and merging method can be customized according to different business logics and data characteristics. For example, it can be grouped according to the business domain of the data source, such as merging all finance-related tables into one metadata group, or grouped according to the data update frequency, such as processing real-time updated data and non-real-time updated data separately. Furthermore, clustering and merging can also be based on data access patterns or user query habits to optimize data retrieval and access efficiency. The result of clustering and merging can be a new metadata model or an update and extension of an existing metadata model. This application does not limit this.

[0038] For example, consider the data fusion scenario of an e-commerce platform:

[0039] S1. First, it's necessary to obtain the pre-created meta-model and all basic information extracted from at least one data source. In this scenario, the meta-model includes at least two sub-data structures, such as table-level metadata and field-level metadata. The full basic information includes at least two granularities, such as field granularity and record granularity. Table-level metadata corresponds to table information in the database, field-level metadata corresponds to field information in the table, field granularity provides information such as field name and field type, and record granularity provides the specific data record content.

[0040] S2, the full set of basic information is encapsulated using the metamodel to obtain full data instances. In this process, the full set of basic information is organized into unified data instances according to the rules of the metamodel. For example, the metamodel defines a JSON-formatted template containing the structure of table-level metadata and field-level metadata. Data from the full set of basic information is populated into this template to form full data instances. These instances include user order data and product information data from e-commerce platforms. Each data instance follows the structure of the metamodel, ensuring data consistency and understandability.

[0041] S3. Based on a preset clustering and merging method, the full set of data instances is clustered and merged using a retrieval-enhanced generative model to generate target metadata. In this step, the preset clustering and merging method may include grouping according to the business description information of the full set of data instances. For example, order data from different tables can be clustered and merged based on order status (e.g., "pending payment," "paid") to generate a new metadata model containing a unified view of all related orders. The retrieval-enhanced generative model plays a crucial role in this process; it not only clusters according to preset rules but also understands and processes natural language descriptions, making the clustering and merging more intelligent and accurate.

[0042] The above process automates metadata generation, significantly improving the efficiency and quality of data fusion. In e-commerce platform scenarios, this method can quickly integrate user order information from different data sources, reducing the complexity and error rate of manual operations and improving the speed and accuracy of data processing. Furthermore, the automated generation of metadata ensures data consistency and maintainability, providing a solid foundation for subsequent data analysis and business decisions. This method also boasts excellent scalability, adapting to different business needs and data structure changes, offering a flexible and efficient solution for data fusion.

[0043] As an optional approach, based on a preset clustering and merging method, all data instances are clustered and merged using a retrieval-enhanced generative model to generate target metadata, including:

[0044] Based on a preset clustering and merging method, all data instances are clustered and merged using a retrieval-enhanced generative model to determine target aggregate instances. Target aggregate instances include instances corresponding to table-level metadata, and at least two sub-data structures include table-level metadata.

[0045] Target metadata is generated based on the target aggregation instance, where the target metadata is set to be assembled according to the target machine language.

[0046] Optionally, in this embodiment, the aforementioned preset clustering and merging method refers to a method for logically or physically aggregating all data instances according to specific rules or standards. This method includes, but is not limited to, grouping data instances based on descriptive information such as business semantics and attribute characteristics. For example, in a data fusion scenario on an e-commerce platform, the preset clustering and merging method may aggregate order data from different data sources based on the business description information of the orders, such as "order status" or "product category." Such clustering and merging helps to form a more complete and consistent data view, facilitating subsequent data analysis and decision support.

[0047] Optionally, in the embodiments of this application, the aforementioned target aggregation instance refers to the aggregated data instance obtained after processing by the retrieval-enhanced generative model. These instances include instances corresponding to table-level metadata, and at least two sub-data structures include table-level metadata. The target aggregation instance may include, but is not limited to, table-level metadata, field-level metadata, and dictionary-level metadata. For example, in the data fusion of a tourism platform, the target aggregation instance may include tourism order information from different data sources, which is aggregated into a unified view. The table-level metadata may contain basic order information, the field-level metadata may contain specific field information of the order, such as order number, customer name, etc., and the dictionary-level metadata may contain enumerated values ​​of order status, such as "pending payment" and "paid".

[0048] Optionally, in this embodiment, the aforementioned target metadata refers to metadata generated based on target aggregation instances, which is configured to be assembled according to the target machine language. Target metadata is data used to describe data, providing a framework and rules for data management and use. For example, when building a data warehouse, target metadata may include table definitions, field data types and lengths, index information, etc. This metadata can be used to generate SQL statements to create tables and fields in the database, or to guide data flow during data migration and transformation. The target machine language can be SQL, NoSQL, or any other programming language used for data manipulation and storage. In this way, target metadata not only provides a structured description of the data but also ensures that the data can be effectively migrated and used between different systems and platforms.

[0049] It should be noted that during the clustering and merging of all data instances using a retrieval-enhanced generative model based on a preset clustering and merging method, this preset method can be designed in various ways according to different business needs and data characteristics. For example, the clustering and merging method can be based on dimensions such as data source, data business attributes, or data timestamps. In e-commerce platforms, order data can be clustered and merged according to dimensions such as product category, order status, or customer region to form target aggregate instances. In the financial field, transaction records can be clustered and merged according to dimensions such as transaction type, amount range, or account status. In the healthcare field, medical records can be clustered and merged according to dimensions such as patient diagnosis, treatment stage, or drug category. This application does not limit this.

[0050] It should be further clarified that the generation of target aggregation instances is not limited to table-level metadata; it can also include various sub-data structures such as field-level metadata, element-level metadata, and dictionary-level metadata. For example, when building a comprehensive customer management system, the target aggregation instance may include basic customer information (table-level metadata), customer purchase history (field-level metadata), customer's specific interaction records (element-level metadata), and customer satisfaction survey options (dictionary-level metadata). This design allows the target aggregation instance to comprehensively reflect the characteristics of the data, providing rich information for subsequent data applications. This application does not limit this aspect.

[0051] Finally, it should be noted that the assembly process of target metadata generated based on target aggregation instances can employ different target machine languages ​​depending on the technology platform and application scenario. For example, in cloud service platforms, target metadata may need to be assembled according to the RESTful API style to ensure compatibility with cloud service interfaces; in traditional relational databases, target metadata may need to be assembled according to SQL to create table structures and indexes; in big data processing platforms, target metadata may need to be assembled according to the language specifications in the Hadoop ecosystem to work with components such as HDFS and MapReduce. This flexibility allows target metadata to adapt to different technology environments and meet diverse data processing needs. This application does not impose any limitations on this.

[0052] Through the embodiments of this application, a metadata self-generation data fusion method based on the RAG (Retrieval-augmented Generation) model is adopted. By processing all data instances through preset clustering and merging rules, and utilizing the retrieval enhancement generation capability of the RAG model, the data instances are clustered and merged to determine the target aggregation instance. This ensures the comprehensiveness and consistency of data fusion, and achieves the goal of logically or physically concentrating data from different sources, formats, and characteristics. This realizes the automation and intelligent technical effects of data fusion.

[0053] Furthermore, target metadata is generated based on the target aggregated instance, where the target metadata is configured to be assembled according to the target machine language. Through an automated metadata generation process, the aggregated instance is converted into a machine-readable metadata format, enabling the target metadata to be directly used for database table creation and data field definition. This not only improves the efficiency of metadata generation but also ensures high compatibility between the generated metadata and the target machine language. This further automates the data fusion process and achieves seamless integration between systems, thereby improving data processing efficiency, reducing human error, and enhancing data consistency. This demonstrates a high degree of automation and precision in data fusion technology.

[0054] As an optional approach, the full set of data instances are clustered and merged using a retrieval-enhanced generative model based on a preset clustering and merging method to determine the target aggregated instance. This includes: setting a preset clustering and merging method; storing the preset clustering and merging method in the knowledge base corresponding to the retrieval-enhanced generative model; and automatically clustering and merging the full set of data instances by referencing the knowledge base through the retrieval-enhanced generative model to determine the target aggregated instance.

[0055] Optionally, in this embodiment, setting a preset clustering merging method refers to defining a set of rules or standards to guide the retrieval enhancement generative model in how to logically or physically centralize all data instances. This includes, but is not limited to, determining which data should be aggregated together and how to organize this data to form meaningful information units. For example, in medical data analysis, a preset clustering merging method might be set based on patient diagnoses, treatment types, or medication usage to ensure that similar medical events are aggregated together. In the field of business intelligence, this might involve setting clustering rules based on product categories, sales regions, or customer groups. These preset clustering merging methods help extract valuable information from large amounts of complex data and support decision-making. Storing the preset clustering merging methods in the knowledge base corresponding to the retrieval enhancement generative model means storing these rules or standards in a structured database that can be accessed and used by the model. This knowledge base provides the retrieval enhancement generative model with the necessary contextual information, enabling it to perform clustering merging tasks more accurately. The data in the knowledge base may include, but is not limited to, classification rules, attribute weights, historical clustering results, etc. For example, in a customer relationship management system (CRM) system, the knowledge base might contain classification rules for customer feedback. These rules are set based on customers' purchase history, service interactions, and feedback content, enabling the model to automatically identify and classify new customer feedback. Enhanced generative models, by referencing the knowledge base through retrieval, automatically cluster and merge all data instances to determine target aggregation instances. This refers to using the model's retrieval capabilities to find and reference relevant information in the knowledge base, and combining this with generative capabilities to create new data structures. This process involves analyzing and understanding all data instances, and integrating the data according to preset clustering and merging methods. For example, in social media analytics, the model might reference rules for topic classification in the knowledge base to automatically cluster and merge related posts and comments, forming a comprehensive view of a specific event or trend. In the financial field, the model might automatically aggregate transaction records into an overview of customer account activity based on transaction type and amount range rules in the knowledge base. Such automated clustering and merging not only improves data processing efficiency but also enhances data readability and usability.

[0056] It should be noted that preset clustering and merging methods can be designed in various ways to meet different business needs and data characteristics. For example, in e-commerce, clustering and merging methods can be set based on dimensions such as product category, customer rating, and sales region to aggregate similar products or customer feedback. In the healthcare field, clustering rules can be designed based on dimensions such as patient age, disease type, and treatment outcome to facilitate the analysis and research of similar cases. In the financial industry, clustering and merging methods can be set based on dimensions such as transaction amount, transaction time, and transaction type to facilitate risk assessment and fraud detection of transaction data. This application does not limit this approach.

[0057] Storing pre-defined clustering and merging methods in the knowledge base corresponding to the retrieval-enhanced generative model means that these rules can be customized and updated according to different scenarios and needs. For example, in customer service, the knowledge base can include sentiment analysis rules based on customer feedback, enabling the model to identify and classify customer emotional tendencies. In supply chain management, the knowledge base can include clustering rules based on logistics information, enabling the model to track and optimize the flow of goods. In education, the knowledge base can include clustering rules based on student performance, enabling the model to identify students' learning patterns and needs. The updating and maintenance of these knowledge bases can be dynamic to adapt to constantly changing data and business environments. This application does not limit this.

[0058] By retrieving the knowledge base referenced by the enhanced generative model, the process of automatically clustering and merging all data instances to determine the target cluster instance can involve various data processing techniques and algorithms. For example, in text data processing, the model can utilize natural language processing techniques to understand and classify text content. In the field of image recognition, the model can apply computer vision techniques to identify and cluster image data. In structured data analysis, the model can use machine learning algorithms to discover patterns and relationships in the data. The application of these techniques enables the model to accurately perform clustering and merging tasks and generate meaningful target cluster instances, providing a powerful tool for decision support and data analysis. This application does not limit this scope.

[0059] Through the embodiments of this application, by storing these preset clustering and merging methods into the knowledge base corresponding to the Retrieval Enhanced Generative Model (RAG model), the goal of automatically clustering and merging all data instances is achieved. The RAG model utilizes its retrieval enhancement capabilities, referencing the clustering and merging rules in the knowledge base to automatically identify and process data instances, thereby determining the target aggregation instances. This eliminates reliance on manual operation in the data fusion process, instead automating the intelligent classification and merging of data according to preset rules. This improves the efficiency and accuracy of data processing, achieving the goal of reducing manual intervention and increasing data processing efficiency and accuracy. This results in improved data fusion quality and automation levels, reducing errors caused by manual operation and enhancing the flexibility and scalability of data fusion, providing a more accurate and comprehensive data foundation for subsequent data analysis and decision support.

[0060] As an optional approach, the full data instances are clustered and merged using a retrieval-enhanced generation model based on a preset clustering and merging method to determine the target aggregated instance. This includes: obtaining the descriptive information corresponding to each table-level metadata in the full data instances through the retrieval-enhanced model; grouping the full data instances according to the descriptive information to obtain at least two groups of data instances corresponding to at least two sub-data structures; and merging each group of data instances according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance.

[0061] Optionally, in this embodiment, the aforementioned retrieval enhancement model refers to a model that combines retrieval and generation technologies, specifically RAG (Retrieval-augmented Generation). This model generates answers or content by referencing information from a knowledge base, exhibiting strong interpretability and customizability. It is applicable to various natural language processing tasks such as question-answering systems, document generation, and intelligent assistants. The advantages of the RAG model lie in its strong versatility, ability to achieve real-time knowledge updates, and provision of more efficient and accurate information services through end-to-end evaluation methods. In this application, the RAG model is used to process the metadata of each table in the full data instance to obtain its corresponding descriptive information, which is a key input in the subsequent data fusion and automatic metadata generation process.

[0062] Optionally, in this embodiment, the aforementioned descriptive information refers to the schema attribute value contained in the table-level metadata. It provides a business description of the table, i.e., the table's business meaning and purpose. For example, in a table of user order data on an e-commerce platform, the schema might be described as "e-commerce platform user order data," while in a table of user order information on a travel platform, the schema might be described as "travel platform user order information." This descriptive information is crucial for understanding and distinguishing different data tables. It helps the RAG model identify and understand the content and context of the data tables, thereby correctly grouping and merging similar or related data tables during the data fusion process.

[0063] Optionally, in this embodiment, the aforementioned grouping refers to logically classifying data tables with similar or identical business semantics based on the descriptive information of the table-level metadata in the full data instance. For example, all data tables related to orders can be grouped into one group, while data tables related to user information can be grouped into another. This grouping is based on business logic and the descriptive information of the data tables, and its purpose is to merge data tables with the same or similar business attributes into a new aggregate instance during the subsequent data fusion process, thereby achieving organic centralization and sharing of data.

[0064] Optionally, in this embodiment, the aforementioned sub-data structures refer to the basic elements constituting a data table, including table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. These sub-data structures collectively define the structure and content of a data table. For example, table-level metadata defines the basic information of the data table, field-level metadata defines the attributes of each field in the table, element-level metadata describes the data content in the table, and dictionary-level metadata defines the dictionary entries used in the table. In this application, for each set of data instances, the attribute parameters of these sub-data structures are merged to generate a target aggregate instance.

[0065] Optionally, in this embodiment, the aforementioned target aggregation instance refers to a new data instance obtained through the grouping and merging operations described above. It includes merged table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. This aggregation instance is the result of data fusion, logically or physically consolidating data from multiple sources, formats, and characteristics to form a unified data model. For example, if there are two different e-commerce platform order data tables, they can be merged into a single target aggregation instance using the method of this application. This instance contains order data from both platforms while maintaining data integrity, accuracy, and consistency. Such an aggregation instance provides enterprises with a comprehensive data sharing view, supporting more efficient data analysis and decision-making.

[0066] It should be noted that the process of retrieving descriptive information corresponding to each table-level metadata in the full data instance through the retrieval enhancement model can involve various different data sources and data types. For example, data sources may include relational databases, non-relational databases, data warehouses, or data lakes, while data types may cover structured data, semi-structured data such as JSON, and unstructured data such as text documents. This application does not limit this.

[0067] Furthermore, the full set of data instances is grouped based on descriptive information. In this step, the classification of descriptive information can be based on different business logics, data attributes, or data sources. For example, grouping can be based on industry sectors (such as finance, healthcare, and education), data sensitivity (such as public data, internal data, and confidential data), or data update frequency (such as real-time data, daily updated data, and historical archived data). This application does not limit this approach.

[0068] Finally, for each set of data instances, they are merged according to the attribute parameters of the corresponding sub-data structure. During this process, the attribute parameters of the sub-data structure can include, but are not limited to, data format, version, quality, and timeliness. For example, data instances with the same data format (such as CSV, XML) can be merged, or data instances from the same version control system can be merged, or data instances with the same data quality standards (such as accuracy, completeness) can be merged. Furthermore, they can be merged based on the timeliness of the data (such as real-time data, historical data). This application does not limit this. In this way, data instances can be flexibly aggregated and merged effectively according to different needs and scenarios to generate target aggregated instances that meet specific requirements.

[0069] This application employs a Retrieval-Augmented Generation (RAG) model to enhance data retrieval and generation capabilities. The model retrieves descriptive information corresponding to each table-level metadata in the full data instance. This step leverages the RAG model's retrieval capabilities to identify and extract key metadata descriptions, providing precise input for data fusion. Next, the full data instance is grouped based on these descriptive information, resulting in at least two groups of data instances corresponding to at least two sub-data structures. This process, based on the business semantics in the descriptive information, groups data tables with similar business attributes into the same group, achieving logical centralization and preprocessing of the data. Finally, for each group of data instances, they are merged according to the attribute parameters of the corresponding sub-data structure, such as the attributes of field-level metadata and dictionary-level metadata, to obtain the target aggregated instance. This step, through automated clustering and merging rules, achieves automatic metadata generation and physical data centralization. Through this series of processes, the goal of reducing manual intervention and improving data processing efficiency and accuracy is achieved, thereby realizing the technical effects of improving data fusion quality and automation levels.

[0070] As an optional approach, for each group of data instances, the target aggregate instance is obtained by merging them according to the attribute parameters of the corresponding sub-data structure. This includes: when the sub-data structure corresponds to the table-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is considered the first group of data instances: adding and filling the table-level data source attribute parameters of each data instance in the first group of data instances to obtain the target table-level data source attribute parameters; setting the table-level identifier attribute parameter of each data instance in the first group of data instances to be filled with business text in the first language to obtain the target table-level identifier attribute parameter corresponding to the target aggregate instance; setting the table-level description attribute parameter of each data instance in the first group of data instances to be filled with business text in the second language to obtain the target table-level description attribute parameter, wherein the first language business text and the second language business text use different languages; and generating the target aggregate instance based on the target table-level data source attribute parameters, the target table-level identifier attribute parameters, and the target table-level description attribute parameters.

[0071] Optionally, in this embodiment, the target table-level data source attribute parameter refers to the attribute parameter obtained after merging the source attributes of the table-level metadata in each group of data instances during the data fusion process. This parameter typically contains all data source information for the merged data instance. For example, if order tables from different databases are merged, the target table-level data source attribute parameter may contain the names or identifiers of all these databases, separated by commas, such as "Database A, Database B, Database C".

[0072] Optionally, in this embodiment, the target table-level identifier attribute parameter refers to the attribute parameter obtained by merging the identifier attributes of the table-level metadata in each data instance during the data fusion process. This parameter is typically used to provide a business identifier for one or more merged data tables. For example, if multiple order tables are merged, the target table-level identifier attribute parameter might be set to "order," which is an identifier filled with business text in a first language (such as English) that concisely describes the main business content of the merged data table.

[0073] Optionally, in this embodiment, the target table-level descriptive attribute parameter refers to the attribute parameter obtained by merging the descriptive attributes of the table-level metadata in each data instance during the data fusion process. This parameter provides a detailed business description of the merged data table. For example, if multiple order tables are merged, the target table-level descriptive attribute parameter may be set to "Contains order information from different platforms," ​​which is a description filled with business text in a second language (such as Chinese), detailing the data content and business background contained in the merged data table.

[0074] It should be noted that, given the correspondence between sub-data structures and table-level metadata, the operations performed on each set of data instances can be diversified according to different business needs and data characteristics, and this application does not impose any limitations on this. For example, the addition and population of table-level data source attribute parameters can include dimensions such as the data source name, version number, and data format, such as merging data source names from different versions of databases to form "database V1, database V2". Such operations can adapt to the merging requirements of different versions of data sources and can also be adjusted according to different data format requirements.

[0075] Furthermore, the settings for table-level identifier and description attribute parameters can be diversified according to different business scenarios and language requirements. For example, the first language business text can be English, used for identifier attribute parameters, such as unifying the identifier of multiple order tables to "Order," while the second language business text can be Chinese, used for description attribute parameters, such as setting the description of the order table to "Contains order information from e-commerce platforms and travel platforms." Such settings are not only suitable for data fusion in different language environments, but can also be customized according to different business domains and data content. The target aggregation instance generated based on the target table-level data source attribute parameters, target table-level identifier attribute parameters, and target table-level description attribute parameters can be generated in a diversified manner according to different data fusion rules and business logic. For example, aggregation can be performed based on dimensions such as data business importance, data update frequency, or data access permissions to generate aggregation instances that meet specific business needs. Such aggregation instances can be used in different data analysis, report generation, or decision support systems, improving the flexibility and practicality of data fusion.

[0076] For example, consider a data fusion scenario for an e-commerce platform that needs to integrate sales data from subsidiaries in different regions to facilitate global sales analysis. The specific implementation process is as follows:

[0077] S1, Design a unified meta-model: First, define a meta-model that includes table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. This meta-model will serve as the basic framework for data fusion.

[0078] S2, Extract full basic information from the data source: Use data source probing technologies, such as DataX or FlinkX, to extract full basic information from the databases of subsidiaries in various regions, including fine-grained basic information such as databases, tables, fields, records, dictionaries, etc.

[0079] S3, Construct a unified full data source instance: Store the extracted basic information of the data source according to the definition and rules of the meta-model to form a unified data source instance.

[0080] S4, Define metamodel instance clustering and merging rules: Define rules to logically or physically aggregate data from different sources, formats, and characteristics. For example, group all sales data tables according to business semantics.

[0081] S5, Store in RAG knowledge base: Store the defined clustering and merging rules in the knowledge base of the RAG model so that the RAG model can process the data according to these rules.

[0082] S6, Clustering and Merging Processing via RAG Model: The encapsulated full meta-model instances are clustered and merged using the RAG model. The RAG model performs clustering and merging operations on the data according to rules in the knowledge base, generating new aggregated instances.

[0083] S7, add up the table-level data source attribute parameters for the first set of data instances: for example, if the data comes from two subsidiaries, "North America" ​​and "Europe", the target table-level data source attribute parameters may be filled with "North America, Europe".

[0084] S8, Set table-level identifier attribute parameter: Set the table-level identifier attribute parameter of each data instance in the first group of data instances to be filled with business text in the first language (such as English), for example, "Sales Data".

[0085] S9, Set table-level description attribute parameters: Set the table-level description attribute parameters of each data instance in the first group of data instances to be filled with business text in a second language (such as Chinese), for example, "Contains sales data of North American and European subsidiaries".

[0086] S10, Generate target aggregation instance: Based on the target table-level data source attribute parameters, target table-level identifier attribute parameters, and target table-level description attribute parameters, generate a target aggregation instance containing sales data of all regional subsidiaries.

[0087] The above process effectively integrates sales data from subsidiaries scattered across different regions. This process not only improves data processing efficiency and reduces human error but also enhances data consistency and accuracy. Through automated metadata generation and data fusion, enterprises can quickly obtain a holistic sales analysis perspective, supporting more precise business decisions. Furthermore, by employing multilingual business text population, the target aggregation instances can better adapt to the language habits of different regions, improving data readability and usability. Ultimately, this approach enhances the automation and intelligence of data fusion, providing enterprises with a comprehensive, accurate, and consistent data view.

[0088] As an optional approach, for each group of data instances, the attribute parameters of the corresponding sub-data structures are merged to obtain the target aggregated instance. This includes: when the sub-data structures correspond to the field-level metadata, performing the following operations on each group of data instances, wherein each group of data instances for which the following operations are performed is considered the second group of data instances: semantically deduplicating the field-level description attribute parameters and field-level data item attribute parameters of each data instance in the second group of data instances, setting the deduplicated field-level description attribute parameters to be filled according to the second language business text to obtain the target field-level description attribute parameters; and merging the field-level labels of each data instance in the second group of data instances. The identification attribute parameter is set to be filled according to the first language business text to obtain the target field-level identification attribute parameter, where the first language business text and the second language business text use different languages; the field-level data type attribute parameter of each data instance in the second group of data instances is filled with the original value to obtain the target field-level data type attribute parameter; the field-level size attribute parameter of each data instance in the second group of data instances is filled with the maximum value to obtain the target field-level size attribute parameter; a target aggregation instance is generated based on the target field-level description attribute parameter, target field-level identification attribute parameter, target field-level data type attribute parameter, and target field-level size attribute parameter.

[0089] Optionally, in this embodiment, the aforementioned field-level metadata refers to the data structure that constitutes the detailed description information of each field in the data table. Field-level metadata includes attributes such as the field's name, type, size, and precision. These attributes define the field's data characteristics and storage requirements. For example, a field-level metadata entry might contain the field name "customer_id", field type "integer", size "10", precision "0", and field description "customer's unique identifier". Field-level metadata is an important component of the data table structure, providing essential information for data storage, retrieval, and management.

[0090] Optionally, in this embodiment, semantic deduplication refers to the process of identifying and merging field descriptions with the same or similar business meanings during data fusion. This process ensures the consistency and accuracy of field descriptions after data fusion. For example, when merging order information from different databases, there may be a situation where two fields describe customer names, but the field names may be different; one may be "customer_name" and the other may be "name". Semantic deduplication identifies these two fields as having the same meaning and retains only one in the target aggregation instance.

[0091] Optionally, in the embodiments of this application, the first language business text and the second language business text refer to different language texts used to populate field-level identifier attribute parameters and field-level descriptive attribute parameters during the data fusion process. These texts reflect the business meaning of the fields and use different languages. For example, the first language business text may be English, used to identify attribute parameters, such as identifying the field "order_date" as "Order Date"; while the second language business text may be Chinese, used to describe attribute parameters, such as describing the same field as "Order Date".

[0092] Optionally, in this embodiment, the field-level data type attribute parameter refers to the metadata defining the field's data type. This parameter specifies the data type that the field can store, such as integers, floating-point numbers, strings, etc. For example, a field-level data type attribute parameter can be "character", corresponding to a field that can store text data.

[0093] Optionally, in this embodiment, the field-level size attribute parameter refers to metadata that defines the maximum storage size of the field. This parameter specifies the maximum number of characters or bytes that the field can store. For example, a field-level size attribute parameter could be "255", indicating that the maximum character length that the field can store is 255 characters.

[0094] It should be noted that, when the sub-data structure corresponds to the field-level metadata, the operations on the second set of data instances can be diversified according to different data characteristics and business needs, and this application does not impose any limitations on this. For example, when performing semantic deduplication of field-level descriptive attribute parameters and field-level data item attribute parameters, consideration can be given from three dimensions: data content, data format, and data source. In terms of data content, it may be necessary to compare the semantic similarity of field descriptions; in terms of data format, it may be necessary to unify the field naming conventions of different data sources; and in terms of data source, it may be necessary to identify and merge identical fields from different systems or departments.

[0095] When setting field-level identifier attribute parameters, you can diversify the processing from three dimensions: language, region, and business domain. In terms of language, the first language business text can be English, while the second language business text can be Chinese, to adapt to the reading habits of users in different regions. In terms of region, it can be adjusted according to the language preferences of the subsidiary's location. In terms of business domain, the business text of the field identifier can be customized according to different business scenarios, such as finance, healthcare, or education.

[0096] When populating field-level data type attribute parameters, three dimensions should be considered: data storage requirements, query efficiency, and data consistency. Regarding storage requirements, the appropriate data type should be selected based on the actual data type. For query efficiency, choosing the right data type can optimize database query performance. Regarding data consistency, maintaining data type consistency helps reduce errors during data fusion.

[0097] When populating field-level size attribute parameters, various approaches can be taken from three dimensions: maximum field length, performance optimization, and data integrity. Regarding maximum length, choosing the maximum value ensures all data can be stored; for performance optimization, a reasonable size can improve the efficiency of database operations; and for data integrity, ensuring the field size can accommodate all possible data values ​​maintains data integrity.

[0098] Target aggregation instances, generated based on target field-level description, identifier, data type, and size attributes, can be evaluated from three dimensions: data availability, consistency, and business adaptability. In terms of availability, the aggregation instance provides a unified data view, facilitating user access and analysis. Regarding consistency, it ensures data accuracy and reliability. Finally, it allows for customization and optimization to meet diverse business needs. These diverse operations enable the generation of target aggregation instances that satisfy various business scenarios, improving the flexibility and effectiveness of data processing.

[0099] Through the embodiments of this application, based on the aforementioned target field-level description attribute parameters, target field-level identifier attribute parameters, target field-level data type attribute parameters, and target field-level size attribute parameters, a target aggregation instance that integrates information from multiple data sources can be generated. This instance provides a unified data view for data analysis and reporting, while also considering the needs of users in different languages, thus improving data usability and accessibility.

[0100] As an optional approach, for each group of data instances, the target aggregate instance is obtained by merging according to the attribute parameters of the corresponding sub-data structure. This includes: when the sub-data structure corresponds to the dictionary-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is considered as the third group of data instances: semantically deduplicating the dictionary-level value parameters of each data instance in the third group of data instances to obtain the target dictionary-level value attribute parameters; filling the dictionary-level description attribute parameters and dictionary-level data type attribute parameters of each data instance in the third group of data instances with their original values ​​to obtain the target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; and generating the target aggregate instance based on the target dictionary-level value attribute parameters, target dictionary-level description attribute parameters, and target dictionary-level data type attribute parameters.

[0101] Optionally, in this embodiment, the aforementioned dictionary-level metadata refers to metadata related to dictionary-type fields in a data table. Dictionary-level metadata contains a collection of dictionary entries, each corresponding to a specific value and description. For example, in an order management system, order status might be a dictionary field, and its dictionary-level metadata might include statuses such as "Pending Payment," "Paid," "Shipping," and "Completed," each corresponding to an integer value. Dictionary-level metadata is an important component of the data model, providing standardized classification and encoding for data, which helps with data consistency and comparative analysis.

[0102] Optionally, in this embodiment, the dictionary-level value parameter refers to the value of a dictionary item. This value is usually unique and is used to identify a specific dictionary item in the database. For example, in the dictionary field of the order status mentioned above, each status such as "pending payment" may correspond to an integer value, such as 1. This value is the dictionary-level value parameter. During the data fusion process, semantic deduplication of the dictionary-level value parameter means identifying and merging different values ​​with the same business meaning, ensuring the uniqueness and consistency of the dictionary items after data fusion.

[0103] Optionally, in this embodiment, the dictionary-level descriptive attribute parameter refers to the textual description of a dictionary item, providing a detailed explanation and business meaning of the dictionary item. For example, for the dictionary item "1" of the order status, its dictionary-level descriptive attribute parameter might be "pending payment." This description helps users understand the specific meaning of each dictionary value. During data fusion, maintaining the original value of the dictionary-level descriptive attribute parameter means preserving the specific description of the dictionary item in each data source to ensure that the business meaning of the data is not lost.

[0104] Optionally, in this embodiment, the dictionary-level data type attribute parameter refers to metadata that defines the data type of the dictionary item value. This parameter specifies the storage type of the dictionary item value, such as integer, string, etc. For example, if the dictionary-level value parameter of the order status is stored in integer form, then its dictionary-level data type attribute parameter is "integer". During the data fusion process, filling the dictionary-level data type attribute parameter with the original value means that the data type of each dictionary item value remains unchanged in the generated target aggregation instance to ensure data compatibility and consistency.

[0105] It should be noted that, given the correspondence between sub-data structures and dictionary-level metadata, operations on the third set of data instances can be diversified based on different business rules, data specifications, and system requirements; this application does not impose any limitations on this. For example, when performing semantic deduplication of dictionary-level value parameters, considerations can be made from three dimensions: business meaning, data format, and system compatibility. Regarding business meaning, it is necessary to identify the same or similar business concepts in different data instances, such as unifying different codes representing the "completed" status in different systems into a single standard value; regarding data format, it is necessary to convert codes of different formats into a unified format to facilitate data comparison and processing; regarding system compatibility, it is necessary to ensure that the deduplicated dictionary-level value parameters can be correctly identified and processed in various systems. When populating dictionary-level descriptive attribute parameters, diversified processing can be carried out from three dimensions: the level of detail in the description, language differences, and cultural differences. Regarding the level of detail, the level of detail in the description needs to be determined according to business requirements; for example, in some cases, more detailed dictionary item descriptions may be required. Regarding language differences, dictionary item descriptions need to be localized according to the language habits of different regions; regarding cultural differences, it is necessary to consider that different cultural backgrounds may lead to different understandings and expressions of the same dictionary item. When populating dictionary-level data type attribute parameters, three dimensions should be considered: data storage efficiency, processing performance, and data security. Regarding storage efficiency, an appropriate data type needs to be selected to optimize storage space usage. For processing performance, the processing efficiency of the data type needs to be considered; for example, integer types are typically processed faster than string types. Regarding data security, it is necessary to ensure that the choice of data type does not introduce security risks, such as avoiding storing sensitive information in plaintext.

[0106] Through the embodiments of this application, based on the target dictionary-level value attribute parameters, target dictionary-level descriptive attribute parameters, and target dictionary-level data type attribute parameters, a unified target aggregation instance can be generated that contains deduplicated dictionary entries while maintaining the original business description and data type. This instance provides a standardized dictionary reference for data analysis and reporting, while ensuring the accuracy and usability of the data model after data fusion.

[0107] The following specific examples will further explain this application:

[0108] In the process of implementing data sharing, since the data provided by different users may come from different sources, their data content, data format and data quality vary greatly. Sometimes, we may even encounter tricky problems such as data format not being able to be converted or information being lost after data conversion, which seriously hinders the flow and sharing of data in various departments and software systems.

[0109] A data warehouse is a structured data environment that serves as a data source for decision support systems and online analytical applications. Data warehousing studies and solves the problem of retrieving information from databases. Data warehouses are integrated; their data originates from dispersed operational data. The required data is extracted from the original data, processed, integrated, unified, and synthesized before being stored in the data warehouse.

[0110] Data fusion is the logical or physical aggregation of data from different sources, formats, and characteristics to provide enterprises with comprehensive data sharing. Designing a consistent data model to ensure data integrity, accuracy, and consistency is crucial for data fusion. Metadata serves as the foundation and link for building data models and enabling data sharing.

[0111] RAG (Related Acyclic Graph) is a model that combines retrieval and generation techniques. It generates answers or content by referencing information from a knowledge base, possessing strong interpretability and customizability, making it suitable for various natural language processing tasks such as question-answering systems, document generation, and intelligent assistants. The advantages of RAG models lie in their versatility, ability to achieve real-time knowledge updates, and the provision of more efficient and accurate information services through end-to-end evaluation methods.

[0112] This application proposes a data fusion method for self-generated metadata based on RAG. The method first extracts the basic information of the entire data source from different sources, formats, and characteristics through a DSE (Data Source Exploration) process, and encapsulates this information using a uniformly designed metamodel (including table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata), completing the basic preparation work for metadata. Second, it defines metamodel clustering and merging rules and stores these rules in the RAG knowledge base, completing the metadata rule definition work. Finally, it performs clustering and merging processing on the encapsulated full metamodel instances using the RAG model and outputs the results, completing the automatic generation of metadata.

[0113] This application proposes a metadata self-generation and data fusion method based on RAG. This method integrates multiple technologies such as DSE and RAG, transforming the traditional process of manually generating metadata into an automatic generation method through artificial intelligence. The principle behind this application is explained below:

[0114] 1. Design a unified meta-model; 2. Extract all basic information from the data source; 3. Construct a unified instance of the full data source based on the meta-model; 4. Define clustering and merging rules for meta-model instances and store them in the RAG knowledge base; 5. Cluster and merge all meta-model instances using the RAG model and generate aggregated instances; 6. Automatically generate metadata based on the aggregated instances.

[0115] Figure 3 This is a flowchart illustrating a method for generating metadata according to an embodiment of this application, such as... Figure 3 As shown, the specific implementation process of this application is described in detail below:

[0116] S302: Design a unified metamodel. The metamodel is defined and stored using JSON format, including table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata information, as shown below:

[0117]

[0118]

[0119] S304: Extract full basic information from the data source. Utilize data source exploration technologies such as DataX or FlinkX to obtain full basic information about the data source, including fine-grained basic information such as databases, tables, fields, records, and dictionaries. For example, database name, table name, table description, field name, field length, field precision, field description, data item content, data item type, data item description, dictionary content, dictionary type, and dictionary description.

[0120] S306: Construct a unified full data source instance based on the metamodel. Store the extracted basic information from the data source according to the definition and rules of the metamodel. An example is shown below:

[0121]

[0122]

[0123]

[0124] S308: Define the metamodel instance clustering and merging rules and store them in the RAG knowledge base. The purpose of defining metamodel clustering and merging rules is to organically centralize data from different sources, formats, and characteristics, either logically or physically. In other words, it's about merging multiple tables with the same business attributes into one table, and the metadata of this table is what we need to create. Rule definition can be understood as a procedural prompt word project. After definition, the rules are stored in the RAG knowledge base. A specific example is as follows:

[0125] (1) Group business semantics based on the schema value of table_infos, the table-level metadata of the full meta-model instance.

[0126] (2) Generate new metamodel table-level metadata table_infos according to the group. The source value is filled with the sum of the source values ​​of all instances in the group and separated by ","; the name value is filled with business English words; and the schema value is filled with business Chinese words.

[0127] (3) Generate new metamodel field-level metadata table_fields according to the group. Semantic deduplication is performed based on the schema value of each field in the metamodel instance table_fields of the group and the data value of each element in the corresponding table_elemenst. The name value is filled with business English words; the schema value is filled with business Chinese words; the type is filled with the original value; and the size value is filled with the maximum value.

[0128] (4) Generate new metamodel dictionary-level metadata tables_dicts according to the group, where semantic deduplication is performed based on the values ​​of each dictionary in the metamodel instance tables_dicts of the group, and the type and schema are filled with the original values.

[0129] S310: Cluster and merge the full meta-model instances using the RAG model to generate new aggregated instances. RAG is a model that combines retrieval and generation technologies. It generates answers or content by referencing information from a knowledge base, possessing strong interpretability and customizability. The full meta-model instances are loaded into the RAG model, which then clusters and merges them according to the rules designed in the knowledge base, generating new aggregated instances. An example of the aggregated meta-models 01 and 02 in step three is shown below:

[0130]

[0131]

[0132] S312: Automatically generate metadata based on aggregated instances. The specific steps are as follows:

[0133] (1) Automatically create tables by assembling SQL statements based on the contents of table_infos;

[0134] (2) Automatically create fields by assembling SQL statements based on the contents of table_fields;

[0135] (3) Automatically create dictionaries by assembling SQL statements based on the contents of table_dicts.

[0136] This application proposes a data fusion method for self-generated metadata and a data fusion design concept based on a meta-model, which realizes the automatic generation of metadata, can significantly improve the efficiency and quality of data fusion, and effectively solve the various shortcomings of manual methods.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0138] This embodiment also provides a metadata generation apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] Figure 4 This is a structural block diagram of a metadata generation apparatus according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0140] The acquisition module 402 is used to acquire a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures and the full basic information includes at least two granularities, and each sub-data structure has a corresponding granularity.

[0141] The encapsulation module 404 is used to encapsulate the full amount of basic information using the meta-model to obtain a full data instance, wherein the full data instance represents the full amount of basic information stored according to the rules indicated by the meta-model.

[0142] Clustering module 406 is used to cluster and merge the above full data instances according to a preset clustering and merging method using a retrieval enhancement generation model to generate target metadata. The retrieval enhancement generation model is used to cluster the above full data instances according to the preset clustering and merging method, which includes grouping according to the description information of the above full data instances.

[0143] As an optional solution, the above-mentioned apparatus is used to cluster and merge the above-mentioned full data instances using a retrieval-enhanced generation model based on a preset clustering and merging method to generate target metadata: clustering and merging the above-mentioned full data instances using the above-mentioned retrieval-enhanced generation model based on the preset clustering and merging method to determine target aggregate instances, wherein the above-mentioned target aggregate instances include instances corresponding to table-level metadata, and the above-mentioned at least two sub-data structures include the above-mentioned table-level metadata; generating target metadata based on the above-mentioned target aggregate instances, wherein the above-mentioned target metadata is set to be assembled according to the target machine language.

[0144] As an optional solution, the above-mentioned device is used to cluster and merge the above-mentioned full data instances based on a preset clustering and merging method using the above-mentioned retrieval enhancement generation model to determine the target cluster instance: setting the above-mentioned preset clustering and merging method; storing the above-mentioned preset clustering and merging method in the knowledge base corresponding to the above-mentioned retrieval enhancement generation model; and automatically clustering and merging the above-mentioned full data instances by referencing the above-mentioned knowledge base through the above-mentioned retrieval enhancement generation model to determine the above-mentioned target cluster instance.

[0145] As an optional solution, the above-mentioned apparatus is used to cluster and merge the above-mentioned full data instances using the above-mentioned retrieval enhancement generation model based on a preset clustering and merging method to determine the target aggregate instance: obtaining the above-mentioned descriptive information corresponding to each table-level metadata in the above-mentioned full data instances through the above-mentioned retrieval enhancement model; grouping the above-mentioned full data instances according to the above-mentioned descriptive information to obtain at least two groups of data instances corresponding to the above-mentioned at least two sub-data structures respectively; and merging each group of the above-mentioned data instances according to the attribute parameters of the corresponding sub-data structure to obtain the above-mentioned target aggregate instance.

[0146] As an optional solution, the above-mentioned apparatus is used to merge the target aggregation instance for each group of the above-mentioned data instances according to the attribute parameters of the corresponding sub-data structure in the following manner: when the above-mentioned sub-data structure corresponds to the above-mentioned table-level metadata, the following operations are performed on each group of the above-mentioned data instances, wherein each group of the above-mentioned data instances performing the following operations is regarded as the first group of data instances: the table-level data source attribute parameters of each data instance in the first group of data instances are added and filled to obtain the target table-level data source attribute parameters; the table-level identifier attribute parameters of each data instance in the first group of the above-mentioned data instances are set to be filled according to the first language business text to obtain the target table-level identifier attribute parameters corresponding to the target aggregation instance; the table-level description attribute parameters of each data instance in the first group of the above-mentioned data instances are set to be filled according to the second language business text to obtain the target table-level description attribute parameters, wherein the first language business text and the second language business text use different languages; the target aggregation instance is generated based on the target table-level data source attribute parameters, the target table-level identifier attribute parameters, and the target table-level description attribute parameters.

[0147] As an optional solution, the above-mentioned apparatus is used to merge the above-mentioned data instances for each group of the above-mentioned data instances according to the attribute parameters of the corresponding sub-data structures in the following manner to obtain the above-mentioned target aggregated instance: when the above-mentioned sub-data structures correspond to field-level metadata, the following operations are performed on each group of the above-mentioned data instances, wherein each group of the above-mentioned data instances for which the following operations are performed is regarded as the second group of data instances: semantic deduplication is performed on the field-level description attribute parameters and field-level data item attribute parameters of each data instance in the above-mentioned second group of data instances, and the deduplicated field-level description attribute parameters are set to be filled according to the second language business text to obtain the target field-level description attribute parameters; the fields of each data instance in the above-mentioned second group of the above-mentioned data instances are... The level identifier attribute parameter is set to be filled according to the first language business text to obtain the target field-level identifier attribute parameter, wherein the first language business text and the second language business text use different languages; the field-level data type attribute parameter of each data instance in the second group of data instances is filled with the original value to obtain the target field-level data type attribute parameter; the field-level size attribute parameter of each data instance in the second group of data instances is filled with the maximum value to obtain the target field-level size attribute parameter; the target aggregation instance is generated based on the target field-level description attribute parameter, the target field-level identifier attribute parameter, the target field-level data type attribute parameter, and the target field-level size attribute parameter.

[0148] As an optional solution, the above-mentioned apparatus is used to merge the above-mentioned target aggregate instance for each group of the above-mentioned data instances according to the attribute parameters of the corresponding sub-data structure in the following manner: when the above-mentioned sub-data structure corresponds to dictionary-level metadata, the following operations are performed on each group of the above-mentioned data instances, wherein each group of the above-mentioned data instances for which the following operations are performed is regarded as the third group of data instances: semantic deduplication is performed on the dictionary-level value parameters of each data instance in the third group of data instances to obtain target dictionary-level value attribute parameters; the dictionary-level description attribute parameters and dictionary-level data type attribute parameters of each data instance in the third group of the above-mentioned data instances are filled with their original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; the above-mentioned target aggregate instance is generated based on the above-mentioned target dictionary-level value attribute parameters, the above-mentioned target dictionary-level description attribute parameters, and the above-mentioned target dictionary-level data type attribute parameters.

[0149] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0150] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0151] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0152] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0153] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0154] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0155] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0156] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A metadata generation method, characterized in that, comprising: obtaining a pre-created meta model and full basic information extracted from at least one data source, wherein the meta model includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure corresponds to a granularity; encapsulating the full basic information using the meta model to obtain full data instances, wherein the full data instances represent the full basic information stored according to the rules indicated by the meta model; based on a preset clustering and merging manner, clustering and merging the full data instances through a retrieval enhancement generation model to generate target metadata, including: obtaining description information corresponding to each table-level metadata in the full data instances through the retrieval enhancement model; grouping the full data instances according to the description information to obtain at least two groups of data instances corresponding to the at least two sub-data structures; for each group of data instances, merging according to the attribute parameters of the corresponding sub-data structure to obtain target aggregation instances, wherein the target aggregation instances include instances corresponding to table-level metadata, and the at least two sub-data structures include the table-level metadata; generating the target metadata based on the target aggregation instances, wherein the target metadata is set to be assembled according to a target machine language, the retrieval enhancement generation model is used to cluster the full data instances according to the preset clustering and merging manner, and the preset clustering and merging manner includes grouping according to the description information of the full data instances.

2. The method of claim 1, characterized in that, the clustering and merging of the full data instances through the retrieval enhancement generation model based on the preset clustering and merging manner to determine the target aggregation instances, comprising: setting the preset clustering and merging manner; storing the preset clustering and merging manner in a knowledge base corresponding to the retrieval enhancement generation model; automatically clustering and merging the full data instances through the retrieval enhancement generation model to determine the target aggregation instances by referencing the knowledge base.

3. The method of claim 1, characterized in that, the merging of each group of data instances according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregation instances, comprising: in the case where the sub-data structure corresponds to the table-level metadata, performing the following operations on each group of data instances, wherein each group of data instances performing the following operations is regarded as a first group of data instances: adding and filling the table-level data source attribute parameters of each data instance in the first group of data instances to obtain target table-level data source attribute parameters; set a table-level identification attribute parameter of each data instance in the first group of data instances to be filled according to a first language service text, to obtain a target table-level identification attribute parameter of the target aggregated instance, and set a table-level description attribute parameter of each data instance in the first group of data instances to be filled according to a second language service text, to obtain a target table-level description attribute parameter, wherein the first language service text and the second language service text use different languages; generate the target aggregated instance based on the target table-level data source attribute parameter, the target table-level identification attribute parameter, and the target table-level description attribute parameter.

4. The method of claim 1, wherein the merging, for each group of data instances, according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance comprises: in a case where the sub-data structure corresponds to field-level metadata, performing the following operations on each group of data instances, wherein each time the following operations are performed on a group of data instances, the group of data instances is regarded as a second group of data instances: performing semantic deduplication on a field-level description attribute parameter and a field-level data item attribute parameter of each data instance in the second group of data instances, setting the deduplicated field-level description attribute parameter to be filled according to a second language service text to obtain a target field-level description attribute parameter; setting a field-level identification attribute parameter of each data instance in the second group of data instances to be filled according to a first language service text to obtain a target field-level identification attribute parameter, wherein the first language service text and the second language service text use different languages; setting a field-level data type attribute parameter of each data instance in the second group of data instances to be filled according to an original value to obtain a target field-level data type attribute parameter; setting a field-level size attribute parameter of each data instance in the second group of data instances to be filled according to a maximum value to obtain a target field-level size attribute parameter; generating the target aggregated instance based on the target field-level description attribute parameter, the target field-level identification attribute parameter, the target field-level data type attribute parameter, and the target field-level size attribute parameter.

5. The method of claim 1, wherein the merging, for each group of data instances, according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregated instance comprises: in a case where the sub-data structure corresponds to dictionary-level metadata, performing the following operations on each group of data instances, wherein each time the following operations are performed on a group of data instances, the group of data instances is regarded as a third group of data instances: performing semantic deduplication on a dictionary-level value parameter of each data instance in the third group of data instances to obtain a target dictionary-level value attribute parameter; setting a dictionary-level description attribute parameter and a dictionary-level data type attribute parameter of each data instance in the third group of data instances to be filled according to original values to obtain a target dictionary-level description attribute parameter and a target dictionary-level data type attribute parameter; ​ ​ generate the target aggregated instance based on the target dictionary level value attribute parameter, the target dictionary level description attribute parameter, and the target dictionary level data type attribute parameter. 6.A metadata generation apparatus, comprising: obtaining a pre-created meta model and full volume basic information extracted from at least one data source, wherein the meta model comprises at least two sub data structures, the full volume basic information comprises at least two granularities, and each sub data structure corresponds to one granularity; encapsulating the full volume basic information using the meta model to obtain a full volume data instance, wherein the full volume data instance represents the full volume basic information stored according to a rule indicated by the meta model; clustering and merging the full volume data instance based on a preset clustering and merging manner to generate target metadata, including: obtaining description information corresponding to each table level metadata in the full volume data instance through a retrieval enhancement model; grouping the full volume data instance according to the description information to obtain at least two groups of data instances corresponding to the at least two sub data structures respectively; merging each group of data instances according to attribute parameters of the corresponding sub data structure to obtain a target aggregated instance, wherein the target aggregated instance comprises an instance corresponding to table level metadata, and the at least two sub data structures comprise the table level metadata; and generating the target metadata based on the target aggregated instance, wherein the target metadata is set to be assembled according to a target machine language, the retrieval enhancement model is used to cluster and merge the full volume data instance according to the preset clustering and merging manner, and the preset clustering and merging manner comprises grouping according to the description information of the full volume data instance. 7.A computer readable storage medium, comprising: the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method in any one of claims 1 to 5. 8.An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: the processor executes the computer program to implement the steps of the method in any one of claims 1 to 5. ​

Images