Metadata generation method and device

Abstract

The embodiment of the invention provides a metadata generation method and device.The method comprises the steps that a pre-created meta-model and total basic information extracted from at least one data source are obtained, the meta-model comprises at least two sub-data structures, and the total basic information comprises at least two granularities; each seed data structure has one granularity corresponding to the seed data structure; packaging the total basic information by using the meta-model to obtain a total data instance; based on a preset clustering combination mode, clustering combination is carried out on the full-amount data instances through a retrieval enhancement generation model, target metadata are generated, the retrieval enhancement generation model is used for clustering the full-amount data instances according to the preset clustering combination mode, and the target metadata are generated; the preset clustering combination mode comprises grouping according to the description information of the total data instance, the technical effect of automatically generating the metadata is achieved, and the problems that in the related technology, the metadata generation efficiency is low, and the data fusion quality is difficult to guarantee are solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the computer field, and in particular, to a method and device for generating metadata. Background Art

[0002] Data fusion is to logically or physically integrate data from different sources, formats, and characteristics, so as to provide comprehensive data sharing for enterprises. Designing a consistent data model to ensure the integrity, accuracy, and consistency of data is the key to data fusion, and metadata is the basis and link for building data models and sharing data. Currently, metadata is manually generated.

[0003] However, manually generating unified metadata has disadvantages such as large workload, difficulty in normalization, high error rate, and poor maintainability, which leads to technical problems such as low metadata generation efficiency and difficulty in ensuring data fusion quality. Summary of the invention

[0004] The embodiments of the present application provide a method and device for generating metadata, so as to at least solve the problem in the related art that the metadata generation efficiency is low and the quality of data fusion is difficult to ensure.

[0005] According to one embodiment of the present application, a method for generating metadata is provided, comprising: obtaining a pre-created metamodel and full basic information extracted from at least one data source, wherein the metamodel includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding to it; using the metamodel to encapsulate the full basic information to obtain a full data instance, wherein the full data instance represents the full basic information stored according to the rules indicated by the metamodel; clustering and merging the full data instance through a retrieval enhanced generation model based on a preset clustering and merging method to generate target metadata, wherein the retrieval enhanced generation model is used to cluster the full data instance according to the preset clustering and merging method, and the preset clustering and merging method includes grouping according to the description information of the full data instance.

[0006] In an exemplary embodiment, the clustering and merging of the full amount of data instances through the retrieval enhanced generation model based on a preset clustering and merging method to generate target metadata includes: clustering and merging the full amount of data instances through the retrieval enhanced generation model based on a preset clustering and merging method to determine a target aggregate instance, wherein the target aggregate instance includes an instance corresponding to table-level metadata, and the at least two sub-data structures include the table-level metadata; generating target metadata based on the target aggregate instance, wherein the target metadata is set to be assembled according to a target machine language.

[0007] In an exemplary embodiment, clustering and merging the full amount of data instances through the retrieval enhancement generation model based on a preset clustering and merging method to determine the target aggregate instance includes: setting the preset clustering and merging method; storing the preset clustering and merging method in a knowledge base corresponding to the retrieval enhancement generation model; and automatically clustering and merging the full amount of data instances by referencing the knowledge base through the retrieval enhancement generation model to determine the target aggregate instance.

[0008] In an exemplary embodiment, the clustering and merging of the full data instances through the retrieval enhancement generation model based on a preset clustering and merging method to determine the target aggregate instance includes: obtaining the description information corresponding to each table-level metadata in the full data instance through the retrieval enhancement model; grouping the full data instances according to the description information to obtain at least two groups of data instances corresponding to the at least two sub-data structures respectively; for each group of the data instances, merging according to the attribute parameters of the corresponding sub-data structure to obtain the target aggregate instance.

[0009] In an exemplary embodiment, for each group of the data instances, merging them according to the attribute parameters of the corresponding sub-data structures to obtain the target aggregate instance includes: when the sub-data structure corresponds to the table-level metadata, performing the following operations on each group of the data instances, wherein a group of the data instances that performs the following operations each time is regarded as a first group of data instances: adding and filling the table-level data source attribute parameters of each data instance in the first group of data instances to obtain target table-level data source attribute parameters; setting the table-level identification attribute parameters of each data instance in the first group of the data instances to be filled according to the first language business text to obtain the target table-level identification attribute parameters corresponding to the target aggregate instance; setting the table-level description attribute parameters of each data instance in the first group of the data instances to be filled according to the second language business text to obtain the target table-level description attribute parameters, wherein the first language business text and the second language business text use different languages; generating the target aggregate instance based on the target table-level data source attribute parameters, the target table-level identification attribute parameters and the target table-level description attribute parameters.

[0010] In an exemplary embodiment, for each group of the data instances, the attribute parameters of the corresponding sub-data structures are merged to obtain the target aggregate instance, including: when the sub-data structure corresponds to the field-level metadata, the following operations are performed on each group of the data instances, wherein a group of the data instances that perform the following operations each time is regarded as a second group of data instances: semantically deduplicating the field-level description attribute parameters and field-level data item attribute parameters of each data instance in the second group of data instances, and setting the deduplicated field-level description attribute parameters to be filled in according to the second language business text to obtain the target field-level description attribute parameters; setting the field-level label of each data instance in the second group of data instances to be filled in according to the second language business text to obtain the target field-level description attribute parameters; and setting the field-level label of each data instance in the second group of data instances to be filled in according to the second language business text to obtain the target field-level description attribute parameters. The identification attribute parameter is set to be filled according to the first language business text to obtain the target field-level identification attribute parameter, wherein the first language business text and the second language business text use different languages; the field-level data type attribute parameter of each data instance in the second group of data instances is filled according to the original value to obtain the target field-level data type attribute parameter; the field-level size attribute parameter of each data instance in the second group of data instances is filled according to the maximum value to obtain the target field-level size attribute parameter; the target aggregate instance is generated based on the target field-level description attribute parameter, the target field-level identification attribute parameter, the target field-level data type attribute parameter and the target field-level size attribute parameter.

[0011] In an exemplary embodiment, for each group of the data instances, the attribute parameters of the corresponding sub-data structures are merged to obtain the target aggregate instance, including: when the sub-data structure corresponds to the dictionary-level metadata, the following operations are performed on each group of the data instances, wherein a group of the data instances that performs the following operations each time is regarded as a third group of data instances: semantically deduplicating the dictionary-level value parameters of each data instance in the third group of data instances to obtain target dictionary-level value attribute parameters; filling the dictionary-level description attribute parameters and the dictionary-level data type attribute parameters of each data instance in the third group of data instances with the original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; generating the target aggregate instance based on the target dictionary-level value attribute parameters, the target dictionary-level description attribute parameters and the target dictionary-level data type attribute parameters.

[0012] According to another embodiment of the present application, a metadata generation device is provided, including: an acquisition module, used to acquire a pre-created metamodel and full basic information extracted from at least one data source, wherein the metamodel includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding to it; an encapsulation module, used to encapsulate the full basic information using the metamodel to obtain a full data instance, wherein the full data instance represents the full basic information stored according to the rules indicated by the metamodel; a clustering module, used to cluster and merge the full data instance through a retrieval enhanced generation model based on a preset clustering merging method to generate target metadata, wherein the retrieval enhanced generation model is used to cluster the full data instance according to the preset clustering merging method, and the preset clustering merging method includes grouping according to the description information of the full data instance.

[0013] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0014] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0015] Through this application, first, the design of the metamodel includes at least two sub-data structures, such as table-level metadata and field-level metadata, and the full amount of basic information includes at least two granularities, such as the level of the library and the table. Such a design ensures that each sub-data structure can correspond to the basic information of a granularity, thereby achieving accurate data encapsulation. Then, the full amount of basic information is encapsulated by the metamodel to generate a full amount of data instance. In this step, the full amount of basic information is organized and stored according to the rules of the metamodel, forming a structured data instance, which provides a standardized basis for subsequent data processing and analysis. Finally, based on the preset clustering and merging method, the retrieval enhanced generation model (RAG model) is used to cluster and merge the full amount of data instances. In this process, the RAG model groups the description information of the full amount of data instances, realizes the logical concentration and integration of the data, and finally automatically generates the target metadata, achieving the purpose of unified management and fusion of data from different sources and formats, thereby achieving the technical effect of automatically generating metadata, and solving the problem of low metadata generation efficiency and difficult to ensure data fusion quality in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a hardware structure block diagram of a server device of a metadata generation method according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of a method for generating metadata according to an embodiment of the present application;

[0018] Figure 3 is a flowchart of a method for generating metadata according to an embodiment of the present application;

[0019] Figure 4 It is a structural block diagram of a metadata generation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a server device of a metadata generation method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating metadata in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0025] In this embodiment, a method for generating metadata is provided. Figure 2 is a flowchart of a method for generating metadata according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0026] Step S202, obtaining a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding thereto;

[0027] Step S204, using the meta-model to encapsulate the full amount of basic information to obtain a full amount of data instance, wherein the full amount of data instance represents the full amount of basic information stored according to the rules indicated by the meta-model;

[0028] Step S206, clustering and merging all data instances through the retrieval enhanced generation model based on a preset clustering and merging method to generate target metadata, wherein the retrieval enhanced generation model is used to cluster the full data instances according to the preset clustering and merging method, and the preset clustering and merging method includes grouping according to the description information of the full data instances.

[0029] Through this application, first, the design of the metamodel includes at least two sub-data structures, such as table-level metadata and field-level metadata, and the full amount of basic information includes at least two granularities, such as the level of the library and the table. Such a design ensures that each sub-data structure can correspond to the basic information of a granularity, thereby achieving accurate data encapsulation. Then, the full amount of basic information is encapsulated by the metamodel to generate a full amount of data instance. In this step, the full amount of basic information is organized and stored according to the rules of the metamodel, forming a structured data instance, which provides a standardized basis for subsequent data processing and analysis. Finally, based on the preset clustering and merging method, the retrieval enhanced generation model (RAG model) is used to cluster and merge the full amount of data instances. In this process, the RAG model groups the description information of the full amount of data instances, realizes the logical concentration and integration of the data, and finally automatically generates the target metadata, achieving the purpose of unified management and fusion of data from different sources and formats, thereby achieving the technical effect of automatically generating metadata, and solving the problem of low metadata generation efficiency and difficult to ensure data fusion quality in related technologies.

[0030] The execution subject of the above steps may be a server, a terminal, etc., but is not limited thereto.

[0031] The execution order of step S202 and step S204 is interchangeable, that is, step S204 may be executed first, and then step S202.

[0032] Optionally, in an embodiment of the present application, the above step S202 refers to the process of obtaining a pre-created metamodel and the full amount of basic information extracted from at least one data source. The metamodel includes at least two sub-data structures, which refer to data descriptions of different levels and types that constitute the metamodel, such as table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. These sub-data structures correspond to different data granularities, for example, table-level metadata corresponds to tables in a database, and field-level metadata corresponds to fields in a table. The full amount of basic information includes at least two granularities, which refer to the level of detail of the information extracted from the data source, such as database name, table name, field name, etc., and their attributes such as field length, field precision, etc. Each sub-data structure has a granularity corresponding to it, ensuring that the metamodel can comprehensively describe and encapsulate the basic information of the data source.

[0033] Optionally, in an embodiment of the present application, the above step S204 refers to the process of encapsulating the full amount of basic information using a metamodel to obtain a full amount of data instance. The full amount of data instance represents the full amount of basic information stored according to the rules indicated by the metamodel. This process involves organizing and storing the basic information extracted from the data source according to the structure and requirements of the metamodel to form a data instance in a unified format. These data instances can reflect the full picture of the data source, including table structure, field attributes, data content, etc., which are the basis for subsequent data fusion and automatic generation of metadata.

[0034] Optionally, in an embodiment of the present application, the above step S206 refers to the process of clustering and merging all data instances through a retrieval enhanced generation model based on a preset clustering and merging method to generate target metadata. The retrieval enhanced generation model is used to cluster all data instances according to the preset clustering and merging method. This model combines retrieval and generation techniques and can perform logical or physical centralized processing of data instances according to the rules in the knowledge base. The preset clustering and merging method includes grouping according to the descriptive information of the full data instance, which means that data instances with the same or similar characteristics are grouped together according to the descriptive information such as the business semantics and attribute characteristics of the data instance for further merging processing to generate target metadata, which will be used to guide subsequent data fusion and application development.

[0035] It should be noted that in the process of obtaining a pre-created metamodel and the full amount of basic information extracted from at least one data source, the sub-data structure of the metamodel and the granularity of the full amount of basic information can have a variety of different combinations and forms. For example, the sub-data structure of the metamodel may include table-level metadata and field-level metadata, and the granularity of the full amount of basic information may be the library name and table name at the database level, or the field name and field type at the field level. Each sub-data structure corresponds to a granularity, such as table-level metadata corresponding to database and table information, and field-level metadata corresponding to detailed information of the field. In addition, the metamodel may also include element-level metadata and dictionary-level metadata, corresponding to the granularity of data item content and dictionary content, respectively. This application does not limit this.

[0036] In the process of using the meta-model to encapsulate the full basic information and obtain the full data instance, the storage rules and format of the full data instance may vary according to different needs and scenarios. For example, the full data instance can be stored in JSON format, which contains detailed information on table-level, field-level, element-level and dictionary-level metadata. This information may include the name of the data source, the description of the table, the type and size of the field, the content and type of the data item, the content and type of the dictionary, etc. The full data instance may also include additional information such as version information, creation time and modification time of the data to meet the needs of data management and tracking. This application is not limited to this.

[0037] The full data instance is clustered and merged based on a preset clustering and merging method through a retrieval enhancement generation model. In the process of generating the target metadata, the preset clustering and merging method can be customized according to different business logics and data characteristics. For example, the data can be grouped according to the business field of the data source, such as merging all finance-related tables into one metadata group, or grouped according to the frequency of data updates, such as processing real-time updated data and non-real-time updated data separately. In addition, clustering and merging can also be based on data access patterns or user query habits to optimize data retrieval and access efficiency. The result of clustering and merging can be a new metadata model, or it can be an update and extension of an existing metadata model. This application is not limited to this.

[0038] For example, take the e-commerce platform data fusion scenario as an example:

[0039] S 1, first you need to obtain the pre-created metamodel and the full amount of basic information extracted from at least one data source. In this scenario, the metamodel includes at least two sub-data structures, such as table-level metadata and field-level metadata. The full amount of basic information includes at least two granularities, such as field granularity and record granularity. Table-level metadata corresponds to the table information in the database, and field-level metadata corresponds to the field information in the table. The field granularity provides information such as field name and field type, and the record granularity provides specific data record content.

[0040] S2, use the metamodel to encapsulate the full basic information to obtain a full data instance. In this process, the full basic information is organized into a unified data instance according to the rules of the metamodel. For example, the metamodel defines a template in JSON format, which contains the structure of table-level metadata and field-level metadata. The data in the full basic information is filled into this template to form a full data instance. These instances include user order data, product information data, etc. of the e-commerce platform. Each data instance follows the structure of the metamodel to ensure the consistency and comprehensibility of the data.

[0041] S3, clustering and merging the full data instance through the retrieval enhanced generation model based on the preset clustering and merging method to generate target metadata. In this step, the preset clustering and merging method may include grouping according to the business description information of the full data instance. For example, the order data in different tables can be clustered and merged according to the order status (such as "to be paid", "paid") to generate a new metadata model, which contains a unified view of all related orders. The retrieval enhanced generation model plays a key role in this process. It not only performs clustering according to preset rules, but also can understand and process natural language descriptions, making clustering and merging more intelligent and accurate.

[0042] Through the above process, the automatic generation of metadata is realized, which significantly improves the efficiency and quality of data fusion. In the e-commerce platform scenario, this method can quickly integrate user order information from different data sources, reduce the complexity and error rate of manual operations, and improve the speed and accuracy of data processing. In addition, due to the automatic generation of metadata, the consistency and maintainability of data are guaranteed, providing a solid foundation for subsequent data analysis and business decision-making. This method also has good scalability and can adapt to different business needs and data structure changes, providing a flexible and efficient solution for data fusion.

[0043] As an optional solution, cluster and merge all data instances through the retrieval enhancement generation model based on the preset clustering and merging method to generate target metadata, including:

[0044] Based on a preset clustering and merging method, clustering and merging the full amount of data instances through the retrieval enhancement generation model to determine a target aggregate instance, wherein the target aggregate instance includes an instance corresponding to the table-level metadata, and at least two sub-data structures include the table-level metadata;

[0045] Generate target metadata based on the target aggregate instance, wherein the target metadata is configured to be assembled according to a target machine language.

[0046] Optionally, in an embodiment of the present application, the above-mentioned preset clustering and merging method refers to a method of logically or physically centralizing all data instances according to specific rules or standards. This method includes but is not limited to grouping according to descriptive information such as business semantics and attribute characteristics of data instances. For example, in the data fusion scenario of an e-commerce platform, the preset clustering and merging method may aggregate order data from different data sources based on the business description information of the order, such as "order status" or "product category". Such clustering and merging helps to form a more complete and consistent data view, which is convenient for subsequent data analysis and decision support.

[0047] Optionally, in an embodiment of the present application, the above-mentioned target aggregate instance refers to an aggregate data instance obtained after processing by the retrieval enhancement generation model. These instances include instances corresponding to table-level metadata, and at least two sub-data structures include table-level metadata. The target aggregate instance may include but is not limited to table-level metadata, field-level metadata, and dictionary-level metadata. For example, in the data fusion of a travel platform, the target aggregate instance may include travel order information from different data sources, which is aggregated into a unified view, where the table-level metadata may contain basic information about the order, the field-level metadata contains specific field information of the order, such as order number, customer name, etc., and the dictionary-level metadata may contain enumeration values ​​of order status, such as "pending payment", "paid", etc.

[0048] Optionally, in an embodiment of the present application, the above-mentioned target metadata refers to metadata generated based on the target aggregate instance, and these metadata are set to be assembled according to the target machine language. Target metadata is data used to describe data, and they provide a framework and rules for the management and use of data. For example, when building a data warehouse, the target metadata may include the definition of the table, the data type and length of the field, index information, etc. These metadata can be used to generate SQL statements to create tables and fields in the database, or to guide the flow of data during data migration and conversion. The target machine language can be SQL, NoSQL or any other programming language for data manipulation and storage. In this way, the target metadata not only provides a structured description of the data, but also ensures that the data can be effectively migrated and used between different systems and platforms.

[0049] It should be noted that in the process of clustering and merging the full amount of data instances through the retrieval enhancement generation model based on the preset clustering and merging method, the preset clustering and merging method can be designed in a diversified way according to different business needs and data characteristics. For example, the clustering and merging method can be based on dimensions such as the source of the data, the business attributes of the data, or the timestamp of the data. In the e-commerce platform, order data can be clustered and merged according to dimensions such as product category, order status, or customer region to form a target aggregation instance. In the financial field, transaction records can be clustered and merged according to dimensions such as transaction type, amount range, or account status. In the field of medical health, medical records can be clustered and merged according to dimensions such as patient diagnosis, treatment stage, or drug category. This application is not limited to this.

[0050] It should be further explained that the generation of target aggregate instances is not limited to table-level metadata, but can also include multiple sub-data structures such as field-level metadata, element-level metadata, and dictionary-level metadata. For example, when building a comprehensive customer management system, the target aggregate instance may include the customer's basic information (table-level metadata), the customer's purchase history (field-level metadata), the customer's specific interaction records (element-level metadata), and customer satisfaction survey options (dictionary-level metadata). Such a design enables the target aggregate instance to fully reflect the characteristics of the data and provide rich information for subsequent data applications. This application does not limit this.

[0051] Finally, it should be pointed out that the target metadata generated based on the target aggregation instance can be assembled in different target machine languages ​​according to different technical platforms and application scenarios. For example, in a cloud service platform, the target metadata may need to be assembled in the style of a RESTful API to be compatible with the cloud service interface; in a traditional relational database, the target metadata may need to be assembled in accordance with the SQL language to create table structures and indexes in the database; in a big data processing platform, the target metadata may need to be assembled in accordance with the language specifications in the Hadoop ecosystem to work with components such as HDFS and MapReduce. This flexibility enables the target metadata to adapt to different technical environments and meet diverse data processing needs. This application does not limit this.

[0052] Through the embodiments of the present application, a metadata self-generated data fusion method based on the RAG (Retrieval-augmented Generation) model is adopted, and all data instances are processed through preset clustering and merging rules. The retrieval-augmented generation capability of the RAG model is utilized to cluster and merge data instances, and target aggregation instances are determined to ensure the comprehensiveness and consistency of data fusion. The purpose of organically and logically or physically concentrating data of different sources, formats, characteristics and properties is achieved, thereby realizing the automation and intelligent technical effects of data fusion.

[0053] Furthermore, target metadata is generated based on the target aggregated instance, wherein the target metadata is set to be assembled according to the target machine language. Through the automated metadata generation process, the aggregated instance is converted into a machine-readable metadata format, so that the target metadata can be directly used for the creation of database tables and the definition of data fields, which not only improves the efficiency of metadata generation, but also ensures the high compatibility of the generated metadata with the target machine language, further realizes the automation of the data fusion process and the seamless integration between systems, achieves the purpose of improving data processing efficiency, reducing human errors, and enhancing data consistency, and realizes the high automation and precision of data fusion technology.

[0054] As an optional solution, all data instances are clustered and merged through a retrieval enhanced generation model based on a preset clustering merging method to determine a target aggregate instance, including: setting a preset clustering merging method; storing the preset clustering merging method in a knowledge base corresponding to the retrieval enhanced generation model; referencing the knowledge base through the retrieval enhanced generation model to automatically cluster and merge all data instances to determine a target aggregate instance.

[0055] Optionally, in an embodiment of the present application, the above-mentioned setting of the preset clustering merging method refers to defining a set of rules or standards to guide the retrieval enhancement generation model on how to perform logical or physical centralized processing of the full amount of data instances. This includes but is not limited to determining which data should be aggregated together and how to organize these data to form meaningful information units. For example, in medical data analysis, the preset clustering merging method may be set according to the patient's diagnosis, treatment type, or drug use to ensure that similar medical events are aggregated together. In the field of business intelligence, this may involve setting clustering rules based on product categories, sales areas, or customer groups. These preset clustering merging methods help to extract valuable information from large amounts of complex data and support decision making. Storing the preset clustering merging method in the knowledge base corresponding to the retrieval enhancement generation model means storing these rules or standards in a structured database that can be accessed and used by the model. This knowledge base provides the necessary contextual information for the retrieval enhancement generation model, enabling it to perform clustering merging tasks more accurately. The data in the knowledge base may include but is not limited to classification rules, attribute weights, historical clustering results, etc. For example, in a customer relationship management system, the knowledge base may contain classification rules for customer feedback, which are set based on the customer's purchase history, service interaction, and feedback content so that the model can automatically identify and classify new customer feedback. The retrieval-enhanced generation model refers to the knowledge base, automatically clusters and merges the full data instances, and determines the target aggregated instances. This refers to using the model's retrieval ability to find and reference relevant information in the knowledge base, and combining it with the generation ability to create a new data structure. This process involves analyzing and understanding the full data instances, and integrating the data according to the preset clustering and merging methods. For example, in social media analysis, the model may refer to the rules for topic classification in the knowledge base, automatically cluster and merge related posts and comments, and form a comprehensive view of a specific event or trend. In the financial field, the model may automatically aggregate transaction records into an activity overview of customer accounts based on the transaction type and amount range rules in the knowledge base. Such automated clustering and merging not only improves the efficiency of data processing, but also enhances the readability and usability of data.

[0056] It should be noted that the preset clustering and merging methods can be designed in a variety of ways according to different business needs and data characteristics. For example, in the field of e-commerce, the clustering and merging methods can be set according to dimensions such as product categories, customer ratings, and sales regions, so as to aggregate similar products or customer feedback. In the field of healthcare, clustering rules can be designed based on dimensions such as the patient's age, disease type, and treatment results, so as to analyze and study similar cases. In the financial industry, clustering and merging methods can be set according to dimensions such as transaction amount, transaction time, and transaction type, so as to facilitate risk assessment and fraud detection of transaction data. This application does not limit this.

[0057] Storing the preset clustering merging method in the knowledge base corresponding to the retrieval enhancement generation model means that these rules can be customized and updated according to different scenarios and needs. For example, in the field of customer service, the knowledge base may contain sentiment analysis rules based on customer feedback so that the model can identify and classify customers' emotional tendencies. In supply chain management, the knowledge base may contain clustering rules based on logistics information so that the model can track and optimize the flow of goods. In the field of education, the knowledge base may contain clustering rules based on student performance so that the model can identify students' learning patterns and needs. The update and maintenance of these knowledge bases can be dynamic to adapt to the ever-changing data and business environment. This application is not limited to this.

[0058] By retrieving the enhanced generation model reference knowledge base, all data instances are automatically clustered and merged, and the process of determining the target aggregate instance can involve a variety of data processing technologies and algorithms. For example, in text data processing, the model can use natural language processing technology to understand and classify text content. In the field of image recognition, the model can apply computer vision technology to identify and cluster image data. In structured data analysis, the model can use machine learning algorithms to discover patterns and associations in the data. The application of these technologies enables the model to accurately perform clustering and merging tasks and generate meaningful target aggregate instances, providing a powerful tool for decision support and data analysis. This application is not limited to this.

[0059] Through the embodiments of the present application, by storing these preset clustering and merging methods into the knowledge base corresponding to the retrieval enhancement generation model (RAG model), the purpose of automatically clustering and merging all data instances is achieved. The RAG model uses its retrieval enhancement capability to reference the clustering and merging rules in the knowledge base to automatically identify and process data instances, thereby determining the target aggregation instance, so that the data fusion process no longer relies on manual operation, but through an automated method, intelligently classifies and merges data according to preset rules, thereby improving the efficiency and accuracy of data processing, achieving the purpose of reducing manual intervention and improving data processing efficiency and accuracy, thereby achieving the technical effect of improving the quality of data fusion and the level of automation, not only reducing errors caused by manual operation, but also improving the flexibility and scalability of data fusion, providing a more accurate and comprehensive data foundation for subsequent data analysis and decision support.

[0060] As an optional solution, clustering and merging the full data instances through a retrieval enhancement generation model based on a preset clustering and merging method to determine a target aggregate instance, including: obtaining description information corresponding to each table-level metadata in the full data instance through a retrieval enhancement model; grouping the full data instances according to the description information to obtain at least two groups of data instances corresponding to at least two sub-data structures; for each group of data instances, merging them according to the attribute parameters of the corresponding sub-data structure to obtain a target aggregate instance.

[0061] Optionally, in an embodiment of the present application, the above-mentioned retrieval enhancement model refers to a model that combines retrieval and generation techniques, specifically, RAG (Retrieval-augmented Generation). This model generates answers or content by referencing information from a knowledge base, and has strong interpretability and customization capabilities. It is suitable for multiple natural language processing tasks such as question-and-answer systems, document generation, and intelligent assistants. The advantage of the RAG model is that it is highly versatile, can achieve instant knowledge updates, and provides more efficient and accurate information services through end-to-end evaluation methods. In this application, the RAG model is used to process each table-level metadata in the full data instance to obtain its corresponding descriptive information, which is the key input in the subsequent data fusion and metadata automatic generation process.

[0062] Optionally, in an embodiment of the present application, the above-mentioned description information refers to the schema attribute value contained in the table-level metadata, which provides a business description of the table, that is, the business meaning and purpose of the table. For example, in a table of e-commerce platform user order data, the schema may be described as "e-commerce platform user order data", while in a table of travel platform user order information, the schema may be described as "travel platform user order information". These description information are the key to understanding and distinguishing different data tables. They help the RAG model identify and understand the content and context of the data tables, so as to correctly group and merge similar or related data tables during the data fusion process.

[0063] Optionally, in an embodiment of the present application, the above grouping refers to logically classifying data tables with similar or identical business semantics according to the description information of the table-level metadata in the full data instance. For example, all order-related data tables can be grouped into one group, and data tables related to user information can be grouped into another group. This grouping is based on business logic and the description information of the data table. The purpose is to be able to merge data tables with the same or similar business attributes into a new aggregate instance in the subsequent data fusion process, thereby realizing the organic concentration and sharing of data.

[0064] Optionally, in an embodiment of the present application, the above-mentioned sub-data structure refers to the basic elements that constitute a data table, including table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. These sub-data structures together define the structure and content of a data table. For example, table-level metadata defines the basic information of a data table, field-level metadata defines the attributes of each field in the table, element-level metadata describes the data content in the table, and dictionary-level metadata defines the dictionary items used in the table. In the present application, for each group of data instances, they will be merged according to the attribute parameters of these sub-data structures to generate a target aggregate instance.

[0065] Optionally, in an embodiment of the present application, the above-mentioned target aggregate instance refers to a new data instance obtained by the above-mentioned grouping and merging operations, which includes the merged table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata. This aggregate instance is the result of data fusion, which logically or physically brings together data from multiple sources, formats, and characteristics to form a unified data model. For example, if there are two different e-commerce platform order data tables, they can be merged into a target aggregate instance through the method of the present application, which includes the order data of the two platforms while maintaining the integrity, accuracy, and consistency of the data. Such an aggregate instance provides enterprises with a comprehensive data sharing view, supporting more efficient data analysis and decision making.

[0066] It should be noted that the process of obtaining the descriptive information corresponding to each table-level metadata in the full data instance through the retrieval enhancement model may involve a variety of different data sources and data types. For example, the data source may include a relational database, a non-relational database, a data warehouse, or a data lake, and the data type may include structured data, semi-structured data such as JSON, and unstructured data such as text documents. This application does not limit this.

[0067] Furthermore, the full amount of data instances are grouped according to the description information. In this step, the classification of the description information can be based on different business logics, data attributes or data sources. For example, it can be grouped according to industry fields (such as finance, medical care, education), or according to the sensitivity of the data (such as public data, internal data, confidential data) or the update frequency of the data (such as real-time data, daily updated data, historical archived data). This application does not limit this.

[0068] Finally, for each group of data instances, merge according to the attribute parameters of the corresponding sub-data structure. In this process, the attribute parameters of the sub-data structure may include but are not limited to the format, version, quality, timeliness, etc. of the data. For example, data instances with the same data format (such as CSV, XML) can be merged, or data instances from the same version control system can be merged, or data instances with the same data quality standards (such as accuracy, completeness) can be merged. In addition, it is also possible to merge according to the timeliness of the data (such as real-time data, historical data). This application is not limited to this. In this way, data instances can be flexibly aggregated and fused effectively according to different needs and scenarios to generate target aggregation instances that meet specific needs.

[0069] Through the embodiment of the present application, the RAG (Retrieval-augmented Generation) model is used to enhance the ability of data retrieval and generation. The descriptive information corresponding to each table-level metadata in the full data instance is obtained through the retrieval enhancement model. This step uses the retrieval ability of the RAG model to identify and extract key metadata descriptions, providing accurate input for data fusion. Then, the full data instance is grouped according to these descriptive information to obtain at least two groups of data instances corresponding to at least two sub-data structures. This process is based on the business semantics in the descriptive information, and data tables with similar business attributes are grouped into the same group, realizing the logical concentration and preprocessing of data. Finally, for each group of data instances, the attribute parameters of the corresponding sub-data structure are merged, such as the attributes of field-level metadata and dictionary-level metadata, to obtain the target aggregate instance. This step realizes the automatic generation of metadata and the physical concentration of data through automated clustering and merging rules. Through this series of processing, the purpose of reducing manual intervention and improving data processing efficiency and accuracy is achieved, thereby achieving the technical effect of improving data fusion quality and automation level.

[0070] As an optional solution, for each group of data instances, the attribute parameters of the corresponding sub-data structure are merged to obtain a target aggregate instance, including: when the sub-data structure corresponds to the table-level metadata, the following operations are performed on each group of data instances, wherein a group of data instances that performs the following operations each time is regarded as the first group of data instances: the table-level data source attribute parameters of each data instance in the first group of data instances are added and filled to obtain target table-level data source attribute parameters; the table-level identification attribute parameters of each data instance in the first group of data instances are set to be filled according to the first language business text to obtain the target table-level identification attribute parameters corresponding to the target aggregate instance; the table-level description attribute parameters of each data instance in the first group of data instances are set to be filled according to the second language business text to obtain the target table-level description attribute parameters, wherein the first language business text and the second language business text use different languages; and the target aggregate instance is generated based on the target table-level data source attribute parameters, the target table-level identification attribute parameters and the target table-level description attribute parameters.

[0071] Optionally, in an embodiment of the present application, the target table-level data source attribute parameter refers to the attribute parameter obtained by merging the source attributes of the table-level metadata in each group of data instances during the data fusion process. This parameter usually contains all the data source information of the merged data instance. For example, if order tables from different databases are merged, the target table-level data source attribute parameter may contain the names or identifiers of all these databases, separated by commas, such as "database A, database B, database C".

[0072] Optionally, in an embodiment of the present application, the target table-level identification attribute parameter refers to an attribute parameter obtained by merging the identification attributes of the table-level metadata in each group of data instances during the data fusion process. This parameter is generally used to provide a business identifier for one or more merged data tables. For example, if multiple order tables are merged, the target table-level identification attribute parameter may be set to "Order", which is an identifier filled in with business text in the first language (such as English), which concisely describes the main business content of the merged data table.

[0073] Optionally, in an embodiment of the present application, the target table-level description attribute parameter refers to the attribute parameter obtained by merging the description attributes of the table-level metadata in each group of data instances during the data fusion process. This parameter provides a detailed business description of the merged data table. For example, if multiple order tables are merged, the target table-level description attribute parameter may be set to "contains order information from different platforms", which is a description filled in with business text in a second language (such as Chinese), which details the data content and business background contained in the merged data table.

[0074] It should be noted that, in the case where the sub-data structure corresponds to the table-level metadata, the operations performed on each group of data instances can be processed in a diversified manner according to different business requirements and data characteristics, and this application does not limit this. For example, for the addition and filling of table-level data source attribute parameters, dimensions such as the name, version number, and data format of the data source can be included, such as merging the data source names from different versions of databases to form "database V1, database V2". Such operations can adapt to the needs of merging data sources of different versions, and can also be adjusted according to different data format requirements.

[0075] Furthermore, the setting of table-level identification attribute parameters and table-level description attribute parameters can be diversified according to different business scenarios and language requirements. For example, the first language business text can be English, which is used to identify attribute parameters, such as unifying the identification of multiple order tables as "Order", and the second language business text can be Chinese, which is used to describe attribute parameters, such as setting the description of the order table to "contains order information of e-commerce platforms and travel platforms". Such settings are not only suitable for data fusion in different language environments, but can also be customized according to different business fields and data contents. The target aggregation instance generated based on the target table-level data source attribute parameters, the target table-level identification attribute parameters and the target table-level description attribute parameters can be generated in a diversified manner according to different data fusion rules and business logic. For example, it can be aggregated according to dimensions such as the business importance of the data, the update frequency of the data or the access rights of the data to generate an aggregation instance that meets specific business needs. Such an aggregation instance can be used for different data analysis, report generation or decision support systems, which improves the flexibility and practicality of data fusion.

[0076] For example, consider a data fusion scenario of an e-commerce platform, which needs to integrate sales data from subsidiaries in different regions to facilitate global sales analysis. The following is the specific implementation process:

[0077] S 1. Design a unified meta-model: First, define a meta-model that includes table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata information. This meta-model will serve as the basic framework for data fusion.

[0078] S2, extracting the full amount of basic information from the data source: Use data source exploration technologies, such as DataX or FlinkX, to extract the full amount of basic information from the databases of subsidiaries in each region, including fine-grained basic information such as libraries, tables, fields, records, and dictionaries.

[0079] S3, build a unified full data source instance: store the extracted full basic information of the data source according to the definition and rules of the metamodel to form a unified data source instance.

[0080] S4, define metamodel instance clustering and merging rules: define rules to logically or physically aggregate data of different sources, formats, and characteristics. For example, group all sales data tables according to business semantics.

[0081] S5, storing in RAG knowledge base: storing the defined clustering merging rules in the knowledge base of the RAG model so that the RAG model can process the data according to these rules.

[0082] S6, clustering and merging through the RAG model: The packaged full meta-model instance is clustered and merged through the RAG model. The RAG model performs clustering and merging operations on the data according to the rules in the knowledge base and generates new aggregate instances.

[0083] S7, add and fill the table-level data source attribute parameters for the first group of data instances: for example, if the data comes from two subsidiaries, "North America" ​​and "Europe", the target table-level data source attribute parameters may be filled with "North America, Europe".

[0084] S8, setting table-level identification attribute parameters: setting the table-level identification attribute parameters of each data instance in the first group of data instances to be filled with business text in the first language (such as English), for example, "Sales Data".

[0085] S9, setting table-level description attribute parameters: setting the table-level description attribute parameters of each data instance in the first group of data instances to be filled in according to the business text in the second language (such as Chinese), for example, "including sales data of North American and European subsidiaries".

[0086] S10, generating a target aggregation instance: based on the target table-level data source attribute parameters, the target table-level identification attribute parameters, and the target table-level description attribute parameters, generating a target aggregation instance including the sales data of all regional subsidiaries.

[0087] Through the above process, the goal of effectively integrating the sales data of subsidiaries scattered in different regions was achieved. This process not only improves the efficiency of data processing and reduces manual operation errors, but also enhances the consistency and accuracy of data. Through automated metadata generation and data fusion, enterprises can quickly obtain sales analysis from a global perspective and support more accurate business decisions. In addition, due to the use of multilingual business text filling, the target aggregation instance can better adapt to the language habits of different regions, improving the readability and usability of the data. Ultimately, this method improves the automation and intelligence level of data fusion, providing enterprises with a comprehensive, accurate and consistent data view.

[0088] As an optional solution, for each group of data instances, the attribute parameters of the corresponding sub-data structures are merged to obtain a target aggregate instance, including: when the sub-data structure corresponds to the field-level metadata, the following operations are performed on each group of data instances, wherein a group of data instances that performs the following operations each time is regarded as a second group of data instances: semantically deduplicating the field-level description attribute parameters and the field-level data item attribute parameters of each data instance in the second group of data instances, setting the deduplicated field-level description attribute parameters to be filled in according to the second language business text, and obtaining the target field-level description attribute parameters; setting the field-level label of each data instance in the second group of data instances ...; obtaining the target field-level description attribute parameters; setting the field-level label of each data instance in the second group of data instances to be filled in according to the second language business text; obtaining the target field-level description attribute parameters; and setting the field-level label of each data instance in the second group of data instances to be filled in according to the second language business text. The identification attribute parameter is set to be filled according to the first language business text to obtain the target field-level identification attribute parameter, wherein the first language business text and the second language business text use different languages; the field-level data type attribute parameter of each data instance in the second group of data instances is filled according to the original value to obtain the target field-level data type attribute parameter; the field-level size attribute parameter of each data instance in the second group of data instances is filled according to the maximum value to obtain the target field-level size attribute parameter; the target aggregate instance is generated based on the target field-level description attribute parameter, the target field-level identification attribute parameter, the target field-level data type attribute parameter and the target field-level size attribute parameter.

[0089] Optionally, in an embodiment of the present application, the above-mentioned field-level metadata refers to a data structure that constitutes detailed description information of each field in a data table. Field-level metadata includes attributes such as the name, type, size, precision, etc. of the field, which define the data characteristics and storage requirements of the field. For example, a field-level metadata may contain a field name "customer_id", a field type "integer", a size "10", a precision "0", and a field description "unique identifier of the customer". Field-level metadata is an important part of the data table structure, which provides necessary information for the storage, retrieval and management of data.

[0090] Optionally, in an embodiment of the present application, semantic deduplication refers to the process of identifying and merging field descriptions with the same or similar business meanings during the data fusion process. This process ensures the consistency and accuracy of the field descriptions after data fusion. For example, when merging order information from different databases, you may encounter a situation where two fields describe the customer name, but the field names may be different, one may be "customer_name" and the other may be "name". Semantic deduplication identifies these two fields as fields with the same meaning and retains only one in the target aggregation instance.

[0091] Optionally, in an embodiment of the present application, the first language business text and the second language business text refer to different language texts used to fill in field-level identification attribute parameters and field-level description attribute parameters during the data fusion process. These texts reflect the business meaning of the field and are in different languages. For example, the first language business text may be English, which is used to identify attribute parameters, such as identifying the field "order_date" as "Order Date"; while the second language business text may be Chinese, which is used to describe attribute parameters, such as describing the same field as "Order Date".

[0092] Optionally, in the embodiment of the present application, the field-level data type attribute parameter refers to metadata that defines the field data type. This parameter specifies the data type that the field can store, such as integers, floating point numbers, strings, etc. For example, a field-level data type attribute parameter can be "character type", corresponding to a field that can store text data.

[0093] Optionally, in an embodiment of the present application, a field-level size attribute parameter refers to metadata that defines the maximum storage size of a field. This parameter specifies the maximum number of characters or bytes that a field can store. For example, a field-level size attribute parameter may be "255", indicating that the maximum character length that the field can store is 255 characters.

[0094] It should be noted that, when the sub-data structure corresponds to the field-level metadata, the operations on the second group of data instances can be diversified according to different data characteristics and business needs, and this application does not limit this. For example, when performing semantic deduplication of field-level description attribute parameters and field-level data item attribute parameters, three dimensions can be considered: data content, data format, and data source. In terms of data content, it may be necessary to compare the semantic similarity of field descriptions; in terms of data format, it may be necessary to unify the field naming specifications of different data sources; in terms of data sources, it may be necessary to identify and merge the same fields from different systems or departments.

[0095] When setting field-level identification attribute parameters, you can diversify the processing from three dimensions: language, region, and business field. In terms of language, the first language business text can be English, while the second language business text can be Chinese to adapt to the reading habits of users in different regions; in terms of region, it can be adjusted according to the language preference of the subsidiary's location; in terms of business field, the business text of the field identification can be customized according to different business scenarios, such as finance, medical care, or education.

[0096] When filling in the field-level data type attribute parameters, you can consider the three dimensions of data storage requirements, query efficiency, and data consistency. In terms of storage requirements, you need to select the appropriate data type based on the actual type of data; in terms of query efficiency, selecting the appropriate data type can optimize the query performance of the database; in terms of data consistency, maintaining the consistency of data types helps reduce errors in the data fusion process.

[0097] When filling in the field-level size attribute parameters, you can perform diversified processing from three dimensions: maximum length of the field, performance optimization, and data integrity. In terms of maximum length, selecting the maximum value can ensure that all data can be stored; in terms of performance optimization, a reasonable size can improve the efficiency of database operations; in terms of data integrity, ensure that the field size can accommodate all possible data values ​​to maintain data integrity.

[0098] The target aggregate instance generated based on the target field-level description attribute parameters, target field-level identification attribute parameters, target field-level data type attribute parameters, and target field-level size attribute parameters can be considered from the three dimensions of data availability, consistency, and business adaptability. In terms of availability, the aggregate instance provides a unified data view for easy user access and analysis; in terms of consistency, the aggregate instance ensures the accuracy and reliability of the data; in terms of business adaptability, the aggregate instance can be customized and optimized according to different business needs. Through these diverse operations, target aggregate instances that meet the needs of different business scenarios can be generated, which improves the flexibility and effectiveness of data processing.

[0099] Through the embodiment of the present application, based on the above target field-level description attribute parameter, target field-level identification attribute parameter, target field-level data type attribute parameter, and target field-level size attribute parameter, a target aggregation instance integrating multiple data source information can be generated. This instance provides a unified data view for data analysis and reporting, while taking into account the needs of users of different languages, and improving the availability and accessibility of data.

[0100] As an optional solution, for each group of data instances, the attribute parameters of the corresponding sub-data structures are merged to obtain a target aggregate instance, including: when the sub-data structure corresponds to the dictionary-level metadata, the following operations are performed on each group of data instances, wherein a group of data instances that performs the following operations each time is regarded as the third group of data instances: semantically deduplicating the dictionary-level value parameters of each data instance in the third group of data instances to obtain target dictionary-level value attribute parameters; filling the dictionary-level description attribute parameters and the dictionary-level data type attribute parameters of each data instance in the third group of data instances with the original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; generating a target aggregate instance based on the target dictionary-level value attribute parameters, the target dictionary-level description attribute parameters and the target dictionary-level data type attribute parameters.

[0101] Optionally, in an embodiment of the present application, the above-mentioned dictionary-level metadata refers to metadata related to a dictionary type field in a data table. The dictionary-level metadata contains a collection of dictionary items, each of which corresponds to a specific value and description. For example, in an order management system, the order status may be used as a dictionary field, and its dictionary-level metadata may include states such as "pending payment", "paid", "shipping", and "completed", each of which corresponds to an integer value. Dictionary-level metadata is an important component of the data model. It provides standardized classification and coding for data, which helps with data consistency and comparative analysis.

[0102] Optionally, in an embodiment of the present application, a dictionary-level value parameter refers to the value of a dictionary item, which is usually unique and is used to identify a specific dictionary item in the database. For example, in the dictionary field of the order status described above, each status such as "pending payment" may correspond to an integer value, such as 1, which is the dictionary-level value parameter. In the data fusion process, semantic deduplication of dictionary-level value parameters means identifying and merging different values ​​with the same business meaning to ensure the uniqueness and consistency of dictionary items after data fusion.

[0103] Optionally, in the embodiment of the present application, the dictionary-level description attribute parameter refers to a text description of the dictionary item, providing a detailed explanation and business meaning of the dictionary item. For example, for the dictionary item "1" of the order status, its dictionary-level description attribute parameter may be "to be paid", and this description helps the user understand the specific meaning of each dictionary value. In the data fusion process, maintaining the original value of the dictionary-level description attribute parameter means retaining the specific description of the dictionary item in each data source to ensure that the business meaning of the data is not lost.

[0104] Optionally, in an embodiment of the present application, a dictionary-level data type attribute parameter refers to metadata that defines the data type of a dictionary item value. This parameter specifies the storage type of the dictionary item value, such as an integer, a string, etc. For example, if the dictionary-level value parameter of the order status is stored in the form of an integer, then its dictionary-level data type attribute parameter is "integer". During the data fusion process, filling the dictionary-level data type attribute parameter with the original value means that in the generated target aggregate instance, the data type of each dictionary item value is kept unchanged to ensure data compatibility and consistency.

[0105] It should be noted that, in the case where the sub-data structure corresponds to the dictionary-level metadata, the operation of the third group of data instances can be processed in a diversified manner according to different business rules, data specifications and system requirements, and this application does not limit this. For example, when performing semantic deduplication of dictionary-level value parameters, it can be considered from three dimensions: business meaning, data format and system compatibility. In terms of business meaning, it is necessary to identify the same or similar business concepts in different data instances, such as unifying different codes representing the "completed" state in different systems into a standard value; in terms of data format, it is necessary to convert codes of different formats into a unified format to facilitate data comparison and processing; in terms of system compatibility, it is necessary to ensure that the deduplicated dictionary-level value parameters can be correctly identified and processed in each system. When filling in the dictionary-level description attribute parameters, it can be processed in a diversified manner from three dimensions: the level of detail of the description, language differences and cultural differences. In terms of detail, the level of detail of the description needs to be determined according to business needs, such as in some cases it may be necessary to provide a more detailed description of the dictionary item; in terms of language differences, the dictionary item description needs to be localized according to the language habits of different regions; in terms of cultural differences, it is necessary to consider that the same dictionary item may have different understandings and expressions under different cultural backgrounds. When filling in dictionary-level data type attribute parameters, you can consider the three dimensions of data storage efficiency, processing performance, and data security. In terms of storage efficiency, you need to select the appropriate data type to optimize the use of storage space; in terms of processing performance, you need to consider the processing efficiency of the data type, such as the processing speed of integer types is usually faster than that of string types; in terms of data security, you need to ensure that the selection of data types does not introduce security risks, such as avoiding storing sensitive information in plain text.

[0106] Through the embodiment of the present application, based on the target dictionary-level value attribute parameter, the target dictionary-level description attribute parameter and the target dictionary-level data type attribute parameter, a unified target aggregation instance can be generated that includes the deduplicated dictionary items and retains the original business description and data type. This instance provides a standardized dictionary reference for data analysis and reporting, while ensuring the accuracy and availability of the data model after data fusion.

[0107] The following is a further explanation of this application with reference to specific examples:

[0108] In the process of implementing data sharing, since the data provided by different users may come from different channels, their data content, data format and data quality vary greatly. Sometimes we even encounter thorny problems such as the inability to convert data format or information loss after data format conversion, which seriously hinders the flow and sharing of data among departments and software systems.

[0109] Data warehouse is a structured data environment for decision support systems and online analytical application data sources. Data warehouse studies and solves the problem of obtaining information from databases. Data warehouse is integrated. The data in data warehouse comes from scattered operational data. The required data is extracted from the original data, processed and integrated, and can enter the data warehouse after unification and synthesis.

[0110] Data fusion is to logically or physically integrate data from different sources, formats, and characteristics, so as to provide comprehensive data sharing for enterprises. Designing a consistent data model and ensuring the integrity, accuracy, and consistency of data are the keys to data fusion. Metadata is the foundation and link for building data models and sharing data.

[0111] RAG is a model that combines retrieval and generation techniques. It generates answers or content by referencing information in the knowledge base. It has strong interpretability and customization capabilities and is suitable for multiple natural language processing tasks such as question-answering systems, document generation, and intelligent assistants. The advantages of the RAG model are its strong versatility, instant knowledge updates, and more efficient and accurate information services through end-to-end evaluation methods.

[0112] The present application proposes a metadata self-generated data fusion method based on RAG. The method firstly extracts the basic information of the full data source from the data of different sources, formats and characteristics that need to be accessed through the DSE (Data Source Exploration) process, and encapsulates it with a uniformly designed metamodel (including table-level metadata, field-level metadata, element-level metadata and dictionary-level metadata) to complete the basic preparation of metadata; secondly, the metamodel clustering and merging rules are defined, and the metamodel clustering and merging rules are stored in the RAG knowledge base to complete the metadata rule definition work; finally, the encapsulated full metamodel instances are clustered and merged through the RAG model, and the results are output to complete the automatic generation of metadata.

[0113] This application proposes a metadata self-generation data fusion method based on RAG. This method integrates multiple technologies such as DSE and RAG, and converts the traditional process of manually generating metadata into an automatic generation method through artificial intelligence. The principle of promoting this application is explained as follows:

[0114] 1. Design a unified metamodel; 2. Extract the full basic information of the data source; 3. Build a unified full data source instance based on the metamodel; 4. Define the clustering and merging rules of the metamodel instances and store them in the RAG knowledge base; 5. Cluster and merge the full metamodel instances through the RAG model and generate aggregate instances; 6. Automatically generate metadata based on the aggregate instances.

[0115] Figure 3 is a flowchart of a method for generating metadata according to an embodiment of the present application, such as Figure 3 The specific implementation process of this application is described in detail as follows:

[0116] S302: Design a unified metamodel. The metamodel is defined and stored in JSON format, including table-level metadata, field-level metadata, element-level metadata, and dictionary-level metadata information, as shown below:

[0117]

[0118]

[0119] S304: Extract the full amount of basic information of the data source. Use data source exploration technology such as DataX or FlinkX to obtain the full amount of basic information of the data source, including fine-grained basic information such as libraries, tables, fields, records, dictionaries, such as database name, table name, table description, field name, field length, field precision, field description, data item content, data item type, data item description, dictionary content, dictionary type, dictionary description, etc.

[0120] S306: Build a unified full data source instance based on the metamodel. Store the extracted full data source basic information according to the definition and rules of the metamodel. The example is as follows:

[0121]

[0122]

[0123]

[0124] S308: Define metamodel instance clustering and merging rules and store them in the RAG knowledge base. The purpose of defining metamodel clustering and merging rules is to logically or physically concentrate data from different sources, formats, and characteristics. In other words, it is to merge multiple tables with the same business attributes into one table. The metadata of this table is what we need to create. Rule definition can be understood as a procedural prompt word project. After the definition is completed, the rules are stored in the RAG knowledge base. The specific examples are as follows:

[0125] (1) Business semantic grouping is performed based on the schema value of the table-level metadata table_infos of the full metamodel instance.

[0126] (2) Generate new metamodel table-level metadata table_infos according to the grouping, where the source value is filled with the sum of all instance source values ​​in the group and separated by ","; the name value is filled with business English words; and the schema value is filled with business Chinese words.

[0127] (3) Generate new metamodel field-level metadata table_fields according to the grouping, where semantic deduplication is performed based on the schema value of each field in the group of metamodel instances table_fields and the data value of each element in the corresponding table_elemenst. The name value is filled with business English words; the schema value is filled with business Chinese words, the type value is filled with the original value, and the size value is filled with the maximum value.

[0128] (4) Generate new metamodel dictionary-level metadata table_dicts according to the group, where semantic deduplication is performed based on the values ​​of each dictionary in the group of metamodel instance table_dicts, and type and schema are filled with original values.

[0129] S310: Cluster and merge all metamodel instances through the RAG model, and generate new aggregate instances. RAG is a model that combines retrieval and generation techniques. It generates answers or content by referencing information from the knowledge base, and has strong interpretability and customization capabilities. Load all metamodel instances into the RAG model, and the RAG model will perform clustering and merging operations on them according to the rules designed in the knowledge base, and generate new aggregate instances. The following is an example of the aggregation of metamodels 01 and 02 in the third step:

[0130]

[0131]

[0132] S312: Automatically generate metadata based on the aggregated instance. The specific steps are as follows:

[0133] (1) Assemble SQL statements based on the contents of table_infos to automatically create a table;

[0134] (2) Assemble SQL statements based on the contents of table_fields to automatically create fields;

[0135] (3) Assemble SQL statements based on the contents of table_dicts to automatically create a dictionary.

[0136] This application proposes a method for data fusion with self-generated metadata and a data fusion design concept based on meta-model, which realizes the automatic generation of metadata, can significantly improve the efficiency and quality of data fusion, and effectively solve the various shortcomings of manual methods.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above methods of each embodiment of the present application.

[0138] In this embodiment, a metadata generation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0139] Figure 4 is a structural block diagram of a metadata generation device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0140] An acquisition module 402 is used to acquire a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding thereto;

[0141] The encapsulation module 404 is used to encapsulate the full amount of basic information using the meta model to obtain a full amount of data instance, wherein the full amount of data instance represents the full amount of basic information stored according to the rules indicated by the meta model;

[0142] Clustering module 406 is used to cluster and merge the above-mentioned full data instances through the retrieval enhancement generation model based on the preset clustering merging method to generate target metadata, wherein the above-mentioned retrieval enhancement generation model is used to cluster the above-mentioned full data instances according to the above-mentioned preset clustering merging method, and the above-mentioned preset clustering merging method includes grouping according to the description information of the above-mentioned full data instances.

[0143] As an optional solution, the above-mentioned device is used to cluster and merge the above-mentioned full data instances through the retrieval enhanced generation model based on a preset clustering and merging method to generate target metadata in the following manner: cluster and merge the above-mentioned full data instances through the retrieval enhanced generation model based on a preset clustering and merging method to determine the target aggregate instance, wherein the above-mentioned target aggregate instance includes an instance corresponding to the table-level metadata, and the above-mentioned at least two sub-data structures include the above-mentioned table-level metadata; generate target metadata based on the above-mentioned target aggregate instance, wherein the above-mentioned target metadata is set to be assembled according to the target machine language.

[0144] As an optional scheme, the above-mentioned device is used to cluster and merge the above-mentioned full data instances through the above-mentioned retrieval enhancement generation model based on a preset clustering merging method in the following manner to determine the target aggregated instance: setting the above-mentioned preset clustering merging method; storing the above-mentioned preset clustering merging method into the knowledge base corresponding to the above-mentioned retrieval enhancement generation model; referencing the above-mentioned knowledge base through the above-mentioned retrieval enhancement generation model, automatically clustering and merging the above-mentioned full data instances, and determining the above-mentioned target aggregated instance.

[0145] As an optional scheme, the above-mentioned device is used to cluster and merge the above-mentioned full data instances through the above-mentioned retrieval enhancement generation model based on a preset clustering and merging method in the following manner to determine the target aggregate instance: obtain the above-mentioned description information corresponding to each table-level metadata in the above-mentioned full data instance through the above-mentioned retrieval enhancement model; group the above-mentioned full data instances according to the above-mentioned description information to obtain at least two groups of data instances corresponding to the above-mentioned at least two sub-data structures respectively; for each group of the above-mentioned data instances, merge them according to the attribute parameters of the corresponding sub-data structure to obtain the above-mentioned target aggregate instance.

[0146] As an optional solution, the above-mentioned device is used to merge the above-mentioned data instances in each group according to the attribute parameters of the corresponding sub-data structure in the following manner to obtain the above-mentioned target aggregate instance: when the above-mentioned sub-data structure corresponds to the above-mentioned table-level metadata, perform the following operations on each group of the above-mentioned data instances, wherein a group of the above-mentioned data instances that performs the following operations each time is regarded as the first group of data instances: add and fill the table-level data source attribute parameters of each data instance in the above-mentioned first group of data instances to obtain target table-level data source attribute parameters; set the table-level identification attribute parameters of each data instance in the above-mentioned first group of data instances to be filled in accordance with the first language business text to obtain the target table-level identification attribute parameters corresponding to the above-mentioned target aggregate instance; set the table-level description attribute parameters of each data instance in the above-mentioned first group of data instances to be filled in accordance with the second language business text to obtain the target table-level description attribute parameters, wherein the first language business text and the second language business text use different languages; generate the above-mentioned target aggregate instance based on the above-mentioned target table-level data source attribute parameters, the above-mentioned target table-level identification attribute parameters and the above-mentioned target table-level description attribute parameters.

[0147] As an optional solution, the above-mentioned device is used to merge the above-mentioned data instances in each group according to the attribute parameters of the corresponding sub-data structure in the following manner to obtain the above-mentioned target aggregate instance: when the above-mentioned sub-data structure corresponds to the field-level metadata, perform the following operations on each group of the above-mentioned data instances, wherein a group of the above-mentioned data instances that performs the following operations each time is regarded as the second group of data instances: semantically deduplicate the field-level description attribute parameters and field-level data item attribute parameters of each data instance in the above-mentioned second group of data instances, and set the above-mentioned field-level description attribute parameters after deduplication to be filled in according to the second language business text to obtain the target field-level description attribute parameters; set the field of each data instance in the above-mentioned second group of data instances The level identification attribute parameter is set to be filled according to the first language business text to obtain the target field level identification attribute parameter, wherein the first language business text and the second language business text use different languages; the field level data type attribute parameter of each data instance in the second group of the data instances is filled according to the original value to obtain the target field level data type attribute parameter; the field level size attribute parameter of each data instance in the second group of the data instances is filled according to the maximum value to obtain the target field level size attribute parameter; the target aggregate instance is generated based on the target field level description attribute parameter, the target field level identification attribute parameter, the target field level data type attribute parameter and the target field level size attribute parameter.

[0148] As an optional solution, the above-mentioned device is used to merge the attribute parameters of the corresponding sub-data structure for each group of the above-mentioned data instances in the following manner to obtain the above-mentioned target aggregate instance: when the above-mentioned sub-data structure corresponds to the dictionary-level metadata, perform the following operations on each group of the above-mentioned data instances, wherein a group of the above-mentioned data instances that performs the following operations each time is regarded as the third group of data instances: semantically deduplicate the dictionary-level value parameters of each data instance in the above-mentioned third group of data instances to obtain target dictionary-level value attribute parameters; fill the dictionary-level description attribute parameters and dictionary-level data type attribute parameters of each data instance in the above-mentioned third group of data instances according to the original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; generate the above-mentioned target aggregate instance based on the above-mentioned target dictionary-level value attribute parameters, the above-mentioned target dictionary-level description attribute parameters and the above-mentioned target dictionary-level data type attribute parameters.

[0149] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0150] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0151] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0152] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0153] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0154] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0155] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0156] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating metadata, characterized in that: include: Acquire a pre-created metamodel and full basic information extracted from at least one data source, wherein the metamodel includes at least two sub-data structures, the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding thereto; Encapsulating the full amount of basic information using the meta-model to obtain a full amount of data instance, wherein the full amount of data instance represents the full amount of basic information stored according to the rules indicated by the meta-model; Based on a preset clustering and merging method, the full amount of data instances are clustered and merged through a retrieval enhanced generation model to generate target metadata, wherein the retrieval enhanced generation model is used to cluster the full amount of data instances according to the preset clustering and merging method, and the preset clustering and merging method includes grouping according to the description information of the full amount of data instances.

2. The method according to claim 1, characterized in that The step of clustering and merging the full amount of data instances by searching and enhancing the generation model based on a preset clustering and merging method to generate target metadata includes: Based on a preset clustering and merging method, clustering and merging the full amount of data instances through the retrieval enhancement generation model to determine a target aggregate instance, wherein the target aggregate instance includes an instance corresponding to the table-level metadata, and the at least two sub-data structures include the table-level metadata; Generate target metadata based on the target aggregate instance, wherein the target metadata is configured to be assembled according to a target machine language.

3. The method according to claim 2, characterized in that The clustering and merging of the full amount of data instances through the retrieval enhancement generation model based on a preset clustering and merging method to determine a target aggregated instance includes: Setting the preset cluster merging mode; Storing the preset clustering merging method in the knowledge base corresponding to the retrieval enhancement generation model; The knowledge base is referenced by the retrieval enhancement generation model, the full amount of data instances are automatically clustered and merged, and the target aggregated instance is determined.

4. The method according to claim 1, characterized in that: The clustering and merging of the full amount of data instances through the retrieval enhancement generation model based on a preset clustering and merging method to determine a target aggregated instance includes: Acquire the description information corresponding to each table-level metadata in the full data instance through the retrieval enhancement model; Grouping the full amount of data instances according to the description information to obtain at least two groups of data instances corresponding to the at least two sub-data structures respectively; For each group of the data instances, the data instances are merged according to the attribute parameters of the corresponding sub-data structures to obtain the target aggregate instance.

5. The method according to claim 4, characterized in that The step of merging each group of data instances according to the attribute parameters of the corresponding sub-data structures to obtain the target aggregate instance includes: In the case where the sub-data structure corresponds to the table-level metadata, the following operations are performed on each group of the data instances, wherein a group of the data instances on which the following operations are performed each time is regarded as the first group of data instances: Add and fill the table-level data source attribute parameters of each data instance in the first group of data instances to obtain target table-level data source attribute parameters; Setting the table-level identification attribute parameter of each data instance in the first group of data instances to be filled in accordance with the first language business text, obtaining the target table-level identification attribute parameter corresponding to the target aggregate instance, and setting the table-level description attribute parameter of each data instance in the first group of data instances to be filled in accordance with the second language business text, obtaining the target table-level description attribute parameter, wherein the first language business text and the second language business text use different languages; The target aggregate instance is generated based on the target table-level data source attribute parameter, the target table-level identification attribute parameter, and the target table-level description attribute parameter.

6. The method according to claim 4, characterized in that The step of merging each group of data instances according to the attribute parameters of the corresponding sub-data structures to obtain the target aggregate instance includes: In the case where the sub-data structure corresponds to the field-level metadata, the following operation is performed on each group of the data instances, wherein a group of the data instances for which the following operation is performed each time is regarded as a second group of data instances: Semantically deduplicate the field-level description attribute parameters and the field-level data item attribute parameters of each data instance in the second group of data instances, and set the deduplicated field-level description attribute parameters to be filled in according to the second language business text to obtain target field-level description attribute parameters; Setting the field-level identification attribute parameter of each data instance in the second group of data instances to be filled according to the first language business text to obtain a target field-level identification attribute parameter, wherein the first language business text and the second language business text use different languages; Filling the field-level data type attribute parameter of each data instance in the second group of data instances with the original value to obtain the target field-level data type attribute parameter; Filling the field-level size attribute parameter of each data instance in the second group of data instances according to the maximum value to obtain a target field-level size attribute parameter; The target aggregate instance is generated based on the target field-level description attribute parameter, the target field-level identification attribute parameter, the target field-level data type attribute parameter, and the target field-level size attribute parameter.

7. The method according to claim 4, characterized in that The step of merging each group of data instances according to the attribute parameters of the corresponding sub-data structures to obtain the target aggregate instance includes: In the case where the sub-data structure corresponds to the dictionary-level metadata, the following operation is performed on each group of the data instances, wherein a group of the data instances for which the following operation is performed each time is regarded as the third group of data instances: Performing semantic deduplication on the dictionary-level value parameter of each data instance in the third group of data instances to obtain a target dictionary-level value attribute parameter; Fill the dictionary-level description attribute parameters and dictionary-level data type attribute parameters of each data instance in the third group of data instances according to the original values ​​to obtain target dictionary-level description attribute parameters and target dictionary-level data type attribute parameters; generate the target aggregate instance based on the target dictionary-level value attribute parameters, the target dictionary-level description attribute parameters and the target dictionary-level data type attribute parameters.

8. A metadata generation device, characterized in that: include: An acquisition module, used to acquire a pre-created meta-model and full basic information extracted from at least one data source, wherein the meta-model includes at least two sub-data structures, and the full basic information includes at least two granularities, and each sub-data structure has a granularity corresponding thereto; An encapsulation module is used to use the meta-model to encapsulate the full basic information to obtain a full data instance, wherein the full data instance represents the full basic information stored according to the rules indicated by the meta-model; a clustering module is used to cluster and merge the full data instance through a retrieval enhanced generation model based on a preset clustering merging method to generate target metadata, wherein the retrieval enhanced generation model is used to cluster the full data instance according to the preset clustering merging method, and the preset clustering merging method includes grouping according to the description information of the full data instance.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images