Metadata management system and method for integrating large models
Through the integrated metadata management system of large models, the process of metadata generation and updates is automatically processed, and the problems of insufficient metadata perfection and long update time caused by relying on manual operations in traditional metadata management methods are solved, achieving efficient and low-cost metadata Data management.
Patent Information
- Application Number
- CN202411096512.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-12
AI Technical Summary
Traditional metadata management methods rely on cumbersome manual operations, resulting in insufficient metadata improvement, high labor costs, and long update time, which affects data operation efficiency and decision-making quality.
Using a metadata management system that integrates large models, including information indexing module, data processing unit, model scheduling module and large models, through artificial intelligence technology and large model technology, metadata generation and update are automatically processed, reducing manual input errors, and achieving low Real-time updates at cost, high quality and high efficiency.
It improves the integrity and usage efficiency of metadata, reduces the cost of metadata management, improves the intelligence level of enterprise data governance, and achieves low-cost, high-quality and high-efficiency real-time updates.
Smart Images

Figure CN118643071B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of metadata management technology, and in particular to a metadata management system and method for integrating a large model. Background Art
[0002] Traditional metadata management methods often rely on cumbersome manual operations to input, update and maintain data. This process is not only time-consuming and labor-intensive, but also prone to errors, which in turn lead to data interpretation errors and misoperation.
[0003] In modern enterprise data management, traditional metadata systems face multiple challenges. First, the metadata of these systems is generally not complete enough, making it difficult for business personnel to understand and effectively use data, affecting data operational efficiency and decision-making quality. In addition, the metadata update and accuracy maintenance of enterprise-level data requires extensive expertise and relies on cross-departmental collaboration, resulting in high communication costs, especially when relying on third-party experts. Currently, there are three main strategies for metadata maintenance: post-supplementation, pre-input, and automation-driven. Although the post-supplementation strategy has little interference with the production process, it has high labor costs and is difficult to sustain; the pre-input strategy can record metadata from the beginning of the project, but it may affect the efficiency of the business process; although automation-driven, such as code analysis, is highly idealistic, it is still technically challenging to achieve this goal accurately and comprehensively.
[0004] Therefore, a new solution for automated metadata management that integrates generative large models is needed. Summary of the invention
[0005] In view of this, the embodiments of this specification provide a metadata management system and method for integrating a large model.
[0006] The embodiments of this specification provide the following technical solutions:
[0007] The embodiment of this specification provides a metadata management system integrating a large model, including:
[0008] Information indexing module, data processing unit, model scheduling module and large model;
[0009] The data processing unit is used to receive the industry source data transmitted by the large model, generate a structured metadata set after processing the data, and process the metadata into a table that is easy to analyze;
[0010] A model scheduling module, configured to receive a question, convert the question into a preset vector form, and then execute a query mechanism to the information index module so as to extract key information from the information index module;
[0011] After acquiring the key information from the information index module, the key information is analyzed on the data warehouse knowledge base by using the model scheduling, a prompt word template is generated, and an API interface of the large model service is scheduled to generate metadata information;
[0012] The large model is used to generate metadata information by understanding the intent and parsing the content of the prompt word template, and return the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers;
[0013] Among them, the big model sets the performance standard parameters of the initial big model according to the new business needs; and fixes the weights of the performance standard parameters in the initial big model, obtains the knowledge and representation capabilities in the initial big model, fine-tunes the initial big model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target big model of the preset domain, and provides accurate metadata for the data processing unit to perform interactive answers.
[0014] The embodiment of this specification also provides a metadata management method for an integrated large model, including:
[0015] The data processing unit receives the industry source data transmitted by the large model, generates a structured metadata set after processing the data, and processes the metadata into a table that is easy to analyze;
[0016] The model scheduling module receives the question, converts the question into a preset vector form, and then executes a query mechanism to the information index module to extract key information from the information index module;
[0017] After obtaining the key information from the information indexing module, the model scheduling module uses the model scheduling to analyze the key information on the data warehouse knowledge base, generates a prompt word template, and schedules the API interface of the large model service to generate metadata information;
[0018] The large model generates metadata information by understanding the intent and parsing the content of the prompt word template, and returns the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers;
[0019] Among them, the big model sets the performance standard parameters of the initial big model according to the new business needs; and fixes the weights of the performance standard parameters in the initial big model, obtains the knowledge and representation capabilities in the initial big model, fine-tunes the initial big model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target big model of the preset domain, and provides accurate metadata for the data processing unit to perform interactive answers.
[0020] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:
[0021] This application introduces artificial intelligence technology and big model technology, which can generate and maintain high-quality metadata at low cost, thereby improving the integrity and efficiency of metadata. The application of big models can not only reduce manual input errors, but also update and calibrate metadata in real time through algorithm optimization, greatly improving the practicality of metadata and the intelligent level of enterprise data governance, and realizing a low-cost, high-quality, and efficient real-time updated metadata management method; improving the quality of metadata and the response speed of the system, and greatly reducing the cost of metadata management. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 It is the architecture diagram of the metadata domain big model service provided by this application;
[0024] Figure 2 It is a flow chart of the automated data management method for the integrated large model provided by the present application;
[0025] Figure 3 is a schematic diagram of the automated metadata management provided by this application;
[0026] Figure 4 It is a flow chart of the management method of the integrated large model provided in this application. DETAILED DESCRIPTION
[0027] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0028] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.
[0029] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.
[0030] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0031] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.
[0032] In modern enterprise data management, traditional metadata systems face multiple challenges. First, the metadata of these systems is generally not complete enough, making it difficult for business personnel to understand and effectively use data, affecting data operational efficiency and decision-making quality. In addition, the metadata update and accuracy maintenance of enterprise-level data requires extensive expertise and relies on cross-departmental collaboration, resulting in high communication costs, especially when relying on third-party experts. Currently, there are three main strategies for metadata maintenance: post-supplementation, pre-input, and automation-driven. Although the post-supplementation strategy has little interference with the production process, it has high labor costs and is difficult to sustain; the pre-input strategy can record metadata from the beginning of the project, but it may affect the efficiency of the business process; although automation-driven, such as code analysis, is highly idealistic, it is still technically challenging to achieve this goal accurately and comprehensively.
[0033] Based on this, the embodiments of this specification propose a new processing solution: to solve the problems of high labor costs, low metadata quality and utilization, and long update time in the prior art, manual operations are converted into automated processing, artificial intelligence technology and big model technology are introduced, and the open source big model is adaptively improved to integrate preset domain knowledge, and then supervised learning is performed or reinforcement learning is used to achieve autonomous learning of metadata management, that is, adaptive business needs determine the target model that is suitable for the business industry field, and the target model autonomously adapts to metadata input, update, maintenance and autonomous learning reasoning processes in the metadata field service process.
[0034] The application of big models can not only reduce manual input errors, but also update and calibrate metadata in real time through algorithm optimization. It can generate and maintain high-quality metadata at low cost, improve the integrity and efficiency of metadata. At the same time, it improves the quality of metadata and the response speed of the system, greatly reducing the cost of metadata management. It improves the practicality of metadata and the intelligent level of enterprise data governance. It realizes a low-cost, high-quality, and efficient real-time updated metadata management method.
[0035] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.
[0036] like Figure 1 and Figure 2 As shown, in order to solve the problems that the metadata management method in the prior art uses manual participation and relies on cumbersome manual operations to input, update and maintain data, resulting in insufficient metadata perfection, affecting the efficiency of data operations and decision-making quality, etc. This application provides an automated metadata management system and method that integrates a large model.
[0037] like Figure 1 As shown, an autonomous learning big model is introduced to set up a metadata management system that integrates the big model, including an information indexing module, a data processing unit, a model scheduling module and a big model.
[0038] The data processing unit is used to receive the industry source data transmitted by the large model, generate a structured metadata set after processing the data, and process the metadata into a table that is easy to analyze;
[0039] The model scheduling module is used to receive questions, convert the questions into a preset vector form, and then execute a query mechanism to the information index module to extract key information from the information index module; wherein, the model scheduling module receives questions raised by users, such as where a specific business data exists, etc. The model scheduling module receives the questions raised by the user.
[0040] After acquiring the key information from the information index module, the key information is analyzed on the data warehouse knowledge base by using the model scheduling, a prompt word template is generated, and an API interface of the large model service is scheduled to generate metadata information;
[0041] The large model is used to generate metadata information by understanding the intent and parsing the content of the prompt word template, and return the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers;
[0042] Among them, the big model sets the performance standard parameters of the initial big model according to the new business needs; and fixes the weights of the performance standard parameters in the initial big model, obtains the knowledge and representation capabilities in the initial big model, fine-tunes the initial big model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target big model of the preset domain, and provides accurate metadata for the data processing unit to perform interactive answers.
[0043] In some embodiments, the large model is also used to extract and transform historical data and change logs to obtain problems corresponding to business needs; and to define performance standard parameters for evaluating the large model according to the business needs, assign a weight to each performance standard parameter, and finally select an initial model according to key hyperparameters;
[0044] Among them, the performance standard parameters include at least one of the following: Chinese text processing, cultural industry metadata domain knowledge understanding, logical analysis, content creation and response capabilities to commands.
[0045] In some embodiments, the large model is also used to determine the weights of the performance standard parameters in the large model by applying the LORA algorithm based on the key hyperparameters, so as to fix the weights of the known performance standard parameters without participating in training, and maintain the knowledge and representation ability of the current model;
[0046] By adding new neural network layers and using preset domain-related data sets to supervise the new neural network layers, or by using reinforcement learning to allow the current model to learn autonomously according to manual instructions, the target large model is finally generated;
[0047] Among them, key hyperparameters include but are not limited to learning rate, batch size, number of fine-tuning layers, regularization or optimizer, etc.
[0048] In some embodiments, the new neural network layer includes a fully connected layer, a convolutional layer, and a recurrent neural network, which are used to process data in a preset field and recognize manual instructions.
[0049] In some embodiments, the mutual communication and data transmission between the data processing unit, the model scheduling module and the information indexing module are carried out in the form of an API interface or an RPC remote call.
[0050] In some embodiments, the reasoning process of the integrated large model metadata management system is encapsulated as an API interface using the Flask Web framework and Docker container deployment and deployed to the server for operation. Figure 3As shown in the figure, business personnel set the requirements for scanning scope and rules, and the system automatically initiates metadata scanning on a periodic basis to discover new metadata, so that the business system obtains metadata change information. During the scanning process, the metadata processing big model service is automatically called to generate relevant metadata information. The system automatically reviews the metadata information according to standards and specifications and publishes it. Finally, metadata users and data management experts continuously provide feedback on metadata accuracy to optimize the metadata big model capabilities.
[0051] like Figure 1 As shown, the information index module includes vector index, system metadata, label annotation and data dictionary; the vector index is used to quickly retrieve data; the data dictionary is used to store keywords of data; the system metadata represents descriptive information about the data; the label annotation is used to label the data for classification.
[0052] In combination with the above embodiments, the embodiments of this specification also provide a metadata management method for an integrated large model, such as Figure 4 As shown, it includes steps S401 to S404. Step S401: The data processing unit receives the industry source data transmitted by the large model, generates a structured metadata set after processing the data, and processes the metadata into a table that is easy to analyze.
[0053] Step S402: The model scheduling module receives the question, converts the question into a preset vector form, and then executes a query mechanism to the information index module to extract key information from the information index module.
[0054] Step S403: After obtaining the key information from the information indexing module, the model scheduling module uses the model scheduling to analyze the key information on the data warehouse knowledge base, generates a prompt word template, and schedules the API interface of the large model service to generate metadata information.
[0055] Step S404: The large model generates metadata information by understanding the intent and parsing the content of the prompt word template, and returns the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers;
[0056] Among them, the big model sets the performance standard parameters of the initial big model according to the new business needs; and fixes the weights of the performance standard parameters in the initial big model, obtains the knowledge and representation capabilities in the initial big model, fine-tunes the initial big model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target big model of the preset domain, that is, the big model adaptively determines the target model suitable for the business industry field according to the business needs, and provides accurate metadata for the data processing unit to perform interactive answers.
[0057] like Figure 2 and Figure 4As shown in the figure, the metadata domain big model service process includes:
[0058] 1. Automated metadata processing requirements analysis and model evaluation: The business requirements for metadata processing are often extracted and converted from historical data and change logs in the business system. Based on this requirement, the standards for evaluating the performance of the large model are defined. These standards include but are not limited to performance standard parameters such as Chinese text processing, cultural industry metadata domain knowledge understanding, logical analysis, content creation, and responsiveness to commands, and these performance standard parameters are given different weights, such as 25% weight for Chinese text processing, 20% weight for metadata domain knowledge understanding, 20% weight for logical analysis, 25% weight for content creation, and 10% weight for responsiveness to commands. Finally, data management experts score according to the output results to select the optimal model. That is, the initial model is finally selected based on the key hyperparameters. According to the optimal result of model evaluation, the ChatGLM open source large model is selected.
[0059] In some embodiments, after the initial model is selected, the metadata framework is organized and a keyword template is formulated to construct an instruction template data set for metadata processing, so as to fine-tune the initial model to meet the preset domain knowledge and format requirements of the metadata, wherein the instruction template data set includes a business description of a single entity, an explanation of the relationship between entities, and data lineage. The business description of a single entity refers to a detailed description of the data entity, such as business meaning, purpose, relationship from other data entities, and role in the business process. The relationship between entities describes the relationship between data entities (such as tables, fields, data items, etc. in a database). Data lineage describes the origin of the data, its flow path, and its transformation process in the system, and the entire life cycle of key key data from the source to the destination.
[0060] 2. Build a metadata processing instruction template data set: Create standardized questions and answer sets for large models to fine-tune the ChatGLM generative large model to meet the specific domain knowledge and format requirements of enterprise metadata. It mainly includes building a data set containing more than 3,000 standardized questions and answers, organizing metadata frameworks, and formulating keyword templates. The design of the industry metadata instruction set includes the business description of a single entity, the relationship between entities, and the elaboration of data lineage, etc., to ensure that the model can understand and accurately execute enterprise-level data processing requirements.
[0061] 3. Fine-tune the parameters of the generative large model to the cultural metadata domain model: 1) Fine-tuning process: Apply the LORA algorithm to fine-tune the key hyperparameters such as learning rate, batch size, number of fine-tuning layers, regularization, and optimizer of the ChatGLM open source large model. The key to this step is to fix the main weight parameters of the large model. For the existing main weight parameters, they are fixed and do not participate in the training process, so that the knowledge and representation ability of the original model can be maintained. At the same time, some new neural network layers are added to the top of the model. These layers can be fully connected layers, convolutional layers, recurrent neural networks, etc., which are used to process data in specific fields and recognize manual instructions, so as to integrate knowledge in specific fields and train models to obey manual instructions. Then train the newly added neural network layers to adapt them to data and tasks in specific fields. These layers can be supervised by using domain-related data sets, or the model can be allowed to learn autonomously according to manual instructions through methods such as reinforcement learning. 2) Accuracy verification: The accuracy of the fine-tuned model is verified by randomly sampling the selected data tables in each field and inviting data management experts to review. The verification results show that the accuracy of the newly generated metadata is close to 100%, which proves that the output of the large model in this application scenario meets the preset standards. In some embodiments, the domain-related database includes a data set with high industry relevance, such as the medical beauty industry, bicycle manufacturing field, etc., and supervised learning is performed using the data set related to the field.
[0062] 4. Reasoning metadata domain model service: This service integrates the data processing unit, model scheduling module and information indexing module.
[0063] In some embodiments, the data processing unit performs data cleaning and aggregation operations, uses regular expression matching and denoising algorithms to process data, removes HTML tags, illegal data, and special data to generate a cleaned data set; uses Kafka or Flink to process the metadata in the cleaned data set into dimension tables, fact tables, and large wide tables that are easy to analyze.
[0064] After receiving the industry source data, the data processing unit first performs data cleaning and aggregation operations, using regular expression matching, denoising algorithms and other technologies to remove HTML tags, illegal data, special data, etc. to generate a cleaned data set, using data standardization algorithms to generate structured metadata data sets, and using big data processing technologies such as Kafka and Flink to process metadata into dimension tables, fact tables, and large wide tables that are easy to analyze. The model scheduler can use the Bert model to convert the questions raised by users into a specific vector form and execute the query mechanism. At the same time, it extracts key information from the information indexer, including vector indexes (quick retrieval of data), data dictionaries (keywords for storing data), system metadata (descriptive information about data), and label annotations (labeling data for classification). The service further uses model scheduling to analyze the data warehouse knowledge base, generate prompt word templates, and then schedule the API interface of the large model service to generate metadata information. Finally, the data processing unit is responsible for executing multiple rounds of interactive questions and answers, combining the metadata information generated by the large model service, and comprehensively processing the question and answer results to achieve automatic generation of metadata and intelligent filling of data gaps.
[0065] Each module and service can interact with each other through API interfaces or RPC remote calls to achieve more efficient and accurate completion of metadata, significantly shortening the time required to fill in information. The average time can be shortened from days to seconds, and the information accuracy rate can reach more than 90%.
[0066] 5. Integrate into the existing metadata management process: Use the Flask Web framework, Docker containerization and other technologies to encapsulate the reasoning process as an API interface and deploy it to the server to run, replacing the manual metadata entry process with an automated method, and retaining a quick manual review step to correct any errors that may be caused by the large model, such as accuracy issues based on pinyin reasoning. Manually retaining error information provides feedback for subsequent optimization, which can continuously improve the capabilities of the large model.
[0067] like Figure 2 and Figure 3 As shown, an automated metadata processing method for an integrated generative large model is provided, which converts manual maintenance into automated processing, and is used to solve the problems of high labor costs, low metadata quality and utilization, and long update time in the prior art, and to achieve a low-cost, high-quality, and efficient real-time updated metadata management method.
[0068] The same or similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiment described later, since it corresponds to the method, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.
[0069] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A metadata management system for integrating large models, characterized in that: include: Information indexing module, data processing unit, model scheduling module and large model; The data processing unit is used to receive industry source data transmitted by the large model, generate structured metadata sets after data processing, and process the metadata into tables that are easy to analyze; A model scheduling module is used to receive questions, convert the questions into a preset vector form, and then execute a query mechanism to the information indexing module to extract key information from the information indexing module; After obtaining the key information from the information index module, the key information is analyzed on the data warehouse knowledge base using model scheduling to generate a prompt word template, and the API interface of the large model service is scheduled to generate metadata information; The large model is used to generate metadata information by understanding the intent and parsing the content of the prompt word template, and return the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers; The large model sets performance standard parameters of the initial large model according to the new business requirements; and fixes the weights of the performance standard parameters in the initial large model, obtains the knowledge and representation capabilities of the initial large model, fine-tunes the initial large model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target large model for the preset business industry domain, and provides accurate metadata for the data processing unit to perform interactive answers; The large model is also used to extract and transform historical data and change logs in the business system to obtain business requirements for metadata processing; and based on the business requirements, the performance standard parameters of the large model are evaluated, each performance standard parameter is assigned a weighted proportion, and finally an initial model is selected based on key hyperparameters; The large model is also used to determine the weights of the performance standard parameters in the large model by applying the LORA algorithm based on key hyperparameters, so as to fix the weights of the known performance standard parameters without participating in training, thus maintaining the knowledge and representation capabilities of the current model; By adding new neural network layers and using preset domain-related data sets to supervise the learning of the new neural network layers, or through reinforcement learning, allowing the current model to autonomously learn metadata management according to manual instructions, the target large model is finally generated. The target large model adapts to the metadata input, update, maintenance and autonomous learning reasoning process during the metadata field service process.
2. The metadata management system for integrating large models according to claim 1, characterized in that: in, Performance standard parameters include at least one of the following: Chinese text processing, cultural industry metadata domain knowledge understanding, logical analysis, content creation and responsiveness to commands.
3. The metadata management system for integrating large models according to claim 2, characterized in that: in, Key hyperparameters include at least one of the following: learning rate, batch size, number of fine-tuning layers, regularization, and optimizer.
4. The metadata management system for integrating large models according to claim 3, characterized in that: The new neural network layer includes a fully connected layer, a convolutional layer, and a recurrent neural network, which is used to process data in a preset field and recognize manual instructions.
5. The metadata management system for integrating large models according to claim 1, characterized in that: The communication and data transmission between the data processing unit, model scheduling module and information index module are carried out by API interface or RPC remote call.
6. The metadata management system for integrating large models according to claim 1, characterized in that: Use the Flask Web framework and Docker container deployment to encapsulate the reasoning process of the integrated large model metadata management system as an API interface and deploy it to the server for operation.
7. The metadata management system for integrating large models according to claim 1, characterized in that: The information index module includes vector index, system metadata, label annotation and data dictionary; The vector index is used to quickly retrieve data; the data dictionary is used to store keywords of data; the system metadata represents descriptive information about the data; and the label annotation is used to label and classify the data.
8. A metadata management method for an integrated large model, characterized in that: include: The data processing unit receives the industry source data transmitted by the large model, generates a structured metadata set after processing the data, and processes the metadata into tables that are easy to analyze; The model scheduling module receives the question, converts the question into a preset vector form, and then executes a query mechanism to the information index module to extract key information from the information index module; After obtaining the key information from the information indexing module, the model scheduling module analyzes the key information on the data warehouse knowledge base using the model scheduling, generates a prompt word template, and schedules the API interface of the large model service to generate metadata information; The large model generates metadata information by understanding the intent and parsing the content of the prompt word template, and returns the generated metadata information to the data processing unit so that the data processing unit can perform multiple rounds of interactive answers; The large model sets performance standard parameters of the initial large model according to the new business requirements; and fixes the weights of the performance standard parameters in the initial large model, obtains the knowledge and representation capabilities of the initial large model, fine-tunes the initial large model to integrate the preset domain knowledge and trains it to obey manual instructions to generate a target large model for the preset business industry domain, and provides accurate metadata for the data processing unit to perform interactive answers; The large model is also used to extract and transform historical data and change logs in the business system to obtain business requirements for metadata processing; and based on the business requirements, the performance standard parameters of the large model are evaluated, each performance standard parameter is assigned a weighted proportion, and finally an initial model is selected based on key hyperparameters; The large model is also used to determine the weights of the performance standard parameters in the large model by applying the LORA algorithm based on key hyperparameters, so as to fix the weights of the known performance standard parameters without participating in training, thus maintaining the knowledge and representation capabilities of the current model; By adding new neural network layers and using preset domain-related data sets to supervise the learning of the new neural network layers, or through reinforcement learning, allowing the current model to autonomously learn metadata management according to manual instructions, the target large model is finally generated. The target large model adapts to the metadata input, update, maintenance and autonomous learning reasoning process during the metadata field service process.
9. The metadata management method for an integrated large model according to claim 8, characterized in that: After selecting the initial model, we organize the metadata framework, formulate keyword templates, and construct a metadata processing instruction template dataset to fine-tune the initial model to meet the metadata preset domain knowledge and format requirements; The instruction template data set includes a business description of a single entity, an explanation of the relationship between entities and data lineage.
10. The metadata management method of the integrated large model according to claim 8, characterized in that: The data processing unit performs data cleaning and aggregation operations, processes the data using regular expression matching and denoising algorithms, removes HTML tags, illegal data, and special data to generate a cleaned data set; Use Kafka or Flink to process the metadata in the cleaned data set into dimension tables, fact tables, and large wide tables that are easy to analyze.
