An automatic generation and management system for data assets based on data operation technology

Abstract

The present invention discloses an automatic generation and management system of data assets based on data operation technology, including: 1) completing the construction of data flow based on metadata to form a logical data resource set oriented to application scenarios, and performing quality inspection on logical data resources by configuring quality inspection rules; 2) determining the type and classification and grading strategy of logical data resources in combination with the format information and sampling method of metadata, and performing final data classification and grading identification; 3) for logical data resources that have been aggregated and quality inspected and classified and graded, configuring specific product information on their data query interfaces to construct data products with asset value; 4) performing value assessment on data products through cost assessment, benefit assessment, and transaction assessment, and performing asset configuration in units of data resources; 5) regularly packaging and generating data resources, and performing global statistics and management on currently effective data assets.

Description

Technical Field

[0001] The present invention relates to the field of digital economy, and in particular to an automatic generation and management system of data assets based on data operation technology. Background Art

[0002] Data assetization refers to the process of gradually transforming data from raw data to data assets. From the perspective of the formation process of data assets, data assetization is a value creation activity around data, including data collection, processing, governance, development, and trading. The ultimate goal is to promote the transformation of raw data into data assets and stimulate and release the value potential of data.

[0003] The existing data assetization mainly adopts the vertical field data value assessment and assetization method, mainly for vertical fields such as medical, consumption, and finance, and focuses on the management, valuation, and circulation of data assets. Through demand research and expert review, a general standard for data asset classification and valuation methods that meet the needs of this vertical field is formulated, and corresponding software platform facilities are provided to provide unified management and circulation functions of data assets for relevant entities. Such software platform facilities usually use blockchain or other means of confirmation of rights. Enterprises, organizations, and individuals in the industry can sort out their own data according to this standard and upload it to the software platform facilities, relying on the platform for subsequent circulation transactions of data assets.

[0004] The existing vertical data value assessment and assetization methods have failed to completely solve the difficulties and pain points of data element assetization. The existing problems are as follows:

[0005] 1) Lack of versatility. By refining and summarizing common situations in the field, this method can ensure that it can meet most common scenarios, but it is difficult to be compatible with edge cases. For enterprises and organizations with special business needs or belonging to different subcategories, their data types, data item composition, and feature sampling may have irreconcilable deviations from general standards and cannot be fully applied to unified standards;

[0006] 2) Data preparation is difficult. Existing methods usually rely on manual or data-based aggregation and processing to form standard data assets for uploading to the software platform. This process not only relies heavily on data processing capabilities and facilities, but also requires a lot of time and manpower. Relevant personnel need to regularly sort out newly generated data resources and upload them. In addition, due to the existence of a large amount of tribal knowledge and data islands, it is difficult for enterprise managers or employees to ensure the comprehensiveness and real-time nature of the data.

[0007] 3) Difficulty in expansion and optimization. With the development of the industry, technological progress and the continuous development of the value of data elements, relying on individuals or groups to research and formulate unified standards for data assets in various vertical fields not only requires a lot of costs, but also requires alignment of the level of understanding of the updated content of the standards by all participants. Otherwise, it is easy to cause deviations in the content and value assessment of data assets due to misunderstandings.

[0008] Therefore, it is necessary to improve the above-mentioned prior art to overcome the above-mentioned defects. Summary of the invention

[0009] The purpose of the present invention is to provide an automatic generation and management system for data assets based on data operation technology. The system can be used to complete the full-chain evolution and management from raw data to data assets, and reduce the time, manpower, and capital investment required for data assetization through the establishment and scheduling of processes based on artificial intelligence. In addition, the vertical domain large model and industry mechanism knowledge graph constructed in the present invention can also solve the problem of efficient and accurate discovery, construction and management of data assets when enterprises lack knowledge related to data assetization, and then help enterprises provide full-link automated operation assistance and decision-making suggestions. Through a large model that can be fine-tuned autonomously and an intelligent and complete knowledge graph, it is ensured that the relevant knowledge and mechanisms can be continuously optimized and improved with user use, properly solving edge cases and adapting to environmental changes.

[0010] The above technical objectives of the present invention are achieved through the following technical solutions:

[0011] A data asset automatic generation and management system based on data operation technology, including the following functional modules:

[0012] 1) Data flow orchestration and quality control: According to data requirements and application scenarios, data flows are built based on metadata to form a logical data resource set for application scenarios, and quality inspection of logical data resources is performed by configuring quality inspection rules;

[0013] 2) Data classification and grading identification: Determine the type and classification and grading strategy of logical data resources based on the format information and sampling method of metadata; select the scope of the data list to be followed according to the industry and region in which the data is located, and conduct the final data classification and grading identification;

[0014] 3) Circulation and trading: For logical data resources that have been aggregated, quality-checked, and classified and graded, data products with asset value are constructed by configuring specific product information on their data query interfaces;

[0015] 4) Value assessment: Data assets are valued through cost assessment, benefit assessment, and transaction assessment, and asset allocation is performed based on data resources;

[0016] 5) Asset generation and table entry: Automatically and regularly package and generate data resources, and perform global statistics and management of currently effective data assets.

[0017] Furthermore, the method of data stream arrangement and quality control is as follows:

[0018] 1.1) Metadata ingestion: By configuring the protocol type, connection location and authentication information, adding user files to be managed, database data, information system data, and real-time information of IoT edge devices, the system automatically connects and scans all readable content in the target location, and records the corresponding storage location and explicit features for each readable content;

[0019] 1.2) Semantic Information Capture: Use a large model that has been fine-tuned for tasks to perform semantic analysis on the currently collected content. Task fine-tuning uses transfer learning methods to perform supervised learning adjustments based on large-scale general corpus and data types and semantic requirements.

[0020] 1.3) Intelligently assisted drag-and-drop data flow construction: When users build data flows, multi-level intelligent recommendations and interactive construction functions are used to help users quickly complete the construction of data flows;

[0021] 1.4) Data resource construction and quality management: After the data flow is constructed, the system will generate a logical data resource collection based on the configuration and connection status of the data flow and present it in the user interface.

[0022] Furthermore, the method for data classification and grading identification is as follows:

[0023] 2.1) Determine the corresponding classification and grading identification strategy and objects according to the format of logical data resources;

[0024] 2.2) After clarifying the classification and grading identification strategy and objects, select the scope of the data inventory that needs to be followed;

[0025] 2.3) Perform classification and grading recognition through a large vertical domain model agent that has been fine-tuned for specific tasks.

[0026] Furthermore, the circulation transaction method is as follows:

[0027] 3.1) For data resources that have completed quality inspection rules and classification and grading identification configuration, by configuring their data query interface with specific product information, build data products with asset value;

[0028] 3.2) Use real-time recommendation algorithms to recommend the most suitable data products to users based on their historical behaviors, preferences, and current needs;

[0029] 3.3) Collect the description information of all data products, match and search for data products, and after determining the data products, conduct transactions using offline business processes or online transactions;

[0030] 3.4) Record the transaction information, including the time of transaction, information of both parties to the transaction, transaction type information, and sampling and storage of query results, to meet the storage of data transaction process and ensure subsequent traceability.

[0031] Furthermore, the value assessment method is as follows:

[0032] 4.1) Ownership confirmation: Users first need to confirm and declare the ownership of the data assets they are going to generate.

[0033] 4.2) Cost aggregation: The system allows users to configure the cost of each step of data resource from generation to maintenance in detail.

[0034] 4.3) Revenue measurement: Through real-time statistics and analysis functions, comprehensively evaluate the value contribution of data resources to corporate operations and overall business.

[0035] 4.4) Transaction Statistics: The system can comprehensively record the transaction status of statistical resources on the platform, including historical transaction frequency, transaction price and future transaction expectations.

[0036] Furthermore, the method of generating and entering assets into the table is as follows:

[0037] 5.1) Asset allocation: According to your own needs, the data resources in the system are assetized;

[0038] 5.2) Automatically package assets to be reviewed: extract new content within the cycle from the original data and package it into static, unchangeable data assets to be reviewed;

[0039] 5.3) Data asset audit: Use the random forest algorithm based on Bootstrap sampling to analyze the characteristics of historical audit records and data assets, automatically perform preliminary classification and scoring of data assets to be audited, and provide audit suggestions to users;

[0040] 5.4) Unified management of data assets: The generated data assets can be managed uniformly by the system as valuable intangible assets. Users can view the current data asset list, total value and amortization status.

[0041] Furthermore, the task fine-tuning method of the large model in step 1.2) is as follows:

[0042] a) Data preparation: Prepare domain data related to the expected task through manual annotation, including thousands of data table structures and their corresponding semantic information as the initial data for the fine-tuning task;

[0043] b) Model fine-tuning: Based on the pre-trained model of the open source large model, the domain data is used for fine-tuning. By automatically dividing the training set and the validation set, the cross-entropy loss function is used to optimize the semantic understanding performance of the model on the specified training sentence. For a single sample, the cross-entropy loss is defined as shown in formula (1):

[0044] (1)

[0045] For the cross entropy loss of the entire training set, the average value of all samples is usually taken and calculated using formula (2):

[0046] (2)

[0047] c) Model optimization: Through mixed precision training and gradient accumulation, while ensuring the accuracy of the model, the training efficiency is optimized and cross-compared with the validation set, and the effect is verified through a predefined loss function.

[0048] Furthermore, the intelligently assisted drag-and-drop data flow construction method in step 1.3) is as follows:

[0049] Based on the definition and description of the application scenario, the AP-GNN algorithm is used to assist users in quickly recommending metadata related to their goals. The AP-GNN algorithm first performs association rule mining to find all frequent item sets by gradually increasing the length of the item set. If the frequent item set meets the specified support threshold, it means that these metadata association patterns appear frequently in the historical data stream configuration.

[0050] The support calculation is shown in formula (3):

[0051] (3)

[0052] Generate association rules from frequent item sets, and evaluate the reliability of the rules by confidence. The confidence calculation formula is shown in formula (4):

[0053] (4)

[0054] After obtaining reliable metadata association patterns, a graph structure is used to represent the correspondence between these patterns and specific data sources. The nodes in the graph structure represent metadata and data sources, and the edges in the graph structure represent metadata association patterns. The weights on the edges are set as the confidence or support of the association rules. Then, the convolution operation of the graph neural network is used to gradually capture the local features and global structural features of the nodes.

[0055] The implementation of the convolution operation is shown in formula (5):

[0056] (5)

[0057] Through the above methods, users can locate the required data content in a transparent manner, and through an intuitive drag-and-drop interface, connect metadata, data operation steps and logical output data sets to build data flows on demand without having to worry about the data format, data storage location and data access method.

[0058] Furthermore, the method for performing quality inspection on the logical data resource by configuring quality inspection rules in step 1) is as follows:

[0059] a) Data consistency check: Use the hash check algorithm to convert the data block into a hash value of fixed length, and compare the hash values ​​to determine whether the data is consistent;

[0060] Assume the data block is , the hash function is , then the data block Hash value It can be calculated using formula (6):

[0061] (6)

[0062] b) Data integrity check: Check the integrity of data through foreign key constraints to avoid partial data loss or errors;

[0063] c) Data accuracy verification: Apply Bayesian network to evaluate the accuracy of data in real time and identify potential abnormal data. The implementation method is shown in formula (7);

[0064] (7)

[0065] d) Data timeliness verification: Use time series analysis and real-time stream processing technology to monitor the frequency of data updates to ensure that the real-time requirements of the business for data are met;

[0066] e) User-defined validation: By configuring data quality standards and detection rules by yourself, the ingested data can be validated in real time. Customizable rules can be used to verify whether the data meets the expected format, value range and statistical information to meet specific business needs.

[0067] Furthermore, the method of data classification and grading identification in step 2.3) is as follows:

[0068] a) Use large models to accurately classify and grade each data object;

[0069] b) For the classification and grading results generated by the large model, the CE-F1 function was used for classification optimization;

[0070] The CE-F1 function combines the functions of the cross entropy loss function and the classification evaluation indicator F1-score. The cross entropy loss function is used to measure the single error coefficient of the classification result. The F1-Score represents the quality weight of the recall result obtained by calculating the harmonic mean of the precision and recall rate. It is used to measure the performance of the classifier when processing an unbalanced data set.

[0071] The quality weight calculation is shown in formulas (8)-(10):

[0072] (8)

[0073] (9)

[0074] (10)

[0075] Furthermore, the cost aggregation method is as follows:

[0076] For each data resource, assume that it contains activity steps, and the cost of each step is , then the total cost can be calculated using formula (11): :

[0077] (11)

[0078] The cost of each activity step is It can be calculated using formula (12):

[0079] (12)

[0080] in, represents the resource cost associated with activity i; represents the activity driver, indicating the degree or amount of resources consumed by activity i;

[0081] The method of calculating the income is as follows:

[0082] The ARI-MLR algorithm is used to analyze and predict the usage of data resources in different time periods, thereby identifying peak and trough periods, as shown in formulas (13) and (14):

[0083] (13)

[0084] (14)

[0085] The transaction statistics method is as follows:

[0086] The Prophet model, a time series prediction algorithm, is used to predict future transaction trends, and combined with historical data transaction records, transaction evaluation results are generated, as shown in formula (15):

[0087] (15)

[0088] Furthermore, the method for automatically packaging assets to be reviewed is as follows:

[0089] The distributed data processing algorithm based on MapReduce is used to segment and aggregate the data. According to the characteristics of the data, each data block is assigned to a different computing node for processing. The implementation method is shown in formula (16):

[0090] (16)

[0091] The Reduce function formula (17) is shown as follows:

[0092] (17)

[0093] The method of data asset audit is as follows:

[0094] The random forest algorithm based on Bootstrap sampling analyzes the characteristics of historical audit records and data assets, automatically performs preliminary classification and scoring of the data assets to be audited, and provides audit suggestions; the splitting of the random forest algorithm based on Bootstrap sampling adopts the information gain method shown in formula (18):

[0095] (18)

[0096] Furthermore, it also includes a dynamic large model access and intelligent agent construction and management module, and its implementation steps are as follows:

[0097] Step 1: Encapsulate the model: Encapsulate the differences between the call parameters and return structures of different large model interfaces in the form of connectors, provide a unified usage method, and support switching between different large models on demand;

[0098] Step 2: Generate an agent: Generate an agent for specific tasks such as data flow construction, asset discovery, and graph fusion, or create a copy based on an existing agent for experimental development or further fine-tuning;

[0099] Step 3: Obtain the agent execution results: Other modules can pass in inputs that meet the expected format through the calling function and obtain the agent execution results; after obtaining the execution results, other modules can dynamically generate the original corpus required for fine-tuning the agent based on the user's feedback and modification of the execution results or other predefined rules, provide feedback through the agent feedback function, and locate the information at the time of execution based on the task ID, and store it as a fine-tuning corpus by combining the input to the large model during the call and the results in the feedback;

[0100] Step 4: Determine the fine-tuning requirements of the agent: The dynamic large model access and agent construction and management module can determine its fine-tuning frequency, tendency and other strategies through global configuration. The module will globally coordinate the fine-tuning requirements of each agent according to the configured time and frequency, and dynamically plan the fine-tuning execution time of each agent;

[0101] Step 5: Fine-tune the training agent: At the beginning of fine-tuning, the agent uses model reasoning to clean the data, eliminate errors, low-quality, and repeated content, solidify the processed high-quality fine-tuning corpus, convert the format of the fine-tuning corpus based on the fine-tuning interface parameter format requirements of the basic model, and finally use the fine-tuning function of the basic large model for fine-tuning training;

[0102] Step 6: Manage and reuse fine-tuning corpus: The stored and solidified high-quality fine-tuning corpus will be anonymized and only associated with the corresponding tasks and agents. When the module is migrated to the new basic large model as a whole, each agent will re-execute the fine-tuning task according to the historical fine-tuning records, so that the agents based on the new basic large model can receive necessary training to ensure the execution effect and quality.

[0103] Furthermore, it also includes a knowledge graph that can grow autonomously and intelligently, and the implementation steps are as follows:

[0104] Step 1: Construct the basic structure of the graph, including industries and specific links, raw data description information, data resource description information, and composition / inclusion relationships;

[0105] Step 2: Prepare the overview map content to ensure the basic user experience when there is insufficient accumulation of actual user usage data;

[0106] Step 3: Graph-based data assetization intention selection: You can choose whether to use the knowledge graph to make assetization suggestions and provide operation assistance, and at the same time allow the system to collect macro-descriptive information that does not involve the specific information carried by the data;

[0107] Step 4: Generate data assetization suggestions based on the graph: When the user chooses to use the relevant services and agrees to collect and share, the system will use the semantic description information of the current global data ingested during the data flow orchestration and quality control process, including the content of the accessed original data, to determine the data type and basic situation of the current user, and compare the scan results with the existing knowledge graph content in the system. The user can select the corresponding industry, business type and link to locate the Sub-Graph, and make suggestions for subsequent data operations and assetization based on the comparison results;

[0108] Step 4: Expand and optimize graph content based on user information: After the user builds the data resource, the relevant information will be anonymized and transmitted to the server. The system will update existing nodes and relationships or create new nodes and relationships based on the situation.

[0109] In summary, the present invention has the following beneficial effects:

[0110] The present invention significantly improves the automation and convenience of the entire process of data assetization through standardized products and auxiliary modules and functions, solves the shortcomings of traditional methods in terms of versatility, ease of use, and scalability, comprehensively reduces various costs of data assetization, and improves its output efficiency and subsequent reuse capabilities.

[0111] 1) The adaptability to edge scenarios is greatly improved. Through the knowledge graph that can autonomously collect user information and intelligently integrate and grow, the present invention can adapt to any edge scenario, be compatible with the special needs of different users and the actual situation, and support various needs through the continuous improvement and supplementation of the knowledge graph significantly better than traditional methods;

[0112] 2) The cost of generating data assets is significantly reduced. By combining the full life cycle management of data assets with data operation technology, the present invention significantly reduces the cost investment required for data sorting and inventory. By arranging and configuring data flows and asset generation, it ensures long-term reusability and reduces the investment required for enterprises to collect, process and maintain data resources in the long term. By operating global business data, the management of data assets is combined with the normal data usage process of the enterprise. The discovery and governance of all data resources can be completed in the normal business operations, ensuring the timeliness and comprehensiveness of the data.

[0113] 3) It has good scalability and freedom. Based on a large-model intelligent agent that autonomously learns user operation patterns and rules, the present invention can adapt to the data asset construction model under low-probability scenarios, and combined with the continuous supplementation and optimization of the knowledge graph that integrates autonomous intelligence and growth, it avoids users' dependence on experts and teams, and transforms the traditional model that relies on manual labor into a system model that can automatically and intelligently optimize and grow, thereby ensuring that the system has the ability to optimize with the development of the industry, technological progress, and the continuous development of the value of data elements. In addition, the intelligent agent that can be automatically fine-tuned and the knowledge graph that can be autonomously grown and optimized enable the system to have long-term learning capabilities. By supporting hot-plugging and retaining the characteristics of fine-tuning effects, it can significantly reduce the investment required for subsequent system upgrades and optimizations while adapting to the development of related AI technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 It is a schematic diagram of the data asset automatic generation and management system described in the present invention.

[0115] Figure 2 It is a schematic diagram of the data assetization system based on data operation technology described in the present invention.

[0116] Figure 3 This is the module diagram of the dynamic large model access, intelligent body construction and management described in the present invention.

[0117] Figure 4 It is the knowledge graph module diagram that can grow autonomously and intelligently through integration as described in the present invention. DETAILED DESCRIPTION

[0118] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with illustrations and specific embodiments.

[0119] See also Figure 1 and Figure 2 The present invention proposes an automatic data asset generation and management system based on data operation technology, which includes five steps: data flow orchestration and quality control, data classification and grading identification, circulation transaction, value assessment, asset generation and entry into the table.

[0120] Step 1: Data flow orchestration and quality control. Connect the user's multi-source heterogeneous data, and ingest metadata containing basic information and semantic information based on artificial intelligence; after the ingestion is completed, the user can build the data flow based on their own data needs and application scenarios through the code-free data flow with AI-assisted capabilities to build a drag-and-drop page; the completed data flow can form a logical data resource collection for the application scenario, and at the same time confirm its source location, operation process and output format, so as to achieve aggregation and governance that can be continuously reused for new data. In addition, the system supports configuring quality inspection rules for the formed data resources to ensure that the quality of the data always meets the requirements and expectations. The steps to achieve data flow orchestration and quality control are as follows:

[0121] Metadata ingestion: Users can add user files, database data, information system data, real-time information of IoT edge devices, etc. to be managed by configuring the protocol type, connection location, authentication information, etc. The system will automatically connect and scan all readable content in the location, and generate corresponding storage locations and explicit features for each content (such as path names, table / file names, table fields, text titles, etc., which can be obtained without natural language processing and semantic understanding).

[0122] Semantic information capture: Use a large model to perform semantic analysis on the currently collected content. The system uses a large model that has been fine-tuned for specific tasks to perform semantic analysis on the collected content. The fine-tuning process uses a transfer learning method. Based on a large-scale general corpus, supervised learning fine-tuning is performed for the data types and semantic requirements in this system. The fine-tuning process includes the following steps:

[0123] Data preparation: Prepare domain data related to the expected task through manual annotation, including thousands of data table structures and their corresponding semantic information as the initial data for the fine-tuning task;

[0124] Model fine-tuning: Based on the pre-trained model of the open source large model, the domain data is used for fine-tuning. By automatically dividing the training set and the validation set, the cross-entropy loss function is used to optimize the semantic understanding performance of the model on the specified training sentences. For a single sample, the cross-entropy loss is defined as shown in formula (1):

[0125] (1)

[0126] Where C is the number of categories for classification, is the actual label. If the sample belongs to the i-th class,

[0127] but =1 otherwise = 0. The probability that the model predicts that the sample belongs to the i-th class will be calculated by the softmax function. The cross entropy loss of the entire training set is calculated using formula (2):

[0128] (2)

[0129] Where N is the number of samples in the training set. and are the actual label and predicted probability distribution of the jth sample, respectively. During fine-tuning, each sample in the training set generates a predicted probability distribution after passing through the model, and then calculates the cross entropy loss with the actual label. The gradient of the loss function relative to the model parameters is calculated by the backpropagation algorithm, and the optimization algorithm stochastic gradient descent SGD is used to update the parameters of the model, so that the loss function is gradually reduced, thereby optimizing the performance of the model in semantic understanding.

[0130] Model optimization: Through mixed precision training and gradient accumulation techniques, while ensuring model accuracy, we optimize training efficiency and cross-check with the validation set, and verify the effect through a predefined loss function.

[0131] In the semantic parsing stage, the fine-tuned model can more accurately infer the semantic description of the data content and generate the corresponding vector representation. The system will further process and archive this semantic information, and vectorize and store all content with semantic information based on the acge_text_embeddin model, so that efficient fuzzy search can be achieved through subsequent 2048 high-dimensional vector distance calculation. Finally, the system encodes all acquired information according to the unified metadata encoding specification and stores it uniformly, thereby achieving standardized management and efficient retrieval of metadata.

[0132] Intelligently assisted drag-and-drop data stream construction: When users build data streams, the system helps users quickly complete the construction of data streams through multi-level intelligent recommendations and interactive construction functions. First, based on the definition and description of the application scenario, the system uses machine learning techniques such as association rule mining algorithms and cluster analysis to quickly recommend metadata related to user goals. The system uses the self-developed AP-GNN algorithm to assist users in quickly recommending metadata related to user goals. The AP-GNN algorithm is a novel algorithm that combines the Apriori algorithm with the graph neural network (GNN). Its core idea is to use the Apriori algorithm to extract metadata association patterns from historical data stream configurations, and use the graph neural network to model the complex mapping between these patterns and data sources to achieve intelligently assisted drag-and-drop data stream construction. The Ap-GNN algorithm first performs association rule mining and searches for all frequent item sets by gradually increasing the length of the item set. If the frequent item set meets the specified support threshold, it means that these metadata association patterns appear frequently in the historical data stream configuration. The support calculation is shown in formula (3):

[0133] (3)

[0134] in, Represents a specific set of metadata items, which is used to represent a combination of one or more fields / data tables and to calculate their corresponding support , It is the frequency of occurrence of metadata item sets in the historical data stream configuration, representing the frequency of occurrence of a specific field / data table combination. is the total number of historical data flow configurations, representing the total number of times the system has observed all past configurations in the smart-assisted drag-and-drop data flow construction.

[0135] Generate association rules from frequent item sets, and evaluate the reliability of the rules by confidence. The confidence calculation formula is shown in formula (4):

[0136] (4)

[0137] in, and are two item sets, represents a set of metadata items, and Representatives and Another metadata item or combination associated with it, association rule It means "if the item set Appears, then the item set It is also likely that the rule It is an item set The support of It is an item set and item sets Support for simultaneous occurrences in the same data stream configuration.

[0138] After obtaining reliable metadata association patterns, first, use a graph structure to represent the correspondence between these patterns and specific data sources. The nodes in the graph represent metadata and data sources, and the edges represent metadata association patterns. The weights on the edges are set to the confidence or support of the association rules. Then, the convolution operation of the graph neural network is used to gradually capture the local features and global structural features of the nodes. The implementation of the convolution operation is shown in formula (5):

[0139] (5)

[0140] in, represents the feature representation of node v at the kth layer, represents the set of neighboring nodes of node v, is the normalization constant between node v and its neighboring node u, It is The weight matrix of the layer, σ is the ReLU activation function.

[0141] In this way, users can locate the required data content in a transparent way, and through an intuitive drag-and-drop interface, connect nodes such as metadata, data operation steps, and logical output data sets to build data flows on demand without having to worry about the underlying details such as the data format, data storage location, and data access method. In addition, the system also provides an intelligent data flow building function in natural language. Users can use a deep learning model agent that has been fine-tuned with vertical field data to build a one-stop data flow by describing the expected application scenario and the field requirements of the final data. This agent enhances the model's understanding and generation capabilities of natural language by using technologies such as generative adversarial networks (GANs). During the fine-tuning process, a large number of data flow construction examples and natural language description comparison data sets are applied to improve the performance of the model under specific tasks. When the user enters the description, the system will generate a data flow that meets the requirements and display it on the interface for the user to review and confirm or further modify. The modification record will be used as the corpus training set for subsequent fine-tuning of the system.

[0142] Data resource construction and quality management: After the data flow is built, the system will ingest metadata according to its configuration, generate a logical data resource collection, namely the adaptive semantic data space DIKube, and present it in the user interface. Users can query data in real time through the query interface attached to the data resource. The query interface uses Apache Calcite, a distributed query optimization technology, to ensure that users can efficiently access the latest aggregated data. The system will dynamically obtain and update this data in the background to ensure that it is consistent with the current business scenario. In order to ensure the accuracy and reliability of logical data resources, the system allows users to configure data quality standards and detection rules. These rules will be automatically applied when the data flow is running to detect data quality in real time. The detection types supported by the system include:

[0143] Data consistency check: In order to improve the efficiency of data consistency check, the present invention uses a hash check algorithm, as shown in formula (6), which can convert a data block into a hash value of fixed length. The uniqueness of the hash value ensures the integrity and consistency of the detected data. If the data changes during transmission or storage, its hash value will also change. That is, by comparing the hash values, it is determined whether the data is consistent. For example, assuming that the data block is , the hash function is , then the hash value h of data block D can be calculated using formula (6):

[0144] (6)

[0145] in, is a SHA-256 hash function. is a fixed-length hash value generated by a hash function. and The hash value is the same, that is , then the two data blocks are considered consistent.

[0146] Data integrity check: Check the integrity of data through foreign key constraints to avoid partial data loss or errors;

[0147] Data accuracy verification: The system uses Bayesian networks to perform unsupervised real-time evaluation of data accuracy and identify potential abnormal data. A Bayesian network is a model based on probability graphs that is used to represent a set of random variables and their conditional dependencies. It consists of nodes (representing variables) and directed edges (representing conditional dependencies), and each node has a corresponding conditional probability distribution. Through the Bayesian network, the accuracy of the data can be probabilistically inferred, thereby identifying potential abnormal data. The implementation method is shown in formula (7).

[0148] (7)

[0149] in is the posterior probability of event A occurring given event B. It is the conditional probability of event B occurring given event A. is the prior probability of event A. is the marginal probability of event B. In data accuracy verification, Bayesian networks are used to model the conditional dependencies between data variables. The steps for constructing a Bayesian network for data accuracy verification are as follows: define the target metadata range to be verified for accuracy, locate the corresponding original data set through the location information in the metadata, and treat each attribute or feature in the data set as a random variable Define a parent node for each random variable (i.e., the variable it conditionally depends on) and form a directed acyclic graph (DAG). Each node in the graph represents a variable, and each edge represents a conditional dependency. For each node Generate a conditional probability table , which represents the probability of a node taking different values ​​given its parent node.

[0150] Data timeliness verification: Use time series analysis and real-time stream processing technology to monitor the frequency of data updates to ensure that the business's real-time data requirements are met;

[0151] User-defined validation: Users can configure data quality standards and detection rules to perform real-time validation on ingested data. They can define custom rules to verify whether the data meets the expected format, value range, statistical information, etc. to meet specific business needs.

[0152] Step 2: Data classification and grading identification. When data resources are established, the system will combine metadata format information and sampling methods to determine the type of data resources and classification and grading strategies; after that, users can choose the scope of the data list to follow based on their own industry and region; after completing the preliminary operations, the system can perform the final data classification and grading identification based on the big model.

[0153] The steps to achieve data classification and grading identification are as follows:

[0154] Classification and grading identification strategy of data resources: The data formats and corresponding identification strategies supported by the system include the following four situations:

[0155] a) For structured data such as table files, data tables in structured databases, and tables stored in information systems, the classification and grading objects are the individual fields therein;

[0156] b) For document-type unstructured data, the classification and grading objects are the individual paragraphs therein;

[0157] c) For unstructured data such as design drawings, industrial control programs, and system monitoring data, the classification and grading object is the data as a whole;

[0158] d) Other unidentifiable types of data will not be classified or graded.

[0159] Determine the data list type: After clarifying the classification and grading strategy and objects, users can choose the data list scope to follow according to their own situation. The data lists included in the system include:

[0160] a) General list, mainly including the definition and corresponding description of personal data, sensitive personal data and data types that are obviously in violation of relevant laws and regulations or public order and good morals;

[0161] b) Regional lists, which are data standards and management requirements for enterprises in a specific region;

[0162] c) Industry lists mainly include data standards and management requirements that companies in the industry should comply with, issued by the competent authorities of the financial, automotive and other industries.

[0163] Classification and grading recognition: When performing classification and grading recognition, the system relies on a vertical domain large model agent that has been fine-tuned for a specific task. This agent has the characteristics of being compatible with many open source or closed source large models in a hot-swappable manner, and can automatically generate existing corpus according to the specific requirements of the large model. By using a large amount of field data related to data classification and grading and training in actual business scenarios, it has the function of automatically retrieving relevant knowledge and accurately classifying and grading each data object. For the classification and grading results generated by the large model, the system uses the self-developed CE-F1 function for classification optimization. By combining the weighted results of multiple indicators, it ensures that the classification ability of the model meets business needs. The CE-F1 function combines the functions of the cross-entropy loss function (Cross-Entropy Loss) and the classification evaluation indicator F1-score. Among them, the cross-entropy loss function is used to measure the single error coefficient of the classification result, and F1-Score represents the quality weight Partial-weight of the recall result obtained by calculating the harmonic mean of the precision (Precision) and the recall rate (Recall), which is used to measure the performance of the classifier when processing unbalanced data sets. The quality weight calculation is shown in formulas (8)-(10):

[0164] (8)

[0165] (9)

[0166] (10)

[0167] Among them, Partial-weight represents the weight of the F1-score part in the CE-F1 function, and its final value will be harmonized and averaged with the calculation result of the cross entropy loss function. TP (True Positives) represents the number of positive samples predicted as positive samples. FP (False Positives) represents the number of negative samples incorrectly predicted as positive samples. FN (False Negatives) represents the number of positive samples incorrectly predicted as negative samples. TN (True Negatives) represents the number of negative samples correctly predicted as negative samples. The recognition and evaluation process uses knowledge graphs and adaptive learning mechanisms to improve recognition accuracy and evaluation dimensions. The knowledge graph provides concepts and relationships in the field, allowing the intelligent agent to consider a wider range of semantic associations when classifying; the adaptive learning mechanism ensures that the intelligent agent can adjust its classification strategy to maintain high accuracy when facing new or changing data. The knowledge graph provides concepts and relationships in the field, allowing the intelligent agent to consider a wider range of semantic associations when classifying. The adaptive learning mechanism ensures that the intelligent agent can adjust its classification strategy to ensure high accuracy when facing new or changing data. In order to ensure that the classification and grading results are not only technically accurate but also meet actual business needs, the system allows users to manually intervene and adjust the automatically generated classification and grading results. The system provides a user-friendly review interface where users can intuitively view and modify classification results. Through functions such as clicking and batch operations, users can quickly adjust classification labels or grading levels. Users can also customize classification rules or grading standards based on specific business scenarios to support data management standards or other related requirements that may exist within the enterprise. The system supports continuous improvement of classification agents through user feedback. Each manual adjustment will be recorded and used to update the model or adjust the classification rules to improve the accuracy of the agent in future recognition tasks.

[0168] Step 3: Circulation and trading. For data resources that have been aggregated and configured with quality inspection and classification and grading, the system supports upgrading the built-in data query interface by configuring specific product information and building the interface into a data product; data products can be uniformly included by the platform and can be traded to a specified range of users or all other users; the platform will record all transaction behaviors for easy subsequent tracing. The steps to achieve data circulation and trading are as follows:

[0169] Data resources upgraded to data products: For data resources that have completed quality inspection rules and classification and grading configuration, the system supports the rapid upgrade of the corresponding data query interface to data products through additional configuration. Data products can freely configure their query conditions, sorting basis, return quantity, starting position, etc., so that product owners can flexibly adjust the tradable scope, while improving the flexibility and convenience of product consumers.

[0170] Data product recommendation: The system uses a self-developed real-time recommendation algorithm MetaGraphFusion, which can recommend the most suitable data products to users based on their historical behavior, preferences, and current needs. The specific working principle of the algorithm MetaGraphFusion can be divided into two parts: offline and online:

[0171] Offline part: The system analyzes the historical data stream configuration, mines common metadata association patterns, and uses graph neural networks (GNNs) to build and optimize the complex relationship between data sources and metadata. The main tasks of the offline part are data preprocessing and model training. Frequently occurring metadata item sets are extracted from the historical data stream configuration in a manner similar to the Apriori algorithm, common association patterns are identified, and the support of each metadata item set is calculated by analyzing historical data, and association rules are generated to record the co-occurrence relationship between metadata. After that, the system will build a graph structure of metadata and data sources based on the output of the association rules. By using metadata item sets as nodes in the graph, association rules as edges, and setting the weight of the edges as confidence, a graph reflecting the complex relationship between metadata is formed. Finally, the system uses graph neural networks to train the constructed graph to learn the feature representation of nodes (metadata) and edges (association relationships). The system continuously optimizes the feature representation of nodes through graph convolution operations (Graph Convolution), so that it can better reflect the correlation and complex relationship between metadata and data sources.

[0172] Online part: Based on the data stream being constructed by the user, the system analyzes the metadata selection and the matching relationship of the data source in real time, detects the metadata item set selected by the current user, and maps it to the trained GNN model to obtain relevant feature representations. The system uses the association rules and GNN feature representations in the offline model for real-time online matching, recommends high-confidence metadata combinations and data source matching solutions, queries the GNN model based on the metadata selected by the current user, finds other metadata item sets that are highly related to it, and provides recommendations based on confidence scores. In the process of users constructing data streams, the system optimizes the recommendation strategy in real time to adapt to the dynamic needs of users. By continuously monitoring the user's selections and adjustments, the recommendation list is dynamically updated, and retraining or online fine-tuning is triggered when necessary to ensure the accuracy of the recommendation.

[0173] Data product transactions: All data products will be uniformly recorded in the platform to support data product transactions. Both parties can use the system's built-in functions to match and search for data products. After determining the data product, you can use offline business processes to make long-term or batch purchases based on the available time or total number of calls, or make batch purchases through the platform's online transactions.

[0174] Transaction information management: The system will automatically record transaction information, including transaction time, information of both parties to the transaction, transaction type information, and sampling and storage of query results, to meet the needs of data transaction process and ensure subsequent traceability.

[0175] Step 4: Value assessment. The system provides three main assessment methods: cost assessment, benefit assessment, and transaction assessment, and supports asset configuration based on data resources. The configuration items include the following four aspects:

[0176] Ownership confirmation: Users first need to confirm and declare the ownership of the data assets to be generated. The system uses smart contracts and blockchain technology to ensure the legitimacy and immutability of ownership declarations. When confirming ownership, users can use the ownership verification algorithms provided by the system (such as digital signature verification and public key encryption) to verify their ownership or control over data resources. Only content in data resources that users legally own or control can be assetized to ensure the legitimacy and compliance of assetization.

[0177] Cost aggregation: The system allows users to configure the cost of each step of data resources from generation to maintenance in detail. Specifically, it includes data generation, collection, processing, aggregation, and maintenance. The system uses the self-developed activity-based costing algorithm (ABC) for cost accounting. The proposed algorithm automatically aggregates various costs and generates total cost data. These cost data will be used for subsequent data asset cost evaluation to accurately reflect the real economic input of data resources. Activity-based costing is a refined cost calculation method that can allocate indirect costs to specific activities related to products or services, thereby accurately measuring the contribution of each activity to the total cost. For data resources, activity-based costing can help users identify costs in each link and aggregate them into overall cost data. The cost aggregation system developed based on the activity-based costing algorithm allows users to customize the cost of each step, which usually includes data generation, collection, processing, aggregation, and maintenance. Based on these parameters, the system aggregates costs into total cost data based on the activity-based costing algorithm. For each data resource, assume that it contains N activity steps, and the cost of each step is , then the total cost can be calculated using formula (11): :

[0178] (11)

[0179] The cost of each activity step is It can be calculated using formula (12):

[0180] (12)

[0181] in Represents the resource costs associated with activity i, such as manpower, equipment, energy, etc. It represents the activity driver, indicating the degree or amount of resources consumed by activity i, such as working hours, amount of data processed, etc.

[0182] Revenue measurement: The system uses real-time statistics and analysis functions to comprehensively evaluate the value contribution of data resources to corporate operations and overall business. The system counts the user access to the system interface and the frequency of use of the query interface, and uses the self-developed ARI-MLR algorithm to evaluate the use of data resources in different time periods and different user groups. It further combines the company's operating indicators to quantify the contribution of data resources to the company's overall business and give it additional economic value. The core idea of ​​the ARI-MLR algorithm is to use formulas (13) and (14) to analyze and predict the use of data resources in different time periods, thereby identifying important time periods such as peak periods and trough periods.

[0183] (13)

[0184] in It's time The observed value represents the number of transactions, visits, or usage frequency that occurred in a specific time period. is a constant term used to represent the long-term average level of the data in the time series. It is the order of the autoregressive term, which indicates how many past observations (or lagged values) are introduced into the model to predict the current value, which can help capture the continuity and autocorrelation of the data. is the autoregressive coefficient, which is used to represent The impact of the visits at a certain time point on the current visit forecast, is the order of the moving average term, which indicates how many past error terms are introduced into the model to predict the current value. is the moving average coefficient, indicating the past The impact of the error term at each time point on the current forecast, is a white noise error term, which represents unpredictable random changes such as sudden increase or decrease in access volume. Subsequently, the system evaluates the impact of data resource usage frequency, business relevance, and data uniqueness on the enterprise operating indicators. The calculation method is shown in formula (14):

[0185] (14)

[0186] in is the dependent variable (such as business performance), is the intercept term. , ,…, are the regression coefficients of the respective variables, , ,…, are independent variables (such as frequency of use of data resources, business relevance, data uniqueness, etc.), is the error term.

[0187] Transaction statistics: The system can comprehensively record the transaction status of data resources on the platform, including historical transaction frequency, transaction price and future transaction expectations. The system uses the self-developed time series prediction algorithm Prophet model to predict future transaction trends, and combines the records of historical data transactions to generate transaction evaluation results. Through these statistics, users can give additional added value to data resources based on their market performance and future potential. Prophet is a flexible and powerful time series prediction algorithm, especially suitable for time series data with obvious trends and seasonality. The Prophet model predicts by decomposing the time series into trend, seasonality and holiday effects. The core idea of ​​the Prophet model is reflected in formula (15):

[0188] (15)

[0189] in It's time Observations of (such as transaction frequency or price), is a trend function, used to capture long-term growth or decline, It is a seasonal function, reflecting the periodic fluctuations in trading data. It is the holiday effect function, which is used to deal with the impact of special time periods such as holidays. is the error term, representing random fluctuations or noise.

[0190] After completing ownership confirmation, cost collection, revenue measurement, and transaction statistics, the system will automatically generate a comprehensive value calculation formula. This formula combines the cost and added value of data resources, and supports users to package and evaluate data sets of any range. The system uses linear programming and optimization algorithms to automatically calculate the final value of the packaged data set based on factors such as the size, number of items, and frequency of use. This calculation formula and results provide users with a standardized asset evaluation tool that supports enterprises to reasonably price and manage data assets in different business scenarios.

[0191] Step 5: Asset generation and table entry. The system supports regular packaging and generation of data resources, and can perform global statistics and management of currently effective data assets, as well as view the content details of any specified data asset. The steps to achieve asset generation and table entry are as follows:

[0192] Asset configuration: Users can configure the data resources in the system as assets according to their own needs. The system supports users to set the data asset generation cycle by themselves. This cycle determines the time and frequency of data resources to be regularly packaged to form data assets. The system uses the self-developed ARI-Prophet model to dynamically optimize the data asset generation cycle. The ARI-Prophet model is based on the Prophet model described in step 4) of step 4 and the formula (13) described in step 4) of step 4 to balance the weights of the Prophet model, predict future data growth trends, and recommend the optimal asset generation cycle for users. Users can choose to package every natural month, quarter, half a year or year. In addition, users can set the expected life cycle of data generation assets based on the content characteristics and usage of the data. Periodic task scheduling is implemented through the Cron Job scheduling mechanism to ensure that data assets are automatically generated at the scheduled time point.

[0193] Automatically package assets to be reviewed: Automatically extract new content from the original data within the cycle and package it into static, unchangeable data assets to be reviewed. In the process, Delta Lake is used as a means of data segmentation and snapshot implementation. A distributed data processing algorithm based on MapReduce is introduced as a data segmentation and aggregation algorithm. According to the characteristics of the data (such as data type, data volume, access frequency, etc.), data resources are segmented and aggregated into the most suitable asset package in an intelligent way. Ensure that the content of each package is complete and consistent. After the data assets to be reviewed are formed, the system marks them as "pending review" and notifies the user to perform subsequent operations. MapReduce is a distributed computing framework that is mainly used for large-scale data processing and summary generation. The system introduces a distributed data processing algorithm based on MapReduce to efficiently segment and aggregate large-scale data and decompose the original data into multiple small data blocks (i.e., "key-value pairs"). According to the characteristics of the data, each data block is assigned to a different computing node for processing. The implementation method is shown in formula (16).

[0194] (16)

[0195] in, is the input key of the data block ID, is the input value of the data content, It is an intermediate key that contains information such as the segmented data identifier. It is the middle value of the segmented data block. The system uses the middle key The Map output is grouped and redistributed, data blocks with the same key are aggregated together, and the aggregated data blocks are processed and merged into the final output data. This process ensures the integrity and consistency of the data and generates the final data assets to be reviewed, as shown in the Reduce function formula (17):

[0196] (17)

[0197] in It's the middle key. It is a list of intermediate values ​​output by the Map stage. is the data to be packaged, It is the final output value (i.e. the packaged data asset).

[0198] Data asset audit: The system uses a self-developed random forest algorithm based on Bootstrap sampling to analyze the characteristics of historical audit records and data assets, automatically classify and score the data assets to be audited, and provide audit suggestions to users. Users can view the data assets to be audited and the audit suggestions provided by the system, and decide whether to generate the asset based on the actual situation. The system will automatically record all audit situations. The random forest algorithm based on Bootstrap sampling is an integrated learning method that improves the classification accuracy and robustness of the model by building multiple decision trees and voting on their results. Each decision tree uses a different random subset (including samples and features) during the construction process, which enables random forests to effectively reduce overfitting and perform well when processing high-dimensional data. Multiple subsets are randomly extracted from historical audit records and data asset features, and each subset is used to train a decision tree. This process is called Bootstrap sampling, that is, sampling with replacement. When constructing each decision tree, for each node split, some features (rather than using all features) are randomly selected to search for the best split. This step reduces the correlation between decision trees and improves the generalization ability of the integrated model. Each decision tree recursively splits the selected features until the stopping condition (such as node purity, maximum depth, etc.) is met. The splitting of the random forest algorithm based on Bootstrap sampling adopts the information gain method shown in formula (18):

[0199] (18)

[0200] in It is a node The entropy of the data set purity or uncertainty, used to judge Are the types of data in similar? Features represent attributes or fields of specific metadata, such as the type, source, and timestamp of the data. It is a feature The value is The subset of time, according to the characteristics Different values ​​from Extract a subset from It is a node The number of samples in represents the total amount of data in the dataset.

[0201] During the audit phase, the data asset to be audited passes through all decision trees, and each decision tree will classify the asset. The random forest model ultimately classifies the asset by majority voting and generates a classification score for it (usually in the form of probability, such as the probability of belonging to a certain category). Through the above process, the system will generate a classification label (such as "high priority", "low risk" or "needs to be audited") and a corresponding score (such as 0.85, indicating that the asset belongs to a certain category with 85% confidence) for each data asset to be audited. This score is based on the output of the random forest algorithm with Bootstrap sampling and is a weighted average of the voting results of multiple decision trees. Users can view the data assets to be audited and the audit suggestions provided by the system, and decide whether to generate the asset based on the actual situation. The system will record each audit decision of the user, including whether to adopt the system suggestion, the specific time of the audit, the auditor, and other information. These records will be used to update the random forest model to improve the accuracy of future audits.

[0202] Unified management of data assets: The data assets formed can be managed uniformly by the system as valuable intangible assets. Users can view the current data asset list, total value, amortization status, etc.:

[0203] The system automatically amortizes the assets using the linear amortization method based on their life cycle, and generates statistics monthly in a commonly used financial format. It also supports exporting and downloading in table form.

[0204] For any specified data asset, users can upload a third-party evaluation report to support their value proposition.

[0205] Based on the user's configuration, the data will be automatically integrated and generated to provide users with viewing or exporting services, including the price composition of their data assets, total asset price, quality inspection results, classification and grading, and third-party evaluation information.

[0206] See also Figure 3 , the present invention also includes a dynamic large model access and intelligent body construction and management module. As an important auxiliary module of the system described in the present invention, this module provides support for various tasks involved in the present invention, such as data flow construction, classification and grading identification, data product matching and recommendation, etc. This module can provide support for unsupervised / semi-supervised feature modeling algorithms such as ARI-Prophet model and AP-GNN, and recommendation algorithms such as bidirectional multi-dimensional filtering mechanisms used in the process steps, integrating the respective advantages of large language models and other artificial intelligence algorithm models, and can be applied to specific tasks with strong pertinence and high verticality, as well as fuzzy tasks with high degree of freedom, and can meet the characteristics and requirements of the system described in the present invention with many task types and strict output format requirements, and support long-term use and continuous optimization.

[0207] The dynamic large model access and intelligent agent construction and management module has the ability of automatic and long-term self-optimization. The implementation steps of the dynamic large model access and intelligent agent construction and management module are as follows:

[0208] Step 1: Encapsulate the model: Encapsulate the differences between the call parameters and return structures of different large model interfaces in the form of connectors, provide a unified usage method, and support switching between different large models on demand;

[0209] Step 2: Generate an agent: Generate an agent corresponding to a specific type of task for tasks such as data flow construction, asset discovery, and graph fusion, or create a copy based on an existing agent for experimental development or further fine-tuning. Any agent supports two behavior modes: execution with predefined processes and autonomous determination of processes. Its behavior mode can be dynamically specified when used. When creating an agent, a fine-tuning model will be automatically created at the same time, and two main functions, call and feedback, are provided. Other functional components can call the agent to complete the task.

[0210] Step 3: Obtain the agent execution result: Other modules in the system described in the present invention can pass in inputs that meet the expected format through the call function and obtain the agent execution result. After obtaining the execution result, other modules can dynamically generate the original corpus required for fine-tuning the agent based on the user's feedback and modification of the execution result or other predefined rules, provide feedback through the agent feedback function, and locate the information at the time of execution based on the task ID, and store it as a fine-tuning corpus by combining the input to the large model during the call and the results in the feedback.

[0211] Step 4: Determine the fine-tuning requirements of the agent: The dynamic large model access and agent construction and management modules can determine their fine-tuning frequency, tendency and other strategies through global configuration. The module will globally coordinate the fine-tuning requirements of each agent according to the configured time and frequency, and dynamically plan the fine-tuning execution time of each agent.

[0212] Step 5: Fine-tune the training agent: At the beginning of fine-tuning, the agent uses model reasoning to clean the data, eliminate errors, low-quality, and repeated content, solidify the processed high-quality fine-tuning corpus, convert the format of the fine-tuning corpus based on the fine-tuning interface parameter format requirements of the basic model, and finally use the fine-tuning function of the basic large model for fine-tuning training;

[0213] Step 6: Manage and reuse fine-tuning corpus: The stored and solidified high-quality fine-tuning corpus will be anonymized, and the stored content will not be associated with user personal information and other content, but will only be associated with the corresponding tasks and agents. When the module is migrated to the new basic large model as a whole, each agent will re-execute the fine-tuning task according to the historical fine-tuning records, so that the agents based on the new basic large model can receive necessary training to ensure the execution effect and quality.

[0214] See also Figure 4 The present invention also includes a knowledge graph that can grow autonomously and intelligently. As an important part of the system described in the present invention, the knowledge graph that can grow autonomously and intelligently supports targeted recommendations to users based on the common data and assets of enterprises in the same industry, business type and link, based on the metadata information and user asset construction status ingested in data flow orchestration and quality control, with explicit authorization from the user, and can intelligently collect, process and integrate user usage into the appropriate position in the graph. The steps to realize a knowledge graph that can grow autonomously and intelligently are as follows:

[0215] Step 1: Construct the basic structure of the graph. It includes the following 4 parts:

[0216] 1) Industry and specific links: As the identifier of the sub-graph, it is used to distinguish the enterprise situations of different industries, different business types and links. Different sub-graphs may have similar or completely different structures and components. The overall knowledge graph is composed of multiple sub-graphs;

[0217] 2) Raw data description information: As one of the main node types, it is used to identify the description of the raw data type owned by the enterprise customers in a specific sub-graph. Each raw data description information node has a weight value to explain to the system its prevalence in the current sub-graph. In addition, the raw data description information node additionally carries the Schema information after fusion weighting and standardization. When using this node to match and recommend the user's current data situation, the Schema information will be used to assist in the comparison;

[0218] 3) Data resource description information: As another type of node, it is used to identify the description information of the specific scenarios in which enterprise customers use data in a specific sub-graph. Each data resource description information node has a weight value to indicate to the system its prevalence in the current sub-graph.

[0219] 4) "Composition / Contains" relationship: connects a raw data description information node with a data resource description information node, identifying the composition relationship between the raw data and the data resource. This type of relationship has additional description information to summarize and explain the data processing and data flow construction process.

[0220] Step 2: Prepare the overview map content. This is to help the system of the present invention to ensure the basic user experience when it is launched and when there is insufficient accumulation of actual user usage data.

[0221] Step 3: Graph-based selection of willingness to capitalize data: Users can choose whether to use the knowledge graph to make assetization recommendations and provide operational assistance, and at the same time allow the system to collect macro-descriptive information that does not involve the specific information carried by the data, such as data item status, data table description, description of constructed assets, etc.

[0222] Step 4: Generate data assetization suggestions based on the graph: When the user chooses to use the relevant services and agrees to collection and sharing, the system will use the semantic description information of the current global data ingested during the data flow orchestration and quality control process, including the content of the accessed original data, to determine the type of data and basic situation owned by the current user, and compare the scanning results with the existing knowledge graph content in the system. Users can choose their corresponding industry, business type and link to locate (or create a new) Sub-Graph, and make suggestions for their subsequent data operations and assetization based on the comparison results.

[0223] Step 5: Expand and optimize the graph content based on user information: After the user builds the data resource, the relevant information will be anonymized and transmitted to the server. The system will update the existing nodes and relationships or create new nodes and relationships according to the situation. The entire process of this task is supported by a fine-tuned vertical domain large model agent, thereby improving the accuracy of recommendation and fusion.

[0224] In this document, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "vertical", and "horizontal" are based on the directions or positional relationships shown in the accompanying drawings and are only for the clarity of expressing the technical solutions and the convenience of description, and therefore should not be understood as limitations of the present invention.

[0225] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0226] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A data asset automatic generation and management system based on data operation technology, characterized in that: Includes the following functional modules: 1) Data flow orchestration and quality control: According to data requirements and application scenarios, data flows are built based on metadata to form a logical data resource set for application scenarios. Quality inspection rules are configured to inspect the quality of logical data resources. The system will inspect the data in the logical data resource set according to the quality inspection rules to ensure that the data quality always meets requirements and expectations. 2) Data classification and grading identification: Combine the metadata format information and sampling method to determine the type and classification and grading strategy of logical data resources; users can select the data list range to be followed according to the industry and region they are in, and conduct the final data classification and grading identification; 3) Circulation and trading: For logical data resources that have been aggregated, quality-checked, and classified and graded, data products with asset value are constructed by configuring specific product information on their data query interfaces; 4) Value assessment: Data assets are assessed through cost assessment, benefit assessment, and transaction assessment, and asset allocation is performed based on data resources; 5) Asset generation and table entry: Automatically and regularly package and generate data resources, and perform global statistics and management on currently effective data assets; The method of data flow arrangement and quality control is as follows: 1.1) Metadata ingestion: By configuring the protocol type, connection location and authentication information, adding user files, database data, information system data and real-time information of IoT edge devices to be managed, the system automatically connects and scans all readable content in the target location, and records the corresponding storage location and explicit features for each readable content; 1.2) Semantic Information Capture: Use a large model that has been fine-tuned for tasks to perform semantic analysis on the currently collected content. Task fine-tuning uses transfer learning methods to perform supervised learning adjustments based on large-scale general corpus and data types and semantic requirements. 1.3) Intelligently assisted drag-and-drop data flow construction: When users build data flows, multi-level intelligent recommendations and interactive construction functions are used to help users quickly complete the construction of data flows; 1.4) Data resource construction and quality control: After the data flow is constructed, the system will ingest metadata according to its configuration, generate a logical data resource set and present it in the user interface. At the same time, the system supports the configuration of quality inspection rules for data resources to ensure that the quality of the data always meets the requirements and expectations; The intelligently assisted drag-and-drop data flow construction method in step 1.3) is as follows: Based on the definition and description of the application scenario, the AP-GNN algorithm is used to assist users in quickly recommending metadata related to their goals. The AP-GNN algorithm first performs association rule mining and searches for all frequent item sets by gradually increasing the length of the item set. If the frequent item set meets the specified support The threshold value indicates that these metadata association patterns appear more frequently in the historical data stream configuration; Among them, support The calculation is shown in formula (3): (3) in, Represents a specific set of metadata items, which is used to represent a combination of one or more fields / data tables and to calculate their corresponding support , It is the frequency of occurrence of metadata item sets in the historical data stream configuration, representing the frequency of occurrence of a specific field / data table combination. is the total number of historical data stream configurations, representing the total number of times the system has observed the intelligently assisted drag-and-drop data stream construction; Generate association rules from frequent item sets and use confidence Evaluate the reliability and confidence of rules The calculation formula is as shown in formula (4): (4) in, and are two item sets, represents a set of metadata items, and Representatives and Another metadata item or combination associated with it, association rule Representation rule: If the item set Appears, then the item set It is also likely that It is an item set The support of It is an item set and item sets Support for simultaneous occurrences in the same data stream configuration; After obtaining reliable metadata association patterns, a graph structure is used to represent the correspondence between these patterns and specific data sources. The nodes in the graph structure represent metadata and data sources, and the edges in the graph structure represent metadata association patterns. The weights on the edges are set as the confidence or support of the association rules. Then, the convolution operation of the graph neural network is used to gradually capture the local features and global structural features of the nodes. The implementation of the convolution operation is shown in formula (5): (5) in, represents the feature representation of node v at layer k, represents the set of neighboring nodes of node v, is the normalization constant between node v and its neighboring node u, It is The weight matrix of the layer, σ is the ReLU activation function; Through the above methods, users can locate the required data content in a transparent manner, and through an intuitive drag-and-drop interface, connect metadata, data operation steps and logical output data sets to build data flows on demand without having to worry about the data format, data storage location and data access method.

2. The data asset automatic generation and management system based on data operation technology according to claim 1 is characterized in that: The method for data classification and grading identification is as follows: 2.1) Determine the corresponding classification and grading identification strategy and objects according to the format of logical data resources; 2.2) After clarifying the classification and grading identification strategy and objects, select the scope of the data inventory that needs to be followed; 2.3) Perform classification and grading recognition through a large vertical domain model agent that has been fine-tuned for specific tasks.

3. The data asset automatic generation and management system based on data operation technology according to claim 1 is characterized in that: The method of the circulation transaction is as follows: 3.1) For data resources that have completed quality inspection rules and classification and grading identification configuration, by configuring their data query interface with specific product information, build data products with asset value; 3.2) Use real-time recommendation algorithms to recommend the most suitable data products to users based on their historical behaviors, preferences, and current needs; 3.3) Collect the description information of all data products, match and search for data products, and after determining the data products, conduct transactions using offline business processes or online transactions; 3.4) Record the transaction information, including the time of transaction, information of both parties to the transaction, transaction type information, and sampling and storage of query results, to meet the storage of data transaction process and ensure subsequent traceability.

4. The data asset automatic generation and management system based on data operation technology according to claim 1 is characterized in that: The method of value assessment is as follows: 4.1) Ownership confirmation: Users first need to confirm and declare the ownership of the data assets to be generated; 4.2) Cost aggregation: The system allows users to configure the cost of each step of data resource from generation to maintenance in detail; 4.3) Revenue measurement: Through real-time statistics and analysis functions, comprehensively evaluate the value contribution of data resources to corporate operations and overall business; 4.4) Transaction Statistics: The system can comprehensively record the transaction status of statistical resources on the platform, including historical transaction frequency, transaction price and future transaction expectations.

5. The data asset automatic generation and management system based on data operation technology according to claim 1 is characterized in that: The method of generating and entering assets into the table is as follows: 5.1) Asset allocation: According to the user's own needs, the data resources in the system are assetized; 5.2) Automatically package assets to be reviewed: extract new content within the cycle from the original data and package it into static and unchangeable data assets to be reviewed; 5.3) Data asset audit: Use the random forest algorithm based on Bootstrap sampling to analyze the characteristics of historical audit records and data assets, automatically perform preliminary classification and scoring of data assets to be audited, and provide audit suggestions to users; 5.4) Unified management of data assets: The generated data assets, as valuable intangible assets, can be managed uniformly by data operation technology. Users can view the current data asset list, total value and amortization status.

6. According to the data asset automatic generation and management system based on data operation technology according to claim 1, the task fine-tuning method of the large model in step 1.2) is as follows: a) Data preparation: Prepare domain data related to the expected task through manual annotation, including thousands of data table structures and their corresponding semantic information as the initial data for the fine-tuning task; b) Model fine-tuning: Based on the pre-trained model of the open source large model, the domain data is used for fine-tuning. By automatically dividing the training set and the validation set, the cross-entropy loss function is used to optimize the semantic understanding performance of the model on the specified training sentence. For a single sample, the cross-entropy loss is defined as shown in formula (1): (1) in, C is the number of categories for classification, is the actual label. If the sample belongs to i Class, then =1, otherwise =0; the model predicts that the sample belongs to i The probability of the class will be calculated by the softmax function; For the cross entropy loss of the entire training set, take the average of all samples and calculate it using formula (2): (2) in, N is the number of samples in the training set; and are the actual label and predicted probability distribution of the jth sample respectively; c) Model optimization: Through mixed precision training and gradient accumulation, while ensuring the accuracy of the model, the training efficiency is optimized and cross-compared with the validation set, and the effect is verified through a predefined loss function.

7. According to the data asset automatic generation and management system based on data operation technology of claim 1, the method of performing quality inspection on logical data resources by configuring quality inspection rules is as follows: a) Data consistency check: Use the hash check algorithm to convert the data block into a hash value of fixed length, and compare the hash values ​​to determine whether the data is consistent; The data block is , the hash function is , then the hash value h of data block D can be calculated using formula (6): (6) b) Data integrity check: Check the integrity of data through foreign key constraints to avoid partial data loss or errors; c) Data accuracy verification: Apply Bayesian network to evaluate the accuracy of data in real time and identify potential abnormal data. The implementation method is shown in formula (7); (7) in, is the posterior probability of event A occurring given event B; It is the conditional probability of event B occurring given event A. is the prior probability of event A; is the marginal probability of event B; d) Data timeliness verification: Use time series analysis and real-time stream processing technology to monitor the frequency of data updates to ensure that the real-time requirements of the business for data are met; e) User-defined validation: By configuring data quality standards and detection rules by yourself, the ingested data can be validated in real time. Customizable rules can be used to verify whether the data meets the expected format, value range and statistical information to meet specific business needs.

8. According to the data asset automatic generation and management system based on data operation technology of claim 2, the method of data classification and grading identification in step 2.3) is as follows: a) Use large models to accurately classify and grade each data object; b) For the classification and grading results generated by the large model, the CE-F1 function was used for classification optimization; The CE-F1 function combines the functions of the cross entropy loss function and the classification evaluation indicator F1-score; The cross entropy loss function is used to measure the single error coefficient of the classification result. The F1-Score represents the quality weight of the recall result obtained by calculating the harmonic mean of the precision and recall. Partial-weight , used to measure the performance of the classifier when dealing with imbalanced data sets; The quality weight calculation is shown in formulas (8)-(10): (8) (9) (10) Among them, Partial-weight represents the weight of the F1-score part in the CE-F1 function, and its final value will be harmonized and averaged with the calculation result of the cross entropy loss function; TP represents the number of positive samples predicted as positive samples; FP represents the number of negative samples incorrectly predicted as positive samples; FN represents the number of positive samples incorrectly predicted as negative samples.

9. According to the data asset automatic generation and management system based on data operation technology in claim 4, the cost collection method is as follows: For each data resource, it contains N activity steps, and the cost of each step is ,in , then the total cost can be calculated using formula (11): : (11) in, Cost per activity step Calculate using formula (12): (12) in, Representatives and Activities i associated resource costs; Represents the activity driver, indicating the activity i the degree or amount of resources consumed; The method of calculating the income is as follows: The ARI-MLR algorithm is used to analyze and predict the usage of data resources in different time periods, thereby identifying peak and trough periods, as shown in formulas (13) and (14): (13) in, It's time The observed value represents the number of transactions, visits, or usage frequency that occurred in a specific time period. is a constant term used to represent the long-term average level of the data in the time series. is the order of the autoregressive term, which indicates how many past observations or lagged values ​​are introduced into the model to predict the current value, which helps capture the continuity and autocorrelation of the data. is the autoregressive coefficient, which is used to represent The impact of the visits at a certain time point on the current visit forecast, is the order of the moving average term, which indicates how many past error terms are introduced into the model to predict the current value. is the moving average coefficient, indicating the past The impact of the error term at each time point on the current forecast, is a white noise error term, which represents unpredictable random changes such as sudden increase or decrease in access volume; (14) in is the dependent variable, is the intercept term, are the regression coefficients of the respective variables, is the independent variable, is the error term, The transaction statistics method is as follows: The Prophet model, a time series prediction algorithm, is used to predict future transaction trends, and combined with historical data transaction records, transaction evaluation results are generated, as shown in formula (15): (15) in, It's time The observed value of is a trend function, used to capture long-term growth or decline, is a seasonal function, reflecting the periodic fluctuations in transaction data. is the holiday effect function, which is used to deal with the impact of special time periods such as holidays. is the error term, representing random fluctuations or noise.

10. According to the data asset automatic generation and management system based on data operation technology in claim 5, the method of automatically packaging assets to be reviewed is as follows: The distributed data processing algorithm based on MapReduce is used to segment and aggregate the data. According to the characteristics of the data, each data block is assigned to a different computing node for processing. The implementation method is shown in formula (16): (16) in, is the input key of the data block ID, is the input value of the data content, It is an intermediate key that contains the segmented data identification information. is the median value of the segmented data block; The Reduce function is shown in formula (17): (17) in, It is an intermediate key that contains the segmented data identification information. It is a list of intermediate values ​​output by the Map stage. is the data to be packaged, is the final output value; The method of data asset audit is as follows: The random forest algorithm based on Bootstrap sampling analyzes the characteristics of historical audit records and data assets, automatically performs preliminary classification and scoring of the data assets to be audited, and provides audit suggestions; the splitting of the random forest algorithm based on Bootstrap sampling adopts the information gain method shown in formula (18): (18) in, Is a node The entropy of the data set purity or uncertainty, used to judge Are the types of data in similar? are features, representing properties or fields of specific metadata, It is a feature The value is The subset of time, according to the characteristics Different values ​​from Extract a subset from It is a dataset The number of samples represents the total amount of data in the dataset.

11. The data asset automatic generation and management system based on data operation technology according to claim 1, characterized in that: It also includes dynamic large model access and intelligent agent construction and management modules, and its implementation steps are as follows: Step 1: Encapsulate the model: Encapsulate the differences between the call parameters and return structures of different large model interfaces in the form of connectors, provide a unified usage method, and support switching between different large models on demand; Step 2: Generate an agent: Generate an agent for a specific type of task, such as data flow construction, asset discovery, and graph fusion tasks, or create a copy of an existing agent for experimental development or further fine-tuning. Step 3: Obtain the agent execution results: Other modules can pass in inputs that meet the expected format through the calling function and obtain the agent execution results; after obtaining the execution results, other modules can dynamically generate the original corpus required for fine-tuning the agent based on the user's feedback and modification of the execution results or other predefined rules, provide feedback through the agent feedback function, and locate the information at the time of execution based on the task ID, and store it as a fine-tuning corpus by combining the input to the large model during the call and the results in the feedback; Step 4: Determine the fine-tuning requirements of the agent: The dynamic large model access and agent construction and management module can determine its fine-tuning frequency and tendency strategy through global configuration. The dynamic large model access and agent construction and management module will globally coordinate the fine-tuning requirements of each agent according to the configured time and frequency, and dynamically plan the fine-tuning execution time of each agent; Step 5: Fine-tune the training agent: At the beginning of fine-tuning, the agent uses model reasoning to clean the data, eliminate errors, low-quality and repeated content, solidify the processed high-quality fine-tuning corpus, convert the format of the fine-tuning corpus based on the fine-tuning interface parameter format requirements of the basic model, and finally use the fine-tuning function of the basic large model for fine-tuning training; Step 6: Manage and reuse fine-tuning corpus: The stored and solidified high-quality fine-tuning corpus will be anonymized and only associated with the corresponding tasks and agents. When the dynamic large model access and agent construction and management modules are migrated to the new basic large model as a whole, each agent will re-execute the fine-tuning task according to the historical fine-tuning records, so that the agents based on the new basic large model can receive necessary training to ensure the execution effect and quality.

12. The data asset automatic generation and management system based on data operation technology according to claim 1, characterized in that: It also includes a knowledge graph that can grow autonomously and intelligently. The implementation steps are as follows: Step 1: Construct the basic structure of the graph, including industries and specific links, raw data description information, data resource description information, and composition / inclusion relationships; Step 2: Prepare the overview map content to ensure the basic user experience when there is insufficient accumulation of actual user usage data; Step 3: Graph-based data assetization intention selection: You can choose whether to use the knowledge graph to make assetization suggestions and provide operation assistance, and at the same time allow the system to collect macro-descriptive information that does not involve the specific information carried by the data; Step 4: Generate data assetization suggestions based on the graph: When the user chooses to use the relevant services and agrees to collect and share, the system will use the semantic description information of the current global data ingested during the data flow orchestration and quality control process, including the content of the accessed original data, to determine the data type and basic situation of the current user, and compare the scan results with the existing knowledge graph content in the system. The user can select the corresponding industry, business type and link to locate the Sub-Graph, and make suggestions for subsequent data operations and assetization based on the comparison results; Step 5: Expand and optimize graph content based on user information: After the user builds the data resource, the relevant information will be anonymized and transmitted to the server. The system will update existing nodes and relationships or create new nodes and relationships based on the situation.

Images