Data asset table entering method based on large language model

Abstract

The invention discloses a data asset table entering method based on a large language model, which belongs to the technical field of data processing, comprises data analysis, semantic annotation, data processing, asset evaluation and dynamic optimization, and realizes an automatic data capitalization process from data extraction, semantic annotation and structured processing to asset evaluation and optimization. The technical problems of efficient analysis, semantic annotation, data processing, asset evaluation, data asset in-table disclosure and dynamic optimization of text data and real-time streaming data are solved, the requirement for manual annotation is reduced, automatic annotation and processing of data are achieved, large-scale text data and real-time streaming data can be efficiently processed, and the data processing efficiency is improved. The method meets the low-delay requirement, provides a comprehensive score and a visual tool, helps a user to visually understand the value of the data assets, and gives higher commercial value to the data assets.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method for entering data assets into a table based on a large language model. Background Art

[0002] With the advent of the big data era, enterprises and organizations have accumulated massive amounts of structured and unstructured data, which contain rich business value. How to effectively transform this data into usable assets is a key issue in the current data management and processing field. Traditional data assetization methods mainly rely on manual annotation and rule-driven algorithms, which usually require a lot of manual intervention and expertise, and have low processing efficiency and accuracy for massive and complex data sets.

[0003] Most existing technologies rely on standardized data cleaning and formatting, but their processing capabilities for unstructured text data and real-time streaming data are limited. In the processing of text data, machine learning-based entity recognition and relationship extraction have made some progress, such as BiLSTM-CRF, BERT and other models, but for large-scale, high-dimensional complex data, there are still problems of low efficiency and low accuracy. For real-time streaming data, traditional data stream parsing technology often cannot meet the requirements of low latency and high throughput.

[0004] In addition, although current natural language processing (NLP) technology has achieved success in tasks such as sentiment analysis and text classification, in enterprise-level data assetization scenarios, how to effectively perform automated semantic annotation, aggregation processing, and dynamic optimization remains a difficulty. Summary of the invention

[0005] The purpose of the present invention is to provide a method for entering data assets into a table based on a large language model, which solves the technical problems of efficient parsing, semantic annotation, data processing, asset evaluation and dynamic optimization of text data and real-time streaming data.

[0006] To achieve the above object, the present invention adopts the following technical solution: A method for entering data assets into a table based on a large language model comprises the following steps: Step 1: Establish a data analysis module. The data analysis module obtains text data and real-time streaming data, segments the text data, and uses BiLSTM - CRF The model identifies entities and extracts entities, attributes and their relationships; Step 2: Establish a semantic annotation module, which automatically adds semantic labels to the data through zero-shot learning method and few-shot learning method; Step 3: Establish a data processing module, which aggregates entities with semantic tags and processes them into structured data assets; Step 4: Establish an asset evaluation module, which quantitatively evaluates the value of data resources, generates a data asset disclosure report, and provides a visual display; Step 5: Establish a dynamic optimization module, which continuously optimizes the data assetization process from step 1 to step 4.

[0007] Preferably, when executing step 1, the following steps are specifically included: Step 1-1: Data input, obtaining text data and real-time streaming data; text data includes corporate documents, logs and contracts; real-time streaming data includes transaction records and reports; Step 1-2: Semantic segmentation: segment the long text data into paragraphs and sentences. The segmentation function is as follows: Seg ( T )={ S 1, S 2, ..., S n} in, T Represents text data, S n represents the segmented sentence, n Indicates the number of sentences after segmentation; Step 1-3: Entity recognition, combined BiLSTM - CRF Perform key entity extraction: ; in, S represents the input sentence, h i Indicated by BiLSTM Extracted i The semantic representation of a word, e i Indicates i The entity tags of the words, CRF Represents a conditional random field, which is used to calculate the probability of a label sequence; E From the sentence S All entity label sequences extracted from ; P(E | S) Indicates that given an input sentence S Under the condition of E The probability of Extracting semantic relations between entities through relational classification models: ; in, hi , h j Is Entity e i , e j The context feature representation of r is the relationship category, W represents the weight matrix, b is the bias term; Step 1-4: Real-time streaming data analysis, through Streaming The model achieves low-latency data analysis. Streaming The model is shown in the following formula: Event ( D )={Transaction ID , amount, timestamp, …}; in D For transaction data flow; Step 1-5: Relationship extraction, using Attention - based Model, extracts the semantic relationship between entities. The specific formula is as follows: ; in, Q , K , V denote the query, key, and value vectors of an entity, respectively. d represents the vector dimension, R represents the final attention value, i and j Both represent index positions. T is the transpose operation.

[0008] Preferably, when executing step 2, the specific steps are as follows: Step 2-1: Entities and relations extracted from the data parsing module; Step 2-2: Generate predefined tags based on LLM The model infers the business scenario and generates a possible set of semantic tags, as follows: l = LLM ( Context ); in, l represents the generated semantic label, Context Represents the business scenario context; The resulting semantic label set is L = { l 1, l 2, ..., l n}; Step 2-3: Use the zero-shot learning model to match entities and relations to the optimal labels. The specific formula is as follows: P ( l ∣ e ) = Softmax ( Sim ( h e , h l )); in, Sim is the cosine similarity function, h e , h l They are the semantic embeddings of entities and tags respectively; Step 2-4: Few-shot learning correction, use a small amount of labeled data to fine-tune the results obtained by the zero-shot learning model.

[0009] Preferably, when executing step 3, the following steps are specifically included: Step 3-1: Retrieve the labeled entity and relationship data; Step 3-2: Semantic aggregation, using improved DBSCAN Clustering algorithm, classifies entities with similar semantics, and improves DBSCAN The clustering algorithm is as follows: Sim ( x , y ) > ϵ and ∣ x − y ∣ < Δ ; in, Sim represents the similarity function, specifically the cosine similarity function, x and y are all input entities, ϵ is the semantic similarity threshold, Δ is the numerical difference threshold; improved DBSCAN The clustering algorithm obtains a value of true, then x and y are of the same category; Step 3-3: Pipeline generation, designing an automated data processing pipeline, which is divided into three sub-processes: cleaning, integration, and standardization; Step 3-4: Asset conversion, introduce causal inference method, and convert it into high-value data assets based on the causal relationship between data. The formula of causal inference method is as follows: Y = αX + δ ; in, δ represents the error term, α represents the causal effect, X Indicates the flow of funds. Y Represents performance risk.

[0010] Preferably, when executing step 4, the following steps are specifically included: Step 4-1: Obtain processed structured data assets; Step 4-2: Weight allocation, based on LLM The model's scoring system assigns weights to data assets in combination with business rules and calculates the comprehensive score of data assets: ; in, w i is the business indicator weight, v i ( d ) is the i The score of each indicator; d Represents a structured data asset, n represents the number of indicators, S ( d ) represents the comprehensive score of data assets; Step 4-3: Asset sorting: prioritize data assets according to comprehensive scores and select high-value assets; Step 4-4: User visualization, generate a data asset disclosure report and provide a visual display, including the basic information of the data assets and the value assessment results of the data assets.

[0011] Preferably, when executing step 5, the specific steps are as follows: Step 5-1: Obtain user feedback and system evaluation indicators; Step 5-2: Semantic feedback: Collect feedback data through user interaction. LLM The model is fine-tuned and updated, including: Step 5-2-1: Collect a dataset with feedback annotations; Step 5-2-2: Adjust the model objective and add feedback loss function; Step 5-2-3: Use mini-batch stochastic gradient descent SGD Optimize the model parameters. The specific formula is as follows: L fine-tune = L original + βL feedback ; in, Lfeedback represents the loss based on user feedback, β is the weight coefficient, L fine-tune represents the loss function after fine-tuning, L original represents the original loss function; Step 5-3: Model optimization, using knowledge distillation method LLM The model is lightweight and optimized; Step 5-4: Asset update: design an automatic update mechanism to update the data asset library based on real-time feedback and remove low-value or outdated assets; Step 5-5: Adapt to multi-task scenarios through multi-task learning Multi - task Learning Enhance the model's adaptability to multiple business scenarios.

[0012] The method for entering data assets into a table based on a large language model described in the present invention solves the technical problems of efficient parsing, semantic annotation, data processing, asset evaluation and dynamic optimization of text data and real-time streaming data. The present invention reduces the need for manual annotation by adopting zero-sample learning and few-sample learning methods, realizes automatic annotation and processing of data, can efficiently process large-scale text data and real-time streaming data, meet low latency requirements, and combines LLM models and business rules to perform automatic evaluation of data assets, provide comprehensive scoring and visualization tools, and help users intuitively understand the value of data assets. The system can be continuously optimized according to user feedback and real-time data streams, and the accuracy and adaptability of the model are continuously improved through multi-task learning and knowledge distillation methods. The improved DBSCAN clustering algorithm and causal inference method are used to perform semantic aggregation and value transformation on data, giving data assets higher commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the main flow chart of the present invention; Figure 2 is a flow chart of step 1 of the present invention; Figure 3 is a flow chart of step 2 of the present invention; Figure 4 is a flow chart of step 3 of the present invention; Figure 5 is a flow chart of step 4 of the present invention; Figure 6 It is a flow chart of step 5 of the present invention. DETAILED DESCRIPTION

[0014] Depend on Figures 1 - 6 A method for entering data assets into a table based on a large language model is shown, comprising the following steps: Step 1: Establish a data analysis module. The data analysis module obtains text data and real-time streaming data, segments the text data, and uses BiLSTM - CRF The model identifies entities and extracts entities, attributes and their relationships; When executing step 1, the specific steps include: Step 1-1: Data input, obtaining text data and real-time streaming data; text data includes corporate documents, logs and contracts; real-time streaming data includes transaction records and reports; Step 1-2: Semantic segmentation: segment the long text data into paragraphs and sentences. The segmentation function is as follows: Seg ( T )={ S 1, S 2, ..., S n} in, T Represents text data, S n represents the segmented sentence, n Indicates the number of sentences after segmentation; Step 1-3: Entity recognition, combined BiLSTM - CRF Perform key entity extraction: ; in, S represents the input sentence, h i Indicated by BiLSTM Extracted i The semantic representation of a word, e i Indicates i The entity tags of the words, CRF Represents a conditional random field, which is used to calculate the probability of a label sequence; E From the sentence S All entity label sequences extracted from ; P(E | S) Indicates that given an input sentence S Under the condition of E The probability of Extracting semantic relations between entities through relational classification models: ; in, h i , h j Is Entity e i , ej The context feature representation of r is the relationship category, W represents the weight matrix, b is the bias term; Step 1-4: Real-time streaming data analysis, through Streaming The model achieves low-latency data analysis. Streaming The model is shown in the following formula: Event ( D )={Transaction ID , amount, timestamp, …}; in D For transaction data flow; Step 1-5: Relationship extraction, using Attention - based Model, extracts the semantic relationship between entities. The specific formula is as follows: ; in, Q , K , V denote the query, key, and value vectors of an entity, respectively. d represents the vector dimension, R represents the final attention value, i and j Both represent index positions. T is the transpose operation.

[0015] In this example, an Attention-based model is used to first calculate the similarity between the query and the key for each pair of entities, generate the attention weight (RRR), and then obtain the relationship representation between entities through weighted summation. This representation reflects the semantic association between entities and is further used in the relationship extraction task to identify the specific relationship between entities.

[0016] In relation extraction tasks, the query and key are usually representations of different entities. For example, suppose you want to extract the relationship between two entities in a text, you can use the word vectors or context vectors of the two entities as the query and key.

[0017] By calculating the similarity between the query vector and the key vector, the attention weight matrix is ​​obtained. This similarity reflects the correlation between entities in the semantic space.

[0018] Based on the attention weight (i.e., R), the value vector is weighted and summed to finally extract the relational representation between entities. This representation contains the semantic relationship between entities.

[0019] Step 2: Establish a semantic annotation module, which automatically adds semantic labels to the data through zero-shot learning method and few-shot learning method; When executing step 2, the specific steps are as follows: Step 2-1: Entities and relations extracted from the data parsing module; Step 2-2: Generate predefined tags based on LLM The model infers the business scenario and generates a possible set of semantic tags, as follows: l = LLM ( Context ); in, l represents the generated semantic label, Context Represents the business scenario context; The resulting semantic label set is L = { l 1, l 2, ..., l n}; Step 2-3: Use the zero-shot learning model to match entities and relations to the optimal labels. The specific formula is as follows: P ( l ∣ e ) = Softmax ( Sim ( h e , h l )); in, Sim is the cosine similarity function, h e , h l They are the semantic embeddings of entities and tags respectively; Zero-shot learning refers to the ability of a model to make predictions or inferences without labeled data for a specific task. Inference is usually performed by converting the task into a general text representation (such as through a large language model LLM).

[0020] In the zero-shot learning model, P ( l ∣ e ) indicates that under zero-sample conditions, the entity e Assigned tags l The probability of Step 2-4: Few-shot learning correction, use a small amount of labeled data to fine-tune the results obtained by the zero-shot learning model.

[0021] In this embodiment, when there is a small amount of labeled data, the prediction results are corrected by fine-tuning the zero-shot learning model. In this case, the small sample data (usually labeled entity or relationship pairs) is used to adjust the parameters of the model so that it can make predictions based on these small samples more accurately.

[0022] The fine-tuning process is to compare the output of the model with the labels of a small amount of annotated data by adjusting the loss function. For example, using cross entropy loss to update the parameters of the model: ; in, y i is the distribution of true labels (few sample labels), is the label distribution predicted by the model.

[0023] The specific steps of fine-tuning are as follows: Step 2-4-1: Train the model with a small amount of labeled data, calculate the difference between the predicted result and the true label, and optimize it using the cross entropy loss function; Step 2-4-2: Use a mini-batch stochastic gradient descent (SGD) optimizer to update model parameters; Step 2-4-3: Use the representation generated by the pre-trained large language model (LLM) as the initial state and adjust the model parameters so that the model can better adapt to the specific task.

[0024] Step 3: Establish a data processing module, which aggregates entities with semantic tags and processes them into structured data assets; When executing step 3, the specific steps include: Step 3-1: Retrieve the labeled entity and relationship data; Step 3-2: Semantic aggregation, using improved DBSCAN Clustering algorithm, classifies entities with similar semantics, and improves DBSCAN The clustering algorithm is as follows: Sim ( x , y ) > ϵ and ∣ x − y ∣ < Δ ; in, Sim represents the similarity function, specifically the cosine similarity function, x and y are all input entities, ϵ is the semantic similarity threshold, Δ is the numerical difference threshold; improvedDBSCAN The clustering algorithm obtains a value of true, then x and y are of the same category; Step 3-3: Pipeline generation, designing an automated data processing pipeline, which is divided into three sub-processes: cleaning, integration, and standardization; In this embodiment, the goal of pipeline generation is to design an automated data processing process to process data through three sub-processes: cleaning, integration, and standardization. These sub-processes can ensure that the raw data can be converted into structured data suitable for data assetization. The following are the specific steps and functions of each sub-process: Data cleaning is the first step in data preprocessing, which aims to remove or correct errors, duplications, missing values ​​and inconsistencies in the data. It ensures that the input data meets the quality standards and provides a clean basis for subsequent processing. The specific tools used are the Pandas library in Python (such as .drop_duplicates(), .fillna() and other functions), and regular expressions are used to clean text data. The specific operations are as follows: Deduplication: Delete duplicate data records to ensure that each piece of data is unique. For example, remove duplicate records of the same transaction or duplicate paragraphs of the same text; Missing value filling: For missing fields, interpolation, mean, median filling or other appropriate strategies can be used to fill in missing values. For text data, missing entities or relationships may need to be filled in through context; Outlier detection: Detect and correct outliers in data. For example, for transaction data, abnormal fluctuations in amounts may need to be corrected or deleted; Format standardization: Standardize the formats of date, time, currency and other fields to ensure that the data remains consistent throughout the entire processing process. For example, convert all dates to the "YYYY-MM-DD" format.

[0025] The purpose of data integration is to merge data from different sources into a unified data set, ensuring that all related data can be processed in the same data model. This step usually involves merging multiple data sources (such as databases, log files, APIs, etc.). The specific tools used are database query language (SQL) for cross-table and cross-database integration, ETL tools (such as Apache Nifi, Talend, etc.) for data extraction, conversion and loading, and graph databases (such as Neo4j) can be used for relationship modeling and merging. The specific operations are as follows: Data source merging: merging data sources from different systems or formats. For example, merging entities extracted from corporate documents with data from real-time transaction streams to form a complete database; Entity mapping: Map the same entities in different data sources. For example, extract the customer ID and name from different databases and ensure that they point to the same customer; Relationship fusion: Process relationships in different data sources to ensure they remain consistent after merging. For example, if two data sources both mention the transaction relationship of "Customer A", this information needs to be merged into a unified representation; Time alignment: For time series data from multiple sources, their timestamps need to be aligned to ensure the consistency of data from different sources in the time dimension.

[0026] The purpose of data standardization is to convert data into a unified format and structure to make it suitable for analysis and modeling. At this stage, we convert data of different dimensions and formats into a unified standard. The tools used are StandardScaler or MinMaxScaler in scikit-learn. Text data preprocessing can use NLP toolkits such as spaCy or NLTK for vocabulary normalization. For field mapping and encoding of structured data, Pandas and SQL can achieve data cleaning and conversion. The specific steps are as follows: Field standardization: unify all field names, units, data types, etc. For example, map the "transaction_amount" and "amount" fields to the same standard field "amount".

[0027] Data scaling and normalization: Normalize numerical data (e.g., normalize the amount field to ensure that it has the same scale) to ensure that the values ​​from different data sources can be compared. For example, use Z-score normalization or min-max normalization methods.

[0028] Vocabulary standardization: For text data, perform vocabulary standardization. For example, normalize synonyms (such as treating "purchase" and "buy" as the same concept) and remove stop words.

[0029] Coding standardization: Encoding standardization of categorical variables ensures that each category in the data has a consistent representation method. For example, map "male" and "female" in the "gender" field to numbers (such as 0 and 1), or use One-Hot Encoding to convert to multiple binary features.

[0030] Step 3-4: Asset conversion, introduce causal inference method, and convert it into high-value data assets based on the causal relationship between data. The formula of causal inference method is as follows: Y = αX + δ ; in, δ represents the error term, α represents the causal effect, X Indicates the flow of funds. Y Represents performance risk.

[0031] In this example, we need to determine which variables are possible causal participants. For example, suppose we are evaluating the impact of capital flow (X) on performance risk (Y), and select key variables: capital flow, performance risk, etc.

[0032] Based on data analysis and domain knowledge, a causal inference model is constructed. A simple linear regression model (such as the above formula) can be used to represent the causal relationship, or more variables and control items can be introduced according to the complexity of the data. For example, other factors (such as market conditions, enterprise size, etc.) may need to be introduced into the model as control variables.

[0033] Use appropriate causal inference methods to estimate causal effects. For example, estimate the value of α through a regression model to represent the impact of capital flow on performance risk.

[0034] After estimating the causal effects, you can assess the contribution of these effects to business objectives. For example, if α represents the impact of cash flow on performance risk, you can use this information to assess the performance risk of the enterprise or system and rank data assets based on this assessment.

[0035] By combining causal effects with other business indicators (such as cash flow, profit, etc.), high-value data assets can be generated. These data assets can help decision makers develop risk management strategies, optimize cash flow, etc.

[0036] Step 4: Establish an asset evaluation module, which quantitatively evaluates the value of data resources, generates a data asset disclosure report, and provides a visual display; When executing step 4, the specific steps include: Step 4-1: Obtain processed structured data assets; Step 4-2: Weight allocation, based on LLM The model's scoring system assigns weights to data assets in combination with business rules and calculates the comprehensive score of data assets: ; in, w i is the business indicator weight, v i ( d ) is the i The score of each indicator; d Represents a structured data asset,n represents the number of indicators, S ( d ) represents the comprehensive score of data assets; Step 4-3: Asset sorting: prioritize data assets according to comprehensive scores and select high-value assets; Step 4-4: User visualization, generate a data asset disclosure report and provide a visual display, including the basic information of the data assets and the value assessment results of the data assets.

[0037] Step 5: Establish a dynamic optimization module, which continuously optimizes the data assetization process from step 1 to step 4.

[0038] When executing step 5, the specific steps are as follows: Step 5-1: Obtain user feedback and system evaluation indicators; In this embodiment, the collection of user feedback is the basis for optimizing the model, mainly in the following ways: Direct user feedback: Collect user feedback on the data assetization process directly through interaction between users and the system.

[0039] Direct user feedback can be achieved through: Feedback button / form: Provide a feedback button or form in the system so that users can submit their opinions, questions, or comments on the data asset assessment results at any time (e.g., ratings, suggestions, improvement requirements, etc.); Rating system: Allows users to rate each data asset generated by the system (e.g. 0-5 points) and add additional comments based on their experience; Questionnaires: Questionnaires are sent to users regularly to collect their feedback on data asset management, data evaluation, data processing, etc.

[0040] Indirect user feedback: User feedback can be obtained through the following indirect methods: User behavior analysis: By analyzing the user's behavior in the system (such as click rate, access frequency, length of stay, etc.), we can infer the user's interest in and importance of certain data assets. Frequent viewing and operation of specific assets by users can be regarded as positive feedback, while neglect of certain data assets can be regarded as negative feedback.

[0041] Decision result feedback: When users make decisions in the system (such as selecting a specific data asset), the decision results can be recorded and used to evaluate the quality of the current system output. If the decision results do not meet the user's expectations, it can reflect that the system needs further optimization.

[0042] System evaluation indicators are used to quantify the current performance of the system and evaluate the accuracy, reliability, efficiency, etc. of the model. Key indicators include but are not limited to: Prediction accuracy: measures the accuracy of the model's evaluation results of data assets, that is, the degree of match between the predicted results and the actual results; Recall and precision: In the process of data processing, semantic annotation and relationship extraction, the system's recall and precision are also very critical evaluation indicators; Response time: evaluates the system's response speed to user requests, especially the real-time nature of data processing and evaluation; Computational efficiency: evaluates the efficiency of the system when processing large amounts of data, such as processing speed, resource consumption (CPU, memory, etc.); Satisfaction rating: Collect user satisfaction with the data asset management system through user reviews, rating systems or questionnaires; User activity: measures the user activity of the system and determines the popularity of the system by monitoring information such as user logins and usage frequency.

[0043] Step 5-2: Semantic feedback: Collect feedback data through user interaction. LLM The model is fine-tuned and updated, including: Step 5-2-1: Collect a dataset with feedback annotations; In this embodiment, a data set with explicit feedback is collected through user interaction. User feedback can be based on the rating of data assets, user evaluation of model output, error identification, requirement description, etc. Generally, user feedback can include positive (e.g., correct predictions or useful data annotations) and negative feedback (e.g., wrong predictions or labels that do not meet business requirements). These feedbacks will be annotated on the data set as input for model fine-tuning.

[0044] For example, users can mark the quality of model outputs with ratings of “high” or “low”; users can annotate certain data items with errors or irrelevant information.

[0045] Collecting these labeled datasets with feedback will serve as training data for subsequent fine-tuning.

[0046] Step 5-2-2: Adjust the model objective and add feedback loss function; In this embodiment, the feedback loss function is introduced to adjust the model's goal so that it not only optimizes the original task (such as text classification, relationship extraction, etc.), but also considers the user's feedback information. In this embodiment, a loss function needs to be designed to combine the model's original goal with the feedback-based optimization goal.

[0047] Step 5-2-3: Use mini-batch stochastic gradient descent SGD to optimize model parameters. The specific formula is as follows: L fine-tune = L original + βL feedback ; in, L feedback represents the loss based on user feedback, β is the weight coefficient, L fine-tune represents the loss function after fine-tuning, L original represents the original loss function; L fine-tune The fine-tuned loss function is used to find a balance between the feedback data and the loss of the original task, and is minimized by SGD. L fine-tune , and then update the parameters of the model.

[0048] Step 5-3: Model optimization, using knowledge distillation method LLM The model is lightweight and optimized; In this embodiment, the knowledge distillation method is a prior art and will not be described in detail.

[0049] Step 5-4: Asset update: design an automatic update mechanism to update the data asset library based on real-time feedback and remove low-value or outdated assets; In this embodiment, the specific steps of step 5-4 are as follows: Step 5-4-1: First, the system needs to receive feedback information from different sources in real time. Usually, this feedback can come from user behavior, system logs, changes in business indicators, etc. For example: User behavior feedback: the frequency with which users query, download, modify, or use certain data assets; System performance feedback: usage rate of data assets, access frequency, query duration, etc. Business feedback: changes in business department demand for data assets or the emergence of new business scenarios.

[0050] Step 5-4-2: Based on real-time feedback data, design a dynamic asset evaluation mechanism to evaluate the current value of data assets. This includes the following: Value scoring: Assign a value score to each data asset based on preset evaluation criteria (e.g., timeliness, quality, relevance, etc. of the data); Timeliness assessment: Determine whether the data is outdated based on its update time and usage frequency. Data with poor timeliness should be considered of low value. User demand matching: Based on current business needs and user feedback, evaluate the fit between data assets and actual needs; Step 5-4-3: Based on real-time feedback and evaluation results, set some rules to determine the update strategy of the data asset library. The rules include: Elimination of assets with values ​​below the threshold: When the comprehensive score or value of a data asset is below the set threshold, the asset can be automatically eliminated or marked as obsolete. For example, if the query frequency of a certain data drops to a certain percentage or the relevance is insufficient, it can be considered that its value has decreased; Eliminate expired assets: Regularly check the update time of each data asset. If some data has not been updated for a long time and does not reflect new business needs, it can be considered that the data is outdated. Priority enhancement: For data with higher value or that meets current business needs, its priority can be automatically increased to ensure that it is processed or used first.

[0051] Step 5-4-4: Once the data asset evaluation and elimination or promotion rules are determined, automatic updates can be achieved through the following methods: Scheduled tasks: Set up data update tasks for regular inspection and evaluation, and regularly re-evaluate and update the data asset library; Event triggering: When there is new user feedback or business demand changes, the real-time update task is triggered to automatically adjust the asset value in the data asset library; Integrated data quality check: Every time the data is updated, data quality checks (such as data completeness, accuracy, etc.) are combined to ensure that the updated data meets the expected quality requirements; Step 5-4-5: In order to avoid losing historical data due to excessive automated updates, a version control mechanism can be implemented: Version management: Every time a data asset is updated, the data version before the update is saved to facilitate tracing and comparison of data changes at different time points; Data History: Keep a history of every asset update, including deletion, modification, and re-labeling of historical data, to ensure that decisions about outdated or low-value data are well-founded.

[0052] Step 5-5: Adapt to multi-task scenarios and enhance the model's adaptability to multiple business scenarios through multi-task learning.

[0053] When executing step 5-5, the multi-task learning enhanced model specifically includes: Step 5-5-1: Identify the task types for the multiple tasks to be handled and define the goals of each task, for example: Data classification tasks: identifying data types, label classification, etc.; Data regression tasks: predicting numerical results of data, such as risk scores, transaction amount predictions, etc. Relation extraction task: extracting the relationship between entities in the data; Data evaluation tasks: evaluate the value, accuracy, etc. of data; Step 5-5-2: In multi-task learning, the Multi-task Learning enhancement model includes shared layers and task-specific layers: Shared layers: By sharing part of the network structure (for example, the convolutional layers and fully connected layers in the first few layers), the model can learn the common features between multiple tasks. This shared knowledge is the basis for different tasks to help each other; Task-specific layers: Each task has its own dedicated layer for specific output predictions. For example, for a classification task, the last output layer is a classifier; for a regression task, the last output layer is a regression model; Step 5-5-3: Define a suitable loss function to combine the weighted losses of multiple tasks. The loss function for each task can be different. The specific design can refer to the following methods: Task-independent loss: The loss function for each task is calculated separately, for example: The classification task uses the cross-entropy loss function (Cross-Entropy Loss); The regression task uses the mean squared error loss function (Mean Squared Error); Weighted loss: The losses of different tasks can be weighted by weight coefficients and combined into an overall loss function. The formula for the overall loss is as follows: ; in, L i It's a task i The loss function is λ i is the weight of each task. By adjusting the weight coefficient, the model can balance the learning objectives between various tasks.

[0054] Step 5-5-4: In multi-task learning, there may be a certain correlation between different tasks. The higher the correlation between tasks, the more useful the shared knowledge is. By analyzing the correlation between tasks, we can determine the shared features or network structure levels. For example: Strong task correlation: If the tasks are highly correlated, more model layers can be shared, reducing the design of independent models; Weak task association: If the correlation between tasks is low, fewer shared layers can be used to retain more independent model structures.

[0055] Step 5-5-5: Use a joint training strategy to optimize the shared layers and task-specific layers. Specifically, a common network model is trained by optimizing the loss function of all tasks in each training step. The model will benefit from learning multiple tasks at the same time.

[0056] Step 5-5-6: Prepare the dataset for each task, ensuring that the data for each task can be merged or share the same input features (for example, different labels of the same dataset can be used as targets for different tasks); Design a neural network model with shared layers and task-specific layers. The shared layers can be convolutional layers, fully connected layers, etc., and the task-specific layers are usually output layers.

[0057] Training process: The loss function of each task is weighted and summed to obtain the total loss function; Use the total loss for back propagation and optimization to gradually adjust the parameters of the network; Regularly evaluate the performance of each task to ensure that the objectives of each task are balanced and optimized.

[0058] In this embodiment, through multi-task learning, the model can effectively adapt to multiple task scenarios. For example, in financial data analysis, tasks such as transaction prediction, credit scoring, and risk assessment can be performed simultaneously. In enterprise data management, multiple tasks may include data classification, data cleaning, and data value assessment.

[0059] The method for entering data assets into a table based on a large language model described in the present invention solves the technical problems of efficient parsing, semantic annotation, data processing, asset evaluation and dynamic optimization of text data and real-time streaming data. The present invention reduces the need for manual annotation by adopting zero-sample learning and few-sample learning methods, realizes automatic annotation and processing of data, can efficiently process large-scale text data and real-time streaming data, meet low latency requirements, and combines LLM models and business rules to perform automatic evaluation of data assets, provide comprehensive scoring and visualization tools, and help users intuitively understand the value of data assets. The system can be continuously optimized according to user feedback and real-time data streams, and the accuracy and adaptability of the model are continuously improved through multi-task learning and knowledge distillation methods. The improved DBSCAN clustering algorithm and causal inference method are used to perform semantic aggregation and value transformation on data, giving data assets higher commercial value.

Claims

1. A method for entering data assets into a table based on a large language model, characterized in that: The steps include: Step 1: Establish a data analysis module. The data analysis module obtains text data and real-time streaming data, segments the text data, and uses BiLSTM - CRF The model identifies entities and extracts entities, attributes and their relationships; Step 2: Establish a semantic annotation module, which automatically adds semantic labels to the data through zero-shot learning method and few-shot learning method; Step 3: Establish a data processing module, which aggregates entities with semantic tags and processes them into structured data assets; Step 4: Establish an asset evaluation module, which quantitatively evaluates the value of data resources, generates a data asset disclosure report, and provides a visual display; Step 5: Establish a dynamic optimization module, which continuously optimizes the data assetization process from step 1 to step 4.

2. The method for entering data assets into a table based on a large language model according to claim 1, characterized in that: When executing step 1, the specific steps include: Step 1-1: Data input, obtaining text data and real-time streaming data; text data includes corporate documents, logs and contracts; real-time streaming data includes transaction records and reports; Step 1-2: Semantic segmentation: segment the long text data into paragraphs and sentences. The segmentation function is as follows: Seg ( T )={ S 1， S 2，…， S n} in, T Represents text data, S n represents the segmented sentence, n Indicates the number of sentences after segmentation; Step 1-3: Entity recognition, combined BiLSTM - CRF Perform key entity extraction: ； in, S represents the input sentence, h i Indicated by BiLSTM Extracted i The semantic representation of a word, e i Indicates i The entity tags of the words, CRF Represents a conditional random field, which is used to calculate the probability of a label sequence; E From the sentence S All entity label sequences extracted from ; P(E | S) Indicates that given an input sentence S Under the condition of E The probability of Extracting semantic relations between entities through relational classification models: ； in, h i , h j Is Entity e i , e j The context feature representation of r is the relationship category, W represents the weight matrix, b is the bias term; Step 1-4: Real-time streaming data analysis, through Streaming The model achieves low-latency data analysis. Streaming The model is shown in the following formula: Event ( D )={Transaction ID , amount, timestamp, …}; in D For transaction data flow; Step 1-5: Relationship extraction, using Attention - based Model, extracts the semantic relationship between entities. The specific formula is as follows: ； in, Q , K , V denote the query, key, and value vectors of an entity, respectively. d represents the vector dimension, R represents the final attention value, i and j Both represent index positions. T is the transpose operation.

3. The method for entering data assets into a table based on a large language model according to claim 1, characterized in that: When executing step 2, the specific steps are as follows: Step 2-1: Entities and relations extracted from the data parsing module; Step 2-2: Generate predefined tags based on LLM The model infers the business scenario and generates a possible set of semantic tags, as follows: l = LLM ( Context )； in, l represents the generated semantic label, Context Represents the business scenario context; The resulting semantic label set is L = { l 1, l 2, ..., l n }; Step 2-3: Use the zero-shot learning model to match entities and relations to the optimal labels. The specific formula is as follows: P ( l ∣ e ) = Softmax ( Sim ( h e ， h l ))； in, Sim is the cosine similarity function, h e , h l They are the semantic embeddings of entities and tags respectively; Step 2-4: Few-shot learning correction, use a small amount of labeled data to fine-tune the results obtained by the zero-shot learning model.

4. The method for entering data assets into a table based on a large language model according to claim 1, characterized in that: When executing step 3, the specific steps include: Step 3-1: Retrieve the labeled entity and relationship data; Step 3-2: Semantic aggregation, using improved DBSCAN Clustering algorithm, classifies entities with similar semantics, and improves DBSCAN The clustering algorithm is as follows: Sim ( x ， y ) > ϵand ∣ x − y ∣ < Δ ； in, Sim represents the similarity function, specifically the cosine similarity function, x and y are all input entities, ϵ is the semantic similarity threshold, Δ is the numerical difference threshold; improved DBSCAN The clustering algorithm obtains a value of true, then x and y are of the same category; Step 3-3: Pipeline generation, designing an automated data processing pipeline, which is divided into three sub-processes: cleaning, integration, and standardization; Step 3-4: Asset conversion, introduce causal inference method, and convert it into high-value data assets based on the causal relationship between data. The formula of causal inference method is as follows: Y = αX + δ ； in, δ represents the error term, α represents the causal effect, X Indicates the flow of funds. Y Represents performance risk.

5. The method for entering data assets into a table based on a large language model according to claim 1, characterized in that: When executing step 4, the specific steps include: Step 4-1: Obtain processed structured data assets; Step 4-2: Weight allocation, based on LLM The model's scoring system assigns weights to data assets in combination with business rules and calculates the comprehensive score of data assets: ； in, w i is the business indicator weight, v i ( d ) is the i The score of each indicator; d Represents a structured data asset, n represents the number of indicators, S ( d ) represents the comprehensive score of data assets; Step 4-3: Asset sorting: prioritize data assets according to comprehensive scores and select high-value assets; Step 4-4: User visualization, generate a data asset disclosure report and provide a visual display, including the basic information of the data assets and the value assessment results of the data assets.

6. The method for entering data assets into a table based on a large language model according to claim 1, characterized in that: When executing step 5, the specific steps are as follows: Step 5-1: Obtain user feedback and system evaluation indicators; Step 5-2: Semantic feedback: Collect feedback data through user interaction. LLM The model is fine-tuned and updated, including: Step 5-2-1: Collect a dataset with feedback annotations; Step 5-2-2: Adjust the model objective and add feedback loss function; Step 5-2-3: Use mini-batch stochastic gradient descent SGD Optimize the model parameters. The specific formula is as follows: L fine-tune = L original + βL feedback ； in, L feedback represents the loss based on user feedback, β is the weight coefficient, L fine-tune represents the loss function after fine-tuning, L original represents the original loss function; Step 5-3: Model optimization, using knowledge distillation method LLM The model is lightweight and optimized; Step 5-4: Asset update: design an automatic update mechanism to update the data asset library based on real-time feedback and remove low-value or outdated assets; Step 5-5: Adapt to multi-task scenarios through multi-task learning Multi - taskLearning Enhance the model's adaptability to multiple business scenarios.

Images