Interactive data analysis method based on large model

Abstract

The present application proposes an interactive data analysis method based on a large model, comprising obtaining text samples and corresponding training labels to construct a training set; performing format conversion on the text samples in the training set to convert the training set into an input sequence sample of the large model; inputting the input sequence into the large model so that the large model performs a preset dimension analysis on the input sequence to obtain an analysis result; calculating the loss function of the large model based on the analysis result and the training labels in the training set to train the large model based on the loss function until the training is completed; obtaining a text to be processed and converting it into a sequence to be processed, so as to input it into the trained large model to perform a preset dimension analysis on the text to be processed and obtain a corresponding analysis result.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an interactive data analysis method based on a large model. Background Art

[0002] In the current information age, with the rapid development of the Internet and social media, the amount of data, especially unstructured text data, has shown explosive growth. These massive text data contain valuable information and knowledge, which are of great value in the fields of market analysis, public opinion monitoring, personalized recommendations, etc. However, effectively extracting useful information from these data faces huge challenges, which are mainly reflected in the following aspects:

[0003] The contradiction between processing scale and efficiency: Traditional rule-based and simple statistical methods are unable to cope with large-scale text data and it is difficult to conduct high-dimensional analysis efficiently and accurately.

[0004] Limitations of semantic understanding: The complexity and diversity of text data require analytical methods to not only deal with the grammatical level, but also have a deep understanding of the semantic level, while traditional analytical tools often lack sufficient contextual understanding and reasoning capabilities.

[0005] Model adaptability and generalization ability: Faced with ever-changing language habits and emerging topics, existing models often need to be frequently updated and adjusted, lacking good adaptability and flexibility to generalize to new fields.

[0006] Interactivity and customization requirements: Users’ demands for data analysis are becoming increasingly diverse, and they expect to be able to flexibly customize the analysis dimensions and depth according to specific scenarios. However, traditional methods are difficult to meet such highly personalized interactive analysis requirements.

[0007] In response to the above problems, existing solutions often focus on optimizing a single technical point, lacking a systematic approach to simultaneously address the comprehensive challenges of large-scale processing, deep semantic understanding, model generalization, and interactive analysis. In particular, in the process of using pre-trained large models for text analysis, how to effectively integrate data preprocessing, model training, performance optimization, and user interaction remains a technical problem that needs to be solved urgently. Summary of the invention

[0008] The purpose of this application is to propose an interactive data analysis method based on a large model to solve or alleviate the technical problems existing in the prior art.

[0009] The technical solutions provided by the embodiments of this application are as follows:

[0010] An interactive data analysis method based on a large model, comprising:

[0011] Obtain text samples and corresponding training labels to build a training set;

[0012] Performing format conversion on text samples in the training set to convert the training set into input sequence samples of a large model;

[0013] Inputting the input sequence into the large model, so that the large model analyzes the input sequence in a preset dimension to obtain an analysis result;

[0014] Based on the analysis result and the training labels in the training set, a loss function of the large model is calculated to train the large model based on the loss function until the training is completed;

[0015] The text to be processed is obtained and converted into a sequence to be processed, so as to be input into the trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results.

[0016] Optionally, obtaining text samples and corresponding training labels and constructing a training set includes:

[0017] Performing structural analysis on the sample data source to extract text class names and text attribute labels in the target data source;

[0018] The text class names and text attribute labels are mapped to text samples and corresponding training labels respectively to construct a training set.

[0019] Optionally, performing structural analysis on the sample data source to extract text class names and text attribute labels in the target data source includes:

[0020] Based on the constructed DOM element sequence, the DOM API is enabled to perform structural analysis on the sample data source, and the text class name and text attribute label in the target data source are extracted.

[0021] Optionally, mapping the text class names and text attribute labels to text samples and corresponding training labels, respectively, to construct a training set includes: performing virtual dynamic processing on the text class names and text attribute labels to map the text class names and text attribute labels to text samples and corresponding training labels, respectively.

[0022] Optionally, the text class names and text attribute labels are virtually dynamic processed to map the text class names and text attribute labels to text samples and corresponding training labels, respectively, including: based on a set dynamic rule engine, the text class names and text attribute labels are virtually dynamic processed to generate a dynamic rule text sequence; decision analysis and voting are performed on the dynamic rule text sequence to map the text class names and text attribute labels to text samples and corresponding training labels, respectively.

[0023] Optionally, the format conversion of the text samples in the training set to convert the training set into input sequence samples of the large model includes:

[0024] Performing word segmentation processing on the text samples in the training set to obtain a number of word units;

[0025] Performing frequency statistics on the plurality of word units to determine high-frequency words and forming a high-frequency vocabulary list accordingly, wherein the high-frequency words are word units whose occurrence frequency is greater than a set frequency threshold;

[0026] The high-frequency vocabulary is encoded to obtain a high-frequency word index sequence to serve as an input sequence sample for converting the training set into a large model.

[0027] Optionally, encoding the high-frequency vocabulary to obtain a high-frequency word index sequence includes:

[0028] Using the Word Embeddings model, determining the semantic relevance of high-frequency words in the high-frequency vocabulary;

[0029] Based on the set adaptive coding model, encoding the high-frequency vocabulary to obtain a group code;

[0030] Based on the semantic association of the high-frequency words in the high-frequency vocabulary, built-in sequence start markers and end markers are added to the group code to generate a high-frequency word index sequence.

[0031] Optionally, the using of the Word Embeddings model to determine the semantic association of the high-frequency words in the high-frequency vocabulary includes: using the Word Embeddings model to convert each high-frequency word in the high-frequency vocabulary into a vector representation; calculating the similarity between different vector representations, so as to evaluate the semantic association between the high-frequency words in the high-frequency vocabulary based on the similarity and generate a semantic association graph.

[0032] Optionally, the adding of built-in sequence start markers and end markers in the group code based on the semantic association of the high-frequency words in the high-frequency vocabulary to generate a high-frequency word index sequence includes:

[0033] Generate a semantic association matrix based on the semantic association of high-frequency words in the high-frequency vocabulary;

[0034] Based on the semantic association matrix, cluster analysis is performed on the high-frequency words in the high-frequency vocabulary to determine group representative words therein;

[0035] The semantic flow and logical structure of the group representative word are expanded to determine the sequence start mark position and the end mark position, so as to add the built-in sequence start mark and end mark to the group code corresponding to the group representative word.

[0036] Optionally, the input sequence sample is input into the large model, so that the large model performs a preset dimension analysis on the input sequence to obtain an analysis result;

[0037] The word segmenter in the large model divides the input sequence sample into a plurality of vocabulary units, and assigns a position identifier to each vocabulary unit according to the position in the input sequence sample to form a standard vocabulary unit;

[0038] Based on the preset dimension, the transformer layer in the large model is triggered to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result.

[0039] Optionally, based on the preset dimension, triggering the transformer layer in the large model to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result, including:

[0040] Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary unit to capture the self-attention semantic space characteristics of the standard vocabulary unit, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary unit to capture the cross-attention semantic space characteristics of the standard vocabulary unit, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

[0041] Optionally, the calculating a loss function of the large model based on the analysis result and the training labels in the training set, so as to train the large model based on the loss function until the training is completed, comprises:

[0042] Based on the analysis results and the training labels in the training set, the loss function of the large model is calculated. If the value of the loss function is less than the set loss threshold, the training is completed. Otherwise, the process jumps to the step of formatting the text samples in the training set to convert the training set into input sequence samples of the large model, so as to retrain the large model.

[0043] Optionally, the obtaining of the text to be processed and converting it into a sequence to be processed, so as to input it into a trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results, includes:

[0044] Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary units in the sequence to be processed to capture the self-attention semantic space features of the standard vocabulary units, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary units in the sequence to be processed to capture the cross-attention semantic space features of the standard vocabulary units, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

[0045] The interactive data analysis method based on a large model proposed in this application has the following technical advantages:

[0046] (1) By adopting large models (such as pre-trained models of the Transformer architecture, such as BERT and GPT series), this method fully utilizes the deep learning capabilities of the models. These models are pre-trained on large-scale data and can efficiently process massive amounts of text data. Large models can capture long-distance dependencies through the self-attention mechanism, improve the efficiency of processing complex text sequences, and solve the bottleneck of traditional methods in large-scale data processing. Continuous optimization of the model during the training process ensures that the processing speed is improved while maintaining the accuracy of high-dimensional analysis, balancing the contradiction between scale and efficiency.

[0047] (2) Large models learn rich language representations through deep neural network architectures, especially in the large-scale unsupervised pre-training stage, which not only include statistical features at the vocabulary level, but also capture syntactic and semantic information at a deeper level. This means that they can better understand the context of the text and perform effective semantic reasoning, addressing the limitations of traditional methods in understanding complex contexts and deep meanings.

[0048] (3) The fine-tuning step of the large model in this method (based on task-specific training) not only improves the accuracy of the model in a specific field, but also enhances the generalization ability of the model. Since the large model has learned a wide range of language structures and patterns in advance, it can quickly adapt to new language habits or topics with only a small amount of labeled data, without the need for frequent and drastic adjustments to the model, thereby improving the model's adaptability and flexibility in application in new fields.

[0049] (4) This method supports interactive data analysis, and users can flexibly customize the analysis dimensions according to their specific needs. By directly inputting the text to be analyzed into the trained large model, users can instantly obtain results for different analysis dimensions (such as sentiment, theme, etc.), achieving a highly personalized analysis experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0051] Figure 1 A flowchart of an interactive data analysis method based on a large model according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0053] Figure 1 The following is a flow chart of an interactive data analysis method based on a large model according to an embodiment of the present application. An interactive data analysis method based on a large model includes:

[0054] Obtain text samples and corresponding training labels to build a training set;

[0055] Performing format conversion on text samples in the training set to convert the training set into input sequence samples of a large model;

[0056] Inputting the input sequence into the large model, so that the large model analyzes the input sequence in a preset dimension to obtain an analysis result;

[0057] Based on the analysis result and the training labels in the training set, a loss function of the large model is calculated to train the large model based on the loss function until the training is completed;

[0058] The text to be processed is obtained and converted into a sequence to be processed, so as to be input into the trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results.

[0059] The interactive data analysis method based on a large model proposed in this application has the following technical advantages:

[0060] (1) By adopting large models (such as pre-trained models of the Transformer architecture, such as BERT and GPT series), this method fully utilizes the deep learning capabilities of the models. These models are pre-trained on large-scale data and can efficiently process massive amounts of text data. Large models can capture long-distance dependencies through the self-attention mechanism, improve the efficiency of processing complex text sequences, and solve the bottleneck of traditional methods in large-scale data processing. Continuous optimization of the model during the training process ensures that the processing speed is improved while maintaining the accuracy of high-dimensional analysis, balancing the contradiction between scale and efficiency.

[0061] (2) Large models learn rich language representations through deep neural network architectures, especially in the large-scale unsupervised pre-training stage, which not only include statistical features at the vocabulary level, but also capture syntactic and semantic information at a deeper level. This means that they can better understand the context of the text and perform effective semantic reasoning, addressing the limitations of traditional methods in understanding complex contexts and deep meanings.

[0062] (3) The fine-tuning step of the large model in this method (based on task-specific training) not only improves the accuracy of the model in a specific field, but also enhances the generalization ability of the model. Since the large model has learned a wide range of language structures and patterns in advance, it can quickly adapt to new language habits or topics with only a small amount of labeled data, without the need for frequent and drastic adjustments to the model, thereby improving the model's adaptability and flexibility in application in new fields.

[0063] (4) This method supports interactive data analysis, and users can flexibly customize the analysis dimensions according to their specific needs. By directly inputting the text to be analyzed into the trained large model, users can instantly obtain results for different analysis dimensions (such as sentiment, theme, etc.), achieving a highly personalized analysis experience.

[0064] Optionally, obtaining text samples and corresponding training labels and constructing a training set includes:

[0065] Performing structural analysis on the sample data source to extract text class names and text attribute labels in the target data source;

[0066] The text class names and text attribute labels are mapped to text samples and corresponding training labels respectively to construct a training set.

[0067] To this end, the method of performing structural analysis on the sample data source and constructing a training set has the following technical benefits:

[0068] (1) By analyzing the structure of the data source and systematically extracting text class names and text attribute labels, this method can more accurately locate and utilize key information in the data. Compared with unstructured data capture, structured processing reduces noise interference, ensures the high quality of each sample in the training set, and improves the effectiveness and accuracy of model training.

[0069] (2) The process of mapping text class names and attribute labels to training labels is actually a purposeful label normalization operation. This normalization not only unifies the label system and avoids training errors caused by inconsistent labels, but also promotes the model's understanding of the distinction between categories or attributes, so that the boundaries of each category can be better learned during the training process, improving the accuracy of classification or prediction.

[0070] (3) Structured data preparation and label mapping help the model understand the logical structure and semantic relationship behind the data, which is particularly important for the model to generalize to unseen data. When the model is trained on a training set with a clear structure and clear labels, it is easier to capture the inherent laws between the data, thereby improving the generalization performance in new scenarios or new data.

[0071] (4) The automated structure analysis and label mapping process reduces the workload of manual annotation and improves the efficiency of data preprocessing. Especially when dealing with large-scale data sets, this automated processing method becomes the cornerstone of large-scale data analysis and model training, enabling projects to be quickly iterated and optimized.

[0072] (5) The structured approach makes it possible to flexibly adjust text class names and attribute labels according to different analysis requirements. This flexibility not only supports in-depth analysis of existing data, but also facilitates the introduction of new data categories or analysis dimensions in the future, increasing the scalability and adaptability of the entire data analysis system.

[0073] The following is a sample code that implements structural analysis of the sample data source, extracts text class names and attribute labels, and maps them to text samples and training labels to build a training set:

[0074] import json

[0075] from collections import defaultdict

[0076] #data_source is a file path that stores data in JSON format

[0077]

[0078] Optionally, performing structural analysis on the sample data source to extract text class names and text attribute labels in the target data source includes:

[0079] Based on the constructed DOM element sequence, the DOM API is enabled to perform structural analysis on the sample data source, and the text class name and text attribute label in the target data source are extracted.

[0080] To this end, the DOM API is used to perform structural analysis on the sample data source and extract text class names and attribute tags, which has the following technical benefits:

[0081] (1) DOM (Document Object Model) represents a document as a node tree, allowing programs and scripts to dynamically update its content, structure, and style. Parsing through the DOM API can accurately identify every element and attribute of the document, and achieve fine-grained extraction of text class names and attribute tags. Compared with simple string matching or regular expression extraction, this method can more accurately capture information in nested structures and reduce the number of mis-extractions and omissions.

[0082] (2) Web pages or other structured documents are often dynamic and diverse. The dynamic parsing capability of the DOM API enables the solution to flexibly respond to changes in the page structure. Even if the layout or tags of the source data are adjusted, the corresponding class names and attributes can be found by traversing the DOM tree, thus maintaining the stability and effectiveness of the extraction logic.

[0083] (3) Extracting text class names not only helps to classify and organize information, but also helps to deepen the understanding of text content with the help of the semantic information of class names. For example, an element with a class name of "product-description" obviously contains product description information, which is of great guiding significance for subsequent text analysis. At the same time, extracting attribute tags (such as "data-rating" for evaluation information) can enrich the metadata of the text and assist in deeper analysis tasks.

[0084] (4) Using DOM API to automatically process structured data sources reduces the need for manual intervention and improves the efficiency and scale of data preprocessing. Especially when dealing with large-scale web crawling or complex document structures, automated DOM parsing mechanisms are the key to efficiently building high-quality training sets.

[0085] (5) As a standard interface, DOM API is widely used in various programming environments and document types. This means that the method is not limited to the processing of web page data, but can also be extended to the processing of other XML or HTML derivative format data that follow the DOM specification, increasing the versatility and applicability of the technical solution.

[0086] To this end, the following exemplary implementation code is provided to implement the process of structural parsing based on DOM element sequence:

[0087]

[0088]

[0089] In the above code, an HTTP request is first sent to obtain the HTML content of the specified URL, and then the content is parsed using BeautifulSoup. By traversing each element in the DOM tree, the class name (if it exists), text content, and other attributes of the element except the class name are extracted, and this information is stored in a list in the form of a dictionary. Finally, some of the extracted information is printed out as a sample output.

[0090] Optionally, mapping the text class names and text attribute labels to text samples and corresponding training labels, respectively, to construct a training set includes: performing virtual dynamic processing on the text class names and text attribute labels to map the text class names and text attribute labels to text samples and corresponding training labels, respectively.

[0091] The above virtual dynamic processing method has the following technical advantages when constructing a training set:

[0092] (1) Virtual dynamic processing does not simply use the class name or attribute label directly as a static label, but introduces a certain degree of variation or abstraction to enable the model to learn more abstract and generalized representations during the training process. For example, for text class names, similar but not identical virtual class names can be created to simulate the changes of the same category in different contexts, helping the model to make accurate judgments when faced with new categories of text.

[0093] (2) By dynamically adjusting text attribute labels, we can simulate the diversity of attributes in actual application scenarios, so that the model can be exposed to more diverse attribute expressions during the training process, thereby improving the tolerance to noisy data and incompletely matching labels and enhancing the robustness of the model.

[0094] (3) Virtual dynamic processing helps the model learn common features across multiple domains or tasks, especially when processing data with similar but not identical category systems. This processing method can promote the model's rapid adaptation and transfer learning capabilities in new domains or tasks, reducing the reliance on large amounts of newly annotated data.

[0095] (4) In some cases, generating additional training instances through virtualization processing can increase data diversity and reduce the risk of overfitting, thereby optimizing model training efficiency and final performance without significantly increasing the amount of data.

[0096] To this end, the following exemplary code is provided to implement the process of virtually dynamic processing of text class names and attribute labels to construct a training set:

[0097]

[0098]

[0099] In the above code, the basic class names and attribute labels are first defined, and then the virtualize_text function is used to perform simple virtualization on these texts, such as selecting synonyms or adding random characters to the text. After that, it loops through each class name and generates a specified number of samples, each of which consists of the dynamic text and the corresponding original class name (as a training label). Finally, some of the generated training samples and their labels are printed out.

[0100] Optionally, the text class names and text attribute labels are virtually dynamic processed to map the text class names and text attribute labels to text samples and corresponding training labels, respectively, including: based on a set dynamic rule engine, the text class names and text attribute labels are virtually dynamic processed to generate a dynamic rule text sequence; decision analysis and voting are performed on the dynamic rule text sequence to map the text class names and text attribute labels to text samples and corresponding training labels, respectively.

[0101] To this end, a dynamic rule engine based on settings is used to virtually dynamically process text class names and attribute labels, and further map and generate text samples and their corresponding training labels through a decision analysis voting mechanism, which has the following technical benefits:

[0102] (1) The dynamic rule engine can flexibly transform the original class names and attribute labels according to the preset rule set, for example, through strategies such as synonym replacement, word order adjustment, random insertion of adjectives or adverbs, etc., to generate multiple text variants. This not only increases the size of the training set, but also makes the data cover a wider range of expressions and contexts, helping the model learn richer language patterns.

[0103] (2) Through the decision analysis voting mechanism, multiple candidate text sequences generated by dynamic rules can be evaluated and the variant that best matches the expected semantics or is the most representative can be selected as the final text sample. This voting process mimics the decision-making process of humans when faced with language diversity, and helps to screen out high-quality samples that can both maintain the original meaning and increase the degree of variation, thereby improving the generalization performance of the model on unseen data.

[0104] (3) The dynamic rule engine allows for customized rules to be adjusted according to specific task requirements, such as processing domain-specific terminology, cultural sensitivity adjustments, or filtering rules for noisy data. This high degree of customization ensures that the data generation process is both efficient and accurate, meeting the specific needs of different scenarios.

[0105] (4) Compared with untargeted large-scale data collection and manual labeling, this solution effectively reduces manual intervention and data preparation costs through intelligent generation. At the same time, decision analysis voting can quickly screen out high-quality samples, avoid the negative impact of invalid or low-quality data on the training process, and improve the overall training efficiency.

[0106] (5) By introducing noise, spelling errors, grammatical variations, etc. from different sources, the model is forced to learn to deal with these common variations in natural language during the training process, which enhances its tolerance for errors and irregular expressions in real-world data and improves the robustness and practicality of the model.

[0107] To this end, an exemplary implementation code is provided as follows to implement the above solution:

[0108]

[0109]

[0110] In the above code, a DynamicRuleEngine class is first defined to dynamically transform text class names and attribute labels according to preset rules, and then a DecisionVotingSystem class is defined to simulate the decision analysis voting process, although the voting mechanism here is very simplified. In actual applications, the dynamic rule engine will be more complex and may include advanced functions such as part of speech conversion and syntactic structure adjustment.

[0111] Optionally, the format conversion of the text samples in the training set to convert the training set into input sequence samples of the large model includes:

[0112] Performing word segmentation processing on the text samples in the training set to obtain a number of word units;

[0113] Performing frequency statistics on the plurality of word units to determine high-frequency words and forming a high-frequency vocabulary list accordingly, wherein the high-frequency words are word units whose occurrence frequency is greater than a set frequency threshold;

[0114] The high-frequency vocabulary is encoded to obtain a high-frequency word index sequence to serve as an input sequence sample for converting the training set into a large model.

[0115] To this end, the above solution has the following technical advantages:

[0116] (1) By performing word segmentation on the text samples in the training set, the natural language text is decomposed into basic word units. This step ensures that the original text can be recognized and utilized by subsequent processing procedures in a structured and standardized form. This standardization process enhances the compatibility between text data from different sources or styles, allowing large models to understand and process these inputs more uniformly and efficiently.

[0117] (2) Constructing a high-frequency vocabulary and determining high-frequency words through frequency statistics, this strategy effectively reduces the computational burden of large model training. High-frequency words usually contain key information in the text. By identifying and prioritizing these words, the processing of low-frequency and redundant words can be reduced, thereby reducing the dimensionality of the model input without sacrificing information integrity, saving storage space and accelerating the calculation process.

[0118] (3) Encoding high-frequency words to form a high-frequency word index sequence is essentially a feature dimension reduction and encoding technology. It not only reduces the length of the input sequence, but also enables the model to process information in a more compact and efficient way. The indexing of high-frequency words enables the model to learn the contextual relationships and semantic features of these important words more quickly during training, thereby improving the model's understanding ability and generalization performance.

[0119] (4) By constructing a high-frequency vocabulary and encoding it, we are actually performing a form of knowledge extraction. This method is helpful for subsequent model fine-tuning or transfer learning tasks, because high-frequency words often carry core knowledge and patterns in the field. When the model is migrated to a new task or field, these encoded high-frequency words can serve as a "bridge" to accelerate the model's adaptation to the new environment and improve learning efficiency.

[0120] (5) Although deep learning models are often difficult to explain due to their “black box” nature, a certain degree of model interpretability can be achieved by clarifying the selection and encoding process of high-frequency words. Understanding which words are considered important by the model can help analyze the logic behind the model’s decisions and further optimize the model design or adjust the training strategy.

[0121] The following is a sample code that implements the preprocessing of text samples, including word segmentation, frequency statistics, and encoding of high-frequency words to generate sequence samples suitable for large model input:

[0122]

[0123] In the above code, the input text sample is first segmented, and then the frequency of each word is counted, and a high-frequency vocabulary is constructed according to the set frequency threshold. Finally, the word units in the text are converted into index sequences based on the high-frequency vocabulary as the input of the large model. Note that this example does not make special treatment for words that do not appear in the high-frequency vocabulary. In practice, you may need to consider replacing them with "UNK" (unknown word) tags or adopting other strategies.

[0124] Optionally, encoding the high-frequency vocabulary to obtain a high-frequency word index sequence includes:

[0125] Using the Word Embeddings model, determining the semantic relevance of high-frequency words in the high-frequency vocabulary;

[0126] Based on the set adaptive coding model, encoding the high-frequency vocabulary to obtain a group code;

[0127] Based on the semantic association of the high-frequency words in the high-frequency vocabulary, built-in sequence start markers and end markers are added to the group code to generate a high-frequency word index sequence.

[0128] To this end, the high-frequency vocabulary is deeply processed by integrating the Word Embeddings model and the adaptive encoding model, which has the following technical benefits:

[0129] (1) Using the Word Embeddings model, each high-frequency word is mapped to a high-dimensional vector space, where the spatial distance reflects the semantic similarity between words. This mapping not only captures the literal meaning of the word, but also contains rich contextual information, allowing the system to understand the deep connections between words, such as synonyms, antonyms, and subtle differences between words.

[0130] (2) Based on the set adaptive encoding strategy, the high-frequency vocabulary is encoded to form a group encoding. This encoding method may involve cluster analysis, which classifies semantically similar words into one category, thereby greatly compressing the representation dimension and reducing computational complexity while retaining important semantic structure information. The adaptive feature means that the encoding model can be automatically adjusted according to the characteristics of the vocabulary and task requirements, improving the flexibility and pertinence of the processing.

[0131] (3) Based on group coding, by adding sequence start and end markers, clear boundary information is provided for subsequent processing. This step is particularly important for tasks such as sequence labeling and sentiment analysis, because it helps the model understand the structure of word sequences, distinguish word combinations in different contexts, and improve the model's ability to handle long-distance dependencies and complex syntactic structures.

[0132] (4) Using high-frequency word index sequences as input features enables downstream machine learning or deep learning models to learn more efficient feature representations, accelerate the training process, and improve the model's generalization ability on unseen data. Especially when processing large-scale text data, this preprocessing method can significantly reduce computing resource consumption while maintaining or improving prediction accuracy.

[0133] (5) This method considers the semantic relevance of words for encoding, which is particularly suitable for processing text data with domain characteristics, such as medical literature, legal documents, etc. Through customized high-frequency vocabulary and corresponding encoding model, it can better capture the expertise and terminology relationships in the field, providing strong support for domain-specific NLP tasks.

[0134] To this end, create the following exemplary code to implement the above process:

[0135] Dark version

[0136] from gensim.models import Word2Vec

[0137] from sklearn.cluster import KMeans

[0138] import numpy as np

[0139] #High frequency vocabulary list

[0140] high_frequency_words = [...] # List of high frequency words

[0141] #Step 1: Use Word2Vec to get the semantic vector of the vocabulary

[0142] #Large corpus for training Word2Vec model

[0143] sentences=[...] #List of sentences in the corpus

[0144] model=Word2Vec(sentences,size=100,window=5,min_count=1,workers=4)

[0145] #Get the vector representation of high-frequency words

[0146] word_vectors={word:model.wv[word]for word in high_frequency_words ifword in model.wv}

[0147] #Step 2: Encode the vocabulary based on the adaptive encoding model (KMeans clustering is used here)

[0148] num_clusters = 10 # Divide the vocabulary into 10 groups

[0149] kmeans=KMeans(n_clusters=num_clusters)

[0150] word_cluster_labels=kmeans.fit_predict(list(word_vectors.values()))

[0151] #Build group coding dictionary

[0152] grouped_words={i:[]for iin range(num_clusters)}

[0153] for word,label in zip(high_frequency_words,word_cluster_labels):

[0154] grouped_words[label].append(word)

[0155] #Step 3: Add sequence markers to group encoding

[0156] #Will' <start>'and' <end>'As a marker

[0157] encoded_sequences = []

[0158] for cluster_id, words in grouped_words.items():

[0159] sequence = [' <start>']+sorted(words)+[' <end>']

[0160] encoded_sequences.append(sequence)

[0161] #Print the encoded sequence example

[0162] for seq in encoded_sequences:

[0163] print(seq)

[0164] The above code first uses Word2Vec to learn the semantic vectors of words from the corpus, then clusters them through KMeans to get the group code, and finally adds <start>Tag, add at the end <end>Tokenize to form a high frequency word index sequence.

[0165] Optionally, the using of the Word Embeddings model to determine the semantic association of the high-frequency words in the high-frequency vocabulary includes: using the Word Embeddings model to convert each high-frequency word in the high-frequency vocabulary into a vector representation; calculating the similarity between different vector representations, so as to evaluate the semantic association between the high-frequency words in the high-frequency vocabulary based on the similarity and generate a semantic association graph.

[0166] To this end, the above-mentioned use of the Word Embeddings model to determine the semantic relevance of words in the high-frequency vocabulary has the following technical benefits:

[0167] (1) Word Embeddings technology can learn the meaning of words in context by training on a large amount of text data, mapping each word to a high-dimensional vector space so that semantically similar words are closer in the space. This method goes beyond traditional statistical methods based on word frequency or co-occurrence matrices and can more accurately capture the subtle semantic differences and associations between words.

[0168] (2) In the process of generating high-frequency word index sequences, by evaluating the semantic relevance between words and encoding them accordingly, the encoded groups can be made more reasonable. Words with similar semantics are grouped together, which helps to reduce redundancy and improve the efficiency and accuracy of subsequent processing tasks such as information retrieval, natural language understanding, and text generation.

[0169] (3) The generated semantic relevance graph intuitively displays the relationship network between high-frequency words, providing a powerful tool for domain knowledge mining. Researchers or developers can use it to discover implicit topic structures, concept clusters or potential semantic rules, which is of great value for domain-specific knowledge organization, topic modeling and the design of personalized recommendation systems.

[0170] (4) Incorporating this semantic-based encoding strategy into the preprocessing stage of the NLP model can enable the model to better understand the deep meaning of the text and improve its generalization ability on unseen data. In particular, when dealing with complex scenarios such as fuzzy queries and polysemous word resolution, this semantic-based processing method can significantly enhance the model's understanding.

[0171] (5) The Word Embeddings model itself has good versatility and scalability. It can be customized or fine-tuned for different fields and tasks to adapt to specific language environments and professional terminology, so that it can achieve good technical results in various application scenarios.

[0172] To this end, the following sample code is provided to load the pre-trained Word2Vec model (a common Word Embeddings model), convert high-frequency words into vector representations, calculate the similarity between words, and simply generate a display of word association:

[0173] pip install gensim

[0174] from gensim.models import Word2Vec

[0175] from gensim.similarities import WordEmbeddingSimilarityIndex

[0176] from gensim.similarities import SoftCosineSimilarity

[0177] from gensim.corpora.dictionary import Dictionary

[0178] import matplotlib.pyplot as plt

[0179] from sklearn.manifold import TSNE

[0180] import numpy as np

[0181] #The high frequency vocabulary list is as follows

[0182] high_freq_words=['apple','banana','orange','fruit','juice','vitamin']

[0183] #Load the pre-trained Word2Vec model

[0184] word2vec_model_path='path / to / your / word2vec.model'

[0185] model=Word2Vec.load(word2vec_model_path)

[0186] #Convert high-frequency words into vector representation

[0187] word_vectors={word:model.wv[word]for word in high_freq_words if wordin model.wv}

[0188] # Calculate the similarity matrix between words

[0189] def get_similarity(word1,word2):

[0190] return model.wv.similarity(word1,word2)

[0191] #Calculate the similarity between 'apple' and 'orange'

[0192] similarity_apple_orange=get_similarity('apple','orange')

[0193] print(f"Similarity between'apple'and'orange':{similarity_apple_orange}")

[0194] #Visualize vocabulary vectors (using t-SNE dimensionality reduction)

[0195] words=list(word_vectors.keys())

[0196] vectors=np.array(list(word_vectors.values()))

[0197] tsne=TSNE(n_components=2,random_state=42)

[0198] word_tsne=tsne.fit_transform(vectors)

[0199] #Draw a 2D representation of the vocabulary vector

[0200] plt.figure(figsize=(10,8))

[0201] for i,word in enumerate(words):

[0202] plt.scatter(word_tsne[i,0],word_tsne[i,1],label=word)

[0203] plt.legend()

[0204] plt.title("t-SNE Visualization of High Frequency Words")

[0205] plt.show()

[0206] The above code first loads a pre-trained Word2Vec model, then converts high-frequency words into vector representations, and shows how to calculate the similarity between any two words. Finally, it uses t-SNE dimensionality reduction technology to project these high-dimensional vectors onto a two-dimensional plane to visualize the relative positions between words, indirectly reflecting their semantic relevance.

[0207] Optionally, the adding of built-in sequence start markers and end markers in the group code based on the semantic association of the high-frequency words in the high-frequency vocabulary to generate a high-frequency word index sequence includes:

[0208] Generate a semantic association matrix based on the semantic association of high-frequency words in the high-frequency vocabulary;

[0209] Based on the semantic association matrix, cluster analysis is performed on the high-frequency words in the high-frequency vocabulary to determine group representative words therein;

[0210] The semantic flow and logical structure of the group representative word are expanded to determine the sequence start mark position and the end mark position, so as to add the built-in sequence start mark and end mark to the group code corresponding to the group representative word.

[0211] To this end, the above method is used to generate a high-frequency word index sequence and add built-in sequence start and end markers, which has the following technical benefits:

[0212] (1) By analyzing the semantic relevance of high-frequency words and generating a correlation matrix, we can gain a deeper understanding of the intrinsic connections between words. Based on this correlation, cluster analysis groups words with similar meanings into the same group, ensuring that the words within the group are closely related in semantics. The determination of group representative words further refines the core meaning of each group, making the generated text not only dense in high-frequency words but also fluent and natural in semantics.

[0213] (2) The semantic flow and logical structure expansion based on the group representative words ensures the reasonable distribution of the sequence start and end markers. This not only helps the machine understand the start and end points of the text and improves the accuracy of the model when processing long texts or complex text structures, but also provides clear paragraph divisions for human readers, enhancing the readability and comprehensibility of information.

[0214] (3) Built-in sequence markers help guide the subsequent text generation process, especially in scenarios such as automatic summary generation, dialogue system responses, or document structuring. The sequence start marker is like the introduction of an article, guiding the start direction of the generated content, while the end marker effectively concludes the topic, ensuring the integrity of the text and avoiding abrupt endings or ambiguity in meaning.

[0215] (4) The high-frequency word index sequence constructed by this method can better generalize to unseen text structures and contexts when processing new data. This is because sequence labeling provides a structured hint for the model to understand and imitate the logical structure of human language, thus generating text that is closer to human expression habits.

[0216] (5) For large-scale text data management, high-frequency word index sequences with clear sequence markers can serve as an efficient indexing mechanism to accelerate information retrieval and automatic classification processes. By quickly identifying the beginning and end of the sequence, the system can more accurately locate and extract key information, improving the efficiency and accuracy of data processing.

[0217] To this end, the following sample code is provided to implement the above process:

[0218]

[0219]

[0220] The above code first converts high-frequency words into vector representations using the TF-IDF method, and then uses KMeans to perform cluster analysis to determine different groups. The group representative word is selected based on the closest word to each cluster center. Finally, a simplified logic is used to add sequence start and end markers, where the first word of each group is preceded by <start>, add <end>.

[0221] Optionally, the input sequence sample is input into the large model, so that the large model performs a preset dimension analysis on the input sequence to obtain an analysis result;

[0222] The word segmenter in the large model divides the input sequence sample into a plurality of vocabulary units, and assigns a position identifier to each vocabulary unit according to the position in the input sequence sample to form a standard vocabulary unit;

[0223] Based on the preset dimension, the transformer layer in the large model is triggered to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result.

[0224] To this end, based on the solution here, the input sequence samples are fed into the large model for analysis of the preset dimensions, which has the following technical benefits:

[0225] (1) Large models, such as language models based on the Transformer architecture, have powerful contextual understanding and generation capabilities. By performing in-depth analysis of the input sequence in preset dimensions, the model can not only capture the meaning of individual words, but also understand the complex relationships and dependencies between words, such as long-range semantic associations and syntactic structures. This enhances the depth and breadth of the model's understanding of the text, thereby improving the accuracy and comprehensiveness of the analysis.

[0226] (2) The tokenizer divides the input sequence into lexical units and assigns positional identifiers, which ensures that the model can clearly understand the specific position of each word in the sentence. Positional information is crucial for many natural language processing tasks, such as distinguishing the meanings of homographs, understanding tense changes, or the referents of demonstrative pronouns. The addition of positional identifiers enables the model to use contextual positional clues to improve the accuracy of understanding sentence structure and meaning.

[0227] (3) The multi-head attention mechanism of the Transformer layer allows the model to examine the input sequence from different perspectives in parallel, with each head focusing on different parts or features of the input, such as the semantics of words, syntactic roles, or contextual dependencies. This parallel processing not only improves the processing speed of the model, but also increases the model's ability to capture complex language patterns, making the analysis results more sophisticated and comprehensive. By guiding attention allocation through preset dimensions, the attention mechanism can be optimized for specific tasks, further improving the model's directional analysis performance.

[0228] (5) The preset dimensions serve as the guide for analysis and can be flexibly adjusted according to different application scenarios, such as sentiment analysis, topic extraction, named entity recognition, etc. This flexibility enables the model to better adapt to the needs of specific tasks and output analysis results that are more in line with the task objectives, thereby improving the practicality and versatility of the model.

[0229] (6) The combined use of the above technical solutions will ultimately significantly improve the performance of various natural language processing downstream tasks, such as improving the accuracy, response speed and user satisfaction of question-answering systems, text classification, machine translation, etc. By deeply understanding and accurately analyzing the input sequence, the model can generate higher-quality output, whether it is generating text, making decisions or extracting information, it can achieve higher efficiency and quality levels.

[0230] The following is a sample code to implement the above solution:

[0231] from transformers import AutoTokenizer,AutoModel#Use Hugging FaceTransformers library

[0232] # Initialize the pre-trained model and word segmenter

[0233] model_name="your_pretrained_model_name"#For example "bert-base-uncased"

[0234] tokenizer=AutoTokenizer.from_pretrained(model_name)

[0235] model=AutoModel.from_pretrained(model_name)

[0236] # Input sequence sample

[0237] input_sequence="Your input sequence goes here."

[0238] # Word segmentation and adding position identifiers

[0239] inputs = tokenizer (input_sequence, return_tensors = "pt", padding = True, truncation = True) # Word segmentation and return PyTorch tensor

[0240] #tokenizer automatically assigns a positional encoding to each token, which is reflected in the returned tensor

[0241] #Trigger the Transformer layer for multi-head attention analysis

[0242] with torch.no_grad():#No gradient calculation, save resources

[0243] outputs = model (**inputs) # Use the model to process the input after word segmentation

[0244] #The analysis results are contained in multiple parts of outputs, such as last_hidden_state, which may be used for subsequent analysis

[0245] analysis_result = outputs.last_hidden_state # Take the last hidden state as the analysis result

[0246] # Further processing according to preset dimensions

[0247] #Focus on the analysis results of the first token

[0248] preset_dimension_analysis=analysis_result[0,:]#Take the vector representation of the first sequence element

[0249] print("Analysis results:", preset_dimension_analysis)

[0250] The above code implements sending an input sequence into a pre-trained Transformer model (such as BERT), performing word segmentation, position encoding, and obtaining the output of the model.

[0251] Optionally, based on the preset dimension, triggering the transformer layer in the large model to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result, including:

[0252] Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary unit to capture the self-attention semantic space characteristics of the standard vocabulary unit, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary unit to capture the cross-attention semantic space characteristics of the standard vocabulary unit, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

[0253] To this end, this solution is adopted. By performing multi-head self-attention and multi-head cross-attention analysis in different Transformer layers in the large model in stages, and combining the results of each classification head for fusion, it has the following technical benefits:

[0254] (1) The multi-head self-attention mechanism of the first Transformer layer can perform in-depth self-analysis of standard vocabulary units. Each head focuses on different aspects of the input sequence, effectively capturing the contextual relationship and self-dependency characteristics within the vocabulary. This helps to reveal the intrinsic structure of the text and the meaning of a single word in different contexts, and enhances the model's sensitivity to local subtle differences in the text.

[0255] (2) The multi-head cross-attention analysis of the second Transformer layer focuses on the mutual influence and dependency between vocabulary units. By comparing the correlation between different words, the model can build a broader context perception and understand the role and meaning of words in the overall context. This helps the model grasp the overall context and logical structure of the text, especially when dealing with complex syntactic structures and long-distance dependencies.

[0256] (3) By adding classification heads to the two analysis stages, the model can perform specialized learning on different semantic space features. The first classification head targets self-attention features, while the second classification head targets cross-attention features. This design not only refines the learning objectives of the model, but also enables the model to classify input sequences from different angles, thereby improving the accuracy and robustness of classification.

[0257] (4) The first and second analysis results and their classification head outputs are fused. This process combines the complementary advantages of self-attention and cross-attention analysis. It not only integrates local and global semantic information, but also improves the model's comprehensive understanding of complex text contexts through the classification head fusion strategy (such as weighted averaging, splicing or more complex fusion networks), thereby obtaining a higher quality final analysis result.

[0258] (5) The flexibility and hierarchy of the scheme enable the model to better adapt to the needs of different tasks. By adjusting the preset dimensions and the configuration of the classification head, the performance of the model in specific fields can be optimized, such as sentiment analysis, event extraction, question-answering systems, etc., thereby improving the model's generalization ability and task adaptability.

[0259] To this end, an example code is provided, using a library like Hugging Face Transformers to implement the multi-head self-attention and cross-attention analysis described above, combined with the fusion strategy of the classification head:

[0260]

[0261]

[0262] Optionally, the calculating a loss function of the large model based on the analysis result and the training labels in the training set, so as to train the large model based on the loss function until the training is completed, comprises:

[0263] Based on the analysis results and the training labels in the training set, the loss function of the large model is calculated. If the value of the loss function is less than the set loss threshold, the training is completed. Otherwise, the process jumps to the step of formatting the text samples in the training set to convert the training set into input sequence samples of the large model, so as to retrain the large model.

[0264] To this end, this solution dynamically adjusts the training process and decides whether to reformat the input data and continue training based on the performance of the model loss function. This has the following technical benefits:

[0265] (1) By monitoring the value of the loss function in real time, the strategy provides an immediate feedback mechanism that can quickly respond to the learning status of the model. If the model has reached the preset loss threshold in the current training state, it indicates that the model has achieved the expected learning effect under the current configuration and no further training is required, thus saving computing resources.

[0266] (2) When the model loss fails to reach the preset threshold, the system automatically goes back to the data preprocessing stage and re-converts the format of the training set. This mechanism emphasizes the importance of data preprocessing for model performance, suggesting that it may be necessary to improve the model learning effect by changing the data representation or introducing new data processing strategies. Iterative data processing adjustments help explore more effective data input formats and promote the gradual improvement of model performance.

[0267] (3) Continuously adjusting the training process based on loss function feedback prompts the model to learn different forms of input data at different stages, which helps the model learn a wider range of more complex language patterns and improves the model's generalization ability. Especially for complex and changeable natural language processing tasks, this dynamic adjustment strategy helps the model better adapt to different types of text data.

[0268] (4) By setting the loss threshold as the stopping condition, unnecessary overtraining is avoided and the waste of computing resources is reduced. At the same time, when the model performs poorly, it immediately returns to the data processing stage instead of blindly increasing the number of training rounds. This is a more economical and strategic training management method.

[0269] (5) This solution reflects the adaptability of model training and can dynamically adjust the training path according to the actual performance during training. This flexibility is particularly important for dealing with data sets of different sizes and qualities, and can ensure that the model training process is both efficient and targeted.

[0270] To this end, sample code is provided to implement the above model training, calculate the loss function, and decide whether to reformat the data and retrain according to the loss threshold:

[0271]

[0272] #Training set and labels

[0273] texts=["Example text 1.","Another example text 2."]

[0274] labels = [0,1] #label

[0275] #Create a data loader

[0276] dataset=TextDataset(texts,labels)

[0277] dataloader=DataLoader(dataset,batch_size=1)

[0278] #Training loop

[0279] for epoch in range(10):#Train for 10 rounds

[0280] for batch in dataloader:

[0281] inputs,targets=batch

[0282] outputs=model(**inputs,labels=targets)

[0283] loss = outputs.loss

[0284] optimizer.zero_grad()

[0285] loss.backward()

[0286] optimizer.step()

[0287] # Check if the loss is below the threshold

[0288] if loss.item() <LOSS_THRESHOLD:

[0289] print(f"Loss reached threshold{LOSS_THRESHOLD}after epoch{epoch}.Training complete.")

[0290] break

[0291] else:#If there is no break (that is, the threshold is not reached)

[0292] print("Loss did not meet the threshold.Reformatting dataset...")

[0293] continue#Restart the training loop. In practice, more complex logic should be used to handle data reformatting.

[0294] print("Training finished.")

[0295] Optionally, the obtaining of the text to be processed and converting it into a sequence to be processed, so as to input it into a trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results, includes:

[0296] Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary units in the sequence to be processed to capture the self-attention semantic space features of the standard vocabulary units, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary units in the sequence to be processed to capture the cross-attention semantic space features of the standard vocabulary units, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

[0297] The above solution combines multi-head self-attention and multi-head cross-attention analysis in a large model, as well as a dual classification head fusion strategy to process the text to be analyzed, which has the following technical benefits:

[0298] (1) The multi-head self-attention mechanism of the first transformer layer can conduct in-depth analysis of each standard vocabulary unit in the sequence to be processed. Each multi-head focuses on the contextual information of the vocabulary from different angles, thereby capturing the rich semantic features of each vocabulary in the text. This mechanism strengthens the model's understanding of the local details and internal structure of the text, and improves the sophistication and accuracy of semantic parsing.

[0299] (2) The multi-head cross-attention analysis of the second transformer layer focuses on the relationship and dependency between vocabulary units. By comparing the interactions of different words in the sequence, the model can build a global semantic framework for the entire text. This not only helps to understand the meaning of words in a larger context, but also can effectively identify and utilize long-distance dependencies, improving the ability to process complex text structures.

[0300] (3) By adding classification heads to the two analysis stages, the model is able to perform specialized classification learning for different types of semantic features. The first classification head focuses on the classification of self-attention features, while the second classification head targets cross-attention features. This dual-path classification method can classify text from multiple dimensions, improving the accuracy and robustness of classification, especially when dealing with text with fuzzy category boundaries or complex semantics.

[0301] (4) By fusing the first and second analysis results and their corresponding classification head outputs, the model can integrate local and global, micro and macro semantic information to form a more comprehensive and profound text understanding. The fusion strategy not only enhances the model's ability to adapt to text diversity and complexity, but also makes the final analysis results more accurate and reliable.

[0302] (5) This phased, multi-level analysis strategy enables the model to flexibly adjust and optimize when faced with different types of tasks, better adapting to changes in task requirements. By effectively integrating features of different dimensions, the model exhibits stronger generalization performance when processing new data or unseen text types.

[0303] To this end, the following exemplary code is provided to implement the above multi-stage analysis processing, including multi-head self-attention and cross-attention analysis:

[0304]

[0305] The present application embodiment provides a device for building a heterogeneous data warehouse, which includes:

[0306] A first processing unit is used to obtain a data asset governance process definition file and parse it to extract a link description and link ID of the data governance process;

[0307] A second processing unit is used to access a standard library that the data asset process depends on, so as to obtain a standard file description and a standard file ID constraining the data governance process;

[0308] a third processing unit, configured to perform attention analysis on the link description and the standard document representation to obtain a standard compliance matching graph, wherein in the standard compliance matching graph, the standard document description and the link description having a constraint relationship are associated through the link ID and the standard document ID;

[0309] A fourth processing unit is used to construct a standard metadata model according to the standard compliance matching graph, and encode the standard metadata model to obtain a standard mapping table to construct a standard data governance rule engine;

[0310] A fifth processing unit is used to assign roles to the data-based governance organizational structure and establish a workflow control model between the assigned roles and the link descriptions;

[0311] The sixth processing unit is used to establish an object collaborative mapping framework between the workflow control model and the standard data governance rule engine to perform standard data asset governance tasks according to the object collaborative mapping framework.

[0312] For an exemplary description of each unit in the above device, please refer to the above Figure 1 Records of.

[0313] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer executable program, and the processor is used to run the computer executable program to implement the method described in any embodiment of the present application.

[0314] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.< / end> < / start> < / end> < / start> < / end> < / start> < / end> < / start>

Claims

1. An interactive data analysis method based on a large model, characterized in that: include: Obtain text samples and corresponding training labels to build a training set; Performing word segmentation processing on the text samples in the training set to obtain a number of word units; Performing frequency statistics on the plurality of word units to determine high-frequency words and forming a high-frequency vocabulary list accordingly, wherein the high-frequency words are word units whose occurrence frequency is greater than a set frequency threshold; Encoding the high-frequency vocabulary to obtain a high-frequency word index sequence as an input sequence sample for converting the training set into a large model; Using the Word Embeddings model, determining the semantic relevance of high-frequency words in the high-frequency vocabulary; Based on the set adaptive coding model, encoding the high-frequency vocabulary to obtain a group code; Generate a semantic association matrix based on the semantic association of high-frequency words in the high-frequency vocabulary; Based on the semantic association matrix, cluster analysis is performed on the high-frequency words in the high-frequency vocabulary to determine group representative words therein; Expanding the semantic flow and logical structure of the group representative word to determine the sequence start mark position and the end mark position, and adding the built-in sequence start mark and end mark to the group code corresponding to the group representative word to generate a high-frequency word index sequence; Inputting the input sequence sample into the large model, so that the large model analyzes the input sequence sample in a preset dimension to obtain an analysis result; Based on the analysis result and the training labels in the training set, a loss function of the large model is calculated to train the large model based on the loss function until the training is completed; Obtain the text to be processed and convert it into a sequence to be processed, so as to input it into the trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results; The step of obtaining text samples and corresponding training labels and constructing a training set includes: Perform structural analysis on the sample data source and extract the text class name and text attribute label in the target data source; Mapping the text class name and text attribute label to text samples and corresponding training labels respectively to construct a training set; The structural analysis of the sample data source to extract the text class name and text attribute label in the target data source includes: Based on the constructed DOM element sequence, the DOM API is enabled to perform structural analysis on the sample data source, and the text class names and text attribute tags in the target data source are extracted.

2. The interactive data analysis method based on a large model according to claim 1, characterized in that: The process of mapping the text class names and text attribute labels to text samples and corresponding training labels respectively to construct a training set includes: performing virtual dynamic processing on the text class names and text attribute labels to map the text class names and text attribute labels to text samples and corresponding training labels respectively.

3. The interactive data analysis method based on a large model according to claim 1, characterized in that: The virtual dynamic processing of the text class name and text attribute label to map the text class name and text attribute label to text samples and corresponding training labels respectively includes: based on a set dynamic rule engine, the text class name and text attribute label are virtually dynamic processed to generate a dynamic rule text sequence; decision analysis and voting are performed on the dynamic rule text sequence to map the text class name and text attribute label to text samples and corresponding training labels respectively.

4. The interactive data analysis method based on a large model according to claim 1, characterized in that: The method of using the Word Embeddings model to determine the semantic association of high-frequency words in the high-frequency vocabulary includes: using the Word Embeddings model to convert each high-frequency word in the high-frequency vocabulary into a vector representation; calculating the similarity between different vector representations, and based on the similarity, evaluating the semantic association between the high-frequency words in the high-frequency vocabulary and generating a semantic association graph.

5. The interactive data analysis method based on a large model according to claim 1, characterized in that: The input sequence sample is input into the large model, so that the large model performs a preset dimension analysis on the input sequence to obtain an analysis result; The word segmenter in the large model divides the input sequence sample into a plurality of vocabulary units, and assigns a position identifier to each vocabulary unit according to the position in the input sequence sample to form a standard vocabulary unit; Based on the preset dimension, the transformer layer in the large model is triggered to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result.

6. The interactive data analysis method based on a large model according to claim 5, characterized in that: Based on the preset dimension, triggering the transformer layer in the large model to perform multi-head attention analysis on the standard vocabulary unit to obtain an analysis result, including: Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary unit to capture the self-attention semantic space characteristics of the standard vocabulary unit, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary unit to capture the cross-attention semantic space characteristics of the standard vocabulary unit, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

7. The interactive data analysis method based on a large model according to claim 1, characterized in that: The step of calculating the loss function of the large model based on the analysis result and the training labels in the training set, so as to train the large model based on the loss function until the training is completed, comprises: Based on the analysis result and the training labels in the training set, the loss function of the large model is calculated. If the value of the loss function is less than the set loss threshold, the training is completed. Otherwise, the step of format conversion of the text samples in the training set is performed to convert the training set into input sequence samples of the large model, so as to retrain the large model. The process of obtaining the text to be processed and converting it into a sequence to be processed, so as to input it into the trained large model to analyze the text to be processed in preset dimensions and obtain corresponding analysis results, includes: Based on the preset dimension, the first transformer layer in the large model is triggered to perform multi-head self-attention analysis on the standard vocabulary units in the sequence to be processed to capture the self-attention semantic space features of the standard vocabulary units, obtain a first analysis result and add a first classification head, and the second transformer layer in the large model is triggered to perform multi-head cross-attention analysis on the standard vocabulary units in the sequence to be processed to capture the cross-attention semantic space features of the standard vocabulary units, obtain a second analysis result and add a second classification head, so as to fuse the first analysis result and the first classification head to obtain the final analysis result.

Images