A method and system for automatic annotation of text data integrating pre-trained NLP models

Abstract

The present invention provides a method and system for automatic annotation of text data integrated with a pre-trained NLP model, relating to the field of artificial intelligence technology. The method comprises: obtaining raw text data to be annotated, the text data comprising conversation records, evaluations and work orders of customer consultations, complaints or feedback; performing word segmentation, stop word filtering and part-of-speech tagging on the raw text data to generate structured text data; inputting the structured text data into the pre-trained NLP model, correcting the semantic features output by the model through a domain adaptation layer to obtain corrected semantic features; performing few-sample learning on the corrected semantic features, strengthening the recognition ability of low-frequency text patterns through a comparative learning mechanism, and generating enhanced feature vectors that are resistant to long-tail interference. The present invention realizes automatic annotation of text data by automating data acquisition and preprocessing and integrating a pre-trained NLP model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for automatically annotating text data using an integrated pre-trained NLP model. Background Art

[0002] In the field of customer service and text analysis, automated annotation of text data such as customer inquiries and complaints is a key technology for intelligent customer service, service quality assessment, and problem tracing. However, traditional text annotation methods can face numerous technical bottlenecks. Supervised learning-based models are subject to the "long tail effect," and insufficient training data can lead to reduced recognition accuracy for less frequent customer service issues, making it difficult to fully cover all issues.

[0003] Some pre-trained NLP models in the customer service field have problems with domain terminology deviation and lack of context dependence, which may lead to distortion of semantic feature extraction and affect the accuracy of label extraction; some multi-dimensional label extraction uses independent models for processing, which may have problems such as lack of label consistency and waste of computing resources, making it difficult to adapt to high-real-time customer service scenarios; some model iterations lag behind business changes, which may lead to timeliness lag and low data utilization. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for automatic annotation of text data integrating a pre-trained NLP model, which realizes automatic annotation of text data by automating data acquisition and preprocessing and integrating a pre-trained NLP model.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] In a first aspect, a method for automatically annotating text data using an integrated pre-trained NLP model is provided, the method comprising:

[0007] Step S1: Obtaining original text data to be annotated, wherein the text data includes conversation records, evaluations, and work orders of customer consultations, complaints, or feedback;

[0008] Step S2: performing word segmentation, stop word filtering, and part-of-speech tagging on the original text data to generate structured text data;

[0009] Step S3: Input the structured text data into the pre-trained NLP model, and modify the semantic features output by the model through the domain adaptation layer to obtain modified semantic features;

[0010] Step S4: Performing few-shot learning on the corrected semantic features, enhancing the recognition capability of low-frequency text patterns through a contrastive learning mechanism, and generating enhanced feature vectors that are resistant to long-tail interference;

[0011] Step S5: Based on the enhanced feature vector, a multi-dimensional tag extraction operation is synchronously performed to output a tag set containing the following four types of tags: demand type tag, sentiment tendency tag, product or service entity tag, and question keyword tag;

[0012] Step S6: For each tag in the tag set, locate its key decision words in the enhanced feature vector through the attention mechanism, generate a natural language description of the tag decision basis, and form an enhanced tag set with explanatory text;

[0013] Step S7: Based on the distribution of low-frequency tags and the frequency of newly added domain terms in the enhanced tag set, trigger the incremental training of the pre-trained NLP model.

[0014] The second aspect is an automatic text data annotation system that integrates a pre-trained NLP model, including:

[0015] An acquisition module is used to acquire the original text data to be annotated, including conversation records, evaluations, and work orders of customer inquiries, complaints, or feedback;

[0016] The preprocessing module is used to perform word segmentation, stop word filtering and part-of-speech tagging on the original text data to generate structured text data;

[0017] The correction module is used to input structured text data into the pre-trained NLP model and correct the semantic features output by the model through the domain adaptation layer to obtain the corrected semantic features;

[0018] The feature vector module is used to perform few-shot learning on the corrected semantic features. It strengthens the recognition of low-frequency text patterns through a contrastive learning mechanism and generates enhanced feature vectors that are resistant to long-tail interference.

[0019] The tag module is used to synchronously perform multi-dimensional tag extraction operations based on the enhanced feature vectors, and output a tag set containing the following four types of tags: demand type tags, sentiment tendency tags, product or service entity tags, and question keyword tags;

[0020] The explanatory module is used to locate the key decision words in the enhanced feature vector of each label in the label set through the attention mechanism, generate a natural language description of the label decision basis, and form an enhanced label set with explanatory text;

[0021] The incremental module is used to trigger incremental training of the pre-trained NLP model based on the distribution of low-frequency tags in the enhanced tag set and the frequency of new domain terms.

[0022] According to a third aspect, a computing device includes:

[0023] one or more processors;

[0024] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0025] In a fourth aspect, a computer-readable storage medium stores a program, which implements the method when executed by a processor.

[0026] The above solution of the present invention includes at least the following beneficial effects:

[0027] The entire process automates data acquisition and preprocessing, connects to data sources in real time, and automatically sorts and stores them. It integrates preprocessing steps such as garbled code cleaning and word segmentation, which improves efficiency compared to traditional manual labeling. At the same time, four types of labels are generated simultaneously through multi-task classifiers, saving computing resources and shortening labeling time. It accurately responds to domain characteristics and long-tail data challenges. The domain adaptation layer dynamically adjusts feature weights based on the terminology library to improve the accuracy of domain-specific vocabulary. It combines few-sample learning and comparative learning mechanisms to strengthen low-frequency text pattern recognition and improve the sensitivity of low-frequency semantic responses. Interpretability and business compliance are significantly enhanced. Natural language explanations of label decisions are generated through the attention mechanism. Incremental training is triggered based on the proportion of low-frequency tags or the terminology library update rate to adapt to business needs in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a method for automatically annotating text data by integrating a pre-trained NLP model, provided by an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of an automatic text data annotation system that integrates a pre-trained NLP model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a method for automatically annotating text data by integrating a pre-trained NLP model, the method comprising the following steps:

[0032] Step S1: Obtaining original text data to be annotated, wherein the text data includes conversation records, evaluations, and work orders of customer consultations, complaints, or feedback;

[0033] Step S2: performing word segmentation, stop word filtering, and part-of-speech tagging on the original text data to generate structured text data;

[0034] Step S3: Input the structured text data into the pre-trained NLP model, and modify the semantic features output by the model through the domain adaptation layer to obtain modified semantic features;

[0035] Step S4: Performing few-shot learning on the corrected semantic features, enhancing the recognition capability of low-frequency text patterns through a contrastive learning mechanism, and generating enhanced feature vectors that are resistant to long-tail interference;

[0036] Step S5: Based on the enhanced feature vector, a multi-dimensional tag extraction operation is synchronously performed to output a tag set containing the following four types of tags: demand type tag, sentiment tendency tag, product or service entity tag, and question keyword tag;

[0037] Step S6: For each tag in the tag set, locate its key decision words in the enhanced feature vector through the attention mechanism, generate a natural language description of the tag decision basis, and form an enhanced tag set with explanatory text;

[0038] Step S7: Based on the distribution of low-frequency tags and the frequency of newly added domain terms in the enhanced tag set, trigger the incremental training of the pre-trained NLP model.

[0039] In an embodiment of the present invention, by collecting multiple types of original data such as customer consultations, complaints, evaluations and work orders, the limitations of a single data dimension are avoided; through word segmentation, stop word filtering and part-of-speech tagging, unstructured text is converted into structured data that can be understood by machines, redundant information is eliminated, data noise is reduced, and the efficiency and accuracy of subsequent model processing are improved; at the model optimization level, the domain adaptation layer is used to achieve precise semantic understanding, and the long-tail problem is overcome by using few samples and comparative learning. Incremental training is triggered based on the frequency of low-frequency tags and new terms to dynamically adapt to business changes; at the tag value level, a multi-dimensional tag system including demand type, emotional tendency, etc. is constructed to support refined operations, and enhanced tags with natural language explanations are generated through the attention mechanism to enhance trust in human-machine collaboration.

[0040] In a preferred embodiment of the present invention, the above step S1: obtaining the original text data to be annotated, the text data includes conversation records, evaluations and work orders of customer consultations, complaints or feedback, and may include:

[0041] Step S11: acquiring original text data to be annotated from the e-commerce platform customer service system in real time. The text data includes conversation records, comments, and work orders of customer consultations, complaints, or feedback;

[0042] Step S12: After sorting by timestamp, the original text data is stored in a structured database and the data source type is marked.

[0043] In an embodiment of the present invention, by acquiring customer feedback data in real time, implementing structured storage and multi-source integration, the latest customer feedback data can be obtained in a timely manner and business needs can be quickly responded to. At the same time, structured storage facilitates subsequent data management and analysis, effectively improving processing efficiency. Multi-source integration enables unified management of multiple types of data such as conversations, evaluations, and work orders, comprehensively covering the entire process of customer feedback.

[0044] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0045] The above step S11 connects with the customer service system of the e-commerce platform (such as Qianniu and JD customer service) through the API interface, and pulls the conversation records between customers and customer service (including text, emoticon package translation text), product reviews (such as follow-up reviews, and text extraction of reviews with pictures), and after-sales work orders (such as return and exchange application descriptions and dispute records) in real time.

[0046] In the above step S12, the data are sorted by timestamp and stored in a structured database (such as MySQL), and the data source type (such as consultation, complaint, evaluation or work order) is marked.

[0047] In a preferred embodiment of the present invention, the above step S2: performing word segmentation, stop word filtering and part-of-speech tagging on the original text data to generate structured text data may include:

[0048] Step S21, scanning each character code of the original text data and processing it to generate a purified text;

[0049] Step S22: Based on the purified text, the probability distribution of each character at its current position as the start, middle, or end position of a word is calculated. Combined with the conditional random field state transition score, dynamic programming is used to generate a global maximum probability segmentation path, and a basic word segmentation sequence with clear word boundaries is output;

[0050] Step S23: traverse each word in the basic word segmentation sequence according to the stop word dictionary stored in the hash table. If a word hits the stop word hash table and is not marked as a protected entity by the named entity recognition module, remove the word; if the word belongs to the entity protection list, retain it in the new sequence; generate a filtered word sequence carrying the core semantic unit;

[0051] Step S24: For each target word in the filtered word sequence, extract the context window of the two words before and after it, and combine them into a feature template; query the pre-trained词性 probability matrix, and calculate the joint probability that the target word belongs to each词性 category in its context environment; construct a词性 state transition grid based on dynamic programming, iteratively calculate the cumulative probability of all词性 annotation paths from the first word to the last word in the sequence, and select the path with the maximum probability as the final词性 sequence; combine each word with its optimal词性 label into a binary tuple to generate structured text data with grammatical markings.

[0052] In the embodiments of the present invention, by purifying the original text, noise interference is eliminated, providing a standardized input for subsequent word segmentation, thereby improving the accuracy of word segmentation; the word segmentation method based on probability distribution can make full use of context features compared with traditional algorithms, effectively solve the problem of ambiguous segmentation, and further improve the accuracy of word segmentation; stop word filtering reduces the interference of meaningless words on subsequent analysis, while protecting key entity information and enhancing the effectiveness of text semantic representation; and词性 annotation endows the text with grammatical structure information, facilitating semantic analysis tasks such as dependency syntax analysis and sentiment analysis, and deepening the depth of text understanding.

[0053] In the embodiments of the present invention, when specifically applied, it can be implemented through the following technical solutions, for example:

[0054] In the above step S21, traverse each character of the original text, check whether its Unicode encoding belongs to a valid character set (such as ASCII, common Chinese character set, etc.), and剔除 garbled or illegal characters; standardize punctuation marks (such as converting full-width to half-width), remove invisible control characters (such as line breaks, tab characters) or replace them with spaces according to business requirements.

[0055] Unify case (such as converting all English to lowercase), process repeated characters (such as "好啊啊啊" → "好啊"), and correct spelling mistakes (based on pre-trained language models or dictionaries).

[0056] In the above step S22, receive the purified text sequence, with each character as an input unit, map the character to a multi-dimensional feature vector (such as the character itself, context, character type, etc.), and predict the label (B / M / E / S) corresponding to each character.

[0057] Design feature templates to capture local and global information of the text. Common templates include:

[0058] Current character features: the character itself, whether it is capitalized, whether it is a number, etc.;

[0059] Context features: the first / last 1-2 characters and their combinations;

[0060] Statistical features: character frequency, whether it is a prefix / suffix of a word in the dictionary;

[0061] Position feature: the position of the character in the sentence (beginning or end of the sentence).

[0062] Define the legal transfer rules between tags, for example: B→E / M (the beginning of a word can only be followed by the middle or end of a word), M→M / E (the middle of a word can only be followed by the middle or end of a word), E→B / S (the end of a word can only be followed by the beginning of a new word or a single word), S→B / S (the single word can only be followed by the beginning of a new word or a single word).

[0063] Collect large amounts of segmented text (such as Corpus A and Corpus B), and convert the segmentation results into B / M / E / S label sequences. For example, original text: "C City D Bridge", segmented text: "C City / D / Bridge", label: BEBMEBE; for each character in the training corpus, extract features according to a predefined feature template and convert the text features into numerical features (such as one-hot encoding).

[0064] The model is trained using Maximum Likelihood Estimation (MLE) or Improved Iterative Scaling (IIS) with the following steps: randomly initialize feature weights, input training samples, calculate the probability of each label sequence, adjust feature weights based on the difference between the true label and the predicted label, and repeat training until the model converges (for example, the change in the log-likelihood function is less than a threshold).

[0065] The validation set is used to evaluate model performance. Common metrics include: Accuracy, which is the ratio of correctly predicted labels to the total number of labels; F1, which is a comprehensive measure of precision and recall; and ambiguity resolution, which tests how well the model can segment ambiguous text (e.g., "The table tennis auction is over").

[0066] Remove features that contribute little to performance, reduce overfitting, add L1 / L2 regularization terms to prevent the model from being too complex, and combine the results of multiple CRF models to improve stability.

[0067] Design features based on linguistic knowledge (such as the rules of Chinese character word formation), and use statistical methods to select features that contribute most to classification; update parameters in small batches to accelerate convergence, stop training when the performance of the validation set no longer improves to prevent overfitting, use a larger learning rate in the early stages of training, and reduce it later to stabilize convergence; based on the general domain model, use a small amount of data from the target domain for fine-tuning; integrate domain dictionaries into feature templates (such as a medical term dictionary).

[0068] Extract the same features as those used in training from the new text, use the trained model parameters to calculate the label probability of each character, combine the state transition scores, find the optimal label sequence, and generate word boundaries based on the label sequence (such as B→E represents a word).

[0069] In the above step S23, each word in the basic word segmentation sequence is quickly matched with the stop word dictionary stored in the hash table (time complexity O(1)); the named entity recognition (NER) module is called to determine whether the word belongs to a protected entity (such as a person's name, a place name, an organization name, etc.); words that hit the stop word and are not protected entities are removed, and the words in the entity protection list are retained to generate a filtered word sequence.

[0070] In step S24, feature template construction is to convert the target word and its context into a feature representation that can be used for model calculation. The specific process is as follows:

[0071] For any target word in the filtered word sequence, we create a window containing contextual information. The specific steps are as follows: With the target word as the center, we take two words forward and two words backward, and then add the target word itself to form a window containing five words. This window acts like a "magnifying glass," helping the model to capture the linguistic context surrounding the target word. If the target word is preceded by fewer than two words (for example, if it is the first or second word in the sequence), the empty space in front is filled with the special symbol `BOS` (Begin of Sequence). If the target word is followed by fewer than two words (for example, if it is the penultimate or second-to-last word in the sequence), the empty space after it is filled with the special symbol `EOS` (End of Sequence).

[0072] Example description:

[0073] Suppose the filtered word sequence is: `["I", "like", "eat", "apple", "and", "banana"]`; when the target word is `"eat"` (position 3), its context window is `["I", "like", "eat", "apple", "and"]` (normal situation, no need to fill); when the target word is `"I"` (position 1), its context window is `["BOS", "BOS", "I", "like", "eat"]` (filled with `BOS` in the front); when the target word is `"banana"` (position 6), its context window is `["and", "apple", "banana", "EOS", "EOS"]` (filled with `EOS` in the back).

[0074] This processing ensures that no matter where the target word is in the sequence, a fixed length of contextual information is obtained, thereby unifying the input format of the model and preserving the key contextual information of boundary words.

[0075] Generate the following features based on the context window: The lexical features are the current word itself (such as "apple"), the 1-2 words before / after it (such as "eat", "very"), and word combinations (such as "eat apple", "apple very"); The morphological features are the word length (such as the length of "apple" is 2), whether it contains numbers / letters (such as "iPhone14" contains letters and numbers), prefix / suffix (such as words starting with "old" are mostly surnames); The statistical features are the frequency of the word in the training corpus (such as the frequency of "de" is extremely high), and the probability of the word as a specific词性 (such as the probability of "apple" as a noun); The dictionary features are whether it is in the predefined dictionary (such as whether it is a word in the verb dictionary), and the common词性 of the word in the dictionary (such as "running" is usually a verb).

[0076] For the target word (w_i), the feature template can be constructed in the following way to capture the key information of itself and its context:

[0077] Directly use the target word itself as a feature (i.e., (w[i])). For example, when the target word is "apple", this feature represents the word "apple" itself; Extract the previous word of the target word (i.e., (w[i - 1])). For example, when the target word is "apple" and the previous word is "eat", then this feature is "eat", which is used to reflect the direct previous context of the target word; Extract the next word of the target word (i.e., (w[i + 1])). For example, when the target word is "apple" and the next word is "very sweet", then this feature is "very sweet", which is used to reflect the direct next context of the target word; Combine the previous word and the current target word into a feature (i.e., (w[i - 1]+w[i])). For example, if the previous word is "eat" and the target word is "apple", then the combined feature is "eat apple".

[0078] Judge whether the target word contains numbers. For example, when the target word is "iPhone14", this feature is "contains numbers"; If it is "apple", then it is "does not contain numbers". This feature helps to distinguish词性 (such as numbers are often paired with quantifiers or are part of time / quantity words).

[0079] Through the above feature template, the context information of the target word can be characterized from different dimensions: There is both the semantic information of the word itself, the collocation relationship of the context before and after, and the morphological features of the word (such as whether it contains numbers). The combination of these features can help the model more accurately judge the词性 of the target word (for example, "apple" is more likely to be a noun in "eat apple" and more likely to be an adjective in "apple green").

[0080] Map each feature template to a numerical feature (such as one-hot encoding) to form a high-dimensional feature vector. For example, the word "apple" → [0, 0, 1, 0, ...] (assuming it is the third word in the vocabulary), the word length = 2 → [0, 1, 0, 0, ...] (assuming the length feature dimension is 4).

[0081] The Hidden Markov Model (HMM) is used to model the probability distribution of part-of-speech sequences. The state transition grid is a visual representation of the HMM. The construction steps are as follows:

[0082] State set (Q): a set of part-of-speech tags (such as nouns, verbs, adjectives, etc.);

[0083] Observation set (V): word set (such as "apple", "eat");

[0084] Parameters: The initial probability (pi) is the probability of each part of speech being the first word in a sentence, the state transition probability (A) is the transition probability between parts of speech (such as the probability of noun → verb), and the emission probability (B) is the probability of generating a word given a part of speech (such as the probability of generating "apple" from a noun).

[0085] For each position in the corresponding word sequence (from 1 to n), each time point contains all possible part-of-speech states, connecting the states of adjacent time points, with the weight being the state transition probability; at time point 1, set the initial probability (pi) for each part-of-speech state; for each subsequent time point (t) (from 2 to n), copy all possible part-of-speech states, starting from each state at time point (t-1), connect to all states at time point (t), and the weight of the edge is the state transition probability (A); at each state node, mark the emission probability (B) of the corresponding word generated by the part of speech.

[0086] The Viterbi algorithm searches for the optimal path in the constructed state transition grid:

[0087] At time point 1, calculate the initial score of each part-of-speech state (initial probability × emission probability); for each subsequent time point (t), calculate the cumulative score of transitions from all possible previous states to the current state, and retain the path with the maximum score; starting from the highest-scoring state at time point (n), backtrack to time point 1 to generate the optimal part-of-speech sequence and generate structured text with part-of-speech tags.

[0088] In a preferred embodiment of the present invention, the above step S3: inputting the structured text data into the pre-trained NLP model and modifying the semantic features output by the model through the domain adaptation layer to obtain the modified semantic features may include:

[0089] Step S31: Input the word sequence in the structured text data generated in step S2 into the embedding layer of the pre-trained NLP model, convert each word into a high-dimensional word vector through a word vector mapping operation, perform context feature extraction through a multi-layer Transformer encoder, and output the original semantic feature vector;

[0090] Step S32: Input the original semantic feature vector into the domain adaptation layer, and calculate the cosine similarity between each feature dimension of the original semantic feature vector and all term vectors in the term library; select a set of term vectors whose similarity exceeds a preset threshold, and perform a normalized weighted sum based on the similarity value between each term vector in the set and the original feature to generate a domain weight vector; expand the domain weight vector into a diagonal matrix form according to the feature dimension to form a domain weight matrix;

[0091] In step S33, a matrix multiplication operation is performed on the domain weight matrix and the original semantic feature vector so that the original features are scaled according to the domain relevance in each dimension; a residual connection is performed on the scaled feature vector and the original semantic feature vector, and the original features and the domain enhancement features are fused through element-level addition operation to obtain a fused feature vector; the fused feature vector is layer-normalized and a corrected semantic feature vector is output.

[0092] In an embodiment of the present invention, with the help of a pre-trained NLP model and a Transformer encoder, general semantic features can be extracted, resource consumption and training time can be reduced, and the processing foundation can be laid; the domain weight matrix is ​​generated by calculating the cosine similarity with the terminology library, which can accurately adapt to the professional semantic requirements of different fields; through matrix operations, residual connections and layer normalization, the domain feature expression is enhanced while retaining the original semantics; high-quality input is provided for downstream NLP tasks, and task indicators and model practicality are improved; no large amount of labeled data is required, and domain adaptation can be completed by relying solely on the terminology library, which reduces costs, shortens the deployment cycle, and enhances the model's versatility and scalability.

[0093] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0094] In step S31 above, the pre-training model is usually built based on a multi-layer Transformer encoder. Each encoder layer contains two core components: the self-attention mechanism allows the model to focus on the semantic information of all other words in the sentence when processing each word, thereby capturing long-distance dependencies (for example, "apple" in "eat apples" represents fruit and in "Apple Company" represents the brand); the feedforward neural network performs nonlinear transformation on the features output by the self-attention to enhance the semantic representation capability.

[0095] In addition, each encoder layer also contains residual connections and layer normalization to stabilize the training process and prevent gradient disappearance.

[0096] Convert words into dense vectors that contain the basic semantics of the words (for example, the vector of "apple" is different from that of "banana") and add information about the position of the words in the sentence. This solves the problem that the Transformer itself lacks sequence order perception (for example, distinguishing between "I eat apples" and "apples eat me"). It is used to process sentence pair inputs (such as question-and-answer scenarios) and distinguish the semantic boundaries of different paragraphs.

[0097] Randomly mask some words in a sentence, allowing the model to predict the masked words based on the context, forcing the model to learn the dependencies between words; determine whether two sentences are consecutive paragraphs to enhance the model's understanding of the semantics of the paragraph (some models such as RoBERTa have removed this goal).

[0098] The input word (such as "apple") is converted into a high-dimensional word vector (such as 768 dimensions) through a pre-trained embedding matrix. This vector contains the semantic commonalities (if the real-class words are located close in the vector space) and uniqueness (the vector differences between different words). The word embedding, position embedding, and paragraph embedding are added together to form the final input vector, ensuring that the model simultaneously captures the word semantics, position order, and paragraph structure information.

[0099] For each word, its "attention weight" is calculated relative to all other words in the sentence. A higher weight indicates a stronger semantic connection. For example, in the sentence "I like to eat apples," the attention weight of "eat" relative to "apple" is higher than its weight relative to "I," allowing the model to capture the subject-object relationship of "eating apples." Self-attention is split into multiple "heads," each capturing semantic relationships from a different subspace (e.g., some focusing on grammatical structure, others on entity relationships). The outputs are then merged to enhance feature diversity. The features output by self-attention undergo nonlinear transformations (e.g., ReLU activation functions) to further refine abstract semantics (e.g., elevating the vector of "apple" from a concrete, physical concept to a more abstract semantic representation).

[0100] The output of a multi-layer Transformer encoder contains semantic information at different levels (the bottom layer is more grammatical, while the upper layers are more semantic). The outputs of all layers are fused through a weighted summation, for example, 30% for the bottom layer, 40% for the middle layer, and 30% for the upper layer, to form a raw semantic feature vector containing rich context. The weights can be learned automatically through training or fixed average weights can be used to ensure that the final vector contains both basic grammatical information and high-level semantic understanding (for example, the semantic differences of the word "apple" in different contexts are captured by the high-level network and reflected in the vector).

[0101] Through large-scale unsupervised data pre-training, the model can learn the general rules of language (such as part-of-speech collocation and entity relationships) without the need to retrain the underlying network for specific tasks; compared with traditional word vectors (such as Word2Vec), the word vectors of the pre-trained model are "dynamic", and the same word will generate different vectors in different contexts (such as the vector of "apple" in "eating apples" and "Apple Inc." is different), which is more in line with the semantic variability of natural language; as a "basic component of language understanding", the pre-trained model only needs a small amount of domain data fine-tuning to adapt to specific tasks (such as sentiment analysis and question-answering systems), greatly reducing training costs.

[0102] In step S32, for each dimension of the original semantic feature vector, the cosine similarity between the feature and the domain term vector in the term library (e.g., "machine learning" and "neural network") is calculated to measure the semantic relevance between the feature and the domain term. Term vectors with a similarity exceeding a threshold (e.g., 0.7) are selected to form a set of related terms (e.g., "apple" in the technology field may be associated with "iPhone" and "iOS"). Weight coefficients are normalized based on the similarity between the term vector and the original feature (e.g., if the similarity for "iPhone" is 0.8 and the similarity for "iOS" is 0.9, the weights are 0.47 and 0.53, respectively). These weight coefficients are expanded into a diagonal matrix (e.g., 768×768), where the values ​​on the diagonal are the weights of the corresponding dimensions and the remaining elements are 0.

[0103] In the above step S33, the domain weight matrix is ​​multiplied by the original semantic feature vector so that the eigenvalue of each dimension is scaled according to the domain relevance (for example, the dimensions related to the science and technology field are enhanced, and the general dimensions remain unchanged); the scaled feature vector is added to the original feature vector to retain the original information to prevent overfitting of domain knowledge (for example, the meaning of "apple" in the general context is not fully covered by the science and technology field weight); the fused feature vector is normalized to stabilize the feature distribution, and the corrected semantic feature vector is output.

[0104] In a preferred embodiment of the present invention, the above step S4: performing few-sample learning on the corrected semantic features, enhancing the recognition capability of low-frequency text patterns through a contrastive learning mechanism, and generating an enhanced feature vector resistant to long-tail interference, may include:

[0105] Step S41: extract semantic feature vectors of high-frequency samples from the annotation library to form a high-frequency sample set, and simultaneously extract semantic feature vectors of low-frequency samples to form a low-frequency sample set; for each target sample in the low-frequency sample set, randomly select high-frequency samples of the same category as positive samples and high-frequency samples of different categories as negative samples; combine the target sample, positive sample, and negative sample into a triplet to generate a set of comparative learning sample pairs;

[0106] Step S42: For all sample feature vectors of each category in the high-frequency sample set, their geometric centers are calculated as the prototype vectors of the category; for the target sample in the comparative learning sample pair, the Euclidean distance between the target sample and the prototype vector of the positive sample category and the Euclidean distance between the target sample and the prototype vector of the negative sample category are calculated; and an inverse weight coefficient is applied to the Euclidean distance according to the category distribution frequency of the low-frequency samples;

[0107] In step S43, the goal is to minimize the distance between the target sample and the positive sample prototype and maximize the distance between the target sample and the negative sample prototype; the feature encoding layer parameters are adjusted by gradient back propagation so that low-frequency samples are clustered towards similar prototypes in the feature space and form an interval boundary with heterogeneous prototypes. The iterative update is performed until the distance loss function converges, and an enhanced feature vector with anti-long-tail distribution characteristics is output.

[0108] In an embodiment of the present invention, high-frequency samples are used to provide semantic references for low-frequency samples, and a contrastive learning framework is used to force the model to learn similarities within categories and differences between categories, thereby alleviating the problem of fuzzy feature representation of low-frequency samples caused by insufficient labeled data; the prototype vector, as the central representation of category semantics, can effectively summarize category commonalities; the inverse weight mechanism enables the model to pay more attention to the feature differences of rare categories, avoiding the neglect of low-frequency categories due to the dominance of high-frequency categories in training; through the distance constraint of geometric space, the model's recognition ability of low-frequency text patterns is significantly improved. Even in long-tail distribution scenarios, reliable classification boundaries can be established by comparing high-frequency category features, reducing the problem of misjudgment of rare categories.

[0109] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0110] In the above step S41, the semantic feature vectors are divided into a high-frequency sample set (such as a category with a frequency of occurrence ≥ 100 times) and a low-frequency sample set (such as a category with a frequency of occurrence < 10 times) according to the frequency of occurrence of samples in the annotation library.

[0111] For each target sample in the low-frequency sample set, randomly select 1-5 high-frequency samples of the same category as positive samples, and select 5-10 high-frequency samples of other categories as negative samples; combine each target sample with the corresponding positive sample and negative sample into a "target-positive-negative" triplet, for example: (low-frequency sample A, high-frequency sample B of the same category, high-frequency sample C of a different category), and finally form a set of contrastive learning sample pairs.

[0112] In the above step S42, for each category in the high-frequency sample set, the geometric center (i.e., the mean vector) of the feature vectors of all samples in the category is calculated as the prototype vector of the category (e.g., the prototype of category K is the average feature of all high-frequency samples of the category); for each target sample, its distance from the prototype of the positive sample category (e.g., the distance from target sample A to the prototype of category K) and the distance from the prototype of the negative sample category (e.g., the distance from target sample A to the prototype of category M) are calculated respectively.

[0113] According to the category distribution frequency of low-frequency samples (for example, category K accounts for 1% of low-frequency samples, and category M accounts for 5%), the distance result is multiplied by an inverse weight (for example, the weight of category K is 1 / 0.01=100, and the weight of category M is 1 / 0.05=20), so that the distance error of rare category samples is given a greater weight.

[0114] In the above step S43, two objectives are optimized simultaneously through the loss function: reducing the distance between the target sample and the positive sample prototype (causing similar features to cluster); and increasing the distance between the target sample and the negative sample prototype (causing heterogeneous features to separate).

[0115] Use gradient backpropagation to adjust the parameters of the feature encoding layer, such as the weights of the last few layers of the Transformer, so that the feature vectors of low-frequency samples move toward similar prototypes in space and away from heterogeneous prototypes; iterative convergence: continue training until the distance loss function (such as TripletLoss) no longer decreases significantly. At this time, the low-frequency samples form a clear category boundary in the feature space and output an enhanced feature vector.

[0116] In a preferred embodiment of the present invention, the above step S5: based on the enhanced feature vector, a multi-dimensional tag extraction operation is synchronously performed to output a tag set containing the following four types of tags: demand type tags, sentiment tendency tags, product or service entity tags, and question keyword tags, which may include:

[0117] Step S51, inputting the enhanced feature vector outputted in step S4 into the shared feature layer of the multi-task classifier, compressing the feature dimension through a fully connected neural network, and generating a reduced-dimensional feature representation shared by the four tasks;

[0118] In step S52, the dimensionality-reduced feature representation is input into the first sub-classifier to calculate the probability distribution of the text belonging to each demand category; the probability value is mapped to the interval [0, 1] through a normalization operation, and the category corresponding to the maximum probability value is selected as the demand type label; the same dimensionality-reduced feature representation is input into the second sub-classifier to calculate the sentiment tendency cutoff value: a positive and negative sentiment decision plane is constructed, and the geometric distance from the feature representation to the decision plane is calculated; the feature slice of each word position in the enhanced feature vector is sliced, the probability of entity labeling is calculated, the global optimal label sequence is solved by the Viterbi algorithm, and continuous labels are merged to form an entity label; the cosine similarity between the feature vector of each word and the feature vector of the full text is calculated, the inverse document frequency weight of the word in the full text is integrated, and the top N words ranked in descending order of the comprehensive score are selected as the question keyword labels;

[0119] Step S53, establish a label storage structure, create a label record containing four fields for each text; write the demand type label into the "category" field, and the sentiment tendency label into the "sentiment" field; write the product or service entity label into the "entity" field in the form of a list; write the question keyword label into the "keyword" field in the form of a descending weight list, and output a label set containing four types of structured labels.

[0120] In an embodiment of the present invention, the enhanced feature vector is reduced in dimensionality by sharing a feature layer, thereby avoiding repeated feature extraction for different labeling tasks, reducing computing resource consumption, retaining the core semantics of the text, and realizing efficient parallel extraction of multi-dimensional labels. The extracted demand type label accurately locates the business attributes of the text based on probabilistic classification, providing a basis for work order allocation, etc. The sentiment tendency label quickly judges positive and negative emotions through geometric distance calculation to assist public opinion analysis. The entity label uses sequence annotation and Viterbi algorithm to identify key objects and clarify core entities. The keyword label combines semantic relevance and inverse document frequency weight to refine the core of the text theme. The standardized four-field label record supports efficient database storage and query, and is easy to integrate into downstream scenarios such as customer service systems and data analysis platforms. It can automatically route work orders, assist business decisions and reduce the labeling cost of machine learning models. Combined with the enhanced feature vector, the accuracy of identifying low-frequency demand types or rare entities is improved, and misjudgment caused by long-tail data is reduced. Feature dimensionality reduction and multi-task collaboratively optimize label quality, so that the accuracy of demand type label classification is improved, the speed of sentiment judgment is improved, and keyword extraction can better reflect the text theme than simple word frequency statistics.

[0121] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0122] In the above step S51, the enhanced feature vector (e.g., 768 dimensions) is input into the shared feature layer of the multi-task classifier, which is composed of a fully connected neural network (e.g., containing 128 neurons); through the linear transformation and activation function (e.g., ReLU) of the fully connected layer, the high-dimensional features are compressed into low-dimensional shared features (e.g., from 768 dimensions to 128 dimensions). This process retains the core semantic commonalities of the text; the feature representation after dimensionality reduction is simultaneously output to four sub-classifiers (demand type, sentiment tendency, entity recognition, and keyword extraction) to achieve feature sharing and multi-task parallel processing.

[0123] In the above step S52, the dimensionality reduction features are input to the first sub-classifier (such as the Softmax classifier) ​​to calculate the probability that the text belongs to each demand category (such as "consultation", "complaint", and "suggestion"); after normalizing the probability values, the category corresponding to the maximum probability is selected as the final label (for example, if the probability of "consultation" is 0.7 and "complaint" is 0.2, the label is "consultation").

[0124] Input the features to the second sub-classifier, and generate the emotion decision plane parameters (such as weight vector and bias) through the fully connected layer; calculate the geometric distance from the feature vector to the decision plane: a positive distance is judged as positive emotion, and a negative distance is judged as negative emotion (for example, a distance > 0.5 is "positive", and < -0.5 is "negative").

[0125] For each word position in the enhanced feature vector, the entity tagging probability is calculated (such as "B-product", "I-product", "O"); the Viterbi algorithm is used to solve the global optimal tag sequence, and continuous tags are merged to form an entity (such as "E mobile phone" is marked as "B-product I-product", and merged into the entity "E mobile phone").

[0126] Calculate the cosine similarity between each word's feature vector and the full-text feature vector to measure the relevance of the word to the text topic; combine the word's inverse document frequency (IDF) weight (low-frequency words have higher weights) to calculate a comprehensive score (such as similarity × IDF); select the top N words (such as N = 5) in descending order of score as keywords (such as "price", "service", "refund").

[0127] The above step S53 generates a structured record containing four fields for each text; field filling: "Category" field: write the demand type label (such as "consultation"); "Sentence" field: write the sentiment tendency label (such as "positive" and "negative"); "Entity" field: store the identified entities in the form of a list (such as ["E mobile phone", "after-sales service"]); "Keyword" field: store a list of keywords sorted in descending order by weight (such as [{"price": 0.85}, {"refund": 0.72}]); label set output: summarize the label records of all texts into a structured label set, supporting batch query and analysis.

[0128] In a preferred embodiment of the present invention, step S6: for each tag in the tag set, locating its key decision words in the enhanced feature vector through the attention mechanism, generating a natural language description of the tag determination basis, and forming an enhanced tag set with explanatory text, may include:

[0129] Step S61: for each tag in the tag set, trace back to the subclassifier decision layer that generated the tag in step S5, and extract the attention weight matrix of the decision layer to the enhanced feature vector output by step S4;

[0130] Step S62, traversing the weight vector corresponding to the current label in the attention weight matrix, identifying the weight peak position whose weight value exceeds a preset peak threshold, and mapping the peak position back to the word index in the structured text data generated in step S2;

[0131] Step S63, extracting original words from the structured text data according to the word index, sorting them from high to low by weight value to generate a key decision word sequence, and merging adjacent word indexes into semantic phrases when they are continuous;

[0132] Step S64: Fill the demand type label, sentiment label, and entity label into the corresponding template according to the key decision word sequence;

[0133] Step S65: append the generated explanatory text to the original label to form an enhanced label tuple. All label tuples constitute an enhanced label set with explanatory text.

[0134] In an embodiment of the present invention, the weight matrix corresponding to the label is traced back through the attention mechanism to locate the key decision words in the text that affect the label judgment, thereby improving the interpretability of the model decision; the weight peak is mapped to the original text word index, so that the label is directly associated with the text expression, enhancing semantic traceability; the key decision words are filled in with the help of templates to generate natural language explanations, reducing the understanding cost of business personnel; the key decision word sequence is used to assist in detecting model misjudgments, facilitating model optimization; in business scenarios such as customer service and medical care, manual review is reduced and system credibility is improved; the templated generation method ensures that the label interpretation is standardized and unified, which is conducive to comparative analysis; the natural language explanation promotes human-computer collaboration, so that non-technical personnel can also quickly understand the meaning of the label.

[0135] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0136] In the above step S61, the sub-classifier corresponding to the current label is determined (such as the demand type label corresponds to the first sub-classifier, and the emotion label corresponds to the second sub-classifier); the last decision layer of the sub-classifier is entered (such as the fully connected layer or the Softmax layer), and the attention weight matrix of each dimension of the enhanced feature vector when the layer generates the current label is extracted. The matrix records the degree of attention to each feature dimension when the model makes decisions; the rows of the weight matrix correspond to the dimensions of the enhanced feature vector, and the columns correspond to the label categories. The column vector corresponding to the current label is extracted as the weight vector of the label.

[0137] In step S62, the weight vectors of the current label are traversed one by one to identify the positions where the weight values ​​are higher than a preset threshold (such as 0.7). These positions are called "weight peaks" and represent the feature dimensions that the model focuses on when making decisions.

[0138] Since each dimension of the enhanced feature vector corresponds to the feature of a word in the structured text (such as the i-th dimension corresponds to the word wi), the feature dimension index corresponding to the weight peak is mapped to the word index in the structured text; invalid indexes that exceed the text length are filtered out to ensure that they are mapped to the actual word position.

[0139] In the above step S63, the corresponding original words are extracted from the structured text data according to the word index obtained by mapping (for example, index 0 corresponds to "I", and index 1 corresponds to "purchase"); the extracted words are sorted from high to low according to the weight value to form a key decision word sequence (for example, "fault" with a weight of 0.9 is ranked before "service" with a weight of 0.7); the sorted word indexes are checked to see whether they are continuous (for example, indexes 3, 4, and 5 correspond to "battery", "battery life", and "short"); if they are continuous, they are merged into semantic phrases ("battery life is short") to improve the readability of the explanation text.

[0140] In the above step S64, it is determined whether the current label belongs to the demand type, sentiment tendency or entity label; the demand type label template: "due to the appearance of the keyword [key decision word sequence], it is determined to be [demand type]" (such as "due to the appearance of 'refund' and 'quality problem', it is determined to be 'complaint'"); the sentiment tendency label template: "based on the weight comparison of the keywords [positive words] and [negative words], it is determined to be [sentiment tendency]" (such as "based on the weight of 'satisfaction' 0.8 and the weight of 'disappointment' 0.2, it is determined to be 'positive'"); the entity label template: "by identifying the continuous keyword [entity phrase], it is determined to be [entity type]" (such as "through 'EMate60', it is determined to be 'product entity'"); keyword filling: the sorted key decision word sequence is filled in the corresponding position according to the template rules to generate a natural language explanation text.

[0141] In step S65, the original tag and the generated explanation text are combined into a tuple (e.g., ("complaint", "due to the occurrence of 'refund' 'quality issues', it is determined to be 'complaint'"). Each tuple contains a tag field and an explanation field, forming a structured record. All tag tuples are aggregated into an enhanced tag set, supporting retrieval and analysis by tag type or explanation text.

[0142] In a preferred embodiment of the present invention, the above step S7: triggering incremental training of the pre-trained NLP model based on the distribution of low-frequency tags and the frequency of newly added domain terms in the enhanced tag set may include:

[0143] Step S71: traverse all tags in the enhanced tag set, count the occurrence frequency of each demand type tag, and mark the tags with a cumulative frequency lower than 0.1% of the total sample size as long-tail tags;

[0144] Step S72: monitor the number of newly added records in the domain terminology database in real time, and calculate the ratio of the number of newly added terms in the past 30 days to the total number of terminology database as the monthly update rate;

[0145] Step S73: Incremental training is triggered when the following two conditions are met simultaneously:

[0146] Condition 1: The proportion of long-tail tags to all demand type tags is greater than 15%;

[0147] Condition 2: Monthly terminology update rate > 5%;

[0148] Step S74: extract all original text data with long-tail tags in step S1 as first-category samples; construct pseudo-annotated samples from the newly added terms and their context fragments as second-category samples;

[0149] In step S75, the first and second category samples are input into the pre-trained NLP model for incremental training, the underlying parameters of the pre-trained NLP model are frozen, the weight matrix of the domain adaptation layer in step S3 and the fully connected layer parameters of the multi-task classifier in step S5 are updated, and the training is terminated when the label recall rate of the pre-trained NLP model on the validation set is improved by less than 1%.

[0150] In an embodiment of the present invention, by monitoring the update rate of low-frequency tags and domain terms, incremental training is automatically triggered, so that the model can continuously adapt to newly emerging business scenarios and avoid performance degradation due to data distribution shift; training is triggered only when the proportion of long-tail tags exceeds 15% and the term update rate exceeds 5%, which balances the cost and benefit of model updates, avoids resource waste caused by frequent training, and ensures that the model is optimized at key nodes; incremental training is performed on the original text corresponding to low-frequency tags to improve the model's ability to recognize rare demand types; new terms and their contexts are constructed into pseudo-annotated samples, so that the model can quickly learn new industry vocabulary, shorten the cycle from the emergence of business terms to model adaptation, and maintain technological advancement; the underlying parameters of the pre-trained model are frozen, and only the domain adaptation layer and multi-task classifier parameters are updated to reduce training time, avoid catastrophic forgetting, and ensure that the model retains learned knowledge while adapting to new data; the verification set label recall rate improvement rate of <1% is used as the termination condition to ensure that the training is both sufficient and not excessive, and to prevent overfitting.

[0151] In the embodiments of the present invention, when applied specifically, it can be achieved through the following technical solutions, for example:

[0152] In step S71, all demand type tags (such as "consultation" and "complaint") in the enhanced tag set are traversed, and a counter is used to record the number of occurrences of each tag. For example, assuming there are 100,000 texts, of which "cross-border payment consultation" appears 50 times and "Metaverse product complaint" appears 8 times; 0.1% of the total sample size is calculated as the long-tail tag threshold. If the total sample size is 100,000, the threshold is 100,000 × 0.1% = 100 times; tags with an occurrence frequency below the threshold are marked as long-tail tags. For example, if "Metaverse product complaint" appears 8 times but less than 100 times, it is marked as a long-tail tag; if "cross-border payment consultation" appears 50 times but less than 100 times, it is also marked as a long-tail tag.

[0153] In step S72, the domain terminology database (e.g., the financial domain terminology database) is monitored in real time, recording any new terminology entries added in the past 30 days. For example, if the terminology database originally contained 1,000 terms and 60 were added in the past 30 days (e.g., "blockchain smart contract" and "AI large-scale risk control"). The monthly update rate is calculated by dividing the number of newly added terms by the total number of terms in the terminology database. The formula is: Monthly Update Rate = Number of Newly Added Terms / Total Number of Terms in the terminology database × 100%. For example, 60 / 1,000 = 6%, indicating a monthly update rate of 6%.

[0154] In step S73, the number of long-tail tags in all demand type tags is counted and divided by the total number of tag types to obtain the long-tail tag ratio. For example, there are 200 total demand tags, of which 35 are long-tail tags, and the ratio is 35 / 200=17.5%.

[0155] Check whether the proportion of long-tail tags is greater than 15% (for example, 17.5% > 15%, meeting condition one); check whether the monthly update rate of the terminology database is greater than 5% (for example, 6% > 5%, meeting condition two). When both conditions are met, trigger incremental training.

[0156] In the above step S74, the original text data of step S1 is traced back to filter out all texts with long-tail tags, for example, the original user feedback text with the tag "Metaverse product complaint"; the noise in the text (such as special symbols and garbled characters) is removed, and the structured word sequence is retained (such as "The purchased Metaverse product cannot be logged in, and the customer service does not handle it").

[0157] For each term added in the past 30 days (such as "blockchain smart contract"), contextual fragments containing the term are extracted from historical text data (such as contract clause text containing "blockchain smart contract"); the newly added terms are marked as corresponding entity types (such as "blockchain smart contract" is marked as "technical term - blockchain"), and pseudo-annotated samples are constructed (such as "The execution process of [blockchain smart contract] must comply with regulations" → marked as the entity "blockchain smart contract").

[0158] In the above step S75, the storage file of the pre-trained model is read (such as HuggingFace's `pytorch_model.bin` and TensorFlow's `checkpoint` file), and the file structure is parsed to obtain the model parameters and configuration information; the model configuration file (such as `config.json`) is read to obtain the model architecture parameters (such as the number of Transformer layers, hidden layer dimensions, number of attention heads, etc.), ensuring that the loaded model structure is consistent with that during training; the weight matrix and bias items (such as the embedding layer weights and the weight parameters of each layer of the Transformer encoder) are extracted from the binary parameter file and mapped to the corresponding network layer according to the model architecture hierarchy.

[0159] Reconstruct the model architecture according to the configuration information, such as creating components such as a multi-layer Transformer encoder, an embedding layer, a domain adaptation layer, and a multi-task classifier; create an encoder with a specified number of layers (such as 12 layers) according to the configuration, each layer containing a self-attention module and a feedforward neural network; instantiate the domain weight matrix storage structure (such as a diagonal matrix) in step S3; and construct a fully connected network structure with a shared feature layer and four sub-classifiers.

[0160] The parsed pre-trained parameters are mapped to the reconstructed model architecture hierarchically. For example, the embedding layer weight matrix is ​​loaded into the model input layer; the weight parameters of each layer of the Transformer encoder are loaded into the corresponding self-attention module and feedforward network; the initial parameters of the domain adaptation layer and multi-task classifier are loaded if pre-trained values ​​exist, and randomly initialized if not.

[0161] Target the underlying parameters of the Transformer encoder (e.g., the first 10 layers) and disable gradient updates by setting `requires_grad = False`. For example, in PyTorch, use `forparaminmodel.transformer.encoder.layers[:10].parameters():param.requires_grad = False`. The freezing logic is based on the assumption that "the underlying network learns general language rules, and the higher-level network learns domain-specific knowledge", ensuring that the model retains basic semantic understanding capabilities during incremental training.

[0162] For the weight matrix of the domain adaptation layer (such as the domain weight matrix in step S3) and the fully connected layer parameters of the multi-task classifier, set `requires_grad = True` to allow gradient updates. For example, open `model.domain_adapter.weight` and `model.classifier.fc.parameters()` update permissions.

[0163] Load model parameters to the corresponding computing device based on hardware resources (CPU / GPU). For example, use `model.to(device)` to transfer the model to the GPU for accelerated training. If the hardware supports it (such as NVIDIA GPU's TensorCore), enable mixed precision training configuration (such as PyTorch's `torch.cuda.amp`) when loading the model to reduce memory usage and speed up calculations. If the sample size is large, configure a distributed training strategy (such as DataParallel or DistributedDataParallel) when loading the model to achieve multi-GPU parallel computing.

[0164] Check whether the loaded parameter dimensions match the model architecture (for example, the embedding layer weight dimension is [vocabulary size, 768]) to avoid dimension errors caused by version incompatibility; use a small number of samples (such as 10 texts) for forward propagation to verify whether the model output meets expectations (for example, the probability distribution range of the requirement type label is between [0, 1]) to ensure that the loaded model functions normally; verify the semantic representation ability of the model through word vector similarity testing to ensure that the pre-trained knowledge of the underlying Transformer encoder is not lost.

[0165] After each round of training, the label recall rate (such as the long-tail label recall rate) is calculated on the validation set. When the recall rate of 5 consecutive rounds of training is improved by less than 1% (for example, from 70% to 70.5%, the improvement rate is 0.5% < 1%), the training is terminated.

[0166] like Figure 2As shown, an embodiment of the present invention further provides a text data automatic annotation system integrating a pre-trained NLP model, comprising:

[0167] An acquisition module is used to acquire the original text data to be annotated, including conversation records, evaluations, and work orders of customer inquiries, complaints, or feedback;

[0168] The preprocessing module is used to perform word segmentation, stop word filtering and part-of-speech tagging on the original text data to generate structured text data;

[0169] The correction module is used to input structured text data into the pre-trained NLP model and correct the semantic features output by the model through the domain adaptation layer to obtain the corrected semantic features;

[0170] The feature vector module is used to perform few-shot learning on the corrected semantic features. It strengthens the recognition of low-frequency text patterns through a contrastive learning mechanism and generates enhanced feature vectors that are resistant to long-tail interference.

[0171] The tag module is used to synchronously perform multi-dimensional tag extraction operations based on the enhanced feature vectors, and output a tag set containing the following four types of tags: demand type tags, sentiment tendency tags, product or service entity tags, and question keyword tags;

[0172] The explanatory module is used to locate the key decision words in the enhanced feature vector of each label in the label set through the attention mechanism, generate a natural language description of the label decision basis, and form an enhanced label set with explanatory text;

[0173] The incremental module is used to trigger incremental training of the pre-trained NLP model based on the distribution of low-frequency tags in the enhanced tag set and the frequency of new domain terms.

[0174] It should be noted that this system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0175] An embodiment of the present invention further provides a computing device comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the above-described method. All implementations in the above-described method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0176] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for automatic annotation of text data integrating a pre-trained NLP model, characterized in that: The method comprises: Step S1: Obtaining original text data to be annotated, wherein the text data includes conversation records, evaluations, and work orders of customer consultations, complaints, or feedback; Step S2: performing word segmentation, stop word filtering, and part-of-speech tagging on the original text data to generate structured text data; Perform word segmentation, stop word filtering, and part-of-speech tagging on the original text data to generate structured text data, including: Scan each character encoding of the original text data and process it to generate a purified text; Based on the purified text, the probability distribution of each character at its current position as the start, middle, or end of a word is calculated. Combined with the conditional random field state transition score, dynamic programming is used to generate the global maximum probability segmentation path, outputting a basic word segmentation sequence with clear word boundaries. According to the stop word dictionary stored in the hash table, each word in the basic word segmentation sequence is traversed. If the word hits the stop word hash table and is not marked as a protected entity by the named entity recognition module, the word is removed; if the word belongs to the entity protection list, it is retained in the new sequence; generate a filtered word sequence carrying the core semantic unit; For each target word in the filtered word sequence, the context windows of the two words before and after it are extracted and combined into a feature template. The pre-trained part-of-speech probability matrix is ​​queried to calculate the joint probability that the target word belongs to each part-of-speech category in its context. A part-of-speech state transition grid is constructed based on dynamic programming. The cumulative probability of all part-of-speech tagging paths from the first word to the last word in the sequence is iteratively calculated, and the path with the maximum probability is selected as the final part-of-speech sequence. Each word and its optimal part-of-speech tag are combined into a bigram to generate structured text data with grammatical annotations. Step S3: Input the structured text data into the pre-trained NLP model, and modify the semantic features output by the model through the domain adaptation layer to obtain modified semantic features; Step S4: Performing few-shot learning on the corrected semantic features, enhancing the recognition capability of low-frequency text patterns through a contrastive learning mechanism, and generating enhanced feature vectors that are resistant to long-tail interference; Step S5: Based on the enhanced feature vector, a multi-dimensional tag extraction operation is synchronously performed to output a tag set containing the following four types of tags: demand type tag, sentiment tendency tag, product or service entity tag, and question keyword tag; Step S6: For each tag in the tag set, locate its key decision words in the enhanced feature vector through the attention mechanism, generate a natural language description of the tag decision basis, and form an enhanced tag set with explanatory text; Step S7: Based on the distribution of low-frequency tags and the frequency of newly added domain terms in the enhanced tag set, trigger the incremental training of the pre-trained NLP model.

2. The method for automatically annotating text data using an integrated pre-trained NLP model according to claim 1, wherein: Step S3: Input the structured text data into the pre-trained NLP model and modify the semantic features output by the model through the domain adaptation layer to obtain modified semantic features, including: Input the word sequence in the structured text data generated in step S2 into the embedding layer of the pre-trained NLP model, convert each word into a high-dimensional word vector through the word vector mapping operation, extract context features through the multi-layer Transformer encoder, and output the original semantic feature vector; The original semantic feature vector is input into the domain adaptation layer. For each feature dimension of the original semantic feature vector, its cosine similarity with all term vectors in the term library is calculated. A set of term vectors whose similarity exceeds a preset threshold is selected. Based on the similarity value of each term vector in the set and the original feature, a normalized weighted sum is performed to generate a domain weight vector. The domain weight vector is expanded into a diagonal matrix form according to the feature dimension to form a domain weight matrix. The domain weight matrix is ​​multiplied by the original semantic feature vector so that the original features are scaled according to the domain relevance in each dimension. The scaled feature vector is residually connected with the original semantic feature vector, and the original features and domain enhancement features are fused through element-level addition to obtain the fused feature vector. The fused feature vector is layer-normalized and the corrected semantic feature vector is output.

3. The method for automatically annotating text data using an integrated pre-trained NLP model according to claim 2, wherein: Step S4: Perform few-sample learning on the corrected semantic features, enhance the recognition ability of low-frequency text patterns through the contrastive learning mechanism, and generate enhanced feature vectors that are resistant to long-tail interference, including: The semantic feature vectors of high-frequency samples are extracted from the annotation database to form a high-frequency sample set, and the semantic feature vectors of low-frequency samples are extracted to form a low-frequency sample set. For each target sample in the low-frequency sample set, high-frequency samples of the same category are randomly selected as positive samples, and high-frequency samples of different categories are selected as negative samples. The target sample, positive sample, and negative sample are combined into a triplet to generate a set of contrastive learning sample pairs. For all sample feature vectors of each category in the high-frequency sample set, their geometric center is calculated as the prototype vector of the category; for the target sample in the contrastive learning sample pair, the Euclidean distance between it and the prototype vector of the positive sample category and the Euclidean distance between it and the prototype vector of the negative sample category are calculated respectively; according to the category distribution frequency of the low-frequency samples, an inverse weight coefficient is applied to the Euclidean distance; The goal is to minimize the distance between the target sample and the positive sample prototype, while maximizing the distance with the negative sample prototype; the feature encoding layer parameters are adjusted through gradient backpropagation, so that low-frequency samples are clustered towards similar prototypes in the feature space and form an interval boundary with heterogeneous prototypes. It is iteratively updated until the distance loss function converges, and the enhanced feature vector with anti-long-tail distribution characteristics is output.

4. The method for automatically annotating text data using an integrated pre-trained NLP model according to claim 3, wherein: Step S5: Based on the enhanced feature vector, a multi-dimensional tag extraction operation is performed synchronously to output a tag set containing the following four types of tags: demand type tags, sentiment tendency tags, product or service entity tags, and question keyword tags, including: The enhanced feature vector output from step S4 is input into the shared feature layer of the multi-task classifier, and the feature dimension is compressed through a fully connected neural network to generate a reduced-dimensional feature representation shared by the four tasks; The reduced-dimensional feature representation is input into the first sub-classifier to calculate the probability distribution of the text belonging to each demand category; the probability value is mapped to the interval [0, 1] through normalization operation, and the category corresponding to the maximum probability value is selected as the demand type label; the same reduced-dimensional feature representation is input into the second sub-classifier to calculate the sentiment tendency cutoff value: a positive and negative sentiment decision plane is constructed, and the geometric distance from the feature representation to the decision plane is calculated; the feature slice of each word position in the enhanced feature vector is sliced, the probability of entity labeling is calculated, the global optimal label sequence is solved by the Viterbi algorithm, and continuous labels are merged to form an entity label; the cosine similarity between the feature vector of each word and the feature vector of the full text is calculated, the inverse document frequency weight of the word in the full text is integrated, and the top N words ranked in descending order of the comprehensive score are selected as the question keyword labels; Establish a tag storage structure and create a tag record containing four fields for each text; write the requirement type label into the "Category" field, and the sentiment tendency label into the "Sentiment" field; write the product or service entity label into the "Entity" field in list form; write the question keyword label into the "Keyword" field in descending weight list form, and output a tag set containing four types of structured labels.

5. The method for automatically annotating text data using an integrated pre-trained NLP model according to claim 4, wherein: Step S6: For each tag in the tag set, the attention mechanism is used to locate its key decision words in the enhanced feature vector, and a natural language description of the tag decision basis is generated to form an enhanced tag set with explanatory text, including: For each label in the label set, trace back to the subclassifier decision layer that generated the label in step S5, and extract the attention weight matrix of the decision layer to the enhanced feature vector output by step S4; Traversing the weight vector corresponding to the current label in the attention weight matrix, identifying the weight peak position whose weight value exceeds the preset peak threshold, and mapping the peak position back to the word index in the structured text data generated in step S2; Extract original words from structured text data based on word indexes, sort them by weight from high to low to generate a sequence of key decision words, and merge adjacent word indexes into semantic phrases when they are continuous. According to the sequence of key decision words, fill the demand type label, sentiment label, and entity label into the corresponding template; The generated explanatory text is appended to the original label to form an enhanced label tuple, and all label tuples constitute a set of enhanced labels with explanatory text.

6. The method for automatically annotating text data using an integrated pre-trained NLP model according to claim 5, wherein: Step S7: Based on the distribution of low-frequency tags and the frequency of newly added domain terms in the enhanced tag set, trigger incremental training of the pre-trained NLP model, including: Traverse all tags in the enhanced tag set, count the occurrence frequency of each requirement type tag, and mark tags with a cumulative frequency lower than 0.1% of the total sample size as long-tail tags; Monitor new records in the domain terminology database in real time and calculate the ratio of the number of new terms added in the past 30 days to the total number of terminology databases as the monthly update rate; Incremental training is triggered when both of the following conditions are met: Condition 1: The proportion of long-tail tags to all demand type tags is greater than 15%; Condition 2: Monthly terminology database update rate > 5%; Extract all the original text data with long-tail tags in step S1 as the first category of samples; construct the newly added terms and their context fragments into pseudo-annotated samples as the second category of samples; Input the first and second category samples into the pre-trained NLP model for incremental training, freeze the underlying parameters of the pre-trained NLP model, update the weight matrix of the domain adaptation layer in step S3 and the fully connected layer parameters of the multi-task classifier in step S5, and terminate the training when the label recall rate of the pre-trained NLP model on the validation set is improved by less than 1%.

7. A text data automatic annotation system integrated with a pre-trained NLP model, the system implementing the method according to any one of claims 1 to 6, characterized in that: include: An acquisition module is used to acquire the original text data to be annotated, including conversation records, evaluations, and work orders of customer inquiries, complaints, or feedback; The preprocessing module is used to perform word segmentation, stop word filtering and part-of-speech tagging on the original text data to generate structured text data; The correction module is used to input structured text data into the pre-trained NLP model and correct the semantic features output by the model through the domain adaptation layer to obtain the corrected semantic features; The feature vector module is used to perform few-shot learning on the corrected semantic features. It strengthens the recognition of low-frequency text patterns through a contrastive learning mechanism and generates enhanced feature vectors that are resistant to long-tail interference. The tag module is used to synchronously perform multi-dimensional tag extraction operations based on the enhanced feature vectors, and output a tag set containing the following four types of tags: demand type tags, sentiment tendency tags, product or service entity tags, and question keyword tags; The explanatory module is used to locate the key decision words in the enhanced feature vector of each label in the label set through the attention mechanism, generate a natural language description of the label decision basis, and form an enhanced label set with explanatory text; The incremental module is used to trigger incremental training of the pre-trained NLP model based on the distribution of low-frequency tags in the enhanced tag set and the frequency of new domain terms.

8. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images