Generalized classification method and device based on large language model for domain perception

Abstract

The application provides a generalized classification method and device based on a large language model. The method comprises: obtaining an original data set of a specified scene, the original data set comprising labeled data and unlabeled data, each data in the labeled data having a fine-grained class label; determining a coarse-grained domain class of each data by using a large language model, and labeling respectively to obtain target domain data, the target domain data comprising coarse-grained domain data and fine-grained domain data; constructing a double-head structure representation model, and performing two consecutive training of the double-head structure representation model by using the target domain data; inputting the target domain data into the trained double-head structure representation model, and determining a generalized classification result according to an output result by using the large language model, the generalized classification result comprising a coarse-grained domain class and a fine-grained domain class of each data in the target domain data. The method has better robustness and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a generalized classification method and device based on a large language model for domain awareness. Background Technology

[0002] Generalized Category Discovery aims to automatically identify the category affiliation of all samples, including both known and new categories, using limited labeled data and a large amount of unlabeled data, given that only a portion of the samples are labeled and only some known categories are covered. Because this task is highly relevant to real-world open-world scenarios, it has become an important research direction in computer vision and natural language processing.

[0003] Typical generalized category discovery methods generally include the following steps: (1) Supervised learning based on a limited number of labeled samples to learn preliminary category discrimination features; (2) Self-supervised or contrastive learning combined with all data samples to improve feature representation capabilities; (3) Clustering of sample features using clustering algorithms (such as K-Means) to identify known classes and new classes.

[0004] Existing research mainly focuses on improving the feature learning and clustering stages to enhance the performance of new category recognition. However, due to the lack of prior knowledge about unknown categories, existing methods generally exhibit overfitting to known categories and insufficient ability to recognize new categories, thus limiting the robustness and generalization ability of the model in open-world environments.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The present invention is proposed in view of the above-mentioned problems. According to one aspect of the present invention, a generalized classification method based on a large language model for domain awareness is provided, comprising: Obtain the original dataset for a specified scenario. The original dataset includes labeled data and unlabeled data, and each data point in the labeled data has a fine-grained category label. The coarse-grained domain category of each data point in the original dataset is determined using a large language model and labeled accordingly to obtain target domain data, which includes coarse-grained domain data and fine-grained domain data. A dual-head structure representation model is constructed, and the dual-head structure representation model is trained twice consecutively using the target domain data; The target domain data is input into the trained dual-head structure representation model, and the generalized classification result is determined based on the output result using the large language model. The generalized classification result includes the coarse-grained domain category and the fine-grained domain category of each data point in the target domain data.

[0007] For example, determining the coarse-grained domain category of each data point in the original dataset using a large language model includes: Using the large language model, a coarse-grained domain category system corresponding to the specified scenario is determined, wherein the coarse-grained domain category system includes multiple coarse-grained domain categories; Zero-sample classification is performed based on the specified scenario and the coarse-grained domain category system to obtain the coarse-grained domain category of each data point in the original dataset.

[0008] For example, the large language model determines the coarse-grained domain category system corresponding to the specified scenario in the following way: Generate a query statement based on the specified scenario and the labeled data in the original dataset; The query statement is executed using a search engine to obtain multiple business data streams; The multi-channel service data is sorted from largest to smallest using a reciprocal ranking fusion algorithm to obtain the sorting result; Based on the specified scenario, the sorting results, and the original dataset, a coarse-grained domain category system corresponding to the specified scenario is generated.

[0009] For example, the dual-head structure representation model includes a coarse-grained neighborhood linear classifier, a fine-grained category linear classifier, and a plurality of Transformer encoders connected in sequence. Each Transformer encoder, except for the first and last Transformer encoders, is connected to the coarse-grained neighborhood linear classifier, and the last Transformer encoder is connected to the fine-grained category linear classifier.

[0010] For example, the step of training the dual-head structure representation model twice consecutively using the target domain data includes: The dual-head structure representation model is pre-trained using the target domain data; The pre-trained dual-head structure representation model is trained in a second-order manner using the target domain data.

[0011] For example, The pre-training of the dual-head structure representation model using the target domain data includes: The fine-grained domain data is input into the dual-head structure representation model; A first-order domain classification cross-entropy loss is constructed based on the coarse-grained domain classification results output by the dual-head structure representation model. A first-order fine-grained category cross-entropy loss is constructed based on the fine-grained domain classification results output by the dual-head structure representation model. The masked language modeling loss is constructed based on the target domain data to construct the dual-head structure representation model. A first objective function is constructed based on the masked language modeling loss, the first-order fine-grained category cross-entropy loss, and the first-order domain classification cross-entropy loss; The dual-head structure representation model is trained and optimized with the goal of minimizing the first objective function. And / or, The step of performing second-order training on the pre-trained dual-head structure representation model using the target domain data includes: The target domain data is input into the pre-trained dual-head structure representation model; Construct a neighborhood contrastive learning loss based on the target domain data; A second-order domain classification cross-entropy loss is constructed based on the coarse-grained domain classification results output by the dual-head structure representation model. A second-order fine-grained category cross-entropy loss is constructed based on the fine-grained domain classification results output by the dual-head structure representation model. A second objective function is constructed based on the neighborhood contrast learning loss, the second-order domain classification cross-entropy loss, and the second-order fine-grained category cross-entropy loss; The dual-head structure representation model is trained and optimized with the goal of minimizing the second objective function.

[0012] For example, the neighborhood contrastive learning loss is constructed through the following steps: The pre-trained dual-head structure representation model is used to generate high-dimensional semantic feature vectors for the target domain data. K-Means is used to cluster the high-dimensional semantic feature vectors of the target domain data to obtain several clusters; Set the cluster center for each cluster, and convert the Euclidean distance between the target domain data and each cluster center into a soft assignment probability using the Student-t distribution; Calculate the probability distribution entropy of the target domain data based on the soft assignment probability; Select a first preset number of data points with the highest probability distribution entropy from the target domain data as difficult sample data; From the difficult sample data, select a second preset number of data points that are closest to the difficult sample data in Euclidean distance as candidate sample data; The large language model is used to select samples of the same category from the candidate sample data as positive sample data, and the data in the candidate sample data other than the positive samples are negative sample data. Based on the positive sample data and the negative sample data, the neighborhood contrast loss is calculated.

[0013] For example, the step of inputting the target domain data into the trained dual-head structured representation model and determining the generalized classification result based on the output includes: High-dimensional semantic feature vectors are extracted from the target domain data using the trained dual-head structure representation model. The high-dimensional semantic feature vectors are clustered using the K-Means clustering algorithm to obtain several target clusters; Predetermine the target cluster center for each target cluster, and calculate the Euclidean distance between the data in the target cluster and the target cluster center; Based on the target cluster Euclidean distance, select the data with the closest Euclidean distance among the target clusters as the representative data of the target clusters; Based on representative data of the target cluster, the semantic name and semantic description of the target cluster are generated using the large language model to obtain a generalized classification result.

[0014] According to another aspect of the present invention, an electronic device is provided, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method as described above.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program / instructions that, when executed by a processor, implement the method described above.

[0016] In the above technical solution, the powerful semantic understanding capability of the large language model is used to complete the automatic labeling of coarse-grained domain categories of data. The labeled data with fine-grained category labels and unlabeled data are integrated to construct multi-dimensional target domain data. Then, the dual-head structure representation model is used to carry out hierarchical progressive training, which can improve the feature representation capability and generalization adaptability of the model. Finally, by combining the large language model to determine the generalized classification result, the scene adaptability, classification comprehensiveness and result accuracy of the generalized classification task are greatly improved. This can significantly improve the ability to identify new categories, robustness and generalization ability, and enhance the interpretability and usability of the results.

[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0018] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0019] Figure 1 A schematic flowchart illustrating a generalized classification method according to an embodiment of the present invention is shown; Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0021] According to one aspect of the present invention, a generalized classification method based on a large language model for domain awareness is provided. Figure 1 A schematic flowchart illustrating a generalized classification method according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include the following steps S110, S120, S130 and S140.

[0022] In step S110, the original dataset of the specified scenario is obtained. The original dataset includes labeled data and unlabeled data. Each data in the labeled data has a fine-grained category label.

[0023] The specified scenario can be any business scenario. For example, it could be a banking business scenario, a government service hotline scenario, or a hospital service consultation scenario. For ease of description, the following text will use the banking business scenario as an example for detailed explanation. In the banking business scenario, the original dataset is a fine-grained intent recognition dataset for banking business scenarios. The dataset contains 13,083 user queries, of which 3,228 data entries have fine-grained category labels. The labeled data includes 19 fine-grained categories such as bank card activation, credit card loss reporting, and exchange rate and handling fee inquiries. The remaining data is unlabeled.

[0024] In step S120, the coarse-grained domain category of each data point in the original dataset is determined using a large language model and labeled accordingly to obtain the target domain data, which includes coarse-grained domain data and fine-grained domain data.

[0025] Coarse-grained categories are broad, generalized categories with significant individual differences; they represent macro-level, coarse classification. Fine-grained categories are narrower categories with smaller individual differences; they represent micro-level, fine classification. For example, in banking scenarios, coarse-grained categories might include "accounts," "transfer payments," "security," and "foreign exchange." Fine-grained categories might include "bank card loss reporting," "bank card activation," and "foreign exchange rate consultation."

[0026] In this example, coarse-grained domain data consists of unlabeled data and its corresponding coarse-grained domain categories, while fine-grained domain data consists of labeled data and its corresponding coarse-grained domain categories. In other words, any coarse-grained domain data consists of a coarse-grained domain label and data text. Similarly, any fine-grained domain data consists of a coarse-grained domain label, a fine-grained category label, and data text. The fine-grained category label represents the fine-grained domain category of the corresponding data, and the coarse-grained domain label represents the coarse-grained domain category of the corresponding data.

[0027] In this example, the large language model used can be any existing large language model, such as OpenAIGPT, Google Gemini, Tongyi Qianwen, DeepSeek, etc. This article does not impose any restrictions on it.

[0028] In step S130, a dual-head structure representation model is constructed, and the dual-head structure representation model is trained twice consecutively using target domain data.

[0029] In this example, a dual-headed structural representation model can be constructed first. This model has two output heads, one for outputting coarse-grained domain categories and the other for outputting fine-grained domain categories. In this example, the dual-headed structural representation model is trained twice consecutively using target domain data. That is, it is first trained (pre-trained) using target domain data, and then trained a second time using target domain data. This achieves a hierarchical improvement and precise adaptation of the model's representation capabilities, strengthening the model's ability to deeply capture and structurally represent multi-dimensional features of the target domain.

[0030] In step S140, the target domain data is input into the trained dual-head structure representation model, and the generalized classification result is determined based on the output result using the large language model. The generalized classification result includes the coarse-grained domain category and the fine-grained domain category of each data point in the target domain data.

[0031] The dual-headed structure representation model can output a vector representation of each target domain data (i.e., the output result). Then, a large language model can be used to determine the final generalized classification result based on these vector representations.

[0032] Research has revealed that while it's difficult to exhaustively list all fine-grained categories in real-world business scenarios, the data typically adheres strictly to a defined business context. For example, in government service hotlines, even if it's impossible to fully enumerate all user requests, they remain closely related to government functions such as "education." Therefore, this solution considers leveraging the semantic understanding capabilities of a large language model to enhance new category discovery, scenario adaptability, and generalization. Specifically, the technical solution utilizes the powerful semantic understanding capabilities of a large language model to automatically label coarse-grained domain categories. It then integrates labeled and unlabeled data with fine-grained category labels to construct multi-dimensional target domain data. A hierarchical, progressive training process using a dual-headed representation model further enhances the model's feature representation capabilities and generalization adaptability. Finally, by combining the large language model to determine generalized classification results, the solution significantly improves the scenario adaptability, classification comprehensiveness, and result accuracy of the generalized classification task. This results in a marked improvement in new category identification, robustness, and generalization ability, enhancing the interpretability and usability of the results.

[0033] For example, step S120, which uses a large language model to determine the coarse-grained domain category of each data point in the original dataset, may specifically include the following steps S121 and S122.

[0034] In step S121, the large language model is used to determine the coarse-grained domain category system corresponding to the specified scenario. The coarse-grained domain category system includes multiple coarse-grained domain categories.

[0035] In step S122, zero-sample classification is performed based on the specified scenario and coarse-grained domain category system to obtain the coarse-grained domain category of each data point in the original dataset.

[0036] In this step, a large language model can first be used to generate the various coarse-grained domain categories involved in the specified scene where the original dataset is located. After determining the various coarse-grained domain categories involved in the specified scene, the large language model can be used to perform zero-shot classification on each piece of raw data in the original dataset based on the specified scene and the various coarse-grained domain categories in the coarse-grained domain category system. For example, the coarse-grained domain category system, the background knowledge of the specified scene, and the original dataset can be used to fill the prompt word template corresponding to the specified scene (for ease of distinction, this prompt word template can be called the zero-shot classification prompt word template) to obtain prompt words. Then, the prompt words are input into the large language model, and the coarse-grained domain category of each piece of data in the original dataset is obtained based on the results returned by the large language model.

[0037] In some embodiments, the zero-sample classification prompt word template for banking business scenarios is as follows: "You are a natural language processing expert proficient in text classification. You are required to perform zero-shot classification of the following online banking customer queries."

[0038] Background knowledge in the banking sector: {Retrieval and recall of documents using multi-way fusion ranking} The optional {M} coarse-grained domain labels and their brief descriptions are as follows: {M domain tags and descriptions} Requirements: For each input query, select the best-matching domain and provide a confidence score of 0-1. Return the results in the following format: { "domain": "domain name", "confidence": "Domain confidence level" } Please now categorize the following text and return only the categorization results: {Bank Customer Inquiry}.

[0039] The multi-path fusion ranking retrieval documents in the above template can be obtained by querying a specified scenario using a large language model. The M domain labels represent multiple coarse-grained domain categories in the coarse-grained domain category system. The bank customer query section is used to fill in the original dataset; each input corresponds to a data point in the original dataset and can be considered a query statement. Specifically, assuming the large language model returns the following result for the query "Transfer always prompts an excess amount, how to cancel the limit?": { "domain":"Transfer Payment", "confidence": 0.9}.

[0040] After obtaining the coarse-grained domain category for each data point in the original dataset, the following operations can be performed: coarse-grained domain labeling is applied to each data point in the unlabeled data to obtain coarse-grained domain data; coarse-grained domain labeling is applied to each data point in the labeled data to obtain fine-grained domain data. Thus, the target domain data can be obtained.

[0041] The aforementioned technical solution utilizes a large language model to perform zero-shot domain category labeling on all data, enabling the automatic introduction of domain labels into large-scale data without the need for manual annotation. This significantly reduces the cost of manual annotation and improves data processing efficiency. Furthermore, the generated domain category labels can serve as prior domain information, providing unified coarse-grained semantic constraints for subsequent representation learning.

[0042] For example, the large language model determines the coarse-grained domain category system corresponding to a specified scenario in the following way: generating a query statement based on the specified scenario and the labeled data in the original dataset; executing the query statement using a search engine to obtain multiple business data; sorting the multiple business data from largest to smallest using the Reciprocal Rank Fusion (RRF) algorithm to obtain the sorting result; and generating the coarse-grained domain category system corresponding to the specified scenario based on the specified scenario, the sorting result, and the original dataset.

[0043] In this example, the solution first generates a large language model based on the background information of the specified scene and some labeled data (i.e., the labeled data in the original dataset). q Different search queries. For example, in a fine-grained intent recognition task targeting banking scenarios, a large language model generates three query statements: "What services does a bank offer?", "What functions does online banking customer service offer?", and "What aspects do online banking customers need?". These queries trigger a search engine to return high-quality knowledge content related to the relevant business. The large language model could be a platform like Wenxin Yiyan, Tongyi Qianwen, or DeepSeek.

[0044] For each generated query, a search engine API can be called sequentially to retrieve relevant search results. The search engine can be Google Search API, Bing Search API, etc. For example, by calling the Google Search API to search for "What are the functions of online bank customer service?", ten web pages (i.e., ten documents) such as "Best Practices for Bank Customer Service" are obtained, sorted in descending order of relevance. This data is the multi-path fusion and sorting retrieval document filled in the template above, i.e., multi-path business data.

[0045] After obtaining multi-channel service data, the RRF algorithm can be used to merge and sort the multi-channel service data. In a specific embodiment, the score of the multi-channel service data can be calculated first, and the specific calculation formula is as follows: ; in, Scoring based on multi-channel business data; For document In the The ranking of business data obtained from a query statement, that is, the ranking of several documents obtained by executing the query statement in descending order of relevance; The offset is set to 60 in this embodiment; For the query statement index; This represents the number of query statements.

[0046] Then, the scores of the multiple business data can be sorted from largest to smallest to obtain the sorting result.

[0047] After obtaining the ranking results, the large language model can extract the domain categories (i.e., a coarse-grained domain category system) for the specified scenario based on the business background information, ranking results, and original dataset. For example, the business background information, ranking results, and original dataset of the specified scenario can be input into a prompt word template (which can be called a category-generated prompt word template) to obtain prompt words. Then, these prompt words can be input into the large language model, and the coarse-grained domain category system can be obtained based on the results returned by the large language model.

[0048] In some embodiments, the category-generated prompt word template is as follows: "You are a banking expert and need to categorize the following limited online banking customer service intentions into {Z} domain categories based on semantic similarity and business function."

[0049] Background knowledge in the banking sector: {Retrieval and recall of documents using multi-way fusion ranking} Given a list of intents: {known limited category information} Requirements: Each domain category should represent a high-level, coarse-grained category of banking services; domain names should be clear, concise, and descriptive; all intents must be categorized into exactly one domain; the number of intents across domains should be as balanced as possible to avoid over-concentration in any one domain. Return results in the following format: { "Domain Category": [ { "domain_name": "domain name", "description": "A brief description of the services covered in this area" ], "reasoning": "A brief explanation of the domain partitioning strategy" }".

[0050] Specifically, this embodiment summarizes six categories related to banking business: "accounts," "transfer payments," "security," and "foreign exchange."

[0051] The aforementioned technical solution searches for business knowledge within a specified scenario and automatically summarizes and abstracts the retrieved multi-path business knowledge using a large language model, generating a coarse-grained domain category system. This system fully integrates the inherent knowledge of the large language model with the rich and dynamically updated business information available on the internet, thereby filtering and focusing on business background knowledge highly relevant to the specified scenario and task objectives. This helps improve the accuracy and robustness of the coarse-grained domain categories determined in subsequent steps. Furthermore, the generated coarse-grained domain categories can be used as domain priors in the subsequent generalized category discovery process, effectively constraining the search space and providing semantic guidance.

[0052] For example, the dual-head structure representation model includes a coarse-grained neighborhood linear classifier, a fine-grained category linear classifier, and multiple Transformer encoders connected in sequence. The coarse-grained neighborhood linear classifier is connected to any Transformer encoder other than the first and last Transformer encoders, and the last Transformer encoder is connected to the fine-grained category linear classifier.

[0053] The dual-head structure representation model can be constructed using the BERT-base-chinese model. This model includes multiple sequentially connected Transformer encoders. The hidden states of different layers have different representational characteristics. The intermediate layer representations typically retain strong general semantic information and category commonality, making them suitable for coarse-grained category discrimination; while the higher-level representations, after more thorough contextual interaction and task-related feature extraction, have stronger discriminative capabilities and are therefore more suitable for fine-grained category classification. Based on this, this paper selects the outputs of the intermediate and tail layers as feature representations for coarse-grained and fine-grained classification, respectively, to better match the semantic modeling requirements of different levels of classification tasks. Specifically, in this example scheme, any Transformer encoder other than the first and tail encoders (i.e., the encoder of the intermediate layer, for example, the fourth encoder) is selected and connected to a coarse-grained domain linear classifier for coarse-grained domain classification; the last Transformer encoder in the BERT-base-chinese model is connected to a fine-grained category linear classifier for fine-grained category classification. In this case, the vectors from the intermediate Transformer encoders are input into the coarse-grained linear neighborhood classifier after a nonlinear transformation, and the vectors from the last Transformer encoder are input into the fine-grained category linear classifier after a nonlinear transformation.

[0054] The above technical solution achieves progressive deep feature extraction and semantic enhancement through a multi-level Transformer encoder. By using a layered dual-head design that connects a coarse-grained domain linear classifier after the intermediate Transformer encoder and a fine-grained category linear classifier at the end of the Transformer encoder, it can simultaneously complete the coarse-grained domain division and the fine-grained category accurate recognition. It can effectively take into account the dual mining of global domain features and local category features, and accurately perform coarse-grained classification and fine-grained classification.

[0055] For example, step S130, which involves training the dual-head structure representation model twice consecutively using target domain data, may include steps S131 and S132.

[0056] In step S131, the dual-head structure representation model is pre-trained using target domain data.

[0057] In step S132, the pre-trained dual-head structure representation model is trained in a second order using target domain data.

[0058] The above technical solution uses target domain data to complete model pre-training and second-order training in sequence, which enables the model to fully learn the data distribution characteristics, structural correlation rules and deep semantic information of the target domain. This effectively enhances the dual-head structure representation model's ability to represent the content of the target domain and the accuracy of feature extraction, reduces domain adaptation bias, and improves the model's generalization performance, fitting effect and practical application adaptability in the corresponding task, making the model's output representation results more in line with the business needs of the target domain.

[0059] For example, step S131, pre-training the dual-headed structure representation model using target domain data, includes: inputting fine-grained domain data into the dual-headed structure representation model; constructing a first-order domain classification cross-entropy loss based on the coarse-grained domain classification results output by the dual-headed structure representation model; constructing a first-order fine-grained category cross-entropy loss based on the fine-grained domain classification results output by the dual-headed structure representation model; constructing a masked language modeling loss for the dual-headed structure representation model based on the target domain data; constructing a first objective function based on the masked language modeling loss, the first-order fine-grained category cross-entropy loss, and the first-order domain classification cross-entropy loss; and training and optimizing the dual-headed structure representation model with minimizing the first objective function as the optimization objective.

[0060] In this paper, for ease of distinction, the fine-grained category cross-entropy loss and the domain classification cross-entropy loss obtained in the pre-training stage are referred to as the first-order fine-grained category cross-entropy loss and the first-order domain classification cross-entropy loss, respectively. The fine-grained category cross-entropy loss and the domain classification cross-entropy loss obtained in the second-order training stage are referred to as the second-order fine-grained category cross-entropy loss and the second-order domain classification cross-entropy loss, respectively.

[0061] Fine-grained category cross-entropy loss is used for supervised learning with known categories. In some embodiments, the fine-grained category cross-entropy loss is calculated as follows: ; in, Fine-grained category cross-entropy loss; The number of labeled data (i.e., the total number of fine-grained domain data). For the first A sequence of input lexical terms; ; For the first The true fine-grained category labels for each input word sequence; Input for fine-grained linear classifier Belongs to the category of true fine-grained tags The probability of.

[0062] The domain classification cross-entropy loss is used to introduce a domain supervision signal. In some embodiments, the first-order domain classification cross-entropy loss is calculated as follows: ; in, The first-order domain classification cross-entropy loss is used. The coarse-grained domain labels obtained in the zero-sample classification stage; Input for coarse-grained domain classifier This belongs to the coarse-grained label category. The probability of.

[0063] Masked language modeling loss is used to improve language representation capabilities. In some embodiments, the dual-headed representation model further includes a fully connected layer and a Softmax layer connected in sequence, with the input of the fully connected layer connected to the last Transformer encoder. For the mask position, the high-dimensional semantic feature vector output by the last Transformer encoder enters the fully connected layer for linear transformation, and then enters the Softmax layer. The Softmax layer outputs the probability that the mask position is predicted to be a different word.

[0064] The masked language modeling loss is constructed based on the target domain data to build a dual-headed structure representation model, including: randomly masking the target domain data; inputting the masked target domain data into the dual-headed structure representation model and obtaining the probability that the prediction result at each mask position of the dual-headed structure representation model is a real word; and calculating the masked language modeling loss using the probability that the prediction result at each mask position is a real word.

[0065] In this embodiment, we consider using a dual-head structure to represent the hidden state vector of the masked position output by the Transformer encoder in the model. This vector is linearly transformed through a fully connected layer (typically sharing weights with the input word embedding layer) to map its dimension back to the size of the vocabulary. This transformed vector is called logits. Assuming the vocabulary size is V, then logits is a vector of length V. The Softmax function is applied to the logits vector obtained in the previous step. Softmax transforms the scores in logits into a probability distribution such that the probability values ​​of all tokens are between (0, 1) and sum to 1. Thus, the model provides the probability that the masked position is every word in the vocabulary. Therefore, the probability of the masked position being predicted as a real word can be obtained. Next, the masked language modeling loss can be determined based on the average of the cross-entropy losses of all masked positions. In practical scenarios, the probability of a masked position being predicted as a real word can be used as the label for the masked position. For non-masked positions, the label can be set to a special value (e.g., -100), so that these positions are automatically ignored when calculating the loss.

[0066] In a specific embodiment, the formula for calculating the mask language modeling loss is as follows: ; in, Modeling loss for masked language; This represents the total number of samples in the target domain, which is determined by the number of labeled data samples. Number of unlabeled data samples Together constitute; For the first The word sequence obtained by randomly replacing the word elements in an input word sequence; This is the set of positions of randomly replaced words in the sequence; For word position; For the first A real word element that is randomly replaced; For the model at position Predicting real words The probability of.

[0067] In some embodiments, the sum of the above three losses can be used as the first objective function, which can be expressed as: ; in, For pre-training objectives; The first loss weighting coefficient, ; This is the second loss weighting coefficient. In this embodiment, the following settings are provided. .

[0068] In a specific embodiment, the target domain data can first be segmented using the BERT-base-chinese model's word segmenter, converting the input data into a word sequence. A special marker [CLS] is added to the beginning of the sequence, and a special marker [SEP] is added to the end to form the model's standard input format. For example, "transfer failed" is converted into the word sequence: "[['[CLS]','transfer','account','loss','failure','[SEP]']]".

[0069] Next, the word sequence corresponding to the fine-grained domain data is input into the dual-head structure representation model, and the corresponding first-order domain classification cross-entropy loss and first-order fine-grained category cross-entropy loss are constructed based on the output results of the coarse-grained domain linear classifier and the fine-grained category linear classifier, respectively.

[0070] Simultaneously, a masked language modeling task is introduced during the pre-training phase to enhance the model's language representation capabilities. Specifically, in all word sequences corresponding to the target domain data, words without special or padding markers are randomly selected as mask targets with a 15% probability. For the selected words, replacements are performed according to the following rules: 1) Replaced with the special marker [MASK] with an 80% probability; 2) Replaced with a random word with a 10% probability; 3) The original word remains unchanged with a 10% probability. The goal of the masked language modeling task is to predict the original word corresponding to the masked position, thereby calculating the masked language modeling loss. In practice, the original fine-grained domain data can be input into the model first to calculate the first-order domain classification cross-entropy loss and the first-order fine-grained category cross-entropy loss. Then, all target domain data is randomly masked, the masked target domain data is input into the model, and the masked language modeling loss is calculated using the model's output. (Details omitted.)

[0071] Finally, the masked language modeling loss, fine-grained category cross-entropy loss, and domain classification cross-entropy loss are jointly optimized until they meet the pre-training objective.

[0072] The aforementioned technical solution constructs a dual-headed representation model and jointly introduces masked language modeling loss, domain classification cross-entropy loss, and fine-grained category cross-entropy loss during the pre-training phase. This allows the model to simultaneously learn general language semantic features, domain-discriminating features, and category-discriminating features during parameter initialization, thereby significantly improving the semantic richness and discriminative ability of feature representations under limited labeled data conditions. Furthermore, this multi-task pre-training approach effectively alleviates the feature representation bias problem caused by relying solely on a single supervisory signal, providing a more stable and separable semantic representation foundation for subsequent two-stage training and cluster discovery.

[0073] For example, before pre-training, the following steps may be included: adding a coarse-grained domain category system to the vocabulary of the dual-headed structure representation model, thereby expanding the vocabulary of the dual-headed structure representation model; and concatenating the coarse-grained domain category annotations to the fine-grained domain data, thereby enhancing the fine-grained domain data. In this way, the model explicitly receives domain cues during the encoding stage, thereby improving the accuracy of domain discrimination.

[0074] For example, step S132, which involves performing second-order training on the pre-trained dual-head structure representation model using target domain data, includes: inputting target domain data into the pre-trained dual-head structure representation model; constructing a neighborhood contrastive learning loss based on the target domain data; constructing a second-order domain classification cross-entropy loss based on the coarse-grained domain classification results output by the dual-head structure representation model; constructing a second-order fine-grained category cross-entropy loss based on the fine-grained domain classification results output by the dual-head structure representation model; constructing a second objective function based on the neighborhood contrastive learning loss, the second-order domain classification cross-entropy loss, and the second-order fine-grained category cross-entropy loss; and training and optimizing the dual-head structure representation model with minimizing the second objective function as the optimization objective.

[0075] The second-order fine-grained category cross-entropy loss is calculated using the same formula as the fine-grained category cross-entropy loss in pre-training, so it will not be repeated here.

[0076] In some embodiments, the second-order neighborhood classification cross-entropy loss is: ; The difference between this formula and pre-training is that it uses all the data. N Perform cross-entropy loss calculation for domain classification. The total number of samples in the target domain data is represented by... and Same composition; among which This represents the amount of coarse-grained domain data.

[0077] In some embodiments, the second objective function can be expressed as: ,in, For second-order training objectives; Neighborhood contrast loss; The loss is a second-order fine-grained category cross-entropy loss. The second-order neighborhood classification cross-entropy loss is used. The first task's weight hyperparameter; This is the weight hyperparameter for the second task.

[0078] The above technical solution, by jointly introducing neighborhood contrastive learning loss, domain classification cross-entropy loss, and fine-grained category cross-entropy loss under a unified optimization framework, enables the model to fully utilize unlabeled data while maintaining effective constraints on the domain structure and known category boundaries. The domain classification loss and fine-grained classification loss impose supervisory constraints on the model representation from two semantic levels, coarse-grained and fine-grained, respectively, which can effectively improve the discriminativeness and clustering friendliness of feature representation.

[0079] For example, the neighborhood contrastive learning loss is obtained through the following steps S1321, S1322, S1323, S1324, S1325, S1326, S1327 and S1328.

[0080] In step S1321, a high-dimensional semantic feature vector of the target domain data is generated using the pre-trained dual-head structure representation model.

[0081] In step S1322, K-Means clustering algorithm is used to cluster the high-dimensional semantic feature vectors of the target domain data to obtain several clusters.

[0082] In step S1323, the cluster center of each cluster is set, and the Euclidean distance between the target domain data and each cluster center is converted into a soft assignment probability using the Student-t distribution.

[0083] In step S1324, the probability distribution entropy of the target domain data is calculated based on the soft assignment probability.

[0084] In step S1325, a first preset number of data points with the highest probability distribution entropy are selected from the target domain data as difficult sample data.

[0085] In step S1326, a second preset number of data points that are closest to the Euclidean distance of the difficult sample data are selected from the difficult sample data as candidate sample data.

[0086] In step S1327, the large language model is used to select samples of the same category from the candidate sample data as positive sample data, and the data other than positive samples in the candidate sample data are negative sample data.

[0087] In step S1328, the neighborhood contrast loss is calculated based on the positive sample data and the negative sample data.

[0088] Several clusters can be represented as: ,in It is a set of clusters; This is the first cluster; This is the second cluster; For the first There are 10 clusters. The cluster center of each cluster is... This is the average vector within the cluster.

[0089] The Student-t distribution is used to measure the distance between target domain data and cluster centers. In this example scheme, the formula for the Student-t distribution is: ; in, For the first The data in the [number] domain belongs to [number]. The probability of a cluster; This represents the number of clusters. Index for the number of clusters; For the first High-dimensional semantic representation of data in various domains; For the first Cluster centers; For degrees of freedom.

[0090] In some embodiments, the probability distribution entropy of the target domain data The calculation formula is: .

[0091] After calculating the probability distribution entropy, a first preset number M data points with the highest probability distribution entropy can be selected as hard samples. In this embodiment, hard samples are those with large probability distribution entropy values. The higher the entropy value, the smaller the difference in the model's discrimination between different categories, and the more difficult the sample is to distinguish, thus it is identified as a hard sample. Subsequently, a second preset number T data points with the closest Euclidean distance to the hard sample data can be selected as candidate sample data from the hard sample data.

[0092] Next, a large language model can be used to select positive sample data. That is, the large language model selects samples from the candidate sample data that are semantically closest to the difficult samples and most likely to belong to the same category as the positive samples. In some embodiments, the template for the large language model to select samples of the same category as positive sample data from the candidate sample data is as follows: "Please select the text that is most similar to the user's query semantics from the given candidate texts and return the sequence number of that candidate text."

[0093] User query: {Difficult sample data} Candidate text: {candidate sample data}.

[0094] Finally, the neighborhood contrast loss can be calculated based on the positive and negative sample data. In some embodiments, the neighborhood contrast loss is calculated using the following formula: ; in, Neighborhood contrast loss; This represents the number of positive sample data. Index for the number of positive sample data; For the first High-dimensional semantic representation of difficult sample data; A high-dimensional semantic representation of the positive sample data selected by the large language model; A high-dimensional semantic representation of negative sample data; The cosine similarity function; This is a temperature coefficient used to control the sharpness of the distribution; Index for the number of negative sample data.

[0095] In the above scheme, the neighborhood contrast learning mechanism guided by the large language model provides weak supervision signals for difficult samples with high uncertainty in fine-grained classification. This can encourage difficult samples to cluster towards semantically similar samples in the feature space and move away from semantically irrelevant samples, thereby effectively strengthening intra-class compactness and expanding inter-class spacing. This helps to apply more accurate, comprehensive and semantically consistent supervision constraints at the coarse-grained level.

[0096] For example, step S140, which involves inputting the target domain data into the trained dual-head structure representation model and determining the generalized classification result based on the output, may specifically include the following steps S141, S142, S143, S144, and S145.

[0097] In step S141, a high-dimensional semantic feature vector is extracted from the target domain data using the trained dual-head structure representation model. This high-dimensional semantic feature vector is the output of the last Transformer encoder of the dual-head structure representation model. This step is implemented in the same way as step S1321.

[0098] In step S142, the K-Means clustering algorithm is used to cluster the high-dimensional semantic feature vectors, resulting in several target clusters. This step is implemented in the same way as step S1322. Each cluster can be represented as: This clustering result can automatically fuse known fine-grained category data with similar unlabeled data, as well as discover and classify new category clusters.

[0099] In step S143, a target cluster center is preset for each target cluster, and the Euclidean distance between the data in the target cluster and the target cluster center is calculated. The target cluster center is the cluster center. Calculate the Euclidean distance between the data in the target cluster and the center of the target cluster. .

[0100] In step S144, based on the Euclidean distance of the target cluster, a third preset number of data points with the closest Euclidean distance to the target cluster are selected as representative data for the target cluster. The third target number can be represented by R, and R can be set according to the actual situation, for example, R=10.

[0101] In step S145, based on the representative data of the target cluster, a large language model is used to generate the semantic name and semantic description of the target cluster to obtain the generalized classification result.

[0102] In this example, representative data is input into a generalized classification prompt word template to obtain prompt words. These prompt words are then input into a large language model to obtain the generalized classification results. The generalized classification results include the category name and semantic description for each cluster. The generalized category suggestion template can be: "You are a banking expert. You need to summarize the following online banking customer queries into a category and generate a name and semantic description for that category."

[0103] Online bank customer inquiry: {Representative data} Reference category: {known fine-grained category name}.

[0104] Specifically, assuming that the large language model can summarize fine-grained categories of "lost or stolen bank cards" based on representative data such as "What should I do if my bank card is lost" and "My wallet containing my bank card was stolen".

[0105] In this embodiment, 77 fine-grained categories were ultimately obtained, including "bank card loss reporting," "bank card activation," and "foreign exchange rate consultation," each corresponding to one of 77 clusters. The fine-grained category of the data in each cluster is actually the fine-grained category corresponding to that cluster. Therefore, this generalized classification result can obtain the fine-grained category of all data.

[0106] The above technical solution uses the trained representation model to perform clustering modeling on all data, uses a large language model to merge and optimize the categories, and generates semantically fine-grained category names and descriptions for each optimized category. This generates semantically informative and interpretable category names and descriptions for the clustering results, thereby obtaining domain-aware generalized classifications, which significantly improves the understandability and usability of the generalized category discovery results.

[0107] According to another aspect of the present invention, an electronic device is also provided. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 2 As shown, the electronic device 200 includes a processor 210 and a memory 220. The memory 220 stores a computer program, which the processor 210 executes to implement the method described above.

[0108] According to another aspect of the present invention, a computer-readable storage medium is also provided. The storage medium stores a computer program / instructions that, when executed by a processor, implement the method described above. The storage medium may, for example, include a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0109] Those skilled in the art will readily understand the implementation structure, working principle, and beneficial effects of electronic devices and computer-readable storage media by reading the above methods. For the sake of brevity, further details will not be elaborated here.

[0110] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0112] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0113] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0114] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof.

[0115] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0116] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0117] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the electronic device according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0118] The above description is merely a specific embodiment of the present invention or an explanation of the specific embodiment. The scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A generalized classification method based on a large language model for domain awareness, characterized in that, include: Obtain the original dataset for a specified scenario. The original dataset includes labeled data and unlabeled data, and each data point in the labeled data has a fine-grained category label. The coarse-grained domain category of each data point in the original dataset is determined using a large language model and labeled accordingly to obtain target domain data, which includes coarse-grained domain data and fine-grained domain data. A dual-head structure representation model is constructed, and the dual-head structure representation model is trained twice consecutively using the target domain data; The target domain data is input into the trained dual-head structure representation model, and the generalized classification result is determined based on the output result using the large language model. The generalized classification result includes the coarse-grained domain category and the fine-grained domain category of each data point in the target domain data. The step of inputting the target domain data into the trained dual-head structured representation model and determining the generalized classification result based on the output includes: High-dimensional semantic feature vectors are extracted from the target domain data using the trained dual-head structure representation model. The high-dimensional semantic feature vectors are clustered using the K-Means clustering algorithm to obtain several target clusters; Predetermine the target cluster center for each target cluster, and calculate the Euclidean distance between the data in the target cluster and the target cluster center; Based on the target cluster Euclidean distance, select the data with the closest Euclidean distance among the target clusters as the representative data of the target clusters; Based on representative data of the target cluster, the semantic name and semantic description of the target cluster are generated using the large language model to obtain a generalized classification result. The process of determining the coarse-grained domain category of each data point in the original dataset using a large language model includes: Using the large language model, a coarse-grained domain category system corresponding to the specified scenario is determined, wherein the coarse-grained domain category system includes multiple coarse-grained domain categories; Zero-shot classification is performed based on the specified scenario and the coarse-grained domain category system to obtain the coarse-grained domain category of each data point in the original dataset; The large language model determines the coarse-grained domain category system corresponding to the specified scenario in the following way: Generate a query statement based on the specified scenario and the labeled data in the original dataset; The query statement is executed using a search engine to obtain multiple business data streams; The multi-channel service data is sorted from largest to smallest using a reciprocal ranking fusion algorithm to obtain the sorting result; Based on the specified scenario, the sorting results, and the original dataset, a coarse-grained domain category system corresponding to the specified scenario is generated.

2. The method according to claim 1, characterized in that, The dual-head structure representation model includes a coarse-grained neighborhood linear classifier, a fine-grained category linear classifier, and multiple Transformer encoders connected in sequence. Each Transformer encoder, except for the first and last Transformer encoders, is connected to the coarse-grained neighborhood linear classifier, and the last Transformer encoder is connected to the fine-grained category linear classifier.

3. The method according to claim 1 or 2, characterized in that, The step of training the dual-head structure representation model twice consecutively using the target domain data includes: The dual-head structure representation model is pre-trained using the target domain data; The pre-trained dual-head structure representation model is trained in a second-order manner using the target domain data.

4. The method according to claim 3, characterized in that, The pre-training of the dual-head structure representation model using the target domain data includes: The fine-grained domain data is input into the dual-head structure representation model; A first-order domain classification cross-entropy loss is constructed based on the coarse-grained domain classification results output by the dual-head structure representation model. A first-order fine-grained category cross-entropy loss is constructed based on the fine-grained domain classification results output by the dual-head structure representation model. The masked language modeling loss is constructed based on the target domain data to construct the dual-head structure representation model. A first objective function is constructed based on the masked language modeling loss, the first-order fine-grained category cross-entropy loss, and the first-order domain classification cross-entropy loss; The dual-head structure representation model is trained and optimized with the goal of minimizing the first objective function. And / or, The step of performing second-order training on the pre-trained dual-head structure representation model using the target domain data includes: The target domain data is input into the pre-trained dual-head structure representation model; Construct a neighborhood contrastive learning loss based on the target domain data; A second-order domain classification cross-entropy loss is constructed based on the coarse-grained domain classification results output by the dual-head structure representation model. A second-order fine-grained category cross-entropy loss is constructed based on the fine-grained domain classification results output by the dual-head structure representation model. A second objective function is constructed based on the neighborhood contrast learning loss, the second-order domain classification cross-entropy loss, and the second-order fine-grained category cross-entropy loss; The dual-head structure representation model is trained and optimized with the goal of minimizing the second objective function.

5. The method according to claim 4, characterized in that, The neighborhood contrastive learning loss is constructed through the following steps: The pre-trained dual-head structure representation model is used to generate high-dimensional semantic feature vectors for the target domain data. K-Means is used to cluster the high-dimensional semantic feature vectors of the target domain data to obtain several clusters; Set the cluster center for each cluster, and convert the Euclidean distance between the target domain data and each cluster center into a soft assignment probability using the Student-t distribution; Calculate the probability distribution entropy of the target domain data based on the soft assignment probability; Select a first preset number of data points with the highest probability distribution entropy from the target domain data as difficult sample data; From the difficult sample data, select a second preset number of data points that are closest to the difficult sample data in Euclidean distance as candidate sample data; The large language model is used to select samples of the same category from the candidate sample data as positive sample data, and the data in the candidate sample data other than the positive samples are negative sample data. Based on the positive sample data and the negative sample data, the neighborhood contrast loss is calculated.

6. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The system stores a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-5.

Images