Large model distillation method and system for low-resource short text classification

Abstract

The invention provides a large model distillation method and system for low-resource short text classification. The method comprises the following steps: establishing a semantic syntax enhanced thinking chain and a domain knowledge thinking chain by utilizing the same large model according to a multi-step prompt method; applying a semantic syntactic enhanced thinking chain to extract key concepts in the short text, retrieving common knowledge of the short text through the key concepts, and outputting an improved text fused with semantic and syntactic reasoning knowledge according to the common knowledge; applying the domain knowledge thinking chain, extracting basic concepts in the short text, and generating domain knowledge through the basic concepts; and training the small model through a short text classification task, a semantic and syntactic reasoning task for improving text to be output by the model and a domain knowledge extraction task for improving domain knowledge to be output by the model. According to the scheme provided by the invention, hardware resources can be saved, and more application scenes can be adapted; the classification precision is improved, and reasoning without retraining is supported; the thinking chain design improves reasoning and emerging capabilities.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence-natural language processing, and in particular relates to a large model distillation method and system for low-resource short text classification. Background Art

[0002] Short texts are crucial to contemporary information flows, especially given the rapid development of the Internet. They are indispensable on major social media platforms, including Twitter, TikTok, Instagram, and Weibo, which facilitate social interactions and are an integral part of daily activities. As an important task for intelligent empowerment and short text applications, short text classification is crucial for applications such as news classification, question answering, and sentiment analysis. Traditional pre-trained language models cannot well address the challenges of short text classification itself: semantic sparsity and syntactic defects, and the amount of data is scarce. To address the challenges raised in the short text classification task, graph convolutional networks have shown some progress by using additional knowledge bases and re-defining short text classification as a node classification problem, alleviating the problem of limited training data. However, the effectiveness of graph convolutional networks is limited by the quality and scope of their knowledge.

[0003] Large language models have revolutionized the field of natural language processing. These models perform state-of-the-art downstream tasks such as complex reasoning and question answering. Studies have shown that models with larger parameters effectively act as implicit knowledge bases, provide better knowledge integration and application, and suppress emergent capabilities such as contextual learning and teaching-following, compared to traditional external knowledge bases. However, some studies have observed the limitations of applying large language models to traditional natural language processing tasks. In addition, it remains a challenge to effectively handle datasets for short text tasks, such as Ohsumed and TagMyNews. Therefore, our research aims to leverage the inherent knowledge and emergent capabilities of large models to address this challenge.

[0004] (1) Graph Neural Network Example

[0005] like Figure 2 As shown in the figure, suppose there is a dataset D, which contains n training samples and m test samples. The overall goal is to construct a large graph for n+m samples, where a node represents a sample. In the training phase, the nodes of n training samples are used to calculate the loss and optimize the parameters, and in the test phase, the nodes of m test samples are used to obtain the classification results.

[0006] Basic concepts:

[0007] (1)W represents a word graph. Nodes are words retained after a series of operations such as word segmentation and removal of low-frequency words for all samples in D. They are initially represented as one-hot vectors. Edges are the PMI (point mutual information) values ​​of two nodes (i.e., two words) in D.

[0008] (2) P represents the part-of-speech graph. First, all samples in D are labeled with parts of speech. Then the operation is the same as that of W, except that word is replaced by pos.

[0009] (3) E represents the entity graph. First, we extract the entities contained in all samples in D. Then we find the embeddings (which can be understood as vector representations) of these entities on an existing knowledge graph and calculate them using cosine similarity.

[0010] (4) Short Document Graph refers to the final large graph, that is, it has (n+m) nodes, and this node representation (i.e., vector) is used to perform classification tasks.

[0011] Training phase: For each sample in D (i.e. n+m), first extract the three corresponding subgraphs, i.e. W'P'E'. Then, the vector obtained by concatenating the three subgraph vectors is used as the node representation of the sample in the Short Document Graph. After all the node representations are obtained, two layers of GCN operations are performed, and the final node vector passes through a layer of softmax to obtain the classification result. In this phase, only n training samples are used to calculate the loss and back propagate to adjust the parameters of each layer.

[0012] Testing phase: Follow the above training process, except that the final m node vectors are passed through a softmax layer without calculating the loss.

[0013] (2) Large Language Model

[0014] The LLM is trained with training data, and the model parameters are saved after training. The trained model parameters are used for inference in the test phase, but directly using the LLM in the prior art will have certain defects.

[0015] Disadvantages of the prior art

[0016] (1) Graph Neural Networks

[0017] Its effectiveness is subject to the quality and richness of the external knowledge graph. And from the description in Chapter 3, we can see that the graph-based method needs to be retrained when a new sample comes in, and its practicality is poor.

[0018] (2) Large Language Model

[0019] The large language model contains a rich knowledge base, and the trained model can be directly inferred. However, studies have shown that the richness of its knowledge is proportional to the number of its parameters. Only when the number of parameters is large enough can the short text classification task be solved well, which means that sufficient hardware computing resources are needed to achieve good results. However, in practical applications, many scenarios do not meet the requirements, such as vertical Chinese medicine institutions. Therefore, the present invention designs a distillation method to transfer the knowledge on the large model to the model with a small number of parameters. Summary of the invention

[0020] In order to solve the above technical problems, the present invention proposes a technical solution of a large model distillation method for low-resource short text classification to solve the above technical problems.

[0021] The first aspect of the present invention discloses a large model distillation method for low-resource short text classification, the method comprising:

[0022] Step S1, using the same large model, according to the multi-step prompting method, establish the semantic syntax enhancement thinking chain and the domain knowledge thinking chain;

[0023] Step S2, applying the semantic syntactic enhancement thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge;

[0024] Step S3, applying the domain knowledge thinking chain to extract basic concepts in the short text, and generating domain knowledge through the basic concepts;

[0025] Step S4, training the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

[0026] According to the method of the first aspect of the present invention, in step S2, applying the semantic syntactic enhanced thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge includes:

[0027]

[0028]

[0029]

[0030] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 1 Indicates key concepts; Represents the context for extracting key concepts; represents the key concept extraction instruction; S represents common sense knowledge; represents the context for retrieving common sense knowledge; represents the retrieval of common sense knowledge instructions; R represents the improvement of text; Represents the context of the output improvement text; Indicates output improvement text instructions.

[0031] According to the method of the first aspect of the present invention, in step S3, applying the domain knowledge thinking chain to extract basic concepts in the short text, and generating domain knowledge through the basic concepts includes:

[0032]

[0033]

[0034] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 2 Indicates basic concepts; Represents the context for extracting basic concepts; Indicates the instruction to extract basic concepts; O indicates domain knowledge; Represents the context in which domain knowledge is generated; Represents instructions for generating domain knowledge.

[0035] According to the method of the first aspect of the present invention, in step S4, the short text classification task is:

[0036] The input of the small model is a short text, and the output is a predicted label.

[0037] According to the method of the first aspect of the present invention, in step S4, the semantic and syntactic reasoning tasks of the improved text output by the model are:

[0038] The input of the small model is semantic and syntactic enhanced thinking chain prompts and short texts, and the output is semantic and syntactic reasoning knowledge.

[0039] According to the method of the first aspect of the present invention, in step S4, the domain knowledge is the domain knowledge extraction task output by the model:

[0040] The input of the small model is domain knowledge thinking chain prompts and short texts, and the output is domain knowledge.

[0041] According to the method of the first aspect of the present invention, in step S4, the loss function for training the small model is:

[0042] L=λ1L1+λ2L2+λ3L3

[0043]

[0044]

[0045]

[0046] Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss.

[0047] A second aspect of the present invention discloses a large model distillation system for low-resource short text classification, the system comprising:

[0048] The first processing module is configured to establish a semantic syntactic enhancement thinking chain and a domain knowledge thinking chain based on a multi-step prompting method using the same large model;

[0049] The second processing module is configured to apply the semantic and syntactic enhanced thinking chain to extract key concepts in the short text, retrieve common sense knowledge of the short text through the key concepts, and output an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge;

[0050] The third processing module is configured to apply the domain knowledge thinking chain to extract basic concepts in the short text and generate domain knowledge through the basic concepts;

[0051] The fourth processing module is configured to train the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

[0052] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the large model distillation methods for low-resource short text classification in the first aspect of the present disclosure are implemented.

[0053] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the large model distillation methods for low-resource short text classification in the first aspect of the present disclosure are implemented.

[0054] In summary, the solution proposed by the present invention can

[0055] 1. Save hardware resources and adapt to more application scenarios

[0056] Compared with directly using a large model, our method effectively transfers the capabilities of the large model by training a small model, making the performance of the small model close to that of the large model in short text classification tasks, while significantly reducing hardware resource consumption. The high efficiency of the trained small model enables it to adapt to more resource-limited application scenarios, especially in environments with limited hardware performance.

[0057] 2. Thinking chain design improves reasoning and emergence ability

[0058] Compared with the traditional method of directly inputting short texts, the two thought chain strategies we designed can better stimulate and improve the reasoning and emergence capabilities of the model. Just as humans need to reason step by step when solving complex problems, the model can more accurately handle the semantic and syntactic challenges in short texts in the process of step-by-step reasoning through thought chains, avoiding errors caused by leaps and bounds.

[0059] 3. Improve classification accuracy and support reasoning without retraining

[0060] Compared with traditional graph-based methods, our method significantly improves the classification accuracy in short text classification tasks. In addition, unlike traditional methods that need to be retrained every time a new sample arrives, our method can directly perform inference on new test samples without retraining, which greatly improves the application efficiency and flexibility of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 Flow chart of a large model distillation method for low-resource short text classification according to an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of a graph neural network according to an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of the principle of a large model distillation method for low-resource short text classification according to an embodiment of the present invention;

[0065] Figure 4 is a structural diagram of a large model distillation system for low-resource short text classification according to an embodiment of the present invention;

[0066] Figure 5 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0068] The first aspect of the present invention discloses a large model distillation method for low-resource short text classification. Figure 1 FIG. 1 is a flowchart of a large model distillation method for low-resource short text classification according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] Step S1, using the same large model, according to the multi-step prompting method, establish the semantic syntax enhancement thinking chain and the domain knowledge thinking chain;

[0070] Step S2, applying the semantic syntactic enhancement thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge;

[0071] Step S3, applying the domain knowledge thinking chain to extract basic concepts in the short text, and generating domain knowledge through the basic concepts;

[0072] Step S4, training the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

[0073] In step S1, using the same large model, according to the multi-step prompting method, the semantic syntax enhancement thinking chain and the domain knowledge thinking chain are established.

[0074] Specifically, unlike the traditional knowledge distillation method (i.e. supervising the student model through the last few layers of the teacher model), Figure 3 As shown in Figure 1, this embodiment uses the data generated by the large model to supervise the small model. Specifically, in the stage of building the thought chain, innovative prompts are designed to guide the large model to gradually generate data that meets expectations. On this basis, a multi-task fine-tuning strategy is used to train the small model to further improve its performance.

[0075] The goal of this step is to stimulate the inherent knowledge and emergent capabilities of the large language model. Emergent capabilities refer to certain capabilities that do not appear in smaller models but appear in larger models. In this embodiment, it specifically refers to the "step-by-step thinking" reasoning ability of the large model. For the short text classification task, two sets of thinking chains are designed: semantic and syntactic enhancement thinking chain and domain knowledge thinking chain.

[0076] The design intentions and knowledge focuses of the two thought chains are different: the semantic and syntactic enhancement thought chain aims to overcome the inherent defects of short texts and solve the semantic and syntactic challenges caused by length limitations. The domain knowledge thought chain focuses on stimulating knowledge in specific fields, thereby improving the generalization ability of small language models on test sets.

[0077] The generation of the above thought chain relies on a multi-step prompting method. In the specific implementation, two sets of thought chains are constructed using the same large model (such as GPT-3.5, LLaMA, ChatGLM, etc.). This embodiment is based on API call implementation. Taking GPT-3.5 as an example, its API call process is briefly described.

[0078] Set the API key and some access request parameters. Before using the OpenAI API, you need to initialize the client and complete the relevant configuration. Through the SDK provided by OpenAI, you can set the API key (api_key) for authentication and configure the base URL (base_url) to specify the entrance of the API service. If necessary, you can also set some global parameters, such as the maximum number of retries (MAX_RETRIES) or the request timeout, to ensure that the request can be retried when the network fluctuates.

[0079] When constructing request data, you need to determine the input content based on the specific task. For example, you can extract some content from a document as sample data, and combine multiple text fragments into model input by traversing and splicing. In the preparation process, you also need to organize the data according to OpenAI's API format, such as packaging the content into a "message" list, where each message specifies the role (such as user) and content (such as user input).

[0080] Use the client to call the OpenAI API and send a request to the model through the chat.completions.create method. Specify the model to be used in the request (for example, gpt-3.5-turbo) and send the prepared data as the message body. This request will interact with the OpenAI service to obtain the results generated by the model. When sending a request, you need to handle network exceptions or service errors to ensure the reliability of the call.

[0081] After receiving the response from the API, extract the content generated by the model from the returned data. For OpenAI's interface, this can usually be obtained through response.choices[0].message.content. If the response structure is complex, you can further parse it according to your needs, such as extracting multiple options or capturing the status information returned by the API to ensure that the results meet your expectations.

[0082] In step S2, the semantic and syntactic enhanced thinking chain is applied to extract key concepts in the short text, the common sense knowledge of the short text is retrieved through the key concepts, and an improved text after integrating semantic and syntactic reasoning knowledge is output based on the common sense knowledge.

[0083] In some embodiments, in step S2, applying the semantic syntactic enhanced thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge includes:

[0084]

[0085]

[0086]

[0087] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 1 Indicates key concepts; Represents the context for extracting key concepts; represents the key concept extraction instruction; S represents common sense knowledge; represents the context for retrieving common sense knowledge; represents the retrieval of common sense knowledge instructions; R represents the improvement of text; Represents the context of the output improvement text; Indicates output improvement text instructions.

[0088] Specifically, the semantic-syntactic-enhanced thinking chain method consists of three steps: Key concept extraction: First, identify the key concepts in the short text to establish an understanding of the basic meaning of the text. This step aims to solve the problem of information loss that may be caused by the concise content of the short text. Semantic enrichment: Then, by expanding and associating key concepts, the model's understanding of the text semantics and the relationship between concepts is deepened. This step alleviates the problem of insufficient semantic expression of short texts. Syntactic enrichment: Finally, the grammar and structure of the text are optimized and improved to ensure that the generated content conforms to the language expression specifications, while enhancing the model's ability to grasp syntax.

[0089] The above steps are closely related to the common problems in short text classification tasks, such as lack of semantics and syntactic simplicity caused by text length limitations.

[0090] Specific methods include:

[0091] First, the model M is required to identify relevant concepts using a specified template. Specifically, for a given short text x i , the model needs to identify the key concepts. The context of this step is given by The remaining text constitutes the instruction The goal of this stage is to allow the model to focus on extracting basic content and prepare for the next step of processing.

[0092] The process can be formally expressed as the first step:

[0093]

[0094] Among them, model M uses its natural language understanding and information extraction capabilities to extract information from the context and instructions Identify the key concept K 1 .

[0095] Acquired key concepts K in the previous stage 1 Based on the model M, the template is used to retrieve relevant common sense knowledge S from the inherent knowledge base of the model M. Specifically, the context of this process is composed of Represents, including the context of the first step and the key concepts K identified 1 The rest of the text As the second step instruction. The formal expression is:

[0096]

[0097] This stage relies on the model’s ability to self-retrieve its internal knowledge. Through this knowledge retrieval method, the model is able to fill the semantic gap caused by the brevity of the text in short text classification tasks, while providing a richer and more comprehensive response.

[0098] After retrieving relevant common sense knowledge, this step aims to improve the original short text to make it semantically rich and syntactically rigorous. This process can be modeled as:

[0099]

[0100] in is the instruction of the third step. The final output R represents the improved text after integrating semantic and syntactic reasoning knowledge.

[0101] In step S3, the domain knowledge thinking chain is applied to extract basic concepts in the short text, and domain knowledge is generated through the basic concepts.

[0102] In some embodiments, in step S3, applying the domain knowledge thinking chain to extract basic concepts from the short text and generating domain knowledge through the basic concepts includes:

[0103]

[0104]

[0105] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 2 Indicates basic concepts; Represents the context for extracting basic concepts; Indicates the instruction to extract basic concepts; O indicates domain knowledge; Represents the context in which domain knowledge is generated; Represents instructions for generating domain knowledge.

[0106] Specifically, in the first step, use the specified template and instructions (“Identify basic concepts, considering the main entities, actions, and events described”) Let M conduct queries that take advantage of its ability to pinpoint relevant concepts. Unlike the first step of the semantic syntax-enhanced thought chain, the domain knowledge thought chain contains domain-specific prompt words. For example, in the news domain, special attention needs to be paid to key entities, actions, and events. Context Contains the context information of the first step, and the remaining content constitutes the reasoning instruction This process can be modeled as:

[0107]

[0108] After establishing basic concepts in the initial phase, the second step aims to prompt M to incorporate domain-specific terminology and deeper analytical perspectives into its output. This process can be modeled as:

[0109]

[0110] in, Include previous context and the identified basic concepts K 2 ,instruction "Provide a summary of the identified components, including their interrelationships and overall significance in the context of the text."

[0111] The final output O represents the rich knowledge retrieved or generated by the function, combining the identified basic concepts with in-depth, domain-specific information to form a knowledge system of domain knowledge thinking chain. This process fully utilizes the model's ability to mine domain information.

[0112] In step S4, the small model is trained through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

[0113] In some embodiments, in step S4, the short text classification task is:

[0114] The input of the small model is a short text, and the output is a predicted label.

[0115] The semantic and syntactic reasoning tasks of the improved text model output are:

[0116] The input of the small model is semantic and syntactic enhanced thinking chain prompts and short texts, and the output is semantic and syntactic reasoning knowledge.

[0117] The domain knowledge extraction task of the model output is:

[0118] The input of the small model is the domain knowledge thinking chain prompt and short text, and the output is domain knowledge. The loss function for training the small model is:

[0119] L=λ1L1+λ2L2+λ3L3

[0120]

[0121]

[0122]

[0123] Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss.

[0124] Specifically, the small model used in this embodiment refers to the FLAN-T5 model, which is denoted by the symbol F in the following text.

[0125] The goal of the fine-tuning small parameter language model stage is to achieve the following two optimizations through multi-task learning:

[0126] 1. Directly learn the mapping from input short text to output label;

[0127] 2. Learn the knowledge and emergence capabilities of large models.

[0128] The core idea of ​​multi-task learning is to process multiple input-output pairs simultaneously, and jointly optimize the model parameters by calculating the losses of multiple tasks. In our method, three input-output pairs are designed: short text classification task: input short text, output predicted label; semantic and syntactic reasoning task: input semantic and syntactic thinking chain prompts and short text, output semantic and syntactic reasoning knowledge; domain knowledge extraction task: input domain knowledge thinking chain prompts and short text, output domain knowledge.

[0129] Through joint training of these tasks, the small-parameter model can not only grasp the direct mapping relationship from short text to labels, but also fully absorb the semantic, syntactic capabilities and domain knowledge reasoning capabilities of the large model, thereby significantly improving its generalization performance.

[0130] In the training phase, the short text x i (i=1,2,…,N) are sequentially input into the small model F three times, so that F predicts Then, these predicted values ​​are compared with the true value y i、r i , o i (The symbols correspond to the output of the previous chain of thought) Compare and calculate the loss. The details are as follows:

[0131] L=λ1L1+λ2L2+λ3L3

[0132]

[0133]

[0134]

[0135] Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss. λ1,λ2,λ3 ensure that no single task dominates the training of the model.

[0136] After calculating the small model training loss L, the parameters of model F are updated through the optimization algorithm, thereby gradually improving the performance of the model on all tasks. Specifically, all parameters in model F will participate in the training process, the gradients are calculated through back propagation, and the model parameters are updated using the optimization algorithm (Adam). Full parameter fine-tuning can ensure that the model learns on multiple tasks at the same time, further improving the comprehensive performance of the model on classification tasks, semantic reasoning tasks, and domain knowledge tasks.

[0137] In summary, the solution proposed by the present invention can

[0138] 1. Save hardware resources and adapt to more application scenarios

[0139] Compared with directly using a large model, our method effectively transfers the capabilities of the large model by training a small model, making the performance of the small model close to that of the large model in short text classification tasks, while significantly reducing hardware resource consumption. The high efficiency of the trained small model enables it to adapt to more resource-limited application scenarios, especially in environments with limited hardware performance.

[0140] 2. Thinking chain design improves reasoning and emergence ability

[0141] Compared with the traditional method of directly inputting short texts, the two thought chain strategies we designed can better stimulate and improve the reasoning and emergence capabilities of the model. Just like humans need to reason step by step when solving complex problems, the model can more accurately handle the semantic and syntactic challenges in short texts in the process of step-by-step reasoning through thought chains, avoiding errors caused by leaps and bounds.

[0142] 3. Improve classification accuracy and support reasoning without retraining

[0143] Compared with traditional graph-based methods, our method significantly improves the classification accuracy in short text classification tasks. In addition, unlike traditional methods that need to be retrained every time a new sample arrives, our method can directly perform inference on new test samples without retraining, which greatly improves the application efficiency and flexibility of the model.

[0144] A second aspect of the present invention discloses a large model distillation system for low-resource short text classification. Figure 4 is a structural diagram of a large model distillation system for low-resource short text classification according to an embodiment of the present invention; Figure 4 As shown, the system 100 includes:

[0145] The first processing module 101 is configured to establish a semantic syntax enhancement thinking chain and a domain knowledge thinking chain using the same large model according to a multi-step prompting method;

[0146] The second processing module 102 is configured to apply the semantic and syntactic enhanced thinking chain to extract key concepts in the short text, retrieve common sense knowledge of the short text through the key concepts, and output an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge;

[0147] The third processing module 103 is configured to apply the domain knowledge thinking chain to extract basic concepts in the short text and generate domain knowledge through the basic concepts;

[0148] The fourth processing module 104 is configured to train the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

[0149] According to the system of the second aspect of the present invention, the first processing module 101 is specifically configured to be different from the traditional knowledge distillation method (i.e., supervising the student model through the last few layers of the teacher model), such as Figure 3 As shown in Figure 1, this embodiment uses the data generated by the large model to supervise the small model. Specifically, in the stage of building the thought chain, innovative prompts are designed to guide the large model to gradually generate data that meets expectations. On this basis, a multi-task fine-tuning strategy is used to train the small model to further improve its performance.

[0150] The goal of this step is to stimulate the inherent knowledge and emergent capabilities of the large language model. Emergent capabilities refer to certain capabilities that do not appear in smaller models but appear in larger models. In this embodiment, it specifically refers to the "step-by-step thinking" reasoning ability of the large model. For the short text classification task, two sets of thinking chains are designed: semantic and syntactic enhancement thinking chain and domain knowledge thinking chain.

[0151] The design intentions and knowledge focuses of the two thought chains are different: the semantic and syntactic enhancement thought chain aims to overcome the inherent defects of short texts and solve the semantic and syntactic challenges caused by length limitations. The domain knowledge thought chain focuses on stimulating knowledge in specific fields, thereby improving the generalization ability of small language models on test sets.

[0152] The generation of the above thought chain relies on a multi-step prompting method. In the specific implementation, two sets of thought chains are constructed using the same large model (such as GPT-3.5, LLaMA, ChatGLM, etc.). This embodiment is based on API call implementation. Taking GPT-3.5 as an example, its API call process is briefly described.

[0153] Set the API key and some access request parameters. Before using the OpenAI API, you need to initialize the client and complete the relevant configuration. Through the SDK provided by OpenAI, you can set the API key (api_key) for authentication and configure the base URL (base_url) to specify the entrance of the API service. If necessary, you can also set some global parameters, such as the maximum number of retries (MAX_RETRIES) or the request timeout, to ensure that the request can be retried when the network fluctuates.

[0154] When constructing request data, you need to determine the input content based on the specific task. For example, you can extract some content from a document as sample data, and combine multiple text fragments into model input by traversing and splicing. In the preparation process, you also need to organize the data according to OpenAI's API format, such as packaging the content into a "message" list, where each message specifies the role (such as user) and content (such as user input).

[0155] Use the client to call the OpenAI API and send a request to the model through the chat.completions.create method. Specify the model to be used in the request (for example, gpt-3.5-turbo) and send the prepared data as the message body. This request will interact with the OpenAI service to obtain the results generated by the model. When sending a request, you need to handle network exceptions or service errors to ensure the reliability of the call.

[0156] After receiving the response from the API, extract the content generated by the model from the returned data. For OpenAI's interface, this can usually be obtained through response.choices[0].message.content. If the response structure is complex, you can further parse it according to your needs, such as extracting multiple options or capturing the status information returned by the API to ensure that the results meet your expectations.

[0157] According to the system of the second aspect of the present invention, the second processing module 102 is specifically configured to apply the semantic syntactic enhanced thinking chain to extract key concepts in the short text, retrieve common sense knowledge of the short text through the key concepts, and output an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge, including:

[0158]

[0159]

[0160]

[0161] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 1 Indicates key concepts; Represents the context for extracting key concepts; represents the key concept extraction instruction; S represents common sense knowledge; represents the context for retrieving common sense knowledge; represents the retrieval of common sense knowledge instructions; R represents the improvement of text; Represents the context of the output improvement text; Indicates output improvement text instructions.

[0162] Specifically, the semantic-syntactic-enhanced thinking chain method consists of three steps: Key concept extraction: First, identify the key concepts in the short text to establish an understanding of the basic meaning of the text. This step aims to solve the problem of information loss that may be caused by the concise content of the short text. Semantic enrichment: Then, by expanding and associating key concepts, the model's understanding of the text semantics and the relationship between concepts is deepened. This step alleviates the problem of insufficient semantic expression of short texts. Syntactic enrichment: Finally, the grammar and structure of the text are optimized and improved to ensure that the generated content conforms to the language expression specifications, while enhancing the model's ability to grasp syntax.

[0163] The above steps are closely related to the common problems in short text classification tasks, such as lack of semantics and syntactic simplicity caused by text length limitations.

[0164] Specific methods include:

[0165] First, the model M is required to identify relevant concepts using a specified template. Specifically, for a given short text x i , the model needs to identify the key concepts. The context of this step is given by The remaining text constitutes the instruction The goal of this stage is to allow the model to focus on extracting basic content and prepare for the next step of processing.

[0166] The process can be formally expressed as the first step:

[0167]

[0168] Among them, model M uses its natural language understanding and information extraction capabilities to extract information from the context and instructions Identify the key concept K 1 .

[0169] Acquired key concepts K in the previous stage 1 Based on the model M, the template is used to retrieve relevant common sense knowledge S from the inherent knowledge base of the model M. Specifically, the context of this process is composed of Represents, including the context of the first step and the key concepts K identified 1 The rest of the text As the second step instruction. The formal expression is:

[0170]

[0171] This stage relies on the model’s ability to self-retrieve its internal knowledge. Through this knowledge retrieval method, the model is able to fill the semantic gap caused by the brevity of the text in short text classification tasks, while providing a richer and more comprehensive response.

[0172] After retrieving relevant common sense knowledge, this step aims to improve the original short text to make it semantically rich and syntactically rigorous. This process can be modeled as:

[0173]

[0174] in is the instruction of the third step. The final output R represents the improved text after integrating semantic and syntactic reasoning knowledge.

[0175] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured to apply the domain knowledge thinking chain to extract basic concepts in the short text, and generate domain knowledge through the basic concepts, including:

[0176]

[0177]

[0178] Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 2 Indicates basic concepts; Represents the context for extracting basic concepts; Indicates the instruction to extract basic concepts; O indicates domain knowledge; Represents the context in which domain knowledge is generated; Represents instructions for generating domain knowledge.

[0179] Specifically, in the first step, use the specified template and instructions (“Identify basic concepts, considering the main entities, actions, and events described”) Let M conduct queries that take advantage of its ability to pinpoint relevant concepts. Unlike the first step of the semantic syntax-enhanced thought chain, the domain knowledge thought chain contains domain-specific prompt words. For example, in the news domain, special attention needs to be paid to key entities, actions, and events. Context Contains the context information of the first step, and the remaining content constitutes the reasoning instruction This process can be modeled as:

[0180]

[0181] After establishing basic concepts in the initial phase, the second step aims to prompt M to incorporate domain-specific terminology and deeper analytical perspectives into its output. This process can be modeled as:

[0182]

[0183] in, Include previous context and the identified basic concepts K 2 ,instruction "Provide a summary of the identified components, including their interrelationships and overall significance in the context of the text."

[0184] The final output O represents the rich knowledge retrieved or generated by the function, combining the identified basic concepts with in-depth, domain-specific information to form a knowledge system of domain knowledge thinking chain. This process fully utilizes the model's ability to mine domain information.

[0185] According to the system of the second aspect of the present invention, the fourth processing module 104 is specifically configured as follows: the short text classification task is:

[0186] The input of the small model is a short text, and the output is a predicted label.

[0187] The semantic and syntactic reasoning tasks of the improved text model output are:

[0188] The input of the small model is semantic and syntactic enhanced thinking chain prompts and short texts, and the output is semantic and syntactic reasoning knowledge.

[0189] The domain knowledge extraction task of the model output is:

[0190] The input of the small model is the domain knowledge thinking chain prompt and short text, and the output is domain knowledge. The loss function for training the small model is:

[0191] L=λ1L1+λ2L2+λ3L3

[0192]

[0193]

[0194]

[0195] Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss.

[0196] Specifically, the small model used in this embodiment refers to the FLAN-T5 model, which is denoted by the symbol F in the following text.

[0197] The goal of the fine-tuning small parameter language model stage is to achieve the following two optimizations through multi-task learning:

[0198] 1. Directly learn the mapping from input short text to output label;

[0199] 2. Learn the knowledge and emergence capabilities of large models.

[0200] The core idea of ​​multi-task learning is to process multiple input-output pairs simultaneously, and jointly optimize the model parameters by calculating the losses of multiple tasks. In our method, three input-output pairs are designed: short text classification task: input short text, output predicted label; semantic and syntactic reasoning task: input semantic and syntactic thinking chain prompts and short text, output semantic and syntactic reasoning knowledge; domain knowledge extraction task: input domain knowledge thinking chain prompts and short text, output domain knowledge.

[0201] Through joint training of these tasks, the small-parameter model can not only grasp the direct mapping relationship from short text to labels, but also fully absorb the semantic, syntactic capabilities and domain knowledge reasoning capabilities of the large model, thereby significantly improving its generalization performance.

[0202] In the training phase, the short text x i (i=1,2,…,N) are sequentially input into the small model F three times, so that F predicts Then, these predicted values ​​are compared with the true value y i 、r i , o i(The symbols correspond to the output of the previous chain of thought) Compare and calculate the loss. The details are as follows:

[0203] L=λ1L1+λ2L2+λ3L3

[0204]

[0205]

[0206]

[0207] Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss. λ1,λ2,λ3 ensure that no single task dominates the training of the model.

[0208] After calculating the small model training loss L, the parameters of model F are updated through the optimization algorithm, thereby gradually improving the performance of the model on all tasks. Specifically, all parameters in model F will participate in the training process, the gradients are calculated through back propagation, and the model parameters are updated using the optimization algorithm (Adam). Full parameter fine-tuning can ensure that the model learns on multiple tasks at the same time, further improving the comprehensive performance of the model on classification tasks, semantic reasoning tasks, and domain knowledge tasks.

[0209] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the large model distillation methods for low-resource short text classification disclosed in the first aspect of the present invention are implemented.

[0210] Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 5As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0211] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0212] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the large model distillation method for low-resource short text classification in any one of the first aspects of the present invention are implemented.

[0213] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A large model distillation method for low-resource short text classification, characterized by: The method comprises: Step S1, using the same large model, according to the multi-step prompting method, establish the semantic syntax enhancement thinking chain and the domain knowledge thinking chain; Step S2, applying the semantic syntactic enhancement thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge; Step S3, applying the domain knowledge thinking chain to extract basic concepts in the short text, and generating domain knowledge through the basic concepts; Step S4, training the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

2. A large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S2, applying the semantic syntactic enhancement thinking chain to extract key concepts in the short text, retrieving common sense knowledge of the short text through the key concepts, and outputting an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge includes: Among them, M() represents a large model, including GPT-3.5, LLaMA or ChatGLM; K 1 Indicates key concepts; Represents the context for extracting key concepts; represents the key concept extraction instruction; S represents common sense knowledge; represents the context for retrieving common sense knowledge; represents the retrieval of common sense knowledge instructions; R represents the improvement of text; Represents the context of the output improvement text; Indicates output improvement text instructions.

3. The large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S3, applying the domain knowledge thinking chain to extract basic concepts from the short text and generating domain knowledge through the basic concepts includes: Among them, M() represents a large model, such as GPT-3.5, LLaMA or ChatGLM; K 2 Indicates basic concepts; Represents the context for extracting basic concepts; Indicates the instruction to extract basic concepts; O indicates domain knowledge; Represents the context in which domain knowledge is generated; Represents instructions for generating domain knowledge.

4. The large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S4, the short text classification task is: The input of the small model is a short text, and the output is a predicted label.

5. The large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S4, the semantic and syntactic reasoning tasks of the improved text output by the model are: The input of the small model is semantic and syntactic enhanced thinking chain prompts and short texts, and the output is semantic and syntactic reasoning knowledge.

6. A large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S4, the domain knowledge is the domain knowledge extraction task of the model output: The input of the small model is domain knowledge thinking chain prompts and short texts, and the output is domain knowledge.

7. The large model distillation method for low-resource short text classification according to claim 1, characterized in that: In step S4, the loss function for training the small model is: L=λ1L1+λ2L2+λ3L3 Where L represents the small model training loss function; L1 represents the short text classification task loss function; λ1 represents the weight of the short text classification task loss function; L2 represents the semantic and syntactic reasoning task loss function; λ2 represents the weight of the semantic and syntactic reasoning task loss function; L3 represents the domain knowledge extraction task loss function; λ3 represents the weight of the domain knowledge extraction task loss function; N represents the total number of training samples; y i Represents the output of the small model for the i-th training sample of the short text classification task; Represents the label of the short text classification task; r i Represents the output of the small model for the i-th training sample of the semantic and syntactic reasoning task; Labels representing semantic and syntactic reasoning tasks; o i The output of the small model representing the i-th training sample of the domain knowledge extraction task; Represents the label of the domain knowledge extraction task; l() represents the cross entropy loss.

8. A large model distillation system for low-resource short text classification, characterized by: The system comprises: The first processing module is configured to establish a semantic syntactic enhancement thinking chain and a domain knowledge thinking chain based on a multi-step prompting method using the same large model; The second processing module is configured to apply the semantic and syntactic enhanced thinking chain to extract key concepts in the short text, retrieve common sense knowledge of the short text through the key concepts, and output an improved text after integrating semantic and syntactic reasoning knowledge according to the common sense knowledge; The third processing module is configured to apply the domain knowledge thinking chain to extract basic concepts in the short text and generate domain knowledge through the basic concepts; The fourth processing module is configured to train the small model through short text classification tasks, semantic and syntactic reasoning tasks with the improved text as model output, and domain knowledge extraction tasks with the domain knowledge as model output.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the large model distillation method for low-resource short text classification described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the large model distillation method for low-resource short text classification described in any one of claims 1 to 7 are implemented.

Images