Self-interpretation text classification method and device based on concepts

Abstract

The invention provides a concept-based self-interpretation text classification method and device, and the method comprises the steps: carrying out the text coding of a target text, and obtaining a coding feature; constructing a concept extractor model based on a competitive attention mechanism, extracting concept features by taking the coding features as input, and obtaining a concept attention score of the target text; obtaining a concept intensity feature of the target text based on the concept attention score; and constructing a text classifier model, and performing text classification on the target text through the text classifier model based on the concept strength features. According to the unsupervised text classification method capable of being explained in the process, semantic concepts are automatically extracted by utilizing a competitive attention mechanism, meanwhile, an evaluation framework is designed, the understandability of the extracted concepts is evaluated by utilizing the capability of a large language model, iterative fine tuning is performed on a network by utilizing feedback provided by the large language model, and the classification efficiency is improved. Therefore, the extracted concept is easy to understand by human beings.

Description

Technical Field

[0001] The present invention belongs to the technical field of text semantic recognition, and particularly relates to a concept-based self-explanatory text classification method and device. Background Art

[0002] The rapid development of deep neural language models has profoundly transformed multiple dimensions of society and industry, bringing about unprecedented change. However, these models are highly complex and opaque, making their decision-making mechanisms difficult for humans to understand. To overcome this challenge, numerous explainable AI methods have been proposed, with feature attribution methods being a mainstream approach. These methods generate intuitive heat maps to highlight the importance of each input feature to the model's decision. However, recent research has shown that feature attribution methods often fail to provide sufficiently profound and meaningful explanations. Consequently, the focus of explainability methods has shifted to deeper needs—not only pinpointing the location of important features but also revealing the meaning and logic behind these features. In this context, concept-based explanation methods have emerged. These methods map input features to a set of human-understandable concepts and further quantify the specific impact of each concept on the model's predictions. By explaining model predictions in a way that is closer to human cognition and understanding, concept-based explainability methods can improve the readability and intuitiveness of model explanations, promoting the safe and effective application of AI technology in a wider range of fields.

[0003] Some current concept-based methods perform post-analysis on trained language models, attempting to extract concepts from intermediate-layer representations to explain the model's decision-making results or clarify the meaning of its internal network structure. However, since the model does not directly encounter these concepts during training, there are still doubts as to whether the model truly understands and applies the discovered concepts. Self-explanatory concept methods provide a more reliable solution. By designing the model structure, such methods can provide built-in explanations while making decisions. Despite this, most of these methods are highly dependent on large amounts of concept-annotated data. They often require manual or large language models to pre-define the concepts to be extracted, which greatly restricts their ability to autonomously identify new concepts during training. Therefore, current concept-based explanation methods are unable to automatically mine understandable new concepts from a corpus in a self-explanatory framework without relying on external concept label supervision. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that the existing technology cannot automatically discover understandable new concepts to reliably explain model decisions. A concept-based self-explanatory text classification method is proposed, including: text encoding of the target text to obtain encoding features; constructing a concept extractor model based on a competitive attention mechanism, taking the encoding features as input, extracting concept features, and obtaining the concept attention score of the target text; obtaining the concept strength feature of the target text based on the concept attention score; constructing a text classifier model, and performing text classification on the target text through the text classifier model based on the concept strength feature.

[0005] Furthermore, it also includes: generating concept meaning based on the concept attention score, extracting concept features of a new target text based on the concept meaning, calculating concept comprehensibility loss, and fine-tuning model parameters of the concept extractor model based on the concept comprehensibility loss; and fine-tuning model parameters of the text classifier model based on the classification result of the new target text.

[0006] Furthermore, the loss function for classifying the target text is :

[0007]

[0008] is the classification loss, , is the sample category label, is the prediction result of the text classifier model;

[0009] is the concept consistency loss, , is the number of concepts, and Different word pair concepts for the target text The concept features, k represents the concept The top k samples with the highest concept attention scores;

[0010] is the concept diversity loss, , 、 is the average feature, , A relative concept Different concepts of

[0011] and is the balance parameter.

[0012] Furthermore, the loss function for fine-tuning the model parameters of the concept extractor model and the text classifier model is :

[0013]

[0014] is the loss of conceptual comprehensibility, , For concept The contribution of the model decision, , For concept The conceptual comprehensibility score of , is the concept attention score, Simulate activation matrices for concepts.

[0015] The present invention also proposes a concept-based self-explanatory text classification device, including: a classification module for performing text classification on a target text; including: performing text encoding on the target text to obtain encoding features; constructing a concept extractor model based on a competitive attention mechanism, taking the encoding features as input, extracting concept features, and obtaining a concept attention score of the target text; obtaining a concept strength feature of the target text based on the concept attention score; constructing a text classifier model, and performing text classification on the target text through the text classifier model based on the concept strength feature.

[0016] Furthermore, it also includes: a fine-tuning module for fine-tuning the parameters of the classification model; including: generating concept meaning based on the concept attention score, extracting concept features of the new target text based on the concept meaning, calculating the concept comprehensibility loss, and fine-tuning the model parameters of the concept extractor model based on the concept comprehensibility loss; fine-tuning the model parameters of the text classifier model based on the classification result of the new target text.

[0017] Furthermore, the loss function for classifying the target text is :

[0018]

[0019] is the classification loss, , is the sample category label, is the prediction result of the text classifier model;

[0020] is the concept consistency loss, , is the number of concepts, and Different word pair concepts for the target text The concept features, k represents the concept The top k samples with the highest concept attention scores;

[0021] is the concept diversity loss, , 、 is the average feature, , A relative concept Different concepts of

[0022] and is the balance parameter.

[0023] Furthermore, the loss function for fine-tuning the model parameters of the concept extractor model and the text classifier model is :

[0024]

[0025] is the loss of conceptual comprehensibility, , For concept The contribution of the model decision, , For concept The conceptual comprehensibility score of , is the concept attention score, Simulate activation matrices for concepts.

[0026] The present invention also provides an electronic device comprising the concept-based self-explanatory text classification device as described above.

[0027] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the concept-based self-explanatory text classification method as described above is implemented.

[0028] This paper proposes a process-interpretable unsupervised text classification method that automatically extracts semantic concepts using a competitive attention mechanism. It also designs an evaluation framework that leverages the power of a large language model to assess the comprehensibility of the extracted concepts. Feedback from the large language model is then used to iteratively fine-tune the network to ensure that the extracted concepts are easily understood by humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the concept-based self-explanatory text classification method of the present invention.

[0030] Figure 2It is a flow chart of the classification-oriented concept extraction and model training phase of the present invention.

[0031] Figure 3 It is a flow chart of the model fine-tuning phase based on concept understandability of the present invention.

[0032] Figure 4 Schematic diagram of the concept-based self-explanatory text classification device of the present invention.

[0033] Figure 5 It is a schematic diagram of an electronic device of the present invention.

[0034] Figure 6 It is a schematic diagram of the hardware structure of an electronic device of the present invention.

[0035] Wherein, the accompanying drawings are marked as follows:

[0036] 100: Electronic device 10: Self-explanatory text classification device

[0037] 11: Classification module 12: Fine-tuning module DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0040] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0041] In response to the above problems, the present invention proposes a concept-based self-explanatory text classification method, which can realize the automatic extraction of semantic concepts by encouraging each concept to competitively explain different information of the input features without concept supervision; and designs concept consistency loss and concept diversity loss to constrain the concept learning process. In order to further enhance the interpretability of concepts, the present invention applies a large language model to the extracted concepts to verify whether these concepts can generate explanations consistent with human intuition, and simulate reasonable concept activation patterns to achieve automated evaluation of concept comprehensibility, and thereby construct comprehensibility loss as a feedback mechanism to iteratively fine-tune and optimize the model. Through the self-explanatory model architecture, the present invention can automatically discover easy-to-understand concepts and clearly explain model decisions, thereby significantly improving the transparency and credibility of the text classification model.

[0042] The self-explanatory text classification method of the present invention comprises:

[0043] A concept extraction method based on a competitive attention mechanism. This method defines multiple concept vectors. When text data is input, the concept vectors work in parallel, each attempting to capture specific information or features in the data. Through the attention mechanism, these concept vectors compete with each other to better interpret and represent different aspects of the input data. Over multiple iterations, each concept vector gradually adjusts its focus until it can stably represent a unique and useful concept.

[0044] This method uses a large language model to automatically assess the comprehensibility of concepts. It applies the large language model to extracted concepts to observe whether they can generate intuitive explanations and simulate reasonable activations. If the concept attention scores output by the concept extractor are highly similar to the simulated activation scores of the large language model, it indicates that the concept can be compressed and reconstructed using natural language, and is therefore considered human-understandable.

[0045] A concept-based self-explanatory model training mechanism combines a concept extraction method based on a competitive attention mechanism with an automated concept comprehensibility assessment method using a large language model. In the first stage, the concept extractor and classifier are trained using classification loss, concept consistency loss, and concept diversity loss to learn concepts with certain semantics. To ensure concept comprehensibility, the second stage performs concept comprehensibility assessment and fine-tunes the model using classification loss and concept comprehensibility loss.

[0046] Figure 1 FIG. 1 is a flow chart of the concept-based self-explanatory text classification method of the present invention. Figure 1As shown, in a first embodiment of the present invention, a self-explanatory text classification method is proposed, comprising:

[0047] Step 1: classification-oriented concept extraction and model training;

[0048] Obtain a text sample to be classified and encode it into encoding features using a text encoder; construct a trainable concept vector, perform random initialization, and build a concept extractor network based on a competitive attention mechanism; input the sample encoding features into the concept extractor network, interact to extract concept features, and obtain concept attention scores; calculate the concept consistency loss and diversity loss based on the concept features of the batch samples to train the concept vector; sum the sample concept attention scores along the word length dimension to obtain concept strength features, input them into the text classifier for classification, and output the classification results of the text;

[0049] Step 2: Fine-tune the model based on concept comprehensibility. Use the large language model to evaluate the comprehensibility of the extracted concepts, summarize the concept meaning based on the concept attention score of the sample, and simulate activation on new samples based on the summary. Use the simulation results to calculate the comprehensibility loss to fine-tune the concept extractor. At the same time, fine-tune the classifier for the classification task. The fine-tuning process includes:

[0050] Step 21, classification task-oriented concept extraction and model training, such as Figure 2 As shown:

[0051] For word length After the text encoder, each word is encoded into the dimension , the encoding feature is ;

[0052] The concept extractor module includes a learnable concept query matrix ,in is the number of concepts. Each concept query vector needs to competitively explain the different contents of the input features. Through multiple rounds of iterative attention competition, these concept query vectors are gradually adjusted to represent different typical features. Input sample encoding features , the concept extractor makes the features and the Concept Query Matrix Perform competitive attention interaction and output sample concept matrix And the concept attention matrix ,The specific process of the concept competition attention module is as follows;

[0053] First, the learnable concept query matrix Assign to ,Right now ;

[0054] conduct Iterations:

[0055]

[0056]

[0057]

[0058]

[0059] in, is a learnable weight parameter, It is a concept attention matrix normalized in the concept dimension to achieve competition among concepts for input features; It is the concept-weighted text feature, which is input into the gated recurrent unit (GRU) to update the concept vector;

[0060] In order to ensure the consistency of concepts, the features activated by different samples under the same concept should be similar. , conceptual features and Should be similar to each other. Select the top k samples with the highest concept attention score in a batch for each concept, and the consistency loss Defined as:

[0061]

[0062] In order to ensure the diversity of concepts, different concepts should focus on different features. That is, in a batch, according to the concept The average feature calculated from the top k samples with the highest concept attention score , should be compared with other concepts The average features calculated Dissimilarity, diversity loss Defined as:

[0063]

[0064] The input of the classifier is the concept strength feature, which is obtained by summing the concept attention scores of the samples along the word length dimension:

[0065]

[0066] This classifier uses fully connected layers without bias. is the learnable classifier weight, where Indicates the number of task categories, and the concept strength feature Input the classifier and obtain the classification results:

[0067]

[0068] The classification task constructs the classification loss through cross entropy loss :

[0069]

[0070] in is the sample category label, Predict results for the classifier;

[0071] The total loss function of the first stage training is:

[0072]

[0073] in and is the loss balance parameter;

[0074] Step 22: Fine-tune the model based on concept comprehensibility. The flowchart for this stage is as follows: Figure 3 As shown:

[0075] After the first stage of training, the model's performance on classification tasks tends to be stable, and concepts with certain semantics can be discovered through the concept extractor. In order to evaluate the comprehensibility of the discovered concepts and further guide the model to extract concepts that are easier to understand, the present invention uses a large language model to calculate the concept comprehensibility score; the concept attention score of the previous stage is converted into the concept attention score. Mapped to the original text, the concept is summarized in natural language using a large language model; based on the concept summary, the large language model is used to simulate the concept activation on the sample to obtain the concept simulation activation matrix ;

[0076] For each concept , calculate the concept comprehensibility score :

[0077]

[0078] The present invention hopes that the greater the contribution of the concept to the model decision, the stronger its comprehensibility. Contribution The classifier weights calculate:

[0079]

[0080] The loss of concept comprehensibility after the concept contribution is enhanced is:

[0081]

[0082] The total loss function of the second stage fine-tuning is:

[0083]

[0084] It should be noted that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned steps does not mean the order of execution. The order of execution of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0086] Figure 4 Schematic diagram of the concept-based self-explanatory text classification device of the present invention. Figure 4 As shown, in a second embodiment of the present invention, a self-explanatory text classification device 10 is provided, comprising:

[0087] The classification module 11 is used to classify the target text; it includes: encoding the target text to obtain encoding features; constructing a concept extractor model based on a competitive attention mechanism, taking the encoding features as input, extracting concept features, and obtaining the concept attention score of the target text; obtaining the concept strength feature of the target text based on the concept attention score; constructing a text classifier model, and classifying the target text through the text classifier model based on the concept strength feature.

[0088] The classification module 11 uses the loss function :

[0089]

[0090] The fine-tuning module 12 is used to fine-tune the parameters of the classification model; including: generating concept meaning based on the concept attention score, extracting concept features of the new target text based on the concept meaning, calculating the concept comprehensibility loss, and fine-tuning the model parameters of the concept extractor model based on the concept comprehensibility loss; and fine-tuning the model parameters of the text classifier model based on the classification result of the new target text.

[0091] The loss function used by this fine-tuning module :

[0092]

[0093] In a third embodiment of the present invention, a computer-readable storage medium is provided. If the functionality of the concept-based self-explanatory text classification device of the present invention is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a computer-readable storage medium, includes instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. Therefore, in a third embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program for a large language model multilingual processing method based on multilingual representation alignment. It should be understood that the computer-readable storage medium in this embodiment of the present invention can be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0094] Figure 5 Schematic diagram of an electronic device of the present invention. Figure 5As shown, in the fourth embodiment of the present invention, an electronic device 100 is proposed, including the concept-based self-explanatory text classification device as described above. Those skilled in the art will understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software functional module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any specific form of combination of hardware and software.

[0095] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0096] The electronic device of the present invention may be any device with data processing capabilities, such as a computer or other device. Device embodiments may be implemented through software, hardware, or a combination of software and hardware. For example, a device implemented in software, as a logical device, is formed by a processor of any device with data processing capabilities reading corresponding computer program instructions from a non-volatile memory into memory and executing them. Figure 6 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Figure 6 As shown in the figure, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the concept-based self-explanatory text classification device of the present invention is located. Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0097] When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0098] Compared to existing technologies, this invention implements a process-interpretable unsupervised text classification method. It automatically extracts semantic concepts using a competitive attention mechanism. An evaluation framework leverages the power of a large language model to assess the comprehensibility of the extracted concepts. Feedback from the large language model is then used to iteratively fine-tune the network to ensure that the extracted concepts are easily understood by humans.

[0099] The above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention. The scope of patent protection of the present invention should be defined by the claims.

Claims

1. A concept-based self-explanatory text classification method, characterized in that: include: Encode the target text and obtain encoding features; A concept extractor model based on a competitive attention mechanism is constructed. The encoding feature is used as input to extract concept features and obtain the concept attention score of the target text. The concept strength feature of the target text is obtained based on the concept attention score. A text classifier model is constructed, and the target text is classified by the text classifier model based on the concept strength feature.

2. The self-explanatory text classification method according to claim 1, wherein: Also includes: Generate concept meaning based on the concept attention score, extract concept features from the new target text based on the concept meaning, calculate the concept comprehensibility loss, and fine-tune the model parameters of the concept extractor model based on the concept comprehensibility loss; The model parameters of the text classifier model are fine-tuned based on the classification result of the new target text.

3. The self-explanatory text classification method according to claim 2, characterized in that: The loss function for classifying the target text : is the classification loss, , is the sample category label, is the prediction result of the text classifier model; is the concept consistency loss, , is the number of concepts, and Different word pair concepts for the target text The concept features, k represents the concept The top k samples with the highest concept attention scores; is the concept diversity loss, , 、 is the average feature, , A relative concept Different concepts of and is the balance parameter.

4. The self-explanatory text classification method according to claim 3, wherein: Loss function for fine-tuning model parameters for the concept extractor model and the text classifier model : is the loss of conceptual comprehensibility, , For concept The contribution of the model decision, , For concept The conceptual comprehensibility score of , is the concept attention score, Simulate activation matrices for concepts.

5. A concept-based self-explanatory text classification device, characterized in that: include: The classification module is used to classify the target text; it includes: encoding the target text to obtain encoding features; constructing a concept extractor model based on a competitive attention mechanism, taking the encoding features as input, extracting concept features, and obtaining the concept attention score of the target text; obtaining the concept strength features of the target text based on the concept attention score; constructing a text classifier model, and classifying the target text through the text classifier model based on the concept strength features.

6. The self-explanatory text classification device according to claim 5, characterized in that: Also includes: A fine-tuning module is used to fine-tune the parameters of the classification model; it includes: generating concept meaning based on the concept attention score, extracting concept features of the new target text based on the concept meaning, calculating the concept comprehensibility loss, and fine-tuning the model parameters of the concept extractor model based on the concept comprehensibility loss; and fine-tuning the model parameters of the text classifier model based on the classification result of the new target text.

7. The self-explanatory text classification device according to claim 6, characterized in that: The classification module uses the loss function : is the classification loss, , is the sample category label, is the prediction result of the text classifier model; is the concept consistency loss, , is the number of concepts, and Different word pair concepts for the target text The concept features, k represents the concept The top k samples with the highest concept attention scores; is the concept diversity loss, , 、 is the average feature, , A relative concept Different concepts of and is the balance parameter.

8. The self-explanatory text classification device according to claim 7, characterized in that: The loss function used by this fine-tuning module : is the loss of conceptual comprehensibility, , For concept The contribution of the model decision, , For concept The conceptual comprehensibility score of , is the concept attention score, Simulate activation matrices for concepts.

9. An electronic device comprising the concept-based self-explanatory text classification device according to any one of claims 5 to 8.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer-executable instructions are executed, the concept-based self-explanatory text classification method according to any one of claims 1 to 4 is implemented.

Images