A problem sustainability evaluation system based on supervised contrastive learning
By adopting a system based on supervised contrast learning in the field of natural language processing, the problem of sample distinction and positive sample construction difficulties in QCE tasks is solved, and better prediction effect and robustness are achieved.
Patent Information
- Application Number
- CN202210765094.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-01
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2042-07-01
AI Technical Summary
In the field of natural language processing, problem sustainability evaluation (QCE) tasks are difficult to effectively distinguish similar samples as independent tasks, and it is difficult to construct positive samples of natural language data.
A system based on supervised contrast learning is adopted, including a preliminary feature extraction module, a main computing module, an interlocutor feature extraction module, a feature fusion module and a linear classification layer. The system is optimized through supervised contrast learning methods to reduce contrast losses and improve prediction effect.
It can better distinguish similar samples and improve the prediction effect of problem sustainability evaluation. No other methods are needed to construct positive samples, and it has good robustness and performance advantages.
Smart Images
Figure CN115129844B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and more particularly to a problem sustainability evaluation system based on supervised contrastive learning. Background Art
[0002] The Question Continuity Evaluation (QCE) task is a branch of the Dialog Act Prediction (DAP) task in the field of natural language processing. It aims to predict whether a specific question needs to be continued in a question-answering dialogue around a specific background text. QCE is used to guide the questioner to ask meaningful questions. The QCE task has a certain role in the automated evaluation of teaching, because in teaching, teachers need to guide students' learning process through dialogue, which is similar to the meaning of QCE. In this task, questions are divided into three categories: "follow up" (or "Y"), which means it is necessary to continue asking questions; "maybe follow up" (or "M"), which means that you can continue asking questions or change the topic of questions; "don't follow up" (or "N"), which means that there is no need to continue asking questions and the question direction should be changed. The QCE task is very important for the evaluation of the teaching process, but unfortunately, it has always been regarded as an auxiliary task of the machine reading comprehension task and has not received due attention.
[0003] The main purpose of contrastive learning is to make the features of samples of the same type close to each other and the features of samples of different types away from each other. In the unsupervised field, positive samples are defined as samples obtained by data enhancement from the same original sample, while negative samples are samples enhanced from different original data. However, different original samples may actually depict the same object, so in the unsupervised field, negative samples may contain the same object. In the field of natural language processing, only a small part of the work uses contrastive learning. Compared with image data, natural language data is more difficult to enhance data, and it is also more difficult to construct positive samples.
[0004] Based on this, a problem sustainability evaluation system is needed that treats the QCE task as an independent task, can better distinguish similar samples and does not need to use other methods to construct positive samples. Summary of the invention
[0005] In view of this, the present invention provides a problem sustainability evaluation system based on supervised contrastive learning, which takes the QCE task as an independent task and introduces the supervised contrastive learning method into the optimization of the system, thereby improving the prediction effect.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A problem sustainability evaluation system based on supervised contrastive learning, comprising: a preliminary feature extraction module, a main calculation module, an interlocutor feature extraction module, a feature fusion module, and a linear classification layer;
[0008] The preliminary feature extraction module extracts text features to obtain a background text feature matrix and a plurality of first dialogue text feature matrices;
[0009] The main calculation module converts the first dialogue text feature matrix into a question feature matrix and an answer feature matrix;
[0010] The interlocutor feature extraction module extracts text features based on the question feature matrix and the answer feature matrix to obtain a third dialogue text feature matrix;
[0011] The feature fusion module fuses the background text feature matrix and the third dialogue text feature matrix to obtain a comprehensive feature matrix;
[0012] The linear classification layer outputs an evaluation result based on the comprehensive feature matrix.
[0013] Preferably, the preliminary feature extraction module comprises a first encoder, an XLNet unit and a linear dimensionality reduction layer;
[0014] The first encoder extracts text features to obtain background text feature vector relationships and dialogue text feature vector relationships. The XLNet unit outputs a background text feature matrix and multiple first dialogue text feature matrices based on the background text feature vector relationships and dialogue text feature vector relationships. The linear dimensionality reduction layer reduces the dimension of the background text feature matrix and outputs it to the feature fusion module, and reduces the dimension of the multiple first dialogue text feature matrices and outputs them to the main computing module.
[0015] Preferably, the main calculation module includes a weighted summation unit and a second encoder;
[0016] The weighted summation unit weights and sums the feature vectors in the first dialogue text feature matrix to obtain a dialogue text feature vector, and a plurality of the dialogue text feature vectors form a second dialogue text feature matrix; the second encoder divides the second dialogue text feature matrix into a question feature matrix and an answer feature matrix and outputs the matrix to the interlocutor feature extraction module.
[0017] Preferably, the interlocutor feature extraction module includes a question feature extraction unit, an answer feature extraction unit and a feature fusion unit;
[0018] The question feature extraction unit extracts feature information based on the question feature matrix to obtain a second question feature matrix, the answer feature extraction unit extracts feature information based on the answer feature matrix to obtain a second answer feature matrix, and the feature fusion unit fuses the second question feature matrix and the second answer feature matrix into a third dialogue text feature matrix.
[0019] Preferably, the problem feature extraction unit includes a third encoder, a first LSTM unit and a first secondary feature fusion unit; the third encoder extracts long-distance dependency and semantic information based on the problem feature matrix to obtain a semantic problem feature matrix, the first LSTM unit extracts sequence information based on the problem feature matrix to obtain a sequential problem feature matrix, and the first secondary feature fusion unit fuses the semantic problem feature matrix and the sequential problem feature matrix into a second problem feature matrix;
[0020] The answer feature extraction unit includes a fourth encoder, a second LSTM unit and a second secondary feature fusion unit; the fourth encoder extracts long-distance dependencies and semantic information based on the answer feature matrix to obtain a semantic answer feature matrix, the second LSTM unit extracts sequence information based on the answer feature matrix to obtain a sequential answer feature matrix, and the secondary feature fusion unit fuses the semantic answer feature matrix and the sequential answer feature matrix into a second answer feature matrix.
[0021] Preferably, an optimization module is also included, and the optimization module optimizes the system based on a supervised contrast learning method to reduce the contrast loss, and the contrast loss function is:
[0022]
[0023] Where L is the contrast loss, i is the sample currently used for system training, I is the system training sample set, p is the sample with the same category label as sample i, P(i) is the set of all samples p, a is the sample in I except sample i, A(i) is the set of all samples except sample i, z i 、z p and z a represent the feature vectors of samples i, p and a respectively, and τ is a hyperparameter used to control the strength of contrastive learning.
[0024] Preferably, the feature fusion module, the feature fusion unit, the first secondary feature fusion unit and the second secondary feature fusion unit are specifically:
[0025] W A =AW1+b1
[0026] W B =BW2+b2
[0027] W = softmax([W A ; W B ])
[0028] F=W[0]W A +W[1]W B
[0029] In the formula, A, B∈R n×d are two feature matrices to be fused, n is the length of the feature matrix, d is the dimension of the feature matrix, W1,W2∈R d×1 and b1,b2∈R n×1 is a learnable parameter, [W A ; W B ]∈R d×2 W A and W B The splicing of W A represents the non-normalized fusion weight of matrix A, W B represents the non-normalized fusion weight of matrix B, W represents the normalized fusion weight, F represents the fused matrix, W[0] represents the 0th column of W, that is, the fusion weight of matrix A, and W[1] represents the 1st column of W, that is, the fusion weight of matrix B.
[0030] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a problem sustainability evaluation system based on supervised contrastive learning, which takes the QCE task as an independent task and introduces a supervised contrastive learning method to optimize the system. It can better distinguish similar samples and no longer need to use other methods (such as data enhancement, etc.) to construct positive samples. It solves the problem of the difficulty in constructing positive samples for natural language data to a certain extent; it can well extract the meaning of the text and the relationship features between texts, has superior performance, and has good robustness to data shortage situations. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0032] Figure 1 is a schematic diagram of the evaluation system of the present invention;
[0033] Figure 2 Schematic diagram of the interlocutor feature extraction module of the present invention. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] Embodiment 1:
[0036] The embodiment of the present invention discloses a problem sustainability evaluation system based on supervised contrastive learning, such as Figure 1 As shown, it includes: preliminary feature extraction module, main calculation module, interlocutor feature extraction module, feature fusion module, linear classification layer;
[0037] The preliminary feature extraction module includes a first encoder, an XLNet unit and a linear dimensionality reduction layer; the first encoder extracts text features to obtain background text feature vector relationships and dialogue text feature vector relationships, the XLNet unit outputs a background text feature matrix and multiple first dialogue text feature matrices based on the background text feature vector relationships and the dialogue text feature vector relationships, the linear dimensionality reduction layer reduces the dimension of the background text feature matrix and outputs it to the feature fusion module, and reduces the dimension of multiple first dialogue text feature matrices and outputs them to the main computing module.
[0038] The main calculation module includes a weighted summation unit and a second encoder; the weighted summation unit weights and sums the feature vectors in the first dialogue text feature matrix to obtain a dialogue text feature vector, and multiple dialogue text feature vectors form a second dialogue text feature matrix; the second encoder divides the second dialogue text feature matrix into a question feature matrix and an answer feature matrix and outputs them to the interlocutor feature extraction module.
[0039] The interlocutor feature extraction module extracts text features based on the question feature matrix and the answer feature matrix to obtain a third dialogue text feature matrix;
[0040] The feature fusion module fuses the background text feature matrix and the third dialogue text feature matrix to obtain a comprehensive feature matrix.
[0041] The linear classification layer outputs the evaluation results based on the comprehensive feature matrix:
[0042] is the continuability label of the i-th question.
[0043] Embodiment 2:
[0044] The embodiment of the present invention discloses a problem sustainability evaluation system based on supervised contrastive learning, including: a preliminary feature extraction module, a main calculation module, an interlocutor feature extraction module, a feature fusion module, a linear classification layer, and an optimization module;
[0045] The preliminary feature extraction module includes a first encoder, an XLNet unit and a linear dimensionality reduction layer; the first encoder extracts text features to obtain background text feature vector relationships and dialogue text feature vector relationships, the XLNet unit outputs a background text feature matrix and multiple first dialogue text feature matrices based on the background text feature vector relationships and the dialogue text feature vector relationships, the linear dimensionality reduction layer reduces the dimension of the background text feature matrix and outputs it to the feature fusion module, and reduces the dimension of multiple first dialogue text feature matrices and outputs them to the main computing module.
[0046] The main calculation module includes a weighted summation unit and a second encoder; the weighted summation unit weights and sums the feature vectors in the first dialogue text feature matrix to obtain a dialogue text feature vector, and multiple dialogue text feature vectors form a second dialogue text feature matrix; the second encoder divides the second dialogue text feature matrix into a question feature matrix and an answer feature matrix and outputs them to the interlocutor feature extraction module.
[0047] The interlocutor feature extraction module includes a question feature extraction unit, an answer feature extraction unit and a feature fusion unit. Figure 2 As shown; the question feature extraction unit extracts feature information based on the question feature matrix to obtain a second question feature matrix, the answer feature extraction unit extracts feature information based on the answer feature matrix to obtain a second answer feature matrix, and the feature fusion unit fuses the second question feature matrix and the second answer feature matrix into a third dialogue text feature matrix;
[0048] The question feature extraction unit includes a third encoder, a first LSTM unit, and a first secondary feature fusion unit; the third encoder extracts long-distance dependency and semantic information based on the question feature matrix to obtain a semantic question feature matrix, the first LSTM unit extracts sequence information based on the question feature matrix to obtain a sequential question feature matrix, and the first secondary feature fusion unit fuses the semantic question feature matrix and the sequential question feature matrix into a second question feature matrix;
[0049] The answer feature extraction unit includes a fourth encoder, a second LSTM unit and a second secondary feature fusion unit; the fourth encoder extracts long-distance dependencies and semantic information based on the answer feature matrix to obtain a semantic answer feature matrix, the second LSTM unit extracts sequence information based on the answer feature matrix to obtain a sequential answer feature matrix, and the secondary feature fusion unit fuses the semantic answer feature matrix and the sequential answer feature matrix into a second answer feature matrix.
[0050] The feature fusion module fuses the background text feature matrix and the third dialogue text feature matrix to obtain a comprehensive feature matrix.
[0051] The linear classification layer outputs the evaluation results based on the comprehensive feature matrix:
[0052] is the continuability label of the i-th question.
[0053] The optimization module optimizes the system based on the supervised contrast learning method to reduce the contrast loss. The contrast loss function is:
[0054]
[0055] Where L is the contrast loss, i is the sample currently used for system training, I is the system training sample set, p is the sample with the same category label as sample i, P(i) is the set of all samples p, a is the sample in I except sample i, A(i) is the set of all samples except sample i, z i 、z p and z a Represent the feature vectors of samples i, p and a respectively, τ is a hyperparameter used to control the strength of contrastive learning; in this embodiment, the optimization module is the AdaDelta optimizer, and the gradient descent method is used to optimize the system.
[0056] The feature fusion module, the feature fusion unit, the first secondary feature fusion unit and the second secondary feature fusion unit are specifically:
[0057] W A =AW1+b1
[0058] W B =BW2+b2
[0059] W = softmax([W A ; W B ])
[0060] F=W[0]W A +W[1]W B
[0061] In the formula, A, B∈R n×d are two feature matrices to be fused, n is the length of the feature matrix, d is the dimension of the feature matrix, W1,W2∈R d×1 and b1,b2∈R n×1 is a learnable parameter, [W A ; W B ]∈R d×2 W A and W B The splicing of W A represents the non-normalized fusion weight of matrix A, W Brepresents the non-normalized fusion weight of matrix B, W represents the normalized fusion weight, F represents the fused matrix, W[0] represents the 0th column of W, that is, the fusion weight of matrix A, and W[1] represents the 1st column of W, that is, the fusion weight of matrix B.
[0062] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0063] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A problem sustainability evaluation system based on supervised contrastive learning, characterized in that: include: Preliminary feature extraction module, main calculation module, interlocutor feature extraction module, feature fusion module, linear classification layer; The preliminary feature extraction module extracts text features to obtain a background text feature matrix and a plurality of first dialogue text feature matrices; The main calculation module converts the first dialogue text feature matrix into a question feature matrix and an answer feature matrix; The interlocutor feature extraction module extracts text features based on the question feature matrix and the answer feature matrix to obtain a third dialogue text feature matrix; The feature fusion module fuses the background text feature matrix and the third dialogue text feature matrix to obtain a comprehensive feature matrix; The linear classification layer outputs an evaluation result based on the comprehensive feature matrix; The preliminary feature extraction module includes a first encoder, an XLNet unit, and a linear dimensionality reduction layer; The first encoder extracts text features to obtain a background text feature vector relationship and a dialogue text feature vector relationship, the XLNet unit outputs a background text feature matrix and a plurality of first dialogue text feature matrices based on the background text feature vector relationship and the dialogue text feature vector relationship, the linear dimensionality reduction layer reduces the dimension of the background text feature matrix and outputs it to the feature fusion module, and reduces the dimension of the plurality of first dialogue text feature matrices and outputs them to the main computing module; The main calculation module includes a weighted summation unit and a second encoder; The weighted summing unit weights and sums the feature vectors in the first dialogue text feature matrix to obtain a dialogue text feature vector, and a plurality of the dialogue text feature vectors form a second dialogue text feature matrix; the second encoder divides the second dialogue text feature matrix into a question feature matrix and an answer feature matrix and outputs the matrix to the interlocutor feature extraction module; The interlocutor feature extraction module includes a question feature extraction unit, an answer feature extraction unit and a feature fusion unit; The question feature extraction unit extracts feature information based on the question feature matrix to obtain a second question feature matrix, the answer feature extraction unit extracts feature information based on the answer feature matrix to obtain a second answer feature matrix, and the feature fusion unit fuses the second question feature matrix and the second answer feature matrix into a third dialogue text feature matrix.
2. The problem sustainability evaluation system based on supervised contrastive learning according to claim 1, characterized in that: The problem feature extraction unit includes a third encoder, a first LSTM unit and a first secondary feature fusion unit; the third encoder extracts long-distance dependency and semantic information based on the problem feature matrix to obtain a semantic problem feature matrix, the first LSTM unit extracts sequence information based on the problem feature matrix to obtain a sequential problem feature matrix, and the first secondary feature fusion unit fuses the semantic problem feature matrix and the sequential problem feature matrix into a second problem feature matrix; The answer feature extraction unit includes a fourth encoder, a second LSTM unit and a second secondary feature fusion unit; the fourth encoder extracts long-distance dependencies and semantic information based on the answer feature matrix to obtain a semantic answer feature matrix, the second LSTM unit extracts sequence information based on the answer feature matrix to obtain a sequential answer feature matrix, and the secondary feature fusion unit fuses the semantic answer feature matrix and the sequential answer feature matrix into a second answer feature matrix.
3. The problem sustainability evaluation system based on supervised contrastive learning according to claim 1, characterized in that: It also includes an optimization module, which optimizes the system based on a supervised contrastive learning method to reduce contrast loss. The contrast loss function is: Where L is the contrast loss, i is the sample currently used for system training, I is the system training sample set, p is the sample with the same category label as sample i, P(i) is the set of all samples p, a is the sample in I except sample i, A(i) is the set of all samples except sample i, z i 、z p and z a represent the feature vectors of samples i, p and a respectively, and τ is a hyperparameter used to control the strength of contrastive learning.
4. The problem sustainability evaluation system based on supervised contrastive learning according to claim 2, characterized in that: The feature fusion module, the feature fusion unit, the first secondary feature fusion unit and the second secondary feature fusion unit are specifically: IN A =AW1+b1 <h2 style=";text-align:left;direction:ltr">W<h2 style=";text-align:left;direction:ltr"> B <h2 style=";text-align:left;direction:ltr"> =BW2+b2 W=softmax([W A ;W B ]) F=W[0]W A +W[1]W B In the formula, A, B∈R n×d are two feature matrices to be fused, n is the length of the feature matrix, d is the dimension of the feature matrix, W1, W2∈R d×1 and b1,b2∈R n×1 is a learnable parameter, [W A ; W B ]∈R d×2 W A and W B The splicing of W A represents the non-normalized fusion weight of matrix A, W B represents the non-normalized fusion weight of matrix B, W represents the normalized fusion weight, F represents the fused matrix, W[0] represents the 0th column of W, that is, the fusion weight of matrix A, and W[1] represents the 1st column of W, that is, the fusion weight of matrix B.