A method for evaluating subjective test answers based on a large language model

Abstract

The present invention provides a method for evaluating subjective test answering based on a large language model, which relates to the technical field of natural language data processing. The method comprises: collecting answers to subjective test questions, and performing image conversion operations on the answers to the subjective test questions to obtain a black-and-white image having only black foreground information and white background information; performing a preprocessing operation on the black-and-white image to obtain a character string, and treating the character string as a structured text; performing a text error correction operation on the structured text to obtain a correct text; and evaluating the correct text according to the final large language model to obtain a final score. The method reduces the consumption of human resources, improves the efficiency and accuracy of marking, and reduces the impact of human subjective factors.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language data processing, and in particular to a subjective test question answering evaluation method based on a large language model. Background Art

[0002] At present, with the rapid development of the Internet industry, the use of the Internet in modern education is becoming more and more common. Among them, the situations where students do exercises online and teachers mark answers online have become more and more common.

[0003] However, in practice, it is found that more and more students are doing exercises online, which has led to a surge in the number of answer sheets, resulting in fewer teachers to grade the answer sheets, thus affecting the overall learning efficiency of students. In addition, in the traditional subjective open-ended test scoring process, it is usually necessary for educational experts in professional fields to perform manual scoring, which is time-consuming and easily affected by subjective factors. Therefore, it is very necessary to design a subjective test answer evaluation method based on a large language model. Summary of the invention

[0004] The purpose of the present invention is to provide a subjective test answer evaluation method based on a large language model to reduce the consumption of human resources, improve the efficiency and accuracy of marking, and reduce the impact of human subjective factors.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for evaluating subjective test answering based on a large language model comprises the following steps:

[0007] Collect answers to subjective test questions, and perform image conversion operations on the answers to the subjective test questions to obtain a black-and-white image with only black foreground information and white background information;

[0008] Perform preprocessing operations on the black and white image to obtain a character string, and regard the character string as structured text;

[0009] Perform text error correction operations on structured text to obtain correct text;

[0010] The correct text is evaluated according to the final large language model to obtain the final score; the steps for building the final large language model include:

[0011] Collecting answers to sample test questions, and performing image conversion operations on the answers to the sample test questions to obtain a sample black-and-white image having only black foreground information and white background information;

[0012] Perform preprocessing operations on the sample black-and-white image to obtain a sample character string, and regard the sample character string as a sample structured text;

[0013] Perform text error correction on the sample structured text to obtain the sample correct text, and divide the sample correct text into a verification set and a training set;

[0014] Pre-train the preset large language model through the training set to obtain a trained large language model; the preset large language model is the ChatGLM model;

[0015] Evaluate the validation set based on the trained large language model to obtain a training score;

[0016] The trained large language model is fine-tuned according to the training score to obtain the final large language model.

[0017] Optionally, answers to subjective test questions are collected, and text recognition is performed on the answers to the subjective test questions to obtain recognized texts. The specific steps include:

[0018] Capture images of answers to subjective test questions;

[0019] Compare the pixel values ​​of all pixels in the image with the preset grayscale threshold respectively;

[0020] According to the comparison result, the image is converted into recognized text having only black foreground information and white background information.

[0021] Optionally, preprocessing is performed on the recognized text to obtain structured text, and the specific steps include:

[0022] Perform denoising on the recognized text to obtain denoised text;

[0023] Performing tilt correction on the denoised text to obtain horizontal text;

[0024] Segment the foreground information and background information in the horizontal text, and identify the text features in the segmented foreground information;

[0025] The text features are compared with the database, a character string is output according to the comparison result, and the character string is regarded as structured text; the database has the text features of all the text.

[0026] Optionally, a text error correction operation is performed on the structured text to obtain a correct text, and the specific steps include:

[0027] All the characters in the structured text are represented by fine-grained information, and any character is selected as the target character, and the unselected characters are regarded as the comparison characters, and the fine-grained difference between the target character and all the comparison characters is calculated respectively; the fine-grained information is: the sequence information of the character structure and the character strokes are arranged in an orderly manner;

[0028] All the characters in the structured text are represented by phonetic features, and the differences between the target characters and all the comparison characters are calculated; the phonetic features are: sequence information that arranges the initial consonants, finals and tones in an orderly manner;

[0029] Calculate the overlap area ratio between the target text and all the comparison texts respectively;

[0030] The context semantics are analyzed through the BERT model, and the erroneous words in the structured text are replaced according to the weights of the analysis results and the proportion of fine-grained differences, phonetic feature differences and overlapping area ratios to obtain the correct text.

[0031] Optionally, pre-training a preset large language model using a training set to obtain a trained large language model includes the following specific steps:

[0032] The book corpus is concatenated, and the concatenated corpus is divided into multiple blocks of fixed length, and the blocks are trained in batches through autoregression using the ChatGLM model;

[0033] The ChatGLM model is trained through the output of a multi-task human-command question-answering dataset; the human-command question-answering dataset includes: one-question-one-answer training and multi-round question-answering training.

[0034] Optionally, the blocks are trained in batches by autoregression using the ChatGLM model, specifically by: training the blocks using the loss function and using the latent output of the current training batch as part of the input of the next training batch;

[0035] The loss function is calculated as:

[0036] Loss=(1-∈)*NLL+∈*Smoothed;

[0037] Among them, Loss is the loss function, NLL is the negative log-likelihood loss, Smoothed is the smooth loss, and ∈ is the weight of the smooth loss;

[0038] The calculation formula for negative log-likelihood loss is:

[0039]

[0040] The calculation formula of smoothing loss is:

[0041] Where N is the total output of the ChatGLM model, Y i is the true label of the i-th output of the ChatGLM model, NLL i is the negative log-likelihood loss of the i-th output of the ChatGLM model, L i is the ith output of the ChatGLM model.

[0042] Optionally, the validation set is evaluated according to the trained large language model to obtain a training score. The specific steps include:

[0043] Perform multiple summarization and keyword extraction on the validation set;

[0044] Calculate the similarity between all abstracts and keywords and the validation set respectively;

[0045] All abstracts and keywords were scored separately according to their similarity.

[0046] Optionally, the trained large language model is fine-tuned according to the training score to obtain a final large language model. The specific steps include:

[0047] Arrange the scores of abstracts and keywords in descending order;

[0048] Filter out abstracts and keywords with scores below 60 points;

[0049] The filtered abstracts and keywords are re-input into the trained large language model with different weights according to their scores for secondary training to obtain the final large language model.

[0050] The present invention discloses the following technical effects: the method for evaluating subjective test answering based on a large language model provided by the present invention comprises: collecting answers to subjective test questions, and performing image conversion operations on the answers to the subjective test questions to obtain a black-and-white image having only black foreground information and white background information; performing preprocessing operations on the black-and-white image to obtain a character string, and treating the character string as a structured text; performing text error correction operations on the structured text to obtain a correct text; and evaluating the correct text according to the final large language model to obtain a final score. The method reduces the consumption of human resources, improves the efficiency and accuracy of marking, and reduces the impact of human subjective factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 This is a flow chart of a method for evaluating subjective test answering based on a large language model according to an embodiment of the present invention;

[0053] Figure 2 is a flow chart of a text recognition method according to an embodiment of the present invention;

[0054] Figure 3 is a flow chart of a preprocessing method according to an embodiment of the present invention;

[0055] Figure 4 A flowchart of a text error correction method according to an embodiment of the present invention;

[0056] Figure 5 This is an example diagram of fine-grained text in an embodiment of the present invention;

[0057] Figure 6 The figure is a flow chart of the final large language model construction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 As shown, the embodiment of the present invention provides a method for evaluating subjective test answering based on a large language model, comprising the following steps:

[0061] Step 100: Collect answers to subjective test questions, and perform image conversion operations on the answers to the subjective test questions to obtain a black-and-white image having only black foreground information and white background information;

[0062] Specifically, Figure 2 As shown, the specific steps of image conversion include:

[0063] Step 101: Collect images of answers to subjective test questions;

[0064] Step 102: Compare the pixel values ​​of all pixels in the image with a preset grayscale threshold; the preset grayscale threshold is 145;

[0065] Step 103: According to the comparison result, the image is converted into a recognized text with only black foreground information and white background information. When the pixel value of a pixel is greater than 145, the pixel is converted to black; when the pixel value of a pixel is less than or equal to 145, the pixel is converted to white, so that the text information is clearer.

[0066] Step 200: preprocessing the black and white image to obtain a character string, and treating the character string as structured text;

[0067] Specifically, Figure 3 As shown, the specific steps of the preprocessing operation include:

[0068] Step 201: De-noising the black-and-white image to obtain the denoised text; wherein the denoising adopts the principal component analysis method, firstly, the black-and-white image data is standardized so that the mean of its features is 0 and the variance is 1; secondly, the covariance matrix of the standardized data is calculated, and the eigenvalue decomposition of the covariance matrix is ​​performed to obtain the eigenvalues ​​and eigenvectors; thirdly, the principal component eigenvectors are selected by the cumulative variance explanation rate; finally, the standardized data is multiplied by the principal component eigenvectors to obtain the denoised text.

[0069] Step 202: performing tilt correction processing on the noise-reduced text to obtain horizontal text;

[0070] Step 203: segmenting the foreground information and the background information in the horizontal text, and identifying the text features in the segmented foreground information;

[0071] More specifically, the foreground information and background information are segmented by the Canny edge detection method, and the contour features, composition structure, character spacing, font style, stroke composition and stroke connection points of the handwritten text in the segmented foreground information are identified by a convolutional neural network.

[0072] Step 204: compare the text features with the database, output a character string based on the comparison result, and regard the character string as structured text; the specific steps are: separate the handwritten text according to the character spacing to obtain separate handwritten text, and then determine whether the handwritten text is a connected character according to the font style, and then compare the outline features, composition structure, stroke composition and stroke connection points of the handwritten text with the outline features, composition structure, stroke composition and stroke connection points of the standard text stored in the database, select the closest standard text, and compose them into a character string for output.

[0073] Step 300: performing text error correction operations on the structured text to obtain a correct text;

[0074] Specifically, Figure 4 As shown, the specific steps of the text error correction operation include:

[0075] Step 301: All characters in the structured text are represented by fine-grained representation, any character is selected as a target character, and all unselected characters are regarded as comparison characters, and the fine-grained difference between the target character and all comparison characters is calculated respectively;

[0076] More specifically, the fine-grained information is the sequence information obtained by arranging the text structure and strokes in an orderly manner. The specific representation method is as follows Figure 5 Taking the character '贫' as an example, first perform a first-level split ① on the character '贫' according to the text structure to obtain the characters '分' and '贝'. Then, perform a second-level split ② on the characters '分' and '贝' respectively, and then perform a third-level split ③ until no further split is possible, obtaining the basic text strokes. Finally, arrange the text structure and strokes in the splitting process in an orderly manner according to the splitting tree to obtain the fine-grained information of the specific text.

[0077] Step 302: Represent all the characters in the structured text through phonetic features, and calculate the differences in phonetic features between the target character and all the comparison characters respectively;

[0078] More specifically, the phonetic feature is the sequence information obtained by arranging the initials, finals, and tones in an orderly manner. The specific representation method is as follows: First, encode the initials of the specific character according to the initial sequence. Then, classify the finals of the specific character according to the categories of single finals, compound finals, and nasal finals, and encode them according to the single-final sequence, compound-final sequence, or nasal-final sequence. Then, encode the tones of the specific character. Finally, arrange the initial encoding, final encoding, and tone encoding in an orderly manner to obtain the phonetic features of the specific character.

[0079] More specifically, the initial sequence is: b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, s, y, w; the single-final sequence is: a, o, e, i, u, ü, er; the compound-final sequence is: ai, ei, ao, ou, ia, ie, ua, uo, üe, iao, iou, uai, uei; the nasal-final sequence is: an, ian, uan, üan, en, in, uen, ün, ang, iang, uang, eng, ing, ueng, ong, iong. Taking the character '贫' as an example, the initial is p, the initial encoding is 2, the final is the nasal final in, the final encoding is 3.6, the tone is the second tone, and the tone encoding is 2.

[0080] Step 303: Calculate the overlapping area ratio between the target character and all the comparison characters respectively. The specific steps are as follows: Place the target character and the comparison character overlappingly, detect their overlapping area, and take the ratio of the overlapping area to the area of the target character as the overlapping area ratio.

[0081] Step 304: Analyze the context semantics through the BERT model, and replace the incorrect characters in the structured text according to the proportion weights of the analysis results, the fine-grained difference, the phonetic feature difference, and the overlapping area ratio to obtain the correct text.

[0082] Specifically, the proportion weights of the fine-grained difference, the phonetic feature difference, and the overlapping area ratio are determined by the actual context semantic analysis results. For example, when the analysis target is '突破自已' (break through oneself), after context semantic analysis, it is known that the character '已' is an incorrect character. Multiple replacement characters are selected respectively with a high proportion of fine-grained difference, a high proportion of phonetic feature difference, and a high proportion of overlapping area ratio, and the incorrect character is replaced one by one. Then, the second context semantic analysis is performed on the analysis target after replacement respectively to obtain the correct text '突破自己' (break through oneself).

[0083] Step 400: Evaluate the correct text according to the final large language model to obtain the final score.

[0084] Specifically, as Figure 6 shown, the construction steps of the final large language model include:

[0085] Step 401: Collect the sample test question answers, and perform image conversion operations on the sample test question answers to obtain a sample black-and-white image with only black foreground information and white background information;

[0086] Step 402: Perform preprocessing operations on the sample black-and-white image to obtain a sample string, and regard the sample string as a sample structured text;

[0087] Step 403: Perform text error correction operations on the sample structured text to obtain a sample correct text, and divide the sample correct text into a validation set and a training set;

[0088] Specifically, the specific steps of Step 401 to Step 403 are the same as those of Step 100 to Step 300.

[0089] Step 404: Pre-train the preset large language model through the training set to obtain a trained large language model; the preset large language model is the ChatGLM model;

[0090] Specifically, pre-training the preset large language model through the training set to obtain a trained large language model, the specific steps include:

[0091] Concatenate the book corpora, divide the concatenated corpora into multiple blocks of fixed length according to a certain block_size (block size), and perform autoregressive training in different blocks in batches through the ChatGLM model;

[0092] Perform output training on the ChatGLM model through a multi-task human instruction question-and-answer data set; the human instruction question-and-answer data set includes: one-question-one-answer training and multi-round question-and-answer training.

[0093] Furthermore, the traditional autoregressive training only performs autoregressive training on each block separately, and the training model can only learn the continuous semantic information in a single block, while the semantic relationship between different blocks will be seriously broken. The embodiment of the present invention uses the hidden output of the block of the current training batch as part of the input of the block of the next training batch to enhance the semantic coherence.

[0094] Furthermore, the autoregressive training uses a loss function for training, and the loss function is calculated as follows:

[0095] Loss=(1-∈)*NLL+∈*Smoothed;

[0096] Among them, Loss is the loss function, NLL is the negative log-likelihood loss, Smoothed is the smooth loss, ∈ is the weight of the smooth loss, and the weight of the smooth loss is 0.45; the calculation formula of the negative log-likelihood loss is:

[0097]

[0098] The calculation formula of smoothing loss is:

[0099] Where N is the total output of the ChatGLM model, Y i is the true label of the i-th output of the ChatGLM model, NLL i is the negative log-likelihood loss of the i-th output of the ChatGLM model, L i is the ith output of the ChatGLM model.

[0100] Step 405: Evaluate the validation set according to the trained large language model to obtain a training score. The specific steps include:

[0101] Perform multiple summarization and keyword extraction on the validation set to obtain multiple summaries and keywords;

[0102] Calculate the similarity between all abstracts and keywords and the validation set respectively;

[0103] All abstracts and keywords were scored separately according to their similarity.

[0104] Specifically, the similarities between all abstracts and keywords and the verification set are calculated respectively. The specific steps are: text vectorization is performed on the abstracts, keywords and verification set to obtain abstract vectors, keyword vectors and verification vectors, and the cosine similarities between the abstract vectors and keyword vectors and the verification vectors are calculated respectively, and are regarded as the similarities between the abstracts and keywords and the verification set respectively.

[0105] Specifically, the scoring criteria for scoring all abstracts and keywords based on similarity are: the full score is 100 points, the similarity value range is 0 to 1, the precision is two decimal places, and the score of the abstract and keyword is 100 times the similarity. For example, if the similarity between the abstract and the validation set is 0.55, the score of the abstract is 55 points.

[0106] Step 406: fine-tune the trained large language model according to the training score to obtain the final large language model. The specific steps include:

[0107] Arrange the scores of abstracts and keywords in descending order;

[0108] Filter out abstracts and keywords with scores below 60 points;

[0109] The filtered abstracts and keywords are re-input into the trained large language model with different weights according to their scores for secondary training to obtain the final large language model.

[0110] Specifically, the higher the scores of the abstract and keywords, the greater the input weight they occupy during the secondary training.

[0111] The beneficial effects of the present invention are as follows:

[0112] 1) Reduce the consumption of human resources and improve the efficiency and accuracy of marking;

[0113] 2) Reduce the impact of subjective factors on marking;

[0114] 3) Improved the recognition accuracy of large language models for subjective test questions.

[0115] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0116] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A subjective test question answering evaluation method based on a large language model, characterized in that: The steps include: Collect answers to subjective test questions, and perform image conversion operations on the answers to the subjective test questions to obtain a black-and-white image having only black foreground information and white background information; Performing a preprocessing operation on the black-and-white image to obtain a character string, and considering the character string as structured text; Performing text error correction operations on the structured text to obtain correct text; Evaluate the correct text according to the final large language model to obtain a final score; The steps of constructing the final large language model include: Collecting answers to sample test questions, and performing the image conversion operation on the answers to the sample test questions to obtain a sample black-and-white image having only black foreground information and white background information; Performing the preprocessing operation on the sample black-and-white image to obtain a sample character string, and treating the sample character string as a sample structured text; Performing the text error correction operation on the sample structured text to obtain a sample correct text, and dividing the sample correct text into a verification set and a training set; Pre-training a preset large language model through the training set to obtain a trained large language model; the preset large language model is a ChatGLM model; pre-training the preset large language model through the training set to obtain a trained large language model includes: The book corpus is concatenated, and the concatenated corpus is divided into multiple blocks according to a fixed length, and the blocks are trained for autoregression in batches through the ChatGLM model; the specific steps are: the blocks are trained through a loss function, and the hidden output of the current training batch is used as part of the input of the next training batch; the calculation formula of the loss function is: Loss = (1-∈)*NLL+∈*Smoothed; wherein Loss is the loss function, NLL is the negative log-likelihood loss, Smoothed is the smoothing loss, and ∈ is the weight of the smoothing loss; the calculation formula of the negative log-likelihood loss is: The calculation formula of the smoothing loss is: Where N is the total output of the ChatGLM model, Y i is the true label of the i-th output of the ChatGLM model, NLL i is the negative log-likelihood loss of the i-th output of the ChatGLM model, L i is the i-th output of the ChatGLM model; The ChatGLM model is output trained through a multi-task human instruction question-answering dataset; the human instruction question-answering dataset includes: one-question-one-answer training and multi-round question-answering training; Evaluate the validation set according to the trained large language model to obtain a training score; The trained large language model is fine-tuned according to the training score to obtain a final large language model.

2. The subjective test question answering evaluation method based on a large language model according to claim 1, characterized in that: Collect answers to subjective test questions, and perform image conversion operations on the answers to the subjective test questions to obtain a black-and-white image having only black foreground information and white background information. The specific steps include: Capturing images of answers to the subjective test questions; Comparing the pixel values ​​of all pixels in the image with the preset grayscale thresholds respectively; The image is converted into a black-and-white image having only black foreground information and white background information according to the comparison result.

3. The subjective test question answering evaluation method based on a large language model according to claim 2 is characterized in that: The black-and-white image is preprocessed to obtain a character string, and the character string is regarded as a structured text. The specific steps include: Performing denoising on the black-and-white image to obtain denoised text; Performing tilt correction processing on the noise-reduced text to obtain horizontal text; Segmenting the foreground information and the background information in the horizontal text, and identifying text features in the segmented foreground information; The text features are compared with a database, the character string is output according to the comparison result, and the character string is regarded as the structured text; the database has the text features of all the characters.

4. The subjective test question answering evaluation method based on a large language model according to claim 1, characterized in that: Performing text error correction on the structured text to obtain a correct text, the specific steps include: All characters in the structured text are represented by fine-grainedness, any character is selected as a target character, and all unselected characters are regarded as comparison characters, and the fine-grained difference between the target character and all the comparison characters is calculated respectively; the fine-grainedness is: sequence information of orderly arranging character structure and character strokes; All characters in the structured text are represented by phonetic features, and the differences between the phonetic features of the target character and all the comparison characters are calculated respectively; the phonetic features are: sequence information of orderly arranging initials, finals and tones; Calculating respectively the overlapping area ratios of the target text and all the comparison texts; The context semantics are analyzed by the BERT model, and the erroneous text in the structured text is replaced according to the analysis result and the weight of the proportion of the fine-grained difference, the phonetic feature difference and the overlapping area ratio to obtain the correct text.

5. The subjective test question answering evaluation method based on a large language model according to claim 1, characterized in that: The validation set is evaluated according to the trained large language model to obtain a training score, and the specific steps include: Performing multiple summarization and keyword extraction on the validation set; Calculating the similarity between all the abstracts and the keywords and the verification set respectively; All the abstracts and the keywords are scored respectively according to the similarities.

6. The subjective test question answering evaluation method based on a large language model according to claim 5, characterized in that: Fine-tuning the trained large language model according to the training score to obtain a final large language model, specifically comprising: Arrange the scores of the abstract and the keywords in descending order; Filter out the abstracts and keywords with scores lower than 60 points; The filtered abstracts and keywords are re-input into the trained large language model with different weights according to the scores for secondary training to obtain the final large language model.

Images