Data processing method and device based on large language model

Abstract

The embodiment of the invention discloses a data processing method and device based on a large language model. According to the method, a to-be-scored question group is obtained, the to-be-scored question group comprises questions, answers of answering parties and reference answers, and according to the to-be-scored question group and key words, a prompt template, a false question generation prompt template and an evaluation generation prompt template are extracted, respectively constructing a keyword extraction prompt statement, a pseudo question generation prompt statement and an evaluation generation prompt statement, respectively inputting the constructed statements into a large language model to obtain corresponding keyword groups, question groups and global evaluation, determining text similarity of answers of answering parties and reference answers, and determining the similarity of the answers of the answering parties and the reference answers; and inputting the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain a score corresponding to the answer of the answering party. According to the method, the advantages of the large language model and the comprehensive scoring model are combined, the generalization generation capability of the large language model is exerted, and the vertical domain scene learning capability of the comprehensive scoring model is utilized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more particularly, to a data processing method and device based on a large language model. Background Art

[0002] In the current enterprise landscape, compliance certification and knowledge-based exams are crucial for the operations and management of medium- to large-scale enterprises. With the advent of the information age, post-certification or exam grading has shifted from manual grading to online grading. There are two main online grading methods. One is rule-based grading, which is efficient and uses consistent criteria, but suffers from poor performance on complex questions and high rule-building and maintenance costs. The other is large language model-based grading, which leverages the powerful semantic understanding capabilities of large language models but lacks in-depth domain knowledge and understanding of the specific domains within an enterprise.

[0003] Therefore, how to automatically score more accurately is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a data processing method and device based on a large language model, which combines the advantages of a large language model and a comprehensive scoring model to improve the scoring accuracy, thereby giving full play to the generalization generation capability of the large language model and making full use of the vertical scenario learning capability of the comprehensive scoring model.

[0005] In a first aspect, a data processing method based on a large language model is provided, the method comprising:

[0006] Obtaining a set of questions to be graded, wherein the set of questions to be graded includes a question, an answer from a questioner, and a reference answer;

[0007] According to the to-be-graded question group and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, corresponding keyword extraction prompt sentences, pseudo-question generation prompt sentences and evaluation generation prompt sentences are respectively constructed;

[0008] Inputting the keyword extraction prompt sentence, the pseudo-question generation prompt sentence and the evaluation generation prompt sentence into a large language model respectively to obtain corresponding keyword groups, question groups and global evaluations;

[0009] Determining the text similarity between the answer of the respondent and the reference answer;

[0010] The keyword group, the question group, the global evaluation and the text similarity are input into a pre-trained comprehensive scoring model to obtain a score corresponding to the answer of the answerer.

[0011] In a second aspect, a data processing device based on a large language model is provided, the device comprising:

[0012] An acquisition module is used to acquire a set of questions to be graded, wherein the set of questions to be graded includes a question, an answer from a questioner, and a reference answer;

[0013] A prompt module, configured to construct corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements according to the to-be-rated question group and preset keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates;

[0014] A model processing module, configured to input the keyword extraction prompt sentence, the pseudo-question generation prompt sentence, and the evaluation generation prompt sentence into a large language model to obtain corresponding keyword groups, question groups, and global evaluations;

[0015] A determination module, configured to determine the text similarity between the answer of the respondent and the reference answer;

[0016] The scoring module is used to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score of the answer of the answerer.

[0017] In a third aspect, a data processing system based on a large language model is provided, the system comprising:

[0018] A data acquisition unit is configured to acquire a set of questions to be scored, wherein the set of questions to be scored includes a question, an answer from a questioner, and a reference answer;

[0019] A keyword extraction unit is configured to construct a corresponding keyword extraction prompt sentence based on the to-be-graded question group and a preset keyword extraction prompt template, and input the keyword extraction prompt sentence into the large language model to obtain a corresponding keyword group;

[0020] a pseudo-question generation unit configured to construct a corresponding pseudo-question generation prompt sentence based on the to-be-graded question group and a preset pseudo-question generation prompt template, and input the pseudo-question generation prompt sentence into the large language model to obtain a corresponding question group;

[0021] An evaluation generation unit is configured to construct a corresponding evaluation generation prompt sentence based on the to-be-rated question group and a preset evaluation generation prompt template, and input the evaluation generation prompt sentence into a large language model to obtain a corresponding global evaluation;

[0022] a similarity determination unit, configured to determine the text similarity between the answer of the answerer and the reference answer;

[0023] The comprehensive scoring unit is configured to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain a score corresponding to the answer of the answerer.

[0024] In a fourth aspect, a data processing architecture based on a large language model is provided, the architecture comprising:

[0025] a data communication component configured to obtain a set of questions to be scored, send the set of questions to be scored to the first processing core and the third processing core, and output a score corresponding to the answer of the respondent, wherein the set of questions to be scored includes the question, the answer of the respondent, and the reference answer;

[0026] The first processing core is configured to construct and send corresponding keyword extraction prompt sentences, pseudo question generation prompt sentences, and evaluation generation prompt sentences to the second processing core based on the to-be-rated question group and the preset keyword extraction prompt template, pseudo question generation prompt template, and evaluation generation prompt template, and to determine the text similarity between the answer of the answerer and the reference answer;

[0027] The second processing core is equipped with a large language model and is configured to extract prompt sentences based on the keywords, generate prompt sentences based on the pseudo questions, and generate prompt sentences based on the evaluations, and generate corresponding keyword groups, question groups, and global evaluations;

[0028] The third processing core is equipped with a comprehensive scoring model and is configured to determine the score corresponding to the answer of the respondent based on the keyword group, the question group, the global evaluation and the text similarity.

[0029] In a fifth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect above.

[0030] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0031] In a seventh aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the method described in the first aspect above.

[0032] The technical solution of the embodiment of the present invention is to obtain a group of questions to be scored, which includes questions, answers of the answering party and reference answers. According to the group of questions to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, respectively construct a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence, respectively input the constructed sentences into the large language model, obtain the corresponding keyword group, question group and global evaluation, determine the text similarity between the answer of the answering party and the reference answer, input the keyword group, question group, global evaluation and text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the answer of the answering party. The above technical solution combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also makes full use of the vertical scenario learning ability of the comprehensive scoring model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0034] Figure 1 Schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention;

[0035] Figure 2 Flowchart of a data processing method based on a large language model according to an embodiment of the present invention;

[0036] Figure 3 A flowchart of a method for matching using a comprehensive scoring model according to an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention;

[0038] Figure 5 is a data flow diagram within a data processing system according to an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of a data processing architecture based on a large language model according to an embodiment of the present invention;

[0040] Figure 7 Schematic diagram of a data processing device based on a large language model according to an embodiment of the present invention;

[0041] Figure 8 FIG. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present application is described below based on the following embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0043] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0044] Unless the context clearly requires otherwise, words like “include”, “comprising” and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0045] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0046] Where the solutions described in this specification and in the examples involve the processing of personal information, such processing will be conducted with a legitimate basis (e.g., with the consent of the personal information subject or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of these basic functions.

[0047] Figure 1 Schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention. Figure 1 As shown, the data processing system based on the large language model includes a first user terminal 11, a second user terminal 12 and a server 13.

[0048] Among them, the first user terminal 11 and the second user terminal 12 are both universal terminals for running applications or small programs of the examination platform, and the terminals can be smart phones, tablet computers, car computers, smart wearable devices, laptops, car computers or other types of mobile communication terminals. In some possible implementations, the first user terminal 11 and the second user terminal 12 can also be dedicated terminals with corresponding applications solidified in dedicated integrated circuits. Among them, the open platform enables third-party developers to develop their own small programs based on the open platform by providing them with an application programming interface (Application Programming Interface, API). The small program can specifically be a program developed on the basis of an application within the platform and used to perform corresponding operations. The first user terminal 11 is the terminal held by the question setter, and the second user terminal 12 is the terminal held by the answerer.

[0049] The server 13 refers to a general data processing device that can provide computing or application services for the charging service provider platform. It can be a single computer, a cluster of multiple computers, or a cloud server that can flexibly adjust computing resources through cloud technology.

[0050] The server 13 exchanges information with the first user terminal 11 and the second user terminal 12 via the network to obtain the set of questions to be scored, and then processes the obtained set of questions to be scored to obtain corresponding scores.

[0051] Specifically, the first user terminal 11 obtains the question and reference answer set by the question setter based on human-computer interaction, and then sends an examination task release instruction carrying the question and reference answer to the server 13. After receiving the examination task release instruction, the server 13 releases the examination task to the second user terminal 12. The second user terminal 12 obtains the answer of the question setter based on human-computer interaction and sends it to the server 13. After receiving the answer of the question setter, the server 13 scores it in combination with the question and the reference answer. The scoring process of the server 13 is as follows: according to the question group to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, respectively construct a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence, respectively input the constructed sentences into the large language model, obtain the corresponding keyword group, question group and global evaluation, determine the text similarity between the answer setter's answer and the reference answer, and input the keyword group, question group, global evaluation and text similarity into the pre-trained comprehensive scoring model to obtain the corresponding score of the answer setter.

[0052] In a possible implementation, the first user terminal 11 and the second user terminal 12 may be the same device, and the question setter and the question answerer use different accounts to log in to the examination platform to complete the above-mentioned interaction with the server 13.

[0053] In one possible implementation, the process of scoring the answers based on the question and reference answers can be performed on the user end. For example, the test platform on the user end can be operated offline. In this case, the large language model and comprehensive scoring model deployed on the user end can be used to implement the above scoring process.

[0054] Exemplarily, after the first user terminal 11 obtains the question and reference answer set by the questioner, it sends the question and reference answer to the second user terminal 12. Then, after the second user terminal 12 obtains the answer of the respondent, it scores the answer locally based on the question and reference answer.

[0055] The method for sending the questions and reference answers to the second user terminal 12 may be to send them to the server 13 via a network, and then the server 13 sends them to the second user terminal 12. Alternatively, they may be sent directly to the second user terminal 12 via a network or a non-networking method. The non-networking method may be data transmission via a data cable, a mobile storage device, Bluetooth, a local area network, a crossover cable, or third-party transmission software.

[0056] Optionally, the first user terminal 11 and the second user terminal 12 can be the same device, that is, the user terminal. After the user terminal obtains the questions and reference answers set by the questioner, it stores them in a storage area with restricted permissions, and then obtains the answer of the answerer, and then completes the scoring locally.

[0057] The system of the embodiment of the present invention is used to obtain a group of questions to be scored, which includes questions, answers of the answering party and reference answers. According to the group of questions to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence are respectively constructed, and the constructed sentences are respectively input into the large language model to obtain the corresponding keyword group, question group and global evaluation, and the text similarity between the answer of the answering party and the reference answer is determined. The keyword group, question group, global evaluation and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the answer of the answering party. The system of this embodiment combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also makes full use of the vertical scenario learning ability of the comprehensive scoring model.

[0058] Figure 2 FIG. 1 is a flow chart of a data processing method based on a large language model according to an embodiment of the present invention. Figure 2 As shown, the data processing method based on the large language model includes the following steps:

[0059] Step S201, obtaining a set of questions to be graded.

[0060] The question set to be graded includes questions, answers from test takers and reference answers.

[0061] In one possible implementation, data corresponding to an entire test paper is obtained. The data corresponding to an entire test paper includes multiple questions and their corresponding answers and reference answers. At this time, it is necessary to first divide the data according to the questions to divide the data corresponding to the entire test paper into multiple groups of questions to be scored.

[0062] In one possible implementation, after obtaining the question and the answer, if there is no corresponding reference answer, it is necessary to search for the corresponding reference answer based on the question, or based on the question and the answer. The reference answer can be retrieved from a database or from the Internet. The database can be a database, a dedicated database of the examination platform, or an authorized database of the user, i.e., the enterprise, group, or individual using the examination platform. It should be understood that this embodiment does not limit the method for obtaining the reference answer.

[0063] Step S202 , constructing corresponding keyword extraction prompt statements, pseudo question generation prompt statements and evaluation generation prompt statements respectively according to the to-be-rated question group and the preset keyword extraction prompt template, pseudo question generation prompt template and evaluation generation prompt template.

[0064] In step S203, the keyword extraction prompt sentence, the pseudo-question generation prompt sentence and the evaluation generation prompt sentence are respectively input into a large language model to obtain corresponding keyword groups, question groups and global evaluations.

[0065] The following will explain in combination with step S202 and step S203.

[0066] The keyword extraction prompt is used to prompt the large language model to extract keywords from the set of questions to be scored. Keyword extraction requires extracting at least the keywords of the answer. During actual scoring, keyword extraction helps to extract key information from redundant information. The closer the key information is to the reference answer, or the more relevant it is to the question, the higher the corresponding score.

[0067] In a possible implementation, there are multiple keyword extraction prompt templates. Correspondingly, there are also multiple keyword extraction prompt sentences constructed based on the keyword extraction templates, which are respectively adapted to different keyword extraction requirements.

[0068] Optionally, keywords are directly extracted from the answerer's answer. The keyword extraction prompt template can be "Extract keywords from [answerer's answer to be input], extraction requirements: xxx." A keyword extraction prompt statement constructed based on this keyword extraction prompt template is used to prompt the large language model to extract keywords from the answerer's answer. In this case, the keyword group output by the large language model includes multiple keywords corresponding to the answerer's answer.

[0069] Optionally, keywords can be extracted from the answer of the respondent based on the title and / or reference answer, and the keyword extraction prompt template can be "Please extract keywords from [answer to be input] based on [title to be input and / or reference answer], extraction requirements: xxx", and the keyword extraction prompt sentence constructed based on the keyword extraction prompt template is used to prompt the large language model to extract keywords from the answer of the respondent based on the title and / or the reference answer. Using the title and / or reference answer as a reference to extract keywords from the answer of the respondent can make the keywords extracted from the answer of the respondent more accurate, effective and adapted to the professional knowledge of the field to which the question group to be scored belongs. At this time, the keyword group output by the large language model contains multiple keywords corresponding to the answer of the respondent.

[0070] Optionally, in addition to extracting keywords from the answer of the questioner, keywords from the question and / or reference answer can also be extracted. Then the keyword extraction prompt template can be "Compare and analyze [question to be input and / or reference answer to be input] with [answer to be input] to extract keywords from [question to be input and / or reference answer to be input] and [answer to be input]". Then the keyword extraction prompt sentence constructed based on the keyword extraction prompt template is used to prompt the large language model to extract keywords from the answer of the questioner and the question and / or reference answer respectively. The keyword group extracted by this extraction method contains multiple keywords corresponding to the answer of the questioner and multiple keywords corresponding to the question, or the keyword group contains multiple keywords corresponding to the answer of the questioner, multiple keywords corresponding to the question and multiple keywords corresponding to the reference answer. The keyword group extracted by this method contains more information, and scoring based on this keyword group will make the scoring more accurate.

[0071] Now let's take the scenario of "extracting keywords from the answerer's answer based on the question and reference answer" as an example of a keyword extraction prompt statement. The example is as follows:

[0072] [You are an intelligent exam assistant and now you need to help the examiner extract key knowledge points from the answers of the test takers.

[0073] The following are the specific requirements:

[0074] 1. The answers may contain redundant information or overly detailed descriptions. Please only focus on the key knowledge points.

[0075] 2. Extract core knowledge points from the answers of the test takers based on the questions, reference answers and scoring criteria.

[0076] 3. Each knowledge point should be as concise as possible, preferably within 25 Chinese characters, and avoid being too long.

[0077] 4. The extracted knowledge points should directly reflect the core content of the test questions and avoid redundant descriptions.

[0078] 5. Each knowledge point should be standardized as much as possible to ensure that it is concise and clear, and to avoid irrelevant or repeated information as much as possible.

[0079] 6. Extract at most 2-3 knowledge points. If it is not enough, please ignore it.

[0080] Specific operations:

[0081] 1. Provide test questions and reference answers. Please extract key knowledge points from the answers of the test takers.

[0082] 2. The scoring points need to be listed in concise Chinese and arranged in the order of appearance.

[0083] 3. The output format should follow standard requirements and avoid any irrelevant content.

[0084] 4. If the answer is short enough to be used as a knowledge point, just output the original answer directly.

[0085] Output format requirements:

[0086] 1. Each knowledge point is separated by a semicolon and arranged in order.

[0087] The topics are as follows:

[0088] {title}

[0089] The reference answers are as follows:

[0090] {analysis}

[0091] The answers to the questions are as follows:

[0092] {answer}】

[0093] The pseudo-question generation prompt is used to prompt the large language model to generate a pseudo-question corresponding to the respondent's answer. A pseudo-question is a question that is more suitable for the respondent's answer. By comparing the similarity between the pseudo-question and the original question, the accuracy of the respondent's answer can be determined and the situation where the answer is very similar but completely incorrect can be avoided. The higher the similarity between the pseudo-question and the original question, the higher the score of the respondent's answer.

[0094] Similarly, there are multiple pseudo-question generation prompt templates, and correspondingly, there are also multiple pseudo-question generation prompt statements constructed based on the pseudo-question generation prompt templates, which are adapted to different pseudo-question generation requirements. Specifically, pseudo-questions can be generated directly based on the answer of the answerer, or pseudo-questions can be generated based on the answer of the answerer, with reference to the question and / or the reference answer. The process is similar to the above-mentioned keyword extraction and will not be repeated here. The question group includes the pseudo-questions corresponding to the question and the answer of the answerer.

[0095] In one possible implementation, pseudo questions for the to-be-graded question set can be generated using the questions and reference answers from other similar question sets. Specifically, similar question sets are searched for in a question bank (database or network), and pseudo question generation prompts are constructed based on similar questions and similar reference answers in the similar question sets. This allows the large language model to mimic similar question sets and generate pseudo questions that are the answers of the test takers.

[0096] For example, the pseudo-question generation prompt statement may be:

[0097] You are an intelligent assistant responsible for evaluating the answers of test takers. Here are your job requirements:

[0098] Based on similar questions and similar reference answers, analyze pseudo-questions that are closer to the answer of the respondent.

[0099] Specific operation and output format requirements:

[0100] 1. Determine the question-answering logic between similar questions and similar reference answers, and use this as a sample to determine the pseudo-questions corresponding to the answers of the answerers.

[0101] 2. Questions should be as concise as possible, preferably within 25 Chinese characters, and avoid being too long.

[0102] Similar test questions are as follows:

[0103] {title sample}

[0104] Similar reference answers are as follows:

[0105] {analysis sample}

[0106] The answers to the questions are as follows:

[0107] {answer}】

[0108] The evaluation generation prompt is used to prompt the large language model to analyze the respondent's answer from multiple dimensions to generate a corresponding global evaluation. Dimensions may include logical coherence, accuracy, key points, etc. The evaluation generation prompt contains an evaluation statement within a predetermined word count threshold.

[0109] Similarly, there are multiple evaluation generation prompt templates, and correspondingly, there are also multiple evaluation generation prompt statements constructed based on the evaluation generation prompt templates, which are adapted to different evaluation generation requirements. Specifically, a global evaluation can be generated directly based on the answer of the answerer, or a global evaluation can be generated based on the answer of the answerer, with reference to the question and / or the reference answer. The process is similar to the above-mentioned keyword extraction and will not be repeated here. The evaluation generation prompt statement includes an evaluation statement within a predetermined word count threshold. The predetermined word count threshold is set or updated according to the actual situation. For example, the predetermined word count threshold can be 100, 200 or 100, etc.

[0110] For example, the evaluation generation prompt statement may be:

[0111] You are an intelligent assistant responsible for evaluating the answers of test takers. Here are your job requirements:

[0112] 1. Based on the question and reference answers, analyze the overall performance of the respondent's answer and give a clear evaluation (such as "good", "average", "poor").

[0113] 2. In the evaluation, point out the strengths (correct points) and weaknesses (incorrect points) of the respondent’s answer to help the examiner understand the key issues of the answer.

[0114] Specific operation and output format requirements:

[0115] 1. Provide questions, scoring criteria, reference answers and answers from the respondent.

[0116] 2. Extract the correct and incorrect points in the answer according to the scoring criteria, and give an overall evaluation based on these points.

[0117] 3. You can think about whether the answer is reasonable from the perspectives of logic, language fluency, and content rationality.

[0118] 4. Output format: A short description, try not to exceed 200 Chinese characters. No need to describe in points

[0119] 5. The output sample is as follows:

[0120] The overall rating of the respondent's answer is "Good." Strengths: The respondent detailed the specific violations in the SMS channel, such as misleading red envelope information and inappropriate use of brand terms, which meets the violation regulations mentioned in the scoring criteria. The respondent also mentioned that penalties would be imposed in accordance with the violation regulations and recommended self-inspection, demonstrating the platform's rigor and fairness in handling issues. Weaknesses: The respondent's answer failed to mention the cautious nature of the penalties and the multiple reviews process. The explanation for the inability to provide personal evidence was insufficient, which may affect the respondent's understanding of the reasonableness of the penalties.

[0121] The test questions are as follows:

[0122] {title}

[0123] The scoring criteria are as follows:

[0124] {analysis}

[0125] The answers to the questions are as follows:

[0126] {answer}】

[0127] Step S204: Determine the text similarity between the answer of the respondent and the reference answer.

[0128] In a possible implementation, the text similarity between the answer of the respondent and the reference answer is directly determined.

[0129] In one possible implementation, the respondent's answer and the reference answer may be truncated and modified, respectively, and then the text similarity between the rewritten respondent's answer and the reference answer may be determined. Since the lengths of the respondent's answer and the reference answer may differ significantly, some reference answers and / or respondent's answers may be complex and redundant. The respondent's answer and the reference answer may contain some garbled characters and extra characters. Therefore, before performing a plain text match, any erroneous symbol information and overly long sentences that may exist in the sentence may be appropriately truncated and rewritten to preserve the original meaning of the sentence without being affected by excessive noise.

[0130] Step S205 , inputting the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain a corresponding score for the answer of the respondent.

[0131] The comprehensive scoring model comprises a universal coding layer, a transformer encoder and a multi-layer perceptron layer, and the comprehensive scoring model is trained based on sample data of the field to which the to-be-scored question set belongs.

[0132] The universal coding layer refers to pre-trained language models designed based on the Transformer architecture in the field of natural language processing (NLP). These models have strong semantic representation capabilities and wide applicability. Examples include BERT, RoBERTa, ALBERT, and ELECTRA. As a universal coding layer, it is used to encode the information obtained from the above different processing branches, including keyword groups, question groups, global evaluations, and the text similarity, to provide a unified feature representation method for subsequent feature fusion.

[0133] The Transformer Encoder, also known as the Transformer Encoder and the Cross-Attention Transformer, are both key components of the Transformer architecture. The Transformer Encoder is constructed by stacking multiple layers of self-attention mechanisms and feedforward neural networks. It can capture long-range dependencies in input sequences and improve the model's semantic understanding capabilities. Cross-Attention calculates the attention weights between the decoder's current position and all encoder positions, enabling the decoder to dynamically weight the encoder's outputs, generating an output sequence related to the input sequence.

[0134] The Multilayer Perceptron (MLP) layer transforms input data into a higher- or lower-dimensional space to better capture nonlinear relationships in the data. In multimodal learning or tasks involving the fusion of multiple features, the MLP layer can fuse information from different sources to produce a more comprehensive and accurate representation.

[0135] In general, the comprehensive scoring model can fuse features of data from multiple sources for comprehensive analysis, making the output score more accurate.

[0136] Compared to large language models, comprehensive scoring models are relatively small in scale. Different domains can correspond to different comprehensive scoring models. This means using sample data from different domains to train the comprehensive scoring model. This allows the trained comprehensive scoring model to have a deep understanding of that domain, resulting in more accurate scoring.

[0137] Figure 3 This is a flow chart of a method for matching using a comprehensive scoring model according to an embodiment of the present invention. Figure 3 As shown, the method for matching using a comprehensive scoring model includes the following steps:

[0138] Step S301: Determine the field to which the question group to be graded belongs.

[0139] Among them, the fields can be education, medical care, construction, services, etc. The degree of field division can be determined according to actual needs. For example, when the demand for education is high, education can be further subdivided into subjects such as mathematics, Chinese, and English.

[0140] Step S302: determining a corresponding comprehensive scoring model according to the field.

[0141] Step S303: input the keyword group, the question group, the global evaluation and the text similarity into the comprehensive scoring model to obtain the score corresponding to the answer of the answerer.

[0142] The method of an embodiment of the present invention is to obtain a group of questions to be scored, which includes questions, answers of the answering party and reference answers. According to the group of questions to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence are respectively constructed, and the constructed sentences are respectively input into the large language model to obtain the corresponding keyword group, question group and global evaluation, and the text similarity between the answer of the answering party and the reference answer is determined. The keyword group, question group, global evaluation and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the answer of the answering party. The above method combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also makes full use of the vertical scenario learning ability of the comprehensive scoring model.

[0143] Figure 4 Schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention. Figure 4 As shown, the data processing system based on the large language model includes:

[0144] The data acquisition unit 401 is configured to acquire a set of questions to be scored, wherein the set of questions to be scored includes questions, answers from answerers, and reference answers.

[0145] The keyword extraction unit 402 is configured to construct a corresponding keyword extraction prompt sentence according to the to-be-graded question group and a preset keyword extraction prompt template, and input the keyword extraction prompt sentence into the large language model to obtain a corresponding keyword group.

[0146] The pseudo question generating unit 403 is configured to construct a corresponding pseudo question generating prompt statement according to the to-be-rated question group and a preset pseudo question generating prompt template, and input the pseudo question generating prompt statement into the large language model to obtain a corresponding question group.

[0147] The evaluation generation unit 404 is configured to construct a corresponding evaluation generation prompt sentence according to the set of questions to be rated and a preset evaluation generation prompt template, and input the evaluation generation prompt sentence into the large language model to obtain a corresponding global evaluation.

[0148] The similarity determination unit 405 is configured to determine the text similarity between the answer of the respondent and the reference answer.

[0149] The comprehensive scoring unit 406 is configured to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain a score corresponding to the answer of the answerer.

[0150] The specific functions of each unit refer to the corresponding contents of the above embodiments and will not be repeated here.

[0151] Figure 5 The following is a data flow diagram of the data processing system in the embodiment of the present invention. Figure 4 and Figure 5 The following explains how to construct a prompt statement using a reference question and a reference answer, and how to process the answer accordingly.

[0152] The data acquisition unit 401 acquires the set of questions to be graded, wherein Figure 5 The Q in the data representation represents the question, the A represents the answer of the answerer, and the R represents the reference answer. Then, the data acquisition unit 401 inputs the acquired set of questions to be scored into the keyword extraction unit 402, the pseudo-question generation unit 403, the evaluation generation unit 404, and the similarity determination unit 405 respectively.

[0153] In the keyword extraction unit 402, a keyword extraction prompt sentence is constructed based on the question, the answer of the answerer and the reference answer, that is, Prompt1 (Q, A, R) shown in the figure, and then it is input into the large language model. The large language model outputs the keyword group according to Prompt1.

[0154] In the pseudo-question generation unit 403, a pseudo-question generation prompt statement is constructed based on the question, the answer of the respondent and the reference answer, that is, Prompt2 (Q, A, R) shown in the figure, and then input into the large language model. The large language model outputs a question group according to Prompt2.

[0155] In the evaluation generation unit 404, an evaluation generation prompt statement is constructed based on the question, the answer of the respondent and the reference answer, that is, Prompt3 (Q, A, R) shown in the figure, and then input into the large language model. The large language model outputs a global evaluation based on Prompt3.

[0156] In the similarity determination unit 405 , the answer of the respondent and the reference answer are first converted into corresponding vectors, and then the corresponding text similarity is determined based on the vector similarity.

[0157] The output data of the keyword extraction unit 402, the pseudo-question generation unit 403, the evaluation generation unit 404 and the similarity determination unit 405 are input into the comprehensive scoring unit 406. In the comprehensive scoring unit 406, the final score is output based on the sequential processing of the general coding layer, the transformer encoder and the multi-layer perceptron layer.

[0158] The data processing system based on the large language model of the embodiment of the present invention combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also fully utilizes the vertical scenario learning ability of the comprehensive scoring model.

[0159] Figure 6 Schematic diagram of a data processing architecture based on a large language model according to an embodiment of the present invention. Figure 6 As shown, the data processing architecture based on the large language model includes:

[0160] The data communication component 601 is configured to obtain a set of questions to be scored, send the set of questions to be scored to the first processing core and the third processing core, and output the score corresponding to the answer of the answerer, wherein the set of questions to be scored includes questions, answers of the answerer and reference answers.

[0161] The first processing core 602 is configured to construct and send corresponding keyword extraction prompt statements, pseudo-question generation prompt statements and evaluation generation prompt statements to the second processing core according to the group of questions to be scored and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, and determine the text similarity between the answer of the respondent and the reference answer.

[0162] The second processing core 603 is equipped with a large language model and is configured to extract prompt sentences according to the keywords, generate prompt sentences according to the pseudo questions, and generate prompt sentences according to the evaluations, and generate corresponding keyword groups, question groups, and global evaluations.

[0163] The third processing core 604 is equipped with a comprehensive scoring model and is configured to determine the score corresponding to the answer of the respondent based on the keyword group, the question group, the global evaluation and the text similarity.

[0164] The data processing architecture of the embodiment of the present invention combines the advantages of a large language model and a comprehensive scoring model to improve the scoring accuracy, which not only gives full play to the generalization generation capability of the large language model, but also fully utilizes the vertical scenario learning capability of the comprehensive scoring model.

[0165] Figure 7 Schematic diagram of a data processing device based on a large language model according to an embodiment of the present invention. Figure 7 As shown, the data processing device based on the large language model includes:

[0166] The acquisition module 701 is used to obtain a set of questions to be graded, wherein the set of questions to be graded includes questions, answers from answerers and reference answers.

[0167] The prompt module 702 is used to construct corresponding keyword extraction prompt statements, pseudo question generation prompt statements and evaluation generation prompt statements according to the question group to be scored and the preset keyword extraction prompt template, pseudo question generation prompt template and evaluation generation prompt template.

[0168] The model processing module 703 is used to input the keyword extraction prompt sentence, the pseudo question generation prompt sentence and the evaluation generation prompt sentence into the large language model respectively to obtain corresponding keyword groups, question groups and global evaluations.

[0169] The determination module 704 is used to determine the text similarity between the answer of the respondent and the reference answer.

[0170] The scoring module 705 is used to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score of the answer of the respondent.

[0171] The specific functions of each module refer to the corresponding content of the above embodiment and will not be repeated here.

[0172] The device of the embodiment of the present invention is used to obtain a group of questions to be scored, which includes questions, answers of the answering party and reference answers. According to the group of questions to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence are respectively constructed, and the constructed sentences are respectively input into the large language model to obtain the corresponding keyword group, question group and global evaluation, and the text similarity between the answer of the answering party and the reference answer is determined. The keyword group, question group, global evaluation and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the answer of the answering party. The device of this embodiment combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also makes full use of the vertical scenario learning ability of the comprehensive scoring model.

[0173] Figure 8 FIG is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 80 includes a server, a terminal, etc. Figure 8As shown, the electronic device 80 includes: at least one processor 801; a memory 802 communicatively connected to the at least one processor 801; and a communication component 803 communicatively connected to the scanning device, and the communication component 803 receives and sends data under the control of the processor 801; wherein the memory 802 stores instructions that can be executed by the at least one processor 801, and the instructions are executed by the at least one processor 801 to implement the above-mentioned data processing method based on the large language model.

[0174] Specifically, the electronic device includes: one or more processors 801 and a memory 802, Figure 8 A processor 801 is used as an example. The processor 801 and the memory 802 may be connected via a bus or other means. Figure 8 The example of the bus connection is shown in FIG. Memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. Processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 802, thereby realizing the above-mentioned data processing method based on the large language model.

[0175] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 802 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely located relative to the processor 801, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] One or more modules are stored in the memory 802 , and when executed by one or more processors 801 , perform the data processing method based on the large language model in any of the above method embodiments.

[0177] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0178] The technical solution of the embodiment of the present invention is to obtain a group of questions to be scored, which includes questions, answers of the answering party and reference answers. According to the group of questions to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template and the evaluation generation prompt template, respectively construct a keyword extraction prompt sentence, a pseudo-question generation prompt sentence and an evaluation generation prompt sentence, respectively input the constructed sentences into the large language model, obtain the corresponding keyword group, question group and global evaluation, determine the text similarity between the answer of the answering party and the reference answer, input the keyword group, question group, global evaluation and text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the answer of the answering party. The above technical solution combines the advantages of the large language model and the comprehensive scoring model to improve the scoring accuracy, which not only gives play to the generalization generation ability of the large language model, but also makes full use of the vertical scenario learning ability of the comprehensive scoring model.

[0179] Another embodiment of the present invention relates to a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute part or all of the above method embodiments.

[0180] Another embodiment of the present invention relates to a computer program product, comprising a computer program / instruction, which implements part or all of the above method embodiments when the computer program / instruction is executed by a processor.

[0181] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0182] The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application are intended to be within the scope of protection of the present application.

Claims

1. A data processing method based on a large language model, characterized in that: The method comprises: Obtaining a set of questions to be graded, wherein the set of questions to be graded includes a question, an answer from a questioner, and a reference answer; According to the to-be-graded question group and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, corresponding keyword extraction prompt sentences, pseudo-question generation prompt sentences and evaluation generation prompt sentences are respectively constructed; Inputting the keyword extraction prompt sentence, the pseudo-question generation prompt sentence and the evaluation generation prompt sentence into a large language model respectively to obtain corresponding keyword groups, question groups and global evaluations; Determining the text similarity between the answer of the respondent and the reference answer; The keyword group, the question group, the global evaluation and the text similarity are input into a pre-trained comprehensive scoring model to obtain a corresponding score for the answer of the answerer.

2. The method according to claim 1, characterized in that The keyword extraction prompt sentence is used to prompt the large language model to extract keywords from the answer of the answerer based on the question and / or the reference answer; The pseudo-question generation prompt sentence is used to prompt the large language model to generate a pseudo-question corresponding to the answer of the answerer based on the question, the answer of the answerer and the reference answer; The evaluation generation prompt sentence is used to prompt the large language model to analyze the answer of the answerer from multiple dimensions to generate a corresponding global evaluation; The keyword group includes multiple keywords corresponding to the answer of the answerer; The question group includes the question and pseudo questions corresponding to the answer of the answerer; The evaluation generation prompt sentence includes an evaluation sentence within a predetermined word count threshold.

3. The method according to claim 1, characterized in that The keyword extraction prompt sentence is used to prompt the large language model to extract keywords from the question, the answer of the answerer and the reference answer respectively; The keyword group includes multiple keywords corresponding to the answer of the respondent and multiple keywords corresponding to the question, or the keyword group includes multiple keywords corresponding to the answer of the respondent, multiple keywords corresponding to the question and multiple keywords corresponding to the reference answer.

4. The method according to claim 1, wherein Determining the text similarity between the answer of the answerer and the reference answer includes: respectively truncating and rewriting the answer of the respondent and the reference answer; Determine the textual similarity between the rewritten respondent's answer and the reference answer.

5. The method according to claim 1, wherein The step of inputting the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain a corresponding score for the answer of the answerer includes: Determine the field to which the to-be-graded question group belongs; Determine the corresponding comprehensive scoring model according to the field; The keyword group, the question group, the global evaluation and the text similarity are input into the comprehensive scoring model to obtain the corresponding score of the answer of the answerer.

6. The method according to claim 1 or 5, characterized in that The comprehensive scoring model includes a universal coding layer, a transformer encoder and a multi-layer perceptron layer, and the comprehensive scoring model is trained based on sample data in the field to which the to-be-scored question set belongs.

7. A data processing device based on a large language model, characterized in that: The device comprises: An acquisition module is used to acquire a set of questions to be graded, wherein the set of questions to be graded includes a question, an answer from a questioner, and a reference answer; A prompt module, configured to construct corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements according to the to-be-rated question group and preset keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates; A model processing module, configured to input the keyword extraction prompt sentence, the pseudo-question generation prompt sentence, and the evaluation generation prompt sentence into a large language model to obtain corresponding keyword groups, question groups, and global evaluations; A determination module, configured to determine the text similarity between the answer of the respondent and the reference answer; The scoring module is used to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score of the answer of the answerer.

8. A data processing system based on a large language model, characterized in that: The system comprises: A data acquisition unit is configured to acquire a set of questions to be scored, wherein the set of questions to be scored includes a question, an answer from a questioner, and a reference answer; A keyword extraction unit is configured to construct a corresponding keyword extraction prompt sentence based on the to-be-graded question group and a preset keyword extraction prompt template, and input the keyword extraction prompt sentence into the large language model to obtain a corresponding keyword group; a pseudo-question generation unit configured to construct a corresponding pseudo-question generation prompt sentence based on the to-be-graded question group and a preset pseudo-question generation prompt template, and input the pseudo-question generation prompt sentence into the large language model to obtain a corresponding question group; An evaluation generation unit is configured to construct a corresponding evaluation generation prompt sentence based on the to-be-rated question group and a preset evaluation generation prompt template, and input the evaluation generation prompt sentence into a large language model to obtain a corresponding global evaluation; a similarity determination unit, configured to determine the text similarity between the answer of the answerer and the reference answer; The comprehensive scoring unit is configured to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain a corresponding score for the answer of the answerer.

9. A data processing architecture based on a large language model, characterized in that: The architecture includes: a data communication component configured to obtain a set of questions to be scored, send the set of questions to be scored to the first processing core and the third processing core, and output scores corresponding to the answers of the answerers, wherein the set of questions to be scored includes the questions, the answers of the answerers, and the reference answers; The first processing core is configured to construct and send corresponding keyword extraction prompt sentences, pseudo question generation prompt sentences, and evaluation generation prompt sentences to the second processing core based on the to-be-rated question group and the preset keyword extraction prompt template, pseudo question generation prompt template, and evaluation generation prompt template, and to determine the text similarity between the answer of the answerer and the reference answer; The second processing core is equipped with a large language model and is configured to extract prompt sentences based on the keywords, generate prompt sentences based on the pseudo questions, and generate prompt sentences based on the evaluations, and generate corresponding keyword groups, question groups, and global evaluations; The third processing core is equipped with a comprehensive scoring model and is configured to determine the corresponding score of the answer of the respondent based on the keyword group, the question group, the global evaluation and the text similarity.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images