A large language model knowledge point automatic annotation method and system

Abstract

The present invention provides a method and system for automatically labeling knowledge points of a large language model, including: obtaining the content of a question stem to be identified, performing preprocessing, traversing a knowledge tree to generate option questions for selection by a large language model. The large language model uses a three-layer traversal method in the form of a question sentence to traverse the knowledge tree and provides labeled knowledge points. The present invention provides a method for automatically labeling knowledge points using a large language model. When traversing the knowledge tree, only three layers are traversed at most. After traversing to the second layer, it directly jumps to the leaf node of the knowledge tree. Not only is the traversal speed fast, but it can also avoid the situation where the more layers are traversed, the higher the probability of error. At the same time, option questions are formed for selection by the large language model, which is convenient for the large language model to understand, analyze and select, and improve the labeling speed.

Description

Technical Field

[0001] The invention relates to a method for automatically labeling knowledge points of a large language model. Background Art

[0002] In the process of primary and secondary school teaching, the construction of question banks is increasingly valued by educational institutions. Question bank data is the data basis of the teaching process, and the teaching process cannot be separated from the support of question bank data. However, question bank construction is a work with large investment and long cycle, especially the knowledge point labeling of test questions in the question bank, which is a big problem. At present, the industry mainly labels the questions manually, which brings huge labor costs to the construction of question banks and is very inefficient. Of course, there are also some systems on the market that use machine learning models to label knowledge points. However, these systems generally have the problems of slow labeling speed and low accuracy. Summary of the invention

[0003] The embodiment of the present invention provides a large language model knowledge point automatic labeling method to solve the problem of slow speed and low efficiency of existing question stem knowledge point labeling.

[0004] First, a method for automatically labeling knowledge points in a large language model is provided. Figure 1 ,include:

[0005] The question stem content to be recognized is obtained, and in order to improve the recognition accuracy and reduce the complexity of model calculation, the said content is preprocessed, and the knowledge points are selected through the large language model for the preprocessed question stem content.

[0006] The large language model uses a three-layer traversal method in the form of a question sentence to traverse the knowledge tree and gives labeled knowledge points, including:

[0007] Step A: combining the knowledge points of the first layer of the knowledge tree with the preprocessed question stem content to form an option question sentence, and the large language model selects the knowledge points listed in the question sentence;

[0008] Step B: According to the knowledge point obtained in step A, the second-layer knowledge point corresponding to the knowledge point in the knowledge tree is combined with the preprocessed question stem content to form an option question sentence, and the large language model selects the knowledge point and gives the middle-layer knowledge point;

[0009] Step C: All leaf-layer knowledge points corresponding to the middle-layer knowledge points form the third layer and are grouped. The knowledge points in the third layer of the knowledge tree are combined with the preprocessed question stem content to form option questions. The large language model selects knowledge points and summarizes and gives labeled knowledge points.

[0010] If the number of the summarized annotated knowledge points is greater than 1, the summarized annotated knowledge points are organized and combined with the preprocessed question stem content to form an option question sentence again. The large language model finally selects the knowledge point and gives the annotated knowledge point that best matches the question stem content.

[0011] The present invention provides a method for automatically labeling knowledge points using a large language model. When traversing a knowledge tree, only three layers are traversed at most. After traversing to the second layer, the method directly jumps to the leaf node of the knowledge tree. Not only is the traversal speed fast, but it can also avoid the situation where the more layers are traversed, the higher the error probability. At the same time, the large language model is combined with option questions to facilitate the large language model to understand, analyze and select, thereby improving the labeling speed. In addition, the present invention groups all leaf-layer knowledge points corresponding to middle-layer knowledge points to form multiple option analyses and selections, thereby improving the accuracy of the given labeled knowledge points. Finally, when the number of labeled knowledge points given due to grouping is more than one, the given labeled knowledge points can be organized and recognized once, so that multiple pre-recognized knowledge points can be recognized again, and finally labeled knowledge points that are more in line with the content of the question to be recognized are given.

[0012] Preferably, the preprocessing means removing the distracting content that is irrelevant to the problem solving and cannot be processed by the large language model, and replacing the removed distracting content with a half-width space, and when multiple half-width spaces are connected, only one half-width space is left. The distracting content includes question numbers, punctuation marks, answer position markers, images, and spaces at the beginning and end of the question.

[0013] Preferably, the format of the question is: "Please read and understand this" + subject name + "topic, select the knowledge points belonging to the topic, there can be multiple knowledge points, and the content of the topic is:" + topic content + "Select one or more knowledge points from the following knowledge points, only output the number of the knowledge point, no other content needs to be output, the knowledge points that need to be selected are:" + numbered knowledge point options. The present invention reduces the token output of the large language model by only outputting the option number, that is, agreeing in the prompt word that the large language model only outputs the option number, so as to improve the judgment efficiency of the large language model.

[0014] Preferably, the grouping is divided into N groups, and among all the leaf-level knowledge points, the left and right adjacent knowledge points are divided into different groups, so that adjacent very similar knowledge points can be divided into different groups, and independent analysis and selection are performed in each group, so that more analyses and selections can be formed for multiple relatively similar knowledge points, giving more "thoughtful" annotated knowledge points and improving the accuracy.

[0015] Preferably, when grouping, knowledge points are selected by jumping with the same step length for grouping, and there are at most eight leaf-level knowledge points in each group, and the number of divided groups = rounded (number of leaf-level knowledge points / 8 + 0.5).

[0016] Preferably, each knowledge point of the first layer, the second layer or the third layer is represented by a letter sequence of "A~Z", and an "undetermined" item is set for the last item.

[0017] Preferably, in order to obtain a reliable large language model, training samples need to be collected, and the sample data generation step for training the large language model includes:

[0018] Extracting sample question stem content and knowledge points, and preprocessing the sample question stem content;

[0019] Use the backtracking method to generate the knowledge point generation path;

[0020] The knowledge point generation path is combined with the preprocessed question stem content, and the sample data of "question + answer" is formed in the order of the three-layer traversal method of the question type.

[0021] Preferably, in the sample data generation step, when traversing the knowledge tree, the content of the question stem and all the knowledge points of the corresponding layer of the knowledge tree are combined to form a sample question sentence, and the sample data is generated in JSON format, and the JSON format is:

[0022] {

[0023] “instruction”: “content of the question”

[0024] “output”: “answer”

[0025] }.

[0026] In a second aspect, a knowledge point automatic annotation system is provided, comprising:

[0027] The data collection module is used to obtain the content of the question stem to be identified.

[0028] The data processing module is used to pre-process the content of the question stem to be recognized, and use the question-style three-layer traversal method to traverse the knowledge tree, and give annotated knowledge points to implement the large language model knowledge point automatic annotation method described in the first aspect.

[0029] In a third aspect, a knowledge point automatic annotation device is provided, comprising: a memory and a processor. The memory stores executable code, and when the executable code is executed by the processor, the processor executes the large language model knowledge point automatic annotation method as described in the first aspect.

[0030] In a fourth aspect, an embodiment of the invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the method for automatic labeling of knowledge points of a large language model in the first aspect.

[0031] In an embodiment of the present invention, a large language model knowledge point automatic annotation method, system and application are provided, which have the following beneficial effects:

[0032] 1. The large language model knowledge point automatic labeling method provided by the present invention only traverses three layers at most when traversing the knowledge tree. After traversing to the second layer, it directly jumps to the leaf node of the knowledge tree, thereby avoiding the situation where the more layers are traversed, the higher the error probability;

[0033] 2. The present invention groups all leaf-level knowledge points corresponding to the middle-level knowledge points to form multiple option analyses and selections, thereby improving the accuracy of the given annotated knowledge points;

[0034] 3. When there are more than one labeled knowledge points given due to grouping, the labeled knowledge points can be organized and recognized once for the last time, so that multiple pre-obtained knowledge points can be reselected and the final selection can be made, so that the labeled knowledge points given at the end are more consistent with the content of the question to be recognized;

[0035] 4. When traversing the knowledge tree, questions are formed to facilitate the understanding, analysis and selection of the large language model, thereby improving the annotation speed;

[0036] 5. After identifying the knowledge point, the large language model only outputs the option number, which can reduce the token output of the large language model and improve the judgment efficiency of the large language model;

[0037] 6. When grouping, adjacent knowledge points on the left and right are divided into different groups, so that adjacent very similar knowledge points can be divided into different groups, and independent analysis and selection are performed in each group, so that more analysis and selection can be formed for multiple relatively similar knowledge points, thereby improving the recognition accuracy;

[0038] 7. Use backtracking to generate the knowledge point generation path, and combine the question-style three-layer traversal method to traverse the knowledge tree, so as to obtain sample data for training large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] in:

[0041] Figure 1 It is a schematic diagram of the steps of a method for automatically labeling knowledge points of a large language model;

[0042] Figure 2 It is the content of the example question;

[0043] Figure 3 It is a tree expansion diagram of the knowledge point of "Simple calculation of electric power"; DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0046] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] Question bank data is the data foundation of the teaching process, and the teaching process cannot be separated from the support of question bank data. However, question bank construction is a work with large investment and long cycle, especially the knowledge point labeling of test questions in the question bank, which is a big problem. At present, the industry mainly labels questions manually, which brings huge labor costs to question bank construction and is very inefficient. Of course, there are also some systems on the market that use machine learning models to label knowledge points. However, these systems generally have the problems of slow labeling speed and low accuracy, and still require repeated manual review.

[0048] In view of the above problems, the present invention proposes a method for automatic labeling of knowledge points of a large language model. When traversing the knowledge tree, only three layers at most are traversed. After traversing to the second layer, it directly jumps to the leaf node of the knowledge tree, thereby avoiding the situation where the more layers are traversed, the higher the error probability. In addition, the present invention groups all leaf-layer knowledge points corresponding to the middle-layer knowledge points to form multiple option analyses and selections, thereby improving the accuracy of the labeled knowledge points.

[0049] The implementation principles of the method, system, device and medium of the present invention are similar and will not be repeated here.

[0050] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are specifically introduced below. It should be noted that the embodiments provided by the present invention are only shown to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not subject to any limitation in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable system.

[0051] For example, see Figure 1 , provides a method for automatically labeling knowledge points of a large language model, including:

[0052] The question stem content to be recognized is obtained, and in order to improve the recognition accuracy and reduce the complexity of model calculation, the question stem content is preprocessed, and knowledge points are selected from the preprocessed question stem content through a large language model.

[0053] In one embodiment, the preprocessing refers to removing distracting content that is irrelevant to the problem and cannot be processed by the large language model, and replacing the removed distracting content with a half-width space, and when multiple half-width spaces are connected, only one half-width space is left. The distracting content includes question numbers, punctuation marks, answer position markers, images, and spaces at the beginning and end of the question.

[0054] The large language model uses a three-layer traversal method in the form of a question sentence to traverse the knowledge tree and gives labeled knowledge points, including:

[0055] Step A: combining the knowledge points of the first layer of the knowledge tree with the preprocessed question stem content to form an option question sentence, and the large language model selects the knowledge points listed in the question sentence (top-level knowledge points);

[0056] Step B: The second-layer knowledge points corresponding to the knowledge points (top-level knowledge points) obtained in step A in the knowledge tree are combined with the preprocessed question stem content to form option-based questions. The large language model selects knowledge points and gives middle-level knowledge points.

[0057] Step C: All leaf-layer knowledge points corresponding to the middle-layer knowledge points form a third layer and are grouped; the knowledge points of the third layer of the knowledge tree are combined with the preprocessed question stem content to form option-based questions, and the large language model selects knowledge points and summarizes and gives labeled knowledge points; if the number of summarized labeled knowledge points is greater than 1, the summarized labeled knowledge points are organized and combined with the preprocessed question stem content to form option-based questions again, and the large language model finally selects knowledge points and gives labeled knowledge points that best match the question stem content.

[0058] In one embodiment, the format of the question is: "Please read and understand this" + subject name + "title, select the knowledge points belonging to the title, there can be multiple knowledge points, the title content is:" + title content + "Select one or more knowledge points from the following knowledge points, only output the number of the knowledge point, no other content needs to be output, the knowledge points that need to be selected are:" + numbered knowledge point options.

[0059] Specifically, each knowledge point of the first layer, the second layer or the third layer is represented by a letter sequence of "A~Z", and an "undetermined" item is set in the last item. The present invention can reduce the token output of the large language model by only outputting the option number, that is, agreeing in the prompt word that the large language model only outputs the option sequence number, so as to improve the judgment efficiency of the large language model.

[0060] On the basis of the above embodiment, when grouping, it is divided into N groups, and among all the leaf-level knowledge points, the knowledge points adjacent to each other on the left and right are divided into different groups, so that adjacent and very similar knowledge points can be divided into different groups, and independent analysis and selection can be performed in each group, so that more times of analysis and selection can be formed for multiple relatively similar knowledge points, giving more "thoughtful" annotated knowledge points and improving the accuracy. For example, when grouping, the knowledge points are selected by jumping with the same step length for grouping, and there are a maximum of eight leaf-level knowledge points in each group, and the number of divided groups = rounded (number of leaf-level knowledge points / 8 + 0.5).

[0061] On the basis of the above embodiments, in order to obtain a reliable large language model, the large language model needs to be fine-tuned. During the fine-tuning, the large language model is trained with sample data.

[0062] In a specific embodiment, the step of generating sample data of the large language model includes:

[0063] Extract sample question stem content and knowledge points. Preprocess the sample question stem content to reduce the interference of irrelevant data on the calculation of the large language model, and use the backtracking method to generate the knowledge point generation path.

[0064] The knowledge point generation path is combined with the preprocessed question stem content, and the sample data of "question + answer" is formed in the order of the three-layer traversal method of the question type.

[0065] In one embodiment, in the sample data generation step, when traversing the knowledge tree, the question stem content and all knowledge points of the corresponding layer of the knowledge tree are combined to form a sample question sentence, and the sample data is generated in JSON format. The JSON format is:

[0066] {

[0067] “instruction”: “content of the question”

[0068] “output”: “answer”

[0069] }.

[0070] In addition, in order to further illustrate the technical solution provided by the present invention, based on the above embodiments, Figure 2 Taking the example shown in the figure as an example, the following technical points are introduced in detail:

[0071] Preprocessing of title content:

[0072] Remove the content that is irrelevant to the problem and cannot be processed by the large language model, such as question numbers, punctuation marks, answer position markers, images, etc., and replace them with half-width spaces. When multiple spaces are connected, only one space is left. The spaces at the beginning and end of the question are removed, and a period is used as the end. The content of the question stem after processing is:

[0073] Xiaoqiang designed a simple electric rice cooker with adjustable power using the equipment in his home laboratory. As shown in the figure, the power supply voltage is 6V. The resistance wires R1=5Ω and R2=15Ω. ab are two closed contacts. When the switch S1 connects the contacts, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J.

[0074] 2. Establish a knowledge point generation path method:

[0075] In the knowledge tree, the backtracking method is used to trace back from the leaf layer knowledge points (bottom layer knowledge points) to the top layer knowledge points, and then the knowledge points of each layer experienced in the backtracking process are arranged in reverse order. Figure 3 The following is a tree expansion diagram of the knowledge point "Simple Calculation of Electric Power". Starting from "Simple Calculation of Electric Power", tracing back to the first layer of nodes, we can obtain the generation path of this knowledge point:

[0076] Electricity->Electric power->Electric power->Calculation of electric power->Simple calculation of electric power

[0077] 3. Three-layer traversal method of knowledge tree question sentence:

[0078] Step A: Traverse the first layer of the knowledge tree and extract the knowledge points of the first layer to establish the options in the question.

[0079] Step B: Based on the knowledge point generation path or the answer result of the large language model, traverse all the knowledge points in the next layer and organize these knowledge points into questions.

[0080] Step C: Generate a path based on the knowledge point or answer the large language model, find the corresponding knowledge point position on the knowledge tree, traverse all the leaf-layer knowledge points in the corresponding lower layer according to the position, organize these knowledge points in order and give them corresponding serial numbers, and first calculate the number of groups that can be grouped:

[0081] Number of groups = integer (number of knowledge points / 8 + 0.5)

[0082] The number of groups is used as the interval number to jump out and extract the knowledge points of each group, and multiple knowledge point grouping options are obtained.

[0083] When traversing the knowledge tree layer by layer, you will get the options of each layer. The options are represented by serial numbers from "A~Z", and the last item is "undetermined".

[0084] Apply the above steps to Figure 3 The question stem shown is:

[0085] After traversing the first and second level knowledge trees based on the path generated by the knowledge point "Simple Calculation of Electric Power", the first and second level option contents can be generated:

[0086] First level: A. Mechanics B. Thermal C. Acoustics D. Optics E. Electrical F. Unable to determine

[0087] Second level: A. Current and circuit B. Voltage and resistance C. Ohm's law D. Electric power E. Cannot be determined

[0088] If the sample data is to be established, a path is generated based on the knowledge points, with the first-level answer labeled "E" and the second-level answer labeled "D"; if the knowledge points are to be labeled, the answers "E" and "D" can be obtained respectively after querying the large language model.

[0089] Get all leaf node contents under the knowledge point of "electric power" and give each knowledge point a serial number in order:

[0090] 1. Electrical energy and its units

[0091] 2. The essence of electrical work

[0092] 3. Calculation of electrical work

[0093] 4. Application of the electric work calculation formula

[0094] 5. Calculate electric energy using the main parameters of the electric energy meter

[0095] 6. Function and reading of electric energy meter

[0096] 7. The total power of the circuit

[0097] 8. The relationship between the work done by each electrical appliance in a series circuit and its resistance

[0098] 9. The relationship between the work done by each electrical appliance and its resistance in a parallel circuit

[0099] 10. Concept and unit of electric power

[0100] 11. Relationship between electric power, electric energy and time

[0101] 12. Relationship between electric power, voltage and current

[0102] 13. Simple calculation of electric power

[0103] 14. Electric power calculation problems involving images

[0104] 15. Calculation of electric power involving tables

[0105] 16. Power calculation of switch-controlled multi-state circuits

[0106] 17. The problem of the maximum value of electric power in the circuit

[0107] 18. Total power relations in circuits

[0108] 19. Electrical power relations in series circuits

[0109] 20. Electrical power relations in parallel circuits

[0110] 21. Measure the power of a small light bulb

[0111] 22. Common faults and troubleshooting in measuring electric power

[0112] 23. Actual power

[0113] 24. Rated power

[0114] The 24 knowledge points above can be grouped into three groups of eight each. The interval between the serial numbers of the knowledge points in each group is 3, that is, skip 2 and take one. Three groups of options are obtained as follows. Add "undetermined items" at the end of each group of options:

[0115] Group 1: A. Electric energy and its units B. Application of electric power calculation formula C. Total power problem of circuit D. Concept and unit of electric power E. Simple calculation of electric power F. Power calculation of multi-state circuit controlled by switch G. Electric power relationship in series circuit H. Common faults and troubleshooting in measuring electric power I. Unable to determine

[0116] Group 2: A. The essence of electric work B. Calculating electric energy using the main parameters of an electric energy meter C. The relationship between the work done by each appliance in a series circuit and its resistance D. The relationship between electric power, electric energy and time E. Electric power calculation problems involving images F. The problem of the maximum value of electric power in a circuit G. Electric power relationships in parallel circuits H. Actual power I. Unable to determine

[0117] Group 3: A. Calculation of electric work B. Function and reading of electric energy meter C. Relationship between work done by each appliance and resistance in parallel circuit D. Relationship between electric power, voltage and current E. Calculation of electric power involving tables F. Total power relationship in circuit G. Measuring electric power of small light bulb H. Rated power I. Unable to determine

[0118] According to the knowledge points of the questions, the answers to the first group are marked as "E", the answers to the second group are marked as "I", and the answers to the third group are marked as "I".

[0119] 4. Question generation:

[0120] Generate questions according to the following format:

[0121] "Please read and understand this" + subject name + "topic, select the knowledge points that belong to the topic, there can be multiple knowledge points, the topic content is:" + topic content + "Select one or more knowledge points from the following knowledge points, only output the number of the knowledge point, no need to output other content, the knowledge points to be selected are:" + numbered knowledge point options

[0122] For example, the first level of the knowledge tree of junior high school physics can be as follows:

[0123] Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple electric rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts. The switch S is closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. Mechanics B. Thermodynamics C. Acoustics D. Optics E. Electricity F. Undetermined

[0124] 5. Sample data organization method:

[0125] The sample data is organized in JSON format. For example, the questions shown above can generate a series of sample data, that is, a test question consists of multiple questions: [

[0127] {

[0128] "instruction": "Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts. The switch S is closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. Mechanics B. Thermodynamics C. Acoustics D. Optics E. Electricity F. Unable to determine"

[0129] "output": "E"

[0130] },

[0131] {

[0132] "instruction": "Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts. The switch S is closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. Current and circuit B. Voltage resistance C. Ohm's law D. Electric power E. Undetermined"

[0133] “output”: “D”

[0134] },

[0135] {

[0136] "instruction": "Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple electric rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. Electric energy and its unit B. Application of electric work calculation formula C. Total work problem of the circuit D. Concept and unit of electric power E. Simple calculation of electric power F. Power calculation of multi-state circuit controlled by switch G. Electric power relationship in series circuit H. Common faults and troubleshooting of electric power measurement I. Unable to determine"

[0137] "output": "E"

[0138] },

[0139] {

[0140] "instruction": "Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple electric rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts. The switch S is closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. The essence of electric work B. Calculate electric energy using the main parameters of the electric energy meter C. The relationship between the work done by each electrical appliance in the series circuit and the resistance value D. The relationship between electric power and electric energy and time E. Electric power calculation problems involving images F. The maximum value problem of electric power in the circuit G. The relationship between electric power in the parallel circuit H. Actual power I. Cannot be determined"

[0141] “output”: “I”

[0142] },

[0143] {

[0144] "instruction": "Please read and understand the physics question, and select the knowledge points that belong to the question. There can be multiple knowledge points. The question content is: Xiaoqiang uses the equipment in his home laboratory to design a simple electric rice cooker with adjustable power. As shown in the figure, the power supply voltage is 6V. The resistance wire R1=5Ω R2=15Ω ab is two contacts. The switch S is closed. When the switch S1 connects the contact, the power of the rice cooker is greater. Connect R2 to the circuit and work for 3 minutes. The heat generated is J. Select one or more knowledge points from the following knowledge points. Only the number of the knowledge point is output, and no other content is required. The knowledge points that need to be selected are: A. Calculation of electric work B. Function and reading of the energy meter C. Relationship between the work done by each electrical appliance and resistance in a parallel circuit D. Relationship between electric power and voltage and current E. Electric power calculation problems involving tables F. Total power relationship in the circuit G. Measuring the electric power of a small light bulb H. Rated power I. Unable to determine"

[0145] “output”: “I”

[0146] } ]

[0148] The large language model knowledge point automatic labeling method provided by the present invention only traverses three layers at most when traversing the knowledge tree. After traversing to the second layer, it directly jumps to the leaf node of the knowledge tree, thereby avoiding the situation that the more layers are traversed, the higher the error probability. In addition, the present invention groups all leaf-layer knowledge points corresponding to the middle-layer knowledge points to form multiple option analyses and selections, thereby improving the accuracy of the given labeled knowledge points. Finally, when the number of labeled knowledge points given due to grouping is more than one, the given labeled knowledge points can be organized and traversed one last time, so that a separate comparative analysis can be performed on multiple pre-knowledge points, and then the final selection is given, so that the last labeled knowledge points are more in line with the question stem to be identified.

[0149] Example 2: After fine-tuning the 7BQwen2 large language model using 110,000 junior high school physics sample data, a large language model that can annotate knowledge points in junior high school physics test questions was obtained. By testing 300 junior high school physics questions, the accuracy rate reached 85%, and the average time to annotate a question was 1.5 seconds.

[0150] Embodiment 3, based on embodiment 1, provides a knowledge point automatic annotation system, including:

[0151] The data collection module is used to obtain the content of the question stem to be identified.

[0152] The data processing module is used to pre-process the content of the question stem to be identified, and use the question-style three-layer traversal method to traverse the knowledge tree, and give annotated knowledge points to implement the large language model knowledge point automatic annotation method described in Example 1.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically labeling knowledge points in a large language model, characterized in that: include: Obtain the content of the question to be identified and perform preprocessing; The large language model uses a three-layer traversal method in the form of a question sentence to traverse the knowledge tree, select knowledge points, and only output the numbers of the knowledge points, including: Step A: combining the knowledge points of the first layer of the knowledge tree with the preprocessed question stem content to form an option question sentence, and the large language model selects the knowledge points listed in the question sentence; Step B: According to the knowledge point obtained in step A, the second-layer knowledge point corresponding to the knowledge point in the knowledge tree is combined with the preprocessed question stem content to form an option question sentence, and the large language model selects the knowledge point and gives the middle-layer knowledge point; Step C: All leaf-layer knowledge points corresponding to the middle-layer knowledge points form a third layer and are grouped. When grouping, they are divided into N groups, and among all the leaf-layer knowledge points, the knowledge points adjacent to each other on the left and right are divided into different groups; the knowledge points of the third layer of the knowledge tree are combined with the preprocessed question stem content to form option-based questions, and the large language model selects knowledge points and summarizes and gives labeled knowledge points; if the number of summarized labeled knowledge points is greater than 1, the summarized labeled knowledge points are organized and combined with the preprocessed question stem content to form option-based questions again, and the large language model finally selects knowledge points and gives labeled knowledge points that match the question stem content.

2. The method for automatically labeling knowledge points of a large language model according to claim 1, characterized in that: The preprocessing refers to removing interference content that is irrelevant to problem solving and cannot be processed by the large language model, and replacing the removed interference content with half-width spaces, and when multiple half-width spaces are connected, only one half-width space is left; the interference content includes question numbers, punctuation marks, answer position markers, images, and spaces at the beginning and end of the question.

3. The method for automatically labeling knowledge points of a large language model according to claim 1, characterized in that: The format of the question is: "Please read and understand the" + subject name + "title, select the knowledge points belonging to the title, at least one knowledge point, the title content is:" + title content + "Select one or more knowledge points from the following knowledge points, only output the number of the knowledge point, no other content needs to be output, the knowledge points that need to be selected are:" + numbered knowledge point options.

4. The method for automatically labeling knowledge points of a large language model according to claim 1, characterized in that: When grouping, knowledge points are selected by jumping with the same step length for grouping, the leaf-level knowledge points in each group are equal to or less than eight, and the number of divided groups = rounded (number of leaf-level knowledge points / 8 + 0.5).

5. The method for automatically labeling knowledge points of a large language model according to claim 1, characterized in that: Each knowledge point of the first layer, the second layer or the third layer is represented by a letter sequence of "A~Z", and an "undetermined" item is set for the last item.

6. The method for automatically labeling knowledge points of a large language model according to claim 1, characterized in that: The large language model is trained using sample data.

7. The method for automatically labeling knowledge points of a large language model according to claim 6, characterized in that: The sample data generation step for training the large language model includes: Extracting sample question stem content and knowledge points, and preprocessing the sample question stem content; Use the backtracking method to generate the knowledge point generation path; The knowledge point generation path is combined with the preprocessed question stem content, and the sample data of "question + answer" is formed in the order of the three-layer traversal method of the question type.

8. The method for automatically labeling knowledge points of a large language model according to claim 7, characterized in that: In the sample data generation step, when traversing the knowledge tree, the content of the question stem and all the knowledge points of the corresponding layer of the knowledge tree are combined to form a sample question sentence, and the sample data is generated in JSON format. The JSON format is: { "instruction":"question content" "output":"answer" }。 9. A knowledge point automatic annotation system, characterized in that: include: A data collection module is used to obtain the content of the question stem to be identified; The data processing module is used to pre-process the content of the question stem to be identified, and to traverse the knowledge tree using a three-layer traversal method in the form of a question sentence, and to give annotated knowledge points, thereby realizing the large language model knowledge point automatic annotation method described in any one of claims 1 to 8.

Images