Method and apparatus for generating knowledge point tags included in test questions

Abstract

This application provides a method and apparatus for generating knowledge point labels included in test questions. The method first uses a large language model to semantically expand the original knowledge point labels of the test questions, generating an expanded set. Then, it calculates recall similarity based on weighted similarity of rules, semantics, and / or graphical features, filtering out a candidate set. Next, it generates a relevance level based on a re-ranking model, filtering Top-K knowledge points to generate a coarse-ranked set. Then, it uses the large language model to generate confidence scores, filtering out a fine-ranked set. In N rounds of random sampling inference, the order of knowledge points is shuffled, and the frequency of occurrence and confidence fluctuation of the knowledge point labels selected by the large language model are calculated to filter out a stable set. Finally, the large language model is used to vote or independently judge the knowledge points to select the optimal label. This method, through a six-level cascaded architecture, achieves unsupervised automatic labeling of knowledge points for multimodal test questions, significantly improving the accuracy and robustness of the labels.

Description

Technical Field

[0001] At least one embodiment of this application relates to the field of data processing, and more specifically, to a method and apparatus for generating knowledge point tags included in test questions. Background Technology

[0002] The knowledge points in the test questions refer to the core knowledge content of a specific subject (such as mathematics, physics, chemistry, etc.) tested in the test questions, including concepts, principles, theorems, formulas or skills, such as the "Pythagorean theorem" in mathematics and "Newton's second law" in physics.

[0003] Currently, it is necessary to manually label the knowledge points in test questions as knowledge point tags and associate the test questions with those knowledge points. This is called knowledge point "labeling" of test questions. Manual labeling may result in non-standardized knowledge point tags, insufficient information, or distorted information. It also presents pain points such as low efficiency, high labor costs, and insufficient accuracy in labeling unlabeled test questions.

[0004] Therefore, there is a need to design a method and device that can generate comprehensive and accurate knowledge point tags for test questions at low cost and high efficiency. Summary of the Invention

[0005] According to one aspect of this application, a method for generating knowledge point tags included in test questions is provided, characterized by comprising:

[0006] We use a large language model to semantically expand the original knowledge point labels of each test question to generate an expanded set of knowledge points;

[0007] Based on the rule matching degree and rule recall weight of knowledge points and tag text rules, the semantic similarity and semantic recall weight of knowledge points and test questions, and / or the similarity of graphic features and graphic recall weight of knowledge points and test questions, the recall similarity is calculated, and knowledge points with recall similarity higher than or equal to the predetermined recall similarity threshold are recalled from the expanded knowledge point set to form a candidate knowledge point set.

[0008] The reordering model generates the relevance between each knowledge point in the candidate knowledge point set and the test question. The Top-K knowledge points with a relevance higher than a predetermined relevance threshold are selected from the candidate knowledge point set to generate a coarse-ranked knowledge point set, where Top-K is a positive integer.

[0009] The coarse-ranked knowledge point set is generated using a large language model. Each knowledge point is a confidence score of the knowledge points included in the test question. One or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are selected from the coarse-ranked knowledge point set to generate a fine-ranked knowledge point set.

[0010] In each round of random sampling reasoning in N rounds, after shuffling the order of the knowledge points in the finely sorted knowledge point set, the large language model is used to infer the knowledge points included in the test question and their confidence scores from the shuffled knowledge points. After N rounds of random sampling reasoning, the frequency of occurrence and the fluctuation of the confidence scores of the inferred knowledge points in N rounds of random sampling reasoning are calculated to filter out one or more knowledge points whose frequency of occurrence is greater than or equal to a predetermined frequency threshold and / or whose fluctuation of confidence scores is less than or equal to a predetermined confidence fluctuation threshold, so as to generate a set of filtered knowledge points, where N is a positive integer;

[0011] The large language model is used to vote on every two knowledge points in the filtered knowledge point set. The optimal knowledge point label for the test question is selected from the filtered knowledge point set by combining at least one of the following: the number of times the higher vote score wins, the confidence score, or the degree of relevance. Alternatively, the large language model is used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point labels for the test question from the filtered knowledge point set.

[0012] In some embodiments, the original knowledge point labels of the test questions are either original knowledge point labels generated by a large language model or original knowledge point labels of the test questions manually labeled.

[0013] In some embodiments, a large language model is used to semantically augment each original knowledge point label of the test question to generate an augmented knowledge point set, including:

[0014] Extract at least one of the following tag information from each original knowledge point tag of the test question: knowledge point keywords, knowledge point hierarchical relationship, constraints on keywords that must appear after semantic expansion, and subject attributes including at least one of the following: applicable question type structure, applicable question type name, applicable grade, applicable region, and difficulty rating.

[0015] Obtain explanations of terminology for knowledge points in different disciplines;

[0016] Based on the tag information and terminology explanations, a large language model is used to generate semantically expanded knowledge point tags, including at least one of the following: terminology explanations of knowledge point keywords for each original knowledge point tag, descriptions of hierarchical relationships of knowledge points, descriptions of constraints, and descriptions of subject attributes.

[0017] Semantic validation is performed on the semantically expanded knowledge point tags to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, and these are used as the semantically validated knowledge point tags.

[0018] Duplicate knowledge point tags are removed from the semantically validated knowledge point tags to generate an expanded knowledge point set.

[0019] In some embodiments, semantic verification is performed on the semantically expanded knowledge point tags to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, which are then used as the semantically verified knowledge point tags. The method further includes:

[0020] The score for the expansion effect is determined based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the degree of domain matching between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the degree of semantic coverage of the semantically expanded knowledge point tags over the original knowledge point tags.

[0021] Knowledge point tags whose expansion effect score is greater than or equal to the predetermined expansion effect threshold are used as knowledge point tags after semantic verification.

[0022] In some embodiments, the expansion effect score is determined based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the domain matching degree between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the semantic coverage of the semantically expanded knowledge point tags over the original knowledge point tags. This includes determining the expansion effect score using the following formula:

[0023] ,

[0024] Here, "Score" represents the score for the expansion effect.

[0025] α, β, and γ are weight coefficients, α+β+γ=1,

[0026] This indicates the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags. ,in Vectors representing the original knowledge point labels Vectors of semantically expanded knowledge point labels Cosine similarity between them;

[0027] This indicates the domain matching degree between the subject domain of the test questions and the subject domain of the semantically expanded knowledge point tags, where... , The number of keywords that match between the semantically expanded knowledge point tags and the knowledge point graph in the subject area of ​​the test questions; This represents the total number of keywords in the knowledge point map of the subject area of ​​the test questions; This refers to the number of subject features that match the subject domain of the test questions in the semantically expanded knowledge point tags; The number of all features across all subject areas of the test questions.

[0028] This indicates the semantic coverage of the semantically expanded knowledge point tags compared to the original knowledge point tags, where... ,in, As weight, and , The difference in sentence structure between the semantically augmented knowledge point labels and the original knowledge point labels given by the large language model. The extent to which the semantically augmented knowledge point labels provided for the large language model supplement the original knowledge point labels from a multi-dimensional semantic perspective. The semantically expanded knowledge point labels provided for the large language model supplement the original knowledge point labels to at least one aspect, including subject, question type, and scenario information. The character count penalty coefficient is as follows: the closer the character count of the semantically expanded knowledge point tag is to the predetermined range, the higher the character count penalty coefficient; the further the character count of the semantically expanded knowledge point tag is from the predetermined range, the lower the character count penalty coefficient.

[0029] In some embodiments, ,in It is the number of characters in the semantically expanded knowledge point tags. It is a predetermined range, including the number of characters in the original knowledge point tags, and k is the smoothing coefficient.

[0030] In some embodiments, the values ​​of rule recall weight, semantic recall weight, graph recall weight, predetermined recall similarity threshold, Top-K value, predetermined confidence threshold, and N vary based on at least one of the following: text length feature of the test item, number of recalled knowledge points and character count feature, confidence score feature, and predetermined coding value of the test item's subject and / or question type.

[0031] In some embodiments, rule recall weight ,

[0032] Semantic Recall Weight ,

[0033] Image Recall Weight ,

[0034] Predetermined recall similarity threshold ,

[0035] Top-K values ,

[0036] Predetermined confidence threshold ,

[0037] The value of N ,

[0038] in, The initial value is , It is a weight parameter, and ,

[0039] in, The subsequent value is ,

[0040] It is a weight parameter, and ,

[0041] in, It is a feature of the text length of the test questions.

[0042] These are the characteristics of the number of knowledge points and the number of characters recalled during the process of generating knowledge point tags included in the previous round of test questions.

[0043] It is a feature of the confidence score in the process of generating knowledge point labels included in the previous round of test questions.

[0044] It is the predetermined code value for the subject and / or question type of the test questions.

[0045] In some embodiments, ,

[0046] in, It is the number of characters in the question. This is the maximum character limit for questions in the corresponding subject area.

[0047] ,

[0048] in, It represents the total number of knowledge points recalled during the previous round of generating test questions, which included the knowledge point tags. It is the preset maximum number of knowledge point tags that can be recalled for a single test question. It is the average number of characters of knowledge points recalled during the process of generating knowledge point tags included in the previous round of test questions. This is the preset maximum character limit for each knowledge point.

[0049] ,

[0050] in, It is the average confidence score of all knowledge points in the refined knowledge point set during the process of generating knowledge point tags in the previous round of test question generation. It is the preset minimum value of the confidence score. It is the preset maximum value of the confidence score.

[0051] The preset value is 1.0 for questions in subjects like Mathematics / Physics / Chemistry and those involving graphing, formulas, or problem-solving; 0.8 for questions in Mathematics / Physics / Chemistry and those involving multiple-choice or fill-in-the-blank questions; 0.9 for questions in Chinese / English and those involving reading comprehension or composition; 0.7 for questions in Chinese / English and those involving multiple-choice or fill-in-the-blank questions; and 0.85 for questions in other subjects and all question types.

[0052] According to another aspect of this application, an apparatus for generating knowledge point tags included in test questions is provided, comprising:

[0053] Memory, used to store instructions;

[0054] A processor for reading instructions from memory and executing the methods of any embodiment of this application.

[0055] In summary, at least one embodiment of this application significantly improves the accuracy and completeness of knowledge point labels through the aforementioned six-level cascaded screening architecture and multi-round verification mechanism, effectively avoiding the problems of non-standard manual annotation and information distortion; it adopts an unsupervised learning strategy combined with a large language model, achieving high-accuracy knowledge point label generation without relying on a large amount of labeled data, greatly reducing the cost of manual annotation; it comprehensively utilizes the feature information of multimodal test questions through a multi-dimensional recall mechanism, effectively improving the ability to identify knowledge points for complex test questions; it effectively resists the influence of input order disturbances by evaluating confidence fluctuations through multi-round random sampling inference; and it significantly enhances the system's generalization ability and robustness to different subjects, question types, and difficulties through a dynamic adaptive parameter adjustment mechanism and a voting decision mechanism. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A schematic diagram of an initial knowledge point map with manual annotations for a certain test question according to at least one embodiment of this application is shown.

[0058] Figure 2 A flowchart illustrating a method for generating knowledge point tags included in test questions according to at least one embodiment of this application is shown.

[0059] Figure 3 A flowchart illustrating the steps of semantically expanding each original knowledge point label of a test question using a large language model to generate an expanded knowledge point set according to at least one embodiment of this application is shown.

[0060] Figure 4 A block diagram of an apparatus for generating knowledge point tags included in test questions according to at least one embodiment of this application is shown. Detailed Implementation

[0061] Specific embodiments of this application will now be described in detail, with examples of the application illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this application as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0062] Existing methods for generating knowledge point labels for test questions typically rely on manual annotation or supervised learning models, which consumes significant human resources. Manual annotation can lead to inconsistent, incomplete, or distorted labels, and the quality and scale of the labeled data are limited. Furthermore, for multimodal test questions (such as those containing both text and images), traditional methods struggle to effectively integrate text and image features, resulting in insufficient accuracy in knowledge point recognition. Moreover, existing technologies lack effective unsupervised learning strategies for processing unlabeled data, making it difficult to meet the demands for rapid processing and comprehensive, accurate label information for large-scale test question knowledge point label generation.

[0063] In summary, existing methods for generating knowledge point labels for test questions suffer from the following technical shortcomings: First, they rely on manual annotation or large amounts of labeled data, making them difficult to apply in unlabeled scenarios, resulting in high costs and low efficiency. Second, for multimodal test questions (including text, formulas, and graphs), existing methods struggle to effectively integrate multimodal features, leading to insufficient accuracy in knowledge point recognition. Third, they lack a mechanism to verify the stability of the model output, potentially causing significant fluctuations in annotation results when input perturbations occur. Fourth, they lack an adaptive mechanism to automatically adjust parameters based on different subjects, question types, and difficulties, resulting in poor cross-scenario generalization ability. Therefore, a technical solution is needed that can automatically generate accurate, comprehensive, stable, and adaptable knowledge point labels for multimodal test questions, adaptable to different subject question types.

[0064] At least one embodiment of this application provides a method for unsupervised generation of knowledge point labels included in test questions based on a multimodal cascade architecture. This method aims to solve the problems in the prior art, such as the reliance on manual annotation for knowledge point labeling, poor multimodal feature fusion effect, and lack of unsupervised learning strategies, which lead to inefficient processing and insufficient comprehensiveness and accuracy of label information in the large-scale generation of test question knowledge point labels.

[0065] Figure 1 A schematic diagram of an initial knowledge point map with manual annotations for a certain test question according to at least one embodiment of this application is shown.

[0066] The initial knowledge point map can be manually constructed by teaching experts. These experts build the knowledge point map based on the nested and inclusive relationships of the knowledge points within the test questions.

[0067] like Figure 1 The tree structure displays a knowledge point map of a fifth-grade math problem (the problem states: "The cost price of a commodity is 200 yuan. After selling it at an 80% discount, a 20% profit is still made. Find the marked price of the commodity"). For example, this math problem mainly involves an important knowledge point: linear equations in one variable. The knowledge point map also shows the hierarchical relationship between knowledge points, that is, a knowledge point has subordinate knowledge points. For example, the subordinate knowledge point of linear equations in one variable is the application of linear equations in one variable, and the subordinate knowledge point of the application of linear equations in one variable is the knowledge point of linear equations in one variable and functions. Therefore, an original knowledge point label manually constructed by teaching experts could be, for example, Fifth Grade Math @ Linear Equations in One Variable @ Application of Linear Equations in One Variable @ Linear Equations in One Variable and Functions (where "@" is assumed to be a symbol representing hierarchical relationships, but other symbols can be used instead, such as "->", without restriction).

[0068] For example, let's take a seventh-grade English question as an example. The original knowledge point tags for the question are "Seventh-grade English @ Phonetics @ Pronunciation @ Pronunciation of single consonant letters", where the "@" symbol indicates the hierarchical relationship of the knowledge points.

[0069] Figure 2 A flowchart of a method 200 for generating knowledge point tags included in test questions according to at least one embodiment of this application is shown.

[0070] like Figure 2 As shown, the method 200 for generating knowledge point tags included in the test questions includes steps 210-260.

[0071] In step 210, the semantics of each original knowledge point label of the test question are semantically expanded using a large language model to generate an expanded knowledge point set.

[0072] In step 220, based on the rule matching degree and rule recall weight of the knowledge point and the tag text rule, the semantic similarity and semantic recall weight of the knowledge point and the question, and / or the graphic feature similarity and graphic recall weight of the question, the recall similarity is calculated, and knowledge points with recall similarity higher than or equal to the predetermined recall similarity threshold are recalled from the expanded knowledge point set to form a candidate knowledge point set.

[0073] In step 230, the relevance between each knowledge point in the candidate knowledge point set and the test question is generated based on the re-ranking model. The Top-K knowledge points with a relevance higher than a predetermined relevance threshold are selected from the candidate knowledge point set to generate a coarse-ranked knowledge point set, where Top-K is a positive integer.

[0074] In step 240, the confidence score of each knowledge point in the coarse-ranked knowledge point set is generated using the large language model. Then, one or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are selected from the coarse-ranked knowledge point set to generate the fine-ranked knowledge point set.

[0075] In step 250, in each round of random sampling reasoning in N rounds of random sampling reasoning, after shuffling the order of the knowledge points in the finely sorted knowledge point set, the large language model is used to infer the knowledge points included in the test question and their confidence scores from the shuffled knowledge points. After N rounds of random sampling reasoning, the frequency of occurrence and the fluctuation of the confidence scores of the inferred knowledge points in N rounds of random sampling reasoning are calculated to filter out one or more knowledge points whose frequency of occurrence is greater than or equal to a predetermined frequency threshold and / or whose fluctuation of the confidence scores is less than or equal to a predetermined confidence fluctuation threshold, so as to generate a set of filtered knowledge points, where N is a positive integer.

[0076] In step 260, the large language model is used to vote on every two knowledge points in the filtered knowledge point set. The optimal knowledge point label for the test question is selected from the filtered knowledge point set by combining at least one of the following: the number of times the higher voting score wins, the confidence score, or the degree of relevance. Alternatively, the large language model is used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point labels for the test question from the filtered knowledge point set.

[0077] The method for generating knowledge point tags included in test questions provided in at least one embodiment of this application achieves the following corresponding technical effects through the synergistic effect of the following technical features.

[0078] First, at least one embodiment of this application utilizes a large language model to semantically expand each original knowledge point label of the test question to generate an expanded knowledge point set. Since the original knowledge point labels are usually relatively simple or fixed, direct retrieval may lead to limited recall. By semantically expanding the original labels using a large language model combined with domain knowledge, expanded labels containing richer contextual information, different expression methods, and hierarchical relationships can be generated. This process effectively solves the problem of insufficient semantic coverage of the original labels, significantly expands the candidate range of the retrieval, thereby improving the recall rate of knowledge point labels and ensuring that potentially relevant knowledge points are not missed.

[0079] Secondly, based on the rule matching degree and rule recall weight between the knowledge point and the tag text rule, the semantic similarity and semantic recall weight with the question, and / or the graphic feature similarity and graphic recall weight with the question, a recall similarity is calculated. Knowledge points with a recall similarity higher than or equal to a predetermined recall similarity threshold are recalled from the expanded knowledge point set to form a candidate knowledge point set. Existing technologies often rely solely on text similarity for retrieval, ignoring multimodal information such as graphics and formulas that may be contained in the question. At least one embodiment of this application achieves comprehensive utilization of the multimodal features of the question by fusing rule matching, semantic similarity, and graphic feature similarity, and introducing a weighting mechanism. This enables the system to effectively identify complex questions containing images or formulas, solving the technical problem of insufficient accuracy of single-modal retrieval in multimedia question scenarios, and improving the comprehensiveness and relevance of the candidate knowledge point set.

[0080] Secondly, based on the re-ranking model, the relevance between each knowledge point in the candidate knowledge point set and the test question is generated. The Top-K knowledge points with relevance levels exceeding a predetermined relevance threshold are then selected from the candidate knowledge point set to generate a coarse-ranked knowledge point set. During the recall phase, some noisy data may be introduced to pursue a high recall rate. By introducing a dedicated re-ranking model to refine the ranking and filtering of the candidate set, noisy labels with low relevance can be quickly removed while retaining high-potential knowledge points. This mechanism effectively purifies the input data while ensuring retrieval efficiency, solving the problem of heavy processing burden caused by noise interference under large-scale retrieval, and providing a high-quality input foundation for subsequent large language model inference.

[0081] Furthermore, each knowledge point in the coarse-ranked knowledge point set generated using a large language model is a confidence score of the knowledge points included in the test question. One or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are then selected from the coarse-ranked knowledge point set to generate a fine-ranked knowledge point set. While re-ranking models can assess relevance, they may not be as effective as large language models in deep semantic understanding and logical reasoning. At least one embodiment of this application utilizes a large language model to score the confidence of the coarse-ranked results, enabling a deeper understanding of the logical connections between the test question content and the knowledge points. By setting a confidence threshold for secondary screening, the false positive rate is further reduced, ensuring that the knowledge point set entering the final verification stage has higher semantic purity and credibility.

[0082] Furthermore, in each round of random sampling inference within the N rounds of random sampling inference, after shuffling the order of the knowledge points in the refined knowledge point set, a large language model is used to infer the knowledge points included in the test question and their confidence scores from the shuffled knowledge points. After N rounds of random sampling inference, the frequency of occurrence and the fluctuation of confidence scores of the inferred knowledge points in the N rounds of random sampling inference are calculated to filter out one or more knowledge points whose frequency of occurrence is greater than or equal to a predetermined frequency threshold and / or whose fluctuation of confidence scores is less than or equal to a predetermined confidence fluctuation threshold, thereby generating a filtered knowledge point set. The output of a large language model is often sensitive to the input order and exhibits random fluctuations. At least one embodiment of this application constructs a stability-based verification mechanism through N rounds of random shuffling sampling inference and statistical analysis of frequency of occurrence and confidence fluctuations. This process effectively resists the influence of input order perturbation on the model output, quantitatively evaluates the stability of label judgment, and eliminates labels with wavering model judgments, thereby significantly improving the reliability and robustness of the final knowledge point labels.

[0083] Finally, a large language model is used to vote on every two knowledge points in the filtered knowledge point set. Combining at least one of the following factors—the number of times a higher vote score results in a win, confidence score, or relevance—an optimal knowledge point label for the question is selected from the filtered knowledge point set. Alternatively, the large language model can be used to determine whether each knowledge point in the filtered knowledge point set is included in the question, thereby selecting one or more optimal knowledge point labels for the question from the filtered knowledge point set. In the final decision-making stage, a single scoring metric may be biased. At least one embodiment of this application provides two modes: a voting mechanism and an independent judgment mechanism. By combining pairwise voting with the number of wins, confidence score, or relevance, a comprehensive decision can be made, avoiding getting trapped in local optima and adapting to the needs of different question types (such as single-label or multi-label). This decision-making mechanism integrates multiple dimensions of evaluation metrics, further improving the accuracy and applicability of the final output labels, and providing high-quality technical support for the construction of intelligent question banks.

[0084] In summary, at least one embodiment of this application significantly improves the accuracy and completeness of knowledge point labels through the aforementioned six-level cascaded screening architecture and multi-round verification mechanism, effectively avoiding the problems of non-standard manual annotation and information distortion; it adopts an unsupervised learning strategy combined with a large language model, achieving high-accuracy knowledge point label generation without relying on a large amount of labeled data, greatly reducing the cost of manual annotation; it comprehensively utilizes the feature information of multimodal test questions through a multi-dimensional recall mechanism, effectively improving the ability to identify knowledge points for complex test questions; it effectively resists the influence of input order disturbances by evaluating confidence fluctuations through multi-round random sampling inference; and it significantly enhances the system's generalization ability and robustness to different subjects, question types, and difficulties through a dynamic adaptive parameter adjustment mechanism and a voting decision mechanism.

[0085] Next, for ease of understanding, each step will be described in detail.

[0086] In step 210, the semantics of each original knowledge point label of the test question are semantically expanded using a large language model to generate an expanded knowledge point set.

[0087] In some embodiments, the original knowledge point tags for a test question may be original knowledge point tags generated using a large language model, or original knowledge point tags for a test question manually annotated. The format of the original knowledge point tags can be referenced... Figure 1 The examples shown are not limited to these.

[0088] Semantic augmentation is the process of generating a set of augmented knowledge point tags that are semantically consistent with the original tags but have richer expressions and broader information coverage, based on the original knowledge point tags. This is achieved by extracting their structured information (such as keywords, hierarchical relationships, constraints, and subject attributes), combining them with domain knowledge (such as terminology explanations and subject rules) and contextual semantics, and utilizing natural language processing techniques (such as large language models). These augmented tags not only retain the core semantics of the original tags but also form a more comprehensive expression of knowledge points by adding contextual descriptions, supplementing scene information, and refining hierarchical relationships. The augmented knowledge point set can include both the original knowledge point tags and the augmented knowledge point tags.

[0089] In some embodiments, step 210, which uses a large language model to semantically augment each original knowledge point label of a test question to generate an augmented knowledge point set, includes steps 211-215.

[0090] Figure 3 A flowchart illustrating the steps of semantically expanding each original knowledge point label of a test question using a large language model to generate an expanded knowledge point set according to at least one embodiment of this application is shown.

[0091] like Figure 3 As shown, in step 211, at least one of the following tag information is extracted from each original knowledge point tag of the test question: knowledge point keywords, knowledge point hierarchical relationship, constraints on keywords that must appear after semantic expansion, and subject attributes including at least one of the following: applicable question type structure, applicable question type name, applicable grade, applicable region, and difficulty rating.

[0092] For example, let's take a seventh-grade English question as an example. The original knowledge point tags for the question are "Seventh-grade English @ Phonetics @ Pronunciation @ Pronunciation of single consonant letters", where the "@" symbol indicates the hierarchical relationship of the knowledge points.

[0093] Knowledge point keywords refer to characteristic words or phrases extracted from the original knowledge point tags of the test questions that can represent the core semantic content of the knowledge point. For example, the knowledge point keywords extracted from each original knowledge point tag are: single consonant letter, pronunciation, and phonetic rules. In a specific embodiment, these keywords are used to describe different dimensions of information in subject learning, specifically including: core language elements: referring to the basic building blocks of a subject, such as "single consonant letter" in English; ability emphasis: referring to the specific skills or abilities emphasized by the knowledge point, such as "pronunciation" emphasizing the ability to achieve speech; knowledge category: referring to the basic theories or rule system of the subject to which the knowledge point belongs, such as "phonetic rules" being the basic knowledge of English learning.

[0094] Extract the hierarchical relationships from each original knowledge point label: for example, phonetics → pronunciation → pronunciation of individual consonant letters. This hierarchical relationship reflects a progressive structure from broad phonetic categories to specific pronunciation rules, which aligns with the knowledge system organization of the English teaching syllabus.

[0095] Extract the constraints from each original knowledge point tag: for example, the semantically expanded tag must contain keywords such as "single consonant letter". This constraint ensures that the expanded knowledge point tag does not deviate from the knowledge point keywords of the original tag, avoiding the generation of tags unrelated to speech knowledge.

[0096] The subject attributes extracted from each original knowledge point tag are as follows: For example, applicable question type: multiple choice, requiring students to select words with different pronunciations from four options; applicable question structure: multiple choice, requiring students to identify and compare the pronunciations of letters in words, a typical phonetic discrimination question; applicable grade: seventh grade; applicable region: all; difficulty rating: relatively easy, as this question involves basic consonant pronunciation rules, introductory knowledge for English learning, hence the relatively easy difficulty rating. This information can be extracted from the original knowledge point tags, the questions themselves, or obtained externally. In short, this information supplements the interpretation of the original knowledge point tags.

[0097] In step 212, the terminology explanations of knowledge points in different disciplines are obtained.

[0098] For example, the terminology for individual consonant letters in English is explained as follows: 1. The pronunciation of a single consonant letter in a word: "Consonant letters refer to the 21 letters in the English alphabet excluding a, e, i, o, and u." The letter 'c' is pronounced / s / before the vowels e, i, and y (e.g., city), and / k / in other cases (e.g., cat, cup, cake). This is a basic rule for learning English phonetics. 2. Key points are usually marked by underlining. In phonetics discrimination questions, letters or syllables that need attention are often marked by underlining or bolding. This information can supplement the explanation of the original knowledge point tags in UE (User Experience).

[0099] In step 213, based on the tag information and terminology explanations, the large language model is used to generate semantically expanded knowledge point tags, including at least one of the following: a description of the terminology explanation of the knowledge point keywords for each original knowledge point tag, a description of the hierarchical relationship of knowledge points, a description of the constraints, and a description of the subject attributes.

[0100] For example, the expanded tag is: Common question type is multiple choice, and it covers common test points in seventh-grade English. It often appears in easier questions, and the question stem usually uses underlined text to ask questions, such as "Which of the following words has a different pronunciation from the other options?", and frequently includes words like "pronounced," "pronunciation," "underlined," "letter," and "the same sound." It frequently appears when testing the pronunciation of a single consonant in a word, and in multiple-choice questions requiring students to select the pronunciation differences of the same consonant in different words. It's important to note that: ① Because most question stems don't explicitly contain the word "consonant," it's necessary to distinguish whether the question tests the pronunciation of vowels or consonants. ② If the question tests the pronunciation of letter combinations, or if the underlined part is more than one letter, this tag is not used. This means that the expanded tag supplements the original knowledge point tag's definition, allowing the large language model to better understand the information related to the knowledge point encompassed by the tag, avoiding situations where the large language model cannot understand or fully comprehend the information related to the knowledge point tags manually annotated by teaching experts.

[0101] In step 214, semantic verification is performed on the semantically expanded knowledge point tags to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, which are then used as the semantically verified knowledge point tags.

[0102] In some embodiments, step 214, which involves performing semantic verification on the semantically expanded knowledge point tags to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, further includes: determining an expansion effect score based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the domain matching degree between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the semantic coverage of the semantically expanded knowledge point tags over the original knowledge point tags; and using knowledge point tags whose expansion effect score is greater than or equal to the predetermined expansion effect threshold as semantically verified knowledge point tags.

[0103] In some embodiments, the expansion effect score is determined based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the domain matching degree between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the semantic coverage of the semantically expanded knowledge point tags over the original knowledge point tags. This includes determining the expansion effect score using the following formula:

[0104] ,

[0105] Here, "Score" represents the score for the expansion effect.

[0106] α, β, and γ are weight coefficients, α+β+γ=1.

[0107] For example, the weighting coefficients α, β, and γ can be dynamically adjusted to meet the adaptation requirements of different subjects, question types, and fields. For instance, α can be set to [0.4, 0.5], β to [0.2, 0.3], and γ to [0.2, 0.3]. In one example, α = 0.5, β = 0.3, and γ = 0.2.

[0108] This indicates the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags. ,in Vectors representing the original knowledge point labels Vectors of semantically expanded knowledge point labels The cosine similarity between them. For example, It can be set to use cosine similarity Mapped to scores between 0 and 100, for example .

[0109] Cosine similarity The value range of is [0, 1], and its calculation formula can be as follows.

[0110] .

[0111] For example, the original tags are "Grade 7 English @pronunciation @pronunciation of a single consonant letter". Preprocessing can convert the hierarchical tags into continuous text: "Grade 7 English pronunciation pronunciation of a single consonant letter".

[0112] The expanded tag is: Common question type is multiple choice, and it covers common test points in seventh-grade English. It often appears in easier questions. The question stem usually uses underlined words to indicate the question, such as "Which of the following words has a different pronunciation from the other options?" The question stem frequently includes words like "pronounced," "pronunciation," "underlined," "letter," and "the same sound." It frequently appears when testing the pronunciation of a single consonant in a word, and in multiple-choice questions requiring students to select the pronunciation differences of the same consonant in different words. Note that: ① Because most question stems do not explicitly contain the word "consonant," it is necessary to distinguish whether the question tests the pronunciation of vowels or consonants. ② If the question tests the pronunciation of letter combinations, or if the underlined part is more than one letter, this tag is not used.

[0113] Next, we calculate the cosine similarity between the original label vector and the augmented label vector. For example, we use a pre-trained language model (such as Bidirectional Encoder Representations from Transformers, BERT) to convert the text into a fixed-dimensional vector. Assuming the model outputs a vector with a dimension of 768 (e.g., BERT-base), then: the vector of the original labels (simplified example): =[0.707, 0.123, 0.045, ..., 0.672], the expanded label vector: =[0.698, 0.145, 0.032, ..., 0.681].

[0114] The formula for calculating cosine similarity is: ,in: It is the dot product of vectors. Let be the modulus of the vector (Euclidean norm).

[0115] Assume a simplified vector (using only the first 3 dimensions): =[0.7, 0.1, 0.2], =[0.6, 0.2, 0.1]. Calculate the dot product. =(0.7×0.6)+(0.1×0.2)+(0.2×0.1)=0.42+0.02+0.02=0.46.

[0116] Calculate the modulus: . .

[0117] Calculate cosine similarity: =0.46 / (0.735×0.640)≈0.46 / 0.4704≈0.978.

[0118] but =97.8 (points).

[0119] This ensures that the expanded tag does not deviate from the core semantics of the original tag, avoiding tag distortion due to overgeneralization or the introduction of irrelevant content. For example, if the expanded tag is changed to "rules for writing consonant letters", the cosine similarity with the original tag will drop to 0.3, which will lead to a decrease in the expansion effect score, possibly falling below the predetermined expansion effect threshold, and thus potentially causing the expanded tag to be removed.

[0120] This indicates the domain matching degree between the subject domain of the test questions and the subject domain of the semantically expanded knowledge point tags, where... , The number of keywords that match between the semantically expanded knowledge point tags and the knowledge point graph in the subject area of ​​the test questions; This represents the total number of keywords in the knowledge point map of the subject area of ​​the test questions; This refers to the number of subject features that match the subject domain of the test questions in the semantically expanded knowledge point tags; The number of all features of the subject area of ​​the test questions.

[0121] For example, .

[0122] Taking the knowledge point tags for the English subject as an example, we count the number of keywords that match between the expanded knowledge point tags and the knowledge point map of the test questions' subject area (such as English phonetics). And calculate the matching rate. The knowledge map of the subject area of ​​the test questions (such as English or further English phonetics) can be pre-generated by teaching experts to represent all the keywords that may be involved in the relevant knowledge points in that subject area.

[0123] For example, the expanded knowledge point tags match keywords in the knowledge point map of the test questions' subject area (such as English phonetics): "consonant letters", "pronunciation", "pronounced", and "underlined". There are 4, while the total number of keywords in the knowledge point map of the subject area of ​​the test questions is 4. There are 12 matches. Match rate The figure of 4 / 12 = 33.3% may indicate that the expanded knowledge point tags do not match the subject area of ​​the test questions well.

[0124] and This refers to the number of subject features that match the subject areas of the test questions in the semantically expanded knowledge point tags. Subject features include, for example, five features: applicable question structure, applicable question name, applicable grade, applicable region, and difficulty rating. This refers to the total number of features across all subject areas of the test questions. Taking the knowledge point tags for the English subject as an example, since the semantically expanded knowledge point tags include information such as applicable question structure, applicable question type name, applicable grade level, applicable region, and difficulty level, therefore... It is 5, and It is also 5, therefore, =5 / 5=1 may indicate a high degree of subject matching between the subject area of ​​the test question and the subject area of ​​the semantically expanded knowledge point tag.

[0125] For example, if 80% of the keywords in the expanded knowledge point tags belong to the English phonetics domain (such as "pronunciation", "consonant letters", "underlined parts in the question stem"), then A score of 0.8 indicates a high degree of relevance in subject areas for the expanded knowledge point tags. If only 20% of the keywords match, then... =0.2, the expanded knowledge point tags may introduce content unrelated to the subject area (such as "mathematical formulas" or "physical laws" in the field of mathematics).

[0126] In summary, for example, assuming the above English subject knowledge point tags are calculated as follows: Assume the expanded tags have characteristics related to question type and grade level. ,but 46.67 points.

[0127] in this way, It can be adapted to different subjects (such as mathematics, physics, and English), ensuring that the content of the expanded knowledge point tags is highly relevant to the subject area of ​​the test questions. For example, if the expanded knowledge point tags contain many "mathematical formulas" or "physical laws" that are irrelevant to the subject area, and / or do not include enough characteristic information of the subject area, then... The calculated values ​​will decrease, thus affecting the expansion effect score. A decrease in the number of knowledge points may result in the knowledge point tags falling below the predetermined threshold for expansion effect, which may trigger the removal of the expanded knowledge point tags.

[0128] This indicates the semantic coverage of the semantically expanded knowledge point tags compared to the original knowledge point tags, where... ,in, As weight, and , The difference in sentence structure between the semantically augmented knowledge point labels and the original knowledge point labels given by the large language model. The extent to which the semantically augmented knowledge point labels provided for the large language model supplement the original knowledge point labels from a multi-dimensional semantic perspective. The semantically expanded knowledge point labels provided for the large language model supplement the original knowledge point labels to at least one aspect, including subject, question type, and scenario information. The character count penalty coefficient is as follows: the closer the character count of the semantically expanded knowledge point tag is to the predetermined range, the higher the character count penalty coefficient; the further the character count of the semantically expanded knowledge point tag is from the predetermined range, the lower the character count penalty coefficient.

[0129] In one example, for instance, ,and The values ​​are 0.4, 0.4, and 0.2 respectively.

[0130] in, The degree of difference in sentence structure between the semantically expanded knowledge point labels provided by the large language model and the original knowledge point labels can be given by the large language model. The general rule is to compare the sentence structure of the original label and the expanded label, and score them according to the degree of difference in sentence structure. The greater the difference, the higher the score. The sentence structure types are divided into "declarative sentence, verb-object structure, modifier-head structure, and supplementary explanation structure". If the sentence structure is completely different, it gets full marks; if the difference is slight, it gets close to full marks; if the sentence structure is completely the same, it gets low marks. For example, the original label "solving a quadratic equation in one variable" (declarative sentence) and the expanded label "completing the square and formula methods for solving a quadratic equation in one variable" (verb-object structure) have completely different sentence structures and get 95 points.

[0131] The extended knowledge point labels provided by the large language model supplement the original knowledge point labels from multiple semantic perspectives. This can be assessed by the large language model itself: evaluating whether the extended labels supplement the original labels from new semantic perspectives. Supplementary dimensions include "problem-solving methods, applicable conditions, common mistakes, and extended knowledge points." The more supplementary dimensions and the closer they are to the core semantics of the original label, the higher the score. For example, the original label is "solving quadratic equations," and the extended label is "solution methods for quadratic equations (completing the square method / formula method), applicable to equations with integer coefficients, common mistake is missing the discriminant," with three supplementary dimensions (problem-solving methods, applicable conditions, common mistakes), earning a perfect score of 100.

[0132] The extent to which the semantically expanded knowledge point labels provided by the large language model supplement the original knowledge point labels in terms of subject, question type, scenario, and supplementary scenario information can be evaluated by the large language model: assessing whether the expanded labels are suitable for the subject and question type scenario corresponding to the original labels, and supplementing scenario-related information. The stronger the scenario suitability and the more specific the supplementary information, the higher the score; for example, the original label "Triangle congruence determination" (mathematics subject, geometry problem, medium difficulty), the expanded label "SSS determination method for triangle congruence in mathematical geometry problems, applicable to medium difficulty questions", supplemented with 3 scenario information, gets a full score of 100 points.

[0133] The character count penalty coefficient is applied. The closer the character count of the semantically expanded knowledge point tag is to the predetermined range, the higher the character count penalty coefficient; the more the character count of the semantically expanded knowledge point tag exceeds the predetermined range (e.g., the predetermined reasonable range), the lower the character count penalty coefficient. Points are deducted for cases where the expanded tag has too many characters to ensure tag conciseness. For example, the penalty coefficient ranges from 0.6 to 1.0.

[0134] In some embodiments, ,in It is the number of characters (positive integer) of the semantically expanded knowledge point tags. It is a predetermined range, including the number of characters in the original knowledge point tags, and k is the smoothing coefficient. The original knowledge point tag can be a positive integer, considered to be within a predetermined reasonable range. After experimentation, setting k to a fixed value of 4 is considered optimal, as multiple verifications have shown that this setting allows the coefficient to approach 1.0 within a reasonable character count range, and then smoothly decreases beyond that range, adapting to the normal fluctuation range of tag character count.

[0135] in this way, While maintaining semantic alignment with the original knowledge point tags, the expanded knowledge point tags can differ significantly from the original sentence structure (indicating a change in the original sentence structure and an increase in the comprehensiveness of the interpretation information). This allows for multi-dimensional supplementation of scenario information (such as subject, question type, and scenario details), while avoiding information overload caused by redundant or excessively long character counts.

[0136] For example, the final expansion effect score is calculated as follows. Assuming the weighting coefficients are α=0.4, β=0.3, and γ=0.3, the calculated result is Score=0.4⋅85+0.3⋅33+0.3⋅92.5=79.

[0137] Assuming a predetermined expansion effect threshold of 80 points, if the expansion effect score is below 80, the current expanded knowledge point label is removed. Then, a new expanded knowledge point label is generated, or another expanded knowledge point label is input, and the expansion effect score is calculated again. This process continues until the expansion effect score calculated for the generated expanded knowledge point label is greater than or equal to the predetermined expansion effect threshold. Only then are qualified expanded knowledge point labels retained as original knowledge point labels. In this way, through quantitative scoring, expanded knowledge point labels with large semantic deviations (such as "consonant letter writing"), domain mismatches (such as "mathematical formulas"), and insufficient coverage (such as simply repeating the original label) are removed. The retained expanded labels retain the original semantics and are supplemented with scenario information (such as question type), test question and subject information (such as "common question type is multiple choice, and it is a common test point in seventh-grade English, often appearing in easier questions"), significantly improving the comprehensiveness of the annotation.

[0138] In step 215, duplicate knowledge point tags are removed from the semantically validated knowledge point tags to generate an expanded knowledge point set.

[0139] Here, if there are completely duplicate knowledge point tags in the knowledge point tags after semantic validation, the duplicate knowledge point tags can be removed to generate the final expanded knowledge point set.

[0140] In this way, semantic verification and deduplication, through multi-dimensional quantitative evaluation, ensure that the expanded knowledge point tags are of high quality in terms of semantic consistency, domain relevance, and information coverage, thereby providing reliable input for the subsequent selection of knowledge point tags and ultimately improving the accuracy and practicality of test question knowledge point annotation.

[0141] In step 220, based on the rule matching degree and rule recall weight of the knowledge point and the tag text rule, the semantic similarity and semantic recall weight of the knowledge point and the question, and / or the graphic feature similarity and graphic recall weight of the question, the recall similarity is calculated, and knowledge points with recall similarity higher than or equal to the predetermined recall similarity threshold are recalled from the expanded knowledge point set to form a candidate knowledge point set.

[0142] Rule matching measures whether the expanded knowledge point tags conform to the preset tag text rules. For example: Keyword matching: Does the expanded tag contain the core keywords of the original tag (such as "consonant letters" or "pronunciation")? Hierarchical relationship structure matching: Does the hierarchical relationship of the expanded tag (such as "speech → pronunciation → consonant letter pronunciation") match the original tag? Constraint matching: Does the expanded tag meet the preset constraints (such as it must contain "consonant letters")?

[0143] The calculation method is as follows: use a rule engine or regular expressions to match keywords, hierarchical relationships, and constraints, and count the number of matches. Rule match score = number of matches / total number of rules. For example, if the expanded tag matches 3 / 4 of the rules, the rule match score is 0.75.

[0144] Rule recall weights are used to adjust the importance of rule matching in the overall recall similarity.

[0145] Semantic similarity measures the semantic relevance between the expanded knowledge point tags and the test question text.

[0146] The calculation method is as follows: A pre-trained language model (such as BERT) is used to convert the expanded label and question text into vector representations. The cosine similarity between the vectors is then calculated. The formula is as described above and will not be repeated here.

[0147] Semantic recall weights are used to adjust the weights of semantic similarity.

[0148] Graphic feature similarity measures the correlation between images (such as formulas and charts) in test questions and expanded knowledge point labels.

[0149] The calculation method is as follows: Feature vectors of the test question images are extracted using an image recognition model (such as the VL-Embedding model). For example, the extracted vector dimension is 768, consistent with the dimension of the text vectorization. The similarity between the augmented label text and image features is calculated using a text-image cross-modal alignment model (such as Contrastive Language-Image Pre-training (CLIP)).

[0150] The formula can be: .

[0151] Image recall weights are used to adjust the weights of image feature similarity.

[0152] In this way, for multimodal test questions (including text, formulas, and graphs), multimodal features can be effectively integrated to obtain a set of candidate knowledge point tags that better match the test questions.

[0153] In step 230, the relevance between each knowledge point in the candidate knowledge point set and the test question is generated based on the re-ranking model. The Top-K knowledge points with a relevance higher than a predetermined relevance threshold are selected from the candidate knowledge point set to generate a coarse-ranked knowledge point set, where Top-K is a positive integer.

[0154] This paper employs a reranking model to generate the relevance between each knowledge point in the candidate knowledge point set and the test question. The reranking model is a key component in information retrieval (IR) and recommender systems, primarily used for secondary ranking and fine-tuning of candidate results recalled in the initial retrieval stage. It receives paired inputs of a query and a set of candidate documents (or tags, knowledge points), and calculates the relevance score between them through more complex interaction mechanisms, thus ranking the most relevant results first. A cross-encoder architecture is typically used, which jointly encodes the query and candidate text inputs, capturing deep semantic interactions and dependencies between them with higher accuracy, but also with relatively higher computational cost. Here, the reranking model is used to generate the "relevance" between each knowledge point in the candidate knowledge point set and the test question, selecting the Top-K knowledge points for a coarse-ranked set, serving as a "noise eliminator" and "preliminary selection" tool. In at least one embodiment of this application, the re-ranking model acts as a "high-efficiency filter," leveraging its computational efficiency to quickly narrow down the search scope; while the large language model acts as an "intelligent analyst," utilizing its powerful semantic understanding and reasoning capabilities to perform in-depth confidence assessment, stability verification, and final decision-making. The combination of these two models forms a cascaded architecture of "wide recall - fine ranking - deep verification," ensuring both processing efficiency and the accuracy of tag generation.

[0155] The training data for this re-ranking model was constructed by using a large language model distillation technique to score the similarity of a large number of unlabeled "question-label" pairs, which were then used as the training data for the re-ranking model.

[0156] During training, the following methods are employed: For example, the BGE-reranker (Beijing Academy of Artificial Intelligence (BAAI) General Embedding reranker) model is used as the base model for fine-tuning the reranker model. For instance, the input is "question text + label text," and the output is a relevance score (0~1). During training, for example, a batch size of 16 and a learning rate of 5e-6 are used. The mean squared error loss function (used to measure the difference between the model's predicted score and the true score generated by the large language model (distilled labels)) is used to optimize the training of the reranking model, ensuring that the reranking model can accurately distinguish the relevance between labels and questions. The above parameters are example parameters, but this application is not limited to these parameter values.

[0157] Each knowledge point label from the recalled candidate knowledge point set is paired with the test question content and then input into the fine-tuned Reranker model to obtain the relevance score of each "test question-label" pair. The relevance scores are sorted from high to low, and the Top-K knowledge point labels are selected as the coarse-ranked knowledge point set based on the dynamic Top-K number.

[0158] Thus, at least one embodiment of this application employs a re-ranking model (such as a Cross-Encoder based on the BGE architecture) that jointly encodes "question text" and "label text" as paired inputs, enabling deep semantic interaction. This mechanism can identify deep semantic connections that may be missed in the triple recall stage (such as the implicit conditions in the question and the logical derivation relationship of knowledge points), thereby significantly improving the accuracy of relevance scoring and solving the problem of shallow semantic understanding caused by independent feature calculation in the recall stage. With the preceding triple recall steps retaining potentially relevant knowledge points and introducing some "noise" candidates with low relevance or semantic ambiguity, the re-ranking model acts as a filter to refine the candidate set generated by the triple recall. Through the relevance score (0~1) obtained by training the re-ranking model, the re-ranking model can accurately distinguish between "superficially relevant" and "substantially relevant" knowledge points. By combining the Top-K strategy, this step effectively eliminates high-scoring noise labels mistakenly included in the triple recall stage, ensuring that the knowledge point set entering the subsequent fine ranking stage has higher purity and confidence, thus compensating for the insufficient accuracy in the recall stage. Directly using a high-precision re-ranking model to perform full computation on the massive original knowledge point database would incur enormous computational overhead, failing to meet the real-time requirements of large-scale test question processing. At least one embodiment of this application achieves a reasonable allocation of computational resources through a two-stage architecture of "triple recall + re-ranking." Triple recall utilizes lightweight rules and high-concurrency vector retrieval to quickly narrow the search scope, while the re-ranking model performs high-cost computation only on a small number of candidate sets (Top-K). This complementary design ensures both the accuracy of the final result (guaranteed by the re-ranking model) and the efficiency of the system (guaranteed by triple recall), solving the dilemma of low efficiency of a single high-precision model or poor accuracy of a single recall model.

[0159] In step 240, the confidence score of each knowledge point in the coarse-ranked knowledge point set is generated using the large language model. Then, one or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are selected from the coarse-ranked knowledge point set to generate the fine-ranked knowledge point set.

[0160] Specifically, the first step is to construct a large language model for tagging prompts: the prompt template can be fixed and includes complete information about the question, label definitions, and discrimination rules (e.g., "Please read the following question content: [Question Text]. Please determine whether the question belongs to one or more of the following knowledge point labels [Knowledge Point Label Text], and output the confidence score (0~1) and the reason for the judgment based on the question content and label semantics").

[0161] Then, the large language model performs reasoning. Input the "prompt word + question + multiple knowledge point labels (including extended knowledge point labels in the coarse-ranked knowledge point set)" into the large language model for reasoning. The large language model will output which knowledge point labels it judges the question to belong to, as well as the discrimination confidence score (score between 0 and 1) and the reason for the judgment for each knowledge point label.

[0162] In another embodiment, since the coarse-ranked knowledge point set may contain multiple knowledge point tags, at least one embodiment of this application can adopt a step-by-step input method to improve the accuracy of judgment and avoid attention distraction caused by excessively long context. Specifically, the "prompt word + complete question content + single knowledge point tag" are input into the large language model one by one for reasoning. For example, if the coarse-ranked knowledge point set contains tags A, B, and C, three input requests are constructed respectively: the first input contains tag A, the second contains tag B, and the third contains tag C. For each individual input request, the large language model outputs its judgment on whether the question belongs to the specific knowledge point tag, as well as the discrimination confidence score (a score between 0 and 1) and the reason for the judgment for each knowledge point tag. This step-by-step reasoning method helps the large language model focus on the matching degree between a single tag and the question, reducing mutual interference between multiple tags.

[0163] Then, remove low-confidence knowledge point tags. For example, remove knowledge point tags with confidence scores below a predetermined confidence threshold, and filter out one or more knowledge points with confidence scores higher than or equal to the predetermined confidence threshold to generate a fine-ranked knowledge point set, which will then proceed to the subsequent re-ranking stage.

[0164] For example, a predetermined confidence threshold is set (e.g., 0.7 or 0.8, which can be preset or dynamically adjusted according to actual business needs or historical data distribution; an embodiment of its dynamic adjustment will be described later). The confidence score of each knowledge point tag is compared with the predetermined confidence threshold: if the confidence score of a knowledge point tag is lower than the predetermined confidence threshold, it is determined that the knowledge point tag has a weak correlation with the test question or poses a risk of misjudgment, and it is removed; if the confidence score of a knowledge point tag is higher than or equal to the predetermined confidence threshold, the tag is retained.

[0165] Through the above screening process, one or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are selected from the coarse-ranked knowledge point set to generate the fine-ranked knowledge point set. Compared with the coarse-ranked knowledge point set, the fine-ranked knowledge point set has higher semantic relevance and confidence, and removes a large number of noise labels that are superficially related but essentially irrelevant.

[0166] Finally, the generated set of refined knowledge points will enter subsequent processing stages (such as the random sampling inference stage in step 250) for further stability verification and final label decision.

[0167] At least one embodiment of this application utilizes a large language model to score and filter the confidence level of a coarsely ranked set of knowledge points. This effectively leverages the semantic understanding advantages of large language models while compensating for the shortcomings of traditional re-ranking models in deep semantic interaction. By setting a confidence threshold for filtering, the false positive rate can be significantly reduced, ensuring a high purity of the knowledge point set entering the subsequent verification stage. This lays the foundation for generating accurate and reliable test knowledge point labels. Simultaneously, outputting the judgment reasons facilitates subsequent manual review or system debugging, improving the interpretability of the entire labeling process.

[0168] In step 250, in each round of random sampling reasoning in N rounds of random sampling reasoning, after shuffling the order of the knowledge points in the finely sorted knowledge point set, the large language model is used to infer the knowledge points included in the test question and their confidence scores from the shuffled knowledge points. After N rounds of random sampling reasoning, the frequency of occurrence and the fluctuation of the confidence scores of the inferred knowledge points in N rounds of random sampling reasoning are calculated to filter out one or more knowledge points whose frequency of occurrence is greater than or equal to a predetermined frequency threshold and / or whose fluctuation of the confidence scores is less than or equal to a predetermined confidence fluctuation threshold, so as to generate a set of filtered knowledge points, where N is a positive integer.

[0169] Specifically, the refined knowledge point set undergoes N rounds of random sampling and inference (the value of N can be dynamically adjusted, and how to dynamically adjust it will be described in detail later). During each sampling, the order of the knowledge point labels in the refined knowledge point set is randomly shuffled, and the data is input into the large language model to re-determine which (or which) specific knowledge point labels the question belongs to, along with the confidence score (between 0 and 1) and the reasoning for each knowledge point label. This can be achieved by either inputting the "prompt word + question + multiple knowledge point labels" into the large language model all at once, as described in step 240, or by inputting the "prompt word + question + one knowledge point label" into the large language model multiple times to obtain the large language model's judgment that the question belongs to one or more knowledge point labels. The frequency of occurrence and the fluctuation of the confidence score of each knowledge point label judged by the large language model to belong to the question are statistically analyzed. Labels with a frequency ≥ 80% * N (number of samplings) and a confidence score fluctuation ≤ 0.1 are retained, while labels with large confidence score fluctuations are removed (because this is considered a misjudgment caused by model jitter).

[0170] This step aims to evaluate the stability of the large language model's judgment on knowledge point labels through multiple random sampling inferences, thereby eliminating the output bias of the large language model caused by different inputs or arrangement orders of knowledge point labels, and filtering out labels whose judgments by the large language model are unstable. The above sampling inference process can be regarded as a stability-based rejection sampling mechanism.

[0171] For example, in some embodiments, N rounds of random sampling inference are performed on the finely ranked knowledge point set, where the value of N can be dynamically adjusted according to computing resources or accuracy requirements (e.g., N can be 5, 10, or 20). In each round of random sampling inference, the order of the knowledge point labels in the finely ranked knowledge point set is first randomly shuffled. For example, if the finely ranked knowledge point set contains labels A, B, and C, the first round input order might be [A, B, C], the second round might be [C, A, B], the third round might be [B, C, A], and so on. The purpose of shuffling the order is to prevent the large language model from making biased judgments about labels that are earlier or later in the list.

[0172] Subsequently, the shuffled set of knowledge point tags and the test question content are input into the large language model. Here, the method described in step 240—"prompt word + test question + multiple knowledge point tags"—can be used to input the large language model all at once, or "prompt word + test question + one knowledge point tag" can be input multiple times to obtain the large language model's judgment of one or more knowledge point tags to which the test question belongs. For this round of input, the large language model will output which knowledge point tags it judges the test question to belong to, as well as the confidence score (between 0 and 1) and the reasoning for each knowledge point tag.

[0173] After N rounds of reasoning, the frequency of occurrence and the fluctuation of the confidence score of each knowledge point tag as determined by the large language model to belong to the question are statistically analyzed. The frequency of occurrence is the number of times the tag is selected in N rounds of reasoning divided by N. The fluctuation of the confidence score can be measured by calculating the standard deviation, variance, or the difference between the maximum and minimum confidence scores of the tag in N rounds of reasoning.

[0174] A specific example is given below: Suppose we are considering a math problem, "Find the roots of the equation x² - 4 = 0". After step 240, the refined set of knowledge points contains three candidate labels: label A (referred to as quadratic equation), label B (referred to as factorization), and label C (referred to as square root). We set N = 5 rounds of random sampling and reasoning.

[0175] The judgment results and confidence scores of the large language model for these three labels in 5 rounds of inference are recorded in Table 1 below:

[0176] Table 1

[0177]

[0178] The statistical results are as follows:

[0179] Tag A (Quadratic Equation): Frequency of occurrence: 5 times (all selected), frequency = 5 / 5 = 100%. Confidence score: [0.95, 0.92, 0.94, 0.93, 0.96]. Confidence fluctuation: maximum value 0.96, minimum value 0.92, fluctuation value = 0.04.

[0180] Tag B (Factorization): Frequency of occurrence: 3 times (selected in rounds 1, 2, and 4), frequency = 3 / 5 = 60%. Confidence score: [0.85, 0.88, 0.40, 0.86, 0.35] (Note: lower confidence scores in rounds not selected). Confidence fluctuation: maximum value 0.88, minimum value 0.35, fluctuation value = 0.53.

[0181] Label C (square root): Frequency of occurrence: 5 times (all selected), frequency = 5 / 5 = 100%. Confidence score: [0.90, 0.89, 0.91, 0.92, 0.90]. Confidence fluctuation: maximum value 0.92, minimum value 0.89, fluctuation value = 0.03.

[0182] The screening threshold is set as follows: the frequency of occurrence is ≥ 80% * N (i.e., 80%), and the fluctuation of the confidence score is ≤ 0.1.

[0183] Label A: Frequency of occurrence ≥ 80%, confidence score fluctuation ≤ 0.1, retained.

[0184] Tag B: Frequency of occurrence 60% < 80%, confidence score fluctuation 0.53 > 0.1, removed.

[0185] Label C: Frequency of occurrence ≥ 80%, confidence score fluctuation ≤ 0.03, retained.

[0186] Through the above process, a final set of filtered knowledge points is generated, including label A and label C. Label B was removed during this process because its judgment was inconsistent across different rounds (sometimes considered a knowledge point, sometimes not), and its confidence score fluctuated significantly. This was considered a misjudgment due to fluctuations in the large language model or a weak correlation between the label and the test question.

[0187] Thus, at least one embodiment of this application effectively resists the sensitivity of large language models to the input order by introducing N rounds of random sampling inference and a shuffling mechanism, ensuring the reliability of the generated labels. By statistically analyzing the frequency and confidence fluctuations, the stability of the labels can be quantitatively evaluated, eliminating labels whose judgments by the large language model are wavering, thereby significantly improving the accuracy and robustness of the final knowledge point labels.

[0188] In step 260, the large language model is used to vote on every two knowledge points in the filtered knowledge point set. The optimal knowledge point label for the test question is selected from the filtered knowledge point set by combining at least one of the following: the number of times the higher voting score wins, the confidence score, or the degree of relevance. Alternatively, the large language model is used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point labels for the test question from the filtered knowledge point set.

[0189] For example, pair knowledge point tags from the filtered knowledge point set and input them into a large language model to construct prompts (such as "Is the question [Question Content] more suitable for knowledge point tag A [Tag A Definition] or knowledge point tag B [Tag B Definition]? Please give a clear choice and reason"). Each pair of knowledge point tags is voted on, and the number of times each tag with the higher vote score wins is counted. The knowledge point tag with the most wins is selected as the optimal knowledge point tag. If there is a tie, the knowledge point tag with the higher confidence score given by the large language model is selected as the final output. If the confidence scores are the same, the knowledge point tag with the higher relevance score in the coarse ranking stage is selected. If the relevance scores in the coarse ranking stage are still the same, it is considered that there is no optimal knowledge point tag, and the process can proceed to the manual judgment stage.

[0190] This method can obtain an optimal knowledge point label for a test question. However, in some cases, some test questions may involve at least one knowledge point. In such cases, a large language model can be used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point labels for the test question from the filtered knowledge point set.

[0191] This step is the final decision-making stage in the entire knowledge point tag generation process, aiming to determine the final knowledge point tag results from the filtered knowledge point set that has undergone stability verification. Considering that different question types may require different knowledge point tag strategies, at least one embodiment of this application provides two decision-making modes: one is to determine a single optimal knowledge point tag through a pairwise voting mechanism, which is suitable for multiple-choice questions or questions that mainly test a single knowledge point; the other is to determine one or more optimal knowledge point tags through an independent judgment mechanism, which is suitable for comprehensive questions or questions involving one or more knowledge points.

[0192] In some embodiments, when the test questions mainly test a core knowledge point, a pairwise voting mechanism can be used. Specifically, the knowledge point tags in the filtered knowledge point set are paired up in pairs. Assuming that the filtered knowledge point set contains tag A, tag B, and tag C, then three pairs are constructed: (A, B), (A, C), and (B, C).

[0193] For each pairing, construct a prompt word for the large language model, such as: "Is the question [Question Content] more suitable for knowledge point label A [Label A Definition] or knowledge point label B [Label B Definition]? Please give a clear choice and reason." Input the prompt word and the pairing label into the large language model, which will output the winning knowledge point label. Count the number of times each knowledge point label wins in all pairs.

[0194] If a tie occurs (e.g., label A and label B have the same number of wins), the label with the higher confidence score given in step 240 is selected as the final output, based on the confidence score provided by the large language model. If the confidence scores are still the same, the label with the higher relevance score from the coarse ranking stage in step 230 is selected. If the relevance scores are still the same in the coarse ranking stage, it is considered that there is no optimal label, and the process proceeds to the manual judgment stage.

[0195] Of course, the final judgment process described above can also include obtaining a final score by weighting the number of wins, confidence score, and relevance score, and using the knowledge point label with the highest score as the optimal knowledge point label. Examples are not provided here.

[0196] A specific example is given below: Suppose a math problem is "Given the function f(x) = x² + 2x + 1, find its minimum value". After step 250, the set of knowledge points contains three candidate labels: label A (properties of quadratic functions), label B (quadratic equations in one variable), and label C (maximum and minimum values ​​of functions). N rounds of voting are set (3 rounds of pairing in this case).

[0197] In the first round of voting (A vs B), the large language model determined that "properties of quadratic functions" was a better fit because the question involved function graphs and properties, rather than simply solving equations. Tag A wins.

[0198] Second round of voting (A vs C): The large language model judges "properties of quadratic functions" to be more fundamental and broader in scope, or "maximum and minimum values ​​of functions" to be more specific. Assuming the model judges "maximum and minimum values ​​of functions" to be more directly related to the problem statement "find its minimum value," then label C wins.

[0199] In the third round of voting (B vs C), the large language model determined that "function extrema" was more accurate than "quadratic equation". Tag C wins.

[0200] Statistical results: Tag A: 1 win. Tag B: 0 wins. Tag C: 2 wins.

[0201] Ultimately, the label C (function maximum / minimum value) with the most wins was selected as the optimal knowledge point label.

[0202] If a tie occurs, for example, both label A and label C have a score of 1.5 wins (assuming more labels could cause a tie), then compare the confidence scores output in step 240. Assuming label A has a confidence score of 0.85 and label C has a confidence score of 0.90, then label C is selected. If both have a confidence score of 0.90, then compare the relevance scores from step 230. If they are still the same, then mark it as requiring manual review.

[0203] In some cases, certain test questions may involve at least one knowledge point. For example, a comprehensive physics question might simultaneously involve "Newton's Second Law" and "force analysis." In such cases, a large language model is used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point tags for the test question from the filtered knowledge point set.

[0204] Specifically, for each knowledge point tag in the filtered knowledge point set, a prompt is constructed: "Does the question [Question Content] belong to the knowledge point tag [Tag Definition]? Please answer yes or no and give your reason." The large language model independently determines the applicability of each tag. The system collects all tags that answer "yes" as one or more final optimal knowledge point tags.

[0205] A specific example is given below: Suppose a physics exam question is, "An object with a mass of 2kg is subjected to a horizontal pulling force of 10N on a horizontal surface. The coefficient of friction is 0.2. Find the acceleration of the object." The set of knowledge points after step 250 filtering includes: Tag A (Newton's Second Law), Tag B (Friction Calculation), and Tag C (Kinematic Formulas).

[0206] Judgment label A: The large language model analysis question involves the relationship between force and acceleration. The answer is "yes" because "the core of the question is the application of F=ma".

[0207] Judgment label B: The large language model analysis question involves the calculation of friction (f=μN). Answer "yes" because "the friction force needs to be calculated first before the resultant force can be obtained".

[0208] Judgment label C: The large language model analysis of the test question does not involve changes in displacement or velocity. The answer is "no" because "the question only asks for acceleration and does not involve the kinematic process".

[0209] Ultimately, label A and label B were selected as one or more optimal knowledge point labels for this question.

[0210] Thus, at least one embodiment of this application flexibly adapts to the knowledge point distribution characteristics of different test questions through two decision-making modes. By employing a pairwise voting mechanism and leveraging the comparative reasoning capabilities of a large language model, the local optima problem that may exist in single-label scoring is effectively avoided, improving the accuracy of single-label selection. The robustness of the decision-making is further enhanced by introducing confidence scores and relevance levels as tie-breaking mechanisms. The independent judgment mechanism can accurately identify complex test questions involving multiple knowledge points, ensuring the comprehensiveness of the labels. This multi-strategy decision-making mechanism significantly improves the intelligence level and applicability of the knowledge point labeling system, providing high-quality technical support for subsequent intelligent question bank construction, personalized learning path planning, and other educational scenarios.

[0211] In some embodiments, the values ​​of rule recall weight, semantic recall weight, graph recall weight, predetermined recall similarity threshold, Top-K value, predetermined confidence threshold, and N vary based on at least one of the following: text length feature of the test item, number of recalled knowledge points and character count feature, confidence score feature, and predetermined coding value of the test item's subject and / or question type.

[0212] First, there are significant differences in feature distribution among test questions from different subjects (such as mathematics, physics, Chinese, and English) and different question types (such as graphic questions, formula questions, multiple-choice questions, and essays). For example, mathematical graphic questions rely heavily on visual features, while English reading comprehension questions rely more on semantic features. By dynamically adjusting the "rule recall weight, semantic recall weight, and graphic recall weight" based on the "pre-defined coding value of the subject and / or question type," the system can automatically adapt retrieval strategies to different scenarios. For example, it automatically increases the graphic recall weight for mathematical graphic questions and automatically increases the semantic recall weight for Chinese essay questions. This mechanism avoids the problem of decreased accuracy in specific subject recognition caused by a "one-size-fits-all" approach with fixed parameters, significantly enhancing the system's versatility and adaptability across different educational scenarios.

[0213] Second, the complexity of test questions varies, with simple questions requiring less computational resources. By dynamically adjusting the value of "N" (number of random sampling inference rounds) and "Top-K" (number of coarse-ranked knowledge points) based on the text length and the number of recalled knowledge points and characters, computational resources can be allocated on demand. For example, for simple questions with short text lengths and few recalled knowledge points, the value of N or the Top-K value can be reduced, thereby reducing the number of inference rounds in the large language model and the computational load of the re-ranking model; conversely, more resources are allocated for complex questions. This dynamic balance significantly reduces the average computational cost and response time for large-scale test question processing while ensuring label quality.

[0214] Third, the confidence distribution of the model output varies depending on the difficulty of the questions. By dynamically adjusting the "predetermined confidence threshold" and "predetermined recall similarity threshold" based on the "features of the confidence scores," noise can be filtered more accurately. For example, when the overall confidence scores of a certain type of questions are generally low, the threshold can be automatically fine-tuned to avoid mistakenly deleting valid labels; when the confidence scores are generally high but fluctuate greatly, the threshold can be tightened to ensure purity. In addition, feedback adjustments are made based on the "features of the number of knowledge points and the number of characters recalled during the generation of knowledge point labels in the previous round of question generation," giving the system self-correcting capabilities. It can continuously optimize parameters as the processing progresses, thereby reducing false positives (mislabeling) and false negatives (missed labels) and improving the accuracy of the final labels.

[0215] Fourth, multimodal questions contain various types of information, including text and images, and their length and complexity vary considerably. By adjusting relevant weights and thresholds based on the "text length feature of the question," the system can effectively address the semantic dilution problem caused by long texts or the information deficiency problem caused by short texts. For example, for long text questions, the semantic recall weight can be adjusted to enhance deep semantic matching; for short text questions, the rule recall weight can be adjusted to utilize the strong matching characteristics of keywords. This mechanism ensures that the system maintains stable knowledge point recognition performance when facing questions of various lengths and modal combinations, avoiding drastic fluctuations in system performance caused by changes in input data features.

[0216] In some embodiments, the following formulas can be used to adjust the various parameters.

[0217] Rule-based recall weight .

[0218] Semantic Recall Weight .

[0219] Image Recall Weight .

[0220] Predetermined recall similarity threshold .

[0221] Top-K values .

[0222] Predetermined confidence threshold .

[0223] The value of N .

[0224] in, It is related to at least one of the following: text length features of the test item, number of recalled knowledge points and character count features, confidence score features, and predefined coding values ​​of the subject and / or question type of the test item.

[0225] in, The initial value is , It is a weight parameter, and For example, setting The fixed value is 0.5, representing the weight of the question length (reflecting the richness of information in the question and affecting the initial parameter settings). [Settings] The fixed value is 0.5, representing the subject / question type weight (reflecting scenario complexity and adapting to different initial parameters for different subjects). Initially, before going through steps 210-260 above, a structure is constructed using relevant information about the questions, such as the text length features of the questions and the predetermined encoding values ​​of the subject and / or question type. Therefore, based on the above formula, the values ​​of rule recall weight, semantic recall weight, image recall weight, predetermined recall similarity threshold, Top-K value, predetermined confidence threshold, and N value are set.

[0226] After the process of generating knowledge point tags for the test questions in steps 210-260 above, the following settings can be configured: text length features of the test questions, number of knowledge points and character count features recalled during the previous round of generating knowledge point tags, confidence score features calculated during the previous round of generating knowledge point tags, and a predetermined encoding value for the subject and / or question type of the test questions. The value of . Where, The subsequent values ​​other than the initial value are . It is a weight parameter, and In one example, (Regarding the length of the test questions) (Regarding label features) (Regarding model confidence) (Regarding subject / question type).

[0227] in, This is a feature of the text length of the test questions. For example, The normalized value of the test text length is given by the formula: .

[0228] in, It is the number of characters in the question. This is the maximum character limit for the corresponding subject of the test question. For example, it can be set to 500 for math questions.

[0229] This refers to the number of knowledge points and the number of characters recalled during the process of generating knowledge point tags included in the previous round of test questions. For example, It is the normalized value of the label feature, and the formula is: .

[0230] in, It represents the total number of knowledge points recalled during the previous round of generating test questions, which included the knowledge point tags. This is the preset maximum number of knowledge point tags that can be recalled for a single test question (e.g., 20). It is the average number of characters of knowledge points recalled during the process of generating knowledge point tags included in the previous round of test questions. This is the preset maximum character limit for knowledge points (e.g., 50).

[0231] It is a feature of the confidence score in the process of generating knowledge point labels included in the previous round of test questions. For example, It is the normalized confidence value of the model, and the formula is: .

[0232] in, It is the average confidence score of all knowledge points in the knowledge point set during the process of generating test questions in the previous round (for example, the range of confidence scores output by the large model is [0,1]). It is the preset minimum value of the confidence score (e.g., 0). It is the preset maximum value of the confidence score (e.g., 1).

[0233] It is the predetermined code value for the subject and / or question type of the test questions. The preset value can be set as follows: 1.0 for questions in subjects like Mathematics / Physics / Chemistry and those involving graphs, formulas, or problem-solving; 0.8 for questions in subjects like Mathematics / Physics / Chemistry and those involving multiple-choice or fill-in-the-blank questions; 0.9 for questions in subjects like Chinese / English and those involving reading comprehension or composition questions; 0.7 for questions in subjects like Chinese / English and those involving multiple-choice or fill-in-the-blank questions; and 0.85 for questions in other subjects and all question types.

[0234] In some embodiments, if the adjustment coefficient If the difference between the maximum and minimum values ​​is less than 0.05 for several consecutive rounds (e.g., 4 rounds), then these rounds can be used. The mean of the values ​​is used as a fixed coefficient and will not be dynamically changed thereafter, in order to save computing resources.

[0235] In summary, the dynamic adaptive adjustment mechanism of this parameter enables the method of this application to be intelligently optimized based on the real-time characteristics of the input data (including specific test questions and knowledge point labels and the generation process) without relying on fixed empirical parameters. This results in a significant improvement in technical performance across multiple dimensions such as accuracy, efficiency, generalization ability, and robustness, and better adapts to the needs of automatic labeling of test questions and knowledge points on a large scale, in multiple modes, and across multiple disciplines.

[0236] Figure 4 A block diagram of an apparatus for generating knowledge point tags included in test questions according to at least one embodiment of this application is shown.

[0237] The apparatus for generating knowledge point tags included in test questions may include a processor (H1); a storage medium (H2) coupled to the processor (H1) and storing computer-executable instructions therein for performing the steps of various methods of at least one embodiment of this application when executed by the processor.

[0238] The processor (H1) may include, but is not limited to, one or more processors or microprocessors.

[0239] Storage media (H2) may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0240] In addition, the device for generating knowledge point tags included in the test questions may also include (but is not limited to) a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (e.g., keyboard, mouse, speaker, etc.).

[0241] The processor (H1) can communicate with external devices (H5, H6, etc.) via the I / O bus (H4) through a wired or wireless network (not shown).

[0242] The storage medium (H2) may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described in this technology when executed by the processor (H1).

[0243] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0244] In one embodiment, a computer-readable storage medium is provided on which instructions are stored, such as computer-readable instructions. When the computer-readable instructions are executed by a processor, the various methods described above can be performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0245] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0246] Furthermore, the steps and apparatus in the various embodiments of this application are not limited to a single embodiment. In fact, new embodiments can be conceived by combining relevant steps and apparatus in the various embodiments of this application based on the concept of this application, and these new embodiments are also included within the scope of this application.

Claims

1. A method for generating knowledge point tags included in test questions, characterized in that, include: The original knowledge point labels of the test questions are semantically expanded using a large language model to generate an expanded set of knowledge points; Based on the rule matching degree and rule recall weight of the knowledge point and the tag text rule, the semantic similarity and semantic recall weight of the knowledge point and the test question, and / or the graphic feature similarity and graphic recall weight of the knowledge point and the test question, the recall similarity is calculated, and the knowledge points with the recall similarity higher than or equal to the predetermined recall similarity threshold are recalled from the expanded knowledge point set to form a candidate knowledge point set. The re-ranking model generates the relevance degree between each knowledge point in the candidate knowledge point set and the test question. The Top-K knowledge points with a relevance degree higher than a predetermined relevance degree threshold are selected from the candidate knowledge point set to generate a coarse-ranked knowledge point set, where Top-K is a positive integer. The large language model is used to generate a confidence score for each knowledge point in the coarse-ranked knowledge point set, which is the confidence score of the knowledge points included in the test question. One or more knowledge points with confidence scores higher than or equal to a predetermined confidence threshold are selected from the coarse-ranked knowledge point set to generate a fine-ranked knowledge point set. In each round of random sampling reasoning in N rounds of random sampling reasoning, after shuffling the order of the knowledge points in the finely ordered knowledge point set, the large language model is used to infer the knowledge points included in the test question and their confidence scores from the shuffled knowledge points. After the N rounds of random sampling reasoning, the frequency of occurrence and the fluctuation of the confidence scores of the inferred knowledge points in the N rounds of random sampling reasoning are calculated to filter out one or more knowledge points whose frequency of occurrence is greater than or equal to a predetermined frequency of occurrence threshold and / or whose fluctuation of the confidence scores is less than or equal to a predetermined confidence fluctuation threshold, so as to generate a filtered knowledge point set, where N is a positive integer; The large language model is used to vote on every two knowledge points in the filtered knowledge point set. The optimal knowledge point label for the test question is selected from the filtered knowledge point set by combining at least one of the following: the number of times the test question wins due to a higher vote score, the confidence score, or the relevance level. Alternatively, the large language model is used to determine whether each knowledge point in the filtered knowledge point set is included in the test question, thereby selecting one or more optimal knowledge point labels for the test question from the filtered knowledge point set.

2. The method according to claim 1, characterized in that, The original knowledge point tags of the test questions are either generated using the large language model or manually labeled.

3. The method according to claim 1, characterized in that, The step of semantically expanding each original knowledge point label of the test question using a large language model to generate an expanded knowledge point set includes: Extract at least one of the following tag information from each original knowledge point tag of the test question: knowledge point keywords, knowledge point hierarchical relationship, constraints on keywords that must appear after semantic expansion, and subject attributes including at least one of the following: applicable question type structure, applicable question type name, applicable grade, applicable region, and difficulty rating. Obtain explanations of terminology for knowledge points in different disciplines; Based on the tag information and the terminology explanation, the large language model is used to generate semantically expanded knowledge point tags, including at least one of the following: the terminology explanation of the knowledge point keywords for each original knowledge point tag, the description of the hierarchical relationship of the knowledge points, the description of the constraint conditions, and the description of the subject attributes. The semantically expanded knowledge point tags are semantically validated to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, and these are used as the semantically validated knowledge point tags. Duplicate knowledge point tags are removed from the semantically validated knowledge point tags to generate an expanded knowledge point set.

4. The method according to claim 3, characterized in that, The step of performing semantic verification on the semantically expanded knowledge point tags to obtain knowledge point tags whose deviation from the original knowledge point tags is less than or equal to a predetermined deviation threshold, and using these as semantically verified knowledge point tags, further includes: The score for the expansion effect is determined based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the degree of domain matching between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the degree of semantic coverage of the semantically expanded knowledge point tags over the original knowledge point tags. The knowledge point tags whose expansion effect score is greater than or equal to the predetermined expansion effect threshold are used as the knowledge point tags after semantic verification.

5. The method according to claim 4, characterized in that, The step of determining the expansion effect score based on at least one of the following: the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags; the domain matching degree between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tags; and the semantic coverage of the semantically expanded knowledge point tags to the original knowledge point tags. This includes determining the expansion effect score using the following formula: ， Wherein, Score represents the score for the expansion effect. α, β, and γ are weight coefficients, α+β+γ=1, This indicates the degree of semantic consistency between the original knowledge point tags and the semantically expanded knowledge point tags. ,in Vectors representing the original knowledge point labels The vector of the semantically expanded knowledge point tags Cosine similarity between them; This indicates the domain matching degree between the subject domain of the test question and the subject domain of the semantically expanded knowledge point tag, where... , The number of keywords that match between the semantically expanded knowledge point tags and the knowledge point graph in the subject area of ​​the test question; This represents the total number of keywords in the knowledge point map of the subject area of ​​the test question; The number of subject features that match the subject domain of the test question in the semantically expanded knowledge point tags; The number of all features of the subject area of ​​the test questions. This indicates the semantic coverage of the semantically expanded knowledge point tags to the original knowledge point tags, where, ,in, As weight, and , The difference in sentence structure between the semantically expanded knowledge point labels and the original knowledge point labels given by the large language model. The extent to which the semantically expanded knowledge point labels provided by the large language model supplement the original knowledge point labels from a multi-dimensional semantic perspective. The extent to which the semantically expanded knowledge point labels provided by the large language model supplement the original knowledge point labels from at least one aspect, including subject, question type, scenario, and supplementary scenario information. The character count penalty coefficient is used to determine the character count of the semantically expanded knowledge point tags. The closer the character count of the semantically expanded knowledge point tags is to the predetermined range, the higher the character count penalty coefficient is; the further the character count of the semantically expanded knowledge point tags is from the predetermined range, the lower the character count penalty coefficient is.

6. The method according to claim 5, characterized in that, ,in This refers to the number of characters in the semantically expanded knowledge point tags. The predetermined range includes the number of characters in the original knowledge point tags, and k is the smoothing coefficient.

7. The method according to claim 1, wherein, The rule recall weight, the semantic recall weight, the graphic recall weight, the predetermined recall similarity threshold, the Top-K value, the predetermined confidence threshold, and the value of N vary based on at least one of the text length feature of the question, the number of recalled knowledge points and the number of characters, the confidence score feature, and the predetermined coding value of the subject and / or question type of the question.

8. The method according to claim 7, characterized in that, The rule recall weight , The semantic recall weight , The graphic recall weight , The predetermined recall similarity threshold , The Top-K value , The predetermined confidence threshold , The value of N , in, The initial value is , It is a weight parameter, and , in, The subsequent value is , It is a weight parameter, and , in, This is the text length feature of the test question. The features described in the previous round of generating test questions are the number of knowledge points and the number of characters recalled during the process of generating knowledge point tags. This is a feature of the confidence score mentioned in the process of generating knowledge point labels in the previous round of test questions. It is the predetermined code value for the subject and / or question type of the test question.

9. The method according to claim 8, characterized in that, ， in, This is the number of characters in the question. This is the maximum character count threshold for questions in the corresponding subject of the test question. ， in, It represents the total number of knowledge points recalled during the previous round of generating test questions, which included the knowledge point tags. It is the preset maximum number of knowledge point tags that can be recalled for a single test question. It is the average number of characters of knowledge points recalled during the process of generating knowledge point tags included in the previous round of test questions. This is the preset maximum character limit for each knowledge point. ， in, It is the average confidence score of all knowledge points in the refined knowledge point set mentioned in the previous round of generating test questions, which included the knowledge point tags. It is the preset minimum value of the confidence score. It is the preset maximum value of the confidence score. The preset value is 1.0 for questions in the subjects of mathematics / physics / chemistry and questions of graphs / formulas / solutions; 0.8 for questions in the subjects of mathematics / physics / chemistry and questions of multiple choice / fill-in-the-blank; 0.9 for questions in the subjects of Chinese / English and questions of reading comprehension / composition; 0.7 for questions in the subjects of Chinese / English and questions of multiple choice / fill-in-the-blank; and 0.85 for questions in other subjects and all question types.

10. An apparatus for generating knowledge point tags included in test questions, characterized in that, include: Memory, used to store instructions; A processor for reading instructions from the memory and executing the method as described in any one of claims 1-9.

Images