Example screening prompt generation method based on knowledge point graph guidance

Abstract

The invention discloses a sample screening prompt generation method based on knowledge point graph guidance, and belongs to the field of computer natural language processing. The method is divided into two stages: in the first stage, knowledge points are extracted and a knowledge point graph is constructed by analyzing book contents of Common Course Standard Invention, and data are aligned to the graph; in the second stage, input query is mapped to corresponding knowledge points through knowledge point association, samples with the knowledge point overlapping degree being 0 are removed through a sample initial screening model, deep knowledge semantic features are extracted through a self-attention neural network, an optimal sample is screened out from a training set in combination with a knowledge point similarity and diversity comprehensive selection model, and the deep knowledge semantic features are extracted; and inputting a large language model to generate an answer after combining with the query. According to the method, the context learning performance of a large language model is remarkably improved, the problem that sample selection is interfered by surface language characteristics is solved, the knowledge point dimension information is introduced, so that the understanding of the model on tasks is more accurate, the generation illusion is reduced, and experimental verification is achieved.

Description

Technical Field

[0001] The present invention is based on a generative large language model and studies a method for generating sample screening prompts guided by a knowledge point map. It includes two stages in total. In the first stage, the books on "Interpretation of General High School Curriculum Standards" of various subjects are first collected, and the corresponding knowledge points are extracted according to the content of the books. A knowledge point map is constructed on this basis. The data is then aligned to the knowledge point map. In the second stage, the newly input query is passed through the first stage to obtain the corresponding knowledge points, and then the samples with point overlap of 0 are filtered according to the sample initial screening model based on the knowledge point overlap; secondly, the deep semantic features of the knowledge point dimension of each example are extracted through the self-attention neural network (BERT); finally, the comprehensive selection model based on the similarity and diversity of knowledge points is used to screen out samples in the training set, and the samples are combined with the query and sent to the large language model to obtain the answer. The present invention belongs to the field of computer natural language processing, and specifically involves deep learning, prompt learning and other technologies. Background Art

[0002] Large language models have recently demonstrated remarkable capabilities in the field of natural language, primarily in natural language understanding and generation. These models can be adapted through contextual learning, typically using a frozen backbone model that can be applied to a variety of downstream tasks without requiring parameter updates for each task. Accordingly, large language models generate outputs for input queries by conditioning on prompts containing input-output examples. Due to the limited length of context for large language models, only a few examples can be displayed in the prompt. Previous research has shown that the performance of contextual learning is strongly dependent on the context examples selected. Therefore, a pressing challenge is how to select appropriate examples from a large pool of examples to help the model solve the problem. Using the raw input as the basis for selection is a common strategy. Its core approach is to leverage a pre-trained embedding model to extract raw features from the input examples and then select the examples that are most similar to the target. This approach can be flexibly integrated into the processing pipelines of various downstream tasks. However, it also faces the risk of being easily contaminated by surface-level linguistic features that are not closely related to the target task. We believe that better context should focus on logical formal relevance. Currently, logical formal datasets primarily focus on reasoning tasks, so we turned our attention to the Gaokao-Bench dataset. Its difficulty, abstractness, and wide range of subject matter, including computation, reasoning, knowledge assessment, and writing tasks, meet the test standards. Knowledge is incorporated to demonstrate the logical structure of the test questions. The knowledge point dimension provides greater flexibility, adapting to questions of varying types and difficulty levels. By mastering this knowledge point dimension, the large language model can gain a deeper understanding of the question's content and test intent. To this end, a knowledge point map was constructed based on the "Interpretation of the General High School Curriculum Standards," encompassing multiple levels of fine-grained knowledge points covering all high school content. Furthermore, the knowledge points were aligned to the dataset, with each test question including the question, answer, explanation, and knowledge point. Prompts were constructed to guide the large language model to provide more accurate responses. Various approaches were devised to construct prompts based on the knowledge points, including MMR-based selection and Overlap-MMR-based selection. A task description auxiliary model was also designed to better understand the task and reduce hallucinations. Selecting the query's nearest neighbor (KNN) to prompt the LLM has proven effective in standard prompt settings, but experiments with Kmeans demonstrate that diversity also plays a significant role. Therefore, we believe that a comprehensive consideration of similarity and diversity yields better results, which is consistent with the results of the MMR method. We also explored the impact of the MMR method. However, we consider that when the logical form of examples is completely unrelated to the input query, this can lead to erroneous effects on large language models. Therefore, we proposed a two-stage comprehensive example selection framework, which was experimentally proven to be the most effective. Summary of the Invention

[0003] The present invention is different from the existing context sample selection method. In this paper, a sample prompt enhancement method (KEP) based on knowledge retrieval is proposed. First, a knowledge point map is constructed according to the "Interpretation of General High School Curriculum Standards", which contains multi-level fine-grained knowledge points and covers all the content in high school. On this basis, the knowledge points are aligned to the data set, and each test question contains questions, answers, analysis, and knowledge points. Then a two-stage sample selection method is proposed, which includes the filtering of knowledge points and the selection of samples that integrate knowledge, as shown in the attached figure. Figure 2 As shown. Specifically, when a user inputs a question, KEP first searches and filters in the multi-level knowledge point map, excludes those questions with zero overlap with the input question knowledge points, and obtains a set of candidate sets. This means that questions that are completely irrelevant to the knowledge can be excluded, preventing the candidate sets with too low knowledge point relevance from being obtained during diversity selection. Secondly, feature extraction at the knowledge level is performed on the candidate samples. Multi-level sample selection is performed in the feature space of the knowledge points, and the most suitable k samples are obtained by comprehensively considering the similarity and diversity of each level. The overall method is shown in the attached figure. Figure 1 Experimental results show that KEP significantly outperforms existing methods in multiple disciplines, further verifying the importance of knowledge logical relevance and diversity in improving LLM performance.

[0004] In summary, the main process of this method can be divided into three steps: construction and alignment of knowledge point maps based on high school curriculum standards, filtering based on knowledge point overlap, and sample selection method for integrated knowledge.

[0005] (1) Construction and alignment of knowledge point maps based on high school curriculum standards

[0006] The object of this study is high school exam questions. In order to solve the problem of insufficient consideration of the logical relevance of the target task and susceptibility to interference from surface language characteristics, the research needs to further explore how to select examples based on the correlation between logical form and semantic depth to enhance the model's understanding of the core requirements of the task. To this end, the concept of knowledge points is proposed to solve the problem from the perspective of knowledge. Therefore, for knowledge points, knowledge graphs of five subjects are constructed based on the high school curriculum standards and aligned with the questions.

[0007] (2) Filtering based on knowledge point overlap

[0008] This paper proposes filtering based on knowledge point overlap, which is the first step in the screening process. It first removes examples that are completely unrelated to the target problem's knowledge points (examples with a knowledge point overlap of 0), thus reducing the size of the sample pool. While reducing the sample size, it also retains potentially high-quality examples, reducing the complexity of subsequent calculations (such as similarity and diversity). Furthermore, it ensures that the selected examples contain at least some knowledge points that overlap with the target problem, improving the relevance of the context to the problem. In particular, for problems with multiple knowledge points, knowledge point overlap filtering can prioritize examples that cover multiple target knowledge points.

[0009] (3) Sample selection method based on knowledge fusion

[0010] After filtering for knowledge point overlap, we further optimize the similarity and diversity of candidate examples across multiple levels of knowledge points to improve the effectiveness and coverage of contextual hints. By introducing multi-level knowledge point features and combining weighted optimization of similarity and diversity scores, we can select contextual hint examples that are more suitable for the model to generate the target answer.

[0011] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0012] 1) To solve the problem of insufficient consideration of the logical relevance of the target task and easy interference from surface language characteristics, an example enhancement prompt method based on knowledge retrieval is proposed. The present invention improves the quality of the model's answers through information in the knowledge dimension, and the effectiveness of this method is proved through experiments. 2) This invention first established a knowledge point map of the high school curriculum standards, with a total of 5 subjects (Chinese, mathematics, English, history, and chemistry), and used a large model to align the knowledge points with the questions, which is conducive to subsequent retrieval through knowledge points. 3) A two-stage sample selection method is proposed, which includes the filtering of knowledge points and the sample selection method of integrated knowledge, which has improved performance compared with traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the overall approach.

[0014] Figure 2 is a diagram of the two-stage sample selection method. DETAILED DESCRIPTION

[0015] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0016] Step 1: Construction and alignment of knowledge point maps based on high school curriculum standards

[0017] Step 1.1: Knowledge point graph construction

[0018] In the first phase, a multi-level knowledge point map was constructed to analyze the content of the "Interpretation of General High School Curriculum Standards". First, the content of the interpretation of general high school curriculum standards was input into GPT4 through voice or image technology, and the model was used to extract and summarize knowledge points, forming a knowledge point set K = {k1, k2, ..., k n}.

[0019] Step 1.2: Knowledge point graph alignment

[0020] The goal of this stage is to align the test questions Q = {q1, q2, ..., q m} and the knowledge point set K = {k1, k2, ..., k n} to align and build a mapping relationship between questions and knowledge points. Specifically, using the large language model f LLM For each question q i Generate a matching score with the knowledge point:

[0021] P(k j |q i )=f LLM (q i ,Context(k j )),

[0022] Where Context(k j ) is the knowledge point k j The context representation of , including its definition and related attributes. Then, the knowledge point with the highest probability is selected as q i Tags:

[0023]

[0024] According to this alignment method, the knowledge point information contained in the question is obtained.

[0025] Step 2: Filtering based on knowledge point overlap

[0026] Knowledge points, as important features of questions, can effectively improve the ability of large language models to understand and answer questions. However, since questions often involve multiple knowledge points, selecting examples based solely on semantic similarity can result in insufficient knowledge point coverage. To address this, we introduced a knowledge point overlap filtering mechanism in the first step of example selection to significantly reduce the size of the candidate sample pool and focus on context related to the knowledge points.

[0027] Step 2.1: Exclude samples with zero overlap

[0028] For the target problem q, the knowledge point set K(q) = k1, k2, ..., k n, calculate each candidate example q j The knowledge point set K(q j ) and K(q):

[0029] Overlap(q,q j )=|K(q)∩K(q j )|.

[0030] If Overlap(q,q j )=0, then the sample q j Eliminate.

[0031] Step 2.2: Multi-level knowledge point overlap filtering

[0032] Considering the multi-level nature of the knowledge point system, the knowledge points involved in different questions may differ only in the high-level labels. To this end, a weighted overlap calculation method is designed to divide the knowledge point overlap of the sample into two cases: complete overlap and partial overlap.

[0033] Complete overlap: If K(q)∩K(q j ) are consistent at all levels, then it is considered a complete overlap

[0034] Partial overlap: If knowledge points overlap only at certain levels, the degree of overlap is recorded using a weighted hierarchy.

[0035]

[0036] where K l (q) is the set of knowledge points of question q at level l, w l is the level weight. Here, w l Set to 1

[0037] By filtering knowledge points, samples that are completely irrelevant to the target problem or have low knowledge point relevance are excluded from the sample pool, which significantly reduces the search space for sample selection while ensuring the knowledge point relevance of the samples.

[0038] Step 3: Sample selection method based on knowledge fusion

[0039] After filtering for knowledge point overlap, we further optimize the similarity and diversity of candidate examples across multiple levels of knowledge points to improve the effectiveness and coverage of contextual hints. By introducing multi-level knowledge point features and combining weighted optimization of similarity and diversity scores, we can select contextual hint examples that are more suitable for the model to generate the target answer.

[0040] Step 3.1: Feature Extraction

[0041] First, the input text is represented as X={x1,x2,...,x n}, where x i is the i-th word. The feature extraction model BERT model converts each input word into a combination of its word embedding, positional embedding, and segment embedding. The total input is represented as:

[0042] E(x i )=W(x i )+P(x i )+S(x i )

[0043] W(x i ) represents the word embedding vector, P(x i ) represents the position embedding vector, S(x i ) represents the segment embedding vector. The input sequence adds a special token CLS at the beginning to represent the features of the entire sentence. The complete input form is:

[0044] X′=[CLS,x1,x2,...,x n ]

[0045] X'∈Rseq_len*768,x n This means that the sentence is divided into n tokens and the special marker [CLS] is added before each token. The input sequence is then encoded through a 12-layer Transformer encoder. In each layer of the network, the self-attention module is first used to calculate the attention between the text segmentation and other segmentations in the sentence. The formula is:

[0046]

[0047] Where Q, K, V are Query matrix, Key matrix, and Value matrix. Each word vector V and W Q , W K ,W V ∈R 768*768 Linear change matrix obtains Q, K, V matrix ∈ R seq_len*768 .W Q , W K ,W V It is a learnable matrix. T Multiply by to get the attention weight of each word in the sentence, and Dividing the dot product reduces its range, and using softmax ensures the non-negativity of the attention weights while increasing nonlinearity. The attention weight matrix is ​​multiplied by the matrix V, and the weights of each word are used to weight the sum of the features of the words in the sentence to produce the output. Each Transformer encoding layer consists of a two-layer feedforward neural network (FFN):

[0048] FFN(h) = ReLU(hW1 + b1)W2 + b2, where h is the output from the attention module. After 12 layers of encoding, the final context vector for each token is obtained. The final CLS vector is the output of the first token CLS after encoding.

[0049] Use the above process to extract knowledge point features from candidate examples, and use the above CLS to represent the feature vector of the entire knowledge point. Construct a multi-level knowledge point vector representation for each example:

[0050] v l (q j )=BERT(q j , K l (q j )),

[0051] where v l (q j ) represents the sample q j The feature vector at the level l knowledge point. BERT indicates the final output obtained by using the previous neural network.

[0052] Step 3.2: Similarity and diversity calculation

[0053] The goal of similarity calculation is to measure the candidate sample q j The semantic relevance to the target question q, especially around the specific dimensions of the knowledge point. In a multi-level knowledge point system, the similarity score of each layer is calculated separately, and then weighted by the level weight to obtain the overall similarity score. First, the level similarity score is calculated on each level l. j Similarity to the target problem q:

[0054] S sim，l (q,q j )=cos(v l (q), v l (q j )),

[0055] where v l (q) and v l (q j ) are question q and example q respectively j The feature vector at the l-th level knowledge point.

[0056] In order to avoid the selected samples being too similar and lacking diversity, a diversity score is introduced to measure the sample q j Difference from the selected sample set T:

[0057]

[0058] The composite score combines similarity and diversity and performs a weighted summation of the different levels:

[0059] S l (q j )=αS sim,l (q,q j )+βS div,l (q j ,T),

[0060]

[0061] According to the above comprehensive scores, the sample selection is completed by using the greedy iterative optimization algorithm. Input the filtered sample pool D filtered , target problem q, number of samples to be selected, level weight w l , similarity and diversity weights α and β. The value of α is set to increase from 0.1 to 0.9, increasing by 0.1 each time. β = 1-α. The set of selected samples is initialized to be empty: T = {}. In each round, the sample with the highest comprehensive score is selected from the remaining samples:

[0062]

[0063] And add it to the selected sample set: T=T∪{q *}. When |T|=k, stop iteration.

[0064] In this way, we can balance the coverage and information richness of contextual cues while selecting high relevance.

[0065] Here k=3

[0066] Step 3.3: Construction of context sample:

[0067] The first sentence is [Task Description]: 'You are an excellent expert in Chinese, mathematics, English, chemistry, and history. Your task is to answer multiple-choice questions in writing, mathematics, English, chemistry, and history. The test points given will help you think step by step. You will choose the correct answer from A, B, C, D and write it in [Answer] and [Answer]. <eoa> 'Corresponding questions and answers are conducted for the corresponding subjects, and the model is guided to give standardized answers to facilitate result verification. It is worth noting that if there is no specific Task Description and only the general paradigm of in-context learning is followed, the large language model will produce more wrong answers and even "talk to itself" (2) The second sentence: "Here are some examples", which means the end of the description and tells the model to learn context. The second part of the prompt template is the few-shot demonstration. The specific demonstration examples are selected through the previous algorithm. Use the few-shot as a demonstration and combine it with the question and sentence to form a context.< / eoa>

Claims

1. A method for generating sample screening prompts based on knowledge point graph guidance, characterized in that The following steps are involved: (1) Construction and alignment of knowledge point maps based on high school curriculum standards The research object is high school exam questions. The concept of knowledge points is proposed. Based on these knowledge points, knowledge maps of five subjects are constructed based on high school curriculum standards and aligned with the questions. (2) Filtering based on knowledge point overlap We propose filtering based on knowledge point overlap, which is the first step in screening. We first filter out samples that are completely unrelated to the target question's knowledge points. Furthermore, we ensure that the selected samples contain at least some knowledge points that overlap with the target question, thus improving the relevance of the context to the question. (3) Sample selection method based on knowledge fusion After knowledge point overlap filtering, the similarity and diversity of candidate examples are optimized at multiple levels of knowledge points. By introducing multi-level knowledge point features and combining weighted optimization of similarity and diversity scores, contextual prompt examples suitable for the model to generate target answers are selected.

2. The method according to claim 1, characterized in that The following steps are involved: Step 1: Construction and alignment of knowledge point maps based on high school curriculum standards Step 1.1: Knowledge point graph construction In the first phase, a multi-level knowledge point map was constructed to analyze the content of the "Interpretation of General High School Curriculum Standards". First, the content of the interpretation of general high school curriculum standards was input into GPT4 through voice or image technology, and the model was used to extract and summarize knowledge points, forming a knowledge point set K = {k1, k2, ..., k n }; Step 1.2: Knowledge point graph alignment Alignment of college entrance examination knowledge points; the goal of this stage is to align the examination questions Q = {q1,q2,...,q m } and the knowledge point set K={k1,k2,...,k n } to align and build a mapping relationship between questions and knowledge points; specifically, using the large language model f LLM For each question q i Generate a matching score with the knowledge point: P(k j |q i )=f LLM (q i ,Context(k j )), Where Context(k j ) is the knowledge point k j The context representation of , including its definition and related attributes; then, the knowledge point with the highest probability is selected as q i Tags: According to this alignment method, the knowledge point information contained in the question is obtained; Step 2: Filter based on knowledge point overlap, as follows: Step 2.1: Exclude samples with zero overlap For the target problem q, the knowledge point set K(q) = k1, k2, ..., k n , calculate each candidate example q j The knowledge point set K(q j ) and K(q): Overlap(q,q j )=|K(q)∩K(q j )|. If Overlap(q,q j )=0, then the sample q j Eliminate; Step 2.2: Multi-level knowledge point overlap filtering A weighted overlap calculation method is designed to divide the knowledge point overlap of the samples into two cases: complete overlap and partial overlap. Complete overlap: If K(q)∩K(q j ) are consistent at all levels, then it is considered as complete overlap. Partial overlap: If the knowledge points overlap only at some levels, the degree of overlap is recorded by weighting the levels. where K l (q) is the set of knowledge points of question q at level l, w l is the level weight; here, w l Set to 1 Step 3: Sample selection for fusion knowledge, as follows: Step 3.1: Feature Extraction First, the input text is represented as X = {x1, x2, ..., x n }, where x i is the i-th word; the feature extraction model BERT model converts each input word into a combination of its word embedding, positional embedding, and segment embedding; the total input is expressed as: E(x i )=W(x i )+P(x i )+S(x i ) W(x i ) represents the word embedding vector, P(x i ) represents the position embedding vector, S(x i ) represents the segment embedding vector; the input sequence adds a special tag CLS at the beginning to represent the features of the entire sentence; the complete input form is: X′=[CLS,x1,x2,...,x n ] X'∈Rseq_len*768, seq_len represents the overall length of the sentence, 768 is the feature dimension; n It means that the sentence is divided into n tokens and the special marker [CLS] is added in front of the token. Then the input sequence is encoded through a 12-layer Transformer encoder. In each layer of the network, the self-attention module is first used to calculate the attention size of the text segmentation and other segmentations in the sentence. The formula is: Where Q, K, V are Query matrix, Key matrix, Value matrix; each word vector V and W Q , W K ,W V ∈R 768*768 Linear change matrix obtains Q, K, V matrix ∈ R seq_len*768 ;W Q , W K ,W V Belong to the learnable matrix; matrix Q and K T Multiply by to get the attention weight of each word in the sentence, and The division reduces the dot product range; the attention weight matrix is ​​multiplied by the matrix V, and the weight of each word is used to weight the sum of the features of the words in the sentence to obtain the output; each Transformer encoding layer contains a two-layer feedforward neural network (FFN): FFN(h)=ReLU(hW1+b1)W2+b2 in, h is the output from the attention module; after 12 layers of encoding, the final context vector of each token is obtained; the final CLS vector is the output of the first token CLS after encoding; W1, W2 are the weights of the neural network, b1, b2 are considered as bias terms in the neural network; Use the above process to extract knowledge point features from candidate samples, use the above CLS to represent the feature vector of the entire knowledge point; construct a multi-level knowledge point vector representation for each sample: v l (q j )=BERT(q j ,K l (q j )), where v l (q j ) represents the sample q j The feature vector at the first level of knowledge point; BERT indicates the final output obtained by using the previous neural network; Step 3.2: Similarity and diversity calculation The goal of similarity calculation is to measure the candidate sample q j Semantic relevance to the target question q; in a multi-level knowledge point system, the similarity score of each level is calculated separately, and then weighted by the level weight to obtain the overall similarity score; First, the hierarchical similarity score is calculated on each layer l for the sample q j Similarity to the target problem q: S sim，l (q,q j )=cos(v l (q),v l (q j )), where v l (q) and v l (q j ) are question q and example q respectively j Feature vector at the lth level knowledge point; Introduce a diversity score to measure the sample q j Difference from the selected sample set TP: The composite score combines similarity and diversity and performs a weighted summation of the different levels: S l (q j )=αS sim,l (q,q j )+βS div,l (q j ,TP) According to the comprehensive score, the sample selection is completed by using the greedy iterative optimization algorithm; the filtered sample pool D is input filtered , target problem q, number of samples to be selected, level weight w l , similarity and diversity weights α, β; the value of α is set to increase from 0.1 to 0.9, with an increase of 0.1 each time; β = 1-α; the selected sample set is initialized to empty: TP = {}; in each round, the sample with the highest comprehensive score is selected from the remaining samples: And add it to the selected sample set: TP=TP∪{q * }; When |TP| = k, stop iteration; k = 3; Step 3.3: Construction of context sample: The first sentence is [Task Description], which defines the task: to ask and answer questions in the corresponding subjects and guide the model to give standardized answers; to guide the model to output answers in the format; (2) The second sentence: 'Here are some examples', which means the end of the description and tells the model to learn the context; the second part of the prompt template is the few-shot demonstration, and the specific demonstration examples are selected through the previous algorithm; using few-shot as a demonstration, combined with the question and sentence to form the context.

Images