Mathematical problem generation method and system based on adaptive multilevel retrieval enhancement

Abstract

The invention provides a mathematical problem generation method and system based on self-adaptive multi-level retrieval enhancement, and relates to the technical field of large language models.The method comprises the steps that a problem setting request of a user is input into a generation model, and an initial mathematical problem is obtained; judging whether the initial mathematical problem meets a problem setting target based on a multifunctional tutor model; if yes, directly outputting the initial mathematical problem as a target mathematical problem; if not, the multifunctional tutor model outputs optimization suggestions, and a multi-level retrieval enhancement model is triggered; inputting a question setting request into the multi-stage retrieval enhancement model to obtain a retrieval result; and inputting the optimization suggestion and the retrieval result into the generative model for secondary generation and optimization of mathematical problems, and outputting a target mathematical problem. According to the invention, the problem that the accuracy, the matching degree, the quality and the efficiency of the math problem generated at present are poor is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method and system for generating math problems based on adaptive multi-level retrieval enhancement. Background Art

[0002] As an important carrier of knowledge and a means of learning assessment, math problems play an indispensable role as educational resources in the teaching process. Their value is mainly reflected in two aspects: for teachers, math problems can provide feedback on students' learning performance, helping teachers to dynamically adjust teaching strategies and provide personalized guidance for students; for students, practicing math problems can help them discover and improve their weaknesses in a targeted manner, avoid blind training, and improve their learning methods. In addition, math problems are becoming more and more widely used in various learning systems and can be used in various scenarios such as personalized learning resource recommendation, learning theory verification, problem difficulty assessment, knowledge structure mining, and learning path planning. Therefore, the automatic generation of high-quality math problems is of great significance to the realization of large-scale personalized teaching.

[0003] To improve the efficiency and quality of math problem generation, researchers have proposed a variety of generation methods. Traditional math problem generation methods are primarily rule- and template-based, generating math problems by manually abstracting feature templates from existing problems. Alternatively, deep learning techniques such as recurrent neural networks are introduced to generate math problems through a sequence-to-sequence approach. Furthermore, large language models (LLMs) offer a new technical approach to math problem generation. The main principle behind LMMs is: first, by training on massive amounts of text, LLMs are equipped with the ability to understand and generate natural language and acquire rich world knowledge. Then, through techniques such as in-context learning (ICL) and retrieval augmented generation (RAG), relevant domain knowledge is supplemented, guiding content creation in the target domain, and finally, math problems are directly generated. For example, by leveraging GPT-3's hint learning and few-shot learning capabilities to generate math problems, LLMs can generate more diverse and personalized math problems than traditional math problem generation methods.

[0004] However, existing LLM methods focus too much on the generation of mathematical word problems, have a relatively simple task goal orientation, and have little diversity in the generated results, making it difficult to meet actual teaching needs. In terms of technology, LLM is usually pre-trained on general corpus and focuses more on generalized generation. There are no clear constraints and controls on the difficulty of test questions, knowledge point coverage, and other aspects. It lacks an in-depth understanding of specific domain knowledge, which not only leads to problems with the accuracy and relevance of mathematical knowledge in the generated test questions; but also the retrieval mechanism lacks adaptability and cannot better adapt to the mathematical field, resulting in poor quality and efficiency of generated mathematical test questions. Summary of the Invention

[0005] In response to the above-mentioned problems, the present invention provides a method and system for generating math problems based on adaptive multi-level retrieval enhancement, which are used to solve the problems of poor accuracy, relevance, generation quality and generation efficiency of currently generated math problems.

[0006] In one aspect, the present invention provides a method for generating math problems based on adaptive multi-level retrieval enhancement, the method comprising: Input the user's question request into the generative model to obtain the initial math problem; Determining whether the initial math problem meets the problem-setting goal based on the multifunctional tutor model; If the conditions are met, the initial math problem is directly output as the target math problem; If not, the multifunctional tutor model outputs optimization suggestions and triggers the multi-level retrieval enhancement model; the question request is input into the multi-level retrieval enhancement model to obtain the retrieval results; and the optimization suggestions and the retrieval results are input into the generation model to perform secondary generation and optimization of the math problem, and output the target math problem.

[0007] Furthermore, the multi-level retrieval enhancement model includes: A constraint target extractor, configured to locate the constraint condition field according to the question request and extract the constraint target value; An example question recaller is used to calculate the similarity and relevance between the question request and the questions in the question corpus according to the constraint target value to obtain a set of candidate example questions; The question scorer is used to calculate the matching scores of different candidate examples according to the constraint target value, select examples that meet the question setting target from the candidate example set, and output the search results.

[0008] Furthermore, obtaining the candidate example question set includes: A first candidate example question set is obtained based on the similarity between the question request and the questions in the question corpus; a second candidate example question set is obtained based on the relevance between the question request and the questions in the question corpus; The candidate example question set is obtained by merging the first candidate example question set and the second candidate example question set.

[0009] Furthermore, the first candidate example question set is obtained in a manner including: Each example question in the test question corpus Embedded as vector And store all vectors to obtain the Chroma DB vector database; Based on the Chroma DB vector database, a question request is obtained. With each example The similarity is: ; In this way, we get the top k candidate examples with the highest similarity, and get the first candidate example set .

[0010] Furthermore, the second candidate example question set is obtained in a manner including: For each question in the question corpus Perform preprocessing to remove stop words and special symbols; Based on the constraint target value Get a question request With each example The correlation is: ; In this way, we get the top k candidate examples with the highest relevance, and get the second candidate example set .

[0011] Furthermore, the matching score is calculated as follows: Converting the constraint target value extracted from the question request into an embedding space using an embedding model; The candidate example set After deduplication, we get the set of examples to be rearranged. ; Using the embedding model, the example set to be rearranged The constraint target values extracted from each example in are converted to the embedding space; The matching score is obtained by calculating the vector similarity between each constraint target value in the embedding space and weighting them based on the importance of the constraint target value.

[0012] Furthermore, the calculation formula of the matching score is: ; in, Respectively represent the constraint target values: question type, knowledge point and difficulty; Indicates example questions; The weight coefficient representing the constraint target value; represents the vector similarity function; represents the embedding model; Represents the constraint target value extracted from the question request Indicates that from the set of examples to be rearranged The constraint objective values extracted from each example in .

[0013] Furthermore, the target math problem The formula is: ; in, Represents the initial math problem , represents the multi-level retrieval enhancement model Based on the question request From the test corpus The set of examples to be rearranged retrieved from , representing the multifunctional tutor model Based on the question request and the initial math problem Predicted labels and optimization suggestions are given.

[0014] Furthermore, the multifunctional tutor model is obtained by using a large language model as an initialization of the multifunctional tutor model, fine-tuning the model using a domain data set that has been processed both manually and by machines, completing the injection of mathematical knowledge, and training the multifunctional tutor model with domain data to enable the model to effectively judge the quality of the initial mathematical problems and decide whether to activate the multi-level retrieval enhancement model.

[0015] On the other hand, the present invention provides a math problem generation system based on adaptive multi-level retrieval enhancement, comprising a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any one of the above methods.

[0016] In general, the present invention provides a method and system for generating math problems based on adaptive multi-level retrieval enhancement. The technical solution conceived by the present invention can achieve the following beneficial effects compared with the existing technology: (1) This paper designs a plug-and-play adaptive multi-level retrieval enhancement framework. This framework firstly implements simple task filtering and retrieval function activation by fine-tuning the multifunctional tutor model. Secondly, it designs a multi-level retrieval mechanism and a fine-grained feature re-ranking algorithm to perform multi-level retrieval from both structured and unstructured perspectives. Finally, it combines the optimization suggestions of the multifunctional tutor model with the retrieval results to construct prompts, thereby enhancing the math problem generation performance of the large language model. This paper does not require parameter adjustment of the target large language model, but directly improves the accuracy, relevance, generation quality and efficiency of math problems through information retrieval and fusion of fine-grained features.

[0017] (2) The adaptive multi-level retrieval enhancement framework of the present invention can adaptively determine whether retrieval is needed through the decision-making ability of the multifunctional tutor model, which can significantly improve the performance of math problem generation. In addition, the adaptive multi-level retrieval enhancement framework of the present invention has plug-and-play characteristics and is applicable to any open source or closed source large language model. It does not require modifying the parameter weights of the target large language model, which is simpler and more convenient.

[0018] (3) The multi-level retrieval enhancement model proposed in this paper includes a constraint target extractor, an example question recaller, and a question scorer. The constraint target extractor can accurately analyze the user's question-setting requirements and ensure that the retrieved questions meet the desired question-setting objectives. The example question recaller includes similarity retrieval and constraint target retrieval, and the combination of the two improves the example question retrieval performance. The question scorer uses fine-grained feature sorting to optimize the final retrieval results, thereby greatly improving the quality and efficiency of math problem generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a schematic diagram of the method and steps of a method and system for generating math problems based on adaptive multi-level retrieval enhancement provided by the present invention; Figure 2 This is a schematic diagram comparing experimental results of a method and system for generating math problems based on adaptive multi-level retrieval enhancement provided by the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be noted that, in the description of the embodiments of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a method, step, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such method, step, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the method, step, or apparatus comprising the element.

[0023] This paper proposes a multi-objective math problem generation (MMPG) method and system based on adaptive multi-level retrieval augmentation. By incorporating the triple goal of "question type, knowledge point, and difficulty" into the problem generation process, the system addresses the needs of real-world teaching scenarios. To effectively achieve this goal, the paper designs a plug-and-play adaptive multi-level retrieval augmentation framework (AMRAF). This framework first adaptively implements simple task filtering and retrieval activation by fine-tuning a multi-functional tutor model. Secondly, a multi-level retrieval mechanism and a fine-grained feature reranking algorithm are designed to perform multi-level retrieval from both structured and unstructured perspectives. Finally, the optimization suggestions from the multi-functional tutor model are combined with retrieval results to construct prompts, enhancing the math problem generation performance of a large language model. This method eliminates the need for parameter adjustment of the target large language model and instead directly improves the quality and efficiency of math problem generation through information retrieval and fusion of fine-grained features.

[0024] Specifically, if Figure 1 As shown, the method of the present invention includes: Input the user's question request into the generative model to obtain the initial math problem; Determine whether the initial math problem meets the problem setting goal based on the multifunctional tutor model; If satisfied, the initial math problem is directly output as the target math problem; If not, the multifunctional tutor model outputs optimization suggestions and triggers the multi-level retrieval enhancement model; the question request is input into the multi-level retrieval enhancement model to obtain the retrieval results; and the optimization suggestions and retrieval results are input into the generation model for secondary generation and optimization of math problems, and the target math problems are output.

[0025] It should be noted that the existing method of generating multi-objective math problems based on RAG is to generate the model and retrieval model Specifically, firstly, a question request containing a set of question targets is given. and a corpus of test questions containing a large number of example questions ; then retrieve the model From the test corpus Search and question request Related examples ;Finally generate the model Based on question request and the retrieved examples , generate a math problem that meets the problem setting goal ; The form of the math problem can be defined as: ;in, Indicates the question type; Indicates knowledge points; Indicates difficulty.

[0026] In order to improve the accuracy, relevance, generation quality and generation efficiency of generated math problems, on the first hand, the present invention introduces a multifunctional tutor model on this basis to determine when in-depth retrieval is needed, thereby forming a more flexible formula. On the second hand, the present invention also optimizes the retrieval model, performs multi-level retrieval from structured and unstructured perspectives, and retrieves content that is more closely related to the question-setting objectives. On the third hand, the present invention inputs the optimization suggestions and retrieval results output by the multifunctional tutor model into an adaptive generation model, thereby performing secondary generation and optimization of the math problems.

[0027] In other words, in order to achieve the goal of setting multi-objective mathematics questions, the AMRAF framework designed in the present invention mainly includes three core components: a multi-functional tutor model, a multi-level retrieval enhancement model, and an adaptive generation model.

[0028] S10, input the user's question request into the generation model to obtain the initial math problem. That is, according to the user's multi-objective constraint question request , using generative models Directly generate initial math problems .

[0029] It should be noted that in order to respond to the diverse needs of real teaching scenarios, the three-dimensional constraints of question type, knowledge points and difficulty need to be fully considered in the process of generating math questions. Therefore, the question request of the present invention At least include question type, knowledge points and difficulty.

[0030] Generative Models That is, the generation model in the existing method of generating multi-objective math problems based on RAG , which is already known in the prior art and will not be described in detail here.

[0031] S20, based on the multifunctional tutor model, determines whether the initial math problem meets the problem setting goal.

[0032] If satisfied, the initial math problem is directly output as the target math problem; If not, the multifunctional tutor model outputs optimization suggestions and triggers the multi-level retrieval enhancement model.

[0033] When generating target math problems, the timing of activating the multi-level search enhancement model and the relevance of the search content are crucial. Blindly enhancing search results can complicate simple problems and negatively impact some tasks. Therefore, this paper designs a multifunctional tutor model to provide intelligent supervision and assistance.

[0034] The multifunctional tutor model is used to determine whether the initial math problem meets the problem-setting objectives, and output optimization suggestions based on the results of whether the objectives are met.

[0035] It's important to note that the multifunctional tutor model uses the initial output of the generated model as a benchmark to assess whether to activate the multi-stage retrieval enhancement model to optimize resource utilization. Furthermore, the multifunctional tutor model analyzes the initial generated math problems and provides detailed revision suggestions to improve output quality. This multifunctional design significantly enhances the efficiency and accuracy of the target math problem generation process.

[0036] As an embodiment, the multifunctional tutor model is obtained by using a large language model as an initialization multifunctional tutor model, fine-tuning the model using a domain data set that has been processed both manually and machine-processed, completing the injection of mathematical knowledge, and enabling the multifunctional tutor model to effectively judge the quality of the initial mathematical problems and decide whether to activate the multi-level retrieval enhancement model through training with domain data.

[0037] For example, large language models such as Qwen1.5-14B-Chat and Deep Seek V2-16B-Chat can be used to initialize the multifunctional tutor model for training, so that the multifunctional tutor model can become a model that can determine whether the math problems match the question objectives, and realize its multiple functions through fine-tuning with domain data.

[0038] The domain data is constructed by constructing the MMPG dataset to train the Multifunctional Tutor model. For example, 9,000 math problems were first collected and manually annotated by question type, knowledge points, and difficulty. GPT-4 was then used for consistency checking, retaining 90% of the annotated problems as the corpus. 7,000 generated targets with inconsistent question designs were randomly selected as negative samples. The data was expanded to 16,000 samples and split into a training and test set at a ratio of 7:1. Finally, the data was formatted into Alpaca-style question-answer pairs, allowing the Multifunctional Tutor model to effectively judge the quality of the initial math problems and decide whether to activate the multi-level retrieval enhancement model.

[0039] Through learning and training on the dataset, the initialized tutor model can better judge the quality of questions given by the target large language model, and activate the subsequent multi-level retrieval enhancement model in an adaptive manner, so that the context is enhanced only when necessary to optimize performance, thereby improving the robustness of the multi-level retrieval enhancement model in the multi-target math problem generation task.

[0040] It should be noted that the setting target is the question type, knowledge points, and difficulty level extracted from the question request. Determining whether the initial math question meets the setting target means determining whether the question type, knowledge points, and difficulty level of the initial math question match those in the question request.

[0041] Specifically, the multifunctional tutor model is used to determine the initial math problems Whether the question goal is met, adaptively decide whether to activate the multi-level retrieval enhancement model to optimize resource utilization. If it is met, the retrieval label is output ; If not satisfied, output optimization suggestions and search tags , triggering a multi-level retrieval enhancement model.

[0042] For example, if the target question type is a multiple-choice question, the knowledge point involves trigonometric functions, and the difficulty level is simple; while the initial math question type is also a multiple-choice question, the knowledge point involves quadratic functions, and the difficulty level is simple; then the optimization suggestion is: emphasize that the knowledge point is trigonometric functions, so that the knowledge point of the output target math question involves trigonometric functions.

[0043] S30: Input the question request into the multi-level retrieval enhancement model to obtain the retrieval results.

[0044] The multi-level retrieval enhancement model integrates the retrieval system of structured and unstructured perspectives. When the output retrieval label is When , the multi-level retrieval enhancement model is activated.

[0045] As an embodiment, the multi-stage retrieval enhancement model includes a constraint target extractor, an example question recaller, and a question scorer.

[0046] The constraint target extractor is used to locate the constraint condition field according to the question request and extract the constraint target value, which provides key information support for the retrieval of the example question recaller and the scoring of the question scorer. In other words, using the constraint target extractor Request for question Perform constraint target extraction to obtain constraint target value ; Among them, the constraint target value Including constraints on question type T, knowledge point K and difficulty D.

[0047] In order to retrieve as many test questions as possible that meet the constraint target value in the test question corpus, help the example question recaller return the most relevant example questions, and alleviate the problem of irrelevant results when using only vector semantic retrieval, the constraint target extractor Used to extract the specific values of the three constraint fields (question type, knowledge point, and difficulty) from the question request for direct use by the example question recaller and question scorer.

[0048] For example, the question request is: [I want to ask a question about quadratic equations. The question type is multiple choice and the difficulty is about 0.5]; then the constraint target extractor The fields extracted from the question request are [Question type: Multiple choice; Knowledge point: Quadratic equation; Difficulty: 0.5].

[0049] As a specific embodiment, a method for extracting the constraint target value can be: using the multifunctional tutor model as a keyword extraction model to extract the question type, knowledge points, and difficulty level from the question request. Specifically, a prompt template can be constructed to prompt the multifunctional tutor model to extract the corresponding keywords from the question request.

[0050] The example question recaller calculates the similarity and relevance between the question request and the questions in the question corpus based on the constraint target value, thereby obtaining a set of candidate example questions. In other words, the example question recaller uses similarity and relevance to generate a hybrid retrieval strategy for the question request, thereby retrieving basic candidate example questions.

[0051] Different from traditional RAG, the present invention segments the questions into complete examples, thus preserving the complete semantics of the questions. represents the test question corpus, Represents a complete example.

[0052] The candidate example question set includes a first candidate example question and a second candidate example question. In one embodiment, obtaining the candidate example question set includes: obtaining the first candidate example question set based on the similarity between the question request and the test questions in the test question corpus; obtaining the second candidate example question set based on the relevance between the question request and the test questions in the test question corpus; and merging the first candidate example question set and the second candidate example question set to obtain the candidate example question set.

[0053] As an embodiment, the first candidate example set is obtained by using a similarity search engine, specifically by: Each example in the test corpus Embedded as vector , and store all vectors to obtain the Chroma DB vector database; for example, you can use Open AI's text-embedding-ada-002 model to store the example Embedded as vector .

[0054] Based on the Chroma DB vector database, get the question request With each example The similarity is: ; In this way, we get the top k candidate examples with the highest similarity, and get the first candidate example .

[0055] While semantic similarity retrieval enables multilingual and multimodal search and is robust to spelling errors, it is limited in capturing subtle differences between mathematical concepts, such as the confusion between "solutions to fractional equations" and "solving fractional equations." Furthermore, this method relies on the quality of vector embeddings and is sensitive to out-of-domain terminology. To overcome these shortcomings, the present invention introduces a retrieval method based on constrained target values, resulting in a second set of candidate questions with higher relevance.

[0056] As an embodiment, the second candidate example set is obtained by using a constrained target retriever, and the specific method of obtaining the second candidate example set includes: Test Question Corpus Each question in Perform preprocessing to remove stop words and special symbols; Based on the constraint target value Get a question request With each example The correlation is: ; In this way, we get the top k candidate examples with the highest relevance, and get the second candidate example set .

[0057] Merge multiple examples with high similarity and relevance to obtain a set of candidate examples .

[0058] The question scorer is used to calculate the matching scores of different candidate examples based on the constraint target value, select examples that meet the question setting target from the candidate example set, and output the retrieval results. In other words, in order to comprehensively consider fine-grained features such as question type, knowledge points and difficulty, the relevance and similarity of the mixed retrieval results are uniformly evaluated, the mixed retrieval results are optimized and the number of examples injected into the prompt is controlled. The present invention also designs a fine-grained feature question scorer, which calculates the matching score for each recalled example by considering multiple fine-grained features such as question type, knowledge points and difficulty, thereby outputting the content most relevant to the user's question request.

[0059] As an example, the matching score is calculated as follows: Use the embedding model to convert the constraint target value extracted from the question request into the embedding space; Candidate Example Set After deduplication, we get the set of examples to be rearranged. ; Using the same embedding model, we will The constraint target values extracted from each example in are converted to the embedding space; The matching score is obtained by calculating the vector similarity between each constraint target value in the embedding space and weighting them based on their importance. The vector similarity can be a cosine similarity function, a Euclidean distance function, or an inner product function, with cosine similarity being preferred.

[0060] More specifically, the matching score is calculated as follows: ; in, Respectively represent the constraint target values: question type, knowledge point and difficulty; Indicates example questions; The weight coefficient representing the constraint target value; represents the vector similarity function; Represents an embedding model, such as text-embedding-ada-002; Represents the constraint target value extracted from the question request Indicates the set of examples to be rearranged The constraint objective values extracted from each example in .

[0061] S40: Input the optimization suggestions and search results into the generation model for secondary generation and optimization of the math problem, outputting the target math problem. The present invention regenerates the math problem based on the optimization suggestions and search results output by the multifunctional tutor model to improve output quality and ensure that the resulting math problem meets multiple constraints specified by the user, such as problem type, knowledge points, and difficulty.

[0062] As a specific embodiment, the target math problem The formula is: ; in, Represents the initial math problem , represents the multi-level retrieval enhancement model Based on question request From the test corpus The set of examples to be rearranged retrieved from , representing a multifunctional mentor model Based on question request and initial math problems Predicted labels and optimization suggestions are given.

[0063] Through the close cooperation between the multifunctional tutor model, the multi-level retrieval enhancement model and the adaptive generation model, the present invention constructs an adaptive retrieval-enhanced math problem generation method that can effectively balance generation quality and generation efficiency. It can not only accurately respond to the user's multi-dimensional needs, but also achieve efficient utilization of computing resources through an intelligent retrieval decision-making mechanism.

[0064] In another aspect, the present invention provides a math problem generation system based on adaptive multi-level search enhancement, comprising a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the aforementioned methods. The technical features of the system are consistent with those of the method and are not further detailed here.

[0065] To validate the effectiveness of the adaptive multi-level retrieval enhancement framework, we randomly generated a math problem generation capability assessment dataset containing 200 evaluation samples. This dataset includes diverse question generation requests across six question types, 193 different knowledge points, and difficulty levels ranging from 0.1 to 1. This effectively evaluates the generative model's ability to generate multi-objective math problems.

[0066] The present invention uses GPT-4 as a quality evaluator for model-generated content. GPT-4 is set as an expert in the quality review of math test questions. It can score each generated test question (out of 10 points) from the following dimensions: whether it meets the constraints required by the user (including question type, knowledge points and difficulty), the innovation, readability, applicability and helpfulness of the generated questions. Finally, the average score is calculated by combining the scores of each dimension as an evaluation indicator. On the one hand, the present invention guides GPT-4 to conduct a step-by-step analysis and evaluation of the solvability rate of the generated math test questions by designing sophisticated prompt words; it also calculates the average perplexity (PPL) of the model-generated test questions and uses it as one of the evaluation indicators. On the other hand, the present invention also evaluates the time (in minutes) required for each large language model to complete the multi-objective math problem generation task in the evaluation data under different methods as a measure of the efficiency of the method.

[0067] This paper selected two categories of baseline models for comparison: search-free baselines (generating math problems without relying on example samples from an external test corpus) and search-based baselines (first retrieving relevant example questions from an external corpus and then generating math problems based on these examples). For closed-source models, we evaluated GPT-3.5-Turbo developed by Open AI and Gemini-Pro developed by Google; both excel at mathematical reasoning. For open-source models, we evaluated several high-performing pre-trained conversational language models, including ChatGLM3-6B, Baichuan2-7B-Chat, Baichuan2-13B-Chat, and Llama3-8B-Instruct.

[0068] The experiment is divided into three parts: generation performance evaluation experiment, ablation experiment and hyperparameter exploration experiment.

[0069] (1) Generate performance evaluation experiments Table 1 Overall experimental comparison results of multiple large language models

[0070] Table 1 shows the overall experimental results of the six large language models on the multi-objective math problem generation task evaluation dataset. The bold numbers represent the best performance among the large language models. It can be seen that: First, the present invention performs well on multiple benchmark models, with overall performance improvement, especially on open source and closed source large models (such as GPT-3.5, Gemini-Pro, Baichuan2, and ChatGLM3), achieving higher scores and solvability rates.

[0071] Second, compared with the non-retrieval method, the present invention significantly improves the quality of generated questions by adopting a multi-level retrieval enhancement method, making the math problems more in line with teaching needs; compared with the full retrieval method, the adaptive multi-level retrieval effectively reduces unnecessary retrieval and improves computing efficiency.

[0072] Third, the present invention reduces the perplexity level (PPL) without affecting the quality, indicating that the generated math problems are more fluent and in line with language expression habits.

[0073] Fourth, while improving the quality, the present invention significantly reduces the computational overhead, making the generation time much shorter than the traditional full-data retrieval enhancement method, and has higher practicality.

[0074] Table 2 Comparison of average quality scores and similarity of large language models

[0075] Table 2 shows the average quality scores of math problems regenerated by the large language generation model based on the candidate examples provided by the multi-level retrieval enhancement model, as well as their similarity to the candidate examples. BF1 is the BERT Score F1, which indicates similarity. It can be seen that: First, the present invention counts the average quality scores of new math problems regenerated by the generation model based on the candidate examples provided by the multi-level retrieval enhancement model after the multi-level retrieval enhancement model is activated, showing that the overall quality of the regenerated new math problems is high and can meet the user's multi-objective needs.

[0076] Second, after activating the multi-level retrieval enhancement model, it provides the generative model with relevant example questions for reference and regenerates a new test question. The BERT F1 score shows that the math questions generated by each large language model generally have low similarity to the candidate example questions. This indicates that although the large language model references the candidate example questions, the generated new math questions differ significantly from the candidate examples in content and form, reflecting the emergence of new ideas and expressions when referring to the candidate examples.

[0077] Combining the analysis of average quality scores and similarity, we conclude that introducing candidate examples effectively inspires the model to generate novel, high-quality math problems, rather than simply imitating them. This validates the rationality and effectiveness of our method, which helps provide a wider range of exercises and promotes comprehensive learning and understanding among students.

[0078] (2) Ablation experiment Table 3 Comparison results of ablation experiments

[0079] Table 3 shows ablation experiments on key components of the generative adaptive multi-stage retrieval enhancement framework on GPT-3.5-Turbo (a closed-source model with over 7B parameters) and ChatGLM3-6B (an open-source model with 6B parameters). "-" indicates that the corresponding component is eliminated from the framework. MM, OR, and SR represent the multifunctional tutor model, the constrained target retriever in the example recaller, and the similarity retriever in the example recaller, respectively. It can be found that: First, after removing the multifunctional tutor model, the accuracy and solvability of the generated questions decreased, indicating that the multifunctional tutor model played a key role in determining whether retrieval was needed and providing optimization suggestions; after removing the constrained target retriever, the matching degree between the generated math problems and user needs decreased, indicating that this module is crucial in ensuring that the questions meet the question type, knowledge points and difficulty requirements; after removing the similarity retriever, the relevance of the retrieved candidate examples decreased, affecting the quality of the final generated content, indicating that similarity retrieval helps improve the applicability of example questions.

[0080] Second, the various models within the complete adaptive multi-stage retrieval enhancement framework achieved the best performance in terms of generation quality, solvability, and computational efficiency, demonstrating that the collaborative work of various components is key to improving performance. The multifunctional tutor model and constrained target retriever contributed most to the quality improvement, while the similarity retriever primarily impacted the relevance of candidate examples and retrieval efficiency.

[0081] (3) Hyperparameter exploration experiment In order to study the impact of using different measurement methods on retrieval performance in similarity-based retrievers, this paper conducts a comparative experiment on three commonly used vector similarity measurement methods: cosine similarity (CS), Euclidean distance (ED) and inner product (IP).

[0082] Table 4 Comparison results of hyperparameter exploration experiments

[0083] Table 4 compares the experimental results of using different vector similarity metrics in the similarity retriever on two large language models, GPT-3.5-Turbo and ChatGLM3-6B. CS represents the use of cosine similarity, ED represents the use of Euclidean distance, and IP represents the use of inner product.

[0084] Experimental results show that cosine similarity performs best when calculating the similarity of text embedding vectors. It can more accurately identify the candidate examples closest to the question request, thereby improving the relevance and controllability of generated math problems. The superior performance of cosine similarity can be attributed to its greater focus on directional similarity rather than size, which makes it particularly suitable for semantic matching tasks in math problem retrieval. In addition, cosine similarity shows more stable performance on models of different scales, indicating that it may be a more suitable choice for various deployment scenarios. These findings justify the rationality of the present invention's prioritization of cosine similarity as the primary similarity metric in the proposed framework.

[0085] In order to analyze the impact of different numbers of candidate examples on the performance of the present invention, different numbers of candidate examples (top-1 to top-10) were recalled and provided to the target large language models (GPT-3.5-Turbo and ChatGLM3-6B) to evaluate their scoring performance and solvability performance of generated math problems.

[0086] like Figure 2 (A) and (B) respectively show the effects of different numbers of candidate examples input into the generative model on the quality and solvability of the generated math problems. Figure 2 Experimental results show that a moderate number of candidate examples (e.g., three) improves generation quality and solvability, while too many examples can cause interference. Similar results were obtained on other target large-scale models, providing a useful reference for selecting the number of examples in subsequent similar tasks.

[0087] It should be noted that for the aforementioned embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0088] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative, and the division of the units described is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0090] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0093] Those skilled in the art will appreciate that all or part of the various circuits in the above embodiments may be implemented by instructing related hardware through a program. The program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0094] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0095] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating math problems based on adaptive multi-level retrieval enhancement, characterized in that: The method comprises: Input the user's question request into the generative model to obtain the initial math problem; Determining whether the initial math problem meets the problem-setting goal based on the multifunctional tutor model; If the conditions are met, the initial math problem is directly output as the target math problem; If not, the multifunctional tutor model outputs optimization suggestions and triggers the multi-level retrieval enhancement model; the question request is input into the multi-level retrieval enhancement model to obtain the retrieval results; and the optimization suggestions and the retrieval results are input into the generation model to perform secondary generation and optimization of the math problem, and output the target math problem.

2. A method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 1, characterized in that: The multi-level retrieval enhancement model includes: A constraint target extractor, configured to locate the constraint condition field according to the question request and extract the constraint target value; An example question recaller is used to calculate the similarity and relevance between the question request and the questions in the question corpus according to the constraint target value to obtain a set of candidate example questions; The question scorer is used to calculate the matching scores of different candidate examples according to the constraint target value, select examples that meet the question setting target from the candidate example set, and output the search results.

3. A method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 2, characterized in that: Obtaining the candidate example set includes: A first candidate example question set is obtained based on the similarity between the question request and the questions in the question corpus; a second candidate example question set is obtained based on the relevance between the question request and the questions in the question corpus; The candidate example question set is obtained by merging the first candidate example question set and the second candidate example question set.

4. A method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 3, characterized in that: The first candidate example question set is obtained in the following manner: Each example question in the test question corpus Embedded as vector And store all vectors to obtain the Chroma DB vector database; Based on the Chroma DB vector database, a question request is obtained. With each example The similarity is: ; In this way, we get the top k candidate examples with the highest similarity, and get the first candidate example set .

5. The method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 3, characterized in that: The second candidate example question set is obtained in the following manner: For each question in the question corpus Perform preprocessing to remove stop words and special symbols; Based on the constraint target value Get a question request With each example The correlation is: ; In this way, we get the top k candidate examples with the highest relevance, and get the second candidate example set .

6. The method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 2, characterized in that: The matching score is calculated as follows: Converting the constraint target value extracted from the question request into an embedding space using an embedding model; The candidate example set After deduplication, we get the set of examples to be rearranged. ; Using the embedding model, the example set to be rearranged The constraint target values extracted from each example in are converted to the embedding space; The matching score is obtained by calculating the vector similarity between each constraint target value in the embedding space and weighting them based on the importance of the constraint target value.

7. A method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 6, characterized in that: The calculation formula of the matching score is: ； in, Respectively represent the constraint target values: question type, knowledge point and difficulty; Indicates example questions; The weight coefficient representing the constraint target value; represents the vector similarity function; represents the embedding model; represents the constraint target value extracted from the question request; Indicates that from the set of examples to be rearranged The constraint objective values extracted from each example in .

8. The method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 1, characterized in that: The target math problem The formula is: ； in, represents the initial math problem; , represents the multi-level retrieval enhancement model Based on the question request From the test corpus The set of examples to be rearranged retrieved from , representing the multifunctional tutor model Based on the question request and the initial math problem Predicted labels and optimization suggestions are given.

9. The method for generating math problems based on adaptive multi-level retrieval enhancement according to claim 1, characterized in that: The multifunctional tutor model is obtained by using a large language model as an initialization of the multifunctional tutor model, fine-tuning the model using a domain data set that has been processed both manually and by a machine, and injecting mathematical knowledge. Through training with domain data, the multifunctional tutor model is equipped with the function of effectively judging the quality of the initial mathematical problems and deciding whether to activate the multi-level retrieval enhancement model.

10. A math problem generation system based on adaptive multi-level retrieval enhancement, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images