Method and system for testing mathematical literal question solver

Abstract

The invention provides a test method and system for a mathematical character question solver, and the system comprises a data preprocessing module which is used for integrating a multi-source mathematical character question data set, carrying out the standardized processing and marking of questions in the mathematical character question data set, and outputting a data instance with a unified data structure; the solver module is used for decomposing a solving algorithm into different basic function components as algorithm components, calling related algorithm components to form a corresponding solver according to a logic sequence of a to-be-tested solving algorithm, and solving a mathematical problem by using the solver based on an input data instance to generate a solving answer; and the perspective confusion comparison module is used for receiving the solving answer and comprehensively evaluating a solver corresponding to the algorithm to be measured through three dimensions of question type classification, key question solving steps and an algorithm component. According to the invention, multi-source data sets can be integrated, automatic and reproducible test workflows are provided, and fine-grained and multi-dimensional performance tests are supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a mathematical word problem solver testing method and system. BACKGROUND

[0002] In the field of artificial intelligence-driven algorithm research, the traditional algorithm development paradigm usually includes multiple stages such as problem definition, literature review, theoretical modeling, algorithm design, implementation, experimental evaluation, and summary report. Although artificial intelligence technology has improved efficiency in some aspects, the entire research paradigm has not undergone fundamental changes. This means that each research team still needs to independently and completely undertake the full process of algorithm design to performance evaluation, resulting in a lot of effort being wasted on repetitive tasks such as algorithm implementation, data preparation, and experimental configuration, causing serious resource waste and efficiency bottlenecks.

[0003] The research of mathematical word problem solvers is a typical representative of this paradigm. In recent years, numerous solving methods have emerged in this field, mainly including rule-based methods, deep learning-based methods, and recently emerging large language model-based methods. However, the current research and practice have the following technical pain points: Evaluation standards are fragmented, and results are difficult to reproduce and fairly compare. There are multiple public datasets in this field, but their problem types, difficulty distributions, and annotation specifications are different. Researchers usually choose to test their algorithms on different datasets or construct datasets of specific formats. This phenomenon makes it difficult to objectively and fairly judge the merits and demerits of different algorithms, and seriously hinders research progress.

[0004] Evaluation dimensions are single, and lack depth diagnosis ability. Existing research generally uses "final answer accuracy" as the core or only evaluation indicator. This "black box" evaluation can only reflect the macro performance of the algorithm and cannot reveal specific defects in the algorithm on specific problem types (such as multi-step reasoning problems), key problem solving steps (such as quantity relationship extraction, equation column writing), or internal functional components.

[0005] R&D and evaluation are too tightly coupled, hindering division of labor and efficiency improvement. Due to the lack of a unified and automated benchmark testing platform, research teams must implement comparative algorithms, build evaluation frameworks, and perform tests themselves. This process not only involves heavy repetitive work, but also introduces error risks due to implementation differences. SUMMARY

[0006] The present application provides a mathematical word problem solver testing method and system to solve at least one of the deficiencies in the prior art.

[0007] In a first aspect, the present application provides a test system for a mathematical word problem solver, comprising: a data preprocessing module, a solver module, and a perspective confusion comparison module. The data preprocessing module is configured to integrate multiple source mathematical word problem data sets, and to standardize and label the problems in the mathematical word problem data sets, and output data instances with uniform data structures. The solver module is configured to decompose the solving algorithm into different basic functional components as algorithm components, to call the relevant algorithm components in the logical order of the to-be-tested solving algorithm to form a corresponding solver, and to use the solver to solve mathematical problems based on the input data instances to generate solving answers. The perspective confusion comparison module is configured to receive the solving answers, and to comprehensively evaluate the solver corresponding to the to-be-tested algorithm through three dimensions of problem type classification, key problem solving steps, and algorithm components.

[0008] According to the test system for the mathematical word problem solver provided by the present application, any solving algorithm is decomposed into different basic functional components, including: decomposing any solving algorithm into different basic functional components based on a state transition framework; wherein the state transition framework regards each key intermediate result in the problem solving process as a specific mathematical state, and each path from the initial state to the final state can be regarded as a solving algorithm, the conversion between mathematical states is realized by the basic functional components of the solving algorithm, and all conversions together form a complete solving algorithm.

[0009] According to the test system for the mathematical word problem solver provided by the present application, the mathematical word problem data set integrated by the data preprocessing module includes: multiple public data sets; and a self-defined data set covering multiple learning stage mathematical knowledge points and containing multiple difficulty levels of problems from single-step to multi-step reasoning.

[0010] According to the test system for the mathematical word problem solver provided by the present application, the standardization processing and labeling of the problems in the mathematical word problem data set includes: predefining a set of standardization target fields to ensure that the output data instances have uniform data structures; the set of standardization target fields includes: serial number, Chinese text, segmented text, English text, formula, answer, quantity relationship, problem type, and source; data cleaning, deduplication, and consistency checking are performed on the problems in the mathematical word problem data set; and a large language model is used to extract the quantity relationship of the problems and classify the difficulty levels.

[0011] According to the mathematical word problem solver test system provided by the application, the solver is evaluated through problem type classification, which comprises the following steps: classifying the problems solved by the solver according to the pre-labeled difficulty levels; counting the correct rates of the solver for the problems of different difficulty levels and the average correct rate; and evaluating the solver in the problem type classification dimension by using the correct rates of the problems of different difficulty levels and the average correct rate.

[0012] According to the mathematical word problem solver test system provided by the application, the solver is evaluated through key problem solving steps, which comprises the following steps: determining the key problem solving steps solved by the solver; the key problem solving steps comprise quantity relationship, equation and answer; and comparing the solved key problem solving steps with the real key problem solving steps to evaluate the solver in the key problem solving steps.

[0013] According to the mathematical word problem solver test system provided by the application, the solver is evaluated through algorithm components, which comprises the following steps: counting the correct rates of each algorithm component of the solver; and evaluating the solver in the algorithm components by using the correct rates of each algorithm component.

[0014] In the second aspect, the application further provides a mathematical word problem solver test method, which comprises the following steps: Integrating multi-source mathematical word problem data sets, and performing standardized processing and labeling on the problems in the mathematical word problem data sets to output data instances with unified data structures; Decomposing a solving algorithm into different basic functional components as algorithm components, calling the related algorithm components in a logical order of the to-be-tested solving algorithm to form a corresponding solver, and generating a solving answer by using the solver to solve a mathematical problem based on an input data instance; Receiving the solving answer, and comprehensively evaluating the solver corresponding to the to-be-tested algorithm in three dimensions of problem type classification, key problem solving steps and algorithm components.

[0015] In the third aspect, the application provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the mathematical word problem solver test method according to any one of the above aspects when executing the program.

[0016] In the fourth aspect, the application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program implements the steps of the mathematical word problem solver test method according to any one of the above aspects when executed by a processor.

[0017] The testing method and system for solving mathematical word problems provided by this invention can integrate multi-source datasets, provide an automated and reproducible testing workflow, and support fine-grained, multi-dimensional performance testing. Compared with existing technologies, it has the following advantages: (1) This invention provides a completely consistent evaluation metric for all solvers by integrating authoritative datasets from multiple sources and constructing a unified standardized test benchmark. This effectively solves the dilemma of incomparable results caused by differences in data sources, formats, and preprocessing methods, enabling the performance of different algorithms to be objectively and fairly measured and ranked.

[0018] (2) The innovative perspective confusion comparison module of this invention performs fine-grained performance analysis from three dimensions: problem type, solution steps, and algorithm components. Compared with the evaluation mode that only relies on the "final answer accuracy", this invention can accurately locate the specific link in the algorithm failure, which greatly improves the pertinence and effectiveness of algorithm optimization.

[0019] (3) By decoupling algorithm design from testing and evaluation and providing an automated end-to-end testing workflow, this invention not only greatly improves R&D efficiency but also fosters a new research paradigm of division of labor and collaboration, enabling researchers to focus on core innovation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of the test system for the mathematical word problem solver provided by the present invention; Figure 2 This is a schematic diagram of the architecture of the benchmark engine provided by the present invention; Figure 3 This is a state transition diagram of the internal workflow of the solver module in the benchmark engine provided by this invention; Figure 4 This is a schematic diagram of the workflow of the perspective obfuscation comparison module in the benchmark engine provided by this invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] The following is combined Figures 1-5 This invention describes the testing method and system for the mathematical word problem solver provided in the embodiments of the present invention.

[0025] Figure 1 This is a schematic diagram of the test system for the mathematical word problem solver provided by the present invention, as shown below. Figure 1 As shown, the system includes: a data preprocessing module 10, a solver module 20, and a perspective confusion comparison module 30; The data preprocessing module 10 is used to integrate multi-source mathematical text problem datasets, and to perform standardized processing and annotation on the problems in the mathematical text problem datasets, outputting data instances with a unified data structure. The solver module 20 is used to decompose the solving algorithm into different basic functional components as algorithm components. The relevant algorithm components are called in the logical order of the solving algorithm under test to form the corresponding solver. The solver is used to solve the mathematical problem and generate the solution based on the input data instance. The perspective confusion comparison module 30 is used to receive the solution answer and comprehensively evaluate the solver corresponding to the algorithm under test through three dimensions: problem type classification, key solution steps, and algorithm components.

[0026] The testing system for solving mathematical word problems provided by this invention can integrate multi-source datasets, provide an automated and reproducible testing workflow, and support fine-grained, multi-dimensional performance testing.

[0027] Based on the above embodiments, this invention proposes a benchmarking engine for evaluating mathematical word problem solvers. It provides the following key features: (1) In addition to integrating the most widely used existing datasets, a customized benchmark dataset was developed, which extensively covers the distribution of mathematical knowledge points and difficulty levels in primary and secondary schools.

[0028] (2) Through modular encapsulation and distributed invocation, the engine decomposes the most advanced algorithms into basic functional components and integrates them into a unified architecture. By sequentially invoking the relevant functional components, it achieves automated testing of various algorithms.

[0029] (3) In view of scalability, the engine supports the integration of newly developed methods, such as large models, thereby ensuring adaptability to future developments in the field.

[0030] (4) The performance of the algorithm is evaluated from three aspects: problem type, key solution steps and algorithm components by using perspective confusion comparison, so as to carry out component-level analysis, which is an effective alternative to ablation research.

[0031] (5) It can inspire researchers to recombine algorithm components to develop new methods.

[0032] Figure 2 This is a schematic diagram of the architecture of the benchmark engine provided by this invention. The data flow passes through three main modules, such as... Figure 2 As shown, for the data preprocessing module, each data instance in the mathematical word problem dataset is transformed into the necessary data structure required by the solver, ensuring compatibility with all subsequent solvers. After preprocessing, the data is passed to the solver module, where state-of-the-art solvers are decomposed into basic functional components (i.e., algorithmic components) and encapsulated, and invoked in a distributed architecture. This architecture integrates rule-based and deep learning-based methods, enabling researchers to automatically test various solvers according to their needs without repeatedly implementing existing solvers. The answers generated by these solvers are then comprehensively evaluated by the perspective confusion comparison module. This module evaluates solver performance from three aspects: problem type, key solution steps, and algorithmic components. To use this system, the target solver to be tested must first be identified and decomposed into algorithmic components in advance according to the system architecture. Any newly introduced algorithmic components should be integrated into the solver module to ensure compatibility. Once the algorithmic components, execution order, and appropriate database are configured, the testing process can begin.

[0033] The following describes each module in further detail with reference to specific embodiments.

[0034] Data preprocessing module: On the one hand, this module integrates several widely used mathematical word problem datasets, including Alg514, Draw1K, Dolphin18, SVAMP, Math23K, and MAWPS-s. On the other hand, this invention constructs a custom dataset, AMPSD24K. The AMPSD24K dataset has three significant characteristics: (1) it comprehensively covers primary and secondary school mathematical concepts; (2) it has a wide range of problem difficulty, covering reasoning tasks from single-step to multi-step; and (3) it has rich annotations to support detailed analysis and rigorous evaluation. These characteristics together establish the robustness and universality of this benchmark in algorithm performance evaluation.

[0035] This invention develops a customized data processing toolkit to automate data processing and annotation. To ensure data structure consistency, this invention first predefines a standardized target field set, including fields such as sequence number, Chinese text, segmented text, English text, formula, answer, quantitative relationships, question type, and source. Subsequently, deduplication is performed based on the Chinese text field to remove redundant samples and eliminate training bias that may be caused by duplicate instances.

[0036] To further improve dataset quality and evaluation reliability, this invention implements a comprehensive data cleaning and filtering process. In addition to routine data cleaning steps, this invention performs specific consistency checks and transformations on formula-answer combinations to ensure their correctness and consistency. To promote structured reasoning and enhance interpretability, this invention extracts quantitative relationships from the question text, transforming natural language descriptions into formalized quantitative relationships. This process establishes an explicit intermediate representation layer connecting the original text input with symbolic reasoning. For large-scale and consistent extraction requirements, this invention automatically derives quantitative relationships from the original Chinese text using structured prompts via the GPT-4o API. These carefully designed prompts generate interpretable intermediate representations suitable for formal reasoning tasks. For hierarchical data processing, all questions are classified using a three-level classification system based on reasoning complexity, with the difficulty increasing as follows: Type 1: Single-step calculation problems without implicit logic.

[0037] Type 2: Apply a known formula for a single conversion.

[0038] Type 3: Multi-step thought chain.

[0039] This classification was accomplished through prompt-based GPT-4o API calls, where a structured prompt-guided model evaluates the reasoning complexity of each question based on its observable logical depth and computational structure.

[0040] Solver module: The current solver module integrates several state-of-the-art rule-based and deep learning-based algorithms. These algorithms are decomposed into basic functional components (algorithm components) and encapsulated, then integrated into a unified state transition framework. This framework allows researchers to flexibly select algorithms for evaluation based on specific objectives, thereby achieving automated and task-oriented testing. This invention validates the feasibility of the proposed framework, and the method described herein can be further extended to integrate more algorithms and expand the algorithm library to further improve the applicability and versatility of the benchmark engine.

[0041] The core idea of ​​the state transition framework is to treat each key intermediate result in the problem-solving process as a specific mathematical state. Each path from the initial state to the final state can be considered an algorithm. The transitions between mathematical states are implemented through the functional components of the algorithm, and all transitions together constitute the complete algorithm. This structure can be represented by a state transition graph, where states constitute nodes and transitions are directed edges, such as... Figure 3 As shown, Figure 3 This invention presents a state transition diagram illustrating the internal workflow of the solver module in the benchmark engine. The benchmark engine currently integrates several different states and transitions. Algorithms composed of these states and transitions can encompass both rule-based and deep learning-based methods. The framework's scalability allows for the integration of state-of-the-art large-scale language model methods as transitions via API-based access. Specifically, the core idea of ​​the state transition diagram is existing technology, as embodied in patent CN118690005A, and will not be elaborated upon here.

[0042] based on Figure 3 The present invention establishes a solver module for a benchmark engine, which integrates multiple solvers based on rules, deep learning, and large models, as illustrated by the states and transitions. These solvers involve a total of eight mathematical states: initial input text, graph, vector, explicit quantitative relation set, implicit quantitative relation set, quantitative relation set, equation, and final output answer. The present invention selects four algorithms as examples to illustrate how to decompose and integrate them into the solver module; these algorithms are respectively composed of the following transitions.

[0043] Algorithm 1: BERT + Tree Decoder + Symbolic Solver Algorithm 2: Gated Loop Unit + Tree Decoder + Symbolic Solver 1 Algorithm 3: Graph Creation + Long Short-Term Memory Network + Recurrent Neural Network + Symbolic Solver 1 Algorithm 4: Syntactic and semantic extraction + fusion and display of quantitative relation sets + contextual reasoning + symbolic solver 2 After breaking down the algorithm into the above basic functional components, the mathematical states connecting the two ends of each component are determined and encapsulated in the corresponding component library. When the algorithm needs to be tested, the required components are arranged according to the rules of graph theory, and then called and executed sequentially. To integrate more algorithms, the algorithms to be integrated are similarly broken down into basic functional components according to the state transition framework. Components already existing in the component library do not require much modification. New components not yet in the component library are added. During testing, the corresponding components are called and executed sequentially.

[0044] Perspective Confusion Comparison Module: Previous studies have primarily used accuracy as a scalar metric to evaluate solver performance. However, this single metric is not comprehensive in measuring algorithm performance, and it particularly fails to meet the needs of intelligent education. Perspective on confusion comparison evaluates algorithms from three dimensions: problem type, key solution steps, and algorithm components.

[0045] The data preprocessing module provides several key annotations, such as sets of quantitative relationships, equations, and the final answer. The solver's performance is evaluated by comparing its output with these annotations.

[0046] Figure 4 This is a schematic diagram of the workflow of the perspective obfuscation comparison module in the benchmark engine provided by this invention, as shown below. Figure 4 As shown, the workflow of the perspective confusion comparison module first classifies the problem types, and then progressively evaluates key problem-solving steps such as quantitative relationships, equations, and answers. The system divides the problem types into three difficulty levels, calculates the accuracy of each algorithm for each level, and then calculates the overall average accuracy. This evaluation dimension accurately reflects the problem type characteristics to which the algorithm is applicable through four sub-dimensions (i.e., the accuracy of questions at the three difficulty levels and the average accuracy).

[0047] Furthermore, the key problem-solving steps dimension can include three sub-dimensions: quantitative relationships, equations, and the answer. The solution obtained by the solver is compared one-to-one with the expert-annotated solution, and the accuracy rate is statistically analyzed to complete the evaluation of this dimension.

[0048] Furthermore, each algorithm can be decomposed into n independent components. The accuracy of each component is calculated separately to evaluate the performance of each component of the algorithm. This evaluation dimension consists of n sub-dimensions, corresponding to the number of components in the algorithm.

[0049] Therefore, the result of perspective confusion comparison can theoretically be represented as a 4×3×n third-order tensor, corresponding to three evaluation dimensions. However, in practice, the accuracy of each component cannot be easily obtained from the true value. Therefore, the evaluation process is simplified to a 4×3 second-order tensor.

[0050] On the other hand, the present invention also provides a testing method for a mathematical word problem solver, comprising the following steps: Integrate multi-source math word problem datasets, standardize and label the problems in the math word problem datasets, and output data instances with a unified data structure; The solution algorithm is decomposed into different basic functional components, which serve as algorithm components. The relevant algorithm components are called in the logical order of the solution algorithm under test to form the corresponding solver. Based on the input data instance, the solver is used to solve the mathematical problem and generate the solution answer. The system receives the solutions and comprehensively evaluates the solvers corresponding to the algorithms under test based on three dimensions: problem type classification, key solution steps, and algorithm components.

[0051] It should be noted that the testing system for the mathematical word problem solver provided in this embodiment of the invention can perform the functions of the testing system for the mathematical word problem solver described in any of the above embodiments during actual operation, which will not be elaborated in this embodiment.

[0052] In summary, the present invention has the following advantages over the prior art: Compared with existing technologies, it has the following advantages: (1) This invention provides a completely consistent evaluation metric for all solvers by integrating authoritative datasets from multiple sources and constructing a unified standardized test benchmark. This effectively solves the dilemma of incomparable results caused by differences in data sources, formats, and preprocessing methods, enabling the performance of different algorithms to be objectively and fairly measured and ranked.

[0053] (2) The innovative perspective confusion comparison module of this invention performs fine-grained performance analysis from three dimensions: problem type, solution steps, and algorithm components. Compared with the evaluation mode that only relies on the "final answer accuracy", this invention can accurately locate the specific link in the algorithm failure, which greatly improves the pertinence and effectiveness of algorithm optimization.

[0054] (3) By decoupling algorithm design from testing and evaluation and providing an automated end-to-end testing workflow, this invention not only greatly improves R&D efficiency but also fosters a new research paradigm of division of labor and collaboration, enabling researchers to focus on core innovation.

[0055] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a testing method for a mathematical word problem solver. This method includes: integrating a multi-source mathematical word problem dataset, standardizing and labeling the problems in the dataset, and outputting data instances with a unified data structure; decomposing the solving algorithm into different basic functional components as algorithm components, calling relevant algorithm components according to the logical order of the algorithm under test to form a corresponding solver, using the solver to solve the mathematical problem based on the input data instance to generate a solution; receiving the solution, and comprehensively evaluating the solver corresponding to the algorithm under test through three dimensions: problem type classification, key solution steps, and algorithm components.

[0056] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A test system for a mathematical word problem solver, characterized by, Comprise: a data preprocessing module, a solver module and a perspective confusion comparison module; the data preprocessing module is used for integrating multi-source mathematical text question data sets, and standardizing and labeling the questions in the mathematical text question data sets, and outputting data instances with uniform data structure; the solver module is used for decomposing a solving algorithm into different basic functional components as algorithm components, calling relevant algorithm components through the logical sequence of the to-be-tested solving algorithm to form a corresponding solver, and generating a solving answer by using the solver to solve a mathematical problem based on the input data instance; the perspective confusion comparison module is used for receiving the solving answer, and comprehensively evaluating the solver corresponding to the to-be-tested algorithm through three dimensions of problem type classification, key problem solving steps and algorithm components.

2. The test system for a mathematical word problem solver according to claim 1, wherein Decompose any solving algorithm into different basic functional components, including: based on a state transition framework, any solving algorithm is decomposed into different basic functional components; wherein the state transition framework regards each key intermediate result in the problem solving process as a specific mathematical state, and each path from the initial state to the final state can be regarded as a solving algorithm, the conversion between mathematical states is realized through the basic functional components of the solving algorithm, and all conversions together form a complete solving algorithm.

3. The test system for a mathematical word problem solver according to claim 1, wherein The mathematical text question data set integrated by the data preprocessing module comprises: a plurality of public data sets; and a self-defined data set covering mathematical knowledge points of multiple learning stages and containing questions of multiple difficulty levels from single-step to multi-step reasoning.

4. The test system for a mathematical word problem solver according to claim 1, wherein The standardization processing and labeling of the questions in the mathematical text question data set include: predefining a standardized target field set to ensure that the output data instances have uniform data structure; the standardized target field set includes: serial number, Chinese text, segmented text, English text, formula, answer, quantity relationship, question type and source; data cleaning, deduplication and consistency checking are performed on the questions in the mathematical text question data set; quantity relationship extraction and difficulty level classification of the questions are performed by using a large language model.

5. The test system for a mathematical word problem solver according to claim 1, wherein The solver is evaluated through problem type classification, including: classifying the problems solved by the solver according to the pre-labeled difficulty levels; statistically analyzing the correct rates of the solver for questions of different difficulty levels and the average correct rate; the correct rates of the solver for questions of different difficulty levels and the average correct rate are used to evaluate the solver in the problem type classification dimension.

6. The test system for a mathematical word problem solver according to claim 1, wherein The solver is evaluated through key problem solving steps, including: determining the key problem solving steps solved by the solver; the key problem solving steps include quantity relationship, equation and answer; the solved key problem solving steps are compared with the real key problem solving steps to evaluate the solver in the key problem solving steps.

7. The test system for a mathematical word problem solver according to claim 1, wherein The solver is evaluated through algorithm components, including: statistically analyzing the correct rates of each algorithm component of the solver; the correct rates of each algorithm component are used to evaluate the solver in the algorithm component.

8. A test method for a mathematical word problem solver, characterized by, Comprise: integrate multi-source mathematical text question data sets, and standardize and label the questions in the mathematical text question data sets, and output data instances with uniform data structure; The solving algorithm is decomposed into different basic functional components as algorithm components, a corresponding solver is formed by calling relevant algorithm components in a logical sequence of the to-be-tested solving algorithm, and a solving answer is generated by solving a mathematical problem based on an input data instance; The solving answer is received, and the solver corresponding to the to-be-tested algorithm is comprehensively evaluated through three dimensions of problem type classification, key problem solving steps, and algorithm components.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the test method of the mathematical word problem solver according to claim 8 when executing the computer program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the test method of the mathematical word problem solver according to claim 8.

Images