Mathematics and logic problem processing method, system, equipment and medium

Abstract

The invention provides a mathematical and logic problem processing method, system and device and a medium, and relates to the technical field of artificial intelligence and natural language process.The method comprises the steps that multiple pieces of mathematical theorem information corresponding to a target problem are obtained, and the target problem represents a mathematical problem or a logic problem in a natural language form; based on the target problem and the multiple pieces of mathematical theorem information, generating a word vector sequence; the word vector sequence is input into a customized attention layer, multiple rounds of attention processes are repeatedly executed, context representation with mathematical theorem knowledge is obtained, and the customized attention layer is formed based on a cross attention module; and generating a target answer corresponding to the target question based on the context representation. According to the method, theorem content can participate in a model reasoning process, so that directional guidance and context enhancement of a current question are realized, knowledge illusion is effectively relieved, and reasoning coherence and answer reliability are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method, system, device, and medium for processing mathematical and logical problems. Background Technology

[0002] With the significant progress made by Large Language Models (LLMs) in general question answering, text generation and other tasks, their application in the fields of mathematics and logical reasoning is becoming increasingly widespread.

[0003] Traditional end-to-end trained large language models often exhibit several shortcomings when dealing with problems that rely on axioms and theorems. Specifically, these include: knowledge illusion, where the model "fabricates" seemingly reasonable but actually erroneous derivations or answers without explicit support; contextual forgetting and reasoning breaks, where large language models struggle to continuously track key premises or theorems in long-chain reasoning, leading to the accumulation of errors in intermediate steps; lack of interpretability, where the answers output by large language models cannot be traced back to specific mathematical principles or theorems, hindering applications in high-reliability scenarios such as education and research; and weak generalization ability, where large language models perform inconsistently for question types or combinations not seen in the training data, especially in tasks requiring the use of multiple theorems for collaborative reasoning.

[0004] Existing solutions often employ strategies such as cue engineering, thought chains, or fine-tuning, which, while effective to some extent, fail to fundamentally address the issue of large language models' proactive utilization and dynamic guidance of structured knowledge. Some research attempts to introduce external knowledge bases, such as mathematical formula databases, but these are typically used only as static retrieval sources and fail to deeply integrate with the internal reasoning mechanisms of large language models. Therefore, there is an urgent need for a method, system, device, and medium for handling mathematical and logical problems to address these issues. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a method, system, device, and medium for processing mathematical and logical problems.

[0006] This invention provides a method for processing mathematical and logical problems, comprising: Obtain multiple mathematical theorems corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; Based on the target problem and multiple mathematical theorems, a word vector sequence is generated; The word vector sequence is input into a custom attention layer and the attention process is repeated multiple times to obtain a contextual representation with mathematical theorem knowledge. The custom attention layer is constructed based on cross-attention modules. Based on the context representation, the target answer corresponding to the target question is generated.

[0007] According to a mathematical and logical problem-solving method provided by the present invention, the step of obtaining multiple mathematical theorem information corresponding to the target problem includes: Based on a large language model or vector retrieval module, the target question is semantically similar to the theorem knowledge in the structured theorem knowledge base to obtain the Top-K theorem knowledge; The Top-K theorems are used as multiple mathematical theorems corresponding to the target problem.

[0008] According to a mathematical and logical problem-solving method provided by the present invention, the step of generating a word vector sequence based on the target problem and multiple mathematical theorems includes: Based on a preset word segmenter, the target problem and multiple mathematical theorem information are segmented to obtain the segmented target problem and segmented mathematical theorem information. Based on the embedding layer in the large language model, the target problem after segmentation and multiple mathematical theorem information after segmentation are embedded and encoded to obtain a problem vector sequence and a theorem vector sequence. The word vector sequence is generated based on the question vector sequence and the theorem vector sequence.

[0009] According to a mathematical and logical problem-solving method provided by the present invention, the customized attention layer includes multiple serial attention sub-modules, each attention sub-module including at least one self-attention head and at least one cross-attention head to perform parallel computation; the attention sub-module is used to model the interaction relationship between the problem vector sequence and the theorem vector sequence.

[0010] According to a mathematical and logical problem-solving method provided by the present invention, the step of inputting the word vector sequence into a customized attention layer and repeatedly executing multiple rounds of attention process to obtain a contextual representation with mathematical theorem knowledge includes: The question vector sequence is used as the Query, and the theorem vector sequence is used as both the Key and Value. These are input into the customized attention layer to repeatedly perform multiple rounds of attention processing for attention weighting, thereby obtaining the context representation.

[0011] According to the mathematical and logical problem-solving method provided by the present invention, the repeated execution of multiple rounds of attention process for attention weighting specifically includes: After attention weighting in the attention submodule of the previous layer, the context representation of the previous layer is output. The context representation of the previous layer is input into the attention submodule of the current layer for attention weighting, and the attention submodule of the current layer outputs the context representation of the current layer. After determining that all attention submodules in the custom attention layer have completed attention weighting, the context representation output by the attention submodule of the last layer in the custom attention layer is used as the context representation.

[0012] According to a mathematical and logical problem-solving method provided by the present invention, generating a target answer corresponding to the target problem based on the context representation includes: Based on the context representation, the corresponding word prediction probability distribution is calculated at each inference time step; Based on the preset search strategy and the predicted probability distribution of the vocabulary, the target answer is generated sequentially during the reasoning stage.

[0013] The present invention also provides a mathematical and logical problem-solving system, comprising: The retrieval module is used to obtain multiple mathematical theorems corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; The vector sequence generation module is used to generate a word vector sequence based on the target problem and multiple mathematical theorems. A multi-round attention module is used to input the word vector sequence into a customized attention layer and repeatedly execute the multi-round attention process to obtain a contextual representation with mathematical theorem knowledge. The customized attention layer is based on a cross-attention module. The answer generation module is used to generate the target answer corresponding to the target question based on the context representation.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the mathematical and logical problem processing methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the mathematical and logical problem processing method described above.

[0016] The mathematical and logical problem-solving methods, systems, devices, and media provided by this invention obtain relevant theorem content from a theorem knowledge base and design a dedicated cross-attention layer, enabling the theorem content to participate in the model reasoning process. This achieves targeted guidance and contextual enhancement for the current problem, effectively alleviates knowledge illusion, and improves reasoning coherence and answer reliability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the mathematical and logical problem-solving method provided by this invention; Figure 2 A schematic diagram of the overall architecture of the mathematical and logical problem model based on theorem-guided attention mechanism provided by this invention; Figure 3 A schematic diagram illustrating the reasoning process of the large language model provided by this invention; Figure 4 This is a schematic diagram of the structure of the mathematical and logical problem processing system provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Figure 1 A flowchart illustrating the mathematical and logical problem-solving method provided by this invention is shown below. Figure 1 As shown, the present invention provides a method for processing mathematical and logical problems, including: Step 101: Obtain multiple mathematical theorems corresponding to the target problem, wherein the target problem represents a mathematical or logical problem in natural language form.

[0021] In this invention, the target problem is a mathematical or logical problem presented in natural language, covering a wide range of fields, such as problems in multiple branches of mathematics, including algebra, geometry, calculus, set theory, propositional logic, and probability and statistics.

[0022] To retrieve relevant mathematical theorem information, lightweight large language models (LLMs) or vector retrieval modules can be used. These tools retrieve the K mathematical theorems most semantically relevant to the target problem from a theorem knowledge base (a structured collection of theorems). The retrieval process can be based on semantic similarity calculations, for example, using text embedding functions. Computational problems q With the Semantic similarity between the embedding vectors of the theorems The formula is: Using the above formula for semantic similarity, we find the most relevant theorem (the first one). k (This refers to a theorem); it is also possible that related theorems can be generated directly from a lightweight large language model.

[0023] Step 102: Generate a word vector sequence based on the target problem and multiple mathematical theorems.

[0024] In this invention, after obtaining the target problem and related mathematical theorem information, it is necessary to represent them digitally so that the computer can process them.

[0025] First, the original target question and the retrieved mathematical theorems are tokenized, which means that the text is split into tokens, or semantic units. For example, sentences are split into words or sub-words.

[0026] Then, these lexical units are converted into word vector sequences of uniform dimension through an embedding layer (embedding encoding module). For example, the target question is converted into a question vector sequence Q, and each retrieved mathematical theorem is converted into a theorem vector sequence T. These word vector sequences can capture the semantic information of the text, making texts with similar semantics closer together in the vector space. For example, the question "Find the area of ​​a circle with a radius of 5" and the formula theorem for the area of ​​a circle have a certain correlation in the vector space after embedding encoding.

[0027] Step 103: Input the word vector sequence into the custom attention layer and repeat the attention process multiple times to obtain a contextual representation with mathematical theorem knowledge. The custom attention layer is based on cross-attention modules.

[0028] In this invention, the customized attention layer is built based on the cross-attention module and is a key part of the entire large language model that enables theorems to guide the reasoning process. The generated question vector sequence Q of the target question and the theorem vector sequence T of the mathematical theorem are input into the customized attention layer. In the customized attention layer, the cross-attention head works in the manner of "the query is Q (target question) and the theorem is K / V (key and value of the mathematical theorem)".

[0029] This setup enables theorems to guide the attention of the current reasoning process, allowing the large language model to focus on the theorems most relevant to the current reasoning. For example, when solving a geometry problem involving the collaborative reasoning of multiple theorems, the cross-attention mechanism dynamically adjusts the degree of attention given to different theorems based on the current reasoning state of the problem.

[0030] In this invention, the attention process is repeated multiple times (N rounds). Each round of the attention process further optimizes the contextual representation, progressively strengthening the association between the problem and relevant theorems. After multiple rounds of processing, a contextual representation rich in mathematical theorems is finally obtained. For example, during the first round of attention, the large language model may initially identify several theorems most relevant to the problem; in subsequent rounds, it will further refine the understanding and utilization of these theorems, making the contextual representation more accurate and richer.

[0031] Step 104: Based on the context representation, generate the target answer corresponding to the target question.

[0032] In this invention, when a contextual representation containing mathematical theorem knowledge is obtained... Then, the large language model can generate the answer to the target question based on this representation. In each generation step, the large language model will use the currently generated partial answer and contextual representation. It is computed through a decoder, and then used... softmax The function predicts the most likely next token at the current time step. The prediction formula is as follows: in, This indicates the probability of the next token. It is a weight matrix; This represents the partially generated answer.

[0033] In this invention, the large language model generates answers word by word until a complete sequence of answers is generated. For example, for the question about the length of the hypotenuse of a right triangle, the large language model will generate an answer such as "the length of the hypotenuse is 5" step by step based on the Pythagorean theorem knowledge implied in the context. Answers generated in this way have high accuracy and interpretability.

[0034] The mathematical and logical problem-solving method provided by this invention obtains relevant theorem content from the theorem knowledge base and designs a dedicated cross-attention layer, enabling the theorem content to participate in the model reasoning process. This achieves targeted guidance and context enhancement for the current problem, effectively alleviates knowledge illusion, and improves reasoning coherence and answer reliability.

[0035] Based on the above embodiments, obtaining multiple mathematical theorem information corresponding to the target problem includes: Based on a large language model or vector retrieval module, the target question is semantically similar to the theorem knowledge in the structured theorem knowledge base to obtain the Top-K theorem knowledge; The Top-K theorems are used as multiple mathematical theorems corresponding to the target problem.

[0036] In this invention, the target problem is a mathematical or logical problem presented in natural language, with the input modality being text. The target problem can be expressed in various forms, not limited to rigorous, textbook-style formulations, but also encompassing colloquial, ambiguous, or complex problems involving multi-step reasoning. For example, a problem might be "A number plus 5 equals 10, what is this number?" (colloquial expression), "There seems to be a theorem about the relationship between the three sides of a triangle, which can be used to calculate the side lengths" (ambiguous expression), or "First calculate the radius of this circle, then use the radius to calculate the area; the circumference of the circle is known to be 18.84" (complex problem involving multi-step reasoning).

[0037] Optionally, the target question input by the user can be preprocessed before matching to better suit subsequent matching operations. Preprocessing may include removing irrelevant characters, standardizing symbol formats, and segmenting sentences to make the question text more standardized and easier to process.

[0038] Lightweight large language models possess a certain level of language understanding and semantic analysis capabilities. They can encode both the input target question text and the theorem text from a structured theorem knowledge base, converting them into vector representations that computers can understand. The similarity between these vectors is used to measure the semantic association between the target question and each theorem. For example, for the target question "Find the length of the hypotenuse of a right triangle, given that the two legs are 3 and 4," the lightweight large language model will encode it as a vector. Simultaneously, it will also encode the Pythagorean theorem from the theorem knowledge base, "In a right triangle, the sum of the squares of the two legs equals the square of the hypotenuse," into a vector, and then calculate the similarity between these two vectors.

[0039] The vector retrieval module primarily utilizes text embedding functions to convert text into vectors, mapping both the target question and theorems from the theorem knowledge base into a vector space. In this vector space, semantically similar texts correspond to vectors that are closer together, while semantically dissimilar texts correspond to vectors that are farther apart. The semantic similarity between the target question vector and each theorem vector is determined by calculating the distance (e.g., cosine similarity) between them. For example, for the problem of finding the hypotenuse of a right triangle, the vector retrieval module converts both the question and the Pythagorean theorem into vectors, calculates their cosine similarity, and a higher similarity indicates a better semantic match.

[0040] In this invention, the structured theorem knowledge base is a pre-built collection of knowledge containing various mathematical and logical theorems. Each theorem has a clear name and a corresponding formula, such as the Pythagorean theorem and the perfect square formula. These theorems are organized and stored according to a certain structure to facilitate rapid retrieval and matching.

[0041] The value of Top-K can be set according to actual needs and application scenarios, representing the number of relevant theorems to be returned from the structured theorem knowledge base. For example, if you want the large language model to refer to more theorem information during the reasoning process, you can set the value of K to be larger; if you want the large language model to focus on the most relevant theorems, you can set the value of K to be smaller.

[0042] Furthermore, based on the semantic similarity calculated in the above embodiments, all theorems in the structured theorem knowledge base are sorted, and the top K theorems are selected in descending order of similarity. These theorems are the Top-K theorems with the highest semantic similarity to the target problem. For example, setting K=3, for the problem of finding the hypotenuse length of a right triangle, after semantic similarity matching, the Pythagorean theorem, the cosine theorem (which may be used in some special cases), and another theorem related to the calculation of the side length of a triangle (assuming such a theorem exists in the knowledge base) may be returned. The top 3 are selected after sorting by similarity.

[0043] In this invention, the Top-K theorems serve as multiple mathematical theorems corresponding to the target problem in subsequent processing, providing important theoretical basis for solving the target problem. In subsequent steps, the large language model will utilize these theorems to perform reasoning, calculations, and other operations to generate the answer to the target problem. For example, when solving the problem of finding the length of the hypotenuse of a right triangle, the large language model will mainly rely on the Pythagorean theorem for calculation, arriving at the answer that the hypotenuse length is 5.

[0044] By introducing these theorems that are highly relevant to the target problem, large language models can better understand the essence and requirements of the problem, improving the accuracy and reliability of the answers. Especially when dealing with complex or fuzzy problems, the relevant theorems can provide clear guidance for large language models, preventing them from making erroneous understandings and inferences.

[0045] Based on the above embodiments, the step of generating a word vector sequence based on the target problem and multiple mathematical theorems includes: Based on a preset word segmenter, the target problem and multiple mathematical theorem information are segmented to obtain the segmented target problem and segmented mathematical theorem information. Based on the embedding layer in the large language model, the target problem after segmentation and multiple mathematical theorem information after segmentation are embedded and encoded to obtain a problem vector sequence and a theorem vector sequence. The word vector sequence is generated based on the question vector sequence and the theorem vector sequence.

[0046] In this invention, a dedicated tokenizer supporting mathematical symbols can be used. This tokenizer is not a typical word segmentation tool only for natural language text, but rather an extension of a general vocabulary, capable of recognizing various special symbols, including mathematical operators, Greek letters, subscripts and superscripts, set symbols, and logical connectors. This extended capability ensures that for complex mathematical expressions, the tokenizer can accurately segment them into reasonable semantic units.

[0047] The target problem is a mathematical or logical problem presented in natural language, which may contain various mathematical expressions and symbols. For example, the problem might be "Given set A={1,2,3}, set B={2,3,4}, find A∩B". When the preset word segmenter segments the problem, it will reasonably divide both the natural language part and the mathematical symbol part of the problem. Natural language words such as "given", "set", and "find" will be segmented. At the same time, the representations of sets "A={1,2,3}" and "B={2,3,4}", as well as the symbol "∩" for finding intersection, will also be correctly identified and segmented, ultimately resulting in a segmented target problem composed of a series of semantic units.

[0048] The mathematical theorem information is derived from a structured theorem knowledge base, containing theorems relevant to the target problem. Each theorem has its specific formulation and notation. For example, the Pythagorean theorem states, "In a right triangle, the sum of the squares of the two legs is equal to the square of the hypotenuse." The word segmenter will segment the theorem, obtaining the segmented mathematical theorem information, such as natural language parts like "in a right triangle," "two legs," "sum of squares," and "hypotenuse." If there are multiple mathematical theorems, the same segmentation operation will be performed on each theorem separately.

[0049] In large language models, the embedding layer is a neural network layer that maps discrete symbols (such as words, subwords, or tokens) to continuous vector representations. These vectors have fixed dimensions and are able to capture the semantic and syntactic relationships between symbols in a vector space. Through embedding encoding, segmented text data can be converted into a numerical form that computers can better process and compute.

[0050] Furthermore, each token (the segmented semantic unit) in the target question after segmentation is sequentially input into the embedding layer of the large language model. The embedding layer generates a fixed-dimensional vector for each token, and these vectors are arranged in the order of the tokens in the original question to form a question vector sequence.

[0051] Similarly, for each segmented mathematical theorem, each token is input into the embedding layer for embedding encoding. Each theorem, after encoding, yields a theorem vector sequence. If there are multiple mathematical theorems, the above operation is performed on each theorem separately, resulting in multiple theorem vector sequences. For example, for the Pythagorean theorem and the perfect square formula, after embedding encoding, two different theorem vector sequences are obtained, with each vector in the sequence corresponding to a token after the theorem segmentation.

[0052] In this invention, the generated word vector sequence can be understood as a more comprehensive set of vector representations formed by integrating the question vector sequence and multiple theorem vector sequences. In practice, the question vector sequence and the various theorem vector sequences may be concatenated in a certain order. For example, the question vector sequence may be placed first, followed by the theorem vector sequences in sequence, forming a longer vector sequence. This integrated word vector sequence contains the semantic information of the target question and the semantic information of the relevant mathematical theorems, providing a rich data foundation for subsequent tasks such as semantic similarity matching and model inference.

[0053] By generating the aforementioned word vector sequence, the target problem and related theorems can be comprehensively analyzed and processed within a unified vector space. For example, when calculating the semantic similarity between the target problem and various theorems, operations can be performed directly on this integrated word vector sequence to more accurately measure the degree of correlation between them, thereby providing more suitable theoretical basis for solving the target problem.

[0054] Based on the above embodiments, the customized attention layer includes multiple serial attention sub-modules, each attention sub-module including at least one self-attention head and at least one cross-attention head to perform parallel computation; the attention sub-module is used to model the interaction relationship between the problem vector sequence and the theorem vector sequence.

[0055] In this invention, the customized attention layer is composed of multiple attention sub-modules connected sequentially (serially). This serial structure allows information to be progressively transferred and processed between different sub-modules. Each layer can further extract and integrate information based on the previous layer, thereby exploring the relationship between the problem and the theorem more deeply. For example, the first attention sub-module may initially capture some basic correlations between the problem and the theorem, while subsequent sub-modules can perform more refined analysis and processing based on this.

[0056] In this invention, each attention submodule contains at least one self-attention head and at least one cross-attention head, and these attention heads perform parallel computation. The self-attention head is primarily used to process the relationships within the input sequence. Within the question vector sequence or theorem vector sequence, the self-attention head can calculate the degree of association between the elements in the sequence. For example, for the question vector sequence, the self-attention head can identify the importance relationships between different parts of the question (such as different words or phrases), helping the large language model understand the semantic structure of the question itself. Similarly, for the theorem vector sequence, the self-attention head can analyze the dependencies between the elements within the theorem, enabling the large language model to better understand the semantics of the theorem.

[0057] The role of cross-attention heads is to establish the interaction between the question vector sequence and the theorem vector sequence. The question vector sequence is treated as a query, and the theorem vector sequence as keys and values. By calculating the similarity between the query and the keys, the degree of association between the question and each part of the theorem is determined. Then, the values ​​are weighted and summed based on this association, allowing the large language model to dynamically focus on the most relevant theorem content according to the semantics of the question, achieving guided attention weighting of the theorem to the current reasoning process. Furthermore, parallel computing allows self-attention heads and cross-attention heads to be computed simultaneously, improving computational efficiency and enabling faster acquisition of different types of information.

[0058] In this invention, the core function of the attention submodule is to model the interaction between the problem vector sequence and the theorem vector sequence. Through the collaborative work of self-attention heads and cross-attention heads, the large language model can comprehensively capture various associations between problems and theorems. For example, when solving a mathematical problem, self-attention heads can analyze the relationships between various mathematical concepts and conditions in the problem, while cross-attention heads can match and associate key information in the problem with relevant theorems. Through this modeling of interaction relationships, the large language model can better understand the theorems required for the problem and enhance its ability to perceive key theorems. For example, for a problem involving the calculation of the area of ​​a triangle, cross-attention heads can associate the description of triangles in the problem with the triangle area formula theorem, enabling the large language model to more accurately select and apply appropriate theorems to solve the problem. At the same time, self-attention heads can ensure the coherence and consistency of information within the problem, providing accurate background information for the application of theorems.

[0059] Based on the above embodiments, the step of inputting the word vector sequence into a customized attention layer and repeatedly executing the attention process multiple times to obtain a contextual representation with mathematical theorem knowledge includes: The question vector sequence is used as the Query, and the theorem vector sequence is used as both the Key and Value. These are input into the customized attention layer to repeatedly perform multiple rounds of attention processing for attention weighting, thereby obtaining the context representation.

[0060] In this invention, the question vector sequence is a series of fixed-dimensional vectors formed by segmenting and embedding the target question in natural language form. These vectors are arranged in the order of the semantic units (tokens) in the question, and can represent the semantic information of the target question in numerical form.

[0061] The theorem vector sequence is also obtained through word segmentation and embedding encoding, containing semantic information of multiple mathematical theorems related to the target problem. Each theorem is also processed to form a vector sequence, and the vectors in these vector sequences correspond to the semantic units in the theorem.

[0062] Furthermore, the question vector sequence is used as the Query, and the theorem vector sequence is used as both the Key and Value input into the custom attention layer. This ensures that the large language model can dynamically "read" the most relevant information from the theorem set based on the current question semantics at each step of reasoning.

[0063] Specifically, in this invention, a Query (i.e., a sequence of question vectors) represents the semantic representation of the problem that needs to be solved. The large language model searches for relevant information based on the content of the Query.

[0064] The key (i.e., a sequence of theorem vectors) is used to perform matching calculations with the query to determine the relevance between them. Each key vector corresponds to a semantic unit in the theorem. By calculating the similarity between the query and each key, we can determine which parts of the theorem are more relevant to the question.

[0065] Once the relevance between the Query and the Key is determined, the Value (i.e., the sequence of theorem vectors) provides specific information. Once the relevant theorem part is found (through Key matching), the detailed theorem content can be retrieved from the corresponding Value.

[0066] In the custom attention layer, attention weights are calculated between the query and each key. This can be done by calculating the similarity between the query and key vectors, such as using dot product or cosine similarity. The calculated similarity values ​​are then normalized using a softmax function, ensuring that the sum of all attention weights is 1. These attention weights represent the degree of relevance between the problem and various parts of the theorem; a larger weight indicates a higher relevance. For example, in a problem of finding the hypotenuse of a right triangle, calculating the attention weight of the key related to the Pythagorean theorem might yield a large weight value, indicating a high degree of relevance between the problem and the theorem.

[0067] In this invention, repeatedly executing multiple rounds of attention processing allows the large language model to more fully explore the relationship between the problem and the theorems. In each round, the large language model further adjusts and optimizes the attention calculation based on the attention weights and relevant information obtained in the previous round. Through multiple iterations, the large language model can gradually focus on the most relevant theorems, improving the accuracy of information extraction. For example, in the first round of attention calculation, the large language model may only roughly determine a few theorems related to the problem; in the second round, it will calculate the attention weights more precisely based on the results of the first round, further filtering out the most relevant information; after multiple rounds of such operations, the large language model can more accurately find the theorems that best match the problem.

[0068] Furthermore, the Value vectors are weighted and summed based on the calculated attention weights. Specifically, each Value vector is multiplied by its corresponding attention weight, and then all weighted Value vectors are summed to obtain a comprehensive vector representation. This comprehensive vector incorporates the theorem information most relevant to the problem, because Value vectors with larger weights contribute more to the summation process. For example, if the attention weight for one theorem part is 0.8 and the weight for another theorem part is 0.2, then the Value vector corresponding to the former theorem part will have a greater impact on the final result during the weighted summation.

[0069] Finally, the comprehensive vector obtained after attention weighting serves as the context representation, which integrates the semantics of the problem with the most relevant theorem information, providing rich contextual information for subsequent model inference. This contextual representation helps large language models better understand the essence of the problem and utilize relevant theorem knowledge for accurate reasoning and computation. For example, when solving mathematical problems, the contextual representation allows the large language model to know which theorem to apply and how to apply it to solve the problem.

[0070] Based on the above embodiments, the repeated execution of the multi-round attention process for attention weighting specifically includes: After attention weighting in the attention submodule of the previous layer, the context representation of the previous layer is output. The context representation of the previous layer is input into the attention submodule of the current layer for attention weighting, and the attention submodule of the current layer outputs the context representation of the current layer. After determining that all attention submodules in the custom attention layer have completed attention weighting, the context representation output by the attention submodule of the last layer in the custom attention layer is used as the context representation.

[0071] In this invention, the multi-round attention process is implemented by multiple cascaded attention enhancement units (i.e., attention sub-modules), where N is a preset positive integer ranging from 2 to 4. Each enhancement unit has the same structure and is connected sequentially. The output of the previous unit serves as the input of the next unit, and the contribution of the theorem information to the context representation is enhanced through multiple rounds of iterative operations.

[0072] In the multi-round attention process, each round (corresponding to an attention submodule) involves an attention weighting operation. The output of the previous round (initially possibly initial contextual information, such as a preliminary representation combining problem and theorem vectors) serves as input, containing the problem semantics and related theorem information. The attention submodule uses the sequence of problem vectors as the query and the sequence of theorem vectors as the key and value. It calculates the similarity between the query and the key (e.g., dot product, cosine similarity), and then normalizes the results using a softmax function to obtain the attention weights. These attention weights reflect the degree of relevance between the problem and various parts of the theorem.

[0073] Then, the Value vector is weighted and summed according to the attention weights to obtain a weighted feature representation, i.e., attention weighting. This highlights the theorem information most relevant to the problem and suppresses irrelevant information. After the attention weighting operation, the attention submodule outputs a comprehensive vector representation, i.e., the context representation of the previous layer. It integrates the semantics of the problem and the theorem information filtered by attention. Compared with the input, it focuses more on the parts closely related to the problem, providing more targeted information for the next layer of processing.

[0074] Since the attention submodule structure is the same in each round, the current layer repeats the attention weighting operation process of the previous layer. The current layer also uses the question vector sequence as the query and the theorem vector sequence as the key and value, and uses the semantic information of the question contained in the context representation of the previous layer to further mine the theorem content that is more relevant to the question.

[0075] After all attention submodules have completed their respective attention weighting, it indicates that the relationship between the problem and the theorem has been fully explored through multiple iterations, and the theorem information has been filtered and strengthened multiple times. At this point, the context representation output by the last attention submodule has undergone multiple rounds of optimization and strengthening, integrating the problem semantics and the most relevant and valuable theorem information, and can most comprehensively and accurately reflect the contextual relationship between the problem and the theorem. Therefore, the context representation output by the last attention submodule is used as the context representation of the entire customized attention layer. This representation will provide an important basis for subsequent model inference, decision-making, and other tasks. For example, in mathematical problem-solving models, this context representation can help large language models accurately select appropriate theorems and computational methods, thereby arriving at the correct answer.

[0076] Based on the above embodiments, generating the target answer corresponding to the target question based on the context representation includes: Based on the context representation, the corresponding word prediction probability distribution is calculated at each inference time step; Based on the preset search strategy and the predicted probability distribution of the vocabulary, the target answer is generated sequentially during the reasoning stage.

[0077] In this invention, the reasoning phase is a process in which the large language model progressively generates answers, divided into multiple time steps. At each time step, the large language model enhances its hidden state using contextual representations. The hidden state can be understood as an intermediate state retained by the model during information processing, recording the large language model's understanding and processing progress of the input information. By fusing and further processing the contextual representations with relevant information from the current time step (such as partially generated answers), the hidden state is enhanced, enabling it to more accurately reflect the needs of the current generation task. For example, when generating a long text answer, the hidden state is continuously updated as the generation process progresses to adapt to the semantic requirements of different stages.

[0078] Furthermore, based on the enhanced contextual hidden state, the large language model calculates the probability distribution of each token in the vocabulary using a linear layer and a softmax function. The linear layer is a fully connected layer that performs a linear transformation on the enhanced hidden state, mapping it to a vector space of the same size as the vocabulary. Then, the softmax function normalizes these values, ensuring that all elements range from 0 to 1 and their sum is 1, thus obtaining a probability distribution where each probability value represents the likelihood of generating the corresponding token at the current time step.

[0079] In this invention, the preset search strategy can be greedy search or bundle search. Greedy search is a simple and direct strategy that always selects the token with the highest probability value in the probability distribution as the generation result for the current time step at each time step.

[0080] Beam search is a more complex strategy. At each time step, it retains the top few tokens with the highest probabilities and then generates tokens for the next time step based on these retained tokens. In this way, beam search can consider more possibilities and avoid prematurely eliminating some potentially high-quality answers, thereby improving the quality of the generated answers.

[0081] During the inference phase, the large language model generates the target answer word by word based on a pre-defined search strategy and the calculated word prediction probability distribution. Taking bundle search as an example, in the first time step, the large language model selects several tokens with the highest probabilities as the beginnings of candidate answers based on the word prediction probability distribution. Then, for each candidate beginning, the large language model moves to the next time step, calculates the word prediction probability distribution again based on the new context representation (which now includes the information of the generated candidate beginnings), and selects several tokens with the highest probabilities to append to each candidate beginning, forming a new sequence of candidate answers. As the time steps progress, the sequence of candidate answers is continuously expanded and filtered until a complete answer is generated or the pre-defined generation length limit is reached.

[0082] Figure 2 This is a schematic diagram of the overall architecture of the mathematical and logical problem model based on theorem-guided attention mechanism provided by the present invention, which can be referred to. Figure 2 As shown, the model framework is mainly used to handle tasks involving mathematical or logical problems. The overall process combines theorem retrieval and a deep learning model based on an attention mechanism to generate accurate answers.

[0083] Specifically, a question in natural language form is input to the theorem retrieval system (a lightweight LLM). The theorem retrieval system, as a lightweight large language model, retrieves the Top-K theorems from a structured theorem knowledge base that are semantically most relevant to the input question.

[0084] Furthermore, the LLM Embedding Layer receives the Top-K theorems from the theorem retrieval system and the original input problem. The LLM Embedding Layer performs embedding encoding on the problem and the theorems, converting the textual input into fixed-dimensional vector representations that capture the semantic information of the text for subsequent model processing.

[0085] The Theorems Attention layer consists of N identical attention units chained together. Each attention unit contains both self-attention and cross-attention mechanisms. The self-attention head handles the relationships within the input sequence, such as the associations between elements within the question vector sequence, helping the model understand the semantic structure of the question itself. The cross-attention head treats the question vector sequence as a query and the theorem vector sequence as both a key and a value. In this way, the large language model can dynamically focus on the most relevant theorems based on the semantics of the question, achieving guided attention weighting of the theorems in the current reasoning process. In this invention, the cross-attention head, as an additional attention head, further strengthens the contribution of theorem information to the contextual representation. Through multiple iterations (multiple attention processes), the contextual representation is gradually optimized to obtain a contextual representation rich in theorem knowledge.

[0086] Based on the contextual representation obtained after multiple rounds of attention processing, the large language model generates the target answer (NTP) word by word using a standard Transformer decoder. At each inference time step, the large language model calculates the probability distribution of each token in the vocabulary based on the current contextual hidden state through a linear layer and a Softmax function. Then, it selects the most suitable token using a pre-defined search strategy (such as greedy search or bundle search) to gradually generate the complete answer.

[0087] Figure 3 A schematic diagram illustrating the reasoning process of the large language model provided by this invention can be referenced. Figure 3 As shown, the reasoning process mainly revolves around handling mathematical or logical problems, combining theorem retrieval and deep learning models to arrive at reasonable reasoning results. Specifically, the entire process begins with inputting a question in natural language form, which can be any mathematical or logical question related to algebra, geometry, and logical reasoning.

[0088] The input question is passed to the theorem retrieval unit (a lightweight LLM). As a lightweight large language model, the core function of the theorem retrieval unit is to retrieve the Top-K theorems most relevant to the input question from a pre-built structured theorem knowledge base, based on principles such as semantic similarity. For example, if the question is about finding the length of the hypotenuse of a right triangle, the theorem retrieval unit might retrieve relevant theorems such as the Pythagorean theorem.

[0089] The retrieved Top-K theorems and the original question are fed into the LLM embedding layer. In this layer, the model performs embedding encoding on the question and theorems, converting the textual data into fixed-dimensional vector representations. These vectors can capture the semantic information of the text in the vector space, making semantically similar texts closer together in the vector space, providing a numerical basis for subsequent attention calculation and inference.

[0090] The vectors after embedding and encoding enter the Theorems Attention layer, which consists of N identical attention units chained together. Within each attention unit, a self-attention mechanism allows the model to capture the dependencies between elements within the input sequence (including the question and theorem vectors); while a cross-attention mechanism uses the question vector as the query and the theorem vector as the key and value, enabling the model to dynamically focus on relevant theorems based on the semantics of the question, thus guiding the reasoning process through the theorems. Through multiple rounds (N rounds) of this attention computation and information fusion, the large language model gradually optimizes the contextual representation, ultimately obtaining a comprehensive representation rich in theorem knowledge and question semantics.

[0091] Based on the context representation after multiple rounds of processing, the large language model performs reasoning to arrive at the final reasoning result, which may be the answer to the question, the solution steps, or the logical analysis of the question.

[0092] This invention introduces a lightweight theorem retrieval system to dynamically acquire Top-K theorems relevant to the input question and embeds them along with the original query into an LLM embedding layer. Subsequently, theorem attention is added to the Transformer decoder. This theorem attention includes additional cross-attention heads (with the theorem as the key / value and the query as the query) and self-attention heads computed in parallel, achieving a deep fusion of theorem knowledge and semantic context. Finally, through N iterations of theorem attention, a more logically constrained next-word prediction (NTP) is output, effectively injecting formal theorem knowledge into the Transformer architecture in a structured manner. This significantly improves the model's accuracy and interpretability in tasks such as mathematical proof and symbolic reasoning without changing the core model parameters, providing an efficient and scalable new paradigm for building large models with "knowledge-guided" reasoning capabilities.

[0093] The mathematical and logical problem-solving system provided by this invention is described below. The mathematical and logical problem-solving system described below can be referred to in correspondence with the mathematical and logical problem-solving method described above.

[0094] Figure 4 This is a schematic diagram of the structure of the mathematical and logical problem-solving system provided by the present invention, such as... Figure 4 As shown, this invention provides a mathematical and logical problem processing system, including a retrieval module 401, a vector sequence generation module 402, a multi-round attention module 403, and an answer generation module 404. The retrieval module 401 is used to obtain multiple mathematical theorem information corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form. The vector sequence generation module 402 is used to generate a word vector sequence based on the target problem and the multiple mathematical theorem information. The multi-round attention module 403 is used to input the word vector sequence into a customized attention layer to repeatedly execute a multi-round attention process to obtain a contextual representation with mathematical theorem knowledge, wherein the customized attention layer is constructed based on a cross-attention module. The answer generation module 404 is used to generate a target answer corresponding to the target problem based on the contextual representation.

[0095] The mathematical and logical problem-solving system provided by this invention obtains relevant theorem content from the theorem knowledge base and designs a dedicated cross-attention layer, enabling the theorem content to participate in the model reasoning process. This achieves targeted guidance and context enhancement for the current problem, effectively alleviates knowledge illusion, and improves reasoning coherence and answer reliability.

[0096] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0097] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504, wherein the processor 501, communications interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute a mathematical and logical problem-solving method. This method includes: acquiring multiple mathematical theorem information corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; generating a word vector sequence based on the target problem and the multiple mathematical theorem information; inputting the word vector sequence into a customized attention layer to repeatedly execute multiple rounds of attention processes to obtain a contextual representation with mathematical theorem knowledge, wherein the customized attention layer is constructed based on a cross-attention module; and generating a target answer corresponding to the target problem based on the contextual representation.

[0098] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the mathematical and logical problem processing methods provided by the above methods, the method comprising: acquiring multiple mathematical theorem information corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; generating a word vector sequence based on the target problem and the multiple mathematical theorem information; inputting the word vector sequence into a customized attention layer to repeatedly execute multiple rounds of attention process to obtain a contextual representation with mathematical theorem knowledge, wherein the customized attention layer is constructed based on a cross-attention module; and generating a target answer corresponding to the target problem based on the contextual representation.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the mathematical and logical problem processing method provided in the above embodiments. The method includes: acquiring multiple mathematical theorem information corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; generating a word vector sequence based on the target problem and the multiple mathematical theorem information; inputting the word vector sequence into a customized attention layer to repeatedly execute multiple rounds of attention process to obtain a contextual representation with mathematical theorem knowledge, wherein the customized attention layer is constructed based on a cross-attention module; and generating a target answer corresponding to the target problem based on the contextual representation.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing mathematical and logical problems, characterized in that, include: Obtain multiple mathematical theorems corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; Based on the target problem and multiple mathematical theorems, a word vector sequence is generated; The word vector sequence is input into a custom attention layer and the attention process is repeated multiple times to obtain a contextual representation with mathematical theorem knowledge. The custom attention layer is constructed based on cross-attention modules. Based on the context representation, the target answer corresponding to the target question is generated.

2. The mathematical and logical problem-solving method according to claim 1, characterized in that, The acquisition of multiple mathematical theorems corresponding to the target problem includes: Based on a large language model or vector retrieval module, the target question is semantically similar to the theorem knowledge in the structured theorem knowledge base to obtain the Top-K theorem knowledge; The Top-K theorems are used as multiple mathematical theorems corresponding to the target problem.

3. The mathematical and logical problem-solving method according to claim 1, characterized in that, The step of generating a word vector sequence based on the target problem and multiple mathematical theorems includes: Based on a preset word segmenter, the target problem and multiple mathematical theorem information are segmented to obtain the segmented target problem and segmented mathematical theorem information. Based on the embedding layer in the large language model, the target problem after segmentation and multiple mathematical theorem information after segmentation are embedded and encoded to obtain a problem vector sequence and a theorem vector sequence. The word vector sequence is generated based on the question vector sequence and the theorem vector sequence.

4. The mathematical and logical problem-solving method according to claim 3, characterized in that, The custom attention layer includes multiple serial attention sub-modules, each of which includes at least one self-attention head and at least one cross-attention head to perform parallel computation; the attention sub-modules are used to model the interaction between the problem vector sequence and the theorem vector sequence.

5. The method for processing mathematical and logical problems according to claim 4, characterized in that, The step of inputting the word vector sequence into a customized attention layer and repeatedly executing the attention process multiple times to obtain a contextual representation with mathematical theorem knowledge includes: The question vector sequence is used as the Query, and the theorem vector sequence is used as both the Key and Value. These are input into the customized attention layer to repeatedly perform multiple rounds of attention processing for attention weighting, thereby obtaining the context representation.

6. The method for processing mathematical and logical problems according to claim 5, characterized in that, The repeated execution of the multi-round attention process involves attention weighting, specifically including: After attention weighting in the attention submodule of the previous layer, the context representation of the previous layer is output. The context representation of the previous layer is input into the attention submodule of the current layer for attention weighting, and the attention submodule of the current layer outputs the context representation of the current layer. After determining that all attention submodules in the custom attention layer have completed attention weighting, the context representation output by the attention submodule of the last layer in the custom attention layer is used as the context representation.

7. The method for processing mathematical and logical problems according to claim 1, characterized in that, The step of generating the target answer corresponding to the target question based on the context representation includes: Based on the context representation, the corresponding word prediction probability distribution is calculated at each inference time step; Based on the preset search strategy and the predicted probability distribution of the vocabulary, the target answer is generated sequentially during the reasoning stage.

8. A mathematical and logical problem-solving system, characterized in that, include: The retrieval module is used to obtain multiple mathematical theorems corresponding to a target problem, wherein the target problem represents a mathematical or logical problem in natural language form; The vector sequence generation module is used to generate a word vector sequence based on the target problem and multiple mathematical theorems. A multi-round attention module is used to input the word vector sequence into a customized attention layer and repeatedly execute the multi-round attention process to obtain a contextual representation with mathematical theorem knowledge. The customized attention layer is based on a cross-attention module. The answer generation module is used to generate the target answer corresponding to the target question based on the context representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the mathematical and logical problem processing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the mathematical and logical problem processing method as described in any one of claims 1 to 7.

Images