Knowledge-driven multi-thinking-chain cross-modal geometric problem solving method and system

Abstract

The invention relates to the technical field of cross-modal geometric problem solving methods, in particular to a knowledge-driven multi-thinking-chain cross-modal geometric problem solving method and system. According to the technical scheme, the knowledge-driven multi-thinking-chain cross-modal geometric problem solving method and system are provided. The method comprises the following steps: firstly, inputting a geometric knowledge point candidate set, and generating a function set by utilizing a multi-modal large language model query theorem; inputting questions, pictures and answers, screening knowledge point subsets, and extracting corresponding theorems and function subsets; a language and program thinking chain prompt template is constructed, and a thinking chain is generated and verified; then taking the successfully verified thinking chain as training data, and carrying out instruction optimization on the small-scale multi-modal model; and finally, generating a thinking chain through the model, performing comprehensive verification, and outputting a final answer.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal geometric problem solving methods, and in particular to a knowledge-driven multi-thinking chain cross-modal geometric problem solving method and system. Background Art

[0002] Solving cross-modal geometry problems is a major challenge in the field of artificial intelligence. This task requires the integrated use of multiple cognitive capabilities, including natural language understanding, visual reasoning, spatial reasoning, logical reasoning, and mathematical computation. To solve cross-modal geometry problems, the model needs to accurately understand the semantic information of the text question and extract relevant facts and conditions from it. Simultaneously, the model needs to perform effective visual reasoning on geometric figures to extract the spatial structure and relational features contained in the figures. Secondly, the model needs to perform multimodal fusion of text and visual information and apply relevant geometric theorems and formulas to perform logical reasoning and numerical calculations.

[0003] Methods for solving geometric problems can be roughly divided into two main types: symbolic geometric solvers and neural geometric solvers. Symbolic geometric solvers typically use a grammatical parser to parse problems and geometric figures into a formal language, and then use techniques such as path search and condition matching to perform symbolic reasoning. These methods have good interpretability, but such methods require the design of complex rules and have poor scalability. In addition, symbolic solvers may have redundant steps, and the search process does not match human solutions. In contrast, benefiting from the development of neural network technology, neural geometric solvers have improved learning efficiency and achieved more significant results.

[0004] Early neural geometry solvers were primarily based on models of visual-language reasoning and generation. These methods used hybrid encoders and self-supervised auxiliary tasks to jointly embed geometric images and text problems and generate solution programs in a sequential manner. While these methods achieved some success, labeling the solution programs required guidance from domain experts, making the process expensive and time-consuming. Furthermore, the lack of detailed reasoning in these methods reduced their interpretability and readability.

[0005] Recently, large language models (LLMs) have advanced mathematical reasoning through a technique called Chain-of-Thought (CoT) reasoning, which generates intermediate reasoning steps before deriving the final answer. However, existing research on chain-of-thought-based mathematical reasoning has primarily focused on textual modalities, with limited application in cross-modal mathematical reasoning scenarios, such as solving cross-modal geometry problems. Compared to single-modal mathematical reasoning, cross-modal geometry problems pose significant challenges to modeling due to their complex structure, including complex problem descriptions, unique graphical information in images, and underlying geometric knowledge. Summary of the Invention

[0006] In order to solve the problems of low interpretability caused by the lack of reasoning processes that fit human thinking in solving cross-modal geometric problems, and low reasoning accuracy caused by insufficient understanding of multimodal geometric problems with complex structures, the present invention proposes a knowledge-driven multi-thinking chain cross-modal geometric problem solving method and system.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0008] One of the solutions of the present invention is to provide a knowledge-driven multi-thinking chain cross-modal geometric problem solving method, comprising:

[0009] Input a candidate set of geometric knowledge points, wherein the candidate set of geometric knowledge points includes a plurality of basic theoretical knowledge points for solving geometric problems;

[0010] Based on the multimodal large language model, querying the geometric knowledge theorem corresponding to each geometric knowledge point in the geometric knowledge point candidate set to generate a geometric knowledge theorem set;

[0011] Convert each theorem in the set of geometric knowledge theorems into an executable Python function to form a set of geometric knowledge theorems Python functions;

[0012] Input the geometry problem to be solved, the corresponding geometry picture and the answer;

[0013] Constructing a geometry knowledge point screening instruction, combining the geometry problem, geometry picture, answer, and geometry knowledge point candidate set, and screening out a subset of geometry knowledge points related to the problem through a multimodal large language model;

[0014] Extracting corresponding theorem subsets and Python function subsets from the geometric knowledge theorem set and the geometric knowledge theorem Python function set according to the geometric knowledge point subsets;

[0015] Constructing a language thinking chain prompt template to guide the multimodal large language model to generate a language thinking chain, and verifying the consistency of the language thinking chain with the answer through regular matching;

[0016] Constructing a program thinking chain prompt template to guide the multimodal large language model to generate an executable Python program chain, and verifying the consistency of the result with the answer by running the program;

[0017] The successfully verified language thinking chain and program thinking chain are used as training data, and combined with general instructions to tune the instructions of the small-scale multimodal model to achieve joint learning of the language thinking chain and the program thinking chain;

[0018] Through the small-scale multimodal model, language thinking chains and program thinking chains are generated respectively, and the final answer is output after comprehensive verification.

[0019] In one possible improvement, the multimodal large language model is GPT-4V, which is used to retrieve geometric knowledge theorems associated with text and image modalities.

[0020] In a possible improvement, the geometric knowledge point screening instructions are described in natural language to guide the model to screen knowledge points related to questions, geometric pictures and answers from the candidate set.

[0021] In a possible improvement, the language thinking chain includes the following sub-steps:

[0022] Infer the underlying geometric knowledge points in the problem;

[0023] Determine the geometric theorems involved;

[0024] Establish the corresponding relationship between geometric knowledge theorems and image elements;

[0025] Add answers to the end of your chain of thoughts and check for consistency.

[0026] In a possible improvement, the input data of the language thinking chain is:

[0027]

[0028] in, Represents the input data of the language thinking chain, I represents the general instruction, Indicates a problem, Represents a geometric picture, and [Language Reasoning] is a special tag, indicating that the model should generate a language thinking chain.

[0029] In a possible improvement, the program thought chain generation requirements include:

[0030] calling a function from said subset of Python functions;

[0031] The variable name should be consistent with the geometric image element;

[0032] Generates comment-free and executable code;

[0033] Output the result directly and match it with the answer.

[0034] In a possible improvement, the input data of the program thought chain is:

[0035]

[0036] in, Represents the input data of the program thinking chain, I represents the general instruction, Indicates a problem, Represents a geometric picture, and [Program Reasoning] is a special tag, indicating that the model should generate a program thinking chain.

[0037] In a possible improvement, the small-scale multimodal model in the step is an LLaVA model, which trains the language thinking chain and the program thinking chain through a joint loss function, and the loss function is:

[0038]

[0039] Among them, formulas (3) and (4) represent the loss functions when the model predicts the Ground Truth label data as language thought chain and program thought chain, respectively, and formula (5) represents the loss when jointly calculating multiple thought chains; Θ is a trainable parameter, Z L The output data is the length of the language thinking chain, Z P The output data is the length of the program thinking chain; and They are the input data of the language thinking chain and the program thinking chain respectively; Indicates that given input data and output language thinking chain c L The probability of generating the i-th word under the condition of the first i-1 words; Indicates that given input data and output program thinking chain c P The probability of generating the i-th word under the condition of the first i-1 words.

[0040] In a possible improvement, the comprehensive verification includes:

[0041] Prioritize executing the Python code generated by the program thinking chain to obtain the results;

[0042] If the program result is invalid, then extract the answer from the language thinking chain;

[0043] The prediction accuracy of the statistical model is used as the final evaluation indicator.

[0044] The second purpose of the present invention is to provide a system for solving multi-thinking chain cross-modal geometric problems driven by knowledge, including

[0045] Data input module: used to input candidate sets of geometric knowledge points, geometric problems to be solved, corresponding geometric pictures and answers;

[0046] Knowledge query and generation module: Based on a multimodal large language model, it queries the geometric knowledge theorem corresponding to each geometric knowledge point in the candidate set of geometric knowledge points, generates a set of geometric knowledge theorems, and writes executable Python functions for the theorems to form a set of geometric knowledge theorems Python functions;

[0047] Knowledge point screening module: Constructs geometric knowledge point screening instructions, combines geometric problems, geometric pictures, answers, and a set of geometric knowledge point candidates, and uses a multimodal large language model to screen out a subset of geometric knowledge points related to the problem;

[0048] Subset extraction module: based on the selected subset of geometric knowledge points, extract the corresponding theorem subset and Python function subset from the geometric knowledge theorem set and the geometric knowledge theorem Python function set;

[0049] Thinking chain construction and verification module: including language thinking chain unit and program thinking chain unit;

[0050] The language thinking chain unit is used to construct a language thinking chain prompt template, guide the multimodal large language model to generate a language thinking chain, and verify the consistency of the language thinking chain with the answer through regular matching;

[0051] The program thinking chain unit is used to build a program thinking chain prompt template, guide the multimodal large language model to generate an executable Python program chain, and verify the consistency of the result and the answer by running the program;

[0052] Model training module: Uses successfully verified language thought chains and program thought chains as training data, combines general instructions to optimize the instructions of a small-scale multimodal model, and realizes the joint learning of language thought chains and program thought chains;

[0053] Result generation and verification module: Through the trained small-scale multimodal model, language thinking chains and program thinking chains are generated respectively. The Python code generated by the program thinking chain is executed first to obtain the results. If the program result is invalid, the answer in the language thinking chain is extracted. The statistical model prediction accuracy is used as the final evaluation indicator to output the final answer.

[0054] The beneficial effects brought about by the technical solutions provided in the embodiments of the present application include at least the following:

[0055] Beneficial effects:

[0056] 1. This invention was designed and implemented after a detailed study of how humans solve geometric problems. It aligns with the human approach of starting from geometric knowledge points, fully utilizing geometric knowledge theorems to perform reasoning and calculations, and thus arrive at the answer.

[0057] 2. This paper uses the multimodal large language model GPT-4V to query geometric knowledge theorems, screen geometric knowledge points, and generate language thought chains and program thought chains. This method of automatically acquiring weakly annotated data is beneficial for reducing the time and labor costs in the data annotation process.

[0058] 3. The language thought chain prompt template constructed by this invention uses geometric knowledge and geometric knowledge theorems related to the problem, and the instructions clearly indicate the detailed ideas for generating the language thought chain, which is beneficial for the model to generate an explainable reasoning process. The constructed program thought chain prompt template uses the geometric knowledge theorem Python function, which is beneficial for the model to generate a precise and rigorous calculation process.

[0059] 4. This invention achieves joint learning of language and program thinking chains by optimizing instructions for small-scale multimodal models. The two thinking chains guide and promote each other during training, which is beneficial to improving the accuracy of small-scale multimodal models in solving geometric problems.

[0060] 5. The method does not require additional parameters, has good versatility, is highly practical, and is easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a system flow chart of a knowledge-driven multi-thinking chain cross-modal geometric problem solving method of the present invention;

[0062] Figure 2 This is an example diagram of question types from the GeoQA dataset of the present invention;

[0063] Figure 3 This is an example diagram of the geometry knowledge point screening prompt template of the present invention;

[0064] Figure 4 This is an example diagram of the language thinking chain prompt template of the present invention;

[0065] Figure 5 This is an example diagram of the program thinking chain prompt template of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0067] Combine Figure 1 、 2 , 3, 4, 5, this method comprises the following steps:

[0068] Step 1: Enter the candidate set of geometric knowledge points and jump to step 2 and step 5 respectively.

[0069] Among them, knowledge points refer to the basic theoretical knowledge that needs to be mastered and applied to solve a certain type of problem, such as parallel lines, circular angles, central angles, Pythagorean theorem, etc. The candidate set of geometric knowledge points is expressed as: K = {k1, k2, ..., k n}, n is the number of geometric knowledge points.

[0070] For example, the input geometric knowledge point candidate set K={Inscribed Angle, Parallel Lines, Right Triangle, Central Angle, Similar Triangle, Tangent, Isosceles Triangle, Pythagorean Theorem, Parallelogram, Perpendicular Diameter, Distance, Trigonometric Functions, Sum Interior Angles Triangle, Parallel Lines Proportion, Angle Bisector, Cyclic Quadrilateral, Triangle Exterior Angle,Triangle Median,Circumcircle,Cone,Equilateral Triangle,PerpendicularBisector,Circumcenter,Rhombus,Polygon,Square,Adjacent SupplementaryAngles,Rectangle,Symmetry,Sector,Rotation,Right Triangle 30Angle,Vertical,DirectionAngles,Translation,Elevation Angles,Depression Angles,Homothety,Three-ViewDrawing,Clock Angles,Golden Ratio,Cylinder,Triangle Special RelationshipsInterior ExteriorAngles,Calculate Arc Length,Congruent Triangles,TriangleIncenter,Triangle Centroid,Regular Polygon,Trapezoid,Projection Theorem}, jump to step 2 and step 5 respectively.

[0071] Among them, the geometric knowledge point candidate set includes a total of 48 geometric knowledge points.

[0072] Step 2: Use the multimodal large language model to query geometric knowledge theorems, form a set of geometric knowledge theorems, and jump to steps 3 and 7 respectively.

[0073] Among them, theorem refers to a general mathematical proposition obtained through rigorous reasoning and proof, such as: if two straight lines are parallel, their alternate interior angles are equal; the circular angles corresponding to the same arc are equal; the sum of the interior angles of a triangle is equal to 180°, etc.

[0074] Specifically, for each geometric knowledge point in the candidate set of geometric knowledge points input in step 1, the multimodal large language model GPT-4V is used to query the geometric knowledge theorem corresponding to the geometric knowledge point, and a set of geometric knowledge theorems is obtained; the queried set of geometric knowledge theorems is expressed as: n is the number of geometric knowledge points; among them, the jth geometric knowledge point k j The candidate set of geometric knowledge theorems is expressed as: Where m is the geometric knowledge point k j The number of geometric theorems in .

[0075] Exemplarily, for each geometric knowledge point in the candidate set of geometric knowledge points input in step 1, the multimodal large language model GPT-4V is used to query the geometric knowledge theorem corresponding to the geometric knowledge point, and obtain a set of geometric knowledge theorems; for example, the knowledge point Inscribed Angle corresponds to the following three geometric knowledge theorems: "1) The inscribed angles corresponding to the same arc are equal; 2) The corresponding inscribed angle of the diameter is 90°; 3) The inscribed angle corresponding to the same arc is half of the central angle"; the knowledge point Parallel Lines corresponds to the following three geometric knowledge theorems: "1) When two lines are parallel, the alternate interior angles formed are equal; 2) When two lines are parallel, the corresponding angles formed are equal; 3) When two lines are parallel, the sum of the consecutive interior angles on the same side of the transversal is 180 degrees"; this step includes a total of 137 geometric knowledge theorems.

[0076] Step 3: Write a geometric knowledge theorem Python function to form a geometric knowledge theorem Python function set, and jump to step 7.

[0077] Specifically, for each geometry theorem in step 2, write a geometry theorem Python function. The function name consists of the geometry knowledge point name and the abbreviation of the geometry theorem. The geometry theorem Python function set consisting of all geometry knowledge points is represented as: Where n is the number of geometric knowledge points. j The geometric knowledge theorem Python function candidate set is expressed as: Among them, m is the geometric knowledge point k j The corresponding number of geometric knowledge theorems.

[0078] For example, the Python function for the knowledge point Inscribed Angle Correspondence Theorem 1) The inscribed angles corresponding to the same arc are equal is:

[0079] "defInscribed_Angle_same_arc_equal_angles(angle):

[0080] return angle”

[0081] Knowledge Point Inscribed Angle Correspondence Theorem 2) The corresponding inscribed angle of the diameter is 90°. The Python function is:

[0082] "defInscribed_Angle_diameter_angle():

[0083] return 90”

[0084] Knowledge Point Inscribed Angle Correspondence Theorem 3) The inscribed angle corresponding to the same arc is half of the central angle. The Python function is:

[0085] "def Inscribed_Angle_relationship_with_central_angle(central_angle=None,inscribed_angle=None):

[0086] ifcentral_angle:

[0087] return 1 / 2*central_angle

[0088] else:

[0089] return 2*inscribed_angle”

[0090] This step includes a total of 138 geometric knowledge theorem Python functions.

[0091] Step 4: Enter the question, geometric picture, and answer, and jump to Step 5, Step 8, Step 10, Step 11, Step 13, and Step 15 respectively.

[0092] The input question corresponds to a geometric picture. The question consists of text described in natural language and consists of letters, symbols, and numbers. The answer is only the final result and does not include the reasoning process. The i-th input sample is represented as: X i , the question, geometric picture and answer in the i-th sample are expressed as:

[0093] Example, input problem "As shown in the figure,AB is the diameter of⊙O,point C is on⊙O,radius OD∥AC,if∠BOD=130°,then the degree of∠B is()", geometric picture Answer 40.0, jump to step 5, step 8, step 10, step 11, step 13, and step 15 respectively.

[0094] Step 5: Write geometric knowledge point filtering instructions and build a geometric knowledge point filtering prompt template.

[0095] Among them, the instruction is usually a description written in natural language, the purpose of which is to let the model perform specific tasks according to the description, such as generating answers, performing a function, or completing a specific operation; the geometric knowledge point screening instruction is represented by I k The specific content is: based on the question, geometric picture, and answer, select the knowledge points needed to solve the current problem from the candidate set of geometric knowledge points.

[0096] Specifically, the geometric knowledge point candidate set K in step 1 and the question in step 4 Geometric pictures Answer And the geometric knowledge point screening instruction I in the current step k , with "I k .question: Geometric pictures: Answer: The candidate set of geometric knowledge points is connected in the form of "[K]" to form a geometric knowledge point screening prompt template.

[0097] The purpose of the geometric knowledge point screening prompt template in step 5 is to guide the multimodal large language model GPT-4V to screen the geometric knowledge points needed for the current problem, so as to avoid excessive geometric knowledge points interfering with the multimodal large language model when generating the thinking chain in subsequent steps.

[0098] An example of a geometry knowledge point screening prompt template is as follows: Figure 3 shown.

[0099] Step 6: Use the multimodal large language model to filter geometric knowledge points to form a subset of geometric knowledge points, and then jump to steps 7, 8, and 11.

[0100] Specifically, the geometric knowledge point screening prompt template in step 5 is input into the multimodal large language model GPT-4V to generate a geometric knowledge point subset, the i-th sample X i The subset of geometric knowledge points is expressed as: in, l is the sample X i The corresponding number of knowledge points.

[0101] Exemplarily, the geometric knowledge point screening prompt template in step 4 is input into the multimodal large language model GPT-4V to generate a subset of geometric knowledge points, wherein the subset of geometric knowledge points for the question "As shown in the figure, AB is the diameter of ⊙O, point C is on ⊙O, radius OD∥AC, if ∠BOD=130°, then the degreeof∠B is()" is: {Adjacent Supplementary Angles, Parallel Lines, InscribedAngle, Right Triangle}.

[0102] Step 7: Extract a subset of geometric knowledge theorems and a subset of geometric knowledge theorems Python functions, and jump to step 8 and step 11 respectively.

[0103] Specifically, the geometric knowledge points selected in step 6 are used to extract a subset of geometric knowledge theorems from the geometric knowledge theorem set T in step 2. The i-th sample X iThe subset of geometric knowledge theorems is expressed as: in, Extract a subset of the geometric knowledge theorem Python function from the geometric knowledge theorem Python function in step 3, the i-th sample X i The geometric knowledge theorem of the Python function subset is expressed as: in, l is the training sample X i The corresponding number of knowledge points.

[0104] For example, the geometry knowledge theorem subset of the geometry knowledge point “Adjacent SupplementaryAngles” is {“The sum of the measures of two adjacent supplementary angles is always equal to 180 degrees;”}, and the geometry knowledge theorem Python function subset is {“defAdjacent_Supplementary_Angles(angle):\n\treturn 180-angle”};

[0105] The geometry theorem subset of the geometry knowledge point "Parallel Lines" is {"When two lines are parallel, the alternate interior angles formed are equal;", "When two lines are parallel, the corresponding angles formed are equal;", "When two lines are parallel, the sum of the consecutive interior angles on the same side of the transversal is 180 degrees."}, and the geometry theorem Python function subset is {"defParallel_Lines_corresponding_angles(angle):\n\treturn angle", "def Parallel_Lines_alternate_interior_angles(angle):\n\treturn angle", "def Parallel_Lines_consecutive_interior_angles_180(angle):\n\t return 180-angle"};

[0106] The geometric knowledge theorem subset of the geometric knowledge point "InscribedAngle" is {"Inscribed angles subtending the same arc of a circle are equal;", "An inscribed angle subtended by a diameter of a circle is 90 degrees;", "The relationship between a central angle and an inscribed angle: The measure of a central angle subtending the same arc is equal to twice the measure of the inscribed angle;", "The sum of the opposite angles of a cyclic quadrilateral (a quadrilateral inscribed in a circle) is 180 degrees."}, and the Python function subset of the geometric knowledge theorem is {"def Inscribed_Angle_same_arc_equal_angles(angle):\n\t return angle", "def Inscribed_Angle_diameter_angle():\n\t return 90", "def Inscribed_Angle_supplementary_angles_90(angle):\n\t return 90-angle", "def Inscribed_Angle_supplementary_angles_180(angle):\n\t return 180-angle", "def Inscribed_Angle_relationship_with_central_angle(central_angle=None, inscribed_angle=None):\n\t if central_angle:\n\t\t return 1 / 2*central_angle\n\t else:\n\t\t return 2*inscribed_angle"};

[0107] The geometric knowledge theorem subset of the geometry knowledge point "Right Triangle" is {"Pythagorean Theorem: In a right triangle, the square of the length of the hypotenuse is equal to the sum of the squares of the lengths of the other two sides;", "Since oneangle is 90degrees the sum of the other two angles must also be 90degrees;", "Midline property: In a right-angled triangle, the length of the segment from the midpoint of the hypotenuse to the right-angle vertex is half the length of the hypotenuse."}, the geometric knowledge theorem Python function subset is {"def Right_Triangle_pythagorean_theorem(leg_a=None,leg_b=None,hypotenuse_c=None):\n\t ifhypotenuse_c:\n\t\t return math.sqrt(hypotenuse_c**2-leg_a**2)\n\t else:\n\t\t return math.sqrt(leg_a**2+leg_b**2)","def Right_Triangle_supplementary_angles_90(angle):\n\t return 90-angle", "def Right_Triangle_medians(hypotenuse=None,median=None):\n\t if hypotenuse:\n\t\t return(1 / 2)*hypotenuse\n\telse:\n\t\treturn2*median”}.

[0108] Step 8: Write language thinking chain instructions and build a language thinking chain prompt template driven by knowledge.

[0109] Among them, the language thinking chain instruction is represented by I y, which includes basic instructions for language thinking chain and detailed instructions for language thinking chain Basic instructions of language thinking chain The specific content is: Based on the question, geometric picture, answer, geometric knowledge points, and the candidate theorems corresponding to these geometric knowledge points, generate a reasoning chain to solve the problem. It consists of three parts: 1. Generate a reasoning chain according to the following three aspects: 1) Infer the underlying geometric knowledge points of the problem, 2) Determine the geometric knowledge theorems involved in the problem, and 3) Discover the correspondence between different geometric knowledge and geometric images; 2. Select the information necessary to solve the problem based on the candidate theorems of the geometric knowledge points. 3. Add the string "The answer is:" after the generated language thinking chain and add the answer after it. The generated reasoning process should be as simple as possible. The purpose of writing detailed instructions for the language thinking chain is: First, to promote the generation of language thinking chains from coarse-grained to fine-grained by the multimodal language model, and to promote the alignment of multimodal information between the problem, geometric knowledge, and geometric images. Second, to ensure that the language thinking chain generated by the multimodal language model is interpretable; finally, to prevent the multimodal language model from generating overly lengthy and irrelevant reasoning processes due to hallucinations.

[0110] Specifically, the question in step 4 Geometric pictures Answer The geometric knowledge point subset K in step 6 i , the subset of geometric knowledge theorems T in step 7 i And the basic instructions of the language thinking chain in the current step and detailed instructions for language thinking chains according to" question: Geometric pictures: Answer: Geometry knowledge points: [K i ]. Theorem candidates corresponding to each knowledge point: geometric knowledge point The candidate theorem is " to form a language thinking chain prompt template. Represents the geometric knowledge point subset K i A geometric knowledge point in Represents a subset of geometric knowledge theorems T i A theorem of geometric knowledge.

[0111] For example, the language thinking chain prompt template is as follows: Figure 4 shown.

[0112] Step 9: Use the multimodal large language model to generate language thought chains.

[0113] Specifically, the language thought chain prompt template in step 8 is input into the multimodal large language model GPT-4V to guide it to generate a language thought chain, which is represented by C L It is composed of a chain of geometric knowledge points → a chain of geometric knowledge theorems → a chain of correspondences between different geometric knowledge and elements of geometric pictures.

[0114] For example, the language thought chain prompt template containing multiple geometric knowledge in step 8 is input into the multimodal large language model GPT-4V to generate a language thought chain, such as Figure 2 shown.

[0115] Step 10: Use regular matching to verify the language thinking chain generated in step 9.

[0116] Specifically, for the language thought chain, regular expression matching is used to extract the characters following "The answer is:" and match them with the answer in step 4. If the match fails or the answer cannot be extracted, the process returns to step 9 and regenerates the language thought chain using the multimodal large language model. If the match is successful, the process jumps to step 14. This process is repeated three times, deleting any data that cannot be matched to the answer.

[0117] Exemplarily, regular matching is used to obtain the answer 40.0 from "The answer is:40.0" in the language thinking chain of step 9 of this embodiment, and match it with the answer 40.0 in step 4. The language thinking chain of this embodiment contains the correct answer, so jump to step 14 and use it as the GroundTruth label data for the geometry problem-solving process.

[0118] Step 11: Write program thinking chain instructions and build a program thinking chain prompt template driven by knowledge.

[0119] Among them, the program thinking chain instruction is represented by I c , which includes the basic instructions of the program thinking chain and detailed instructions for program thinking chain Basic instructions of program thinking chain The specific content is: Generate a complete Python program based on the question, geometric picture, answer, geometric knowledge point and the geometric knowledge theorem Python function corresponding to the geometric knowledge point. The following content is included: Please call the applicable function from the candidate geometry theorem Python function to solve the problem. You can select multiple Python functions related to geometry knowledge points. The variable names in the generated Python program should be consistent with the corresponding elements in the geometry diagram. The program should not include comments, just generate a concise program. When printing, do not add additional strings; just print the variables directly. Make sure that each function is used and that the program is executable. The purpose of writing detailed instructions for the program thinking chain is to ensure that the Python program generated by the multimodal large language model includes reasoning and is executable.

[0120] Specifically, the question in step 4 Geometric pictures Answer The geometric knowledge point subset K in step 6 i 、The geometric knowledge theorem in step 7 Python function subset P i And the basic instructions of the program thinking chain in the current step and detailed instructions for program thinking chain according to" question: Geometric pictures: Answer: Geometry knowledge points include: [K i ]. Candidates for the geometric knowledge theorem Python function corresponding to each geometric knowledge point: Geometric knowledge point The candidate theorem is " is connected in the form of a program thinking chain prompt template; Represents the geometric knowledge point subset K i A geometric knowledge point in Representing geometric knowledge theorem Python function subset P i A geometric theorem in Python function.

[0121] For example, the program thinking chain prompt template example is as follows Figure 5 shown.

[0122] Step 12: Use the multimodal large language model to generate program thought chains.

[0123] Specifically, the program thinking chain prompt template in step 11 is input into the multimodal large language model GPT-4V to guide it to generate a program thinking chain, which is represented by C P , which has an inference process and is executable.

[0124] For example, the program thought chain prompt template in step 11 is input into the multimodal large language model GPT-4V to generate a program thought chain, such as Figure 2 shown.

[0125] Step 13: Run the Python program and verify the program thinking chain generated in step 12.

[0126] Specifically, run the Python program in the program thinking chain and match it with the answer in step 4. If the match fails or the program cannot run, return to step 12 and use the multimodal large language model to regenerate the program thinking chain; if the match is successful, jump to step 14; this process is iterated 3 times, deleting the data that cannot match the answer or the program cannot run.

[0127] For example, running the Python program of the program thinking chain in step 12 obtains the result 40.0, which matches the answer 40.0 in step 4. The result of running the program thinking chain in this embodiment is correct, so jump to step 14 and use it as the Ground Truth label data for the geometry problem-solving process.

[0128] Step 14: Obtain the language thinking chain and program thinking chain that have been successfully verified in steps 10 and 13, and use them as the ground truth label data for the geometry problem-solving process;

[0129] Step 15: Write general instructions and construct instructions to tune input data;

[0130] Specifically, the general instructions are as follows: If you are a geometry expert, read the question carefully, observe and analyze the geometric pictures, generate a detailed reasoning process and give an answer; this instruction is the same as the question in step 4. Geometric pictures They are used as input information, and two special tags [Language Reasoning] and [Program Reasoning] are used to distinguish different thinking chains. These two special tags are spliced to the end of all inputs. The input data of the language thinking chain and the program thinking chain are expressed as formula (1) and formula (2) respectively:

[0131]

[0132] Among them, I represents a general instruction, [Language Reasoning] indicates that the model should generate a language thinking chain, and [Program Reasoning] indicates that the model should generate a program thinking chain.

[0133] For example, in this embodiment, the input data for instruction tuning For: "If you are ageometry expert,carefully read the question,observe and analyze the geometricimage,generate a detailed reasoning process and provide an answer;Question:Asshown in the figure,AB is the diameter of⊙O,point C is on⊙O,radius OD∥AC,if∠BOD=130°,then the degree of∠B is (); Image: [Language Reasoning];"; Input data For: "If you are a geometry expert, carefully read the question, observe and analyze the geometric picture, generate a detailed reasoning process and provide an answer; Question: As shown in the figure, AB is the diameter of ⊙ O, point C is on ⊙ O, radius OD ∥ AC, if ∠ BOD = 130°, then the degree of ∠ B is (); Image: [Program Reasoning];", where [Language Reasoning] indicates that the model should generate a language thinking chain, and [Program Reasoning] indicates that the model should generate a program thinking chain.

[0134] Step 16: Instruct to tune a small-scale multimodal model to achieve joint learning of multiple thought chains.

[0135] Specifically, the 7B parameter-scale LLaVA model is used as a small-scale multimodal model. The model includes three core modules: a language model, a pre-trained visual encoder, and a projection layer. The language model uses the Vicuna model for natural language understanding and generation; the pre-trained visual encoder, such as the Vision Transformer (ViT), is used to extract visual features from images; the projection layer maps the visual features output from the visual encoder to the same embedding space dimension as the language model to achieve alignment of visual and semantic features.

[0136] Input the data in step 15 into the LLaVA model. Following the traditional autoregressive objective of training large language models, the model needs to predict the Ground Truth label data in step 14 as the next word in the language thinking chain and the program thinking chain respectively. The loss function calculation formula is shown in formula (3)-formula (5):

[0137]

[0138] Among them, formulas (3) and (4) represent the loss functions when the model predicts the Ground Truth label data as language thought chain and program thought chain, respectively, and formula (5) represents the loss when jointly calculating multiple thought chains; Θ is a trainable parameter, Z L The output data is the length of the language thinking chain, Z P The output data is the length of the program thinking chain; and are the input data of the language thinking chain and the program thinking chain of formula (1) and formula (2) in step 15 respectively; Indicates that given input data and output language thinking chain c L The probability of generating the i-th word under the condition of the first i-1 words; Indicates that given input data and output program thinking chain c P The probability of generating the i-th word under the condition of the first i-1 words.

[0139] Step 17: Generate language thinking chain and program thinking chain respectively, and verify them through reasoning.

[0140] Specifically, by appending the special tokens [Language Reasoning] and [Program Reasoning] to the corresponding input data, the small-scale multimodal model (LLaVA) was guided to generate a language reasoning chain and a program reasoning chain, respectively. The language reasoning chain appended the characters "The answer is:" at its end. A regular expression was used to extract the predicted answer from the language reasoning chain. If the regular expression failed to match, the answer was invalid. The program reasoning chain directly printed the variables according to the prompt template. The answer was obtained by simply running the generated Python program. If the program failed to run, the answer was invalid. The results of the two reasoning chains were combined for verification. First, the Python program of the program reasoning chain was run to obtain the result. If the output was not included in the answer candidate options, the answer from the language reasoning chain was used as the prediction. It is important to note that to ensure experimental fairness, the model did not pre-patch the answer candidate during training or inference. After obtaining the combined results, accuracy was calculated, where accuracy refers to the ratio of the number of questions correctly predicted by the model to the total number of questions.

[0141] For example, by adding special tags [Language Reasoning] and [Program Reasoning] after the corresponding input data, the small-scale multimodal model LLaVA is guided to generate language thinking chains and program thinking chains respectively. Figure 2 shown.

[0142] After running the Python program of the program thinking chain, the value of angle_B is 40.0; the results obtained by the language thinking chain and the program thinking chain in this embodiment are consistent with the original result 40.0 of the data, so Figure 2 The accuracy of the displayed data is 100%; although the language thinking chain and program thinking chain generated in this embodiment are not completely consistent with the thinking chain generated using the multimodal large language model GPT-4V in steps 9 and 12, the logical thinking of reasoning is consistent.

[0143] At this point, after steps 1 to 17, a knowledge-driven multi-thinking chain cross-modal geometric problem solving method has been completed.

[0144] The method described in the present invention is knowledge-driven, which is consistent with the idea that humans start from geometric knowledge points when solving geometric problems, further apply geometric knowledge theorems, perform reasoning and calculation step by step, and finally get the answer. The method uses the multimodal large language model GPT-4V in the stages of querying geometric knowledge theorems, screening geometric knowledge points, generating language thinking chains and program thinking chains, and the generated language thinking chains and program thinking chains will serve as weakly labeled data for instruction tuning of small-scale multimodal models. Therefore, the cost of manpower and time is greatly saved in terms of data annotation. The language thinking chain generated by the method is composed of a chain of geometric knowledge points, a chain of geometric knowledge theorems, and a chain of relationships between different geometric knowledge and geometric pictures, making the problem-solving process more explainable, and the generated program thinking chain is more formalized, and the calculation process is more precise and rigorous. In the stage of instruction tuning of the small-scale multimodal model, the method jointly trains the language thinking chain and the program thinking chain. The two thinking chains can guide and promote each other during the training process, which helps to improve the logical reasoning ability of the small-scale multimodal model and improve the accuracy of the model in solving geometric problems. The method does not require additional parameters, has good versatility, is highly practical, and is easy to promote.

[0145] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0146] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0147] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0148] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be applied in the widest sense consistent with the principles and novel features of the present invention.

[0149] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A knowledge-driven multi-thinking chain cross-modal geometric problem solving method, characterized by: include: Input a candidate set of geometric knowledge points, wherein the candidate set of geometric knowledge points includes a plurality of basic theoretical knowledge points for solving geometric problems; Based on the multimodal large language model, querying the geometric knowledge theorem corresponding to each geometric knowledge point in the geometric knowledge point candidate set to generate a geometric knowledge theorem set; Convert each theorem in the set of geometric knowledge theorems into an executable Python function to form a set of geometric knowledge theorems Python functions; Input the geometry problem to be solved, the corresponding geometry picture and the answer; Constructing a geometry knowledge point screening instruction, combining the geometry problem, geometry picture, answer, and geometry knowledge point candidate set, and screening out a subset of geometry knowledge points related to the problem through a multimodal large language model; Extracting corresponding theorem subsets and Python function subsets from the geometric knowledge theorem set and the geometric knowledge theorem Python function set according to the geometric knowledge point subsets; Constructing a language thought chain prompt template to guide the multimodal large language model to generate a language thought chain, and verifying the consistency of the language thought chain with the answer through regular matching; Constructing a program thinking chain prompt template to guide the multimodal large language model to generate an executable Python program chain, and verifying the consistency of the result with the answer by running the program; The successfully verified language thinking chain and program thinking chain are used as training data, and combined with general instructions to tune the instructions of the small-scale multimodal model to achieve joint learning of the language thinking chain and the program thinking chain; Through the small-scale multimodal model, language thinking chains and program thinking chains are generated respectively, and the final answer is output after comprehensive verification.

2. The knowledge-driven multi-thinking chain cross-modal geometric problem solving method according to claim 1 is characterized in that: The multimodal large language model is GPT-4V, which is used to retrieve geometric knowledge theorems associated with text and image modalities.

3. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 1 is characterized in that: The geometric knowledge point screening instructions are described in natural language to guide the model to screen knowledge points related to questions, geometric pictures and answers from the candidate set.

4. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 1 is characterized in that: The language thinking chain includes the following sub-steps: Infer the underlying geometric knowledge points in the problem; Determine the geometric theorems involved; Establish the corresponding relationship between geometric knowledge theorems and image elements; Add answers to the end of your chain of thoughts and check for consistency.

5. The knowledge-driven multi-thinking chain cross-modal geometric problem solving method according to claim 4 is characterized in that: The input data of the language thinking chain is: in, Represents the input data of the language thinking chain, I represents the general instruction, Indicates a problem, Represents a geometric picture, and [Language Reasoning] is a special tag, indicating that the model should generate a language thinking chain.

6. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 1 is characterized in that: The requirements for generating the program thinking chain include: calling a function from said subset of Python functions; The variable name should be consistent with the geometric picture element; Generates comment-free and executable code; Output the result directly and match it with the answer.

7. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 6 is characterized in that: The input data of the program thinking chain is: in, Represents the input data of the program thinking chain, I represents the general instruction, Indicates a problem, Represents a geometric picture, and [Program Reasoning] is a special tag, indicating that the model should generate a program thinking chain.

8. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 1 is characterized in that: The small-scale multimodal model in the step is an LLaVA model, which trains the language thinking chain and the program thinking chain through a joint loss function, and the loss function is: Among them, formulas (3) and (4) represent the loss functions when the model predicts the Ground Truth label data as language thought chain and program thought chain, respectively, and formula (5) represents the loss when jointly calculating multiple thought chains; Θ is a trainable parameter, Z L The output data is the length of the language thinking chain, Z P The output data is the length of the program thinking chain; and They are the input data of the language thinking chain and the program thinking chain respectively; Indicates that given input data and output language thinking chain c L The probability of generating the i-th word under the condition of the first i-1 words; Indicates that given input data and output program thinking chain c P The probability of generating the i-th word under the condition of the first i-1 words.

9. The knowledge-driven multi-thinking chain cross-modal geometry problem solving method according to claim 1 is characterized in that: The comprehensive verification includes: Prioritize executing the Python code generated by the program thinking chain to obtain the results; If the program result is invalid, then extract the answer from the language thinking chain; The prediction accuracy of the statistical model is used as the final evaluation indicator.

10. A system for implementing the method according to any one of claims 1 to 9, characterized in that: include: Data input module: used to input candidate sets of geometric knowledge points, geometric problems to be solved, corresponding geometric pictures and answers; Knowledge query and generation module: Based on a multimodal large language model, it queries the geometric knowledge theorem corresponding to each geometric knowledge point in the candidate set of geometric knowledge points, generates a set of geometric knowledge theorems, and writes executable Python functions for the theorems to form a set of geometric knowledge theorems Python functions; Knowledge point screening module: Constructs geometric knowledge point screening instructions, combines geometric problems, geometric pictures, answers, and a set of geometric knowledge point candidates, and uses a multimodal large language model to screen out a subset of geometric knowledge points related to the problem; Subset extraction module: based on the selected subset of geometric knowledge points, extract the corresponding theorem subset and Python function subset from the geometric knowledge theorem set and the geometric knowledge theorem Python function set; Thinking chain construction and verification module: including language thinking chain unit and program thinking chain unit; The language thinking chain unit is used to construct a language thinking chain prompt template, guide the multimodal large language model to generate a language thinking chain, and verify the consistency of the language thinking chain with the answer through regular matching; The program thinking chain unit is used to build a program thinking chain prompt template, guide the multimodal large language model to generate an executable Python program chain, and verify the consistency of the result and the answer by running the program; Model training module: Uses successfully verified language thought chains and program thought chains as training data, combines general instructions to optimize the instructions of a small-scale multimodal model, and realizes the joint learning of language thought chains and program thought chains; Result generation and verification module: Through the trained small-scale multimodal model, language thinking chains and program thinking chains are generated respectively. The Python code generated by the program thinking chain is executed first to obtain the results. If the program result is invalid, the answer in the language thinking chain is extracted. The statistical model prediction accuracy is used as the final evaluation indicator to output the final answer.

Images