Mathematical problem solving method and device based on multi-modal large model and electronic equipment

Abstract

The invention provides a mathematical problem solving method and device based on a multi-modal large model and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a mathematical element image of a mathematical problem; inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by training based on a sample mathematical element image and a positive sample text description and a negative sample text description corresponding to the sample mathematical element image; inputting the image vector into a self-adaptive module to obtain an image conversion coding vector output by the self-adaptive module; the adaptive module is obtained based on training of a sample image vector and a sample text vector; determining question stem characters of the mathematical question, and inputting the question stem characters and the image conversion coding vector into the large language model to obtain a prediction answering process output by the large language model; the large language model is obtained based on sample question stem characters, sample image conversion coding vectors and sample answering process training, and the mathematical problem solving capability of the multi-modal large model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and electronic equipment for solving mathematical problems based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal large models are increasingly used in image processing, natural language processing, and cross-modal tasks. In the field of education, especially in solving mathematical problems, traditional single-modality-based solution methods have gradually shown their limitations. Mathematical problems usually contain rich visual information and complex text descriptions. It is difficult to fully capture the core elements of the problem by relying solely on text input. Therefore, how to combine multimodal methods of images and text has become an important issue that needs to be solved to improve the ability to solve mathematical problems. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a method, device and electronic device for solving mathematical problems based on a multi-modal large model.

[0004] The present invention provides a method for solving mathematical problems based on a multimodal large model, comprising: Determine the mathematical element image of the mathematical problem; Inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; Inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on a sample image vector and a sample text coding vector; Determine the stem text of a math problem, input the stem text and the image conversion coding vector into a large language model, and obtain a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vector and sample solution process.

[0005] According to a method for solving mathematical problems based on a multimodal large model provided by the present invention, the image coding model is obtained by training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, including: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; Inputting the positive sample text descriptions into a pre-trained text encoding model respectively to obtain a text embedding vector corresponding to the positive sample text description output by the text encoding model; Determine the text similarity between each of the text embedding vectors, and select a negative sample text description of each of the sample geometric images based on the text similarity.

[0006] According to a mathematical problem solving method based on a multimodal large model provided by the present invention, determining the text similarity between the text embedding vectors, and selecting a negative sample text description of each sample geometric image based on the text similarity, comprising: Calculate the similarity between the two text embedding vectors; Determine a target text embedding vector from the text embedding vectors, and determine the similarity between the target text embedding vector and the remaining text embedding vectors; Selecting the positive sample text descriptions corresponding to the K text embedding vectors with the highest similarity as the negative sample text descriptions of the target sample geometric image corresponding to the target text embedding vector; The process of determining the target text embedding vector from the text embedding vector is repeated until a negative sample text description of each of the sample geometric images is selected.

[0007] According to a method for solving mathematical problems based on a multimodal large model provided by the present invention, the image coding model is obtained by training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, including: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; The positive sample text description and the modification prompt word are respectively input into a general language model to obtain L negative sample text descriptions output by the general language model; wherein the negative sample text description is obtained by slightly perturbing and modifying the positive sample text description.

[0008] According to a method for solving mathematical problems based on a multimodal large model provided by the present invention, the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, comprising: Inputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample text descriptions into an original image coding model to obtain a predicted image vector output by the original image coding model; Determine a loss function of the predicted image vector and the corresponding positive sample text description and a preset number of negative sample texts, and adjust the parameters of the original image coding model based on the loss function; Repeat the process of outputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample texts to the original image coding model until the original image coding model converges to obtain the image coding model.

[0009] According to a mathematical problem solving method based on a multimodal large model provided by the present invention, the negative sample text description is obtained by slightly perturbing the positive sample text description, including: The negative sample text description is obtained by slightly perturbing and modifying the positive sample text description based on geometric element attributes, letters and letter sequences, geometric element values, and geometric element relationships.

[0010] The present invention also provides a mathematical problem solving device based on a multi-modal large model, comprising: A mathematical problem determination module, used for determining a mathematical element image of a mathematical problem; An image vector acquisition module, used for inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; A vector conversion module, used for inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is obtained by training based on sample image vectors and sample text coding vectors; The solution process prediction module is used to determine the stem text of the mathematical problem, input the stem text and the image conversion coding vector into a large language model, and obtain the predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vector and sample solution process.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for solving mathematical problems based on a multimodal large model as described above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for solving mathematical problems based on a multimodal large model.

[0013] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for solving mathematical problems based on a multimodal large model.

[0014] The mathematical problem-solving method, device and electronic device based on a multimodal large model provided by the present invention understand the image vector corresponding to the mathematical element image output of the mathematical problem through an image coding model; understand the question stem text of the mathematical problem through a large language model; and convert the image vector into an image conversion coding vector that can be understood by the large language model through an adaptive module, so that the large language model can understand the mathematical problem and perform reasoning in combination with the question stem text and image information description of the mathematical problem, thereby improving the mathematical problem-solving ability of the multimodal large model and improving the reliability of the prediction and solution process of the output mathematical problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the method for solving mathematical problems based on a multimodal large model provided by the present invention.

[0017] Figure 2 It is a schematic diagram of the framework of the mathematical problem solving method based on the multimodal large model provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of a mathematical problem solving device based on a multi-modal large model provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figure 1-Figure 4 The invention describes a method, device and electronic device for solving mathematical problems based on a multimodal large model.

[0022] Figure 1 It is a flow chart of the method for solving mathematical problems based on a multi-modal large model provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Determine a mathematical element image of a mathematical problem.

[0023] Mathematical element images refer to images including mathematical elements that need to be understood by the multimodal large model when solving mathematical problems. For example, they can be images including geometric element attributes, geometric element values, or geometric element relationships in geometric mathematical problems.

[0024] It can be understood that the attributes of geometric elements refer to the characteristics of the shapes of geometric elements such as triangles, circles or trapezoids, the values ​​of geometric elements refer to the characteristics of the sizes of geometric elements such as length, angle or radian, and the relationships between geometric elements refer to the characteristics of the movement between two geometric elements such as the intersection of two lines, the tangency of two circles or the similarity of triangles.

[0025] Step 102: input the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions.

[0026] An image vector is a numerical representation of image element information obtained by processing a mathematical element image with an image coding model. An image coding model is a multimodal large model component that converts a mathematical element image into an image vector containing image information.

[0027] A positive sample text description refers to a text description whose text description content is consistent with the content in the corresponding sample mathematical element image. A negative sample text description refers to a text description whose text description content is inconsistent with the content in the corresponding sample mathematical element image. It can be understood that a negative sample text description does not need to have a text description content that is opposite to the content in the corresponding sample mathematical element image.

[0028] For example, the sample geometric feature image contains a square ABCD and a line segment EF, where the side length of the square ABCD is 5 cm and the line segment EF is parallel to the side AB of the square; the positive sample text description is: "There is a square ABCD in the image, whose side length is 5 cm. The line segment EF is parallel to the side AB of the square." The negative sample text description can be: "There is a rectangle ABCD in the image, whose side length is 5 cm. The line segment EF is parallel to the side AB of the rectangle", or: "There is a square ABCD in the image, whose side length is 5 cm. The line segment EF is perpendicular to the side AB of the square", etc.

[0029] Exemplarily, a geometric image can be constructed using a Python script, and then a large language model can be used to convert the Python script into a text description in natural language to obtain the corresponding positive sample text description, and the adjusted Python script can be converted into a text description in natural language to obtain the corresponding negative sample text description. In this way, the deficiency of insufficient geometric images and corresponding positive sample text descriptions and negative sample text descriptions can be compensated in quantity.

[0030] Compared to obtaining an image coding model only through training with positive sample text descriptions, in this embodiment, when training the original image coding model, by performing comparative learning between the sample mathematical element images and the corresponding positive sample text descriptions and negative sample text descriptions, the image vector output by the trained image coding model is aligned with the content in the positive sample text description, that is, the image coding model in the multimodal large model obtains image-text alignment capability or the ability to understand mathematical element images through comparative learning, which can reduce the amount of data required for the image coding model to obtain the image-text alignment capability, reduce the cost of constructing training sample data, improve the learning efficiency of the image coding model, and enhance the mathematical reasoning ability of the multimodal large model obtained based on the image coding model.

[0031] Step 103, input the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on sample image vectors and sample text coding vectors.

[0032] The adaptive module, which may also be referred to as an intermediate module, an intermediate layer or a conversion module, refers to a multimodal large model component that converts an image vector to obtain a conversion coding vector. The conversion coding vector refers to a numerical representation of the acceptable input of the large language model output by the adaptive module. Therefore, the adaptive module can also be said to be a multimodal large model component that connects the image coding model and the large language model, and is used to convert the image vector to a feature space compatible with the text vector to obtain a conversion coding vector.

[0033] Exemplarily, sample image vectors and sample text encoding vectors can be obtained through the G-LLaVA alignment dataset. Specifically, the images in the G-LLaVA alignment dataset can be generated through a trained image encoding model to obtain corresponding image vectors to obtain sample image vectors, and the text corresponding to the images in the G-LLaVA alignment dataset can be converted into conversion encoding vectors through methods such as a text encoder to obtain corresponding sample text encoding vectors. Repeating this process multiple times can obtain multiple groups of sample image vectors and sample text encoding vectors to determine the sample set for training the adaptive module.

[0034] Step 104, determine the stem text of the math problem, input the stem text and the image conversion coding vector into a large language model, and obtain a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vector and sample solution process.

[0035] The stem text refers to the text description part that needs to be understood by the multimodal large model when solving mathematical problems. The large language model is a multimodal model component that can understand and generate natural language text.

[0036] It can be understood that the prediction and answering process output by the large language model is an answering process in the form of text information, which can be obtained by the large language model understanding, reasoning and calculating the image element information represented by the image conversion coding vector based on the text in the question stem.

[0037] Exemplarily, the image coding model may adopt the AltCLIP model to convert the input mathematical element image into an image vector; the adaptive module may adopt a two-layer perceptron to convert the image vector into an input vector of the large language model; the large language model may adopt the Mammoth2-7B model to output a predicted answer process based on the text and image of the question stem; the framework of the multimodal large model may adopt LLaVA.

[0038] The predicted solution process refers to the steps and answers to the math problem output by the large language model after understanding the text and image conversion encoding vectors of the math problem. It is understandable that the predicted solution process is the steps and answers to the math problem that the multimodal large model believes may be different from the steps and answers to the math problem. For example, the steps may be wrong but the answer may be correct, or the steps may be correct but the answer may be wrong.

[0039] Exemplarily, a middle school geometry data set can be selected to determine the mathematical element images, stem text, and solution process of mathematical problems. Specifically, the middle school geometry data set may include 17,000 mathematical problems. The mathematical element image of a mathematical problem can be first input into a trained image coding model to obtain its image vector, and the image vector can be input into a trained adaptive module to obtain its image conversion coding vector, and a sample image conversion coding vector can be obtained. The stem text of the mathematical problem is used as a sample stem text, and the solution process of the mathematical problem is used as a sample solution process; repeating multiple times can obtain multiple groups of sample stem text, sample image conversion coding vectors, and sample solution processes, and determine the sample set for training the large language model.

[0040] The mathematical problem solving method based on the multimodal large model provided in the embodiment of the present invention understands the image vector corresponding to the mathematical element image output of the mathematical problem through the image coding model; understands the question stem text of the mathematical problem through the large language model; and converts the image vector into an image conversion coding vector that can be understood by the large language model through an adaptive module, so that the large language model can understand the mathematical problem and perform reasoning in combination with the question stem text and image information description of the mathematical problem, which can improve the mathematical problem solving ability of the multimodal large model and improve the reliability of the prediction and solution process of the output mathematical problem.

[0041] As mentioned above, the method for solving mathematical problems based on the multimodal large model provided by the present invention requires the recognition and understanding of two parts of input, one is the stem text of the mathematical problem, and the other is the mathematical element image of the mathematical problem. The accuracy of the recognition and understanding of the two parts of input will affect the mathematical problem solving ability of the multimodal large model, or in other words, accurately recognizing and understanding the stem text and mathematical element image of the mathematical problem is a priori step for the multimodal large model to perform effective reasoning.

[0042] The recognition and understanding of mathematical element images depends on the image coding model, which is obtained by comparative learning between sample mathematical element images and the corresponding positive sample text descriptions and negative sample text descriptions. Currently, when the original image coding model is trained by comparative learning of positive sample text descriptions and negative sample text descriptions, the negative sample text descriptions used are the positive sample text descriptions of other sample mathematical element images in the same batch.

[0043] This will result in a low correlation between the negative sample text description and the sample mathematical element image. For example, the geometric image in one sample mathematical element image is a triangle, and the positive sample text description of another sample mathematical element image that is a negative sample text description is that the geometric image is a circle. This will lead to the poor image-text alignment ability of the trained image coding model, and it can only recognize and understand the sample mathematical element image at a coarse-grained level, but cannot recognize and understand the sample mathematical element image at a coarse-grained level, for example, it can compare triangles and circles, but cannot compare different triangles. At the same time, it will also reduce the learning efficiency of comparative learning training of the original image coding model.

[0044] To solve the above technical problem, based on any of the above embodiments, the image coding model is obtained by training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, including: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; Inputting the positive sample text descriptions into a pre-trained text encoding model respectively to obtain a text embedding vector corresponding to the positive sample text description output by the text encoding model; Determine the text similarity between each of the text embedding vectors, and select a negative sample text description of each of the sample geometric images based on the text similarity.

[0045] Specifically, in this embodiment, by controlling the unique correspondence between the sample geometric image and the positive sample text description, the positive sample text description of one sample geometric image can be essentially different from the remaining sample geometric images, thereby avoiding the false negative problem.

[0046] For example, a geometric image in the MAVIS-CAPTION dataset can be used as a sample geometric image, and the unique text description corresponding to the geometric image in the MAVIS-CAPTION dataset can be used as a positive sample text description. The obtained sample geometric image and the unique corresponding positive sample text description are a set of training samples. In this way, multiple sets of training samples can be obtained through the MAVIS-CAPTION dataset. The pre-trained text encoding model can be the SimANS model published in the EMNLP conference.

[0047] The text similarity between the text embedding vectors can be determined based on the similarity between each pair of text embedding vectors and the remaining text embedding vectors, or the text embedding vectors can be clustered to obtain clusters, and the text similarity between the text embedding vectors can be determined through the clusters.

[0048] In this embodiment, a text embedding vector corresponding to each positive sample text description is obtained through a pre-trained text coding model. After determining the text similarity between each of the text embedding vectors, the positive sample text description corresponding to other text embedding vectors with high text similarity can be selected as the negative sample text description of the sample geometric image based on the text embedding vector corresponding to a sample geometric image. This can improve the correlation between the negative sample text description and the sample mathematical element image, thereby improving the image-text alignment capability of the trained image coding model, and at the same time improving the learning efficiency of comparative learning training of the original image coding model.

[0049] In one embodiment, a target sample geometric image and its corresponding target text embedding vector may be determined from the plurality of groups of sample geometric images, and a negative sample text description of the target sample geometric image may be selected based on text similarity between the target text embedding vector and the remaining text embedding vectors; The process of determining the target sample geometric image from the multiple groups of sample geometric images is repeated until a negative sample text description of each of the sample geometric images is obtained.

[0050] Based on any of the above embodiments, determining the text similarity between the text embedding vectors, and selecting a negative sample text description of each of the sample geometric images based on the text similarity, comprising: Calculate the similarity between the two text embedding vectors; Determine a target text embedding vector from the text embedding vectors, and determine the similarity between the target text embedding vector and the remaining text embedding vectors; Selecting the positive sample text descriptions corresponding to the K text embedding vectors with the highest similarity as the negative sample text descriptions of the target sample geometric image corresponding to the target text embedding vector; The process of determining the target text embedding vector from the text embedding vector is repeated until a negative sample text description of each of the sample geometric images is selected.

[0051] Exemplarily, it is preferred to calculate the cosine similarity score between the two text embedding vectors to obtain the similarity between the two text embedding vectors, or to calculate the Euclidean distance between the two text embedding vectors to obtain the similarity between the two text embedding vectors, etc. This is not further limited in the present embodiment.

[0052] Exemplarily, each text embedding vector can be determined as a target text embedding vector one by one in a set order from the text embedding vectors corresponding to multiple sample geometric images to determine the similarity between each text embedding vector and its corresponding remaining text embedding vectors, and the negative sample text description of the corresponding sample geometric image is selected, so as to construct a training sample set in which each sample geometric image has 1 positive sample text description and K negative sample text descriptions.

[0053] The number K of text embedding vectors can be set according to actual needs. This is not further limited in this embodiment. Exemplarily, K can be 100.

[0054] Compared with the subjective or heuristic selection of negative sample text descriptions, in this embodiment, the similarity between the text embedding vectors is calculated, and on this basis, the negative sample text description of the sample geometric image is selected, so that the correlation between the selected negative sample text description and the sample geometric image can be quantified, thereby improving the reliability of the high correlation between the negative sample text description and the sample geometric image.

[0055] Based on any of the above embodiments, the image coding model is obtained by training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, including: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; The positive sample text description and the modification prompt word are respectively input into a general language model to obtain L negative sample text descriptions output by the general language model; wherein the negative sample text description is obtained by slightly perturbing and modifying the positive sample text description.

[0056] In this embodiment, a geometric image in the MAVIS-CAPTION dataset may be used as a sample geometric image, and a text description uniquely corresponding to the geometric image in the MAVIS-CAPTION dataset may be used as a positive sample text description.

[0057] Exemplarily, the general language model may be a language model such as GLM-4 or GPT-4o.

[0058] Similar to determining the number K of text embedding vectors, the number L of negative sample text descriptions output by the general language model can be set according to actual needs. This is not further limited in this embodiment. Exemplarily, L can be 10.

[0059] Specifically, a text description uniquely corresponding to a geometric image can be determined as the positive sample text description, and the positive sample text description and the modification prompt word can be input into GLM-4 to obtain 10 negative sample text descriptions output by GLM-4. The obtained geometric image and its corresponding positive sample text description and 10 negative sample text descriptions can be used to train the image coding model.

[0060] In this embodiment, the positive sample text description is modified through a general language model so that most of the content of the modified negative sample text description is similar to the positive sample text description, but there are several differences. The negative sample text description can be constructed around a single sample geometric image, thereby further improving the correlation between the negative sample text description and the sample mathematical element image, thereby improving the image-text alignment capability of the trained image coding model.

[0061] In order to reduce the risk of outputting hallucinatory answers during the general large model reasoning process, the effectiveness of modifying the positive sample text description is provided. Based on any of the above embodiments, the negative sample text description is obtained by slightly perturbing the positive sample text description, including: the negative sample text description is obtained by slightly perturbing the positive sample text description based on geometric element attributes, letters and letter sequences, geometric element values, and geometric element relationships.

[0062] Exemplarily, geometric element attributes may include rectangles, squares, circles, ellipses, right triangles, isosceles triangles or trapezoids, etc., which can reflect the nature and characteristics of geometric elements. Letters and letter sequences are used to represent the coordinates and order of geometric elements, such as ABC in triangle ABC, ABCD in trapezoid ABCD, and such features that are marked with letters and arranged in a certain order. Geometric element values ​​include side lengths, angles, and radians. For example, the side length is 8 cm. The relationship between geometric elements includes the intersection, translation, tangency, and symmetry of two or more geometric elements, which can represent the action relationship between the elements.

[0063] On this basis, this embodiment provides an example of modifying the prompt words as follows: For the positive sample text description, perform a minor modification of the local content. The text description after each modification should have 1 to 4 positions different. The modification range includes: A. Concepts of specific attributes, such as square, rectangle, triangle, line segment, parallel lines, intersection, etc. For example: square ABCD can be modified to rectangle ABCD, the diameter of circle O1 can be modified to the radius of circle O1, and an equilateral triangle can be modified to an isosceles triangle; B. Modification of specific letter indicators. When it is required to modify letter indicators, change more than two letters of indicators, but avoid the same head and tail order. For example, changes such as ABCD and CDAB are not allowed. The modified letter order should avoid being consistent with the original indicator. For example, square ABCD can be modified to square CBDA, and line segment AB parallel to CD can be modified to line segment AD parallel to CB. C. Modify the value. For example, the side length of square ABCD is 8 cm, which can be modified to 6 cm. D. Modification of transformation verbs, such as verbs of geometric elements such as parallel, intersecting, similar, equal and perpendicular. For example: Line segment AB is parallel to CD, which can be modified to line segment AB is perpendicular to CD; E. Do not use specific letters that are not in the text description, such as the trapezoidal shape GHIJ, which will not work if it is changed to the trapezoidal shape XYZO; F. Make changes throughout the entire text description. Do not focus on just one location, such as the first or second half, but make changes evenly throughout the entire text.

[0064] Currently, when the original image coding model is trained through comparative learning of positive sample text descriptions and negative sample text descriptions, the corresponding negative sample text descriptions are generally not set for the sample mathematical element images alone. Instead, the positive sample text descriptions of other sample mathematical element images in the same batch are used as their negative sample text descriptions. The number of negative sample text descriptions is limited by the batch size, which may lead to insufficient semantic discrimination of negative samples, and thus cause the trained image coding model to have deviations in the deep semantic understanding of mathematical element images.

[0065] To solve this problem, based on any of the above embodiments, the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, including: Inputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample text descriptions into an original image coding model to obtain a predicted image vector output by the original image coding model; Determine a loss function of the predicted image vector and the corresponding positive sample text description and a preset number of negative sample texts, and adjust the parameters of the original image coding model based on the loss function; Repeat the process of outputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample texts to the original image coding model until the original image coding model converges to obtain the image coding model.

[0066] On the basis that the mathematical problem solving method based on the multimodal large model provided by the present invention can generate any number of negative sample text descriptions according to needs, in this embodiment, the original image coding model is trained by using sample mathematical element images and their corresponding positive sample text descriptions and a preset number of negative sample text descriptions. By adjusting the preset number of negative sample text descriptions, the intensity of the gradient learning of the negative sample text descriptions can be adjusted, so that on the basis of balancing the computational cost, the negative term in the loss function provides a stronger gradient signal, thereby guiding the original image coding model to be trained faster to obtain the image coding model.

[0067] Exemplarily, the geometric dataset for training the image coding model can be expressed as:

[0068] in, is a sample math element image, is the positive example text description of the sample mathematical element image, is the negative text description of the sample mathematical element image, The number of negative sample text descriptions for each sample mathematical element image, is the total number of trainings.

[0069] Understandably, .

[0070] Among them, K is the number of negative sample text descriptions of each sample geometric image selected based on text similarity, and L is the number of negative sample text descriptions obtained by slightly perturbing the positive sample text description.

[0071] Loss Function It can be expressed by the following formula:

[0072] in, is the cosine similarity score between the predicted image vector and the vector of the positive example text description, The cosine similarity score between the predicted image vector and the vector of the negative text description.

[0073] For the positive sample text description gradient can be calculated by the following formula:

[0074] For negative sample text description gradient, it can be calculated by the following formula:

[0075] In order to specifically illustrate the function of the mathematical problem solving method based on a multimodal large model provided by this embodiment, a specific example is provided below.

[0076] like Figure 2 As shown, a negative sample text description is obtained by a retrieval method, specifically, multiple groups of sample geometric images and positive sample text descriptions uniquely corresponding to each group of the sample geometric images are determined through a geometric data set; the positive sample text descriptions are respectively input into a pre-trained text encoding model to obtain text embedding vectors corresponding to the positive sample text descriptions output by the text encoding model; the similarity between the two text embedding vectors is calculated; a target text embedding vector is determined from the text embedding vectors, and the similarity between the target text embedding vector and the remaining text embedding vectors is determined; the positive sample text descriptions corresponding to the K text embedding vectors with the highest similarity are selected as the negative sample text descriptions of the target sample geometric image corresponding to the target text embedding vector; the process of determining the target text embedding vector from the text embedding vector is repeated until a negative sample text description of each of the sample geometric images is selected; The negative sample text description is obtained by a micro-perturbation modification method. Specifically, multiple groups of sample geometric images and positive sample text descriptions uniquely corresponding to each group of sample geometric images are determined by a geometric data set; the positive sample text descriptions and modification prompt words are respectively input into a universal language model to obtain L negative sample text descriptions output by the universal language model; wherein the negative sample text description is obtained by micro-perturbation modification of the positive sample text description based on geometric element attributes, letters and letter sequences, geometric element values, and geometric element relationships; Input the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample text descriptions into the original image coding model to obtain a predicted image vector output by the original image coding model; determine a loss function of the predicted image vector and the corresponding positive sample text description and a preset number of negative sample texts, and adjust the parameters of the original image coding model based on the loss function; repeat the process of outputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample texts into the original image coding model until the original image coding model converges to obtain the image coding model; Determine a mathematical element image of a mathematical problem; input the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; input the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on sample image vectors and sample image conversion coding vectors; determine the stem text of the mathematical problem, input the stem text and the image conversion coding vector into a large language model, and obtain a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vectors and sample solution processes.

[0077] The mathematical problem solving method based on the multimodal large model provided by the present invention utilizes a retrieval method and a micro-perturbation modification method to construct a negative sample text description so that the elements in the image can be fully represented; and a method for training an image coding model for any number of negative samples is proposed, which can greatly improve the image-text alignment capability of the image coding model, provide the accuracy of the image vector output by the image coding model in representing the geometric elements, and then enhance the mathematical reasoning effect of the multimodal large model.

[0078] Experiments on the data sets MM-Math, MathVista, and GeoQA of the mathematical problem solving method based on the multimodal large model provided by the present invention show that the mathematical reasoning ability of the multimodal large model obtained on the basis of constructing the negative sample text description using the proposed retrieval method and micro-perturbation modification method is enhanced, especially in terms of the accuracy of recognizing mathematical elements in mathematical element images, the effect is much better than the existing recognition model.

[0079] In addition, the mathematical problem solving method based on a multimodal large model provided by the present invention can effectively expand the depiction of geometric elements, thereby solving the problem of insufficient negative sample training data for the image coding model.

[0080] The mathematical problem solving device based on the multimodal large model provided by the present invention is described below. The mathematical problem solving device based on the multimodal large model described below and the mathematical problem solving method based on the multimodal large model described above can be referenced to each other.

[0081] Figure 3 is a schematic diagram of the structure of a mathematical problem solving device based on a multi-modal large model provided by the present invention, such as Figure 3 As shown, the device comprises: A math problem determination module 310 is used to determine a math element image of a math problem; An image vector acquisition module 320 is used to input the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; A vector conversion module 330, used for inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is obtained by training based on sample image vectors and sample text coding vectors; The solution process prediction module 340 is used to determine the stem text of the math problem, input the stem text and the image conversion coding vector into a large language model, and obtain the predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vector and sample solution process.

[0082] Based on any of the above embodiments, the negative sample generation module is used to: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; Inputting the positive sample text descriptions into a pre-trained text encoding model respectively to obtain a text embedding vector corresponding to the positive sample text description output by the text encoding model; Determine the text similarity between each of the text embedding vectors, and select a negative sample text description of each of the sample geometric images based on the text similarity.

[0083] Based on any of the above embodiments, the negative sample generation module is specifically used to: Calculate the similarity between the two text embedding vectors; Determine a target text embedding vector from the text embedding vectors, and determine the similarity between the target text embedding vector and the remaining text embedding vectors; Selecting the positive sample text descriptions corresponding to the K text embedding vectors with the highest similarity as the negative sample text descriptions of the target sample geometric image corresponding to the target text embedding vector; The process of determining the target text embedding vector from the text embedding vector is repeated until a negative sample text description of each of the sample geometric images is selected.

[0084] Based on any of the above embodiments, the negative sample generation module is further used to: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; The positive sample text description and the modification prompt word are respectively input into a general language model to obtain L negative sample text descriptions output by the general language model; wherein the negative sample text description is obtained by slightly perturbing and modifying the positive sample text description.

[0085] Based on any of the above embodiments, the negative sample generation module is specifically used to obtain the negative sample text description by slightly perturbing the positive sample text description based on geometric element attributes, letters and letter sequences, geometric element values, and geometric element relationships.

[0086] Based on any of the above embodiments, the image coding model training module is used to: Inputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample text descriptions into an original image coding model to obtain a predicted image vector output by the original image coding model; Determine a loss function of the predicted image vector and the corresponding positive sample text description and a preset number of negative sample texts, and adjust the parameters of the original image coding model based on the loss function; Repeat the process of outputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample texts to the original image coding model until the original image coding model converges to obtain the image coding model.

[0087] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a mathematical problem solving method based on a multimodal large model, the method comprising: determining a mathematical element image of a mathematical problem; inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on sample image vectors and sample text coding vectors; determining the stem text of the mathematical problem, inputting the stem text and the image conversion coding vector into a large language model to obtain a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vectors and sample solution processes.

[0088] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mathematical problem solving method based on a multimodal large model provided by the above-mentioned methods, and the method includes: determining a mathematical element image of the mathematical problem; inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on sample image vectors and sample text coding vectors; determining the stem text of the mathematical problem, inputting the stem text and the image conversion coding vector into a large language model, and obtaining a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vectors and sample solution processes.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the mathematical problem solving method based on a multimodal large model provided by the above-mentioned methods, the method comprising: determining a mathematical element image of the mathematical problem; inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on sample image vectors and sample text coding vectors; determining the stem text of the mathematical problem, inputting the stem text and the image conversion coding vector into a large language model to obtain a predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vectors and sample solution processes.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for solving mathematical problems based on a multimodal large model, characterized in that: include: Determine the mathematical element image of the mathematical problem; Inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; Inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is trained based on a sample image vector and a sample text coding vector; Determine the stem text of the math problem, input the stem text and the image conversion coding vector into a large language model, and obtain a predicted solution process output by the large language model; The large language model is trained based on sample question text, sample image conversion coding vectors and sample answering processes.

2. The method for solving mathematical problems based on a multimodal large model according to claim 1, characterized in that: The image coding model is trained based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, and includes: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; Inputting the positive sample text descriptions into a pre-trained text encoding model respectively to obtain a text embedding vector corresponding to the positive sample text description output by the text encoding model; Determine the text similarity between each of the text embedding vectors, and select a negative sample text description of each of the sample geometric images based on the text similarity.

3. The method for solving mathematical problems based on a multimodal large model according to claim 2, characterized in that: Determining text similarity between the text embedding vectors, and selecting a negative sample text description of each of the sample geometric images based on the text similarity, comprising: Calculate the similarity between the two text embedding vectors; Determine a target text embedding vector from the text embedding vectors, and determine the similarity between the target text embedding vector and the remaining text embedding vectors; Selecting the positive sample text descriptions corresponding to the K text embedding vectors with the highest similarity as the negative sample text descriptions of the target sample geometric image corresponding to the target text embedding vector; The process of determining the target text embedding vector from the text embedding vector is repeated until a negative sample text description of each of the sample geometric images is selected.

4. The method for solving mathematical problems based on a multimodal large model according to claim 1 or 2, characterized in that: The image coding model is obtained by training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, and includes: Determine a plurality of groups of sample geometric images and a positive sample text description uniquely corresponding to each group of the sample geometric images; The positive sample text description and the modification prompt word are respectively input into a general language model to obtain L negative sample text descriptions output by the general language model; wherein the negative sample text description is obtained by slightly perturbing and modifying the positive sample text description.

5. The method for solving mathematical problems based on a multimodal large model according to claim 1, characterized in that: The image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions, and includes: Inputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample text descriptions into an original image coding model to obtain a predicted image vector output by the original image coding model; Determine a loss function of the predicted image vector and the corresponding positive sample text description and a preset number of negative sample texts, and adjust the parameters of the original image coding model based on the loss function; Repeat the process of outputting the sample mathematical element image and its corresponding positive sample text description and a preset number of negative sample texts to the original image coding model until the original image coding model converges to obtain the image coding model.

6. The method for solving mathematical problems based on a multimodal large model according to claim 4, characterized in that: The negative sample text description is obtained by slightly perturbing the positive sample text description, and includes: The negative sample text description is obtained by slightly perturbing and modifying the positive sample text description based on geometric element attributes, letters and letter sequences, geometric element values, and geometric element relationships.

7. A mathematical problem solving device based on a multimodal large model, characterized in that: include: A mathematical problem determination module, used for determining a mathematical element image of a mathematical problem; An image vector acquisition module, used for inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by comparative learning and training based on sample mathematical element images and their corresponding positive sample text descriptions and negative sample text descriptions; A vector conversion module, used for inputting the image vector into an adaptive module to obtain an image conversion coding vector output by the adaptive module; the adaptive module is obtained by training based on sample image vectors and sample text coding vectors; The solution process prediction module is used to determine the stem text of the mathematical problem, input the stem text and the image conversion coding vector into a large language model, and obtain the predicted solution process output by the large language model; the large language model is trained based on sample stem text, sample image conversion coding vector and sample solution process.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the method for solving mathematical problems based on a multimodal large model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for solving mathematical problems based on a multimodal large model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for solving mathematical problems based on a multimodal large model as described in any one of claims 1 to 6 is implemented.

Images