A method and system for geometric problem reasoning using aligned multi-modal features

Abstract

The present application relates to the field of natural language processing and machine learning, and discloses a method and system for geometric problem reasoning using aligned multi-modal features, comprising the following specific steps: initializing a training data set of geometric problems and a geometric problem reasoning model based on a pre-trained Chinese CLIP model; inputting multi-modal data into the geometric problem reasoning model for training, introducing a dynamic vocabulary in the training process to obtain a trained geometric problem reasoning model; obtaining a geometric problem to be reasoned and inputting it into the trained geometric problem reasoning model to obtain a geometric problem reasoning result. The present application solves the problem that the prior art cannot fully utilize cross-modal semantic representation to solve geometric problems, and has the characteristics of being able to produce more accurate reasoning results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing and machine learning, and more particularly, to a method and system for geometric problem reasoning using aligned multi-modal features. BACKGROUND

[0002] It is challenging and has attracted more and more attention to solve geometric problems using deep models to realize logical reasoning capabilities, which requires our models to have the ability to analyze multi-modal information, the ability to fully utilize mathematical knowledge, and comprehensive reasoning capabilities. Geometric problems are usually divided into two categories: calculation problems and proof problems. Previous studies have attempted to use top-down tree structures or specific symbolic procedures to handle geometric problems, but these methods rely too much on hand-crafted rules. Recent work has proposed a geometric problem benchmark, which uses knowledge point reasoning auxiliary models for prediction. However, these methods do not fully utilize the benefits of multi-modal feature alignment, which limits further improvement of model performance. The latest multi-modal learning framework Chinese CLIP uses large-scale natural language and image data for pre-training to achieve image and text alignment and consistency.

[0003] For geometric problems, both images and texts are complementary. Images can more clearly display ambiguous descriptions in problem texts and provide more details not in problem texts. Problem texts are precise definitions of images and can provide specific numerical values not described in images. Based on these frontier works and the complementary relationship between images and texts, it is crucial to reasonably align multi-modal features and fully utilize cross-modal semantic representations for solving geometric problems.

[0004] The prior art has a middle school geometry problem automatic solving method based on contrast learning, which is as follows: collect a number of middle school geometry questions and answers to obtain the required middle school geometry dataset; divide into no-image geometry question dataset and image geometry question dataset; input the no-image geometry question into the geometry image generator to obtain a high-accuracy geometry figure; then input into the image solver to obtain the final multi-modal feature vector; the program decoder in the image solver obtains a high-accuracy problem-solving answer; finally, the geometry image generator and the image solver are combined together to test, forming a unified large model that solves geometry problems with self-drawing and self-provided figures.

[0005] However, the prior art cannot fully utilize cross-modal semantic representations to solve geometric problems, so how to invent a method and system for geometric problem reasoning using aligned multi-modal features is a technical problem that needs to be solved in this technical field. SUMMARY

[0006] The present application provides a method and system for geometric problem reasoning using aligned multi-modal features to solve the problem that the existing technology cannot fully utilize cross-modal semantic representation to solve geometric problems.

[0007] To achieve the above-mentioned object of the present application, the technical solution adopted is as follows:

[0008] A method for geometric problem reasoning using aligned multi-modal features, comprising the following specific steps:

[0009] S1, initializing a training data set of geometric problems and a geometric problem reasoning model based on a pre-trained Chinese CLIP model;

[0010] S2, inputting multi-modal data into the geometric problem reasoning model for training, and the training process is specifically:

[0011] S201, preprocessing the training data set to obtain multi-modal data;

[0012] S202, feature extraction on the multi-modal data;

[0013] S203, unifying the dimensions of image and text features through contrastive learning;

[0014] S204, aligning and fusing the image and text features after unifying the dimensions;

[0015] S205, introducing a dynamic vocabulary to decode the fused features to obtain an annotation program;

[0016] S206, obtaining a reasoning result according to the generated annotation program;

[0017] S207, iteratively optimizing the model according to the reasoning result to obtain a trained geometric problem reasoning model;

[0018] S3, obtaining a geometric problem to be reasoned and inputting it into the trained geometric problem reasoning model to obtain a geometric problem reasoning result.

[0019] Preferably, the training data set of geometric problems specifically adopts a GeoQA data set; the specific steps for preprocessing the training data set to obtain multi-modal data are:

[0020] Filtering error and irrelevant text, image;

[0021] Filtering error and irrelevant annotation program;

[0022] Obtaining a multi-modal data set Clean-GeoQA.

[0023] Further, the geometric problem reasoning model comprises an encoder based on a Chinese CLIP framework, a cross-modal fusion block, and a decoder; wherein the encoder comprises a text encoder based on a BERT model and an image encoder based on a VIT-B / 16 model; wherein the image encoder removes the LayerNorm of the second-to-last layer and the feature mapping of the last layer of the VIT-B / 16 model.

[0024] Further, the multi-modal data is subjected to feature extraction, and the specific steps are as follows:

[0025] Special symbols and new words in the multi-modal data are added to the word table of the BERT; for the question text in the multi-modal data The question text is input into the text encoder to obtain the [CLS] tokenH T ∈R BS×HS , and the dimension is expanded to where BS is the batch size, and HS is the hidden size.

[0026] The image of the geometric question of the Clean-GeoQA is input into the VIT-B / 16 model to obtain the [CLS] tokenH D ∈R BS×HS , and the dimension is expanded to where BS is the batch size, and HS is the hidden size.

[0027] Further, the dimensions of the image and text features are unified through contrast learning, and the specific steps are as follows:

[0028] The cosine similarity of the image-text pair is calculated:

[0029]

[0030] where τ is a learnable temperature parameter, Norm(·) is a normalization operation, and Squar(·) is a square operation.

[0031] The square sum of the cosine similarity of each corresponding image-text pair is calculated, and finally the square sum is scaled by the τ parameter to obtain the final contrastive loss for backpropagation optimization.

[0032] The dimensions of the image and text features are unified through contrast learning.

[0033] Further, the image and text features with unified dimensions are aligned and fused, and the specific steps are as follows:

[0034] The text feature H Tand image features H D Input cross-modal fusion block;

[0035] Text features H T After passing through a module consisting of a stacked 6-layer self-attention network and a feedforward network, each layer performs residual connections to perform context alignment and capture semantic relationships in the text, resulting in aligned text features H′. T :

[0036] H′ T =Norm(H T +FFN(Norm(H T +Self-Attention(H T ))))

[0037] FFN(·)=Linear(ReLU(linear(·)))

[0038] Norm(·) is the normalization operation;

[0039] Text features H′ T This information will be used as guidance and input into another module consisting of a stacked 6-layer self-attention network, a cross-attention network, and a feedforward network to process image features H. D Alignment is performed to obtain the aligned image features H′. D It contains a wealth of information, including question text and images:

[0040]

[0041] Use a Buffer = {m1, m2, ..., m DS}∈R HS×DS Store the aligned text features H′ T , where m u =(H′) T ) i HS is the hidden size, and DS is the size of the training set; after the Buffer first stores the maximum length, multimodal feature alignment is performed with a certain probability p.

[0042] Image features H′ D The image features are weighted and combined with all text features in the buffer to obtain the fused feature H in the text feature space. proj :

[0043]

[0044] H′ D and Hproj A residual connection is made, and a learnable vector γ ∈ R HS is used to balance the mapped output H proj and the input feature H′ D The vector is initialized with 0 to ensure that the input feature H′ D will not be greatly modified by the addition of the mapped feature H proj :

[0045] H′ D = (1-γ) × H′ D + γ × H proj .

[0046] Further, a dynamic vocabulary is introduced to decode the fusion feature to obtain the annotated program, and the specific steps are as follows:

[0047] The fusion feature is decoded using a vocabulary including operators OP, constants Const, problem variables N and process variables V; in the decoding, several predictions are made, and in each prediction, the vocabulary used for the next prediction output is dynamically changed according to the current prediction output, until the final annotated program is obtained.

[0048] Further, in the decoding, the following restriction rules are followed:

[0049] In the first prediction, the prediction probability of non-operators is reduced to a set extremely low value ε to ensure that the output is an operator;

[0050] In the second prediction, after determining that the first-step prediction output is an operator, the second-step prediction output is limited to a problem variable N;

[0051] In each subsequent prediction:

[0052] If the last prediction output is @PAD@ or @end@, the prediction has ended, and the subsequent prediction output will be limited to @PAD@;

[0053] If the last prediction output is an operator, the current step can only output a problem variable or a process variable;

[0054] If the last prediction output is a problem variable, a process variable or a constant, only the current output is required to ensure that it cannot repeat the output of the last step.

[0055] Further, when the dynamic vocabulary is used to decode the fusion feature, a probability θ is also set, and the dynamic vocabulary adjustment is applied with a probability of θ in each prediction.

[0056] A system for geometric problem reasoning using aligned multi-modal features, comprising a data acquisition module, a model initialization module, a model training module, and a problem reasoning module.

[0057] The data acquisition module is used for initializing a training data set of a geometric problem and acquiring a geometric problem to be inferred;

[0058] The model initialization module is used for initializing a geometric problem inference model based on a pre-trained Chinese CLIP model;

[0059] The model training module is used for inputting multi-modal data into the geometric problem inference model for training, and the training process is specifically as follows:

[0060] The training data set is preprocessed to obtain multi-modal data;

[0061] The multi-modal data is subjected to feature extraction;

[0062] The dimensions of the image and text features are unified through contrast learning;

[0063] The image and text features after dimension unification are aligned and fused;

[0064] A dynamic vocabulary is introduced to decode the fused features to obtain an annotation program;

[0065] The inference result is obtained according to the generated annotation program;

[0066] The model is iteratively optimized according to the inference result to obtain a trained geometric problem inference model;

[0067] The problem inference module is used for acquiring a geometric problem to be inferred and inputting the geometric problem into the trained geometric problem inference model to obtain a geometric problem inference result.

[0068] The beneficial effects of the present application are as follows:

[0069] A method for inferring geometric problems by using aligned multi-modal features is proposed, which is specially used for solving geometric calculation problems. The present application uses the aligned cross-modal semantic representation to solve geometric problems. Through technical means and according to the characteristics of geometric problems, the solving process is constrained, which can improve the accuracy of geometric problem solving, improve the multi-step reasoning ability of deep models, and reduce the distance between different modal features in the semantic space. At the same time, the method proposed in the present application can also adapt to the task target domain, quickly promote the alignment of features of one modality to features of another modality. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a flowchart of the method for inferring geometric problems by using aligned multi-modal features of the present application.

[0071] Figure 2 is a training flowchart of the method for inferring geometric problems by using aligned multi-modal features of the present application.

[0072] Figure 3 is a schematic diagram of an encoder for geometric problem reasoning using aligned multi-modal features.

[0073] Figure 4 is a schematic diagram of an overall framework for geometric problem reasoning using aligned multi-modal features.

[0074] Figure 5 is a schematic diagram of a multi-modal feature alignment and fusion branch for geometric problem reasoning using aligned multi-modal features. DETAILED DESCRIPTION

[0075] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Embodiment 1

[0077] As shown in Figure 1 , Figure 2 , a method for geometric problem reasoning using aligned multi-modal features includes the following specific steps:

[0078] S1, initializing a training data set of geometric problems and a geometric problem reasoning model based on a pre-trained Chinese CLIP model;

[0079] S2, inputting multi-modal data into the geometric problem reasoning model for training, and the training process is specifically as follows:

[0080] S201, preprocessing the training data set to obtain multi-modal data;

[0081] S202, feature extraction is performed on the multi-modal data;

[0082] S203, the dimensions of image and text features are unified through contrastive learning;

[0083] S204, aligning and fusing the image and text features after unifying the dimensions;

[0084] S205, introducing a dynamic vocabulary to decode the fused features to obtain an annotation program;

[0085] S206, obtaining a reasoning result according to the generated annotation program;

[0086] S207, iteratively optimizing the model according to the reasoning result to obtain a trained geometric problem reasoning model;

[0087] S3, obtaining a geometric problem to be reasoned and inputting it into the trained geometric problem reasoning model to obtain a geometric problem reasoning result.

[0088] Embodiment 2

[0089] More specifically, in one specific embodiment, the training data set of the geometric problem specifically adopts the GeoQA data set.

[0090] In this embodiment, the GeoQA data set contains 4998 data pairs of image data and question text of geometric problems, each calculation question has an annotation program corresponding to its solution process, the reasoning result is obtained by executing the corresponding annotation program, and the reasoning performance of the model is verified. However, since there are many noise data in the GeoQA data set, it is necessary to filter and filter the noise data, and the filtering conditions are as follows:

[0091] 1) The picture data is a black picture, which cannot bring actual performance improvement to the image encoder;

[0092] 2) The question text data is wrong, there are extra symbols or information irrelevant to the question;

[0093] 3) The annotation program is wrong, there are irrelevant or wrong information. After cleaning, the data set is called Clean-GeoQA data set, which contains 4973 geometric problems, each question has its corresponding annotation program.

[0094] In one specific embodiment, as shown in Figure 3 , Figure 4 The geometric problem reasoning model includes an encoder based on the Chinese CLIP framework, a cross-modal fusion block, and a decoder; wherein the encoder includes a text encoder based on the BERT model and an image encoder based on the VIT-B / 16 model; wherein the image encoder removes the LayerNorm of the second-to-last layer and the feature mapping of the last layer of the VIT-B / 16 model.

[0095] In one specific embodiment, feature extraction is performed on the multi-modal data, and the specific steps are as follows:

[0096] Add special symbols and new words in the multi-modal data to the BERT vocabulary; for the question text in the multi-modal data Input the question text into the text encoder to obtain the [CLS] tokenH T ∈R BS×HS , and expand its dimension to Where BS is the batch size, and HS is the hidden size;

[0097] Input the geometric question image of Clean-GeoQA into the VIT-B / 16 model to obtain the [CLS] tokenH D ∈RBS×HS and extend its dimension to where BS is the batch size and HS is the hidden size.

[0098] In one embodiment, the dimensions of the image and text features are unified by contrastive learning, and the specific steps are as follows:

[0099] Calculate the cosine similarity of the image-text pair:

[0100]

[0101] where τ is a learnable temperature parameter, Norm(·) is a normalization operation, and Squar(·) is a square operation.

[0102] Square the cosine similarity of each corresponding image-text pair, and finally scale the calculated sum of squares by the τ parameter to obtain the final contrastive loss for backpropagation optimization.

[0103] The dimensions of the image and text features are unified by contrastive learning.

[0104] In one embodiment, in order to fully utilize the cross-modal semantic representation to solve geometric problems, a combination of self-attention and cross-attention is used to fuse the multi-modal data; the unified dimensional image and text features are aligned and fused, and the specific steps are as follows:

[0105] The text feature H T and the image feature H D are input into the cross-modal fusion block.

[0106] The text feature H T passes through a module stacked by 6 layers of self-attention network and feedforward network, and each layer is connected by residual connection to align the context and capture the semantic relationship in the text, obtaining the aligned text feature H' T :

[0107] H' T = Norm(H T + FFN(Norm(H T + Self-Attention(H T )))

[0108] FFN(·) = Linear(ReLU(Linear(·)))

[0109] where Norm(·) is a normalization operation.

[0110] Text features H′ T This information will be used as guidance and input into another module consisting of a stacked 6-layer self-attention network, a cross-attention network, and a feedforward network to process image features H. D Alignment is performed to obtain the aligned image features H′. D It contains a wealth of information, including question text and images:

[0111]

[0112] Use a Buffer = {m1, m2, ..., m DS}∈R HS×DS Store the aligned text features H′ T , where m i =(H′) T ) i HS is the hidden size, DS is the size of the training set, which is set to 3478; after the buffer is first stored to its maximum length, multimodal feature alignment is performed with a certain probability p, which is set to 0.1;

[0113] Image features H′ D The image features are weighted and combined with all text features in the buffer to obtain the fused feature H in the text feature space. proj :

[0114]

[0115] H′ D and H proj Perform residual connections and use learnable vectors γ∈R HS To balance the output H after mapping proj and input features H′ D This vector is initialized with 0 to ensure that the input feature H′ is initialized. D It will not be due to the mapping feature H proj The addition of [something] led to significant modifications:

[0116] H′ D = (1-γ)×H′ D +γ×H proj .

[0117] In one specific embodiment, a dynamic vocabulary is introduced to decode the fused features to obtain the annotation program. The specific steps are as follows:

[0118] The fusion feature is decoded using a new 45-length vocabulary including an operator OP, a constant Const, a problem variable N and a process variable V; in the decoding, several predictions are performed, in each prediction, the vocabulary used for the next prediction output is dynamically changed according to the current prediction output until the final annotation program is obtained.

[0119] In one specific embodiment, in the decoding, the following restriction rules are followed:

[0120] In the first prediction, the prediction probability of the non-operator is reduced to a set extremely low value epsilon to ensure that the output is an operator;

[0121] In the second prediction, after determining that the first step prediction output is an operator, the second step restricts the prediction output to be a problem variable N;

[0122] In each subsequent prediction:

[0123] If the last prediction output is @PAD@ or @end@, the prediction has ended, and the subsequent prediction output will be limited to @PAD@;

[0124] If the last prediction output is an operator, the current step can only output a problem variable or a process variable;

[0125] If the last prediction output is a problem variable, a process variable or a constant, only need to ensure that the current output cannot repeat the output of the last step.

[0126] In one specific embodiment, when the dynamic vocabulary is used to decode the fusion feature, a probability theta is also set, and the dynamic vocabulary adjustment is applied with a probability of theta in each prediction.

[0127] Compared with the prior art, the present application has the following advantages:

[0128] 1. Compared with the existing geometric problem solving model, the problem of insufficient multi-modal fusion is considered, and therefore the contrast learning, multi-modal feature fusion and multi-modal alignment technology are added. This helps the model to more comprehensively understand and reason the geometric problem, so that the model can obtain more rich and comprehensive information from different data sources, and is expected to improve the accuracy and robustness of problem solving.

[0129] 2. Compared with the GeoQA data set, the proposed Clean-GeoQA data set has less noise data, and compared with directly training using the original data, the cleaned data set helps to reduce the possibility of learning incorrect information by the model, and improves the generalization ability and reasoning effect of the model.

[0130] 3、Compared with the operation of the current geometric problem solving model at the decoding end, the dynamic vocabulary method is used to limit the output of the decoder, and the vocabulary is dynamically adjusted according to the current prediction. This method introduces a more flexible and intelligent decoding method in geometric problem solving, which can better adapt to the requirements of specific tasks and improve the accuracy and efficiency of the model in solving geometric problems.

[0131] Embodiment 3

[0132] A system for geometric problem reasoning using aligned multi-modal features, comprising a data acquisition module, a model initialization module, a model training module, and a problem reasoning module;

[0133] The data acquisition module is used to initialize the training data set of geometric problems and acquire geometric problems to be reasoned;

[0134] The model initialization module is used to initialize the geometric problem reasoning model based on the pre-trained Chinese CLIP model;

[0135] The model training module is used to input multi-modal data into the geometric problem reasoning model for training, and the training process is specifically:

[0136] Preprocess the training data set to obtain multi-modal data;

[0137] Feature extraction is performed on the multi-modal data;

[0138] The dimensions of the image and text features are unified through contrastive learning;

[0139] Align and fuse the image and text features after unifying the dimensions;

[0140] Introduce a dynamic vocabulary to decode the fused features to obtain an annotated program;

[0141] The reasoning result is obtained according to the generated annotated program;

[0142] Iteratively optimize the model according to the reasoning result to obtain a trained geometric problem reasoning model;

[0143] The problem reasoning module is used to acquire geometric problems to be reasoned and input them into the trained geometric problem reasoning model to obtain geometric problem reasoning results.

[0144] In this embodiment, the multi-modal data is Clean-GeoQA, which contains 4973 geometry questions. The data set is composed of three different topics, namely angle problems, length problems and other problems. Among them, there are 2717 angle problems, 1864 length problems and 392 other problems. Each question has its corresponding annotation program, and the reasoning result is obtained by executing the corresponding annotation program, and the reasoning performance of the model is verified.

[0145] In this embodiment, 70% of the multi-modal data is randomly selected as the training data set, 15% as the validation data set, and 15% as the test data set.

[0146] In this embodiment, the geometry problem reasoning model based on the pre-trained Chinese CLIP model is initialized, specifically: using PyTorch, using Adam optimizer for optimization, β1=0.9, β2=0.999, and the learning rate is set to 1e-3. The learning rate scheduler uses ReduceLROnPlateau, where factor is 0.8, patience is 3, mode is set to "max", and when the accuracy does not improve within 3 epochs, the learning rate is reduced according to the factor. The beam size is set to 10. For the pre-trained Chinese CLIP model, in order to make it more suitable for the data set, we set the learning rate of the transformer part in VIT-16 / B to 2e-3, and the learning rate of the remaining network modules of the Chinese CLIP model to 2e-5 to fine-tune the data set. Finally, we set the batch size to 32, and train the model end-to-end for 100 epochs.

[0147] In this embodiment, the training data set is preprocessed, specifically: first, fill the picture with a white background and scale it to 224x224 size, then convert it from single channel to three channels, and finally normalize it. For text data, we encode the text into a vector through the vocabulary of Bert for subsequent processing.

[0148] In this embodiment, when extracting features from multi-modal data, the extracted image and text features are unified in dimension to meet the subsequent input of the model.

[0149] In this embodiment, the image and text features after dimension unification are aligned and fused, and the extracted features are further aligned, which can alleviate the misalignment of modal features caused by the large gap between the data set used by the pre-trained Chinese CLIP model and the geometry problem data set; meanwhile, the features after multi-modal fusion can utilize cross-modal semantic representation, thereby better solving geometry problems.

[0150] In this embodiment, the dynamic vocabulary is introduced to decode the fused features, and when the annotated program is obtained, the vocabulary used for the current prediction output is specified according to the prediction output of the previous step, so as to better constrain the prediction process and generate an annotated program with higher quality.

[0151] In this embodiment, the unified image and text features are aligned and fused as shown in Figure 5 , which is specifically divided into three parts.

[0152] The cosine similarity of the image-text pair is calculated, and the square sum of the cosine similarity of each corresponding image-text pair is calculated, and this is used as a loss value for back propagation optimization.

[0153] The text features are passed through a module stacked by 6 layers of self-attention network and feedforward network, and each layer is connected in residual, and finally the aligned text features are obtained. Subsequently, the text features will be used as guiding information and input into another module stacked by 6 layers of self-attention network, cross-attention network and feedforward network, to align the picture features and obtain the aligned picture features.

[0154] The text features in 2) are stored in the buffer, the image features and all the text features in the buffer are combined to obtain the representation of the image features in the text feature space, and then residual connection is performed, so as to further narrow the semantic space of the two modalities.

[0155] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for geometric problem reasoning using aligned multimodal features, characterized in that: The specific steps include the following: S1. Initialize the training dataset for the geometric problem and construct a geometric problem reasoning model based on the pre-trained Chinese CLIP model; the geometric problem reasoning model includes an encoder, a cross-modal fusion block, and a decoder based on the Chinese CLIP framework; wherein the encoder includes a text encoder based on the BERT model and an image encoder based on the VIT-B / 16 model; wherein the image encoder removes the second-to-last LayerNorm and the feature mapping of the last layer of the VIT-B / 16 model; S2. Input the multimodal data into the geometric problem reasoning model for training. The specific training process is as follows: S201. Preprocess the training dataset to obtain multimodal data; S202. Perform feature extraction on multimodal data; S203. Unify the dimensions of image and text features through comparative learning; S204. Align and fuse the unified dimensional image and text features. The specific steps are as follows: Text features and image features Input cross-modal fusion block; Text features The text is processed through a module consisting of six layers of self-attention and feedforward networks. Each layer has residual connections to perform context alignment and capture semantic relationships in the text, resulting in aligned text features. : Norm(·) is the normalization operation; Text features As guiding information, it is input into another module consisting of a stacked 6-layer self-attention network, a cross-attention network, and a feedforward network to process image features. Alignment is performed to obtain the aligned image features. It contains a wealth of information, including question text and images: Use one Store aligned text features ,in HS is the hidden size, and DS is the size of the training set. After the Buffer first stores the maximum length, modal feature alignment is performed with a certain probability p. Image features The image features are weighted and combined with all text features in the buffer to obtain the fused features in the text feature space. : Will and Perform residual connections and use learnable vectors. To balance the output after mapping and input features This vector is initialized with 0 to ensure the input features are safe. Not because The addition of [something] led to significant modifications: S205. Introduce a dynamic vocabulary to decode the fused features and obtain the annotation program; S206. Obtain the reasoning result based on the generated annotation program; S207. Based on the reasoning results, iteratively optimize the model to obtain a trained geometric problem reasoning model; S3. Obtain the geometric problem to be reasoned and input it into the trained geometric problem reasoning model to obtain the geometric problem reasoning result.

2. The method for geometric problem reasoning using aligned multimodal features according to claim 1, characterized in that: The training dataset for the aforementioned geometric problem specifically uses the GeoQA dataset; the specific steps for preprocessing the training dataset to obtain multimodal data are as follows: Filter out errors and irrelevant text and images; A program to filter out errors and irrelevant comments; The multimodal dataset Clean-GeoQA was obtained.

3. The method for geometric problem reasoning using aligned multimodal features according to claim 2, characterized in that: The specific steps for feature extraction from multimodal data are as follows: Special symbols and new words from multimodal data are added to BERT's vocabulary; for question text in multimodal data... The question text is input into the text encoder to obtain the [CLS] token of the text features. And expand its dimensions to Where BS is the batch size; The geometry problem images from Clean-GeoQA are input into the VIT-B / 16 model to obtain the [CLS] token of the image features. And expand its dimensions to .

4. The method for geometric problem reasoning using aligned multimodal features according to claim 3, characterized in that: By comparing and learning the dimensions of unified image and text features, the specific steps are as follows: Calculate the cosine similarity of image-text pairs: in, It is a learnable temperature parameter, Norm(·) is the normalization operation, and Squar(·) is the square operation; The cosine similarity of each corresponding image-text pair is summed by squares, and finally... The parameters are scaled on the calculated sum of squares to obtain the final contrastive loss, which is used for backpropagation optimization. By comparing and learning, we can unify the dimensions of image and text features.

5. The method for geometric problem reasoning using aligned multimodal features according to claim 4, characterized in that: A dynamic vocabulary is introduced to decode the fused features, resulting in an annotation program. The specific steps are as follows: The fused features are decoded using a vocabulary that includes operators OP, constants Const, problem variables, and process variables V. During decoding, several predictions are made. In each prediction, the vocabulary used for the next prediction is dynamically changed based on the current prediction output, until the final annotation program is obtained.

6. The method for geometric problem reasoning using aligned multimodal features according to claim 5, characterized in that: During decoding, the following restrictions must be followed: In the first prediction, the prediction probability of non-operators is reduced to a set extremely low value ε to ensure that the output is an operator; In the second prediction, after determining that the output of the first prediction is the operator, the second step restricts the output of the prediction to the problem variable; In every subsequent prediction: If the previous prediction output was @PAD@ or @end@, then the prediction has ended, and subsequent prediction outputs will be restricted to @PAD@. If the previous prediction output was an operator, then the current step can only output the problem variable or the process variable; If the previous prediction output was a problem variable, process variable, or constant, you only need to ensure that the current output does not repeat the output of the previous step.

7. The method for geometric problem reasoning using aligned multimodal features according to claim 6, characterized in that: When decoding fused features, the dynamic vocabulary also sets a probability θ, and each prediction is adjusted by applying the dynamic vocabulary with a probability of θ.

8. A system for geometric problem reasoning using aligned multimodal features, characterized in that: The method for implementing the method as described in any one of claims 1 to 7 includes a data acquisition module, a model initialization module, a model training module, and a problem reasoning module; The data acquisition module is used to initialize the training dataset for the geometric problem and acquire the geometric problem to be reasoned. The model initialization module is used to initialize the geometric problem reasoning model based on the pre-trained Chinese CLIP model; The model training module is used to input multimodal data into the geometric problem reasoning model for training. The specific training process is as follows: Preprocess the training dataset to obtain multimodal data; Feature extraction from multimodal data; By comparing and contrasting, we can learn the dimensions of unified image and text features; Align and fuse the image and text features after unifying the dimensions; A dynamic vocabulary is introduced to decode the fused features, resulting in an annotation program; The reasoning result is obtained based on the generated annotation program; The model is iteratively optimized based on the reasoning results to obtain a well-trained geometric problem reasoning model. The problem reasoning module is used to obtain the geometric problem to be reasoned and input it into the trained geometric problem reasoning model to obtain the geometric problem reasoning result.

Images