Automatic problem-solving method for plane geometry based on spatial perception of geometric primitives

Abstract

The invention discloses a method for automatically solving plane geometry problems based on spatial perception of geometric primitives. S1 uses a parser to parse geometric diagrams and geometric problems to be solved into text clauses according to preset rules, and inputs them into a Transformer Encoder for encoding; S2 divides the geometric diagram image into 64 diagram patch features through a convolution neural network and learns the internal features of the visual modality through a self-attention mechanism; S3 inputs these modal features into a multimodal dual-branch spatial perception pre-trained language module to learn cross-modal global features and geometric primitive spatial information, and inputs them into a bidirectional GRU encoder to perform fusion encoding; S4 sends the text code in the hybrid code into a point primitive spatial attention module to enhance the model's spatial perception ability of the geometric primitives; S5 decodes the enhanced hybrid code by a self-limiting GRU decoder, and generates a sequential solution program.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to a method for automatically solving plane geometry problems based on geometric primitive space perception. Background Art

[0002] Applying artificial intelligence technology to solve geometric problems is a long-standing and challenging task in the field of computer vision and natural language processing, not only because of its high academic value in academia, but also because of its great significance in the field of education. A geometry problem consists of a problem text and its corresponding geometric image. The text problem description presents the conditions and problem-solving goals of the geometry problem, and the geometric image provides semantic and structural information that cannot be described by the problem text. Together, they provide rich information for intelligent geometry problem-solving programs. Due to the characteristics of its data set, geometry problems can be regarded as visual-language multimodal reasoning tasks, which require joint reasoning of images and texts. Therefore, being able to better understand each modality and the integration and understanding of multiple modalities is one of the keys to using neural solvers to solve plane geometry problems. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method for automatic problem-solving of plane geometry based on spatial perception of geometric primitives, which enhances the model's perception of the spatial distribution of geometric primitives through a multimodal dual-channel visual text point matching pre-trained language module and a point primitive spatial perception attention module, thereby better solving plane geometry problems.

[0004] To achieve the above object, the present invention adopts the following technical solution: a method for automatically solving plane geometry problems based on spatial perception of geometric primitives, comprising the following steps:

[0005] Step S1: Using a parser to parse the geometry diagram and geometry problem to be solved into text clauses based on the geometric primitives such as points, lines, and circles mentioned in the problem, the text clauses include structural clauses, semantic clauses, and question texts, and input them into the Transformer Encoder to learn the internal features of the text modality;

[0006] Step S2: The geometric diagram image is initially feature extracted through a convolutional neural network and divided into 64 diagram patches, and the internal features of the visual modality are learned through a self-attention mechanism;

[0007] Step S3: merge the text features and the image features into a multimodal dual-branch spatial perception pre-trained language module to obtain cross-modal global features and geometric primitive space information, and input them into a bidirectional GRU encoder to perform fusion encoding;

[0008] Step S4: taking out the text encoding part in the hybrid encoding and inputting it into the point primitive spatial perception attention module to enhance the spatial perception ability of the geometric solver model to the geometric primitives;

[0009] Step S5: concatenate the enhanced text code with the image code in the hybrid code and decode it by the self-limited GRU decoder, and generate a sequential solver;

[0010] In step S3, a multimodal dual-branch spatial perception pre-training language module is used to obtain the global features of vision-language and the spatial relationship of geometric primitives in geometric diagrams, including: a structural-semantic pre-training strategy for learning the global features of vision-language; a dual-branch vision-text point alignment strategy, which uses intra-modal and inter-modal vision-text point matching to achieve alignment of point primitive symbols appearing in the text with the corresponding image patches in the geometric diagram; specifically, the visual encoding of the geometric diagram is converted into With text encoding Splice and input Transformer Encoder to learn cross-modal information, and output visual features that integrate text features and text features fused with visual features , and then the contrastive learning-based approach is used to match the visual points and text points within and between modalities respectively. The loss based on contrastive learning is expressed as:

[0011]

[0012] in represents the intra-modal contrast loss, represents the contrast loss between modalities, and Represent the features of text points within and between modalities, and Respectively represent the characteristics of visual points within the modal and between modalities, and Represent the features of each word in the intra-modal and inter-modal text features, respectively, j represents the index of the word in the text clause, represents the linear projection function, represents the temperature coefficient, and T represents the transposition operation. The visual points and text points are aligned through the dual-branch visual-text point alignment strategy to obtain the spatial distribution relationship between geometric primitives. The loss is expressed as:

[0013] ;

[0014] The total training loss of the multimodal dual-branch spatial perception pre-trained language module is expressed as:

[0015] ,

[0016] Where L SSP represents the training loss of the structure-semantic pre-training strategy, and represents the hyperparameter, and then the output of the multimodal dual-branch spatial perception pre-trained language module is input into the bidirectional GRU encoder to perform fusion encoding to obtain the context encoding H.

[0017] In a preferred embodiment: in step S1, the geometric diagram and the geometric problem to be solved are parsed into text clauses pt using preset rules, including: structural clauses for describing the connection relationship between geometric primitives; semantic clauses for describing the semantic relationship between non-geometric primitives and geometric primitives; problem text for describing the solution target of the plane geometry problem; encoding the text clauses to obtain text encoding , where L represents the length of the text clause and the text encoding E PT It includes the text clause encoding TokenEmb, which is used to describe geometric images and text, the one-dimensional position encoding PosEmb, which is used to describe the order of each word in the text clause, the fragment encoding SectEmb, which is used to describe the source of each word in the text clause, and the category encoding ClassEmb, which is used to describe the category of each word in the text sentence; the text clause encoding is expressed as:

[0018] ;

[0019] in, Represents the words in the text clause parsed from the geometry diagram and geometry problem to be solved, j represents the index of the word in the text clause, the text clause encoding is input into the Transformer Encoder to extract the text features, and then a linear mapping is used to project the text features to obtain a new text feature

[0020] ,in Represents the internal features of the text modality obtained after each text word passes through the Transformer Encoder.

[0021] In a preferred embodiment: In step S2, the convolution neural network uses ResNet10 to resize the image into 256*256 and input it into ResNet10 for convolution operation, dividing it into 64 image patches of equal size. , where i represents the sequential index of the image patch, Represents the features of each patch in the image, N represents the number of image patches; encodes the image patch PD, where the image patch encoding includes the image patch encoding PatchEmb and the one-dimensional position encoding PosEmb used to represent the order of the image patches, and the image patch encoding is expressed as:

[0022] ;

[0023] Learning internal features of visual modalities using self-attention mechanism ,in represents the internal features of the visual modality of each patch in each image after acquiring the features of other patches, and i represents the sequential index of the image patch.

[0024] In a preferred embodiment: in step S4, the text encoding corresponding to the context encoding after passing through the bidirectional GRU encoder is Take out and input into the point primitive spatial perception attention module to enhance the spatial perception ability of geometric primitives, where represents the encoding of the jth word in the context text encoding, and j represents the index of the word in the text clause; specifically, the point primitive space-aware attention module uses self-primitive attention to capture the intrinsic correlation between texts and texts. The self-attention formula It is expressed as:

[0025]

[0026] in, represents the encoding of the word in the mth context text encoding, m and j represent the index of the word in the text clause, Q(*) and K(*) are the query matrix and key matrix in the self-attention mechanism, d head represents the number of heads of multi-head autonomous attention; next, the relative distance bias between the point text and the geometric diagram is used to force the attention to pay attention to the relative position relationship between the point primitives, so as to enhance the perception of the spatial relationship of the geometric primitives. Further expressed as :

[0027] ,

[0028] where b pm-pj (x) and b pm-pj (y) represents the relative distance bias of the point text on the x-axis and y-axis respectively, pm and pj represent the point text in the text; finally, the spatial attention output h m is represented as:

[0029] ,

[0030] in, represents the normalization of each attention weight in the attention mechanism, k represents the kth text encoding, m represents the index of the word in the text clause, softmax(*) represents the activation function, and V(*) represents the value matrix in the attention mechanism.

[0031] In a preferred embodiment: in step S5, the text encoding passed through the point primitive spatial attention module and the visual encoding output by the bidirectional GRU are spliced ​​in the first dimension, and input into the self-limiting GRU decoder for decoding, a sequential solver is generated, and then the result is calculated by the Python settler.

[0032] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a novel spatial perception neural solver model for solving plane geometry problems based on geometric primitive spatial perception, and enhances the model's perception of the spatial distribution of geometric primitives through a multimodal dual-channel visual text point matching pre-trained language module and a point primitive spatial perception attention module, thereby better solving plane geometry problems. Compared with other methods, the present invention has the following characteristics:

[0033] (1) The present invention proposes a spatial perception neural solver model.

[0034] (2) The present invention proposes a multimodal dual-channel visual-text point matching pre-trained language module, which aligns visual points and text points in a fine-grained manner and demonstrates better cross-modal fusion performance.

[0035] (3) The present invention proposes a point primitive spatial perception attention module, which uses the relative distance of point primitives as a bias, makes fuller use of the information provided by geometric diagrams, and demonstrates superior performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of a preferred embodiment of the present invention.

[0037] Figure 2 This is a model diagram of a preferred embodiment of the present invention.

[0038] Figure 3 This is a structural diagram of a multimodal dual-channel visual text point matching pre-trained language module in a preferred embodiment of the present invention.

[0039] Figure 4 An example schematic diagram provided for a point primitive space-aware attention module of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0043] like Figures 1 to 3 As shown, the present invention discloses a method for automatically solving plane geometry problems based on geometric primitive space perception, comprising the following steps:

[0044] The method for automatically solving plane geometry problems based on geometric primitive space perception is characterized by comprising the following steps:

[0045] Step S1: using a parser to parse the question into a format such as line segments ABC, AB=3 based on geometric primitives such as points, lines, and circles mentioned in the question, and parsing the geometric diagrams and geometric questions to be solved into text clauses. Using a parser to parse the geometric diagrams and geometric questions to be solved into text clauses based on geometric primitives such as points, lines, and circles mentioned in the question, the text clauses include structural clauses, semantic clauses, and question texts, and input them into a Transformer Encoder to learn internal features of the text modality;

[0046] In step S1, the geometric diagram and the geometric problem to be solved are parsed into text clauses pt using preset rules, including: structural clauses, which are used to describe the connection relationship between geometric primitives; semantic clauses, which are used to describe the semantic relationship between non-geometric primitives and geometric primitives; problem text, which is used to describe the solution goal of the plane geometry problem; the text clauses are encoded to obtain the text encoding , where L represents the length of the text clause and the text encoding E PT It includes the text clause encoding TokenEmb, which is used to describe geometric images and text, the one-dimensional position encoding PosEmb, which is used to describe the order of each word in the text clause, the fragment encoding SectEmb, which is used to describe the source of each word in the text clause, and the category encoding ClassEmb, which is used to describe the category of each word in the text sentence; the text clause encoding is expressed as:

[0047] ;

[0048] in, Represents the words in the text clause parsed from the geometry diagram and geometry problem to be solved, j represents the index of the word in the text clause, the text clause encoding is input into the Transformer Encoder to extract the text features, and then a linear mapping is used to project the text features to obtain a new text feature

[0049] ,in Represents the internal features of the text modality obtained after each text word passes through the Transformer Encoder.

[0050] Step S2: The geometric diagram image is initially feature extracted through a convolutional neural network and divided into 64 diagram patches, and the internal features of the visual modality are learned through a self-attention mechanism;

[0051] In step S2, the convolution neural network uses ResNet10 to resize the image into 256*256 and input it into ResNet10 for convolution operation, dividing it into 64 image patches of equal size. , where i represents the sequential index of the image patch, Represents the features of each patch in the image, N represents the number of image patches; encodes the image patch PD, where the image patch encoding includes the image patch encoding PatchEmb and the one-dimensional position encoding PosEmb used to represent the order of the image patches, and the image patch encoding is expressed as:

[0052] ;

[0053] Learning internal features of visual modalities using self-attention mechanism ,in represents the internal features of the visual modality of each patch in each image after acquiring the features of other patches, and i represents the sequential index of the image patch.

[0054] Step S3: merge the text features and the image features into a multimodal dual-branch spatial perception pre-trained language module to obtain cross-modal global features and spatial information of geometric primitives (including points, lines, and circles), and input them into a bidirectional GRU encoder to perform fusion encoding;

[0055] In step S3, a multimodal dual-branch spatial perception pre-training language module is used, such as Figure 3As shown in FIG. 1 , the method is used to obtain the global features of vision-language and the spatial relationship of geometric primitives in geometric diagrams, including: a structural-semantic pre-training strategy for learning the global features of vision-language; a dual-branch vision-text point alignment strategy, which uses intra-modal and inter-modal vision-text point matching to achieve the alignment of the point primitive symbols appearing in the text with the corresponding image patches in the geometric diagram; specifically, the visual encoding of the geometric diagram is converted into With text encoding Splice and input Transformer Encoder to learn cross-modal information, and output visual features that integrate text features and text features fused with visual features , and then the contrastive learning-based approach is used to match the visual points and text points within and between modalities respectively. The loss based on contrastive learning is expressed as:

[0056]

[0057] in represents the intra-modal contrast loss, represents the contrast loss between modalities, and Represent the features of text points within and between modalities, and Respectively represent the characteristics of visual points within the modal and between modalities, and Represent the features of each word in the intra-modal and inter-modal text features, respectively, j represents the index of the word in the text clause, represents the linear projection function, represents the temperature coefficient, and T represents the transposition operation. The visual points and text points are aligned through the dual-branch visual-text point alignment strategy to obtain the spatial distribution relationship between geometric primitives. The loss is expressed as:

[0058] ;

[0059] The total training loss of the multimodal dual-branch spatial perception pre-trained language module is expressed as:

[0060] ,

[0061] Where L SSP represents the training loss of the structure-semantic pre-training strategy, and represents the hyperparameter, and then the output of the multimodal dual-branch spatial perception pre-trained language module is input into the bidirectional GRU encoder to perform fusion encoding to obtain the context encoding H.

[0062] Step S4: taking out the text encoding part in the hybrid encoding and inputting it into the point primitive spatial perception attention module to enhance the spatial perception ability of the geometric solver model to the geometric primitives;

[0063] In step S4, the text encoding corresponding to the context encoding after the bidirectional GRU encoder is Take out and input into the point primitive spatial perception attention module to enhance the spatial perception ability of geometric primitives, where Represents the encoding of the jth word in the context text encoding, and j represents the index of the word in the text clause; Specifically, the structure of the point primitive space-aware attention module is similar to that of the single-layer Transformer Encoder, consisting of a self-attention layer, layer normalization, a feedforward layer, and a residual link. Different from the traditional self-attention mechanism, the relative position information between points is additionally added when calculating the attention score. Figure 4 As shown, the coordinates of the point primitives provided by the geometric diagram can be used to calculate their relative distance in 2D space. The relative distance can reflect the relative positions between the point primitives. For example, if the relative distance between point F and point E on the x-axis is a positive value, it means that point F is on the right side of point E. Similarly, if the relative distance on the y-axis is a positive value, it means that point F is above point E. This method reflects the relationship between point primitives in space, allowing the model to better understand the spatial relationship between geometric primitives. Considering the large range of relative positions, we model the relative positions of point primitives in space as bias terms. Specifically, we use the original attention mechanism to capture the correlation between texts. The self-attention formula It is expressed as:

[0064]

[0065] in, represents the encoding of the word in the mth context text encoding, m and j represent the index of the word in the text clause, Q(*) and K(*) are the query matrix and key matrix in the self-attention mechanism, d head represents the number of heads of multi-head autonomous attention; next, the relative distance bias between the point text and the geometric diagram is used to force the attention to pay attention to the relative position relationship between the point primitives, so as to enhance the perception of the spatial relationship of the geometric primitives. Further expressed as :

[0066] ,

[0067] where b pm-pj (x) and b pm-pj(y) represents the relative distance bias of the point text on the x-axis and y-axis respectively, pm and pj represent the point text in the text; finally, the spatial attention output h m is represented as:

[0068] ,

[0069] in, represents the normalization of each attention weight in the attention mechanism, k represents the kth text encoding, m represents the index of the word in the text clause, softmax(*) represents the activation function, and V(*) represents the value matrix in the attention mechanism.

[0070] Step S5: The enhanced text encoding is concatenated with the image encoding in the hybrid encoding and decoded by the self-limited GRU decoder, and a sequential solver is generated.

[0071] In step S5, the text encoding after the point primitive spatial attention module and the visual encoding output by the bidirectional GRU are concatenated in the first dimension and input into the self-limited GRU decoder for decoding to generate a sequential solver, and then the result is calculated by the Python solver.

[0072] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A method for automatically solving plane geometry problems based on spatial perception of geometric primitives, characterized in that: The following steps are involved: Step S1: Using the parser to parse the geometry diagram and geometry problem to be solved into text clauses based on the geometric primitives of points, lines and circles mentioned in the problem, the text clauses include structural clauses, semantic clauses and question text, and input them into TransformerEncoder to learn the internal features of the text modality; Step S2: The geometric diagram image is initially feature extracted through a convolutional neural network and divided into 64 diagram patches, and the internal features of the visual modality are learned through a self-attention mechanism; Step S3: Merge the internal features of the text modality and the internal features of the visual modality and input them into the multimodal dual-branch spatial perception pre-trained language module to obtain cross-modal global features and geometric primitive space information, and input them into the bidirectional GRU encoder to perform fusion encoding; Step S4: taking out the text encoding part in the fusion encoding and inputting it into the point primitive spatial perception attention module to enhance the spatial perception ability of the geometric solver model to the geometric primitives; Step S5: concatenate the enhanced text code with the image code in the fusion code and decode it by the self-limited GRU decoder, and generate a sequential solver; In step S3, a multimodal dual-branch spatial perception pre-training language module is used to obtain cross-modal global features and geometric primitive space information, including: a structure-semantic pre-training strategy for learning visual-language global features; The dual-branch visual-text point alignment strategy uses intra-modality and inter-modality visual-text point matching to achieve the alignment of the point primitive symbols appearing in the text with the corresponding image patches in the geometric diagram; specifically, the visual encoding E of the geometric diagram is PD With text encoding E PT Concatenate and input TransformerEncoder to learn cross-modal information, and output visual features F′ that integrate text features PD and text features F′ that integrate visual features PT , and then the contrastive learning-based approach is used to match the visual points and text points within and between modalities respectively. The loss based on contrastive learning is expressed as: Where L intra represents the intra-modal contrast loss, L inter represents the contrast loss between modalities, and Represent the features of text points within and between modalities, and Respectively represent the characteristics of visual points within the modality and between modalities, and Represent the features of each word in the intra-modal and inter-modal text features, j represents the index of the word in the text clause, s(*, *) = W PT (f pt ) T W PD (f pd ), W PT (*) and W PD (*) represents the linear projection function, τ represents the temperature coefficient, and T represents the transposition operation. The visual points and text points are aligned through the dual-branch visual-text point alignment strategy to obtain the spatial distribution relationship between geometric primitives. The loss is expressed as: L VTP M ＝ L intra +L inter ; The total training loss of the multimodal dual-branch spatial perception pre-trained language module is expressed as: L all =α*L SSP +β*L VTPM , Where L SSP represents the training loss of the structure-semantic pre-training strategy, α and β represent hyperparameters, and then the output of the multimodal dual-branch spatially aware pre-trained language module is input into the bidirectional GRU encoder to perform fusion encoding to obtain the context encoding H.

2. The method for automatically solving plane geometry problems based on spatial perception of geometric primitives according to claim 1, characterized in that: In step S1, the geometric diagram and the geometric problem to be solved are parsed into text clauses pt using preset rules, including: structural clauses, which are used to describe the connection relationship between geometric primitives; semantic clauses, which are used to describe the semantic relationship between non-geometric primitives and geometric primitives; problem text, which is used to describe the solution goal of the plane geometry problem; the text clauses are encoded to obtain the text encoding Where L represents the length of the text clause and the text encoding E PT It includes the text clause encoding TokenEmb, which is used to describe geometric images and text, the one-dimensional position encoding PosEmb, which is used to describe the order of each word in the text clause, the fragment encoding SectEmb, which is used to describe the source of each word in the text clause, and the category encoding ClassEmb, which is used to describe the category of each word in the text sentence; the text clause encoding is expressed as: Among them, pt j Represents the words in the text clause parsed from the geometry diagram and geometry problem to be solved, j represents the index of the word in the text clause, the text clause encoding is input into the Transformer Encoder to extract the text features, and then a linear mapping is used to project the text features to obtain a new text feature 3. The method for automatically solving plane geometry problems based on spatial perception of geometric primitives according to claim 1, characterized in that: In step S2, the convolutional neural network uses ResNet10 to resize the image into 256*256 and input it into ResNet10 for convolution operation, dividing it into 64 image patches of equal size. Where i represents the sequential index of the image patch, pd i Represents the features of each patch in the image, N represents the number of image patches; encodes the image patch PD, where the image patch encoding includes the image patch encoding PatchEmb and the one-dimensional position encoding PosEmb used to represent the order of the image patches, and the image patch encoding is expressed as: Learning internal features of visual modalities using self-attention mechanism in represents the internal features of the visual modality of each patch in each image after acquiring the features of other patches, and i represents the sequential index of the image patch.

4. The method for automatically solving plane geometry problems based on spatial perception of geometric primitives according to claim 1, characterized in that: In step S4, the text encoding corresponding to the context encoding after the bidirectional GRU encoder is Take out and input into the point primitive spatial perception attention module to enhance the spatial perception ability of geometric primitives, where represents the encoding of the jth word in the context text encoding, and j represents the index of the word in the text clause; specifically, the point primitive space-aware attention module uses self-primitive attention to capture the intrinsic correlation between texts and texts. The self-attention formula a mj It is expressed as: in, represents the encoding of the word in the mth context text encoding, m and j represent the index of the word in the text clause, Q(*) and K(*) are the query matrix and key matrix in the self-attention mechanism, d head represents the number of heads of multi-head autonomous attention; next, the relative distance bias between the point text and the geometric diagram is used to force the attention to pay attention to the relative position relationship between the point primitives, so as to enhance the perception of the spatial relationship of the geometric primitives. mj Further expressed as where b pm-pj (x) and b pm-pj (y) represents the relative distance bias of the point text on the x-axis and y-axis respectively, pm and pj represent the point text in the text; finally, the spatial attention output h m is represented as: in, represents the normalization of each attention weight in the attention mechanism, k represents the kth text encoding, m represents the index of the word in the text clause, softmax(*) represents the activation function, and V(*) represents the value matrix in the attention mechanism.

5. The method for automatically solving plane geometry problems based on spatial perception of geometric primitives according to claim 1, characterized in that: In step S5, the text encoding after the point primitive spatial attention module and the visual encoding output by the bidirectional GRU are concatenated in the first dimension and input into the self-limited GRU decoder for decoding to generate a sequential solver, and then the result is calculated by the Python solver.

Images