Multimodal large model driven geometric image reading analysis method and system

Abstract

The application discloses a multi-modal large model driven geometric image reading analysis method and system, relates to the technical field of data processing, and comprises the following steps: after receiving image data uploaded by a user, a candidate set of geometric elements is extracted, and an initial confidence is set for each candidate element; a geometric labeling and recognition channel is started in parallel, visual encoders are used to extract the candidate set of geometric elements, visual embedding is generated, an OCR channel is activated at the same time for text recognition, and text semantic embedding is generated; through confidence weighted cross-modal alignment fusion, structured geometric representation is generated, which is input into a mixed reasoning and identification model, and an analysis result is output. The application solves the technical problems that the existing geometric image analysis method cannot effectively process image and text information at the same time, and the analysis accuracy and efficiency of complex geometric structures are low, and achieves the technical effects of accurately extracting geometric information and text labeling in an image through the fusion of multi-modal large models, and improving the analysis accuracy and efficiency.

Description

Multimodal large model-driven geometric image reading and parsing method and system Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for reading and parsing geometric images driven by multimodal large models. Background Technology

[0002] Geometric image analysis is an important research topic in computer vision and natural language processing, with wide applications in mathematical problem-solving, engineering design, and many other fields. However, traditional image processing and recognition methods face numerous challenges. On the one hand, the graphic, labeled, and textual information in geometric images is complex and diverse, making it difficult for single-modal processing methods to accurately extract and understand this information. On the other hand, geometric image analysis not only requires recognizing graphics and text but also inferring and deducing from geometric knowledge, which places higher demands on the comprehensive capabilities of the model. Existing technologies often rely on hand-designed feature extraction algorithms and simple geometric reasoning models, making it difficult to handle complex image content and diverse geometric structures, resulting in low accuracy and reliability of the analysis results. Summary of the Invention

[0003] This application provides a multimodal large model-driven geometric image reading and parsing method and system to solve the technical problems of existing geometric image parsing methods being unable to effectively process image and text information simultaneously, and having low parsing accuracy and efficiency for complex geometric structures.

[0004] The first aspect of this application provides a multimodal large model-driven geometric image reading and parsing method, the method comprising: after receiving image data uploaded by a user, performing adaptive preprocessing of the image data to establish an adaptive preprocessing result; inputting the adaptive preprocessing result into a geometric element extraction channel, wherein the geometric element extraction channel constructs a geometric element candidate set by jointly extracting edge response, connectivity analysis, and shape decomposition, the geometric element candidate set including vertex candidates, line segment or arc segment candidates, subscript candidates, and annotation text candidates, and setting an initial confidence level for each candidate element; activating a geometric annotation recognition channel in parallel, extracting the geometric element candidate set using a lightweight visual encoder, establishing a visual embedding, and simultaneously activating an OCR semantic parsing channel, performing text recognition on the question stem and the annotation text, and establishing a text semantic embedding; performing cross-modal alignment fusion of the visual embedding and the text semantic embedding under confidence weighting to generate a structured geometric representation; inputting the structured geometric representation into a hybrid reasoning recognition model and outputting a parsing result.

[0005] A second aspect of this application provides a multimodal large model-driven geometric image reading and parsing system, comprising: an image preprocessing module, used to perform adaptive preprocessing of image data after receiving image data uploaded by a user, and establish an adaptive preprocessing result; a geometric element extraction module, used to input the adaptive preprocessing result into a geometric element extraction channel, wherein the geometric element extraction channel constructs a geometric element candidate set by jointly extracting edge response, connectivity analysis, and shape decomposition, wherein the geometric element candidate set includes vertex candidates, line segment or arc segment candidates, subscript candidates, and labeled text candidates, and each candidate element is set with an initial confidence level identifier; a geometric label recognition module, used to start the geometric label recognition channel in parallel, extract the geometric element candidate set using a lightweight visual encoder, establish a visual embedding, and simultaneously activate an OCR semantic parsing channel to perform text recognition on the question stem and labeled text, and establish a text semantic embedding; a cross-modal alignment and fusion module, used to perform cross-modal alignment and fusion of the visual embedding and the text semantic embedding under confidence weighting, and generate a structured geometric representation; and a geometric representation parsing module, used to input the structured geometric representation into a hybrid reasoning recognition model and output the parsing result.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] This application provides a multimodal large model-driven geometric image reading and parsing method and system, which relates to the field of data processing technology. It optimizes image quality through adaptive preprocessing, extracts candidate sets of geometric elements, generates visual embeddings and text semantic embeddings in parallel using a visual encoder and OCR channels, generates structured geometric representations through cross-modal alignment and fusion, and inputs these representations into a hybrid inference model for parsing. This achieves accurate geometric image parsing and annotation recognition, solving the technical problems of existing geometric image parsing methods being unable to effectively process image and text information simultaneously, and exhibiting low accuracy and efficiency in parsing complex geometric structures. It achieves the technical effect of accurately extracting geometric information and text annotations from images by fusing a multimodal large model, thereby improving parsing accuracy and efficiency. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 is a schematic flowchart of the multimodal large model-driven geometric image reading and parsing method provided in an embodiment of this application;

[0010] Figure 2 is a schematic diagram of the structure of the multimodal large model driven geometric image reading and parsing system provided in the embodiment of this application.

[0011] Figure labeling: Image preprocessing module 11, geometric feature extraction module 12, geometric annotation recognition module 13, cross-modal alignment and fusion module 14, geometric representation analysis module 15. Detailed Implementation

[0012] This application provides a multimodal large model-driven geometric image reading and parsing method and system to solve the technical problems of existing geometric image parsing methods being unable to effectively process image and text information simultaneously, and having low parsing accuracy and efficiency for complex geometric structures.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0015] Example 1, as shown in Figure 1, this application provides a multimodal large model-driven geometric image reading and parsing method, which includes:

[0016] P10: After receiving the image data uploaded by the user, perform adaptive preprocessing of the image data and establish an adaptive preprocessing result.

[0017] Furthermore, in performing adaptive preprocessing of the image data, step P10 of this embodiment further includes:

[0018] P11: Perform feature extraction on the image data and establish a feature set, which includes brightness distribution features, noise texture features, tilt angle features, and local contrast features; P12: Use the feature set as input data to activate the image scene adaptive adjustment model and output basic correction adjustment parameters; P13: Perform preprocessing adjustment on the image data according to the basic correction adjustment parameters to complete adaptive preprocessing.

[0019] It should be understood that after a user uploads image data, an adaptive preprocessing program is first initiated to ensure the quality and usability of the image data, laying the foundation for subsequent data processing and analysis. The core of adaptive preprocessing lies in automatically adjusting image parameters through feature extraction and analysis of the image data to achieve accurate image correction and optimization.

[0020] Specifically, the first step is to extract features from the input image data to establish a feature set. This feature set encompasses several key features, including brightness distribution features, noise texture features, tilt angle features, and local contrast features. Brightness distribution features describe the overall distribution of pixel brightness in the image. By calculating the image's histogram or statistically analyzing the distribution range of brightness values, the system can understand the overall brightness level of the image. For example, if the image is generally dark, this feature provides crucial information for subsequent brightness adjustment. Noise texture features identify random noise or texture interference in the image. By analyzing the high-frequency components of the image or using noise detection algorithms, the intensity and distribution pattern of noise can be assessed, providing a reference for subsequent noise reduction processing. Tilt angle features determine the image's tilt angle by detecting the tilt of straight lines or geometric shapes. For example, techniques such as the Hough transform can be used to detect straight lines in the image and calculate their tilt angle, providing important information for subsequent image correction. Local contrast features assess the contrast differences between different regions of the image. By calculating the pixel value differences within local regions, areas with excessively low or high contrast can be identified, providing data support for subsequent contrast adjustment.

[0021] After feature extraction, the feature set can be used as input data to activate the image scene adaptive adjustment model. This model is trained using deep learning techniques and, through extensive image data, has mastered the optimal adjustment parameters for different image scenes. Based on the input feature set, the model automatically outputs a set of basic correction adjustment parameters, including brightness adjustment parameters, noise reduction parameters, tilt correction parameters, and contrast adjustment parameters. For example, if the feature set indicates that the image is generally dark and contains a lot of noise, the model will output higher brightness adjustment parameters and stronger noise reduction parameters to improve the image's brightness and clarity. In this way, the model can adaptively adjust the adjustment parameters according to different image scenes, ensuring the flexibility and effectiveness of image preprocessing.

[0022] Finally, based on the baseline correction parameters output by the image scene adaptive adjustment model, preprocessing adjustments of the image data are performed. This process includes several aspects such as brightness adjustment, noise reduction, tilt correction, and contrast enhancement. For brightness adjustment, the overall brightness of the image can be adjusted according to the brightness adjustment parameters, for example, by using techniques such as histogram equalization or gamma correction to improve the image's brightness level, making the image clearer and brighter. For noise reduction, noise in the image can be removed according to the noise reduction parameters, for example, by using algorithms such as bilateral filtering or non-local mean filtering to remove noise while preserving image details. For tilt correction, the image undergoes geometric transformation according to the tilt correction parameters to adjust its tilt angle, aligning it horizontally and reducing errors caused by tilt during subsequent geometric element extraction. For contrast enhancement, the local contrast of the image is enhanced according to the contrast adjustment parameters, for example, by using adaptive contrast enhancement algorithms to improve the contrast of details in the image, making the image's contours and features more prominent.

[0023] Through the above preprocessing adjustments, the image quality is significantly improved, the geometric elements in the image are clearer, the background noise is effectively suppressed, and the tilt angle is corrected, which can provide a reliable data foundation for subsequent geometric element extraction and multimodal data fusion.

[0024] Furthermore, step P13 in this embodiment of the application also includes:

[0025] P13-1: Perform geometric sensitivity enhancement processing on the preprocessing adjustment results. The geometric sensitivity enhancement processing includes direction consistency enhancement, local structure tensor enhancement, and fine line segment reinforcement enhancement. P13-2: Perform local visibility analysis on the geometric sensitivity enhancement processing results to establish a local visibility analysis map. The local visibility analysis includes edge sharpness, noise density, and occlusion probability analysis. P13-3: Perform differential preprocessing based on the local visibility analysis map to complete adaptive preprocessing.

[0026] Optionally, the image preprocessing workflow can be further refined to ensure the clarity and visibility of the image data in terms of geometric features, thereby better adapting to subsequent geometric feature extraction and parsing tasks.

[0027] First, after performing basic preprocessing adjustments, the results are further enhanced with geometric sensitivity enhancement. This process includes three key steps: orientation consistency enhancement, local structure tensor enhancement, and fine line segment enhancement. Orientation consistency enhancement aims to optimize the directional information of lines and edges in the image, ensuring that geometric elements with clear directions, such as line segments and subscripts, are fully displayed. By enhancing the edge response in the image, directional features are improved, making these geometric elements more prominent in the image.

[0028] Subsequently, local structure tensor enhancement is applied to each local region of the image. The local structure tensor, obtained by calculating the image gradient, provides information about the main orientation and geometric structure of the local region. Enhancing this structural information makes details in the image, such as corners and curves, more prominent. Local structure tensor enhancement effectively improves the visibility of complex shapes and detailed regions in the image, ensuring that these key structures are not affected by detail loss during subsequent analysis.

[0029] Fine line segment enhancement focuses on improving the visibility of small line segments. By increasing the contrast and sharpness of small line segments in an image, it helps to recover detailed elements within the image. Since small line segments are easily affected by noise or low contrast during shooting, fine line segment enhancement effectively highlights these small but important geometric elements, making them more prominent in the image and ensuring that subsequent geometric element extraction can accurately identify these small line segments.

[0030] After completing the geometry sensitivity enhancement processing, local visibility analysis is performed next. The purpose of this analysis is to evaluate the visibility of different regions in the image, particularly edge sharpness, noise density, and occlusion. First, the sharpness of edges in the image is evaluated using an edge detection algorithm. If the edges in certain regions are blurry, geometric elements may not be accurately extracted, thus requiring further edge enhancement processing in these areas. Edge sharpness analysis helps determine which regions require additional processing to ensure that important geometric elements in the image are not missed due to blurred edges.

[0031] Next, noise density analysis is performed. Noise in the image can interfere with subsequent geometric element extraction, especially in detailed areas. Noise density analysis can identify areas of high noise density in the image and provide a basis for denoising these areas. Areas with high noise density may require more robust denoising methods, such as median filtering or adaptive filtering, to reduce the impact of noise on the image and ensure improved image quality.

[0032] Occlusion probability analysis is used to assess potential occlusion in an image. In geometric images, some important elements may become invisible due to occlusion by other structures. Occlusion probability analysis can predict which regions may have occlusion problems and identify important geometric elements in these regions. For these regions, local inpainting or reconstruction methods may be needed to recover the occluded geometric information, thereby ensuring that all critical information in the image is fully presented.

[0033] Finally, differentiated preprocessing is performed based on the local visibility analysis map. In this stage, different processing strategies are adopted according to the visibility analysis results of each region. For regions with low edge sharpness, further edge enhancement processing can be used to improve the detail in the image; for regions with high noise density, a stronger denoising algorithm is needed to reduce the impact of noise on image resolution; and for regions with a high probability of occlusion, image reconstruction or local enhancement may be necessary to recover the occluded geometric elements, ensuring that the geometric information of these regions can be fully extracted. Through this differentiated preprocessing, the system can ensure that the image achieves optimal processing results in each region, thus completing the entire adaptive preprocessing workflow.

[0034] P20: Input the adaptive preprocessing results into the geometric feature extraction channel. The geometric feature extraction channel constructs a set of geometric element candidates by combining edge response, connectivity analysis, and shape decomposition extraction. The set of geometric element candidates includes vertex candidates, line segment or arc segment candidates, subscript candidates, and annotation text candidates, and each candidate element is set with an initial confidence level.

[0035] Furthermore, step P20 in this embodiment of the application also includes:

[0036] P21: Activate the geometric bounding box generation layer, perform multi-scale directional gradient field analysis on the adaptive preprocessing results, construct a joint edge response map, and generate the initial bounding box region of the geometric components through a dual constraint strategy of edge strength consistency and corner gradient aggregation degree; P22: Activate the geometric fragment aggregation layer, perform connectivity analysis on the initial bounding box region, construct a connectivity graph using directional extension rules, minimum jump rules, and local structure similarity rules, aggregate fractured, weak line segments and local curve fragments into geometric candidate fragments with clear shapes, and set topological credibility scoring labels; P23: Activate the shape decomposition layer, and perform... The geometric candidate fragment shape atom decomposition analysis decomposes line segments, arc segments, subscripts, arrow labels, and perpendicular parallel markers into reusable geometric semantic atoms, and constructs geometric semantic atom confidence based on shape reconstruction error, local orientation field variance, and image preprocessing confidence; P24: Candidate set construction layer, used to recombine the geometric semantic atoms according to topological adjacency, shape consistency, and directional continuity to form a geometric element candidate set, and constructs a multi-source confidence field based on image preprocessing confidence, topological confidence score, and geometric semantic atom confidence, generating an initial confidence identifier for each candidate element.

[0037] In one possible embodiment of this application, adaptively preprocessed image data is input to a geometric feature extraction channel to extract and construct a candidate set of geometric elements. The candidate set of geometric elements includes vertex candidates, line segment or arc segment candidates, subscript candidates, and annotation text candidates. Each candidate element is assigned an initial confidence score for subsequent processing and analysis.

[0038] First, the geometric bounding box generation layer is activated, and multi-scale directional gradient field analysis is performed on the adaptive preprocessing results. This analysis captures the direction and intensity of edges by calculating gradient information at different scales and constructs a joint edge response map. This map reflects the location and orientation of potential edges in the image. Next, an initial bounding box region for the geometric components is generated through a dual-constraint strategy of edge intensity consistency and corner gradient clustering. Edge intensity consistency ensures that extracted edges have similar intensity within local regions, effectively suppressing interference from background texture and non-geometric noise; corner gradient clustering further enhances the boundary information of the geometric shape by detecting gradient clustering near corners. This dual-constraint strategy ensures the accuracy and stability of the initial bounding box region, providing a reliable foundation for subsequent geometric feature extraction.

[0039] After generating the initial bounding box regions, the geometric fragment aggregation layer is further activated to perform connectivity analysis on these regions. The purpose of connectivity analysis is to aggregate broken, weak line segments and local curve fragments into geometrically well-defined candidate fragments. To this end, directional extension rules, minimum jump rules, and local structural similarity rules can be used to construct a connectivity graph. Specifically, the directional extension rule ensures the consistency of line segments in direction, allowing broken line segments to be correctly connected; the minimum jump rule restricts directional changes during line segment connections, avoiding erroneous aggregation; and the local structural similarity rule further verifies connectivity by comparing the local structural features of fragments. Through these rules, broken line segments and local curve fragments can be aggregated into complete geometric candidate fragments, and a topological confidence score label is assigned to each fragment to measure its connectivity and geometric integrity, thereby ensuring the quality of the candidate fragments.

[0040] Subsequently, the shape decomposition layer is entered, where shape atom decomposition analysis is performed on the geometric candidate fragments. Complex geometric shapes are broken down into basic geometric semantic atoms—the fundamental units that can independently constitute geometric shapes, such as line segments, arcs, subscripts, arrowheads, and perpendicular / parallel markers. These geometric semantic atoms form the basis for subsequent geometric element construction. To evaluate the reliability of each geometric semantic atom, a confidence score is constructed based on shape reconstruction error, local orientation field variance, and image preprocessing confidence. The shape reconstruction error reflects the degree of matching between the atom and the original image data; the local orientation field variance measures the consistency of the orientation within the atom; and the image preprocessing confidence considers the impact of the preprocessing stage on data quality. By integrating these factors, a confidence score can be assigned to each geometric semantic atom, ensuring that these elements are accurately represented in subsequent geometric parsing processes.

[0041] Finally, the candidate set construction layer is entered, where geometric semantic atoms are recombinated according to topological adjacency, shape consistency, and directional continuity to form the final candidate set of geometric elements. In this layer, geometric semantic atoms are combined based on conditions such as topological adjacency, shape consistency, and directional continuity to generate a complete candidate set of geometric elements. These combined candidate sets include various geometric elements in the image, such as vertices, line segments, subscripts, and annotation text. To ensure the reliability and accuracy of each candidate element, a multi-source confidence field is constructed by combining image preprocessing confidence, topological confidence score, and geometric semantic atom confidence, generating an initial confidence identifier for each candidate element. This initial confidence identifier will help determine the confidence of each candidate element in subsequent steps, thereby supporting more accurate geometric parsing and image understanding.

[0042] Furthermore, step P23 in this embodiment of the application also includes:

[0043] P23-1: Based on the local orientation field, curvature distribution, and structural tensor features of the geometric candidate fragments, a local semantic field for semantic decomposition is generated to distinguish the local stable regions of linear structures, circular arc structures, and symbol structures; P23-2: In the local orientation field, geometric semantic boundaries are automatically identified based on directional discontinuities, curvature transition points, and symbol shape inflection points, and the geometric candidate fragments are decomposed into geometric semantic atoms, including line segment atoms, arc segment atoms, subscript atoms, arrow annotation atoms, and vertical parallel symbol atoms; P23-3: Geometric axiom consistency is checked on the geometric semantic atoms, and geometric semantic atoms that fail the consistency check and have a local confidence level higher than a preset threshold are back-mapped to the original fragments for secondary decomposition. The geometric axiom consistency check includes linear consistency, circular arc fitting consistency, and symbol geometric consistency check; P23-4: The checked geometric semantic atoms are output according to the check results.

[0044] Specifically, the operation process of the shape decomposition layer can be further refined. Through the generation of local semantic fields, automatic identification of geometric semantic boundaries, verification of geometric axiom consistency, and secondary decomposition, the accuracy and reliability of geometric semantic atoms can be ensured.

[0045] When performing the shape decomposition layer operation, a local semantic field for semantic decomposition is first generated based on the local orientation field, curvature distribution, and structural tensor features of the geometric candidate fragments. The local orientation field reflects the directional information of each point in the candidate fragment; the curvature distribution describes the degree of curvature of the fragment; and the structural tensor features capture the local structural features of the fragment. By synthesizing these features, the system can construct a local semantic field that distinguishes the local stable regions of linear structures, circular arc structures, and symbolic structures. With the help of the local semantic field, local stable regions in geometric fragments can be identified; these regions have relatively stable geometric morphological features and are not easily affected by external noise or deformation. In this way, complex geometric structures in the image are further classified, facilitating subsequent geometric semantic analysis.

[0046] After generating the local semantic field, the geometric semantic boundaries are automatically identified in the local orientation field based on directional discontinuities, curvature transition points, and symbol morphology inflection points. Directional discontinuities typically mark the start or end points of line segments; curvature transition points indicate the start or end positions of arc segments; and symbol morphology inflection points can identify the boundaries of geometric symbols such as subscripts, arrow labels, and perpendicular / parallel symbols. Based on these feature points, geometric candidate fragments can be decomposed into basic geometric semantic atoms. Geometric semantic atoms are the basic units constituting geometric structures, including line segment atoms, arc segment atoms, subscript atoms, arrow label atoms, and perpendicular / parallel symbol atoms. Each geometric semantic atom represents a specific type of geometric information in the image, and its morphology and semantics can be understood and reconstructed independently.

[0047] Next, geometric semantic atoms undergo geometric axiom consistency checks, including linear consistency, arc fitting consistency, and symbolic geometric consistency checks. Geometric semantic atoms must conform to certain geometric axioms to ensure the accuracy of their shape and structure. For example, line segment atoms should satisfy linear consistency, arc segment atoms should satisfy arc fitting consistency, and symbolic atoms should satisfy symbolic geometric consistency. These checks ensure that each decomposed geometric semantic atom is geometrically valid and consistent. If some geometric semantic atoms fail the consistency check but their local confidence is higher than a preset threshold, further decomposition is required. Specifically, these geometric semantic atoms are back-mapped to the original candidate fragments and subjected to secondary decomposition to further refine and correct their geometric structure, ensuring that they ultimately meet the requirements of the geometric axioms.

[0048] Finally, the verified geometric semantic atoms are output based on the verification results. These geometric semantic atoms, which have undergone rigorous verification and possible secondary decomposition, have higher accuracy and reliability, and can provide high-quality data support for the subsequent construction of candidate sets of geometric elements.

[0049] P30: The geometric annotation recognition channel is launched in parallel. After extracting the candidate set of geometric elements using a lightweight visual encoder, a visual embedding is established. Simultaneously, the OCR semantic parsing channel is activated. After performing text recognition on the question stem and the annotated text, a text semantic embedding is established.

[0050] Optionally, the geometric annotation recognition channel and the OCR semantic parsing channel can be launched in parallel to efficiently extract geometric and textual information from the image. Through the collaborative work of the visual encoder and OCR technology, geometric elements and textual content are processed separately, ultimately generating visual embeddings and textual semantic embeddings to achieve comprehensive analysis of the geometric image.

[0051] First, in the geometric annotation recognition channel, a lightweight visual encoder is used to process the candidate set of geometric elements. The lightweight visual encoder is an efficient and resource-efficient neural network architecture capable of automatically identifying and extracting various geometric elements from images, such as line segments, subscripts, and annotation text. These geometric elements are converted into low-dimensional feature representations by the encoder, forming visual embeddings. Visual embeddings are compact representations of geometric information in images, preserving the spatial relationships of geometric elements while removing redundant information, thus providing effective support for subsequent tasks such as geometric reasoning and annotation recognition.

[0052] Simultaneously, the OCR semantic parsing channel is activated. OCR (Optical Character Recognition) technology is used to recognize text content in the image, including the question stem and annotation text. In this channel, text recognition is performed on the question stem and annotation text, converting the text content in the image into an editable text format. This process not only recognizes the text content but also performs in-depth analysis of the text through semantic parsing technology, extracting semantic information from the text, including its meaning and contextual relationships. This information is encoded as a text semantic embedding. Text semantic embedding is also a high-dimensional vector representation that can effectively capture the semantic features of the text, providing textual support for subsequent geometric image parsing.

[0053] The collaborative work of these two channels enables the system to comprehensively analyze geometric images, understanding not only the geometric shapes and annotations in the images, but also the related textual information. This greatly improves the accuracy and efficiency of geometric image analysis, providing a solid foundation for subsequent geometric image analysis and applications.

[0054] P40: Perform cross-modal alignment fusion under confidence weighting on the visual embedding and text semantic embedding to generate a structured geometric representation.

[0055] Furthermore, step P40 in this embodiment of the application also includes:

[0056] P41: Establish a visual confidence matrix using the initial confidence identifier; P42: Construct a text confidence matrix based on the OCR recognition confidence, context consistency score, and question stem relevance in the OCR semantic parsing channel; P43: After normalizing the visual confidence matrix and text confidence matrix using a self-supervised confidence scheduler, perform bidirectional attention interaction verification based on visual embedding and text semantic embedding to generate a structured geometric representation.

[0057] It should be understood that by establishing visual and textual confidence matrices, and using a self-supervised confidence scheduler for normalization and bidirectional attention interaction verification, a structured geometric representation is ultimately generated.

[0058] First, confidence-weighted processing is performed on the visual embeddings and textual semantic embeddings generated by the geometric element extraction channel and the OCR semantic parsing channel. Specifically, a visual confidence matrix is ​​first established using initial confidence labels. The initial confidence is derived from the geometric element extraction stage and is based on geometric features in the image, such as edge strength and shape matching. The purpose of generating the visual confidence matrix is ​​to assign a confidence value to each geometric element, which reflects the reliability of the visual information in the overall image analysis. This confidence matrix will serve as the basis for subsequent processing, ensuring that high-confidence elements of visual information receive stronger weights during cross-modal alignment.

[0059] Meanwhile, a text confidence matrix is ​​constructed based on the OCR recognition confidence score, context consistency score, and stem relevance score from the OCR semantic parsing channel. The OCR recognition confidence score reflects the accuracy of text recognition; the context consistency score measures the degree of matching between the text content and its context; and the stem relevance score assesses the semantic connection between the text and the geometric question stem. These factors are combined to form the text confidence matrix, which also records the confidence level of text information in a structured form, providing a basis for text information fusion.

[0060] After establishing the visual and text confidence matrices, a self-supervised confidence scheduler was used to normalize these two matrices. The purpose of normalization is to adjust the confidence values ​​of visual and textual information to the same scale, ensuring their comparability in subsequent fusion processes.

[0061] Finally, based on the normalized visual and text confidence matrices, bidirectional attention-based interactive verification of visual embedding and text semantic embedding is performed. This bidirectional attention mechanism allows visual and textual information to mutually attend to and verify each other, thereby identifying consistency and differences between visual and textual information. For example, if text annotations and visually detected geometric elements are inconsistent in position or semantics, the bidirectional attention mechanism can detect this difference and optimize the fusion result through confidence adjustment. Through this interactive verification, visual and textual information can be effectively aligned and fused, ultimately generating a structured geometric representation. This representation not only includes the visual features of the geometric image but is also closely integrated with semantic content such as annotations and descriptions in the text, providing rich support for subsequent tasks such as geometric reasoning, question answering, and knowledge point annotation.

[0062] Furthermore, based on visual embedding and text semantic embedding, bidirectional attention interaction verification is performed. In this embodiment, step P43 further includes:

[0063] P43-1: Generate cross-modal attention query groups and key-value groups using geometric topology fields in visual embedding and structural semantic units in text semantic embedding, and generate a geometric-aware mask based on set adjacency relationships to restrict attention propagation to execute within the topologically permissible range; P43-2: In the direction of mapping from visual embedding to text semantic embedding, use geometric elements as query vectors and semantic labels as keys, and use geometric structure similarity, distance decay factor, and orientation field consistency factor to correct attention weights, outputting a visually dominant attention map; P43-3: In the direction of mapping from text semantic embedding to visual embedding, use semantic labels as query vectors and candidate geometric elements as keys, and add stem semantic dependency, semantic redundancy exclusion factor, and annotation citation relationship factor to the attention calculation, outputting a text-dominant attention map; P43-4: Perform bidirectional consistency cross-measurement on the visually dominant attention map and the text-dominant attention map to generate a structured geometric representation.

[0064] Specifically, during the bidirectional attention interaction verification process, cross-modal attention query groups and key-value groups are first generated using the geometric topology fields in visual embeddings and the structural semantic units in text semantic embeddings. In this stage, based on the geometric topology fields in visual embeddings (i.e., the geometric structure and spatial relationships in the image) and the structural semantic units in text semantic embeddings (i.e., the semantic information of each part of the text and their correspondences), cross-modal query groups and key-value groups are generated. These query groups and key-value groups form the basis for attention calculations, representing the elements to be aligned in the visual embeddings and text semantic embeddings, respectively.

[0065] Next, to ensure the effectiveness and rationality of the attention mechanism, a geometry-aware mask is generated based on set adjacency relationships. The role of the geometry-aware mask is to restrict attention propagation to the extent allowed by the topology, avoiding unreasonable or incorrect attention allocation. For example, if two geometric elements are not topologically adjacent, the geometry-aware mask will restrict attention propagation between them, thereby improving the accuracy and efficiency of the attention mechanism.

[0066] Next, in the mapping from visual embedding to textual semantic embedding, geometric elements are used as query vectors and semantic labels as keys. Attention weights are adjusted using geometric similarity, distance decay factor, and orientation field consistency factor. Geometric similarity helps measure the similarity between visual elements and their corresponding descriptions in the text; the distance decay factor ensures that elements that are farther away have lower weights; and the orientation field consistency factor corrects the orientation consistency between geometric elements in the image and their descriptions in the text. Through the comprehensive adjustment of these factors, a visual-dominated attention map is output, reflecting the distribution of visual information's attention to textual information.

[0067] Similarly, in the mapping from text semantic embedding to visual embedding, semantic tags are used as query vectors and candidate geometric elements as keys. The attention calculation incorporates stem semantic dependency, semantic redundancy exclusion factor, and annotation citation relationship factor. Stem semantic dependency measures the semantic relevance between semantic tags and stem text; the semantic redundancy exclusion factor avoids excessive influence of repetitive semantic information on attention; and the annotation citation relationship factor considers the citation relationship between annotation text and geometric elements. Through the comprehensive correction of these factors, a text-dominated attention map is output, reflecting the distribution of attention from text information to visual information.

[0068] Finally, a bidirectional consistency cross-metric is performed on the visual-dominated attention map and the text-dominated attention map. By cross-measuring the two attention maps, the similarity between them is calculated, verifying whether the alignment between visual and textual information is consistent. If the alignment between the two attention maps is high, it indicates a strong semantic and geometric correlation. Through this bidirectional consistency cross-metric, a structured geometric representation is ultimately generated. This representation integrates the precise alignment results of visual and textual information into a structured output, providing reliable support for subsequent tasks such as geometric reasoning and problem-solving.

[0069] P50: Input the structured geometric representation into the hybrid reasoning recognition model and output the analytical result.

[0070] The analysis results include key points for understanding the question, analysis of the thought process, and content of distributed derivation. The analysis results are executed through user interaction via voice broadcast function, and the analysis results are output in an adaptive step-guided manner after the user activates the dynamic step unlocking mechanism.

[0071] Optionally, after generating the structured geometric representation, this representation is input into the hybrid reasoning recognition model to output detailed analytical results. The hybrid reasoning recognition model is a comprehensive reasoning model that combines the visual features of geometric images, the semantic information of text annotations, and geometric axioms and theorems, enabling comprehensive analysis and solutions to geometric problems. Trained using deep learning technology, this model is capable of handling complex geometric problems and can generate detailed analytical results including key points for understanding the problem, logical reasoning, and step-by-step derivation.

[0072] First, the hybrid reasoning model processes the structured geometric representation of the input to extract the core information from the problem, forming key points for understanding the question. This part helps users quickly identify the key elements of the problem, such as the type of figure, known conditions, and the problem to be solved, thus providing a clear direction for subsequent problem-solving. Next, the model further analyzes the key points, providing a problem-solving approach. The problem-solving approach section details the basic problem-solving strategies and methods for addressing the geometric problem, helping users understand the logical structure of the solution.

[0073] Building upon this foundation, the system also provides step-by-step derivations, showcasing each specific step in the problem-solving process and providing intermediate results and derivation details. Through these detailed derivations, users can clearly see how the problem-solving approach is translated into concrete operations and understand the basis for each calculation.

[0074] To enhance user experience, the system also features a voice broadcast function, which outputs the solution results in audio format. Through voice broadcast, users can listen and follow along with the problem-solving process, providing a more intuitive and convenient guidance, especially during learning. Users can follow the solution steps via voice feedback without frequently checking the screen, thus improving the interactivity of the problem-solving process.

[0075] Furthermore, the system introduces a dynamic step-by-step unlocking mechanism, allowing users to gradually unlock different parts of the solution based on their learning progress and comprehension ability. Users can first review the key points of the problem, and then gradually unlock the thought process and step-by-step derivation as needed. This mechanism not only improves users' learning autonomy but also enhances their understanding and mastery of geometry problems. When a user activates the dynamic step-by-step unlocking mechanism, the system adaptively guides the output of the solution, ensuring that the user receives clear guidance and feedback at each step.

[0076] Furthermore, step P50 in this embodiment of the application also includes:

[0077] After a user submits an answer, a consistency check is performed between the submitted result and the parsed output. Results that fail the consistency check are recorded in the incorrect question set.

[0078] In one possible embodiment of this application, in addition to generating structured geometric representations and providing detailed analysis processes, a consistency verification function can be added after the user submits their answer, so as to enhance the user's learning experience and feedback mechanism.

[0079] The core of this consistency verification process is to compare the user's submitted answer with the system's output analysis, including the final answer and each derivation step in the problem-solving process. If the user's answer is completely consistent with the system's analysis, the system will consider the user's solution correct and provide positive feedback; if the user's answer deviates from or contains errors in the analysis, the system will mark the result as inconsistent and take appropriate action.

[0080] For results that fail consistency verification, the system will automatically record the question and related error information in the incorrect question collection. The incorrect question collection is a dynamically updated recording system specifically designed to store questions and their incorrect answers that users encounter during the learning process. The purpose of this process is to help users track their errors and review and improve in a targeted manner during subsequent learning. The incorrect question collection not only records incorrect answers but also saves key steps in the explanation process so that users can understand and correct their mistakes during review.

[0081] This approach not only helps users understand the correct problem-solving strategies and steps, but also provides a targeted feedback mechanism to help users identify and review their weaknesses, thereby improving their problem-solving abilities.

[0082] In summary, the embodiments of this application have at least the following technical effects:

[0083] This application improves the accuracy of geometric image parsing by fusing multimodal large models and combining image and text information to accurately extract geometric elements and labeled text from images. Through adaptive preprocessing, geometric element extraction, and automated processing of visual and text embedding, it reduces manual intervention, improves parsing efficiency and automation, and enhances the system's adaptability to different scenarios. It achieves cross-modal alignment and fusion of image and text information, improving the correlation and consistency between the two modalities. Furthermore, through a mistake log and dynamic step-unlocking mechanism, it supports personalized learning, helping users to conduct targeted review and problem-solving based on error records and learning progress.

[0084] This technology achieves the goal of accurately extracting geometric information and text annotations from images by fusing multimodal large models, thereby improving the accuracy and efficiency of image processing.

[0085] Example 2: Based on the same inventive concept as the multimodal large model-driven geometric image reading and parsing method in the preceding examples, as shown in Figure 2, this application provides a multimodal large model-driven geometric image reading and parsing system. The system and method examples in this application are based on the same inventive concept. The system includes:

[0086] The image preprocessing module 11 is used to perform adaptive preprocessing of the image data after receiving the image data uploaded by the user, and to establish an adaptive preprocessing result.

[0087] The geometric feature extraction module 12 is used to input the adaptive preprocessing results into the geometric feature extraction channel. The geometric feature extraction channel constructs a set of geometric element candidates by combining edge response, connectivity analysis, and shape decomposition extraction. The set of geometric element candidates includes vertex candidates, line segment or arc segment candidates, subscript candidates, and annotation text candidates, and each candidate element is set with an initial confidence level identifier.

[0088] The geometric annotation recognition module 13 is used to launch the geometric annotation recognition channel in parallel. After extracting the candidate set of geometric elements using a lightweight visual encoder, a visual embedding is established. Simultaneously, the OCR semantic parsing channel is activated to perform text recognition on the question stem and the annotated text and then establish a text semantic embedding.

[0089] The cross-modal alignment and fusion module 14 is used to perform cross-modal alignment and fusion of the visual embedding and the text semantic embedding under confidence weighting to generate a structured geometric representation.

[0090] The geometric representation parsing module 15 is used to input the structured geometric representation into the hybrid reasoning recognition model and output the parsing result.

[0091] Furthermore, the image preprocessing module 11 is also used to perform the following steps:

[0092] The image data is subjected to feature extraction to establish a feature set, which includes brightness distribution features, noise texture features, tilt angle features, and local contrast features. The feature set is used as input data to activate the image scene adaptive adjustment model and output basic correction adjustment parameters. The image data is preprocessed and adjusted according to the basic correction adjustment parameters to complete the adaptive preprocessing.

[0093] Furthermore, the image preprocessing module 11 is also used to perform the following steps:

[0094] The preprocessing adjustment results are subjected to geometric sensitivity enhancement processing, which includes directional consistency enhancement, local structure tensor enhancement, and fine line segment enhancement. Local visibility analysis is performed on the geometric sensitivity enhancement results to establish a local visibility analysis map, which includes edge sharpness, noise density, and occlusion probability analysis. After performing differential preprocessing based on the local visibility analysis map, adaptive preprocessing is completed.

[0095] Furthermore, the geometric feature extraction module 12 is also used to perform the following steps:

[0096] Activate the geometric bounding box generation layer, perform multi-scale directional gradient field analysis on the adaptive preprocessing results, construct a joint edge response map, and generate the initial bounding box region of the geometric components through a dual constraint strategy of edge strength consistency and corner gradient aggregation degree; activate the geometric fragment aggregation layer, perform connectivity analysis on the initial bounding box region, construct a connectivity graph using directional extension rules, minimum jump rules, and local structure similarity rules, aggregate fractured, weak line segments and local curve fragments into geometric candidate fragments with clear shapes, and set a topological credibility score label; activate the shape decomposition layer, and perform geometric candidate generation. The shape atom decomposition analysis of the fragment decomposes line segments, arc segments, subscripts, arrow labels, and perpendicular parallel markers into reusable geometric semantic atoms, and constructs the geometric semantic atom confidence based on shape reconstruction error, local orientation field variance, and image preprocessing confidence. The candidate set construction layer is used to recombine the geometric semantic atoms according to topological adjacency, shape consistency, and directional continuity to form a geometric element candidate set, and constructs a multi-source confidence field based on image preprocessing confidence, topological confidence score, and geometric semantic atom confidence, generating an initial confidence identifier for each candidate element.

[0097] Furthermore, the geometric feature extraction module 12 is also used to perform the following steps:

[0098] Based on the local orientation field, curvature distribution, and structural tensor features of the geometric candidate fragments, a local semantic field is generated for semantic decomposition to distinguish the local stable regions of linear structures, circular arc structures, and symbol structures. Within the local orientation field, geometric semantic boundaries are automatically identified based on directional discontinuities, curvature transition points, and symbol shape inflection points. The geometric candidate fragments are then decomposed into geometric semantic atoms, including line segment atoms, arc segment atoms, subscript atoms, arrow marker atoms, and vertically parallel symbol atoms. Geometric axiom consistency checks are performed on these geometric semantic atoms. Geometric semantic atoms that fail the consistency check and have a local confidence level higher than a preset threshold are back-mapped to the original fragments for secondary decomposition. The geometric axiom consistency checks include linear consistency, circular arc fitting consistency, and symbol geometric consistency checks. The checked geometric semantic atoms are then output based on the check results.

[0099] Furthermore, the cross-modal alignment and fusion module 14 is also used to perform the following steps:

[0100] A visual confidence matrix is ​​established using the initial confidence identifier; a text confidence matrix is ​​constructed based on the OCR recognition confidence, context consistency score, and question stem relevance in the OCR semantic parsing channel; after normalizing the visual confidence matrix and text confidence matrix using a self-supervised confidence scheduler, bidirectional attention interaction verification is performed based on visual embedding and text semantic embedding to generate a structured geometric representation.

[0101] Furthermore, the cross-modal alignment and fusion module 14 is also used to perform the following steps:

[0102] Cross-modal attention query groups and key-value groups are generated using geometric topology fields in visual embeddings and structural semantic units in text semantic embeddings. A geometry-aware mask is generated based on set adjacency relationships to restrict attention propagation to occur within the allowed topological range. In the direction of mapping from visual embeddings to text semantic embeddings, geometric elements are used as query vectors and semantic labels as keys. Attention weights are corrected using geometric structure similarity, distance decay factors, and orientation field consistency factors, resulting in a visually dominant attention map. In the direction of mapping from text semantic embeddings to visual embeddings, semantic labels are used as query vectors and candidate geometric elements as keys. Question stem semantic dependency, semantic redundancy exclusion factors, and annotation citation relationship factors are added to the attention calculation, resulting in a text-dominant attention map. A bidirectional consistency cross-metric is performed on the visually dominant attention map and the text-dominant attention map to generate a structured geometric representation.

[0103] Furthermore, in the geometric representation analysis module 15:

[0104] The analysis results include key points for understanding the question, thought process analysis, and distributed derivation content. The analysis results are executed through user interaction via voice broadcast function, and the analysis results are output in an adaptive step-guided manner after the user activates the dynamic step unlocking mechanism.

[0105] Furthermore, the geometric representation analysis module 15 is also used to perform the following steps:

[0106] After a user submits an answer, a consistency check is performed between the submitted result and the parsed output. Results that fail the consistency check are recorded in the incorrect question set.

[0107] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0108] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0109] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. A geometric image reading and parsing method driven by a multimodal large model, characterized in that, The method includes: after receiving image data uploaded by a user, performing adaptive preprocessing of the image data to establish an adaptive preprocessing result; inputting the adaptive preprocessing result into a geometric element extraction channel, wherein the geometric element extraction channel constructs a candidate set of geometric elements through joint edge response, connectivity analysis, and shape decomposition extraction, wherein the candidate set of geometric elements includes vertex candidates, line segment or arc segment candidates, subscript candidates, and annotation text candidates, and each candidate element is assigned an initial confidence level identifier; parallelly activating a geometric annotation recognition channel, extracting the candidate set of geometric elements using a lightweight visual encoder, establishing a visual embedding, and simultaneously activating an OCR semantic parsing channel, performing text recognition on the question stem and the annotation text, and establishing a text semantic embedding; performing cross-modal alignment fusion under confidence weighting on the visual embedding and the text semantic embedding to generate a structured geometric representation; inputting the structured geometric representation into a hybrid reasoning recognition model and outputting a parsing result; performing cross-modal alignment fusion under confidence weighting on the visual embedding and the text semantic embedding to generate a structured geometric representation, including: establishing a visual confidence matrix using the initial confidence level identifier; and based on the OCR recognition confidence level and context consistency evaluation in the OCR semantic parsing channel... A text confidence matrix is ​​constructed based on the correlation between the visual and textual confidence matrices. After normalization of the visual and textual confidence matrices using a self-supervised confidence scheduler, bidirectional attention interaction verification is performed based on visual embedding and textual semantic embedding to generate a structured geometric representation. This bidirectional attention interaction verification includes: generating cross-modal attention query groups and key-value groups using geometric topology fields in visual embedding and structural semantic units in textual semantic embedding, and generating a geometric-aware mask based on set adjacency relationships to restrict attention propagation within the allowed topological range; in the direction of mapping from visual embedding to textual semantic embedding, using geometric elements as query vectors and semantic labels as keys, attention weights are corrected using geometric structure similarity, distance decay factors, and orientation field consistency factors to output a visually dominant attention map; in the direction of mapping from textual semantic embedding to visual embedding, using semantic labels as query vectors and candidate geometric elements as keys, question stem semantic dependency, semantic redundancy exclusion factors, and annotation citation relationship factors are added to the attention calculation to output a text-dominant attention map; bidirectional consistency cross-metrics are performed on the visually dominant attention map and the text-dominant attention map to generate a structured geometric representation.

2. The multimodal large model-driven geometric image reading and parsing method as described in claim 1, characterized in that, The adaptive preprocessing results are input into the geometric feature extraction channel, including: activating the geometric border generation layer, performing multi-scale directional gradient field analysis on the adaptive preprocessing results, constructing a joint edge response map, and generating the initial border region of the geometric components through a dual constraint strategy of edge strength consistency and corner gradient aggregation; activating the geometric fragment aggregation layer, performing connectivity analysis on the initial border region, constructing a connectivity graph using directional extension rules, minimum jump rules, and local structure similarity rules, aggregating fractured, weak line segments and local curve fragments into geometric candidate fragments with clear shapes, and setting a topological credibility score label; activating The shape decomposition layer performs shape atom decomposition analysis on geometric candidate fragments, decomposing line segments, arc segments, subscripts, arrow labels, and perpendicular parallel markers into reusable geometric semantic atoms, and constructing geometric semantic atom confidence based on shape reconstruction error, local orientation field variance, and image preprocessing confidence. The candidate set construction layer is used to recombine the geometric semantic atoms according to topological adjacency, shape consistency, and directional continuity to form a geometric element candidate set, and constructs a multi-source confidence field based on image preprocessing confidence, topological confidence score, and geometric semantic atom confidence, generating an initial confidence identifier for each candidate element.

3. The multimodal large model-driven geometric image reading and parsing method as described in claim 2, characterized in that, Activating the shape decomposition layer includes: generating a local semantic field for semantic decomposition based on the local orientation field, curvature distribution, and structural tensor features of the geometric candidate fragments, to distinguish the local stable regions of linear structures, circular arc structures, and symbol structures; automatically identifying geometric semantic boundaries in the local orientation field based on directional discontinuities, curvature transition points, and symbol shape inflection points, and decomposing the geometric candidate fragments into geometric semantic atoms, including line segment atoms, arc segment atoms, subscript atoms, arrow annotation atoms, and vertical parallel symbol atoms; performing geometric axiom consistency checks on the geometric semantic atoms, and back-mapping geometric semantic atoms that fail the consistency check and have a local confidence level higher than a preset threshold to the original fragments for secondary decomposition, the geometric axiom consistency checks including linear consistency, circular arc fitting consistency, and symbol geometric consistency checks; and outputting the checked geometric semantic atoms based on the check results.

4. The multimodal large model-driven geometric image reading and parsing method as described in claim 1, characterized in that, Performing adaptive preprocessing of the image data includes: performing feature extraction of the image data to establish a feature set, the feature set including brightness distribution features, noise texture features, tilt angle features, and local contrast features; using the feature set as input data to activate an image scene adaptive adjustment model and output basic correction adjustment parameters; and performing preprocessing adjustment of the image data according to the basic correction adjustment parameters to complete the adaptive preprocessing.

5. The multimodal large model-driven geometric image reading and parsing method as described in claim 4, characterized in that, The image data preprocessing adjustment based on the basic correction adjustment parameters further includes: performing geometric sensitivity enhancement processing on the preprocessing adjustment results, wherein the geometric sensitivity enhancement processing includes orientation consistency enhancement, local structure tensor enhancement, and fine line segment enhancement; performing local visibility analysis on the geometric sensitivity enhancement processing results to establish a local visibility analysis map, wherein the local visibility analysis includes edge sharpness, noise density, and occlusion probability analysis; and performing differential preprocessing based on the local visibility analysis map to complete adaptive preprocessing.

6. The multimodal large model-driven geometric image reading and parsing method as described in claim 1, characterized in that, The analysis results include key points for understanding the question, thought process analysis, and distributed derivation content. The analysis results are executed through user interaction via voice broadcast function, and the analysis results are output in an adaptive step-guided manner after the user activates the dynamic step unlocking mechanism.

7. The multimodal large model-driven geometric image reading and parsing method as described in claim 6, characterized in that, After a user submits an answer, a consistency check is performed between the submitted result and the parsed output. Results that fail the consistency check are recorded in the incorrect question set.

8. A multimodal large-model driven geometric image reading and parsing system, characterized in that, The system includes: an image preprocessing module, used to perform adaptive preprocessing of the image data after receiving image data uploaded by the user, and establish an adaptive preprocessing result; a geometric element extraction module, used to input the adaptive preprocessing result into a geometric element extraction channel, which constructs a geometric element candidate set by jointly extracting edge response, connectivity analysis, and shape decomposition, the geometric element candidate set including vertex candidates, line segment or arc segment candidates, subscript candidates, and labeled text candidates, and each candidate element is set with an initial confidence level identifier; a geometric label recognition module, used to start the geometric label recognition channel in parallel, extract the geometric element candidate set using a lightweight visual encoder, establish a visual embedding, and simultaneously activate an OCR semantic parsing channel to perform text recognition on the question stem and labeled text, and establish a text semantic embedding; a cross-modal alignment and fusion module, used to perform cross-modal alignment and fusion of the visual embedding and text semantic embedding under confidence weighting, and generate a structured geometric representation; a geometric representation parsing module, used to input the structured geometric representation into a hybrid reasoning recognition model, output the parsing result; establish a visual confidence matrix using the initial confidence level identifier; and perform OCR semantic parsing... The text confidence matrix is ​​constructed using OCR recognition confidence, contextual consistency score, and stem relevance in the channel. After normalization of the visual and text confidence matrices using a self-supervised confidence scheduler, bidirectional attention interaction verification is performed based on visual embedding and text semantic embedding to generate a structured geometric representation. Cross-modal attention query groups and key-value groups are generated using geometric topology fields from visual embedding and structural semantic units from text semantic embedding, and a geometric-aware mask is generated based on set adjacency relationships to restrict attention propagation within the allowed topological range. In the direction of mapping from visual embedding to text semantic embedding, geometric elements are used as query vectors, semantic labels as keys, and attention weights are corrected using geometric structure similarity, distance decay factor, and orientation field consistency factor to output a visually dominant attention map. In the direction of mapping from text semantic embedding to visual embedding, semantic labels are used as query vectors, candidate geometric elements as keys, and stem semantic dependency, semantic redundancy exclusion factor, and annotation citation relationship factor are added to the attention calculation to output a text-dominant attention map. Bidirectional consistency cross-metrics are performed on the visually dominant attention map and the text-dominant attention map to generate a structured geometric representation.

Images