A mathematical question automatic labeling method based on multi-modal image enhancement and semantic understanding

Abstract

The present application relates to the technical field of image processing and multi-modal semantic understanding, in particular to a kind of mathematical problem automatic labeling method based on multi-modal image enhancement and semantic understanding;The method comprises: receiving stem, solution, answer, analysis and comment picture, the white balance correction, contrast enhancement, symbol sharpening, tilt correction and denoising processing are carried out to picture;The enhanced picture is divided into pure text area, formula area and figure area and feature is extracted;Text extraction, LaTeX conversion and figure feature extraction are carried out to each area, and structured question information is formed according to spatial position relationship semantic splicing;According to subject meta information and subject number distribution rule, the subject is determined, matched with knowledge point system, generates knowledge point label and examination frequency statistics, is stored in database and forms knowledge base entry.The present application realizes the automatic identification, structured extraction and automatic labeling of mathematical problem.

Description

Technical Field

[0001] This invention relates to the fields of image processing and multimodal semantic understanding, specifically to an automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding. Background Technology

[0002] With the rapid development of artificial intelligence technology, smart education has become an important direction for educational informatization. In the field of higher mathematics education, the postgraduate entrance examination in mathematics involves three subjects: advanced mathematics, linear algebra, and probability theory and mathematical statistics. Over the years, a large number of past exam questions have been accumulated. These questions exist in paper or image form. Digitizing them and building a structured mathematical knowledge base is the foundation for realizing AI-assisted teaching, intelligent Q&A, and personalized learning. Currently, the digitization of mathematical questions involves optical character recognition technology, mathematical formula recognition technology, multimodal understanding technology, knowledge annotation and classification technology, and image enhancement technology. Among them, optical character recognition technology is used to extract text information from images, mathematical formula recognition technology is used to convert mathematical expressions from images into structured representations, multimodal understanding technology is used to process various modal information such as text descriptions, mathematical formulas, and geometric figures in mathematical questions, knowledge annotation and classification technology is used to automatically classify questions into the subject knowledge point system, and image enhancement technology is used to process mathematical question images from sources such as scanned documents, mobile phone photos, and screenshots. In the process of digitizing mathematical problems, there are still problems such as insufficient image recognition accuracy, fragmentation of multimodal information, and insufficient automatic annotation of knowledge points. Specifically, these problems are as follows: On the one hand, existing solutions directly recognize the original images without a dedicated enhancement preprocessing mechanism for mathematical problem images. They do not consider the extremely high requirements of mathematical formulas for clarity and contrast. When the problem images come from diverse sources and have uneven lighting, blurriness, tilt, or noise, the lack of targeted image enhancement methods leads to a significant decrease in subsequent recognition accuracy. On the other hand, existing solutions often treat text recognition and formula recognition as two independent processes, severing the semantic connections between the text, mathematical formulas, and geometric figures in the problem stem. This makes it impossible to achieve cross-modal joint understanding, which can easily lead to the loss of core conditions of the problem, failure to resolve ambiguities in the meaning of formula symbols, and inability to extract key information from the figures. At the same time, existing formula recognition solutions are unstable when faced with complex formula structures, and the recognition output lacks structured mathematical semantic annotation. Furthermore, existing solutions lack multi-level automatic annotation capabilities based on subject knowledge systems and are weak in the overall semantic splicing processing of multi-image problems, which can easily result in information duplication, omission, or misalignment. Summary of the Invention

[0003] The purpose of this invention is to provide an automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding, so as to solve the problems of insufficient accuracy in mathematical problem image recognition, fragmentation of multimodal information, and insufficient automatic annotation capability of knowledge points mentioned in the background art.

[0004] To achieve the above objectives, the technical solution of the present invention is: an automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding, comprising: S1. Receive images related to math problems uploaded by users; The images related to math problems include images of the problem statement, the solution process, the answer, the analysis, and the commentary. S2. Each image related to a math problem is preprocessed sequentially through the image enhancement module to obtain enhanced images related to the math problems. The preprocessing includes white balance correction, adaptive contrast enhancement, sharpening of mathematical symbol regions, tilt correction, and noise reduction. S3. The enhanced images related to the math problems are fed into the multimodal semantic understanding module. The image content is divided into plain text regions, formula regions, and graphic regions through the region segmentation network, and features are extracted from each region. S4, the recognition and structured output module performs targeted processing on the classified regions; for plain text regions, a text recognition engine is used to extract text; for formula regions, a mathematical formula recognition engine is used to convert them into LaTeX format; and for graphic regions, graphic description features and positional relationships are extracted. S5. Semantically concatenate the recognition results of each region according to the original spatial location relationship to form complete structured question information; S6. The automatic annotation module determines the subject based on the question meta-information and the distribution pattern of subject question numbers. Then, it performs semantic matching between the identified question content and the subject knowledge point system to generate knowledge point annotations and test frequency statistics. The question metadata includes the year, exam type, and question number; S7. Store all the above structured results into the database to form searchable knowledge base entries.

[0005] Preferably, in S2, the image enhancement module is used to restore and highlight the visual information of the mathematical problem, including an image quality assessment and classification submodule and a white balance and color normalization submodule; The image quality assessment and classification submodule is used to assess the quality of the input image, calculate the image's sharpness index, contrast index, tilt angle, and noise level, and classify the image into three levels: high quality, medium quality, and low quality based on the assessment results, adaptively selecting an enhancement strategy. The image sharpness index is the Laplacian variance. Specifically, the enhancement strategy is to directly process high-quality images, perform mild enhancement on medium-quality images, and perform heavy enhancement on low-quality images. The white balance and color normalization submodule is used to automatically correct the white balance of color photographed images using the gray world hypothesis algorithm, eliminate the influence of color cast, perform color space conversion, and convert the image into a grayscale image to provide a unified input format for subsequent processing.

[0006] Preferably, in S2, the image enhancement module further includes an adaptive contrast enhancement submodule, a mathematical symbol region sharpening submodule, a geometric correction submodule, and an adaptive denoising submodule; The adaptive contrast enhancement submodule employs an improved CLAHE algorithm. Targeting the high-frequency details of mathematical symbols, it divides the image into smaller, more granular local regions. Within each local region, contrast-limited histogram equalization is performed, and the contrast limiting threshold is adaptively adjusted based on the gray-level variance of the local region. Specifically, the improved CLAHE algorithm applies a weighted local enhancement strategy to minute symbols in mathematical formulas. This strategy applies enhancement to regions with gray-level gradients greater than a gray-level gradient threshold, which is a manually set threshold. The mathematical symbol region sharpening submodule is used to identify errors caused by blurred edges of mathematical symbols and employs a sharpening strategy. Specifically, the sharpening strategy involves extracting edge information from the image using an edge detection operator and then using an unsharpened mask technique with an adaptive radius Gaussian blur kernel for sharpening. The geometric correction submodule is used to solve the problems of tilt and perspective distortion caused by taking pictures. It uses a line detection algorithm based on Hough transform to detect document boundaries, and then corrects the image to a front view through a perspective transformation matrix. For screenshots of titles without clear boundaries, a method based on projection contour is used to detect the tilt angle of text lines and perform rotation correction. The adaptive denoising submodule employs a nonlocal mean denoising algorithm to remove image noise while preserving the sharpness of mathematical symbol edges.

[0007] Preferably, in step S3, the multimodal semantic understanding module includes a multimodal region segmentation submodule; The multimodal region segmentation submodule uses a deep learning-based semantic segmentation network to classify the enhanced question image pixel by pixel into four categories: plain text, mathematical formulas, geometric figures, and background. The deep learning-based semantic segmentation network adopts an improved U-Net++ architecture, which adds the depth of dense skip connections to the standard U-Net++ and introduces a spatial attention gating mechanism at the decoder end to locate the boundary between the formula region and the text region.

[0008] Preferably, the multimodal region segmentation submodule adopts a two-stage strategy of "coarse segmentation followed by fine-tuning", specifically: in the first stage, the candidate regions of the formula are identified using the first receptive field parameter; in the second stage, the candidate regions of the formula are classified at the pixel level using a second input resolution higher than the first receptive field parameter.

[0009] Preferably, in step S3, the multimodal semantic understanding module includes a multimodal feature extraction submodule; Specifically, the multimodal feature extraction submodule employs differentiated feature extraction strategies for deep feature encoding based on the different types of segmented regions, as follows: For plain text regions, a pre-trained Chinese OCR backbone network is used to extract character-level text feature sequences; For the formula region, a dedicated visual encoder for formula recognition is used to extract two-dimensional spatial features; the dedicated visual encoder for formula recognition is based on the Swin Transformer architecture. For graphic regions, a convolutional neural network is used to extract global geometric features and local structural features of the graphic, including the positional features of key points and the embedding encoding of geometric relationships.

[0010] Preferably, in step S3, the multimodal semantic understanding module further includes a cross-modal attention fusion submodule and a multi-image sequential semantic splicing submodule; The cross-modal attention fusion submodule uses a cross-modal cross-attention mechanism to exchange and align semantic information between text modality, formula modality, and graphics modality. Specifically, the cross-modal cross-attention mechanism is as follows: The feature representations of the three modalities are defined as text T, formula F, and image G, respectively. Three sets of cross-attention calculations are constructed as follows: T→F attention: Using text features as the query and formula features as the key and value, calculate the attention weight of the text to the formula; F→T attention: using formula features as queries and text features as keys and values, and using text context to resolve ambiguity of formula symbols; G→F and G→T attention: Using graphic features as queries, cross-attention calculations are performed with formula features and text features respectively to establish the correspondence between geometric elements in the graphic and variables in the formula and descriptions in the text. The multi-image sequential semantic splicing submodule is used to semantically splice the recognition results of multiple images in sequence for questions containing multiple images. It adopts a splicing strategy based on text overlap detection and semantic coherence analysis. Specifically, it first detects the text overlap areas at the edges of adjacent images. If there is overlap, it performs deduplication and merging. Then, it uses a pre-trained language model to evaluate the semantic coherence score of the spliced ​​text. If the score is lower than the threshold, it adjusts the splicing boundary.

[0011] Preferably, in step S4, the recognition and structured output module is based on the multimodal semantic understanding module and is used to convert the feature representations of each modality into structured text output, including a mathematical formula recognition engine; The mathematical formula recognition engine adopts an end-to-end encoder-decoder architecture to convert formula images into LaTeX code. The encoder extracts the two-dimensional visual features of the formula based on the Swing Transformer. The decoder uses a Transformer decoder to generate LaTeX code sequences token by token in an autoregressive manner. The design of the decoder includes: introducing a position-aware two-dimensional attention bias: in standard self-attention computation, the two-dimensional spatial coordinates of the formula symbols are encoded as attention bias terms, so that the model can consider the spatial positional relationship of the symbols when generating LaTeX code; A bundle search decoding strategy based on grammatical constraints is adopted: During the decoding phase, a LaTeX grammatical state machine is maintained to detect the grammatical correctness of the currently generated sequence in real time; during the bundle search process, candidate sequences that violate grammatical constraints are directly pruned to ensure that the output LaTeX code is grammatically correct. A hierarchical identification strategy is designed for complex structures: for the detected complex formula structure, the structure type is first identified at the coarse-grained level, and then the content is filled in at the fine-grained level.

[0012] Preferably, in step S4, the recognition and structured output module further includes a text recognition submodule, a structure fusion submodule, and a structured annotation submodule; The text recognition submodule adopts a hybrid text recognition architecture that combines sequential classification and attention mechanism to extract text from plain text regions. The structure fusion submodule is used to arrange and combine the identified text fragments and formula fragments according to their spatial position relationship in the original image. A fusion algorithm based on spatial layout tree is adopted: first, a spatial layout tree is constructed according to the position coordinates of each region, and an inorder traversal is performed on the tree to obtain an ordered content sequence. The root node of the spatial layout tree is the entire page, the child nodes are each line, and the leaf nodes are each text block and formula block. The structured annotation submodule performs structured annotation on the fused content and outputs structured data containing the following fields: question description, solution process, answer, and analysis text.

[0013] Preferably, in step S6, the automatic annotation module uses the question meta-information and content recognition results to automatically complete the annotation and classification of knowledge points, and generate structured knowledge entries that can be directly entered into the database, including a rule-based subject rapid determination submodule, a semantic knowledge point classification submodule, an exam frequency statistics submodule, and a teaching manuscript generation submodule. The rule-based rapid subject determination submodule is used to determine the subject to which a question belongs by utilizing a unified question-setting pattern; The semantic-based knowledge point classification submodule, after determining the subject, uses the knowledge point system of the subject as a candidate label set, and uses the semantic features of the question extracted by the multimodal semantic understanding module to calculate the semantic similarity between the question and each knowledge point; it adopts a text embedding method based on a pre-trained language model to encode the question description and solution process into semantic vectors, calculates the cosine similarity between the semantic vectors of each knowledge point and the semantic vectors of each knowledge point, and selects the knowledge point with the highest similarity as the labeling result. The frequency statistics submodule, based on the knowledge point annotation results, counts the frequency of each knowledge point in past years' exam papers, and generates statistical information including the frequency distribution over the past N years, the total number of occurrences, and the examination trend, which is then deposited into the exam frequency metadata of the knowledge base. The teaching script generation submodule automatically generates teaching script content tailored to teaching scenarios based on structured recognition results and knowledge point annotation results using a large language model.

[0014] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects: 1. In this invention, a dedicated image enhancement pipeline for mathematical formulas can effectively address the image processing needs of mathematical problems in various quality degradation scenarios such as scanned documents, mobile phone photos, and screenshots. By setting sharpening, noise reduction, and contrast enhancement strategies for the high-frequency detail characteristics of mathematical symbols, and by performing adaptive detection and targeted enhancement on the formula area, the recognition accuracy of subsequent text, formula, and graphic areas is improved. 2. In this invention, the multimodal fusion understanding framework based on text, formulas and graphics can overcome the semantic fragmentation problem caused by the independent processing of different modal information in the prior art; by fusing the semantic features of text descriptions, mathematical formulas and geometric figures through a multimodal attention mechanism, collaborative extraction and semantic alignment of cross-modal information can be achieved, thereby obtaining a complete semantic understanding result of the question. 3. In this invention, the process of converting a mathematical problem from its original image to a structured knowledge base is broken down into a well-defined and controllable multi-stage pipeline through a hierarchical recognition and structured output mechanism; this allows the output of each stage to be checked and corrected, ensuring the correctness and structure of the final LaTeX output. 4. In this invention, the automatic annotation function based on the subject knowledge system can reduce the time consumption and inconsistent standards caused by manual annotation of each question; by utilizing the correspondence between the question meta-information and the distribution pattern of subject question numbers, it can achieve rapid subject determination, and combine the recognition results to perform refined knowledge point classification and generate knowledge point annotation results. 5. In this invention, through intelligent stitching of multiple images and overall semantic understanding, cross-image semantic connection between question stem images, solution process images, answer images, analysis images, and commentary images can be processed; continuous extraction of complete question information can be achieved, reducing information duplication, omission, or misalignment problems in the process of processing multiple images of questions. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the overall technical architecture and processing flow of the present invention. Figure 2 Flowchart for image enhancement module processing; Figure 3 Architecture diagram of the multimodal semantic understanding module; Figure 4 This is a schematic diagram of a cross-modal attention fusion mechanism; Figure 5 Flowchart for identifying and structuring output modules; Figure 6 This is a flowchart of the automatic annotation module. Detailed Implementation

[0016] like Figure 1 As shown, the present invention proposes an automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding. The specific implementation steps are as follows: S1. Receive images related to math problems uploaded by users; The images related to math problems include images of the problem stem, the solution process, the answer, the analysis, and the commentary. Each type of image can contain multiple images. S2. Each image related to a math problem is preprocessed sequentially through the image enhancement module to obtain enhanced images related to the math problems. The preprocessing includes white balance correction, adaptive contrast enhancement, sharpening of mathematical symbol regions, tilt correction, and noise reduction. S3. The enhanced images related to the math problems are fed into the multimodal semantic understanding module. The image content is divided into plain text regions, formula regions, and graphic regions through the region segmentation network, and features are extracted from each region. S4, the recognition and structured output module performs targeted processing on the classified regions; the plain text region uses a text recognition engine to extract text; the formula region uses a mathematical formula recognition engine to convert it into LaTeX format; and the graphic region extracts graphic description features and positional relationships. S5. Semantically concatenate the recognition results of each region according to the original spatial location relationship to form complete structured question information; S6. The automatic annotation module determines the subject based on the question meta-information and the distribution pattern of subject question numbers. Then, it performs semantic matching between the identified question content and the subject knowledge point system to generate knowledge point annotations and test frequency statistics. The question metadata includes the year, exam type, and question number; S7. Store all the above structured results into the database to form searchable knowledge base entries.

[0017] In this embodiment S2, the image enhancement module is used to restore and highlight the key visual information of the math problem to the greatest extent possible under the condition of non-ideal image in the actual scene. It includes an image quality assessment and classification submodule and a white balance and color normalization submodule. The image quality assessment and classification submodule is used to assess the quality of the input image, calculate the image's sharpness index, contrast index, tilt angle, and noise level, and classify the image into three levels: high quality, medium quality, and low quality based on the assessment results, adaptively selecting an enhancement strategy. The image sharpness index is the Laplacian variance. Specifically, the enhancement strategy is to directly process high-quality images, perform mild enhancement on medium-quality images, and perform heavy enhancement on low-quality images. The white balance and color normalization submodule is used to automatically correct the white balance of color photographed images using the gray world hypothesis algorithm, eliminate the influence of color cast, perform color space conversion, and convert the image into a grayscale image to provide a unified input format for subsequent processing.

[0018] In this embodiment S2, the image enhancement module further includes an adaptive contrast enhancement submodule, a mathematical symbol region sharpening submodule, a geometric correction submodule, and an adaptive denoising submodule; The adaptive contrast enhancement submodule employs an improved CLAHE (Contrast-Limited Adaptive Histogram Equalization) algorithm. Targeting the high-frequency details of mathematical symbols, it divides the image into smaller, localized regions (8×8 pixel blocks). Contrast-limited histogram equalization is performed within each local region, and the contrast limiting threshold is adaptively adjusted based on the gray-level variance of the local region. Specifically, for common minute symbols in mathematical formulas (such as subscripts, superscripts, and expressions within square roots), this module uses a weighted local enhancement strategy—applying greater enhancement to regions with larger gray-level gradients (i.e., regions that may contain mathematical symbols; a larger gray-level gradient refers to regions with a gray-level gradient greater than the gray-level gradient threshold). The CLAHE algorithm is a contrast-limited adaptive histogram equalization algorithm; the gray-level gradient threshold is a manually set threshold. The mathematical symbol region sharpening submodule is used to identify errors caused by blurred edges of mathematical symbols, employing a sharpening strategy. Specifically, this strategy involves extracting edge information from the image using an edge detection operator; then employing unsharp masking technology with an adaptive radius Gaussian blur kernel for sharpening. The use of the adaptive radius Gaussian blur kernel specifically involves using a smaller Gaussian radius (σ=1.0) for high-gradient regions to preserve fine details, and a larger Gaussian radius (σ=2.5) for low-gradient regions (corresponding to text or blank areas) to avoid over-sharpening and introducing noise. The formula for calculating the sharpening intensity is: ; ; in, For adaptive parameters, Use an empirical value of 1.2-1.8; Among them, the edge detection operator is an improved Canny operator with a lower lower threshold to adapt to weak edges; high gradient regions are formula-dense areas; low gradient regions are text or blank areas. The geometric correction submodule is used to solve the problems of tilt and perspective distortion caused by taking pictures. It uses a line detection algorithm based on Hough transform to detect document boundaries, and then corrects the image to a front view through a perspective transformation matrix. For screenshots of titles without clear boundaries, a method based on projection contour is used to detect the tilt angle of text lines and perform rotation correction. The adaptive denoising submodule employs a non-local means denoising algorithm to remove image noise while preserving the sharpness of mathematical symbol edges. The denoising intensity parameter h is adaptively set based on the image noise level estimation result. A smaller h value is used for low-noise images to preserve details, while a larger h value is used for high-noise images to effectively reduce noise.

[0019] In this embodiment S2, the above-mentioned image enhancement sub-modules do not all have to be executed in sequence. Instead, the sub-modules to be executed and their parameters are adaptively selected according to the image quality assessment results, thereby minimizing the processing time while ensuring the enhancement effect.

[0020] In this embodiment S2, the image enhancement module can also use three other alternative schemes: Image enhancement alternative scheme 1: Use an end-to-end image enhancement network based on deep learning (such as MIRNet, U-Net enhancement network, etc.) to replace the traditional CLAHE and unsharpened mask method; the advantage of this alternative scheme is that it can automatically learn the optimal enhancement strategy, but the disadvantage is that the computational overhead is large, which may affect the overall processing speed; Image enhancement alternative scheme 2: Use a multi-frame fusion enhancement strategy. For multiple images of the same question (such as photos taken from different angles), use the complementary information between multiple frames to perform super-resolution fusion, and obtain higher quality enhancement results through image registration and pixel-level fusion; Image enhancement alternative scheme 3: Replace the white balance correction algorithm. The perfect reflection method or the learning-based color constancy method can be used to replace the gray world hypothesis algorithm.

[0021] In this embodiment S3, the multimodal semantic understanding module breaks through the limitation of existing technologies that process text, formulas and graphics independently, and realizes collaborative understanding and semantic fusion of multimodal information.

[0022] In this embodiment S3, the multimodal semantic understanding module includes a multimodal region segmentation submodule; The multimodal region segmentation submodule employs a deep learning-based semantic segmentation network to classify the enhanced question image pixel-by-pixel into four categories: plain text, mathematical formulas, geometric figures, and background. The deep learning-based semantic segmentation network uses an improved U-Net++ architecture, specifically by adding depth to dense skip connections on top of the standard U-Net++ and introducing a spatial attention gating mechanism at the decoder end to locate the boundary between formula regions (typically containing small symbols and dense structures) and text regions. The training data for the deep learning-based semantic segmentation network comes from a manually annotated dataset of postgraduate entrance examination math questions, covering different years and types of questions. Specifically, the multimodal region segmentation submodule adopts a two-stage strategy of "coarse segmentation followed by fine-tuning": the first stage uses a first receptive field parameter to identify candidate formula regions; the second stage uses a second input resolution higher than the first receptive field parameter to perform pixel-level classification of the candidate formula regions.

[0023] In this embodiment S3, the multimodal semantic understanding module includes a multimodal feature extraction submodule; Specifically, the multimodal feature extraction submodule employs differentiated feature extraction strategies for deep feature encoding based on the different types of segmented regions, as follows: For plain text regions, a pre-trained Chinese OCR backbone network (such as DBNet+CRNN architecture) is used to extract character-level text feature sequences. In particular, since math problems contain a large number of mathematical symbols and terms, a domain-adaptive fine-tuning is performed using mathematical domain corpus on the basis of a general Chinese OCR pre-trained model to enhance the recognition ability of mathematical professional words and symbols (such as "differential", "integral", "determinant" etc.). For the formula region, a dedicated visual encoder for formula recognition is used to extract two-dimensional spatial features. The dedicated visual encoder for formula recognition is based on the Swin Transformer architecture. Compared with traditional CNN feature extraction methods, the self-attention mechanism of Swin Transformer can better capture long-distance two-dimensional spatial dependencies between formula symbols (such as the vertical positional relationship between the numerator and denominator of a fraction, the spatial relationship between the summation symbol and the upper and lower limits, etc.). The feature map output by the encoder retains the spatial structure information of the original image, with a size of H / 16 × W / 16 × C, where H and W are the height and width of the input image, and C is the number of feature channels. For graphic regions, convolutional neural networks are used to extract global geometric features and local structural features of the graphics, including the positional features of key points (such as the vertices of triangles and the intersections of auxiliary lines) and the embedding encoding of geometric relationships (parallel, perpendicular, tangent, etc.).

[0024] In this embodiment S3, the multimodal semantic understanding module further includes a cross-modal attention fusion submodule and a multi-image sequential semantic splicing submodule; The cross-modal attention fusion submodule uses a cross-modal cross-attention mechanism to exchange and align semantic information between text modality, formula modality, and graphics modality. Specifically, the cross-modal cross-attention mechanism is as follows: The feature representations of the three modalities are defined as text T, formula F, and image G, respectively. Three sets of cross-attention calculations are constructed as follows: T→F attention: Using text features as queries and formula features as keys and values, the attention weights of the text to the formula are calculated; this allows the model to understand the relationship between mathematical concepts mentioned in the text (such as "quadratic form" and "eigenvalue") and their corresponding expressions in the formula; F→T attention: Using formula features as queries and text features as keys and values, it uses textual context to resolve ambiguities in formula symbols; for example, the same symbol "λ" may represent feature values, Lagrange multipliers, or Poisson distribution parameters in different questions, and textual context can help determine its exact meaning. G→F and G→T attention: Using graphic features as queries, cross-attention calculations are performed with formula features and text features respectively to establish the correspondence between geometric elements in the graphic and variables in the formula and descriptions in the text. The fused multimodal features are adaptively weighted and combined through a lightweight gated fusion layer to output a unified semantic feature representation; The formula for calculating the gating weight is: ; in, and These are learnable parameters; The multi-image sequential semantic splicing submodule is used to semantically splice the recognition results of multiple images in sequence for questions containing multiple images (e.g., the question stem contains multiple accompanying images, and the solution process spans multiple images). It adopts a splicing strategy based on text overlap detection and semantic coherence analysis. Specifically, it first detects the text overlap areas at the edges of adjacent images, and if overlap exists, it performs deduplication and merging. Then, it uses a pre-trained language model to evaluate the semantic coherence score of the spliced ​​text. If the score is lower than the threshold, it adjusts the splicing boundary to ensure that the final output question text is continuous, complete, and without repetition.

[0025] In this embodiment S3, the multimodal semantic understanding module can also use three other alternative schemes: Scheme 1: Using a Transformer-based unified multimodal encoder (such as the ViT architecture) instead of the region segmentation + classification processing scheme; inputting all modal image regions into the same Transformer encoder, and distinguishing different region types through learnable modality embedding to achieve closer multimodal feature fusion; Scheme 2: Replacing the cross-attention mechanism, graph neural networks (GNNs) can be used for multimodal feature fusion; text fragments, formula blocks, and graphic elements are used as nodes in the graph, edges are constructed based on spatial adjacency and semantic association, and collaborative reasoning of multimodal information is achieved through message passing mechanism; Scheme 3: Replacing the region segmentation network, formula region detection methods based on object detection (such as YOLO, DETR) can be used instead of pixel-level semantic segmentation, outputting the bounding box of the formula region instead of pixel-by-pixel classification, which has higher computational efficiency but may have slightly lower boundary accuracy.

[0026] In this embodiment S4, the recognition and structured output module is based on the multimodal semantic understanding module and is used to convert the feature representations of each modality into structured text output, including a mathematical formula recognition engine; The mathematical formula recognition engine adopts an end-to-end encoder-decoder architecture to convert formula images into LaTeX code. The encoder extracts the two-dimensional visual features of the formula based on the Swing Transformer. The decoder uses a Transformer decoder to generate LaTeX code sequences token by token in an autoregressive manner. The design of the decoder includes: introducing a position-aware two-dimensional attention bias: in standard self-attention computation, the two-dimensional spatial coordinates of the formula symbols are encoded as attention bias terms, so that the model can consider the spatial positional relationship of the symbols when generating LaTeX code; The formula for calculating attention bias is: ; in, and Let be the horizontal and vertical offsets between the image regions corresponding to token i and token j. For scale ratio; A bundle search decoding strategy based on grammatical constraints is adopted: During the decoding phase, a LaTeX grammar state machine is maintained to detect the grammatical correctness of the currently generated sequence in real time (including curly brace matching, context switch matching, etc.); during the bundle search process, candidate sequences that violate grammatical constraints are directly pruned to ensure that the output LaTeX code is grammatically correct; this strategy significantly reduces common errors such as mismatched curly braces and unclosed contexts in the model output. A hierarchical recognition strategy is designed for complex structures: For detected complex formula structures (such as nested fractions, multi-row matrices, and piecewise functions), the structure type is first identified at a coarse-grained level, and then the content is filled in at a fine-grained level; for example, for matrices, the matrix type (square bracket matrix, determinant matrix, etc.) and the number of rows and columns are identified first, and then the content of each element is identified in turn. This hierarchical strategy significantly improves the recognition accuracy of complex structures.

[0027] In this embodiment S4, the recognition and structured output module further includes a text recognition submodule, a structure fusion submodule, and a structured annotation submodule; The text recognition submodule employs a hybrid text recognition architecture that combines Connectionist Temporal Classification (CTC) with an attention mechanism to extract text from plain text regions. The recognition results retain the original line breaks and spaces to maintain the readability of the questions. The structure fusion submodule is used to arrange and combine the identified text fragments and formula fragments according to their spatial position relationship in the original image. A fusion algorithm based on spatial layout tree is adopted: first, a spatial layout tree is constructed according to the position coordinates of each region, and an inorder traversal is performed on the tree to obtain an ordered content sequence. The root node of the spatial layout tree is the entire page, the child nodes are each line, and the leaf nodes are each text block and formula block. At the junction of text and formula, it is determined whether to add or delete spaces based on semantic fluency so that the final output conforms to the typesetting specifications of mathematical problems. The structured annotation submodule performs structured annotation on the fused content and outputs structured data containing the following fields: question description (LaTeX format text), solution process (LaTeX format text), answer (LaTeX format text or category annotation), and analysis text. The question description, solution process, answer, and analysis text all retain the corresponding original image references, supporting subsequent manual verification. In this embodiment S6, the automatic annotation module uses the question meta-information and content recognition results to automatically complete the annotation and classification of knowledge points, and generate structured knowledge entries that can be directly entered into the database, including a rule-based subject rapid determination submodule, a semantic knowledge point classification submodule, an exam frequency statistics submodule, and a teaching manuscript generation submodule. The rule-based rapid subject determination submodule is used to utilize the unified question-setting rules of postgraduate mathematics examinations—given the year, exam paper type (Mathematics I / II / III / IV) and question number—to uniquely determine the subject to which the question belongs (Advanced Mathematics / Linear Algebra / Probability Theory and Mathematical Statistics); the rule-based rapid subject determination submodule has a built-in complete subject question number distribution rule table, covering all exam paper types from 1987 to the current year, and can complete subject determination in O(1) time without calling an AI model; The semantic-based knowledge point classification submodule, after determining the subject, uses the subject's knowledge point system as a candidate label set. It then uses the semantic features extracted by the multimodal semantic understanding module to calculate the semantic similarity between the question and each knowledge point. A text embedding method based on a pre-trained language model is used to encode the question description and solution process into semantic vectors. These vectors are then compared with the semantic vectors of each knowledge point using cosine similarity calculations. The knowledge point with the highest similarity is selected as the labeling result (the number of knowledge points K is determined by the knowledge point coverage strategy, typically 2-5). To improve classification accuracy, this submodule employs the following optimization strategies: a) constructing a subject-specific knowledge point vector index and using a vector database for efficient retrieval; b) introducing a contrastive learning training strategy to make questions of the same type of knowledge point more clustered in the semantic space and questions of different types more dispersed; c) using information about the problem-solving methods involved in the question-solving process as auxiliary features, because problem-solving methods are usually strongly correlated with knowledge points (e.g., questions using the "substitution method" are likely to belong to indefinite integral knowledge points). The frequency statistics submodule, based on the knowledge point annotation results, statistically analyzes the frequency of each knowledge point in past exam papers, generating statistical information including the frequency distribution over the past N years, the total number of occurrences, and the examination trend, which is then stored as the frequency metadata of the knowledge base; where N years are determined manually. The teaching script generation submodule automatically generates teaching script content tailored to teaching scenarios based on structured recognition results and knowledge point annotation results using a large language model. The teaching script content specifically includes: a) a problem-solving process script—converting mathematical formulas and solution steps into spoken language, removing mathematical symbols, and describing all mathematical operations in Chinese; b) a typical error analysis script—reiterating common errors recorded in the analysis in a conversational manner; c) a summary script—providing a concise, conversational summary of the examination authority's analysis. The generated scripts can be directly used in text-to-speech (TTS) systems, adapting to digital human teacher scenarios. The teaching script generation submodule is a submodule that users can flexibly select whether they need.

[0028] In this embodiment S6, the automatic annotation module can also have two other schemes: Automatic annotation alternative scheme 1, adopting a knowledge point classification scheme based on knowledge graph reasoning, constructing a hierarchical relationship and association graph between mathematical knowledge points, using graph reasoning algorithms to perform multi-hop knowledge point inference, and improving the classification accuracy of fine-grained knowledge points; Automatic annotation alternative scheme 2, replacing the two-stage annotation method, can adopt an end-to-end multi-task learning scheme, simultaneously completing formula recognition and knowledge point classification in a unified model, and achieving joint optimization by sharing underlying features.

[0029] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. An automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding, characterized in that, Includes the following steps: S1. Receive images related to math problems uploaded by users; The images related to math problems include images of the problem statement, the solution process, the answer, the analysis, and the commentary. S2. Each image related to a math problem is preprocessed sequentially through the image enhancement module to obtain enhanced images related to the math problems. The preprocessing includes white balance correction, adaptive contrast enhancement, sharpening of mathematical symbol regions, tilt correction, and noise reduction. S3. The enhanced images related to the math problems are fed into the multimodal semantic understanding module. The image content is divided into plain text regions, formula regions, and graphic regions through the region segmentation network, and features are extracted from each region. S4, the recognition and structured output module performs targeted processing on the classified regions; the plain text region uses a text recognition engine to extract text; the formula region uses a mathematical formula recognition engine to convert it into LaTeX format; and the graphic region extracts graphic description features and positional relationships. S5. Semantically concatenate the recognition results of each region according to the original spatial location relationship to form complete structured question information; S6. The automatic annotation module determines the subject based on the question meta-information and the distribution pattern of subject question numbers. Then, it performs semantic matching between the identified question content and the subject knowledge point system to generate knowledge point annotations and test frequency statistics. The question metadata includes the year, exam type, and question number; S7. Store all the above structured results into the database to form searchable knowledge base entries.

2. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 1, characterized in that: In S2, the image enhancement module is used to restore and highlight the visual information of the math problem, including an image quality assessment and classification submodule and a white balance and color normalization submodule. The image quality assessment and classification submodule is used to assess the quality of the input image, calculate the image's sharpness index, contrast index, tilt angle, and noise level, and classify the image into three levels: high quality, medium quality, and low quality based on the assessment results, adaptively selecting an enhancement strategy. The image sharpness index is the Laplacian variance. Specifically, the enhancement strategy is to directly process high-quality images, perform mild enhancement on medium-quality images, and perform heavy enhancement on low-quality images. The white balance and color normalization submodule is used to automatically correct the white balance of color photographed images using the gray world hypothesis algorithm, eliminate the influence of color cast, perform color space conversion, and convert the image into a grayscale image to provide a unified input format for subsequent processing.

3. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 2, characterized in that: In S2, the image enhancement module further includes an adaptive contrast enhancement submodule, a mathematical symbol region sharpening submodule, a geometric correction submodule, and an adaptive denoising submodule; The adaptive contrast enhancement submodule employs an improved CLAHE algorithm. Targeting the high-frequency details of mathematical symbols, it divides the image into smaller, more granular local regions. Within each local region, contrast-limited histogram equalization is performed, and the contrast limiting threshold is adaptively adjusted based on the gray-level variance of the local region. Specifically, the improved CLAHE algorithm applies a weighted local enhancement strategy to minute symbols in mathematical formulas. This strategy applies enhancement to regions with gray-level gradients greater than a gray-level gradient threshold, which is a manually set threshold. The mathematical symbol region sharpening submodule is used to identify errors caused by blurred edges of mathematical symbols and employs a sharpening strategy. Specifically, the sharpening strategy involves extracting edge information from the image using an edge detection operator and then using an unsharpened mask technique with an adaptive radius Gaussian blur kernel for sharpening. The geometric correction submodule is used to solve the problems of tilt and perspective distortion caused by taking pictures. It uses a line detection algorithm based on Hough transform to detect document boundaries, and then corrects the image to a front view through a perspective transformation matrix. For screenshots of titles without clear boundaries, a method based on projection contour is used to detect the tilt angle of text lines and perform rotation correction. The adaptive denoising submodule employs a nonlocal mean denoising algorithm to remove image noise while preserving the sharpness of mathematical symbol edges.

4. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 3, characterized in that: In S3, the multimodal semantic understanding module includes a multimodal region segmentation submodule; The multimodal region segmentation submodule uses a deep learning-based semantic segmentation network to classify the enhanced question image pixel by pixel into four categories: plain text, mathematical formulas, geometric figures, and background. The deep learning-based semantic segmentation network adopts an improved U-Net++ architecture, which adds the depth of dense skip connections to the standard U-Net++ and introduces a spatial attention gating mechanism at the decoder end to locate the boundary between the formula region and the text region.

5. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 4, characterized in that: The multimodal region segmentation submodule adopts a two-stage strategy of "coarse segmentation followed by fine-tuning". Specifically, in the first stage, the first receptive field parameter is used to identify the candidate region of the formula. In the second stage, the candidate regions are classified at the pixel level using a second input resolution higher than the first receptive field parameter.

6. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 5, characterized in that: In S3, the multimodal semantic understanding module includes a multimodal feature extraction submodule; Specifically, the multimodal feature extraction submodule employs differentiated feature extraction strategies for deep feature encoding based on the different types of segmented regions, as follows: For plain text regions, a pre-trained Chinese OCR backbone network is used to extract character-level text feature sequences; For the formula region, a dedicated visual encoder for formula recognition is used to extract two-dimensional spatial features; the dedicated visual encoder for formula recognition is based on the Swin Transformer architecture. For graphic regions, a convolutional neural network is used to extract global geometric features and local structural features of the graphic, including the positional features of key points and the embedding encoding of geometric relationships.

7. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 6, characterized in that: In S3, the multimodal semantic understanding module further includes a cross-modal attention fusion submodule and a multi-image sequential semantic splicing submodule; The cross-modal attention fusion submodule uses a cross-modal cross-attention mechanism to exchange and align semantic information between text modality, formula modality, and graphics modality. Specifically, the cross-modal cross-attention mechanism is as follows: The feature representations of the three modalities are defined as text T, formula F, and image G, respectively. Three sets of cross-attention calculations are constructed as follows: T→F attention: Using text features as the query and formula features as the key and value, calculate the attention weight of the text to the formula; F→T attention: using formula features as queries and text features as keys and values, and using text context to resolve ambiguity of formula symbols; G→F and G→T attention: Using graphic features as queries, cross-attention calculations are performed with formula features and text features respectively to establish the correspondence between geometric elements in the graphic and variables in the formula and descriptions in the text. The multi-image sequential semantic splicing submodule is used to semantically splice the recognition results of multiple images in sequence for questions containing multiple images. It adopts a splicing strategy based on text overlap detection and semantic coherence analysis. Specifically, it first detects the text overlap areas at the edges of adjacent images. If there is overlap, it performs deduplication and merging. Then, it uses a pre-trained language model to evaluate the semantic coherence score of the spliced ​​text. If the score is lower than the threshold, it adjusts the splicing boundary.

8. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 7, characterized in that: In S4, the recognition and structured output module is based on the multimodal semantic understanding module and is used to convert the feature representations of each modality into structured text output, including a mathematical formula recognition engine; The mathematical formula recognition engine adopts an end-to-end encoder-decoder architecture to convert formula images into LaTeX code. The encoder extracts the two-dimensional visual features of the formula based on the Swing Transformer. The decoder uses a Transformer decoder to generate LaTeX code sequences token by token in an autoregressive manner. The design of the decoder includes: introducing a position-aware two-dimensional attention bias: in standard self-attention computation, the two-dimensional spatial coordinates of the formula symbols are encoded as attention bias terms, so that the model can consider the spatial positional relationship of the symbols when generating LaTeX code; A bundle search decoding strategy based on grammatical constraints is adopted: During the decoding phase, a LaTeX grammatical state machine is maintained to detect the grammatical correctness of the currently generated sequence in real time; during the bundle search process, candidate sequences that violate grammatical constraints are directly pruned to ensure that the output LaTeX code is grammatically correct. A hierarchical identification strategy is designed for complex structures: for the detected complex formula structure, the structure type is first identified at the coarse-grained level, and then the content is filled in at the fine-grained level.

9. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 8, characterized in that: In S4, the recognition and structured output module further includes a text recognition submodule, a structure fusion submodule, and a structured annotation submodule; The text recognition submodule adopts a hybrid text recognition architecture that combines sequential classification and attention mechanism to extract text from plain text regions. The structure fusion submodule is used to arrange and combine the identified text fragments and formula fragments according to their spatial position relationship in the original image. A fusion algorithm based on spatial layout tree is adopted: first, a spatial layout tree is constructed according to the position coordinates of each region, and an inorder traversal is performed on the tree to obtain an ordered content sequence. The root node of the spatial layout tree is the entire page, the child nodes are each line, and the leaf nodes are each text block and formula block. The structured annotation submodule performs structured annotation on the fused content and outputs structured data containing the following fields: question description, solution process, answer, and analysis text.

10. The automatic annotation method for mathematical problems based on multimodal image enhancement and semantic understanding according to claim 9, characterized in that: In S6, the automatic annotation module uses the question meta-information and content recognition results to automatically complete the annotation and classification of knowledge points, and generate structured knowledge entries that can be directly entered into the database. This includes a rule-based subject rapid determination submodule, a semantic knowledge point classification submodule, an exam frequency statistics submodule, and a teaching manuscript generation submodule. The rule-based rapid subject determination submodule is used to determine the subject to which a question belongs by utilizing a unified question-setting pattern; The semantic-based knowledge point classification submodule, after determining the subject, uses the knowledge point system of the subject as a candidate label set, and uses the semantic features of the question extracted by the multimodal semantic understanding module to calculate the semantic similarity between the question and each knowledge point; it adopts a text embedding method based on a pre-trained language model to encode the question description and solution process into semantic vectors, calculates the cosine similarity between the semantic vectors of each knowledge point and the semantic vectors of each knowledge point, and selects the knowledge point with the highest similarity as the labeling result. The frequency statistics submodule, based on the knowledge point annotation results, counts the frequency of each knowledge point in past years' exam papers, and generates statistical information including the frequency distribution over the past N years, the total number of occurrences, and the examination trend, which is then deposited into the exam frequency metadata of the knowledge base. The teaching script generation submodule automatically generates teaching script content tailored to teaching scenarios based on structured recognition results and knowledge point annotation results using a large language model.

Images