A smart education test question attached figure automatic matching method, a storage medium and an equipment

Abstract

This invention discloses a method, storage medium, and device for automatically matching illustrations in smart education test questions. By integrating image layout analysis, semantic reference recognition, and multi-dimensional spatial association reasoning mechanisms, it achieves accurate automatic matching between the question stem and the illustrations. It can not only identify linguistic clues such as "as shown in the figure" and "see Figure X", but also make comprehensive judgments based on the question number order, the relative position of the text and images, and the arrangement pattern. It can still maintain a high accuracy rate when dealing with integrated illustration layouts.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to a method, storage medium, and device for automatically matching diagrams of intelligent education test questions. Background Technology

[0002] In the process of digitizing test questions in the field of smart education, it is usually necessary to import the entire page of paper test papers or electronic images, and then use image recognition technology to break down and input the questions one by one. Existing methods mostly rely on OCR (Optical Character Recognition) technology to extract text information and combine it with simple image region segmentation to obtain attached images. However, they have significant shortcomings in automatically matching questions with attached images. Especially in teaching materials or exam papers, in order to save layout space, there are often complex layouts such as multiple questions sharing a row of attached images, images being arranged in a concentrated manner, and captions being shared. Traditional methods only match the nearest image based on the physical distance between the image and the text, lacking an understanding of the semantic reference relationship of the question stem and the overall page logical structure. This leads to frequent errors in binding attached images with the question stem, seriously affecting the accuracy of the test data and its subsequent use. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention aims to provide a method, storage medium, and device for automatically matching diagrams for smart education test questions.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: The method for automatically matching images to test questions in smart education includes the following steps: S1. Receive a full-page test paper image containing multiple questions, and preprocess the full-page test paper image to obtain a clear binarized image; S2. Use a deep learning model to identify all text regions and image regions in the binarized image, and extract the position coordinates and size information of each text region and image region respectively; S3. Use layout analysis algorithms to identify the spatial layout relationship of each region and divide the text region into question units. S4. For the question stem text in each question unit, check whether it contains reference text pointing to the attached figure; S5. For the title text containing reference text pointing to the attached figure, construct a spatial association model between the location of the title text and all unassigned image regions: using the position coordinates and size information of the title text of the title unit and each image region, calculate the scores of four dimensions between the title unit and the unassigned image regions: horizontal alignment, vertical proximity, arrangement order consistency, and region enclosure relationship, and calculate the comprehensive matching score based on the scores of the four dimensions. S6. After calculating the comprehensive matching score between each question stem text containing references to the attached figures and each image region, perform the attached figure matching operation sequentially for each question stem text containing references to the attached figures according to the question number order: First, semantic analysis is performed on the reference text of the current question text to the attached figures to determine the number of attached figures referenced by the current question text as J. Then, J matching operations are performed on the current question text. In each matching operation, the image region with the highest comprehensive matching score between the current unassigned image region and the current question text is matched to the current question text, and this image region is marked as assigned. This process continues until the J matching operations of the current question text are completed, and then the process of matching attached figures for the next question text begins. S7. Output the structured data package of each question text and its corresponding image region to complete the automatic recognition and binding of questions with images.

[0005] Furthermore, in step S1, the preprocessing includes denoising, binarization, and tilt correction.

[0006] Furthermore, in step S2, the deep learning model adopts the Mask R-CNN model, which is trained on an educational document image dataset and can accurately distinguish text, illustrations, tables, formula images and decorative borders in the image.

[0007] Furthermore, in step S5, the specific calculation process for the horizontal alignment score, vertical proximity score, arrangement order consistency score, and region enclosure relationship score is as follows: (1) Horizontal alignment score: Let Δy be the minimum vertical distance between the bounding box of the question text and the bounding box of the image region on the Y-axis, and let h be the standard line height of the exam paper. Calculate the horizontal alignment score H using the following formula: ; When Δy=0, the horizontal alignment score H is 1 (full marks); when Δy exceeds k*h, the horizontal alignment score H is 0. (2) Vertical proximity score: Let the Y-coordinate of the bottom edge of the text be y. t The Y-coordinate of the center point of the image region is y i Then the vertical distance d = |y i y t | Calculate the vertical proximity score V using the following formula: V=e d / σ ; Where σ is the distance attenuation constant; (3) Consistency score in arrangement order: Before calculating the consistency score, a layout analysis algorithm is used to determine if there are multiple images arranged horizontally. If such a layout exists, the consistency score calculation process is then initiated. If no such layout exists, the consistency score is automatically awarded as full marks, i.e., 1 point. The calculation process for the consistency score is as follows: Let N be the question number in the current question text. When multiple images are arranged horizontally, the sequence number of the image regions is I, and the order deviation Δo = |N|. If I |, then the consistency score of the permutation order is O = max(0,1). Δo / M), where M is the total number of image regions currently arranged horizontally; when the question number N is exactly the same as the arrangement number of the image region, Δo=0, and the score for consistency of arrangement order is the full score of 1. The greater the deviation, the lower the score for consistency of arrangement order. (4) Region Enclosure Relationship Score: Let the total area of ​​the image region be A. i If the area of ​​the intersection between the image region and the question unit to which the current question text belongs is Ainter, then the region enclosure relationship score is C = Ainter / Ai.

[0008] Furthermore, in step S5, the overall matching score S is calculated using a weighted summation formula: S = w1 * H + w2 * V + w3 * O + w4 * C; Where H, V, O, and C represent the normalized sub-scores of horizontal alignment, vertical proximity, arrangement order consistency, and region enclosure relationship, respectively, and w1, w2, w3, and w4 are the corresponding weight coefficients, satisfying w1+w2+w3+w4=1.

[0009] Furthermore, in step S6, when performing the image matching operation for a certain question text, the number of images referenced by the question text is denoted as K. K matching operations need to be performed. If more than two image regions with the highest comprehensive matching score are found in a certain matching operation, then the unique image region is matched by further combining semantic keywords.

[0010] Furthermore, in step S6, the specific process of performing secondary verification using semantic keyword matching is as follows: A1. Extract semantic keywords from the question text: Use natural language processing technology to extract the core nouns or phrases that describe the content of the attached figure from the question text; A2. Obtain semantic information of image regions, including textual and visual aspects; Text level: Extract the captions, annotations, or explanatory text inside and immediately below the image area using OCR technology; Visual aspect: Using image classification or object detection models, identify key visual element labels in the image area; A3. Calculate semantic matching degree: Compare the extracted semantic keywords of the question stem with the semantic information of the image region, and quantify the semantic similarity score between the two. A4. Selective Confirmation of Binding: Among multiple candidate image regions, select the image region with the highest semantic similarity score that exceeds the preset threshold of semantic similarity score as the unique matching image region in this matching operation; if the semantic similarity scores of all candidate image regions are lower than the threshold or there is a tie for the highest score, the exception handling mechanism is triggered and the case is handed over to manual review.

[0011] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0012] The present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the above-described method.

[0013] The beneficial effects of this invention are as follows: This invention achieves accurate automatic matching between the question stem and the attached figures by integrating image layout analysis, semantic reference recognition and multi-dimensional spatial association reasoning mechanism. It can not only identify language clues such as "as shown in the figure" and "see figure X", but also make comprehensive judgments by combining the question number order, the relative position of the text and the figure and the arrangement pattern. It can still maintain a high matching accuracy when facing integrated attached figure layout. Detailed Implementation

[0014] The present invention will be further described below. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0015] This embodiment provides a method for automatically matching images to intelligent education test questions, including the following steps: S1. Receive a full-page test paper image containing multiple questions, and preprocess the full-page test paper image to obtain a clear binarized image; S2. Use a deep learning model to identify all text regions and image regions in the binarized image, and extract the position coordinates and size information of each text region and image region respectively; S3. Use layout analysis algorithms to identify the spatial layout relationship of each region and divide the text region into question units. S4. For each question unit, check whether the question stem text contains references to the attached figure, such as "as shown in the figure," "see the figure below," "attached figure X," etc. Natural language processing techniques can be used to perform this detection on the question stem text.

[0016] S5. For the title text containing reference text pointing to the attached figure, construct a spatial association model between the location of the title text and all image regions: using the position coordinates and size information of the title text of the title unit and each image region, calculate the scores of four dimensions between the title unit and all image regions: horizontal alignment, vertical proximity, arrangement order consistency, and region enclosure relationship, and calculate the comprehensive matching score based on the scores of the four dimensions. S6. After calculating the comprehensive matching score between each question stem text containing references to the attached figures and each image region, perform the attached figure matching operation sequentially for each question stem text containing references to the attached figures according to the question number order: First, semantic analysis is performed on the reference text of the current question text to the attached figures to determine the number of attached figures referenced in the current question text as J (for example, for "as shown in Figure 1-2", the number of attached figures referenced can be determined to be 2). Then, J matching operations are performed on the current question text. In each matching operation, the image region with the highest comprehensive matching score between the current unassigned image region and the current question text is matched to the current question text and marked as assigned. This process continues until the J matching operations of the current question text are completed, and then the process of matching attached figures for the next question text begins.

[0017] S7. Output the structured data package of each question stem text and its matched image region to complete the automatic recognition and binding of questions with images.

[0018] In this embodiment, step S1 includes denoising, binarization, and tilt correction.

[0019] In this embodiment, in step S2, the deep learning model can be the Mask R-CNN model, which is trained on an educational document image dataset and can accurately distinguish the regions of text, illustrations, tables, formulas, images and decorative borders in the image.

[0020] In this embodiment, step S5, the calculation of the scores for the four dimensions of horizontal alignment, vertical proximity, consistency of arrangement order, and region enclosure relationship, as well as the comprehensive matching score, specifically includes: (1) The degree of horizontal alignment is specifically to determine whether the image area and the question text are on the same horizontal line or adjacent lines, which means that the question text prioritizes matching the image area that is visually on the same horizontal line.

[0021] Let Δy be the minimum vertical distance between the bounding box of the question text and the bounding box of the image region on the Y-axis, and let h be the standard line height of the exam paper. Calculate the horizontal alignment score H using the following formula: ; When Δy=0 (i.e., there is overlap in the same horizontal row), the horizontal alignment score H is the full score of 1; when Δy exceeds k*h (k is the set threshold for adjacent rows, such as k=2), the horizontal alignment score H is 0.

[0022] (2) Vertical proximity is used to measure the vertical distance between the center of the image area and the bottom of the question text. The closer the distance, the higher the score, which conforms to the common layout and reading habit of "text above image" in educational documents. Let the Y-coordinate of the bottom edge of the question text be y. t The Y-coordinate of the center point of the image region is y i Then the vertical distance d = |y i y t | Calculate the vertical proximity score V using the following formula: V=e d / σ ; Where σ is the distance decay constant (which can be set as a fixed proportion of the page height).

[0023] In this embodiment, an exponential decay function is used to measure the vertical proximity score. The smaller the vertical distance d, the closer the score V is to 1.

[0024] (3) The purpose of the consistency score in the arrangement order is to ensure that the question number order of the question stem text matches the physical arrangement order of the image area from left to right when multiple images are arranged horizontally (such as four images side by side), so as to avoid misattribution.

[0025] Before calculating the consistency score, a layout analysis algorithm is used to determine if multiple images are arranged horizontally. If such a layout exists, the consistency score calculation process proceeds. If no such layout exists, there is no risk of horizontal order disorder (i.e., no need to prevent horizontal misplacement), so the order deviation is considered 0, and the consistency score is directly set to full, 1 point. This ensures that this dimension does not incorrectly deduct points from the final overall matching score, guaranteeing the accuracy of the matching algorithm. The calculation process for the consistency score is as follows: Let N be the question number in the current question text. When multiple images are arranged horizontally, the sequence number of the image regions is I, and the order deviation Δo = |N|. If I |, then the consistency score of the permutation order is O = max(0,1). Δo / M), where M is the total number of image regions currently arranged horizontally. When the question number N is exactly the same as the sequence number of the image region (Δo=0), the score for consistency of the arrangement order is the full score of 1. The greater the deviation, the lower the score for consistency of the arrangement order.

[0026] (4) The purpose of the region enclosing relationship score is to measure whether the image is located inside the question unit to which the current question text belongs, thus preventing incorrect matching across question regions. Let the total area of ​​the image region be A. i If the area of ​​the intersection between the image region and the question unit to which the current question text belongs is Ainter, then the region enclosure relationship score is C = Ainter / Ai.

[0027] The region enclosure score is essentially calculated by determining the proportion of the image region enclosed by the question unit. If the image region is completely within the question block, then C=1; if it is completely outside, then C=0.

[0028] Through the above process, abstract spatial layout relationships can be transformed into specific quantitative values, thereby accurately calculating the comprehensive matching score between each image region and the current question text.

[0029] Finally, the overall matching score S is calculated using a weighted summation formula: S = w1 * H + w2 * V + w3 * O + w4 * C; Where H, V, O, and C represent the normalized sub-scores (all ranging from 0) for horizontal alignment, vertical proximity, consistency of arrangement order, and region enclosure relationship, respectively. 1) w1, w2, w3, w4 are the corresponding weight coefficients, and w1+w2+w3+w4=1.

[0030] In this embodiment, in step S6, when performing the image matching operation for a certain question text, the number of images referenced by the question text is denoted as K. K matching operations need to be performed. If more than two image regions with the highest comprehensive matching score are found in a certain matching operation, then the unique image region is further matched by semantic keyword matching.

[0031] In this embodiment, the specific process of performing secondary verification using semantic keyword matching is as follows: A1. Extract semantic keywords from the question stem text: Use natural language processing techniques (such as named entity recognition or keyword extraction) to extract core nouns or phrases describing the content of the attached diagram (e.g., "circuit diagram", "pulley system", "plant cell", "force analysis").

[0032] A2. Obtain semantic information of image regions, including textual and visual aspects; At the text level: OCR (Optical Character Recognition) technology is used to extract the captions, annotations, or explanatory text inside and immediately below the image area.

[0033] Visual aspect: Using image classification or object detection models, identify key visual element labels in the image area (such as identifying "battery", "switch", "moving pulley" etc. in the image).

[0034] A3. Calculate semantic matching degree: Compare the extracted semantic keywords of the question stem with the semantic information of the image region, and use professional knowledge graphs, thesaurus or word vector models (such as calculating the cosine similarity of word vectors) in the education field to quantify the semantic similarity score between the two.

[0035] A4. Selective Binding: Among multiple candidate image regions, the image region with the highest semantic similarity score that exceeds a preset threshold is selected as the unique matching image region in this matching operation. If the semantic similarity scores of all candidate image regions are lower than the threshold or there is a tie for the highest score, an exception handling mechanism is triggered, and the case is handed over to manual review.

[0036] For those skilled in the art, various corresponding changes and modifications can be made based on the above technical solutions and concepts, and all such changes and modifications should be included within the protection scope of the claims of this invention.

Claims

1. A method for automatically matching images to test questions in smart education, characterized in that: Includes the following steps: S1. Receive a full-page test paper image containing multiple questions, and preprocess the full-page test paper image to obtain a clear binarized image; S2. Use a deep learning model to identify all text regions and image regions in the binarized image, and extract the position coordinates and size information of each text region and image region respectively; S3. Use layout analysis algorithms to identify the spatial layout relationship of each region and divide the text region into question units. S4. For the question stem text in each question unit, check whether it contains reference text pointing to the attached figure; S5. For the title text containing reference text pointing to the attached figure, construct a spatial association model between the location of the title text and all unassigned image regions: using the position coordinates and size information of the title text of the title unit and each image region, calculate the scores of four dimensions between the title unit and the unassigned image regions: horizontal alignment, vertical proximity, arrangement order consistency, and region enclosure relationship, and calculate the comprehensive matching score based on the scores of the four dimensions. S6. After calculating the comprehensive matching score between each question stem text containing references to the attached figures and each image region, perform the attached figure matching operation sequentially for each question stem text containing references to the attached figures according to the question number order: First, semantic analysis is performed on the reference text of the current question text to the attached figures to determine the number of attached figures referenced by the current question text as J. Then, J matching operations are performed on the current question text. In each matching operation, the image region with the highest comprehensive matching score between the current unassigned image region and the current question text is matched to the current question text, and this image region is marked as assigned. This process continues until the J matching operations of the current question text are completed, and then the process of matching attached figures for the next question text begins. S7. Output the structured data package of each question text and its corresponding image region to complete the automatic recognition and binding of questions with images.

2. The method according to claim 1, characterized in that, In step S1, the preprocessing includes denoising, binarization, and tilt correction.

3. The method according to claim 1, characterized in that, In step S2, the deep learning model uses the Mask R-CNN model, which is trained on an educational document image dataset and can accurately distinguish between text, illustrations, tables, formula images and decorative borders in the image.

4. The method according to claim 1, characterized in that, In step S5, the specific calculation process for the horizontal alignment score, vertical proximity score, arrangement order consistency score, and region enclosure relationship score is as follows: (1) Horizontal alignment score: Let Δy be the minimum vertical distance between the bounding box of the question text and the bounding box of the image region on the Y-axis, and let h be the standard line height of the exam paper. Calculate the horizontal alignment score H using the following formula: ； When Δy=0, the horizontal alignment score H is 1 (full marks); when Δy exceeds k*h, the horizontal alignment score H is 0. (2) Vertical proximity score: Let the Y-coordinate of the bottom edge of the text be y. t The Y-coordinate of the center point of the image region is y i Then the vertical distance d = |y i y t | Calculate the vertical proximity score V using the following formula: V=e d / σ ; Where σ is the distance attenuation constant; (3) Consistency score in arrangement order: Before calculating the consistency score, a layout analysis algorithm is used to determine if there are multiple images arranged horizontally. If such a layout exists, the consistency score calculation process is then initiated. If no such layout exists, the consistency score is automatically awarded as full marks, i.e., 1 point. The calculation process for the consistency score is as follows: Let N be the question number in the current question text. When multiple images are arranged horizontally, the sequence number of the image regions is I, and the order deviation Δo = |N|. If I |, then the consistency score of the permutation order is O = max(0,1). Δo / M), where M is the total number of image regions currently arranged horizontally; when the question number N is exactly the same as the arrangement number of the image region, Δo=0, and the score for consistency of arrangement order is the full score of 1. The greater the deviation, the lower the score for consistency of arrangement order. (4) Region Enclosure Relationship Score: Let the total area of ​​the image region be A. i If the area of ​​the intersection between the image region and the question unit to which the current question text belongs is Ainter, then the region enclosure relationship score is C = Ainter / Ai.

5. The method according to claim 1 or 4, characterized in that, In step S5, the overall matching score S is calculated using a weighted summation formula: S = w1 * H + w2 * V + w3 * O + w4 * C; Where H, V, O, and C represent the normalized sub-scores of horizontal alignment, vertical proximity, arrangement order consistency, and region enclosure relationship, respectively, and w1, w2, w3, and w4 are the corresponding weight coefficients, satisfying w1+w2+w3+w4=1.

6. The method according to claim 1, characterized in that, In step S6, when performing the image matching operation for a certain question text, the number of images referenced by the question text is denoted as K. K matching operations need to be performed. If more than two image regions with the highest comprehensive matching score are found in a certain matching operation, then the unique image region is further matched by semantic keyword matching.

7. The method according to claim 6, characterized in that, In step S6, the specific process of performing secondary verification using semantic keyword matching is as follows: A1. Extract semantic keywords from the question text: Use natural language processing technology to extract the core nouns or phrases that describe the content of the attached figure from the question text; A2. Obtain semantic information of image regions, including textual and visual aspects; Text level: Extract the captions, annotations, or explanatory text inside and immediately below the image area using OCR technology; Visual aspect: Using image classification or object detection models, identify key visual element labels in the image area; A3. Calculate semantic matching degree: Compare the extracted semantic keywords of the question stem with the semantic information of the image region, and quantify the semantic similarity score between the two. A4. Selective Confirmation of Binding: Among multiple candidate image regions, select the image region with the highest semantic similarity score that exceeds the preset threshold of semantic similarity score as the unique matching image region in this matching operation; if the semantic similarity scores of all candidate image regions are lower than the threshold or there is a tie for the highest score, the exception handling mechanism is triggered and the case is handed over to manual review.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method of any one of claims 1-7.

Images