Efficient document layout analysis method, system, equipment and media for question bank construction

Abstract

The present invention discloses an efficient document layout analysis method, system, device and medium for question bank construction, which is suitable for question bank construction. The analysis method comprises the following steps: scanning document images of different subject test papers, textbook exercises, and books, and pre-processing to obtain non-tilted scanned document images; marking the relative position and category labels of each element in the scanned document image to obtain a document data set; using the document data set to train a document layout target detection model; inputting the image to be detected into the trained document layout target detection model, detecting foreground content including questions, graphics and tables, and outputting recognition results of the layout elements. While improving the foreground content detection accuracy, the present invention also increases the inference speed of the document layout target detection model, realizes the accurate recognition of various elements in test papers and textbook exercises of different subjects, and provides an efficient solution for applications such as question bank construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of document layout analysis, and in particular relates to an efficient document layout analysis method, system, electronic equipment and storage medium for question bank construction. Background Art

[0002] With the rapid development of educational technology, visual information extraction from document images is becoming increasingly important in educational fields such as question bank construction, textbook digitization, and online test answering. Document layout analysis, as a precursor to document understanding, aims to identify and separate distinct layout structures and content areas, such as titles, text, charts, and formulas, from complex document images, providing the necessary foundation for subsequent document information extraction and understanding tasks.

[0003] However, some current document layout analysis methods rely on general unimodal techniques such as convolutional neural networks and Transformers. In the paper "Li C, Guo R, Zhou J, et al. Pp-structurev2: A stronger document analysis system [J]. arXiv preprint arXiv: 2210.05391, 2022," Li et al. proposed using PP-YOLOV2 as a detector for layout analysis. However, because its pre-trained weights are based on a general dataset, its performance in document layout analysis is relatively mediocre. In the paper "Zhao Z, Kang H, Wang B, et al. DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception [J]. arXiv preprint arXiv: 2410.12628, 2024," Zhao et al. combined document data pre-training with the unimodal YOLO for layout detection. However, the YOLO method is limited by non-maximum suppression, making further accuracy improvements difficult. In addition, pre-trained multimodal technology has demonstrated powerful capabilities in document understanding and layout analysis. In the paper "Da C, Luo C, Zheng Q, et al. Vision grid transformer for document layout analysis[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2023:19462-19472.", Da et al. performed pre-training by using high-parameter grid text encoding, but still faced significant speed bottlenecks when reasoning about document images, making it difficult to meet practical application needs. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide an efficient document layout analysis method, system, equipment and medium for question bank construction. The document layout analysis method can adapt to the complex layout structure and high-noise environment in educational scenarios while ensuring high precision and high speed.

[0005] In order to achieve the first objective above, the present invention discloses an efficient document layout analysis method for question bank construction, the efficient document layout analysis method comprising the following steps:

[0006] S1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0007] S2. Annotate the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document dataset;

[0008] S3. Use the document dataset to train a document layout object detection model;

[0009] S4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

[0010] Furthermore, the document layout object detection model includes a lightweight feature extraction network, a feature encoder, and a detection head connected in sequence;

[0011] The lightweight feature extraction network includes a backbone path, a branch path, and a dynamic multi-scale fusion module. We combine the idea of ​​cross-stage local networks to divide the original module with large parameters into lightweight backbone and branch paths, thereby improving the inference speed and accuracy of the document layout object detection model. The backbone path and branch paths respectively extract shallow features from the input image. Shallow features include edge features, texture features, and color features. The branch path and the backbone path use different network architectures to capture shallow features of the image from multiple scales and angles, ensuring feature complementarity and providing diverse expressions. The dynamic multi-scale fusion module integrates multi-scale information from the shallow features of the backbone path and branch paths through a dynamic fusion strategy to obtain the final fused feature map X. This preserves both local details and global context information of the image at the feature level, improving the accuracy and robustness of image analysis.

[0012] The feature encoder includes a multi-scale information interaction module and a dual convolution attention module connected in sequence. The multi-scale information interaction module contains 8 layers of multi-head attention to capture the information relationship of different frequencies in the feature map. The dual convolution attention module contains 2 convolutions and 2 attention mechanisms to simultaneously model global features and local features and remove document image noise. The feature encoder is denoted as function G(), which is defined as follows:

[0013] Where the feature map X has c input channels, The output feature map represents the feature representation after being processed by the multi-scale information interaction module and the dual convolutional attention module;

[0014] The detection head outputs a feature map Perform the query selection operation with the minimum feature uncertainty and calculate the difference between the positioning prediction distribution P() and the classification prediction distribution C(). The formula is as follows:

[0015]

[0016] in, is the output feature map of the feature encoder, which represents the feature representation of the input image after being processed by the feature encoder. D represents the dimension of the feature, Represents feature encoder features The predicted distribution in the localization task, Represents feature encoder features In the prediction distribution of classification tasks, the symbol "||·||" means to find the 2 norm. is feature uncertainty, defined as the difference between the location prediction distribution and the classification prediction distribution, reflecting the uncertainty of the query feature. Represents the prediction result of the document layout object detection model for the input data, y represents the true label of the input data, including the true category c and the true bounding box b, represents the output bounding box of the model in the positioning task, b represents the true bounding box of the input data, represents the output category of the model in the classification task, c represents the true category label of the input data, and L box () indicates the predicted bounding box The difference between the ground-truth bounding box b, L cls () indicates the predicted category The difference between the true category c, L box () and L cls () is defined as follows:

[0017]

[0018] Among them, x0, y0 represent the center coordinates of the bounding box, w0, h0 represent the width and height of the bounding box respectively, i represents the index of the bounding box parameter, L s1 () indicates that when the absolute value of the error is less than 1, the loss is half the square of the error. When the absolute value of the error is greater than or equal to 1, the loss is the absolute value of the error minus 0.5. j represents the index of the category label parameter, and Y0 is the number of categories.

[0019] Furthermore, given an input document feature map I, the first convolutional layer of the backbone receives the initial input document feature map I, the output of the first convolutional layer of the backbone is connected to the input of the second convolutional layer of the backbone, and the output of the second convolutional layer of the backbone is connected to the input of the third convolutional layer of the backbone; the first convolutional layer of the branch receives the initial input I, the output of the first convolutional layer of the branch is connected to the input of the second convolutional layer of the branch, and the output of the second convolutional layer of the branch is connected to the input of the third convolutional layer of the branch; the output of the third convolutional layer of the backbone and the output of the third convolutional layer of the branch are connected to the input of the first dynamic multi-scale fusion module;

[0020] The output of the first dynamic multi-scale fusion module is connected to the input of the fourth convolutional layer of the trunk and the input of the fourth convolutional layer of the branch. The output of the fourth convolutional layer of the trunk is connected to the input of the fifth convolutional layer of the trunk, the output of the fifth convolutional layer of the trunk is connected to the input of the sixth convolutional layer of the trunk, the output of the sixth convolutional layer of the trunk is connected to the input of the seventh convolutional layer of the trunk, and the output of the seventh convolutional layer of the trunk is connected to the input of the eighth convolutional layer of the trunk; the output of the fourth convolutional layer of the branch is connected to the input of the fifth convolutional layer of the branch; the input of the eighth convolutional layer of the trunk and the output of the fifth convolutional layer of the branch are connected to the input of the second dynamic multi-scale fusion module;

[0021] The output of the second dynamic multi-scale fusion module is connected to the input of the ninth convolutional layer of the trunk and the input of the sixth convolutional layer of the branch. The output of the ninth convolutional layer of the trunk is connected to the input of the tenth convolutional layer of the trunk; the output of the sixth convolutional layer of the branch is connected to the input of the seventh convolutional layer of the branch; the output of the tenth convolutional layer of the trunk and the output of the seventh convolutional layer of the branch are connected to the input of the third dynamic multi-scale fusion module. The third dynamic multi-scale fusion module obtains the final fused feature map X.

[0022] Furthermore, the specific process of training the document layout object detection model using the document dataset in step S3 is as follows:

[0023] S301. Initialize the document layout object detection model, set the number of training rounds, input image size to 1280 by 1280, select AdamW optimizer, which comes from the preprint paper "Decoupled Weight Decay Regularization" published by Loshchilov et al. in 2017. AdamW optimizer is a variant of Adam optimizer; Adam is an optimizer that combines momentum and adaptive learning rate. AdamW improves the optimization process by "decoupling" the weight decay in Adam, which can provide better generalization performance and converge faster in different types of deep learning tasks; set the batch size to 4, enable mosaic data enhancement, the mosaic data enhancement method was proposed by Alexey Bochkovskiy in the document "Ultra-Fast Object Detection for Autonomous In the paper "Drivin", it is proposed that mosaic data augmentation is widely used in object detection tasks in computer vision, especially in processing small object detection and complex scenes. Its core idea is to increase data diversity and model robustness by stitching four images together to form a new image. It is turned off in the last 10 training rounds and hybrid image augmentation is turned on at the same time to obtain the initial training configuration.

[0024] S302, randomly extracting a batch of image label pairs from the training set of the document layout analysis dataset, and inputting them into the document layout object detection model for forward propagation to obtain predicted bounding box values;

[0025] S303, calculate the actual border box and the predicted border The IOU intersection-over-union ratio between them is calculated using the GIOU() function, which comes from the paper "Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression" published by the author Ronghang Hu at the 2020 Conference on Computer Vision and Pattern Recognition. As part of the loss calculation, it is used to guide the update of the document layout object detection model parameters. The result is as follows:

[0026]

[0027] Among them, box represents the actual border value, Represents the bounding box value predicted by the model, A represents the predicted bounding box area, B represents the true bounding box area, O represents the bounding box area, and the bounding box area contains the minimum rectangular box of A and B;

[0028] S304. Perform forward propagation on the validation set of the document layout analysis dataset to obtain the average precision mAP and overall loss Loss of this round of training. Loss includes IOU loss Classification loss Bounding Box Loss The formula is as follows:

[0029]

[0030] Among them, cls k Represents the true category label of 0 or 1, Represents the predicted probability of the kth class, Y1 represents the total number of categories, box v represents the vth parameter of the true bounding box, The predicted vth parameter of the measured bounding box, x1, y1 represent the center coordinates of the bounding box, w1, h1 represent the width and height of the bounding box respectively, and v represents the index of the bounding box parameter;

[0031] S305: The document layout object detection model evaluates the performance of the current training times and compares the best performance. If the performance does not improve and reaches the early stopping threshold, the training is stopped early to obtain the early stopping condition.

[0032] S306, the document layout object detection model saves the status of the current training times to obtain the saved document layout object detection model and training status;

[0033] S307, the training process updates the learning rate and obtains a new learning rate according to the learning rate decay strategy;

[0034] S308. The document layout object detection model uses the results of the current training times as a basis to continue the next round of training and obtain the starting state of the next round of training.

[0035] Furthermore, in step S4, the real image to be tested is input into the document layout object detection model for forward propagation to obtain the detection result of the document foreground content.

[0036] A second object of the present invention is to provide an efficient document layout analysis system for question bank construction, which is used to execute the above-mentioned efficient document layout analysis method for question bank construction. The efficient document layout analysis system includes:

[0037] Scanning preprocessing module, used to scan document images of different subject examination papers, textbook exercises, and books, and preprocess them to obtain non-skewed scanned document images;

[0038] The document annotation module is used to annotate the relative positions and category labels of titles and diagrams in the scanned document images to obtain a document dataset;

[0039] Model training module, used to train document layout object detection model using document dataset;

[0040] The image recognition module is used to input the image to be detected into the trained document layout object detection model, detect foreground content including titles, graphics and tables, and output the recognition results of layout elements.

[0041] The third object of the present invention is to provide a computer device, including a processor and a memory for storing processor executable programs, when the processor executes the program stored in the memory, it implements the above-mentioned efficient document layout analysis method for question bank construction.

[0042] The fourth object of the present invention is to provide a storage medium storing a program, which, when executed by a processor, implements the above-mentioned efficient document layout analysis method for question bank construction.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention discloses an efficient document layout analysis method, system, device, and medium for question bank construction, and proposes a document layout target detection model with both high speed and high accuracy. The model adopts a real-time DETR target detector, avoiding the limitations of traditional non-maximum suppression, and improves the model's reasoning speed and generalization ability through a lightweight feature extraction network and a multi-scale information fusion module. In addition, the encoder integrates a multi-scale information interaction module and a dual convolution-attention module to effectively deal with complex layouts and noise interference, enhancing the adaptability and robustness of the model in practical applications.

[0045] 2. Compared with existing technologies, this invention focuses on document layout analysis for test papers and accompanying exercises, effectively adapting to complex and noisy test paper and textbook layouts. The technical solution of this invention has strong disciplinary versatility and can be applied to educational scenarios across different disciplines. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1This is a flow chart of an efficient document layout analysis method for question bank construction disclosed in Example 1 of the present invention;

[0048] Figure 2 This is a model structure diagram of an efficient document layout analysis method for question bank construction disclosed by the present invention;

[0049] Figure 3 It is a diagram of the lightweight feature extraction network structure disclosed in the present invention;

[0050] Figure 4 It is a schematic diagram of the detection results disclosed in the present invention;

[0051] Figure 5 This is a structural block diagram of an efficient document layout analysis system for question bank construction in Example 3 of the present invention;

[0052] Figure 6 This is a structural block diagram of the computer device in Example 4 of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0055] Example 1

[0056] Figure 1 This invention discloses a flow chart of an efficient document layout analysis method for question bank construction, such as Figure 1 As shown, the present embodiment discloses an efficient document layout analysis method for question bank construction, comprising the following steps: scanning and preprocessing a document image, annotating the scanned document image, training with a document dataset, and detecting the document image with a document layout target detection model, as follows:

[0057] T1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0058] T2. Annotate the relative position and category labels of each element in the scanned document image to obtain a document dataset;

[0059] T3. Use the document dataset to train the document layout target detection model. The architecture of the document layout target detection model is as follows: Figure 2 As shown in the figure, it includes a lightweight feature extraction network, a feature encoder, and a detection head connected in sequence. The trunk path combines the idea of ​​a cross-stage local network to achieve richer gradient combinations while significantly reducing computing resources. The specific approach is to change the 6-fold stacking of the original module to 3-fold stacking, which is connected in parallel with the branch path. This design makes the entire model more lightweight without reducing performance. In order to solve the problem of differences in feature extraction between the trunk path and the branch path, a dynamic alignment fusion module is designed to achieve adaptive fusion of features. The coordinate attention mechanism is introduced in the convolution block of the last layer of the trunk path to further reduce redundant information interference and improve adaptability to complex document layouts. The convolution layers included in the lightweight feature extraction network are shown below. Figure 3 As shown in the figure; in addition, by introducing dual convolution and dual attention modules, the complex layout, diverse text and graphic elements in document images, as well as noise interference such as blur and shadows caused by scanning or shooting are effectively dealt with; to verify the effectiveness of the document layout object detection model, the present invention conducted a comprehensive evaluation on the private dataset EED and the public dataset CDLA. As can be seen from the results in Table 1, on the private dataset EED, the proposed document layout object detection model DLA-DETR achieved the best performance in both AP50 and mAP indicators in comparison with YOLOv5m, YOLOv51, YOLOv8m, YOLOv10m, YOLO11m, DocLayout-YOLO, and the baseline model RT-DETR-L with similar parameter numbers. At the same time, it also significantly outperformed existing document layout analysis methods in terms of GFLOPs. Although DLA-DETR is slightly lower than YOLOv10m in inference speed FPS, it still maintains a high speed advantage. Compared to the baseline model RT-DETR-L, DLA-DETR improves accuracy by 0.5%, while reducing GFLOPs and parameters by 54.2% and 35.4%, respectively. As shown in Table 2, on the public dataset CDLA, DLA-DETR achieves the best performance in terms of inference speed, AP50, and mAP compared to existing unimodal models such as PP-YOLOv2, PP-PicoDet-LCNet, and DocLayout-YOLO.

[0060] Table 1. Comparison of the accuracy and inference speed of the proposed method with other methods on the EED dataset

[0061] Model AP50(%) mAP (%) GFLOPs(G) FPS YOLOv5m 91.3 74.6 64.0 34.4 YOLOv5l 92.3 75.0 135.3 19.7 YOLOv8m 91.7 74.6 78.8 33.6 YOLOv10m 90.8 74.2 64.0 44.4 YOLO11m 92.4 74.9 87.3 31.a DocLayout-YOLO 92.7 74.4 68.0 29.8 DocLayout-YOLO 92.8 74.8 127.4 28.7 RT-DETR-L (Benchmark) 92.9 74.9 103.5 21.1 Document layout object detection model 93.4 75.1 47.4 35.9

[0062] Table 2. Comparison of the accuracy and inference speed of the present invention and other methods on the CDLA dataset

[0063] Model mAP (%) FPS PP-YOLOv2 84.7 34.4 PP-PicoDet-LCNet2.5x 87.8 1 9.7 PP-PicoDet-LCNet1.0x 84.5 33.6 PP-PicoDet-LCNet1.0x+FGD 89.9 29.4 Document layout object detection model 91.7 35.7

[0064] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After reasoning, the document layout object detection model outputs the recognition results of the layout elements including questions and charts. The result example is as follows: Figure 4 shown.

[0065] Example 2

[0066] Based on the efficient document layout analysis method for question bank construction disclosed in Example 1, this embodiment continues to refer to steps T1 to T4 of the efficient document layout analysis method for question bank construction disclosed in Example 1. Figure 1 As shown, the method includes the following steps:

[0067] T1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0068] T2. Annotate the relative position and category labels of each element in the scanned document image to obtain a document dataset;

[0069] T3. Use the document dataset to train the document layout target detection model. The architecture of the document layout target detection model is as follows: Figure 2 As shown, it includes a lightweight feature extraction network, a feature encoder and a detection head connected in sequence;

[0070] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0071] Table 3 shows the module ablation experiments for the document layout detection model in step T3. Comparative experiments were conducted on the EED dataset, comparing HG-CSPNet* without the attention mechanism, the multi-scale information interaction module, CA-HGBlock with the attention mechanism, and the dual convolution and dual attention modules. While HG-CSPNet* without the attention mechanism slightly degraded accuracy, inference speed doubled. Introducing either the information interaction module or the attention mechanism further improved both accuracy and inference speed. Combining the two modules achieved comparable accuracy to the baseline model. Finally, adding the dual convolution and dual attention modules surpassed the baseline in both accuracy and speed. The parameters of the document layout object detection model were then initialized and trained on the document layout analysis dataset to obtain a pretrained document layout object detection model.

[0072] Table 3. Comparison of the impact of different modules on the overall architecture of the present invention on the EED dataset

[0073] HG-CSPNet* Hilo-TF CA-HGB lock DCAM AP50 mAP - - - - 92.9 74.9 use - - - 91.7 73.6 use - use - 92.3 74.0 use use - - 92.5 74.3 use use use - 92.9 74.8 use use use use 93.4 75.1

[0074] In summary, Examples 1 and 2 of the present invention propose an efficient document layout analysis method for question bank construction. First, a high-quality document image dataset is constructed through a standardized process, and combined with manual annotation and verification steps to ensure the accuracy and consistency of the data. Subsequently, the dataset is trained and inferred using the designed document layout target detection model, and the advantages of the model in processing complex document layouts, diverse text and graphic elements are fully verified through the examples. Through the designed lightweight feature extraction network, multi-scale information fusion module, and dual convolution and dual attention mechanisms, the network can efficiently parse the document layout structure, provide technical support for applications such as question bank construction and online answering in educational scenarios, and improve the automation level and processing efficiency of the task.

[0075] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0076] Example 3

[0077] like Figure 5 As shown, this embodiment provides an efficient document layout analysis system for question bank construction, which includes: a scanning preprocessing module 501, a document annotation module 502, a model training module 503, and an image recognition module 504. The specific functions of each module are as follows:

[0078] Scanning pre-processing module 501 is used to scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0079] The document annotation module 502 is used to annotate the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document dataset;

[0080] Model training module 503, used to train a document layout object detection model using a document dataset;

[0081] The image recognition module 504 is used to input the image to be detected into the trained document layout object detection model, detect the foreground content including the title, graphics and tables, and output the recognition results of the layout elements.

[0082] Example 4

[0083] This embodiment provides a computer device, which can be a computer, such as Figure 6 As shown, a processor 602, a memory, an input device 603, a display 604, and a network interface 605 are connected via a system bus 601. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 606 and an internal memory 607. The non-volatile storage medium 606 stores an operating system, a computer program, and a database. The internal memory 607 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 602 executes the computer program stored in the memory, an efficient document layout analysis method for question bank construction proposed in the above embodiment 1 is implemented. The efficient document layout analysis method for question bank construction includes the following steps:

[0084] T1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0085] T2. Annotate the relative position and category labels of each element in the scanned document image to obtain a document dataset;

[0086] T3. Use the document dataset to train a document layout object detection model;

[0087] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0088] Example 5

[0089] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the method for analyzing an efficient document layout for question bank construction described in the first embodiment is implemented. The method for analyzing an efficient document layout for question bank construction includes the following steps:

[0090] T1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images;

[0091] T2. Annotate the relative position and category labels of each element in the scanned document image to obtain a document dataset;

[0092] T3. Use the document dataset to train a document layout object detection model;

[0093] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0094] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0095] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An efficient document layout analysis method for question bank construction, characterized by: The efficient document layout analysis method comprises the following steps: S1. Scan document images of different subject examination papers, textbook exercises, and books, and pre-process them to obtain non-skewed scanned document images; S2. Annotate the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document dataset; S3. Using a document dataset to train a document layout object detection model; the document layout object detection model includes a lightweight feature extraction network, a feature encoder, and a detection head connected in sequence; The lightweight feature extraction network includes a trunk path, a branch path, and a dynamic multi-scale fusion module, wherein the trunk path and the branch path respectively extract shallow features from the input image, and the shallow features include edge features, texture features, and color features; the dynamic multi-scale fusion module integrates the shallow features from the trunk path and the branch path through a dynamic fusion strategy to obtain the final fused feature map X; The feature encoder includes a multi-scale information interaction module and a dual convolution attention module connected in sequence. The multi-scale information interaction module contains 8 layers of multi-head attention to capture the information relationship of different frequencies in the feature map. The dual convolution attention module contains 2 convolutions and 2 attention mechanisms to simultaneously model global features and local features and remove document image noise. The feature encoder is denoted as function G( ), which is defined as follows: Where the feature map X has c input channels, The output feature map represents the feature representation after being processed by the multi-scale information interaction module and the double convolution attention module; the detection head outputs the feature map Perform the query selection operation with the minimum feature uncertainty and calculate the difference between the positioning prediction distribution P( ) and the classification prediction distribution C( ). The formula is as follows: in, is the output feature map of the feature encoder, which represents the feature representation of the input image after being processed by the feature encoder. , D represents the dimension of the feature, Represents feature encoder features The predicted distribution in the localization task, Represents feature encoder features Predictive distribution in classification tasks, symbol" " means to find the 2 norm, is feature uncertainty, defined as the difference between the location prediction distribution and the classification prediction distribution, reflecting the uncertainty of the query feature. Represents the prediction result of the document layout object detection model on the input data, Represents the true label of the input data, including the true category and the ground-truth bounding box , represents the output bounding box of the model in the positioning task, represents the ground-truth bounding box of the input data, Represents the output category of the model in the classification task, represents the true category label of the input data, Represents the predicted bounding box and the ground-truth bounding box The difference between Represents the predicted category and the true category The difference between The definition is as follows: in, represents the center coordinates of the bounding box, denote the width and height of the bounding box respectively, represents the index of the bounding box parameters, It means that when the absolute value of the error is less than 1, the loss is half of the square of the error. When the absolute value of the error is greater than or equal to 1, the loss is the absolute value of the error minus 0.

5. Represents the index of the category label parameter, is the number of categories; S4, input the image to be detected into the trained document layout target detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

2. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: Given an input document feature map I, the first convolutional layer of the backbone receives the initial input document feature map I, the output of the first convolutional layer of the backbone is connected to the input of the second convolutional layer of the backbone, and the output of the second convolutional layer of the backbone is connected to the input of the third convolutional layer of the backbone; The first convolution layer of the branch receives the initial input I, the output of the first convolution layer of the branch is connected to the input of the second convolution layer of the branch, and the output of the second convolution layer of the branch is connected to the input of the third convolution layer of the branch; the output of the third convolution layer of the trunk and the output of the third convolution layer of the branch are connected to the input of the first dynamic multi-scale fusion module; The output of the first dynamic multi-scale fusion module is connected to the input of the fourth convolutional layer of the trunk and the input of the fourth convolutional layer of the branch. The output of the fourth convolutional layer of the trunk is connected to the input of the fifth convolutional layer of the trunk, the output of the fifth convolutional layer of the trunk is connected to the input of the sixth convolutional layer of the trunk, the output of the sixth convolutional layer of the trunk is connected to the input of the seventh convolutional layer of the trunk, and the output of the seventh convolutional layer of the trunk is connected to the input of the eighth convolutional layer of the trunk; the output of the fourth convolutional layer of the branch is connected to the input of the fifth convolutional layer of the branch; the input of the eighth convolutional layer of the trunk and the output of the fifth convolutional layer of the branch are connected to the input of the second dynamic multi-scale fusion module; The output of the second dynamic multi-scale fusion module is connected to the input of the ninth convolutional layer of the backbone and the input of the sixth convolutional layer of the branch, and the output of the ninth convolutional layer of the backbone is connected to the input of the tenth convolutional layer of the backbone; The output of the branch's sixth convolutional layer is connected to the input of the branch's seventh convolutional layer; the output of the trunk's tenth convolutional layer and the output of the branch's seventh convolutional layer are connected to the input of the third dynamic multi-scale fusion module, and the third dynamic multi-scale fusion module obtains the final fused feature map X.

3. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: The specific process of using the document dataset to train the document layout object detection model in step S3 is as follows: S301. Initialize the document layout object detection model, set the number of training rounds, set the input image size to 1280 x 1280, select the AdamW optimizer, set the batch size to 4, enable mosaic enhancement and turn it off for the last 10 training rounds, and enable hybrid image enhancement to obtain the initial training configuration; S302, randomly extract a batch of image label pairs from the training set of the document layout analysis dataset, and input them into the document layout object detection model for forward propagation to obtain the predicted border value; S303, calculate the actual border value and predicted bounding box The IOU intersection-of-union ratio between them is calculated by the GIOU() function. As part of the loss calculation, it is used to guide the update of the document layout object detection model parameters. The result is as follows: in, Indicates the actual border value, represents the bounding box value predicted by the model, represents the predicted bounding box area, represents the ground-truth bounding box area, Represents the bounding box area, the bounding box area contains and S304, perform forward propagation on the validation set of the document layout analysis dataset to obtain the average precision mAP and overall loss Loss of this round of training, Including IOU loss , classification loss , bounding box loss , the formula is as follows: Represents the true category label of 0 or 1, represents the predicted probability of the kth class, represents the total number of categories, represents the vth parameter of the true bounding box, Predict the vth parameter of the measured bounding box, represents the center coordinates of the bounding box, denote the width and height of the bounding box respectively, Indicates the index of the bounding box parameters; S305, the document layout object detection model evaluates the performance of the current training times and compares the best performance. If the performance does not improve and reaches the early stopping threshold, the training is stopped early to obtain the early stopping condition; S306, the document layout object detection model saves the status of the current training times to obtain the saved document layout object detection model and training status; S307, the training process updates the learning rate and obtains a new learning rate according to the learning rate decay strategy; S308. The document layout object detection model uses the results of the current training times as a basis to continue the next round of training and obtain the starting state of the next round of training.

4. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: In step S4, the real image to be tested is input into the document layout object detection model for forward propagation to obtain the detection result of the document foreground content.

5. An efficient document layout analysis system for question bank construction, used to execute the efficient document layout analysis method for question bank construction according to any one of claims 1 to 4, characterized in that: The efficient document layout analysis system includes: Scanning preprocessing module, used to scan document images of different subject examination papers, textbook exercises, and books, and preprocess them to obtain non-skewed scanned document images; The document annotation module is used to annotate the relative positions and category labels of titles and diagrams in the scanned document images to obtain a document dataset; Model training module, used to train document layout object detection model using document dataset; The image recognition module is used to input the image to be detected into the trained document layout object detection model, detect foreground content including titles, graphics and tables, and output the recognition results of layout elements.

6. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, it implements the efficient document layout analysis method for question bank construction as described in any one of claims 1 to 4.

7. A storage medium storing a program, characterized in that: When the program is executed by a processor, the efficient document layout analysis method for question bank construction described in any one of claims 1 to 4 is implemented.

Images