Question bank construction-oriented efficient document layout analysis method, system and equipment and medium

Abstract

The invention discloses a question bank construction-oriented efficient document layout analysis method, system, equipment and medium, which are suitable for question bank construction, and the analysis method comprises the following steps: scanning document images of test papers, teaching material matched exercises and books of different subjects, and preprocessing to obtain non-tilt scanned document images; labeling relative positions and category labels of elements in the scanned document image to obtain a document data set; training the document layout target detection model by adopting the document data set; and inputting a to-be-detected image into the trained document layout target detection model, detecting foreground contents including topics, graphs and tables, and outputting an identification result of layout elements. According to the method, while the foreground content detection precision is improved, the reasoning speed of the document layout target detection model is increased, accurate identification of various elements in test papers and textbooks and exercises of different subjects is realized, and an efficient solution is provided for application such as question bank construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of document layout analysis, and in particular relates to an efficient document layout analysis method, system, electronic equipment and storage medium for question bank construction. Background Art

[0002] With the rapid development of educational technology, the importance of visual information extraction from document images in educational fields such as question bank construction, textbook digitization, and online answering is increasing. Document layout analysis, as a pre-step to document understanding, aims to identify and separate different layout structures and content areas from complex document images, such as titles, text, charts, formulas, etc., to provide necessary basic support for subsequent document information extraction and understanding tasks.

[0003] However, some current document layout analysis methods are general single-modal technologies based on convolutional neural networks and Transformers. In the paper "Li C, Guo R, Zhou J, et al. Pp-structurev2: A stronger document analysis system [J]. arXiv preprint arXiv: 2210.05391, 2022.", Li et al. proposed using PP-YOLOV2 as a detector for layout analysis, but because its pre-trained weights are based on general data sets, its performance in the field of document layout analysis is relatively general. In the paper "Zhao Z, Kang H, Wang B, et al. DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception [J]. arXiv preprint arXiv: 2410.12628, 2024.", Zhao et al. combined single-modal YOLO with document data pre-training for layout detection, but the YOLO method is limited by non-maximum suppression and it is difficult to further improve its accuracy. In addition, pre-trained multimodal technology has shown strong capabilities in document understanding and layout analysis. In the paper "Da C, Luo C, Zheng Q, et al. Vision grid transformer for document layout analysis [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2023: 19462-19472.", Da et al. performed pre-training by using high-parameter grid text encoding, but still faced significant speed bottlenecks when reasoning about document images, which made it difficult to meet the needs of actual applications. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide an efficient document layout analysis method, system, device and medium for question bank construction. The document layout analysis method can adapt to the complex layout structure and high noise environment in educational scenarios while ensuring high accuracy and high speed.

[0005] In order to achieve the first objective above, the present invention discloses an efficient document layout analysis method for question bank construction, the efficient document layout analysis method comprising the following steps:

[0006] S1, scanning document images of different subject test papers, textbook exercises, and books, and preprocessing to obtain non-tilted scanned document images;

[0007] S2, marking the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document dataset;

[0008] S3, using the document dataset for training to obtain a document layout object detection model;

[0009] S4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

[0010] Furthermore, the document layout object detection model includes a lightweight feature extraction network, a feature encoder and a detection head connected in sequence;

[0011] The lightweight feature extraction network includes a trunk path, a branch path and a dynamic multi-scale fusion module. We combine the idea of ​​cross-stage local networks and divide the original module with large parameters into lightweight trunk paths and branch paths, thereby improving the reasoning speed and accuracy of the document layout object detection model. The trunk path and the branch path extract shallow features from the input image respectively. The shallow features include edge features, texture features, and color features. The branch path and the trunk path use different network architectures to capture shallow features of the image from multiple scales and angles, ensuring feature complementarity and providing diverse expressions. The dynamic multi-scale fusion module integrates multi-scale information of shallow features from the trunk path and the branch path through a dynamic fusion strategy to obtain the final fused feature map X, which retains local details and global context information of the image at the feature level, thereby improving the accuracy and robustness of image analysis.

[0012] The feature encoder includes a multi-scale information interaction module and a dual convolution attention module connected in sequence. The multi-scale information interaction module includes 8 layers of multi-head attention for capturing information relationships of different frequencies in the feature map. The dual convolution attention module includes 2 convolutions and 2 attention mechanisms for simultaneously modeling global features and local features and removing document image noise. The feature encoder is denoted as a function G(), which is defined as follows:

[0013] The feature map X has c input channels, is the output feature map, which represents the feature representation after being processed by the multi-scale information interaction module and the dual convolution attention module;

[0014] The detection head outputs a feature map Perform the query selection operation with the minimum feature uncertainty and calculate the difference between the positioning prediction distribution P() and the classification prediction distribution C(). The formula is as follows:

[0015]

[0016] in, is the output feature map of the feature encoder, which represents the feature representation of the input image after being processed by the feature encoder. D represents the dimension of the feature. Represents feature encoder features The predicted distribution in the localization task, Represents feature encoder features In the prediction distribution of classification tasks, the symbol "||·||" means to find the 2-norm. is feature uncertainty, defined as the difference between the location prediction distribution and the classification prediction distribution, reflecting the uncertainty of the query feature. represents the prediction result of the document layout object detection model on the input data, y represents the true label of the input data, including the true category c and the true bounding box b, represents the output bounding box of the model in the positioning task, b represents the true bounding box of the input data, represents the output category of the model in the classification task, c represents the true category label of the input data, and L box () indicates the predicted bounding box The difference between the ground-truth bounding box b, L cls () indicates the predicted category The difference between the true category c, L box () and L cls () is defined as follows:

[0017]

[0018] Among them, x0, y0 represent the center coordinates of the bounding box, w0, h0 represent the width and height of the bounding box respectively, i represents the index of the bounding box parameter, and L s1 () means that when the absolute value of the error is less than 1, the loss is half the square of the error. When the absolute value of the error is greater than or equal to 1, the loss is the absolute value of the error minus 0.5. j represents the index of the category label parameter, and Y0 is the number of categories.

[0019] Further, given an input document feature map I, the first convolutional layer of the trunk receives the initial input document feature map I, the output of the first convolutional layer of the trunk is connected to the input of the second convolutional layer of the trunk, and the output of the second convolutional layer of the trunk is connected to the input of the third convolutional layer of the trunk; the first convolutional layer of the branch receives the initial input I, the output of the first convolutional layer of the branch is connected to the input of the second convolutional layer of the branch, and the output of the second convolutional layer of the branch is connected to the input of the third convolutional layer of the branch; the output of the third convolutional layer of the trunk and the output of the third convolutional layer of the branch are connected to the input of the first dynamic multi-scale fusion module;

[0020] The output of the first dynamic multi-scale fusion module is connected to the input of the fourth convolutional layer of the trunk and the input of the fourth convolutional layer of the branch, the output of the fourth convolutional layer of the trunk is connected to the input of the fifth convolutional layer of the trunk, the output of the fifth convolutional layer of the trunk is connected to the input of the sixth convolutional layer of the trunk, the output of the sixth convolutional layer of the trunk is connected to the input of the seventh convolutional layer of the trunk, and the output of the seventh convolutional layer of the trunk is connected to the input of the eighth convolutional layer of the trunk; the output of the fourth convolutional layer of the branch is connected to the input of the fifth convolutional layer of the branch; the input of the eighth convolutional layer of the trunk and the output of the fifth convolutional layer of the branch are connected to the input of the second dynamic multi-scale fusion module;

[0021] The output of the second dynamic multi-scale fusion module is connected to the input of the ninth convolutional layer of the trunk and the input of the sixth convolutional layer of the branch. The output of the ninth convolutional layer of the trunk is connected to the input of the tenth convolutional layer of the trunk; the output of the sixth convolutional layer of the branch is connected to the input of the seventh convolutional layer of the branch; the output of the tenth convolutional layer of the trunk and the output of the seventh convolutional layer of the branch are connected to the input of the third dynamic multi-scale fusion module, and the third dynamic multi-scale fusion module obtains the final fused feature map X.

[0022] Furthermore, the specific process of using the document dataset to train the document layout object detection model in step S3 is as follows:

[0023] S301. Initialize the document layout object detection model, set the number of training rounds, input image size to 1280 by 1280, select AdamW optimizer, which comes from the preprint paper "Decoupled Weight Decay Regularization" published by Loshchilov et al. in 2017. AdamW optimizer is a variant of Adam optimizer; Adam is an optimizer that combines momentum and adaptive learning rate. AdamW improves the optimization process by "decoupling" the weight decay in Adam, which can provide better generalization performance and converge faster in different types of deep learning tasks; set the batch size to 4, enable mosaic data enhancement, and the mosaic data enhancement method was proposed by Alexey Bochkovskiy in the literature "Ultra-Fast Object Detection for Autonomous Drivin” proposed that mosaic data enhancement is widely used in target detection tasks in computer vision, especially in processing small object detection and complex scenes. Its core idea is to increase data diversity and model robustness by stitching four images together to form a new image; and turn off the hybrid image enhancement in the last 10 training rounds to get the initial training configuration;

[0024] S302, randomly extracting a batch of image label pairs from the training set of the document layout analysis data set, and inputting them into the document layout object detection model for forward propagation to obtain predicted bounding box values;

[0025] S303, calculate the actual border box and the predicted border The IOU intersection over union ratio between them is calculated by the GIOU() function, which comes from the paper "Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression" published by the author Ronghang Hu at the Conference on Computer Vision and Pattern Recognition in 2020. As part of the loss calculation, it is used to guide the update of the document layout object detection model parameters. The result is as follows:

[0026]

[0027] Among them, box represents the actual border value, Represents the bounding box value predicted by the model, A represents the predicted bounding box area, B represents the true bounding box area, O represents the bounding box area, and the bounding box area contains the minimum rectangular box of A and B;

[0028] S304: Perform forward propagation on the validation set of the document layout analysis dataset to obtain the average precision mAP and overall loss Loss of this round of training, including IOU loss. Classification Loss Bounding Box Loss The formula is as follows:

[0029]

[0030] Among them, cls k Represents the true category label of 0 or 1, represents the predicted probability of the kth class, Y1 represents the total number of categories, and box v represents the vth parameter of the true bounding box, The vth parameter of the predicted bounding box is used. x1 and y1 represent the center coordinates of the bounding box, w1 and h1 represent the width and height of the bounding box respectively, and v represents the index of the bounding box parameter.

[0031] S305, the document layout object detection model evaluates the performance of the current training times and compares the best performance. If the performance does not improve and reaches the early stopping threshold, the training is stopped early to obtain the early stopping condition;

[0032] S306, the document layout object detection model saves the state of the current training times to obtain the saved document layout object detection model and training state;

[0033] S307, the learning rate is updated during the training process, and a new learning rate is obtained according to the learning rate decay strategy;

[0034] S308. The document layout object detection model uses the results of the current training times as a basis to continue the next round of training and obtain the starting state of the next round of training.

[0035] Furthermore, in step S4, the real image to be tested is input into the document layout object detection model for forward propagation to obtain the detection result of the document foreground content.

[0036] The second object of the present invention is to provide an efficient document layout analysis system for question bank construction, which is used to execute the efficient document layout analysis method for question bank construction. The efficient document layout analysis system comprises:

[0037] The scanning preprocessing module is used to scan the document images of different subject examination papers, textbook exercises, and books, and preprocess them to obtain non-tilted scanned document images;

[0038] The document annotation module is used to annotate the relative positions and category labels of titles and diagrams in the scanned document images to obtain a document dataset;

[0039] A model training module is used to train a document layout object detection model using a document dataset;

[0040] The image recognition module is used to input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

[0041] The third object of the present invention is to provide a computer device, including a processor and a memory for storing processor executable programs, when the processor executes the program stored in the memory, it implements the above-mentioned efficient document layout analysis method for question bank construction.

[0042] The fourth object of the present invention is to provide a storage medium storing a program, which, when executed by a processor, implements the above-mentioned efficient document layout analysis method for question bank construction.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention discloses an efficient document layout analysis method, system, device and medium for question bank construction, and proposes a document layout target detection model with both high speed and high accuracy. The model adopts a real-time DETR target detector to avoid the limitations of traditional non-maximum suppression, and improves the reasoning speed and generalization ability of the model through a lightweight feature extraction network and a multi-scale information fusion module. In addition, the encoder integrates a multi-scale information interaction module and a dual convolution-attention module to effectively deal with complex layouts and noise interference, and enhances the adaptability and robustness of the model in practical applications.

[0045] 2. Compared with the prior art, the present invention mainly analyzes the document layout of test papers and textbook exercises, and can effectively adapt to complex test papers and textbook layouts with noise interference. The technical solution of the present invention has strong subject versatility and can be applied to educational scenarios of different subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1It is a flow chart of an efficient document layout analysis method for question bank construction disclosed in Example 1 of the present invention;

[0048] Figure 2 It is a model structure diagram of an efficient document layout analysis method for question bank construction disclosed by the present invention;

[0049] Figure 3 It is a lightweight feature extraction network structure diagram disclosed in the present invention;

[0050] Figure 4 It is a schematic diagram of the detection results disclosed by the present invention;

[0051] Figure 5 It is a structural block diagram of an efficient document layout analysis system for question bank construction in Embodiment 3 of the present invention;

[0052] Figure 6 It is a structural block diagram of the computer device in Example 4 of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0055] Example 1

[0056] Figure 1 The present invention discloses a flow chart of an efficient document layout analysis method for question bank construction, such as Figure 1 As shown, the present embodiment discloses an efficient document layout analysis method for question bank construction, including the following steps: scanning and preprocessing document images, annotating scanned document images, using document data set training, and document layout target detection model to detect document images, as follows:

[0057] T1. Scan the document images of different subject test papers, textbook exercises, and books, and pre-process them to obtain non-tilted scanned document images;

[0058] T2, annotate the relative position and category label of each element in the scanned document image to obtain a document dataset;

[0059] T3. Use the document dataset to train a document layout object detection model. The architecture of the document layout object detection model is as follows: Figure 2 As shown in the figure, it includes a lightweight feature extraction network, a feature encoder, and a detection head connected in sequence. The trunk path combines the idea of ​​a cross-stage local network to achieve richer gradient combinations while significantly reducing computing resources. The specific approach is to change the 6-fold stacking of the original module to 3-fold stacking, which is connected in parallel with the branch path. This design makes the entire model more lightweight without reducing performance. In order to solve the problem of differences in feature extraction between the trunk path and the branch path, a dynamic alignment fusion module is designed to achieve adaptive fusion of features. The coordinate attention mechanism is introduced in the convolution block of the last layer of the trunk path to further reduce redundant information interference and improve adaptability to complex document layouts. The convolution layers included in the lightweight feature extraction network are shown in the figure. Figure 3 As shown; In addition, by introducing dual convolution and dual attention modules, the complex layout, diverse text and graphic elements in the document image, as well as noise interference including blur and shadow caused by scanning or shooting are effectively dealt with; To verify the effectiveness of the document layout target detection model, the present invention is comprehensively evaluated on the private dataset EED and the public dataset CDLA. From the results in Table 1, it can be seen that on the private dataset EED, the proposed document layout target detection model DLA-DETR has achieved the best performance in both AP50 and mAP indicators, and is significantly better than the existing document layout analysis methods in terms of GFLOPs. Although DLA-DETR is slightly lower than YOLOv10m in inference speed FPS, it still maintains a high speed advantage. Compared with the baseline model RT-DETR-L, DLA-DETR improves accuracy by 0.5%, while GFLOPs and parameters are reduced by 54.2% and 35.4%, respectively. From the results in Table 2, it can be seen that on the public dataset CDLA, the present invention compares DLA-DETR with the existing PP-YOLOv2, PP-PicoDet-LCNet series and DocLayout-YOLO single-modal models, and achieves the best performance in inference speed, AP50 and mAP.

[0060] Table 1. Comparison of the accuracy and inference speed of the present invention with other methods on the EED dataset

[0061] Model AP50(%) mAP(%) GFLOPs(G) FPS YOLOv5m 91.3 74.6 64.0 34.4 YOLOv5l 92.3 75.0 135.3 19.7 YOLOv8m 91.7 74.6 78.8 33.6 YOLOv10m 90.8 74.2 64.0 44.4 YOLO11m 92.4 74.9 87.3 31.a DocLayout-YOLO 92.7 74.4 68.0 29.8 DocLayout-YOLO 92.8 74.8 127.4 28.7 RT-DETR-L(Benchmark) 92.9 74.9 103.5 21.1 Document layout object detection model 93.4 75.1 47.4 35.9

[0062] Table 2. Comparison of the accuracy and inference speed of the present invention with other methods on the CDLA dataset

[0063] Model mAP(%) FPS PP-YOLOv2 84.7 34.4 PP-PicoDet-LCNet2.5x 87.8 1 9.7 PP-PicoDet-LCNet1.0x 84.5 33.6 PP-PicoDet-LCNet1.0x+FGD 89.9 29.4 Document layout object detection model 91.7 35.7

[0064] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of layout elements. After inference, the document layout object detection model outputs the recognition results of layout elements including questions and charts. The result example is as follows: Figure 4 shown.

[0065] Example 2

[0066] Based on the efficient document layout analysis method for question bank construction disclosed in Example 1, this embodiment continues to refer to steps T1 to T4 of the efficient document layout analysis method for question bank construction disclosed in Example 1. Figure 1 As shown, the method comprises the following steps:

[0067] T1. Scan the document images of different subject test papers, textbook exercises, and books, and pre-process them to obtain non-tilted scanned document images;

[0068] T2, annotate the relative position and category label of each element in the scanned document image to obtain a document dataset;

[0069] T3. Use the document dataset to train and obtain a document layout object detection model. The architecture of the document layout object detection model is as follows: Figure 2 As shown, it includes a lightweight feature extraction network, a feature encoder and a detection head connected sequentially;

[0070] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0071] Among them, the module ablation experiment of the document layout detection model in step T3 is shown in Table 3. On the dataset EED, a comparative experiment was conducted on HG-CSPNet* without adding the attention mechanism, the multi-scale information interaction module, the CA-HGBlock with adding the attention mechanism, and the double convolution and double attention modules. Although the accuracy of HG-CSPNet* without adding the attention mechanism was slightly reduced, the reasoning speed was increased by two times; after introducing the information interaction module or the attention mechanism module, the accuracy and reasoning speed were further improved. Combining the two modules, the accuracy is on par with the baseline model. Finally, after adding the double convolution and double attention modules, the accuracy and speed are surpassed by the baseline model. Then, the various parameters of the document layout target detection model are initialized, and the document layout analysis dataset is used for training to obtain the pre-trained document layout target detection model.

[0072] Table 3. Comparison of the impact of different modules on the overall architecture of the present invention on the EED dataset

[0073] HG-CSPNet* Hilo-TF CA-HGB lock DCAM AP50 mAP - - - - 92.9 74.9 use - - - 91.7 73.6 use - use - 92.3 74.0 use use - - 92.5 74.3 use use use - 92.9 74.8 use use use use 93.4 75.1

[0074] In summary, Examples 1 and 2 of the present invention propose an efficient document layout analysis method for question bank construction. First, a high-quality document image dataset is constructed through a standardized process, and combined with manual annotation and verification steps to ensure the accuracy and consistency of the data. Subsequently, the dataset is trained and inferred using the designed document layout target detection model, and the advantages of the model in processing complex document layouts, diverse texts, and graphic elements are fully verified through the examples. Through the designed lightweight feature extraction network, multi-scale information fusion module, and dual convolution and dual attention mechanisms, the network can efficiently parse the document layout structure, provide technical support for applications such as question bank construction and online answering in educational scenarios, and improve the automation level and processing efficiency of the task.

[0075] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0076] Example 3

[0077] like Figure 5 As shown, this embodiment provides an efficient document layout analysis system for question bank construction, which includes: a scanning preprocessing module 501, a document annotation module 502, a model training module 503, and an image recognition module 504. The specific functions of each module are as follows:

[0078] The scanning preprocessing module 501 is used to scan the document images of different subject examination papers, textbook exercises, and books, and preprocess them to obtain the scanned document images without tilt;

[0079] The document annotation module 502 is used to annotate the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document data set;

[0080] Model training module 503, used to obtain a document layout object detection model by training with a document dataset;

[0081] The image recognition module 504 is used to input the image to be detected into the trained document layout object detection model, detect the foreground content including the title, graphics and tables, and output the recognition result of the layout elements.

[0082] Example 4

[0083] This embodiment provides a computer device, which may be a computer, such as Figure 6 As shown, a processor 602, a memory, an input device 603, a display 604 and a network interface 605 connected via a system bus 601 are provided. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 606 and an internal memory 607. The non-volatile storage medium 606 stores an operating system, a computer program and a database. The internal memory 607 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 602 executes the computer program stored in the memory, an efficient document layout analysis method for question bank construction proposed in the above embodiment 1 is implemented. The efficient document layout analysis method for question bank construction includes the following steps:

[0084] T1. Scan the document images of different subject test papers, textbook exercises, and books, and pre-process them to obtain non-tilted scanned document images;

[0085] T2, annotate the relative position and category label of each element in the scanned document image to obtain a document dataset;

[0086] T3, using the document dataset for training to obtain a document layout object detection model;

[0087] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0088] Example 5

[0089] This embodiment provides a storage medium, which is a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, an efficient document layout analysis method for question bank construction in the above embodiment 1 is implemented. The efficient document layout analysis method for question bank construction includes the following steps:

[0090] T1. Scan the document images of different subject test papers, textbook exercises, and books, and pre-process them to obtain non-tilted scanned document images;

[0091] T2, annotate the relative position and category label of each element in the scanned document image to obtain a document dataset;

[0092] T3, using the document dataset for training to obtain a document layout object detection model;

[0093] T4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements. After inference, the document layout object detection model outputs the recognition results of the layout elements including questions and charts.

[0094] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An efficient document layout analysis method for question bank construction, characterized in that: The efficient document layout analysis method comprises the following steps: S1, scanning document images of different subject test papers, textbook exercises, and books, and preprocessing to obtain non-tilted scanned document images; S2, marking the relative positions and category labels of titles and diagrams in the scanned document image to obtain a document dataset; S3, using the document dataset for training to obtain a document layout object detection model; S4. Input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

2. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: The document layout object detection model includes a lightweight feature extraction network, a feature encoder and a detection head connected in sequence; The lightweight feature extraction network includes a trunk path, a branch path and a dynamic multi-scale fusion module, wherein the trunk path and the branch path respectively extract shallow features from the input image, and the shallow features include edge features, texture features, and color features; the dynamic multi-scale fusion module integrates multi-scale information of shallow features from the trunk path and the branch path through a dynamic fusion strategy to obtain a final fused feature map X; The feature encoder includes a multi-scale information interaction module and a dual convolution attention module connected in sequence. The multi-scale information interaction module includes 8 layers of multi-head attention for capturing information relationships of different frequencies in the feature map. The dual convolution attention module includes 2 convolutions and 2 attention mechanisms for simultaneously modeling global features and local features and removing document image noise. The feature encoder is denoted as a function G(), which is defined as follows: where the feature map x has c input channels, is the output feature map, which represents the feature representation after being processed by the multi-scale information interaction module and the dual convolution attention module; The detection head outputs a feature map Perform the query selection operation with the minimum feature uncertainty and calculate the difference between the positioning prediction distribution P() and the classification prediction distribution C(). The formula is as follows: in, is the output feature map of the feature encoder, which represents the feature representation of the input image after being processed by the feature encoder. D represents the dimension of the feature. Represents feature encoder features The predicted distribution in the localization task, Represents feature encoder features In the prediction distribution of classification tasks, the symbol "||·||" means to find the 2-norm. is feature uncertainty, defined as the difference between the location prediction distribution and the classification prediction distribution, reflecting the uncertainty of the query feature. represents the prediction result of the document layout object detection model on the input data, y represents the true label of the input data, including the true category c and the true bounding box b, represents the output bounding box of the model in the positioning task, b represents the true bounding box of the input data, represents the output category of the model in the classification task, c represents the true category label of the input data, and L box () indicates the predicted bounding box The difference between the ground-truth bounding box b, L cls () indicates the predicted category The difference between the true category c, L box () and L cls () is defined as follows: Among them, x0, y0 represent the center coordinates of the bounding box, w0, h0 represent the width and height of the bounding box respectively, i represents the index of the bounding box parameter, and L s1 () means that when the absolute value of the error is less than 1, the loss is half the square of the error. When the absolute value of the error is greater than or equal to 1, the loss is the absolute value of the error minus 0.

5. j represents the index of the category label parameter, and Y0 is the number of categories.

3. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: Given an input document feature map I, the first convolutional layer of the backbone receives the initial input document feature map I, the output of the first convolutional layer of the backbone is connected to the input of the second convolutional layer of the backbone, and the output of the second convolutional layer of the backbone is connected to the input of the third convolutional layer of the backbone; The first convolution layer of the branch receives the initial input I, the output of the first convolution layer of the branch is connected to the input of the second convolution layer of the branch, and the output of the second convolution layer of the branch is connected to the input of the third convolution layer of the branch; the output of the third convolution layer of the trunk and the output of the third convolution layer of the branch are connected to the input of the first dynamic multi-scale fusion module; The output of the first dynamic multi-scale fusion module is connected to the input of the fourth convolutional layer of the trunk and the input of the fourth convolutional layer of the branch, the output of the fourth convolutional layer of the trunk is connected to the input of the fifth convolutional layer of the trunk, the output of the fifth convolutional layer of the trunk is connected to the input of the sixth convolutional layer of the trunk, the output of the sixth convolutional layer of the trunk is connected to the input of the seventh convolutional layer of the trunk, and the output of the seventh convolutional layer of the trunk is connected to the input of the eighth convolutional layer of the trunk; the output of the fourth convolutional layer of the branch is connected to the input of the fifth convolutional layer of the branch; the input of the eighth convolutional layer of the trunk and the output of the fifth convolutional layer of the branch are connected to the input of the second dynamic multi-scale fusion module; The output of the second dynamic multi-scale fusion module is connected to the input of the ninth convolutional layer of the trunk and the input of the sixth convolutional layer of the branch, and the output of the ninth convolutional layer of the trunk is connected to the input of the tenth convolutional layer of the trunk; The output of the branched sixth convolutional layer is connected to the input of the branched seventh convolutional layer; the output of the trunk tenth convolutional layer and the output of the branched seventh convolutional layer are connected to the input of the third dynamic multi-scale fusion module, and the third dynamic multi-scale fusion module obtains the final fused feature map x.

4. The efficient document layout analysis method for question bank construction according to claim 1 is characterized in that: The specific process of using the document dataset to train the document layout object detection model in step S3 is as follows: S301, initialize the document layout object detection model, set the number of training rounds, input image size as 1280 times 1280, select AdamW optimizer, set batch size as 4, enable mosaic enhancement and turn it off in the last 10 training rounds, and enable hybrid image enhancement to obtain the initial training configuration; S302, randomly extracting a batch of image label pairs from the training set of the document layout analysis data set, and inputting them into the document layout object detection model for forward propagation to obtain predicted bounding box values; S303, calculate the actual border box and the predicted border The IOU intersection-and-union ratio between them is calculated by the GIOU() function. As part of the loss calculation, it is used to guide the update of the document layout object detection model parameters. The result is as follows: Among them, box represents the actual border value, Represents the bounding box value predicted by the model, A represents the predicted bounding box area, B represents the true bounding box area, O represents the bounding box area, and the bounding box area contains the minimum rectangular box of A and B; S304: Perform forward propagation on the validation set of the document layout analysis dataset to obtain the average precision mAP and overall loss Loss of this round of training, including IOU loss. Classification Loss Bounding Box Loss The formula is as follows: Among them, cls k Represents the true category label of 0 or 1, represents the predicted probability of the kth class, Y1 represents the total number of categories, and box v represents the vth parameter of the true bounding box, The vth parameter of the predicted bounding box is used. x1 and y1 represent the center coordinates of the bounding box, w1 and h1 represent the width and height of the bounding box respectively, and v represents the index of the bounding box parameter. S305, the document layout object detection model evaluates the performance of the current training times and compares the best performance. If the performance does not improve and reaches the early stopping threshold, the training is stopped early to obtain the early stopping condition; S306, the document layout object detection model saves the state of the current training times to obtain the saved document layout object detection model and training state; S307, the learning rate is updated during the training process, and a new learning rate is obtained according to the learning rate decay strategy; S308. The document layout object detection model uses the results of the current training times as a basis to continue the next round of training and obtain the starting state of the next round of training.

5. The efficient document layout analysis method for question bank construction according to claim 2 is characterized in that: In step S4, the real image to be tested is input into the document layout object detection model for forward propagation to obtain the detection result of the document foreground content.

6. An efficient document layout analysis system for question bank construction, used to execute the efficient document layout analysis method for question bank construction as described in any one of claims 1 to 5, characterized in that: The efficient document layout analysis system comprises: The scanning preprocessing module is used to scan the document images of different subject examination papers, textbook exercises, and books, and preprocess them to obtain non-tilted scanned document images; The document annotation module is used to annotate the relative positions and category labels of titles and diagrams in the scanned document images to obtain a document dataset; A model training module is used to train a document layout object detection model using a document dataset; The image recognition module is used to input the image to be detected into the trained document layout object detection model, detect the foreground content including titles, graphics and tables, and output the recognition results of the layout elements.

7. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, it implements the efficient document layout analysis method for question bank construction as described in any one of claims 1 to 5.

8. A storage medium storing a program, characterized in that: When the program is executed by a processor, the efficient document layout analysis method for question bank construction described in any one of claims 1 to 5 is implemented.

Images