Layout structure analysis method, device, electronic device and storage medium

Abstract

The present invention provides a layout structure analysis method, device, electronic device and storage medium, wherein the method comprises: extracting image features of a layout image to be analyzed; based on the image features, performing layout element detection on the layout image to obtain position information and element categories of each element in the layout image; based on the image features, performing text line segmentation on the layout image to obtain position information of each text line in the layout image; based on the position information and element categories of each element in the layout image, and the position information of each text line, performing layout structure analysis on the layout image. The layout structure analysis method, device, electronic device and storage medium provided by the present invention can simultaneously perform comprehensive and accurate analysis on the logical structure and layout structure of a layout image, especially for layout structure analysis in scenes with complex layout structures.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language understanding, and in particular to a layout structure analysis method, device, electronic device and storage medium. Background Art

[0002] By analyzing the layout structure of teaching aids and test papers, we can quickly process educational graphic data and obtain a large amount of high-quality annotated data.

[0003] Some existing layout structure analysis methods use traditional image processing technology and still require manual operation by users to accurately analyze the structural information within the image and text, which cannot reach the level of automation and intelligence.

[0004] There are also some graphic layout analysis methods that use machine learning technology to implement them, but they focus more on multiple measurements than a single capability, do not have comprehensive layout analysis capabilities, are not easy to use, and limit their usage scenarios. Summary of the invention

[0005] The present invention provides a layout structure analysis method, device, electronic device and storage medium, which are used to solve the defects of the prior art that multiple measurements are more important than a single capability and the comprehensive layout analysis capability is not possessed.

[0006] The present invention provides a layout structure analysis method, comprising:

[0007] Extracting image features of the layout image to be analyzed;

[0008] Based on the image features, performing layout element detection on the layout image to obtain position information and element category of each element in the layout image;

[0009] Based on the image features, segment the layout image into text lines to obtain position information of each text line in the layout image;

[0010] Based on the position information and element category of each element in the layout image, and the position information of each text line, the layout image is subjected to layout structure analysis.

[0011] According to a layout structure analysis method provided by the present invention, the layout element detection is performed on the layout image based on the image features to obtain the position information and element category of each element in the layout image, including:

[0012] Based on the image features, performing layout element detection on the layout image to obtain position information, element category and element confidence of each candidate element in the layout image;

[0013] Based on the position information of each candidate element, determine the candidate elements at the same position from the candidate elements, and filter the candidate elements at the same position based on the element categories and element confidences of the candidate elements at the same position;

[0014] Based on the element confidence of the filtered candidate elements, the position information and element category of each element in the layout image are determined.

[0015] According to a layout structure analysis method provided by the present invention, the filtering of candidate elements at the same position based on the element category and element confidence of the candidate elements at the same position includes:

[0016] When the element category of the candidate elements at the same position is an easily confused category, the candidate elements at the same position except for the candidate element with the highest element confidence are deleted, and based on the element confidence of each candidate element at the same position, the element confidence of the candidate element with the highest element confidence at the same position is updated.

[0017] According to a layout structure analysis method provided by the present invention, the layout structure analysis of the layout image is performed based on the position information and element category of each element in the layout image, and the position information of each text line, including:

[0018] Determining the relationship between the elements in the layout image based on the position information and element category of each element in the layout image;

[0019] Based on the position information of each element in the layout image and the position information of each text line, the text lines are merged or segmented to obtain the text line information of each element in the layout image;

[0020] Based on the relationship between the elements in the layout image and the text line information of the elements in the layout image, the layout structure analysis is performed on the layout image.

[0021] According to a layout structure analysis method provided by the present invention, the relationship between the elements in the layout image is determined based on the position information and element category of each element in the layout image, and then further includes:

[0022] Determining the same category elements belonging to adjacent column elements based on the relationship between the elements in the layout image;

[0023] Based on the element position and / or the element type contained in the same category elements belonging to the latter column elements in the adjacent column elements, the attribution type of the same category elements belonging to the adjacent column elements is determined, and the same category elements belonging to the adjacent column elements are merged when the attribution type is the same.

[0024] According to a layout structure analysis method provided by the present invention, the step of extracting image features of a layout image to be analyzed includes:

[0025] Based on the multi-scale sample image, multi-scale feature extraction is performed on the layout image to be analyzed to obtain image features of the layout image.

[0026] According to a layout structure analysis method provided by the present invention, the method extracts image features of a layout image to be analyzed, performs layout element detection on the layout image based on the image features to obtain position information and element categories of each element in the layout image, and performs text line segmentation on the layout image based on the image features to obtain position information of each text line in the layout image, including:

[0027] Based on the layout structure analysis model, performing layout element detection and text line segmentation on the layout image;

[0028] The layout structure analysis model is trained based on a first sample image marked with element positions and category labels, and a second sample image marked with text line labels.

[0029] The present invention also provides a layout structure analysis device, comprising:

[0030] A feature extraction unit, used to extract image features of the layout image to be analyzed;

[0031] An element detection unit, used to perform layout element detection on the layout image based on the image features, and obtain position information and element category of each element in the layout image;

[0032] A text line segmentation unit, used to perform text line segmentation on the layout image based on the image features to obtain position information of each text line in the layout image;

[0033] The layout structure analysis unit is used to perform layout structure analysis on the layout image based on the position information and element category of each element in the layout image, and the position information of each text line.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned layout structure analysis methods are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described layout structure analysis methods.

[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned layout structure analysis methods are implemented.

[0037] The layout structure analysis method, device, electronic device and storage medium provided by the present invention perform layout structure analysis on the layout image by utilizing the position information and element category of each element in the layout image, as well as the position information of each text line. Compared with the layout analysis in the prior art that only implements a single capability, this method can simultaneously perform a comprehensive and accurate analysis of the logical structure and layout structure of the layout image, especially for layout structure analysis in scenarios with complex layout structures.

[0038] In addition, this method can utilize the extracted image features to realize information sharing between layout element detection and text line segmentation multi-branch integration, achieving comprehensive and accurate layout analysis while reducing the amount of calculation and improving layout analysis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is one of the flow charts of the layout structure analysis method provided by the present invention;

[0041] Figure 2 is a flow chart of step 120 in the layout structure analysis method provided by the present invention;

[0042] Figure 3 is a flow chart of step 140 in the layout structure analysis method provided by the present invention;

[0043] Figure 4 It is a flowchart of the method for merging elements of the same category provided by the present invention;

[0044] Figure 5 This is the second flow chart of the layout structure analysis method provided by the present invention;

[0045] Figure 6 It is a structural schematic diagram of the layout structure analysis device provided by the present invention;

[0046] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In recent years, artificial intelligence technology has developed rapidly, and its applications have spread across multiple industries such as security, finance, and education, bringing great impact and convenience to people's production, work, and even life. Among them, the field of education is deeply affected by artificial intelligence technology. Among them, the sub-fields of artificial intelligence: computer vision and natural language understanding technology are widely used in the field of education, such as photo-cutting technology, text detection and recognition technology, photo-searching technology, and human-based question-pushing technology. Limited by the high reliance of artificial intelligence technology on big data, there is also an important prerequisite for artificial intelligence assistants in the field of education - a large amount of high-quality data. At present, most of the high-quality data on the market is obtained by manual annotation. The disadvantage of this annotation method is that it is costly and time-consuming. Therefore, relying on artificial intelligence technology to automatically and quickly process a large amount of high-quality data is a very potential application scenario.

[0049] Graphic data in the field of education mainly come from supplementary teaching materials and test papers, which often have a very complex layout structure and may include the following elements: page, single column, title, header and footer, page number, title, image, table, etc. At the same time, there are many types of questions, such as: multiple choice questions, fill-in-the-blank questions, true or false questions, answer questions, calculation questions, drawing questions, etc., and the questions will have nested sub-elements such as question numbers, sub-questions, options, images, tables, etc. Different question types will correspond to various types of answers and analysis. At the same time, elements such as titles, headers and footers, page numbers, and titles are composed of text.

[0050] In order to automatically and quickly complete the processing of educational graphic data, the layout structure analysis of educational graphics becomes particularly important, and the high layout complexity is the biggest difficulty in layout structure analysis.

[0051] Most existing related technical solutions adopt a separation method to realize layout structure analysis, such as using target detection technology to realize the photo-taking function and using semantic segmentation technology to realize the text line segmentation function. The disadvantage of this implementation is that it does not have comprehensive layout analysis capabilities and most of them can only realize one analysis capability, which greatly limits the application scenarios.

[0052] Based on this, an embodiment of the present invention provides a layout structure analysis method, which can be applied not only to analysis scenarios of educational graphic data with high layout complexity, but also to layout structure analysis of other graphic data, such as resumes or newspapers, etc. The embodiment of the present invention does not make specific limitations on this.

[0053] Figure 1 One of the flow charts of the layout structure analysis method provided by the present invention is as follows: Figure 1 As shown, the method includes:

[0054] Step 110, extracting image features of the layout image to be analyzed.

[0055] Specifically, the layout image to be analyzed refers to an image that needs to be subjected to layout structure analysis, for example, it may be an image of a test paper or a supplementary teaching book.

[0056] The layout image here may be a color image or a grayscale image, and the embodiment of the present invention does not limit the specific form of the layout image.

[0057] The layout image can be scanned by a scanner, or taken by a high-definition camera, mobile device, etc. It can also be an image downloaded from the Internet, or an image received from a device, or an image in a video. The embodiment of the present invention does not limit the image source of the layout image.

[0058] The size scale of the layout image can be dynamically changed within a preset range. For example, the multi-scale change range of the layout image can be 800pix*1280pix to 1600pix*2560pix.

[0059] Image features may include spatial relationship features and semantic features of the layout image, wherein the spatial relationship features may characterize the spatial position or relative direction relationship between multiple targets segmented from the layout image, for example, the spatial relationship features may characterize the position information features of each element and / or each text line in the layout image; the semantic features may characterize the content contained in the layout image, for example, they may characterize text, chart or image features.

[0060] Image feature extraction of layout images can be achieved through feature extraction algorithms, for example, it can be a Histogram of Oriented Gradient (HOG) feature extraction algorithm or a Local Binary Pattern (LBP) feature extraction algorithm, etc.; of course, feature extraction can also be performed through convolutional neural networks.

[0061] In some preferred implementations, a feature pyramid network (FPN) structure can be used for multi-scale feature extraction, which can well express the high resolution of low-level features and the high semantic information of high-level features of the layout image.

[0062] Step 120, based on the image features, perform layout element detection on the layout image to obtain the position information and element category of each element in the layout image.

[0063] Specifically, after the image features of the layout image to be analyzed are extracted, layout element detection can be performed on the layout image based on the image features.

[0064] Here, elements refer to elements with a certain logical structure that can be extracted from the layout image, such as pages, titles, page numbers, topics, options, etc. Elements can be expressed in the form of text, charts and / or images, etc. For example, a title can be composed of text, and a topic can be composed of text and images.

[0065] The position information of each element refers to the regional position of each element in the layout image. For example, the position information of each element in the layout image can be represented by a rectangular frame.

[0066] Element categories can specifically be pages, titles, page numbers, topics, options, etc.

[0067] The object detection algorithm can be used to detect the elements of the layout image, such as the Cascade RCNN algorithm and the YOLO algorithm. First, the candidate area is obtained, and then the candidate area is classified.

[0068] In some complex layout structure scenarios, for example, there may be similar question types (such as calculation questions and solution questions) in scanned teaching aids, and the predicted element categories may be inaccurate. An improved adaptive non-maximum suppression (NMS) strategy can be used for filtering to improve the accuracy of layout element detection.

[0069] Step 130, based on the image features, segment the layout image into text lines to obtain the position information of each text line in the layout image.

[0070] Specifically, the image features extracted in step 110 may be used to segment the layout image into text lines, thereby obtaining the position information of each text line in the layout image.

[0071] Here, the position information of each text line may also be represented by a rectangular frame.

[0072] Text line segmentation of layout images can be achieved by using text detection algorithms, such as DBNet (Real-time Scene Text Detection with Differentiable Binarization), PSENet or PANNet.

[0073] In the scenario of scanning teaching aids, the text may be particularly dense, with small gaps between multiple lines, and mathematical formulas often appear, which increases the difficulty of detection. It is preferred to use the DBNet text detection algorithm, which has a better effect on segmenting dense text lines.

[0074] It should be noted that the embodiment of the present invention does not limit the execution order of step 120 and step 130, and they can be executed simultaneously.

[0075] Step 140, performing layout structure analysis on the layout image based on the position information and element category of each element in the layout image, as well as the position information of each text line.

[0076] Specifically, considering the element category of each element, the logical structure between the elements can be reflected, for example, the layout contains a single column, the single column contains the question type, the question type contains the question number, etc. The position information of each element and the position information of each text line can reflect the text line information contained in each element in the layout, that is, the layout structure of each element. For example, a single column element is composed of multiple text lines of the question type element, while the title element and the header and footer element are equivalent to a single text line.

[0077] Therefore, the layout structure analysis of the layout image can be performed by comprehensively considering the position information and element category of each element, as well as the position information of each text line, so that the logical structure and layout structure of the layout image can be comprehensively and accurately analyzed at the same time.

[0078] The layout structure analysis method provided in the embodiment of the present invention performs layout structure analysis on the layout image by comprehensively analyzing the position information and element category of each element in the layout image, as well as the position information of each text line. Compared with the layout analysis in the prior art that only implements a single capability, this method can simultaneously perform a comprehensive and accurate analysis of the logical structure and layout structure of the layout image, especially for layout structure analysis in scenarios with complex layout structures.

[0079] In addition, this method can utilize the extracted image features to realize information sharing between layout element detection and text line segmentation multi-branch integration, achieving comprehensive and accurate layout analysis while reducing the amount of calculation and improving layout analysis efficiency.

[0080] Based on the above embodiments, Figure 2is a flow chart of step 120 in the layout structure analysis method provided by the present invention, such as Figure 2 As shown, step 120 specifically includes:

[0081] Step 121, based on the image features, performing layout element detection on the layout image to obtain the position information, element category and element confidence of each candidate element in the layout image;

[0082] Step 122, based on the position information of each candidate element, determine the candidate elements at the same position from the candidate elements, and filter the candidate elements at the same position based on the element categories and element confidences of the candidate elements at the same position;

[0083] Step 123, based on the element confidence of the filtered candidate elements, determine the position information and element category of each element in the layout image.

[0084] Specifically, a target detection algorithm can be used to detect layout elements on a layout image to obtain the position information, element category and element confidence of each candidate element in the layout image. The candidate element can be a set of candidate elements obtained through element detection.

[0085] Based on the location information of each candidate element, the candidate elements at the same location can be obtained. Due to the interference of similar element categories, the element categories predicted by the candidate elements at the same location may belong to the same element category or different element categories. If the predicted elements belong to different element categories, the confidence of the predicted multiple elements will not be too high.

[0086] Considering that if the traditional adaptive non-maximum suppression (NMS) strategy is used to filter features within each feature category, it is impossible to completely filter out features of different categories; if feature filtering is performed across categories, some nested categories will be filtered out, and the confidence of multiple features predicted at the same position after filtering will still be low.

[0087] Therefore, the element confidence and element category of the candidate elements at the same position can be considered at the same time to filter the candidate elements at the same position. When filtering elements, it is sensitive not only to the element confidence but also to the element category.

[0088] For example, the feature category at the same position may be determined based on the feature confidence and feature category of the candidate features at the same position; and the feature confidence at the same position may be determined based on the feature confidence of each candidate feature at the same position.

[0089] After filtering the candidate elements at the same position, the position information and element category of each element in the layout image can be determined according to the element confidence of the filtered candidate elements. For example, the elements with element confidence lower than the preset confidence threshold can be filtered out, and only the elements with element confidence higher than the preset confidence threshold are retained, thereby obtaining the position information and element category of each element in the layout image.

[0090] The layout structure analysis method provided by the embodiment of the present invention takes into account the element confidence and element category of the candidate elements at the same position when filtering the candidate elements at the same position, thereby improving the accuracy of element detection.

[0091] Based on any of the above embodiments, in step 122, the candidate elements at the same position are filtered based on the element categories and element confidences of the candidate elements at the same position, specifically including:

[0092] When the feature category of the candidate elements at the same position is an easily confused category, the candidate elements at the same position except for the candidate element with the highest feature confidence are deleted, and the feature confidence of the candidate element with the highest feature confidence at the same position is updated based on the feature confidence of each candidate element at the same position.

[0093] Specifically, considering that in some complex layout structure scenarios, for example, there will be similar question types (such as calculation questions and solution questions) in the scanned teaching aids, multiple question type elements of different categories will be predicted at the same position. And because similar question types are easy to confuse, the target scores of multiple question types predicted at the same position will not be too high. Therefore, when the element category of the candidate elements at the same position is an easily confused category, the element filtering can be performed as follows:

[0094] First, the candidate elements except the candidate element with the highest element confidence at the same position are deleted. That is, only the candidate element with the highest element confidence is retained, and the element category of the candidate element with the highest element confidence can be used as the element category at the same position.

[0095] Then, based on the element confidences of the candidate elements at the same position, the element confidence of the candidate element with the highest element confidence at the same position is updated.

[0096] For example, the element confidences of the candidate elements at the same position may be fused, and the fused element confidences may be used as the element confidences at the same position.

[0097] In one embodiment, the fused element confidence can be expressed as:

[0098] P out =min(1,P max +0.5*Pmin )

[0099] Among them, P out Represents the confidence of the elements after fusion, P max and P min They respectively represent the maximum and minimum confidence levels of the candidate elements.

[0100] The method provided by the embodiment of the present invention further improves the accuracy of feature detection by filtering duplicate feature detection results by updating the feature category and feature confidence at the same position when the feature category of the candidate feature at the same position is an easily confused category.

[0101] Based on any of the above embodiments, Figure 3 is a flow chart of step 140 in the layout structure analysis method provided by the present invention, such as Figure 3 As shown, step 140 specifically includes:

[0102] Step 141, determining the relationship between the elements in the layout image based on the position information and element category of each element in the layout image;

[0103] Step 142, based on the position information of each element in the layout image and the position information of each text line, each text line is merged or segmented to obtain text line information of each element in the layout image;

[0104] Step 143, performing layout structure analysis on the layout image based on the relationship between the elements in the layout image and the text line information of the elements in the layout image.

[0105] Specifically, when performing layout structure analysis on a layout image, the relationship between various elements and the relationship between various elements and various text lines may be considered simultaneously.

[0106] First, we can aggregate the elements in the layout image based on the positional relationship to restore the inclusion and inclusion relationship, such as the whole page contains the title, single column and page number; the single column contains the question type; the question type contains the question number, image and table, thereby forming a parent node-child node connection relationship.

[0107] Then, the text lines are aggregated and segmented using the relationship between each element and each text line. For example, if a question type contains multiple text lines, the multiple text lines are aggregated into multiple sub-nodes of the question type. If multiple question numbers or answers belong to the same text line, the text line needs to be accurately segmented according to the position of the question number or answer, so that the sub-node of the question number or answer is an accurate text sub-node.

[0108] Finally, according to the relationship between the elements in the layout image and the text line information of each element in the layout image, the layout image is analyzed for layout structure. Specifically, a tree connection relationship can be constructed, and it can be written into different formats for storage as needed, such as Xml format or Json format file.

[0109] The method provided by the embodiment of the present invention performs post-processing by fusing various elements and text lines, thereby performing structural analysis on the layout image, and finally obtaining complete layout image structural information.

[0110] Based on any of the above embodiments, Figure 4 is a flow chart of the method for merging elements of the same category provided by the present invention, such as Figure 4 As shown, after step 141, the following steps are also included:

[0111] Step 410, based on the relationship between the elements in the layout image, determine the same category elements belonging to adjacent column elements;

[0112] Step 420, based on the element position and / or the included element category of the same category elements belonging to the latter column elements in the adjacent column elements, determine the belonging category of the same category elements belonging to the adjacent column elements, and merge the same category elements belonging to the adjacent column elements when the belonging categories are the same.

[0113] Specifically, considering some special scenarios, for example, when the question or answer analysis is separated by two columns or two pages, elements of the same type need to be merged.

[0114] First, determine the same category elements belonging to adjacent columns, and then determine whether they belong to the same category based on the position of the same category elements in the latter column and / or the element types contained. If they belong to the same category, they are merged; otherwise, they are not merged.

[0115] In a specific example, if the question or answer analysis is divided into two columns or two pages, first determine whether the question or answer analysis has a question number and whether it is at the top of a column, and then determine whether the question or answer analysis should be merged into the previous question or answer analysis.

[0116] The method provided by the embodiment of the present invention merges separated elements of the same type through the inclusion relationship between element positions and / or element categories, thereby further realizing a comprehensive and complete layout structure analysis.

[0117] Based on any of the above embodiments, step 110 specifically includes:

[0118] Based on the multi-scale sample images, multi-scale feature extraction is performed on the layout image to be analyzed to obtain the image features of the layout image.

[0119] Specifically, since the size of the paper image is not fixed in the scenario of scanning teaching aids, the size of the elements in the image is also variable. At the same time, the size difference between elements of different categories is also very large. For example, a single-column element may be hundreds of times the size of a page number element. Therefore, the variable scale makes it difficult to detect the overall elements.

[0120] Therefore, in the training phase, multi-scale sample images can be used for training. In the same training batch, an image size scale is first selected, and then all image scales in a training batch are adjusted to the selected scale. At the same time, the image scales selected in different training batches are dynamically changed. In this way, the network can learn objects of different scales during the training process, thereby improving the network's prediction ability for images of different scales and objects of different scales.

[0121] At the same time, since the sizes of each element are also variable, the size of a large element may be hundreds of times the size of a small element. Therefore, it is obviously inappropriate to use the same method to predict two types of elements with such a large size difference.

[0122] Therefore, when extracting features, multi-scale feature extraction can be performed. For example, a feature pyramid structure (FPN) can be used as a general component added after the backbone network to fuse the high-level and low-level features of the backbone network and output them in layers, which can well express the multi-layer features of the image. The high-level features of the output are small in size but have a large receptive field and deeper semantic features, which are suitable for predicting large targets. The low-level features are large in size and small in receptive field but have good detail features, which are friendly to small target prediction.

[0123] Accordingly, when performing feature detection, the multi-scale target detection Cascade RCNN algorithm can be used.

[0124] The method provided by the embodiment of the present invention extracts multi-scale features of the layout image to be analyzed based on the multi-scale sample image to obtain the image features of the layout image. The method can realize the analysis of layout structures with a rich number of elements and variable sizes, and is more suitable for scenes with complex layout structures while ensuring the accuracy of element detection.

[0125] Based on any of the above embodiments, steps 110 to 130 in the layout structure analysis method provided by the present invention specifically include:

[0126] Based on the layout structure analysis model, layout element detection and text line segmentation are performed on the layout image;

[0127] The layout structure analysis model is trained based on a first sample image marked with element positions and category labels, and a second sample image marked with text line labels.

[0128] Specifically, steps 110 to 130 described in the above embodiment can realize layout element detection and text line segmentation for the layout image based on the layout structure analysis model. Layout element detection and text line segmentation can share a feature extraction layer, and the feature extraction layer realizes information sharing between layout element detection and text line segmentation.

[0129] Due to the differences in data annotation required for layout element detection and layout text line segmentation, it is difficult to collect data with both annotation specifications. Therefore, the embodiment of the present invention adopts a multi-branch multi-data source training strategy to reduce the requirements for training data. The training samples can be divided into a first sample image marked with element position and category labels, and a second sample image marked with text line labels. The specific training process is as follows:

[0130] (1) First, data from different sources are packaged according to the data packaging specifications of each scenario for use in the training of each branch, and the data source is written into the training data as a tag;

[0131] (2) In the data sampling step of the training phase, proportional data sampling is achieved according to the ratio of the number of data from different sources;

[0132] (3) When calculating the training loss of each branch, the loss of this branch is selectively retained according to the data source label while ignoring the losses of other branches, thereby completing an accurate parameter update.

[0133] The method provided by the embodiment of the present invention constructs two different tasks within the same model framework by using a multi-branch and multi-data source training strategy, thereby simplifying the system flow and improving the efficiency of layout structure analysis.

[0134] Based on any of the above embodiments, Figure 5 This is the second flow chart of the layout structure analysis provided by the present invention, such as Figure 5 As shown, the method includes:

[0135] Step 510, based on the multi-scale sample image, multi-scale feature extraction is performed on the layout image to be analyzed to obtain image features of the layout image;

[0136] Step 520, based on the multi-scale object detection Cascade RCNN algorithm, perform layout element detection on the layout image to obtain the position information and element category of each candidate element in the layout image;

[0137] Specifically, the Cascade RCNN algorithm is a very powerful target detection algorithm. Cascade RCNN cascades three classifiers and regressors with different IoU thresholds. The feature representation extracted by the backbone network is sent to the RPN. The RPN originates from the FasterRCNN target detection algorithm. It obtains better candidate boxes through network adaptive learning, and then extracts the corresponding regional features through the RoIAlign algorithm. RCNN represents different classification and regression heads of the cascade. Different RCNNs set different IoU threshold hyperparameters. In the inference stage, the best detection results can be obtained by progressively adjusting the results of different cascade stages.

[0138] Step 530, based on the NMS strategy that is sensitive to element category and element confidence, the element detection results at the same position are filtered to obtain the position information and element category of each element in the layout image;

[0139] Step 540, based on the DBnet algorithm, segment the layout image into text lines to obtain the position information of each text line;

[0140] Specifically, DBNet predicts the area where the text shrinks relatively inward, that is, the probability map; at the same time, it predicts the boundary contour of the text line, that is, the threshold map; by subtracting the predicted shrinking area from the boundary area to obtain an approximate binary map, the text shrinking area can be accurately obtained, and then it can be expanded back accordingly according to the scaling ratio to obtain accurate text detection results.

[0141] The main reason why DBNet predicts dense text lines better is that the algorithm proposes a differentiable approximate binarization function, which is expressed as follows:

[0142]

[0143] Among them, B is an approximate binary image, P is a probability map, T is a threshold map, and k is a hyperparameter that can be set to 50. As the value of k is set larger, the image of the above expression will be steeper, and the distinction of the text boundary range will be better. The approximate binary segmentation map is obtained through the differentiable module DB. The regions in the binary map represent the regions after the text instance is shrunk, and the complete text detection result is obtained after the expansion operation.

[0144] Step 550, based on the position information and element category of each element in the layout image, as well as the position information of each text line, a fusion process is performed to obtain the final layout structure analysis result.

[0145] The method provided by the embodiment of the present invention first uses the target detection technology to complete the detection of layout elements, and then uses the semantic segmentation technology to complete the layout text line segmentation. Then the two are integrated to achieve a comprehensive layout structure analysis capability.

[0146] The layout structure analysis device provided by the present invention is described below. The layout structure analysis device described below and the layout structure analysis method described above can be referenced to each other.

[0147] Based on any of the above embodiments, Figure 6 Schematic diagram of the structure of the layout structure analysis device provided by the present invention. Figure 6 As shown, the device comprises:

[0148] A feature extraction unit 610, used to extract image features of the layout image to be analyzed;

[0149] An element detection unit 620 is used to perform layout element detection on the layout image based on the image features to obtain position information and element category of each element in the layout image;

[0150] A text line segmentation unit 630, configured to segment the layout image into text lines based on the image features, and obtain position information of each text line in the layout image;

[0151] The layout structure analysis unit 640 is used to perform layout structure analysis on the layout image based on the position information and element category of each element in the layout image, and the position information of each text line.

[0152] The layout structure analysis device provided in the embodiment of the present invention performs layout structure analysis on the layout image by utilizing the position information and element category of each element in the layout image, as well as the position information of each text line. Compared with the layout analysis in the prior art that only implements a single capability, the device can simultaneously perform a comprehensive and accurate analysis of the logical structure and layout structure of the layout image, especially for layout structure analysis in scenarios with complex layout structures.

[0153] In addition, the device can realize multi-branch integrated calculation of layout element detection and text line segmentation based on the extracted image features, thereby achieving comprehensive and accurate layout analysis while reducing the amount of calculation and improving the efficiency of layout analysis.

[0154] Based on any of the above embodiments, the element detection unit 620 is further configured to:

[0155] Based on the image features, performing layout element detection on the layout image to obtain position information, element category and element confidence of each candidate element in the layout image;

[0156] Based on the position information of each candidate element, determine the candidate elements at the same position from the candidate elements, and filter the candidate elements at the same position based on the element categories and element confidences of the candidate elements at the same position;

[0157] Based on the element confidence of the filtered candidate elements, the position information and element category of each element in the layout image are determined.

[0158] Based on any of the above embodiments, the element detection unit 620 is further configured to:

[0159] When the element category of the candidate elements at the same position is an easily confused category, the candidate elements at the same position except for the candidate element with the highest element confidence are deleted, and based on the element confidence of each candidate element at the same position, the element confidence of the candidate element with the highest element confidence at the same position is updated.

[0160] Based on any of the above embodiments, the layout structure analysis unit 640 is further used for:

[0161] Determining the relationship between the elements in the layout image based on the position information and element category of each element in the layout image;

[0162] Based on the position information of each element in the layout image and the position information of each text line, the text lines are merged or segmented to obtain the text line information of each element in the layout image;

[0163] Based on the relationship between the elements in the layout image and the text line information of the elements in the layout image, the layout structure analysis is performed on the layout image.

[0164] Based on any of the above embodiments, the layout structure analysis device provided by the embodiment of the present invention further includes an element merging unit, wherein the element merging unit is used to:

[0165] Determining the same category elements belonging to adjacent column elements based on the relationship between the elements in the layout image;

[0166] Based on the element position and / or the element type contained in the same category elements belonging to the latter column elements in the adjacent column elements, the attribution type of the same category elements belonging to the adjacent column elements is determined, and the same category elements belonging to the adjacent column elements are merged when the attribution type is the same.

[0167] Based on any of the above embodiments, the feature extraction unit 610 is further configured to:

[0168] Based on the multi-scale sample image, multi-scale feature extraction is performed on the layout image to be analyzed to obtain image features of the layout image.

[0169] Based on any of the above embodiments, the feature extraction unit, the element detection unit and the text line segmentation unit in the layout structure analysis device can be replaced by a model application unit, wherein the model application unit is used to:

[0170] Based on the layout structure analysis model, performing layout element detection and text line segmentation on the layout image;

[0171] The layout structure analysis model is trained based on a first sample image marked with element positions and category labels, and a second sample image marked with text line labels.

[0172] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the layout structure analysis method, which includes: extracting the image features of the layout image to be analyzed; based on the image features, performing layout element detection on the layout image to obtain the position information and element category of each element in the layout image; based on the image features, performing text line segmentation on the layout image to obtain the position information of each text line in the layout image; based on the position information and element category of each element in the layout image, and the position information of each text line, performing layout structure analysis on the layout image.

[0173] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the layout structure analysis method provided by the above-mentioned methods, which includes: extracting image features of the layout image to be analyzed; based on the image features, performing layout element detection on the layout image to obtain position information and element categories of each element in the layout image; based on the image features, performing text line segmentation on the layout image to obtain position information of each text line in the layout image; based on the position information and element category of each element in the layout image, as well as the position information of each text line, performing layout structure analysis on the layout image.

[0175] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the layout structure analysis method provided by the above-mentioned methods, the method comprising: extracting image features of a layout image to be analyzed; based on the image features, performing layout element detection on the layout image to obtain position information and element categories of each element in the layout image; based on the image features, performing text line segmentation on the layout image to obtain position information of each text line in the layout image; based on the position information and element category of each element in the layout image, as well as the position information of each text line, performing layout structure analysis on the layout image.

[0176] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A layout structure analysis method, It is characterized in that include: Extracting image features of the layout image to be analyzed; Based on the image features, performing layout element detection on the layout image to obtain position information and element category of each element in the layout image; Based on the image features, segment the layout image into text lines to obtain position information of each text line in the layout image; Determining the relationship between the elements in the layout image based on the position information and element category of each element in the layout image; Based on the position information of each element in the layout image and the position information of each text line, the text lines are merged or segmented to obtain the text line information of each element in the layout image; The fusion or segmentation is achieved by utilizing the relationship between each element and each text line; Based on the relationship between the elements in the layout image and the text line information of the elements in the layout image, the layout structure analysis is performed on the layout image.

2. The layout structure analysis method according to claim 1, It is characterized in that The detecting of layout elements of the layout image based on the image features to obtain the position information and element category of each element in the layout image includes: Based on the image features, performing layout element detection on the layout image to obtain position information, element category and element confidence of each candidate element in the layout image; Based on the position information of each candidate element, determine the candidate elements at the same position from the candidate elements, and filter the candidate elements at the same position based on the element categories and element confidences of the candidate elements at the same position; Based on the element confidence of the filtered candidate elements, the position information and element category of each element in the layout image are determined.

3. The layout structure analysis method according to claim 2, It is characterized in that The filtering of the candidate elements at the same position based on the element categories and element confidences of the candidate elements at the same position includes: When the element category of the candidate elements at the same position is an easily confused category, the candidate elements at the same position except for the candidate element with the highest element confidence are deleted, and based on the element confidence of each candidate element at the same position, the element confidence of the candidate element with the highest element confidence at the same position is updated.

4. The layout structure analysis method according to claim 1, It is characterized in that The method further comprises: determining the relationship between the elements in the layout image based on the position information and the element category of each element in the layout image, and then: Determining the same category elements belonging to adjacent column elements based on the relationship between the elements in the layout image; Based on the element position and / or the element type contained in the same category elements belonging to the latter column elements in the adjacent column elements, the attribution type of the same category elements belonging to the adjacent column elements is determined, and the same category elements belonging to the adjacent column elements are merged when the attribution type is the same.

5. The layout structure analysis method according to any one of claims 1 to 4, It is characterized in that The step of extracting image features of the layout image to be analyzed includes: Based on the multi-scale sample image, multi-scale feature extraction is performed on the layout image to be analyzed to obtain image features of the layout image.

6. The layout structure analysis method according to any one of claims 1 to 4, It is characterized in that Extracting image features of a layout image to be analyzed, performing layout element detection on the layout image based on the image features to obtain position information and element categories of each element in the layout image, performing text line segmentation on the layout image based on the image features to obtain position information of each text line in the layout image, including: Based on the layout structure analysis model, performing layout element detection and text line segmentation on the layout image; The layout structure analysis model is trained based on a first sample image marked with element positions and category labels, and a second sample image marked with text line labels.

7. A layout structure analysis device, It is characterized in that include: A feature extraction unit, used to extract image features of the layout image to be analyzed; An element detection unit, used to perform layout element detection on the layout image based on the image features, and obtain position information and element category of each element in the layout image; A text line segmentation unit, used to perform text line segmentation on the layout image based on the image features to obtain position information of each text line in the layout image; Layout structure analysis unit, used for: Determining the relationship between the elements in the layout image based on the position information and element category of each element in the layout image; Based on the position information of each element in the layout image and the position information of each text line, the text lines are merged or segmented to obtain the text line information of each element in the layout image; The fusion or segmentation is achieved by utilizing the relationship between each element and each text line; Based on the relationship between the elements in the layout image and the text line information of the elements in the layout image, the layout structure analysis is performed on the layout image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the layout structure analysis method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the layout structure analysis method according to any one of claims 1 to 6 are implemented.

Images