Layout analysis method and device, equipment and storage medium

Through the layout analysis model of dual-stream network and graph neural network, combined with convolutional neural network and question graph cross-page association network, the problem of question graph cross-page association analysis in the test paper layout is solved, and more efficient layout recognition and analysis are achieved, improving The accuracy and efficiency of layout analysis are achieved.

CN119091459AActive Publication Date: 2024-12-06SHENZHEN CHUANGZHI MINIMALIST TECHNOLOGY CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411590795.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-08
Publication Date
2024-12-06
Estimated Expiration
2044-11-08

AI Technical Summary

Technical Problem

Existing technologies are difficult to effectively realize the layout analysis of cross-page correlation of question charts in the test paper layout, and lack applicable models and methods.

Method used

Using a layout analysis model based on dual-stream network and graph neural network, the image to be processed is preprocessed, and multiple convolutional neural network structures and question graph cross-page association network structures are used to realize the cross-page association between question and question graph. , output layout recognition results.

Benefits of technology

It improves the accuracy and efficiency of layout analysis, realizes the layout layout effect of cross-page correlation of the title picture, and overcomes the information splitting and incompleteness caused by page separation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091459A_ABST
    Figure CN119091459A_ABST
Patent Text Reader

Abstract

The invention discloses a layout analysis method and device, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of a to-be-processed image, obtaining a preprocessed initial image, inputting the initial image into a layout analysis model based on a double-flow network and a graph neural network, so as to obtain a layout recognition result of the matching between a question and a question graph, the graph neural network comprises a plurality of convolutional neural network structures and a topic graph cross-page association network structure, and each convolutional neural network structure has different receptive fields in sequence from large to small. Compared with the prior art, the method has the advantages that the initial image is input into the layout analysis model based on the double-flow network and the graph neural network to obtain the matched layout recognition result between the question and the question map, the test paper layout can be effectively analyzed, cross-page association is performed on the question map and the question through the question map cross-page association network, and the test paper layout recognition efficiency is improved. Therefore, the layout effect of cross-page association of the theme diagram is better achieved, and the accuracy and efficiency of layout analysis are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a layout analysis method, device, equipment and medium. Background Art

[0002] With the exponential increase in the production and storage demand of large amounts of electronic documents, higher requirements are placed on the automated retrieval and layout analysis of documents.

[0003] Existing technical solutions tend to use NLP technology to analyze the association between paragraphs in document layout, but lack the implementation of the scenario of cross-page title and image. For example, the VGT model uses the grid Transformer model and network linear projection to convert the document content data of the image into a serialized representation that can be input to the Transformer encoder, and proposes a pre-trained 2D token-level and segment-level semantic understanding model to solve the association relationship between paragraphs, but it is not suitable for the function of cross-page association of title and image. Therefore, how to effectively analyze the test paper layout to achieve the layout effect of cross-page association of title and image has become a technical problem that needs to be solved urgently by technicians in this field. Summary of the invention

[0004] Based on this, it is necessary to provide a layout analysis method, device, equipment and medium to address the above-mentioned technical problems, which can effectively analyze the test paper layout, so as to better achieve the layout effect of cross-page association between questions and pictures.

[0005] A first aspect of an embodiment of the present application provides a layout analysis method, the layout analysis method comprising: Preprocess the image to be processed to obtain an initial image after preprocessing; The initial image is input into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields from large to small.

[0006] A second aspect of an embodiment of the present application provides a layout analysis device, the layout analysis device comprising: A processing module, used for preprocessing the image to be processed to obtain an initial image after preprocessing; A module is obtained, which is used to input the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields from large to small.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the layout analysis method as described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the layout analysis method as described in the first aspect is implemented.

[0009] In summary, the present invention provides a layout analysis method, device, equipment and medium, which preprocesses the image to be processed to obtain a preprocessed initial image, and inputs the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the question and the question image, wherein the graph neural network includes multiple convolutional neural network structures and a question-image cross-page association network structure, and each convolutional neural network structure has different receptive fields from large to small. Compared with the prior art, the present application can effectively analyze the test paper layout by inputting the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the question and the question image, and through the question-image cross-page association network, the question image and the question are cross-page associated, thereby better achieving the layout layout effect of the question-image cross-page association, and improving the accuracy and efficiency of the layout analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 It is a schematic diagram of an application environment of a layout analysis method provided by an embodiment of the present invention; Figure 2 It is a flow chart of a layout analysis method provided by one embodiment of the present invention; Figure 3 It is a structural schematic diagram of a layout analysis device provided by an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term “if” may be interpreted as “when” or “uponce” or “in response to determining”, depending on the context. Similarly, the phrases “if it is determined” or “if matched to [described condition or event]” may be interpreted as meaning “upon determination” or “in response to determination” or “uponce matched to [described condition or event]” or “in response to matching to [described condition or event]”, depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0019] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0020] An embodiment of the present invention provides a layout analysis method which can be applied in Figure 1 In the application environment, it aims to solve common defects in images, thereby improving the accuracy of layout analysis, so as to complete the image of cross-page association of test questions and pictures. Among them, the scanning device communicates with the server through the network. The scanning device is used to obtain the image to be processed and send the image to be processed to the server. The server can use the image to be processed as a training sample image, and use it in the layout analysis model based on the dual-stream network and graph neural network to obtain the layout recognition result matching the question and the question image. Among them, the scanning device can be but not limited to various cameras, sensors, laser scanners, scanning measuring instruments, etc. The server can be implemented with an industrial computer, an independent server, or a server cluster consisting of multiple servers.

[0021] See also Figure 2 , is a flow chart of a layout analysis method provided by an embodiment of the present invention, and the layout analysis method can be applied to Figure 1 Servers in Figure 2 As shown, the layout analysis method can be implemented through the following steps.

[0022] S201: Preprocess the image to be processed to obtain an initial image after preprocessing.

[0023] In the embodiments of the present disclosure, the image to be processed may be any image that needs to be analyzed for layout, such as an image of a test paper written by a student. The image to be processed may be a photograph of teaching materials (e.g., books, test papers, and exercise books) taken by a camera of a smartphone, a camera of a tablet computer, a lens of a digital camera, etc., or may include an image obtained by scanning teaching materials (e.g., books, test papers, and exercise books) using a device with a scanning function such as a printer. It should be noted that although the embodiments of the present disclosure all take the image to be processed as an image of a test paper as an example, it should not be regarded as a limitation of the present disclosure.

[0024] As an optional embodiment, preprocessing the image to be processed includes at least one random combination of: random rotation, random scaling, random cropping, random flipping, random contrast / brightness enhancement, random RGB-grayscale-RGB color space conversion, random addition of different Gaussian or salt and pepper noise, image normalization, and Gaussian bilateral filtering.

[0025] Specifically, random rotation refers to rotating the image within a certain range, which can enhance the diversity of the image and help the model better learn features in various directions. The rotation angle can be randomly selected from the specified range; random scaling is the operation of enlarging or reducing the image. By changing the size of the image, the model can be more robust to objects of different sizes. It is usually scaled within a set range of proportions; random cropping is to randomly select an area from the image for cropping, which can change the perspective of the image and also improve the model's ability to handle different scenes and partial information; random flipping refers to randomly flipping the image horizontally or vertically, which can help the model learn and recognize different image layouts, especially in the case of symmetrical objects; random contrast / brightness enhancement refers to randomly adjusting the contrast or brightness of the image to change the visual effect of the image, which can make the model sensitive to light. The random RGB-grayscale-RGB color space conversion can randomly convert the image into a grayscale image or keep it as a color image. This change can not only introduce color diversity, but also help the model adapt to low light or high contrast conditions. Randomly adding different Gaussian or salt and pepper noise can add Gaussian noise (transparent random noise) or salt and pepper noise (random black and white dots) to the image, which can help the model better handle noise, thereby improving the robustness of the model. Image normalization is to adjust the pixel value of the image to a specific range (such as 0 to 1 or -1 to 1), which helps to improve the stability of the optimization process and make the training process more efficient. Gaussian bilateral filtering is a smoothing operation that removes noise while retaining edge details, taking into account spatial distance and pixel value differences, so that neighboring pixels are more influential during smoothing, thereby making the edges of the image clearer. This application obtains a preprocessed initial image by performing at least one random combination of preprocessing on the image to be processed, so that the initial image can be subsequently input into a layout analysis model based on a dual-stream network and a graph neural network, and a layout recognition result matching the title and the title image can be quickly and accurately obtained.

[0026] It should be noted that the preprocessing of the image to be processed may include at least one random combination according to actual conditions, and the present application does not impose any limitation on this.

[0027] In an embodiment of the present application, by preprocessing the image to be processed to obtain a preprocessed initial image, the image to be processed can be quickly converted into an initial image that is more suitable for analysis and processing, making the processed image clearer, which is helpful for subsequent analysis and processing, thereby improving the efficiency and effectiveness of the entire image processing process.

[0028] S202: Input the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields from large to small.

[0029] In the embodiment of the present application, the dual-stream network is a network structure for processing video and dynamic scene analysis, and the graph neural network is a deep learning model that can directly process graph structure data. Graph structure data (such as social networks, traffic networks, molecular structures, etc.) consists of nodes (elements in the graph) and edges (relationships between nodes). Since the existing technical solutions tend to use NLP technology to analyze the association between paragraphs in the document layout, but lack the scene implementation of the title image across pages, the present application adds a title image across pages association network structure in the graph neural network, given the aggregate feature FM={FM2, FM3, FM4, FM5}, and followed by a target detection or instance segmentation model, such as the Mask-RCNN model, which can be used to generate component candidates in the document (not just using RPN), thereby refining the prediction results. The title image across pages association network structure updates the hidden representation of each node by participating in the neighbors of each node, that is, by updating the node features, we can predict its precise label and position coordinates, and each node uses a representation containing position information and deep features, and finally merges the two nodes to represent the constructed new node features. By inputting the initial image into the layout analysis model based on the dual-stream network and graph neural network, the layout recognition result that matches the title and the title image is obtained.

[0030] It should be noted that the receptive field is the area in the input space that affects a specific unit of the network. The input area can be the input of the network or the output of other units in the network, which is not limited here.

[0031] As an optional embodiment, the initial image is input into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, including: Extracting features from the initial image through a two-stream network in the image segmentation model to obtain semantic features and visual features; Performing feature fusion on the semantic feature map and the visual feature map based on an attention mechanism to obtain a fused feature map; Using a graph neural network to encode the fused feature map into a multi-scene element relationship vector corresponding to the fused feature map; According to the multi-scene element relationship vector, the layout analysis of the image to be processed is performed to obtain a recognition result of the match between the title and the title image.

[0032] Specifically, the two-stream network (two-stream visual grid transformer based on the VGT model) in the image segmentation model extracts visual features and semantic features respectively. These features include modality-specific visual features and language features, which are used for subsequent fusion and relationship modeling, providing a basis for subsequent layout analysis. The extracted visual features and semantic features are adaptively fused based on the attention mechanism to obtain a fused feature map, making full use of the complementary information between the two modalities. This step may involve complex fusion strategies, and fusion can also be performed by feature weighting and other methods. This application does not impose any restrictions on this. A relationship module based on a graph neural network (GNN) is introduced to model the relationship between components, that is, the fused feature map is encoded into a multi-scene element relationship vector corresponding to the fused feature map using a graph neural network, thereby providing useful information representation for subsequent tasks. It should be noted that the graph neural network can learn and reason about the nodes and edges in the interconnected graph of multi-scene elements, and is used to extract the feature representation of the fused feature map for prediction. In addition, the graph neural network updates the representation of each element node by iteratively aggregating the neighbor information of the element node according to the connection relationship between the element nodes in the fused feature map. The multi-scene element relationship vector is used to characterize the characteristics of each element node and the connection relationship between element nodes. Because the multi-scene element interconnection graph integrates the interaction relationship and interaction actions of multiple scene objects, the multi-scene element relationship vector also integrates the information of multiple scenes. Therefore, task prediction based on the multi-scene element relationship vector can improve the efficiency and accuracy of task prediction. In particular, for the task of cross-page association of title and image, the hidden representation of each node is updated by participating in the neighbors of each node, so as to predict the accurate label and position coordinates. Define a specific cross-page association network of title and image (such as DeConwwithRelationLanguageModeling). This module is based on the results of GNN relationship modeling. According to the multi-scene element relationship vector, the cross-page association network of title and image is used to analyze the layout of the processed image, and the cross-page association of the title image and the title is performed. By updating the hidden association representation of the node, such as the segmented association of the question number, the accurate matching between the title image and the question is achieved, and the final layout recognition result is output, including the result of the cross-page association of the title image, which may include the precise location of the document component, the category label and the association relationship between them.

[0033] This application can effectively aggregate the neighbor information of each node and update the hidden association representation of each node through the cross-page association network of the title and image. By stacking multiple graph convolutional layers, it can learn more and more complex feature representations and adapt to more complex task requirements. Through cross-page association and merging processing, it can overcome the information fragmentation and incompleteness caused by page separation, thereby achieving a better cross-page association effect of the title and image and improving the accuracy and completeness of information extraction.

[0034] As an optional embodiment, according to the multi-scene element relationship vector, the layout analysis of the image to be processed is performed to obtain a recognition result of a match between the title and the title image, including: Performing layout analysis on each page of the to-be-processed image according to the multi-scene element relationship vector to determine a content list and a directory list, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; Performing random occlusion and prediction processing on the image to be processed in sequence to determine positioning information and type information of a plurality of sub-region images included in the image to be processed; According to the location information and type information of the multiple sub-region images, the directory list is traversed layer by layer, and the title indicated by the traversed current node and the title indicated by the adjacent node in the same layer are respectively matched with at least a part of the paragraphs in the content list to determine two matching paragraphs; All paragraphs between the two matching paragraphs in the content list are spliced, and the splicing result is used as the associated text block of the current node to obtain a recognition result of the match between the title and the title image.

[0035] Specifically, for an image to be processed that includes multiple pages, the multiple pages usually have a preset page sequence. According to the multi-scene element relationship vector, each page of the image to be processed is analyzed for layout to determine a content list and a directory list. At least a portion of the paragraphs in the content list are consistent with at least a portion of the titles in the directory list. The image to be processed is subjected to random occlusion and prediction processing in sequence, and then the positioning information and type information of the multiple sub-region images included in the image to be processed are determined. According to the positioning information and type information of the multiple sub-region images, the directory list is traversed layer by layer, and the title indicated by the traversed current node and the title indicated by the adjacent node at the same layer are respectively matched with at least a portion of the paragraphs in the content list for text similarity to determine two matching paragraphs, and all paragraphs between the two matching paragraphs in the content list are spliced, and the splicing result is used as the associated text block of the current node to obtain the recognition result of the match between the title and the title image. Compared with matching all paragraphs in the content list for each match, it helps to greatly narrow the scope of matching and improve the efficiency of cross-page association between the title image and the title.

[0036] As an optional embodiment, the image to be processed is subjected to random occlusion and prediction processing in sequence to determine the positioning information and type information of multiple sub-region images included in the image to be processed, including: Selecting N grid data in the image to be processed to perform occlusion processing to obtain occluded grid data; Perform center point prediction and surrounding node prediction on the blocked grid data to obtain center point prediction data and surrounding node prediction data; Make category judgment on the prediction data of the center point and the prediction data of the surrounding nodes; According to the judgment result, the positioning information and type information of the multiple sub-region images included in the image to be processed are determined.

[0037] Specifically, N grid data in the image to be processed are selected for occlusion processing to obtain the occluded grid data, which can allow the unoccluded area to better learn the connection of the occluded area, so that the unoccluded area not only has the connection of the data and position of the area but also has the connection of the occluded area, thereby increasing the generalization ability of the subsequent model. For example, when a picture with a blurred area or black spots is input, the model can also learn the relationship between the blurred area or the black spot area. The center point prediction and the surrounding node prediction of the occluded grid data are first predicted. The category of the center point of each occluded grid is first predicted, and the local maximum value is searched for each occluded grid, and the non-maximum value is suppressed. Only the maximum value output is retained, and then the category of the predicted occluded area grid is judged by setting a threshold. If the same, the type information of the multiple sub-area images included in the image to be processed is determined, and the location information of each sub-area image is determined by statistical boundary information, starting from the position of the corresponding center point, combined with the predicted category and node information. For example, each area can be represented by a bounding box. By using the information of the center point and surrounding nodes, the boundaries and categories of each object in the image can be accurately obtained, improving the accuracy and efficiency of image processing and analysis. It can also provide important support for the application of machine learning and deep learning, making the overall process more efficient.

[0038] As an optional embodiment, the layout analysis model is trained in the following manner: Obtain training sample data set; Inputting the sample images in the training sample data set into the two-stream network to obtain fusion features in the sample images; Inputting the fused features into a graph neural network to obtain aggregated neighbor features; Calculating a loss function of aggregated neighbor features in the sample image; The layout analysis model is trained based on the loss function until the loss value output by the trained layout analysis model is less than a preset loss threshold.

[0039] Specifically, several sample images can be obtained from images publicly available online or collected offline, and the obtained sample images include but are not limited to educational scene documents, such as test papers, and the obtained sample images are processed to obtain a training sample data set. The training sample data set can be composed of multiple sample subsets, and a sample subset includes a sample image. It should be noted that in the embodiment of the present disclosure, the size of the sample image is set in advance to facilitate subsequent processing. Since there are differences in the sizes of different sample images, in order to facilitate model training, the collected sample images can be resized, and sample images of different sizes can be corrected to a uniform size, such as uniformly scaling the sample images to a size of 512*512. Afterwards, the sample images in the training sample data set are input into the two-stream network to obtain the fusion features in the sample images, and the fusion features are input into the graph neural network to obtain the aggregated neighbor features. Since in the graph neural network, the hidden representation of each node is updated by aggregating the information of its neighbor nodes, that is, according to the neighbor node features of each node, aggregation is performed, and then the hidden representation of itself is updated by combining its own features and the aggregated neighbor information. Then, the loss function of the aggregated neighbor features in the sample image is calculated, and the network parameters of the layout analysis model are updated according to the loss function to perform iterative training until the loss value of the layout analysis model to be trained is less than or equal to the preset loss threshold, so as to obtain the target layout analysis model. It should be noted that the preset loss threshold can be set according to the experience of the technician, and this embodiment does not limit the value of the preset loss threshold.

[0040] It should be noted that model training is an iterative process. By calculating the loss function value after each iteration and continuously adjusting the model's network parameters for iterative training when the loss function value is greater than the preset value, the model converges to obtain a trained model until the overall loss function value of the model is less than or equal to the preset value, or the overall loss function value of the model no longer changes or changes slowly.

[0041] In the disclosed embodiment, a sample image in a training sample data set is input into a dual-stream network to obtain fused features in the sample image, and the fused features are input into a graph neural network to obtain aggregated neighbor features. Then, a layout analysis model is trained according to a loss function of the aggregated neighbor features in the sample image, until the loss value output by the trained layout analysis model is less than a preset loss threshold. Thus, the trained layout analysis model can effectively analyze the layout of the test paper, thereby better achieving the layout effect of cross-page association between questions and images, and improving the clarity of the image in visual perception.

[0042] In summary, the present invention provides a layout analysis method, device, equipment and medium, which preprocesses the image to be processed to obtain a preprocessed initial image, and inputs the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the question and the question image, wherein the graph neural network includes multiple convolutional neural network structures and a question-image cross-page association network structure, and each convolutional neural network structure has different receptive fields from large to small. Compared with the prior art, the present application can effectively analyze the test paper layout by inputting the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the question and the question image, and through the question-image cross-page association network, the question image and the question are cross-page associated, thereby better achieving the layout layout effect of the question-image cross-page association, and improving the accuracy and efficiency of the layout analysis.

[0043] See also Figure 3 , Figure 3 Schematic diagram of the structure of the layout analysis device provided by the embodiment of the present invention. In this embodiment, the terminal includes various units for executing Figure 3 For details, please refer to the steps in the corresponding embodiment. Figure 2 as well as Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 3 The layout analysis device 30 includes: a processing module 31 and an obtaining module 32.

[0044] A processing module 31 is used to preprocess the image to be processed to obtain an initial image after preprocessing; Obtaining module 32, for inputting the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields in descending order.

[0045] Optionally, the processing module 31 is specifically used for: The preprocessing of the image to be processed includes at least one random combination: random rotation, random scaling, random cropping, random flipping, random contrast / brightness enhancement, random RGB-grayscale-RGB color space conversion, random addition of different Gaussian or salt and pepper noise, image normalization, and Gaussian bilateral filtering.

[0046] Optionally, the obtaining module 32 is specifically used for: Extracting features from the initial image through a two-stream network in the image segmentation model to obtain semantic features and visual features; Performing feature fusion on the semantic feature map and the visual feature map based on an attention mechanism to obtain a fused feature map; Using a graph neural network to encode the fused feature map into a multi-scene element relationship vector corresponding to the fused feature map; According to the multi-scene element relationship vector, the layout analysis of the image to be processed is performed to obtain a recognition result of the match between the title and the title image.

[0047] Optionally, the obtaining module 32 is further used for: Performing layout analysis on each page of the to-be-processed image according to the multi-scene element relationship vector to determine a content list and a directory list, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; Performing random occlusion and prediction processing on the image to be processed in sequence to determine positioning information and type information of a plurality of sub-region images included in the image to be processed; According to the location information and type information of the multiple sub-region images, the directory list is traversed layer by layer, and the title indicated by the traversed current node and the title indicated by the adjacent node in the same layer are respectively matched with at least a part of the paragraphs in the content list to determine two matching paragraphs; All paragraphs between the two matching paragraphs in the content list are spliced, and the splicing result is used as the associated text block of the current node to obtain a recognition result of the match between the title and the title image.

[0048] Optionally, the obtaining module 32 is further used for: Selecting N grid data in the image to be processed to perform occlusion processing to obtain occluded grid data; Perform center point prediction and surrounding node prediction on the blocked grid data to obtain center point prediction data and surrounding node prediction data; Make category judgment on the prediction data of the center point and the prediction data of the surrounding nodes; According to the judgment result, the positioning information and type information of the multiple sub-region images included in the image to be processed are determined.

[0049] Optionally, the obtaining module 32 is further used for: Obtain training sample data set; Inputting the sample images in the training sample data set into the two-stream network to obtain fusion features in the sample images; Inputting the fused features into a graph neural network to obtain aggregated neighbor features; Calculating a loss function of aggregated neighbor features in the sample image; The layout analysis model is trained based on the loss function until the loss value output by the trained layout analysis model is less than a preset loss threshold.

[0050] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0051] Figure 4 Schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in the above-mentioned layout analysis method embodiment are implemented.

[0052] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 These are merely examples of computer devices and do not constitute limitations on the computer devices. The computer devices may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, they may also include a network interface, a display screen, and an input device.

[0053] In one embodiment, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor in a computer device, the computer device can perform the steps of any embodiment of a layout analysis disclosed in the present invention, which will not be repeated here. The computer-readable storage medium can be non-volatile or volatile.

[0054] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0055] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating device and the computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store operating devices, application programs, boot loaders (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0056] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0057] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0058] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A layout analysis method, characterized in that: include: Preprocess the image to be processed to obtain an initial image after preprocessing; The initial image is input into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields from large to small.

2. The layout analysis method according to claim 1, characterized in that: The inputting the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result matching the title and the title image includes: Extracting features from the initial image through a two-stream network in the image segmentation model to obtain semantic features and visual features; Performing feature fusion on the semantic feature map and the visual feature map based on an attention mechanism to obtain a fused feature map; Using a graph neural network to encode the fused feature map into a multi-scene element relationship vector corresponding to the fused feature map; According to the multi-scene element relationship vector, the layout analysis of the image to be processed is performed to obtain a recognition result of the match between the title and the title image.

3. The layout analysis method according to claim 2, characterized in that: The performing layout analysis on the image to be processed according to the multi-scene element relationship vector to obtain a recognition result of a match between the title and the title image includes: Performing layout analysis on each page of the to-be-processed image according to the multi-scene element relationship vector to determine a content list and a directory list, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; Performing random occlusion and prediction processing on the image to be processed in sequence to determine positioning information and type information of a plurality of sub-region images included in the image to be processed; According to the location information and type information of the multiple sub-region images, the directory list is traversed layer by layer, and the title indicated by the traversed current node and the title indicated by the adjacent node in the same layer are respectively matched with at least a part of the paragraphs in the content list to determine two matching paragraphs; All paragraphs between the two matching paragraphs in the content list are spliced, and the splicing result is used as the associated text block of the current node to obtain a recognition result of the match between the title and the title image.

4. The layout analysis method according to claim 3, characterized in that: The step of sequentially performing random occlusion and prediction processing on the image to be processed to determine positioning information and type information of a plurality of sub-region images included in the image to be processed includes: Selecting N grid data in the image to be processed to perform occlusion processing to obtain occluded grid data; Perform center point prediction and surrounding node prediction on the blocked grid data to obtain center point prediction data and surrounding node prediction data; Make category judgment on the prediction data of the center point and the prediction data of the surrounding nodes; According to the judgment result, the positioning information and type information of the multiple sub-region images included in the image to be processed are determined.

5. The layout analysis method according to claim 1, characterized in that: The layout analysis model is trained in the following way: Obtain training sample data set; Inputting the sample images in the training sample data set into the two-stream network to obtain fusion features in the sample images; Inputting the fused features into a graph neural network to obtain aggregated neighbor features; Calculating a loss function of aggregated neighbor features in the sample image; The layout analysis model is trained based on the loss function until the loss value output by the trained layout analysis model is less than a preset loss threshold.

6. The layout analysis method according to claim 1, characterized in that: The preprocessing of the image to be processed includes at least one random combination: random rotation, random scaling, random cropping, random flipping, random contrast / brightness enhancement, random RGB-grayscale-RGB color space conversion, random addition of different Gaussian or salt and pepper noises, image normalization, and Gaussian bilateral filtering.

7. A layout analysis device, characterized in that: include: A processing module, used for preprocessing the image to be processed to obtain an initial image after preprocessing; A module is obtained, which is used to input the initial image into a layout analysis model based on a dual-stream network and a graph neural network to obtain a layout recognition result that matches the title and the title image, wherein the graph neural network includes multiple convolutional neural network structures and a title-image cross-page association network structure, and each of the convolutional neural network structures has different receptive fields from large to small.

8. The layout analysis device according to claim 7, characterized in that: The obtaining module is also used for: Extracting features from the initial image through a two-stream network in the image segmentation model to obtain semantic features and visual features; Performing feature fusion on the semantic feature map and the visual feature map based on an attention mechanism to obtain a fused feature map; Using a graph neural network to encode the fused feature map into a multi-scene element relationship vector corresponding to the fused feature map; According to the multi-scene element relationship vector, the layout analysis of the image to be processed is performed to obtain a recognition result of the match between the title and the title image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the layout analysis method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the layout analysis method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • PDF document cross-page table merging method and device, electronic equipment and storage medium

    CN112380825A

  • Image processing method and device, terminal and computer readable storage medium

    CN112446259A

  • Document layout analysis method, model training method and device and equipment

    CN113361247A

  • Question set data processing method and device based on artificial intelligence, and computer equipment

    CN113761217A

  • PDF (Portable Document Format) document cross-page table merging method and device and related equipment

    CN114036909A