A document page identification method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a document layout recognition method and device, electronic equipment and storage medium, the method comprises: obtaining a to-be-recognized document, extracting visual features and semantic features of the to-be-recognized document, wherein the visual features identify the visual characteristics on the overall layout of the image corresponding to the to-be-recognized document, and the semantic features at least include character-level features and text line-level features; fusing the image features and the semantic features to obtain multi-modal document features; and based on the multi-modal document features, recognizing the element positions and categories of each element in the to-be-recognized document. The character-level semantic features can extract text-level elements such as formulas embedded in the text, and the visual features can recognize visual elements such as images, and then the multi-modal document features can obtain elements including visual features such as images and character-level elements within text lines such as formulas, so that the recognition result of the document layout is more comprehensive, and the accuracy of the document layout recognition result is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document processing, and in particular to a document layout recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Document layout analysis refers to analyzing and recognizing a document to obtain the positions and categories of elements included in the document. Document layout analysis technology is widely used in document understanding, document compression, document digitization, and other application scenarios, and has wide application value. The elements in a document can include figures, tables, text, titles, and the like.

[0003] In current document layout analysis methods, text information and image information of multiple text lines in a document are first extracted, and then the text information and image information of the multiple text lines are encoded, decoded, and the like to obtain the positions and categories of elements in the document.

[0004] Since current document layout analysis methods are text line-level recognition performed on text lines, the positions and categories of the recognized elements are also coarse-grained text line-level results, and the recognition result of the document layout is not accurate. SUMMARY

[0005] Embodiments of the present application aim to provide a document layout recognition method, device, electronic device, and storage medium to improve the accuracy of the recognition result of the document layout. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide a document layout recognition method, which comprises:

[0007] obtaining a to-be-recognized document;

[0008] extracting visual features and semantic features of the to-be-recognized document, wherein the visual features identify visual characteristics on an overall layout of an image corresponding to the to-be-recognized document, and the semantic features at least include character-level features and text line-level features;

[0009] fusing the image features and the semantic features to obtain multi-modal document features;

[0010] based on the multi-modal document features, recognizing element positions and categories of elements in the to-be-recognized document.

[0011] Optionally, the step of extracting the visual features and the semantic features of the to-be-recognized document comprises:

[0012] When it is judged that the to-be-recognized document is a document in a file format, the to-be-recognized document is converted into an image as a to-be-recognized image; the to-be-recognized document is subjected to file analysis to obtain text content and a text position included in the to-be-recognized document;

[0013] When it is judged that the to-be-recognized document is a document in an image format, the to-be-recognized document is taken as a to-be-recognized image; the to-be-recognized document is subjected to image recognition to obtain text content and a text position included in the to-be-recognized document, wherein the text content at least includes text content at a character level and text content at a text line level;

[0014] The visual feature of the to-be-recognized image is extracted, and the semantic feature of the to-be-recognized document is determined based on the text content and the text position.

[0015] Optionally, the step of extracting the visual feature of the to-be-recognized image and determining the semantic feature of the to-be-recognized document based on the text content and the text position comprises:

[0016] The text content is filled into the to-be-recognized document according to the text position to obtain a semantic feature map;

[0017] The to-be-recognized image and the semantic feature map are input into a first convolutional neural network established in advance to obtain visual features and semantic features output by the first convolutional neural network; or,

[0018] The to-be-recognized image is input into a second convolutional neural network established in advance to obtain visual features output by the second convolutional neural network, and the semantic feature map is input into a third convolutional neural network established in advance to obtain semantic features output by the third convolutional neural network.

[0019] Optionally, the step of fusing the image feature and the semantic feature to obtain a multi-modal document feature comprises:

[0020] The image feature and the semantic feature are input into a neural network established in advance to obtain a multi-modal document feature output by the neural network, wherein the neural network is used for feature fusion of image features and semantic features, and the neural network obtains the multi-modal document feature M based on the following formula:

[0021] M=A*V+(1-A)*S

[0022] wherein V is the visual feature, S is the semantic feature, A=f(V, S), A is a weighted weight obtained by training the neural network based on a visual feature sample and a semantic feature sample, and f is a mapping function corresponding to the neural network.

[0023] Optionally, the step of identifying the element position and the element category of each element in the to-be-identified document based on the multi-modal document feature comprises:

[0024] performing target detection on the to-be-identified document based on the multi-modal document feature, to determine the element position and the element category of each element in the to-be-identified document.

[0025] Optionally, after the step of determining the element position and the element category of each element in the to-be-identified document, the method further comprises:

[0026] adjusting the element position and the element category of each element based on the pre-determined structural relationship between document elements, to obtain the final element position and the final element category of each element.

[0027] Optionally, the step of adjusting the element position and the element category of each element based on the pre-determined structural relationship between document elements, to obtain the final element position and the final element category of each element comprises:

[0028] inputting the multi-modal document feature, the element position and the element category of each element into a pre-established relationship modeling model, so that the relationship modeling model adjusts the element position and the element category of each element based on the pre-determined structural relationship between document elements and the multi-modal document feature, and outputs the final element position and the final element category of each element.

[0029] In a second aspect, an embodiment of the present application provides a document layout recognition device, which comprises:

[0030] an acquisition module configured to acquire a to-be-identified document;

[0031] an extraction module configured to extract visual features and semantic features of the to-be-identified document, wherein the visual features identify visual characteristics on an overall layout of an image corresponding to the to-be-identified document, and the semantic features at least include character-level features and text line-level features;

[0032] a fusion module configured to fuse the image features and the semantic features, to obtain multi-modal document features;

[0033] an identification module configured to identify the element position and the element category of each element in the to-be-identified document based on the multi-modal document feature.

[0034] Optionally, the extraction module comprises:

[0035] The first processing unit is configured to convert the to-be-recognized document into an image as a to-be-recognized image when the to-be-recognized document is determined to be a document in a file format, and perform file analysis on the to-be-recognized document to obtain text content and a text position included in the to-be-recognized document;

[0036] The second processing unit is configured to take the to-be-recognized document as a to-be-recognized image when the to-be-recognized document is determined to be a document in an image format, and perform image recognition on the to-be-recognized document to obtain text content and a text position included in the to-be-recognized document, wherein the text content at least includes character-level text content and text line-level text content.

[0037] The feature extraction unit is configured to extract visual features of the to-be-recognized image, and determine semantic features of the to-be-recognized document based on the text content and the text position.

[0038] Optionally, the feature extraction unit includes:

[0039] The padding subunit is configured to fill the text content into the to-be-recognized document according to the text position to obtain a semantic feature map.

[0040] The first extraction subunit is configured to input the to-be-recognized image and the semantic feature map into a first convolutional neural network established in advance to obtain visual features and semantic features output by the first convolutional neural network; or,

[0041] The second extraction subunit is configured to input the to-be-recognized image into a second convolutional neural network established in advance to obtain visual features output by the second convolutional neural network, and input the semantic feature map into a third convolutional neural network established in advance to obtain semantic features output by the third convolutional neural network.

[0042] Optionally, the fusion module includes:

[0043] The feature fusion unit is configured to input the image features and the semantic features into a neural network established in advance to obtain multi-modal document features output by the neural network, wherein the neural network is used for feature fusion of image features and semantic features, and the neural network obtains the multi-modal document features M based on the following formula:

[0044] M=A*V+(1-A)*S

[0045] wherein V is the visual features, S is the semantic features, A=f(V, S), A is a weighted weight obtained by training the neural network based on visual feature samples and semantic feature samples, and f is a mapping function corresponding to the neural network.

[0046] Optionally, the recognition module includes:

[0047] a target detection unit, configured to perform target detection on the to-be-recognized document based on the multi-modal document features, and determine element positions and categories of elements in the to-be-recognized document.

[0048] Optionally, the apparatus further comprises:

[0049] an adjusting module, configured to, after the element positions and categories of the elements in the to-be-recognized document are determined, adjust the element positions and categories of the elements based on a pre-determined structural relationship between document elements, to obtain final element positions and categories of the elements.

[0050] Optionally, the adjusting module comprises:

[0051] an adjusting unit, configured to input the multi-modal document features, the element positions and categories of the elements into a pre-established relationship modeling model, so that the relationship modeling model adjusts the element positions and categories of the elements based on the pre-determined structural relationship between document elements and the multi-modal document features, and outputs the final element positions and categories of the elements.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0053] the memory is configured to store a computer program;

[0054] the processor is configured to execute the program stored on the memory, and implement the method steps of any one of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the first aspect.

[0056] Embodiments of the present application have the following beneficial effects:

[0057] In the solution provided by this invention, an electronic device can acquire a document to be identified and extract its visual and semantic features. The visual features identify the overall layout characteristics of the image corresponding to the document, while the semantic features include at least character-level and text-line-level features. The image features and semantic features are fused to obtain multimodal document features. Based on these multimodal document features, the element position and category of each element in the document to be identified are determined. Through this solution, the electronic device can acquire text-line-level semantic features, character-level semantic features, and visual features that identify the overall layout characteristics of the image corresponding to the document. Since character-level semantic features can extract text-level elements such as references and formulas embedded in the text, and visual features can identify visually rich elements such as figures and tables, recognition based on the multimodal document features obtained by fusing visual features, text-line-level semantic features, and character-level semantic features can obtain elements based on visual features, such as figures and tables, as well as text-line-level elements such as references and formulas. Therefore, the document layout recognition results are more comprehensive, greatly improving the accuracy of the document layout recognition results. Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart illustrating a document layout recognition method provided in an embodiment of the present invention;

[0060] Figure 2 for Figure 1 A specific flowchart of step S102 in the illustrated embodiment;

[0061] Figure 3 for Figure 2 A specific flowchart of step S203 in the illustrated embodiment;

[0062] Figure 4 A flowchart illustrating the processing method of the relationship modeling model provided in an embodiment of the present invention;

[0063] Figure 5 This is a flowchart illustrating a document layout recognition method provided in an embodiment of the present invention.

[0064] Figure 6 A structural schematic diagram of a document layout recognition device provided by an embodiment of the present application;

[0065] Figure 7 Another structural schematic diagram of a document layout recognition device provided by an embodiment of the present application;

[0066] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0068] In order to improve the accuracy of the recognition result of the document layout, an embodiment of the present application provides a document layout recognition method, device, electronic device, computer readable storage medium and computer program product. First, a document layout recognition method provided by an embodiment of the present application is introduced.

[0069] The document layout recognition method provided by an embodiment of the present application can be applied to any electronic device that needs to recognize a document layout, for example, a computer, a mobile phone, a tablet computer, etc., which is not limited here. In order to describe clearly, the electronic device is called in the following.

[0070] As shown in the figure, a document layout recognition method, the method comprises: Figure 1

[0071] S101, obtaining a to-be-recognized document;

[0072] S102, extracting visual features and semantic features of the to-be-recognized document;

[0073] The visual features identify the visual characteristics of the overall layout of the image corresponding to the to-be-recognized document, and the semantic features at least include character-level features and text line-level features.

[0074] S103, fusing the image features and the semantic features to obtain multi-modal document features;

[0075] S104, based on the multi-modal document features, recognizing the element positions and categories of each element in the to-be-recognized document.

[0076] ​It can be seen that in the scheme provided by the embodiment of the application, the electronic device can acquire the to-be-identified document, extract visual features and semantic features of the to-be-identified document, wherein the visual features identify visual characteristics of the overall layout of the image corresponding to the to-be-identified document, the semantic features at least include character-level features and text line-level features, the image features and the semantic features are fused to obtain multi-modal document features, and the element positions and categories of each element in the to-be-identified document are identified based on the multi-modal document features. Through the above scheme, the electronic device can acquire the text line-level semantic features and the character-level semantic features, and the visual features capable of identifying the overall layout characteristics of the image corresponding to the to-be-identified document. Since the character-level semantic features can extract text-level elements such as references and formulas embedded in the text, and the visual features can identify visual-rich elements such as graphs and tables, the multi-modal document features obtained by fusing the visual features, the text line-level semantic features and the character-level semantic features are used for identification, which can obtain elements such as graphs and tables based on the visual features, and character-level elements in the text line such as references and formulas, so that the recognition result of the document layout is more comprehensive, and the accuracy of the document layout recognition result is greatly improved.

[0077] When the user wants to perform document layout analysis on the document, the document to be identified can be provided, that is, the to-be-identified document. For example, a paper version of the document can be photographed by using the camera of the electronic device, or the document stored on another electronic device can be sent to the electronic device executing the scheme as the to-be-identified document, or the stored document to be subjected to document layout analysis can be selected in the electronic device as the to-be-identified document, so that the electronic device can acquire the to-be-identified document, that is, execute the above step S101.

[0078] The to-be-identified document can be a document in an image format, and the image format can be bmp, jpg, png, etc. The to-be-identified document can also be a document in a file format, and the file format can be xml, pdf, word, etc., which is not limited herein.

[0079] After acquiring the document to be identified, the electronic device can perform step S102, which involves extracting the visual and semantic features of the document. The visual features can identify the overall layout and visual characteristics of the image corresponding to the document. These visual features may include color, texture, and shape features, etc., without specific limitations. Because visual features can identify the overall layout and visual characteristics of the image corresponding to the document, they are more effective in identifying visually rich elements such as figures and tables within the document. Specifically, the overall layout and visual characteristics of the image corresponding to the document refer to the visual features formed by the arrangement and positional relationships of figures, tables, etc., within the image.

[0080] The aforementioned semantic features can include at least character-level features and text line-level features. Character-level features can characterize the features of a character itself and the relationships between characters within a text line. For example, they can include characters and tokens. A token can represent a meaningful combination of consecutive characters. For instance, for the sentence "Fruits containing Vitamin C: Citrus fruits.", the character feature can identify "'Fruits containing Vitamin C': 'Citrus fruits'.", and the token feature can identify "'Fruits containing Vitamin C': 'Citrus fruits'.". Text line-level features can characterize the features of a text line itself and the relationships between text lines. Therefore, the semantic features in this scheme are more conducive to identifying visually similar but different elements, such as character-level formulas within a text line.

[0081] After extracting the visual and semantic features of the document to be identified, the electronic device can perform step S103, which involves fusing the image features and semantic features to obtain multimodal document features. To fully utilize the complementary characteristics between feature modalities, in one embodiment, the electronic device can input the extracted visual and semantic features of the document to be identified into a pre-established neural network to obtain multimodal document features.

[0082] The pre-established neural network can be used to fuse the visual and semantic features of the document to be identified, thereby obtaining multimodal document features. These multimodal document features can characterize the overall visual layout of the image corresponding to the document to be identified. For example, they can characterize the features of visually rich elements such as figures and tables in the document. They can also characterize the semantic features of each line of text, as well as the semantic features of the characters themselves and the semantic features between characters within each line.

[0083] Further, in the step S104, the electronic device can identify the element position and the category of each element in the to-be-identified document based on the multi-modal document feature. Since the multi-modal document feature can represent the visual features of the overall layout of the image corresponding to the to-be-identified document, and can also represent the semantic features of the text line level and the semantic features of the character level, the elements identified by the electronic device based on the multi-modal document feature include not only elements of coarse granularity, such as figures, tables, and texts, but also elements of fine granularity, such as titles, references, and formulas. The element position of each element is the position of the element in the to-be-identified document. In an embodiment, the element position of each element can be represented by a candidate box, and the area framed by the candidate box is the element position of the corresponding element.

[0084] Since the multi-modal document feature is obtained by fusing the image feature and the semantic feature, the multi-modal document feature can be beneficial to identifying not only elements with rich visual features such as figures and tables, but also elements with similar visual features but different contents. Therefore, the electronic device can obtain the semantic features of the text line level and the semantic features of the character level, and the visual features capable of identifying the overall layout features of the image corresponding to the to-be-identified document. Since the semantic features of the character level can extract text-level elements such as references and formulas embedded in the text, and the visual features can identify elements with rich visual features such as figures and tables, the multi-modal document feature obtained by fusing the visual features, the semantic features of the text line level, and the semantic features of the character level can identify not only elements based on the visual features such as figures and tables, but also elements of the character level within the text line such as references and formulas. Therefore, the recognition result of the document layout can be more comprehensive, and the accuracy of the recognition result of the document layout can be greatly improved.

[0085] As an embodiment of the present application, as shown in Figure 2 The step of extracting the visual features and the semantic features of the to-be-identified document can include:

[0086] S201, when it is judged that the to-be-identified document is a document in a file format, converting the to-be-identified document into an image as a to-be-identified image; performing file analysis on the to-be-identified document to obtain the text content and the text position included in the to-be-identified document;

[0087] S202, when it is judged that the to-be-identified document is a document in an image format, taking the to-be-identified document as a to-be-identified image; performing image recognition on the to-be-identified document to obtain the text content and the text position included in the to-be-identified document;

[0088] Since the to-be-recognized document can be a document in a file format or a document in an image format, the step of extracting the visual features and the semantic features of the to-be-recognized document includes at least the following two cases for to-be-recognized documents in different formats:

[0089] In a first case, when the electronic device determines that the to-be-recognized document is a document in a file format, the electronic device can convert the to-be-recognized document into an image as a to-be-recognized image. For example, the electronic device can convert a to-be-recognized document in a file format such as pdf, word, xml, etc. into an image to obtain a to-be-recognized image, which can be an image for subsequent visual feature extraction.

[0090] For the case where the to-be-recognized document is a document in a file format, the electronic device can also parse the to-be-recognized document by using a document parsing software to obtain text content and text positions. The document parsing software can be apache Tika, pdfminer, PDF Expert, etc. and is not limited herein. The text content at least includes text content at a character level and text content at a text line level, which ensures that semantic features at at least the character level and the text line level can be obtained subsequently.

[0091] For example, if the file format of the to-be-recognized document A is pdf, the electronic device can convert the to-be-recognized document A in the file format of pdf into an image by using a conversion software, and use the image as a to-be-recognized image for subsequent visual feature extraction. The electronic device can also parse the to-be-recognized document A by using a document parsing software such as pdfminer to obtain text content at a character level and text content at a text line level and text positions included in the to-be-recognized document A, so that semantic features at at least the character level and the text line level can be obtained subsequently.

[0092] In a second case, when the electronic device determines that the to-be-recognized document is a document in an image format, the electronic device can use an OCR (Optical Character Recognition) technology to recognize the to-be-recognized document to obtain text content and text positions included in the to-be-recognized document, wherein the text content at least includes text content at a character level and text content at a text line level.

[0093] In an embodiment, the electronic device can input the document to be recognized in an image format into a detection network, and then obtain the text position at the character level and the text line level included in the document to be recognized, wherein the detection network can include a Faster RCNN (Faster Region with CNN feature), a YOLO, and a SSD (Single Shot MultiBox Detector) detection network, and the like, and the electronic device can further input the document to be recognized in an image format into a recognition network, and then obtain the text content of the document to be recognized, wherein the recognition network can include a sequence recognition framework, an Attention architecture, and a single word recognition, and the like, without being specifically limited.

[0094] S203, extract the visual features of the image to be recognized, and determine the semantic features of the document to be recognized based on the text content and the text position.

[0095] After obtaining the text content and the text position included in the image to be recognized and the document to be recognized, the electronic device can perform visual feature extraction on the image to be recognized, and then obtain the visual features of the document to be recognized. And can determine at least the semantic features at the character level and the semantic features at the text line level based on the text content and the text position at the character level and the text line level included in the document to be recognized.

[0096] As can be seen, in the present embodiment, when it is judged that the document to be recognized is a document in a file format, the electronic device can convert the document to be recognized into an image as an image to be recognized, perform file analysis on the document to be recognized, and obtain the text content and the text position included in the document to be recognized. When it is judged that the document to be recognized is a document in an image format, the electronic device can take the document to be recognized as an image to be recognized, perform image recognition on the document to be recognized, and obtain the text content and the text position included in the document to be recognized. Further, the electronic device can extract the visual features of the image to be recognized, and determine the semantic features of the document to be recognized based on the text content and the text position. For the document to be recognized in different file formats, the electronic device can obtain accurate visual features and semantic features in different ways, and ensure the accuracy of the subsequent document layout recognition result.

[0097] As an embodiment of the present application, as shown in Figure 3 The above step of extracting the visual features of the image to be recognized, and determining the semantic features of the document to be recognized based on the text content and the text position can include:

[0098] S301, fill the text content into the document to be recognized according to the text position, and obtain a semantic feature map;

[0099] After obtaining the text content and the text position included in the to-be-identified document, the electronic device can fill the text content into the to-be-identified document according to the corresponding text position to obtain a semantic feature map, which can have a size of height x width x D, where the height and the width are the same as the height and the width of the to-be-identified image, and D is a feature dimension, for example, 64, 128, etc. The size of the to-be-identified image is height x width x K, where K is the number of color channels of the to-be-identified image, for example, K is 1 when the to-be-identified image is a black and white image, and K is 3 when the to-be-identified image is a color image. The semantic feature map obtained by filling the text content into the to-be-identified document according to the text position has the same height and width as the to-be-identified image.

[0100] For example, the to-be-identified image corresponding to the to-be-identified document B is to-be-identified image C, the corresponding semantic feature map is semantic feature map A, and the to-be-identified image C is a color image. The size of the to-be-identified image C can be 18cm x 25cm x 3, and the size of the semantic feature map A can be 18cm x 25cm x 68.

[0101] S302, input the to-be-identified image and the semantic feature map into a first convolutional neural network established in advance to obtain visual features and semantic features output by the first convolutional neural network; or input the to-be-identified image into a second convolutional neural network established in advance to obtain visual features output by the second convolutional neural network, and input the semantic feature map into a third convolutional neural network established in advance to obtain semantic features output by the third convolutional neural network.

[0102] In order to obtain the visual features and the semantic features of the to-be-identified document, the electronic device can use any one of the following three methods to obtain the visual features and the semantic features of the to-be-identified document:

[0103] The first method is that the electronic device can input the above-mentioned semantic feature map as a semantic feature and the above-mentioned to-be-identified image as a visual input into a fourth convolutional neural network established in advance, where the fourth convolutional neural network can include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer, etc. The convolutional layer can perform convolutional processing on the to-be-identified image input by the input layer to extract color features, texture features, shape features, etc., and then reduce the size of the features through the pooling layer, classify the image features through the fully connected layer, and output the visual features through the output layer. Then the electronic device can obtain the visual features output by the fourth convolutional neural network.

[0104] In the second mode, the electronic device can input the to-be-identified image as a visual input and the semantic feature map as a semantic input into a first convolutional neural network established in advance, the first convolutional neural network can perform convolutional processing on the to-be-identified image to obtain a visual feature and output the visual feature. Meanwhile, the first convolutional neural network can perform convolutional processing on the semantic feature map to extract a semantic feature and output the semantic feature. Then, the electronic device can obtain the visual feature and the semantic feature output by the first convolutional neural network.

[0105] In the second mode, the electronic device can obtain the visual feature and the semantic feature by inputting the to-be-identified image and the semantic feature map into the first convolutional neural network, that is, the visual feature and the semantic feature can be obtained by using only one network, so the network parameter quantity is relatively small.

[0106] In the third mode, the electronic device can input the to-be-identified image as a visual input into a second convolutional neural network established in advance to obtain a visual feature output by the second convolutional neural network, and input the semantic feature map as a semantic input into a third convolutional neural network established in advance to obtain a semantic feature output by the third convolutional neural network.

[0107] The second convolutional neural network can perform convolutional processing on the to-be-identified image to obtain a visual feature and output the visual feature. The third convolutional neural network can perform convolutional processing on the semantic feature map to extract a semantic feature and output the semantic feature. In this way, the second convolutional neural network and the third convolutional neural network can output the visual feature and the semantic feature specific to the modal, respectively.

[0108] It can be seen that, in this embodiment, the electronic device can fill the text content into the to-be-identified document according to the text position to obtain a semantic feature map, input the to-be-identified image and the semantic feature map into a first convolutional neural network established in advance to obtain a visual feature and a semantic feature output by the first convolutional neural network, or input the to-be-identified image into a second convolutional neural network established in advance to obtain a visual feature output by the second convolutional neural network, and input the semantic feature map into a third convolutional neural network established in advance to obtain a semantic feature output by the third convolutional neural network. In this way, the electronic device can obtain accurate visual features and semantic features of the to-be-identified document in different ways according to different situations, so as to ensure the accuracy of the document layout recognition result.

[0109] As an implementation manner of the embodiment of the present application, the step of fusing the image feature and the semantic feature to obtain a multi-modal document feature can include:

[0110] The image feature and the semantic feature are input into a neural network established in advance to obtain a multi-modal document feature output by the neural network.

[0111] In order to obtain the complementary characteristics of the visual features and the semantic features, a neural network for feature fusion of the visual features and the semantic features can be established in advance, which can make full use of the complementary characteristics of the features of the input model and output the fused multi-modal features. The neural network is used to fuse the image features and the semantic features to obtain the fused multi-modal features.

[0112] In this way, the electronic device can input the image features and the semantic features into the pre-established neural network, which can process the fused multi-modal document features by using the complementary characteristics between the image features and the semantic features, and then output the multi-modal document features. The electronic device can obtain the multi-modal document features output by the feature fusion model.

[0113] The neural network can be a convolutional neural network, a fully connected network, or any other neural network capable of feature fusion, which is not limited herein. Specifically, the neural network can calculate the multi-modal document features M based on the following formula:

[0114] M = A * V + (1-A) * S

[0115] wherein V is the visual feature, S is the semantic feature, A = f(V, S), A is the weighting weight obtained by training the neural network based on the visual feature sample and the semantic feature sample, and f is the mapping function corresponding to the neural network.

[0116] Since the weighting weight corresponding to the semantic feature and the weighting weight corresponding to the visual feature are obtained based on the mapping function corresponding to the neural network, and the neural network is trained based on the image feature sample and the semantic feature sample, the parameters of the mapping function are adjusted during the training process, so that the parameters gradually become accurate. Therefore, the weighting weight obtained by using the mapping function can accurately represent the importance of the visual feature and the semantic feature for the fused features. Therefore, the multi-modal document features M obtained by multiplying the semantic feature by the corresponding weighting weight and the visual feature by the corresponding weighting weight are accurate, which can more accurately represent the visual features of the overall layout of the image corresponding to the to-be-recognized document, as well as the text line-level features and the character-level features of the to-be-recognized document. Therefore, compared with the current feature fusion methods such as direct addition and splicing, the feature can be adaptively fused, and the complementary characteristics between the feature modalities can be fully utilized.

[0117] As can be seen, in this embodiment, the electronic device can input image features and semantic features into a pre-established neural network. The neural network can calculate multimodal document features using the above formula, thereby enabling the electronic device to obtain the multimodal document features output by the neural network. In this way, adaptive fusion of features can be achieved, making full use of the complementary characteristics between feature modalities, thereby improving the accuracy of document layout recognition results.

[0118] As one embodiment of the present invention, the step of identifying the element position and category of each element in the document to be identified based on the multimodal document features may include:

[0119] Based on the multimodal document features, target detection is performed on the document to be identified to determine the element position and category of each element in the document to be identified.

[0120] After obtaining the above multimodal document features, the electronic device can perform target detection on the document to be identified based on the above multimodal document features, and then determine the element position and category of each element in the document to be identified. The target detection algorithm can be R-CNN, SPP-Net, OverFeat, etc., without specific limitations.

[0121] For example, such as Figure 4 As shown, the electronic device can use the OverFeat algorithm to perform target detection on the document C to be recognized, thereby obtaining a recognition result including multiple candidate boxes 410. This means that each element in the document C is distinguished using candidate boxes, and the position selected by the candidate box is the element's location. In one implementation, candidate boxes of different colors can be used to distinguish elements of different categories; in another implementation, the corresponding identifiers for each candidate box can be recorded, and different identifiers can be used to distinguish elements of different categories—both are reasonable. In this way, the electronic device can determine the element position and category of each element in the document to be recognized, thus completing the document layout recognition.

[0122] As can be seen, in this embodiment, the electronic device can perform target detection on the document to be identified based on multimodal document features, and determine the element position and category of each element in the document to be identified. Through the above scheme, since the multimodal document features are obtained by fusing visual features, text line-level semantic features and character-level semantic features, and character-level semantic features can extract text-level elements such as references and formulas embedded in the text, and visual features can identify visually rich elements such as figures and tables, the electronic device can obtain elements based on visual features such as figures and tables, as well as text line-level elements such as references and formulas, thereby improving the accuracy of document layout recognition results.

[0123] As an implementation of an embodiment of the present application, after the step of determining the element positions and categories of the elements in the to-be-recognized document, the method can further include:

[0124] adjusting the element positions and categories of the elements based on the pre-determined structural relationship between the document elements, to obtain the final element positions and categories of the elements.

[0125] Since there is a certain inherent structural relationship between the elements in various documents, in order to further improve the accuracy of the recognition result of the document layout, after the element positions and categories of the elements in the to-be-recognized document are determined, the electronic device can adjust the element positions and categories of the elements based on the pre-determined structural relationship between the document elements, to obtain the final element positions and categories of the elements.

[0126] The pre-determined structural relationship between the document elements can be a binding relationship between elements, a relative position relationship between elements, and a paragraph alignment relationship between elements, and the like, which are not limited here.

[0127] The binding relationship between elements can be a fixed collocation relationship between a figure and a figure caption, a figure and a figure icon, and a formula and a formula serial number, the relative position relationship between elements can be a relative position relationship that a formula is located below a table, a formula meaning explanation is located below a formula, and a sub-title is located below a main title, and the paragraph alignment relationship between elements can be an alignment relationship that each paragraph is aligned at both ends.

[0128] For example, the pre-established structural relationship between the document elements includes that a figure caption (1) is located below a figure a and is centered, and then the electronic device can adjust the element positions and categories of the elements in the to-be-recognized document according to the structural relationship, that is, the figure caption (1) in the to-be-recognized document is located below the figure a and the element position is centered.

[0129] It can be seen that in the embodiment, the electronic device can adjust the element positions and categories of the elements based on the pre-determined structural relationship between the document elements, to obtain the final element positions and categories of the elements, and since the pre-determined structural relationship between the document elements can more accurately represent the relationship between the elements, the accuracy of the recognition result of the document layout can be further improved.

[0130] As an implementation of an embodiment of the present application, the step of adjusting the element positions and categories of the elements based on the pre-determined structural relationship between the document elements, to obtain the final element positions and categories of the elements, can include:

[0131] The multi-modal document features, the element positions and the categories of the elements are input into a pre-established relationship modeling model, so that the relationship modeling model adjusts the element positions and the categories of the elements based on the structural relationship between the document elements and the multi-modal document features, and outputs the final element positions and categories of the elements.

[0132] To improve the accuracy of the recognition result of the document layout, the electronic device can pre-establish a relationship modeling model, which can be trained based on multi-modal document feature samples and element position and category samples of elements, and is used to adjust the element positions and categories of input elements based on the structural relationship between the document elements and the multi-modal document features.

[0133] Therefore, after obtaining the multi-modal document features and the element positions and categories of the elements, the electronic device can input the multi-modal document features, the element positions and the categories of the elements into the pre-established relationship modeling model.

[0134] The relationship modeling model can determine the elements in the to-be-recognized document based on the element positions and the categories of the elements, take each element as a node, and the feature of each node can include the position information of the element and the multi-modal features. Further, the relationship modeling model can depict the relationship between two nodes by using the self-attention technology, that is, compare the nodes two by two. When comparing each two nodes, the feature of each node is updated based on the pre-determined structural relationship between the document elements, and then the final element positions and categories of the elements are determined. The comparison process between each two nodes can be performed simultaneously, and there is no limitation on the sequence.

[0135] That is, after the electronic device inputs the multi-modal document features, the element positions and the categories of the elements into the pre-established relationship modeling model, the relationship modeling model can adjust the element positions and the categories of the elements based on the pre-determined structural relationship between the document elements and the multi-modal document features, and output the final element positions and categories of the elements which are more accurate.

[0136] For example, for the position information and the categories of the elements shown in the candidate box 410 in FIG. 4B, the relationship modeling model can determine the nodes representing the elements based on the position information and the categories of the elements shown in the candidate box 410, and the nodes are shown in FIG. 4C. Figure 4 Figure 4 ​Z1, Z2…Z8 shown in 420. And the nodes corresponding to elements of different categories can be represented in different colors, for example, icons can be represented in green, figure captions can be represented in blue, text can be represented in yellow, etc. The relationship modeling model can depict the relationship between each pair of nodes through self-attention technology, that is, the nodes are compared pairwise, and the pairwise comparison process can be performed simultaneously. Further, the positions and categories of the nodes are adjusted based on the structural relationship of the document. For example, when updating the features of node Z1, the features of Z1 can be adjusted based on the structural relationship between the remaining nodes and Z1 to obtain the adjusted position and category of Z1. For another example, when adjusting node Z2, since node Z2 contains node Z3 and node Z6, and node Z3 is a graph and node Z6 is text, which is not a figure caption, it can be determined that node Z2 is misidentified, so the node features are updated and node Z2 is deleted to obtain the node shown on the right side of 420. The final output of the recognition result is shown in 430, and the element position and category of the element are obtained more accurately.

[0137] It can be seen that in the embodiment, the electronic device can input the multi-modal document features, the element positions and categories of the elements into the pre-established relationship modeling model, so that the relationship modeling model adjusts the element positions and categories of the elements based on the pre-determined structural relationship between the document elements and the multi-modal document features, and outputs the final element positions and categories of the elements. In this way, the electronic device can input the multi-modal document features, the element positions and categories of the elements into the pre-established relationship modeling model, and the relationship modeling model can output the final element positions and categories of the elements, thereby improving the accuracy of the recognition result of the document layout.

[0138] As an implementation manner of the embodiment of the present application, the above-mentioned pre-established first convolutional neural network, second convolutional neural network, third convolutional neural network, fourth convolutional neural network, neural network for feature fusion of image features and semantic features, and relationship modeling model and other network models can be trained separately. The training manner can use any training manner in the field of model training, as long as the corresponding model can be trained, for example, gradient descent algorithm, stochastic gradient descent algorithm, etc. can be used for model training, which is not limited here.

[0139] For example, the training manner of the neural network for feature fusion of image features and semantic features can be: obtaining image feature samples and semantic feature samples and an initial model, taking the multi-modal document features corresponding to each image feature sample and semantic feature sample as sample labels, inputting each image feature sample and semantic feature sample into the initial model to obtain a prediction result, adjusting model parameters of the initial model based on a difference between the prediction result and the multi-modal document features until the initial model converges, and then obtaining the neural network for feature fusion of image features and semantic features.

[0140] In another implementation, the above models can constitute an overall model for document layout recognition, and the above models can be trained as a whole. Specifically, the training process can be: obtaining an initial model, the initial model including an initial first convolutional neural network, an initial neural network, and an initial relationship modeling model, or including an initial second convolutional neural network, an initial third convolutional neural network, an initial neural network, and an initial relationship modeling model. The training sample is a document image and a semantic feature map corresponding to the document image, the recognition result of the document layout corresponding to the document image is taken as a sample label, and then the sample is input into the initial model to obtain a prediction result. Based on a difference between the prediction result and the sample label, the model parameters of the initial model are adjusted until the initial model converges, and the overall model for document layout recognition is obtained.

[0141] The following will be described in combination with Figure 5 The recognition method of the document layout provided by the embodiment of the application is exemplarily introduced. Specifically, the electronic device can input the to-be-recognized image 501 corresponding to the to-be-recognized document into the pre-established second convolutional neural network 503 to obtain visual features output by the second convolutional neural network, input the semantic feature map 502 into the pre-established third convolutional neural network 504 to obtain semantic features output by the third convolutional neural network. Then, the visual features and the semantic features are input into the neural network 505, and multi-modal document features can be obtained. The multi-modal document features, the element positions and categories of the elements are input into the pre-established relationship modeling model 506, and the relationship modeling model can adjust the element positions and categories of the elements based on the pre-determined structural relationship between the document elements and the multi-modal document features, output the final element positions and categories of the elements, and then obtain the recognition result 507.

[0142] It can be seen that in the embodiment, the electronic device can acquire the semantic features at the text line level and the semantic features at the character level, and can identify the visual features of the overall layout of the image corresponding to the to-be-identified document. Therefore, the electronic device can identify based on the multi-modal document features obtained by fusing the visual features, the semantic features at the text line level and the semantic features at the character level, and can obtain elements including graphs, tables and other elements based on the visual features, and elements such as references and formulas in the text line at the character level. Therefore, the electronic device can make the recognition result of the document layout more comprehensive, and greatly improve the accuracy of the recognition result of the document layout.

[0143] Corresponding to the above-mentioned document layout recognition method, the embodiment of the present application further provides a document layout recognition device. The document layout recognition device provided by the embodiment of the present application will be introduced below.

[0144] As shown in Figure 6 A document layout recognition device, the device can include:

[0145] The acquisition module 610 is configured to acquire a to-be-identified document.

[0146] The extraction module 620 is configured to extract visual features and semantic features of the to-be-identified document.

[0147] The visual features identify the visual features on the overall layout of the image corresponding to the to-be-identified document, and the semantic features at least include the features at the character level and the features at the text line level.

[0148] The fusion module 630 is configured to fuse the image features and the semantic features to obtain multi-modal document features.

[0149] The recognition module 640 is configured to identify the element positions and categories of the elements in the to-be-identified document based on the multi-modal document features.

[0150] It can be seen that in the scheme provided by the embodiment of the present application, the electronic device can obtain a to-be-recognized document, extract visual features and semantic features of the to-be-recognized document, wherein the visual features identify visual characteristics of the overall layout of the image corresponding to the to-be-recognized document, and the semantic features at least include character-level features and text line-level features, the image features and the semantic features are fused to obtain multi-modal document features, and the element positions and categories of each element in the to-be-recognized document are recognized based on the multi-modal document features. Through the above scheme, the electronic device can obtain the semantic features at the text line level and the semantic features at the character level, and the visual features capable of identifying the overall layout characteristics of the image corresponding to the to-be-recognized document. Since the semantic features at the character level can extract text-level elements such as references and formulas embedded in the text, and the visual features can recognize visual-rich elements such as graphs and tables, the multi-modal document features obtained by fusing the visual features, the semantic features at the text line level and the semantic features at the character level are used for recognition, which can obtain elements including graphs and tables based on the visual features, and elements such as references and formulas at the character level within the text line, so that the recognition result of the document layout is more comprehensive, and the accuracy of the document layout recognition result is greatly improved.

[0151] As an embodiment of the present application, the above extraction module 620 can include:

[0152] The first processing unit is configured to, when it is judged that the to-be-recognized document is a document in a file format, convert the to-be-recognized document into an image as a to-be-recognized image, and perform file analysis on the to-be-recognized document to obtain text content and text positions included in the to-be-recognized document;

[0153] The second processing unit is configured to, when it is judged that the to-be-recognized document is a document in an image format, take the to-be-recognized document as a to-be-recognized image, and perform image recognition on the to-be-recognized document to obtain text content and text positions included in the to-be-recognized document;

[0154] The text content at least includes text content at a character level and text content at a text line level.

[0155] The feature extraction unit is configured to extract visual features of the to-be-recognized image, and determine semantic features of the to-be-recognized document based on the text content and the text positions.

[0156] As an embodiment of the present application, the above feature extraction unit can include:

[0157] The padding sub-unit is configured to fill the text content into the to-be-recognized document according to the text positions to obtain a semantic feature map.

[0158] The first extraction subunit is configured to input the image to be recognized and the semantic feature map into a first convolutional neural network established in advance to obtain visual features and semantic features output by the first convolutional neural network.

[0159] The second extraction subunit is configured to input the image to be recognized into a second convolutional neural network established in advance to obtain visual features output by the second convolutional neural network, and input the semantic feature map into a third convolutional neural network established in advance to obtain semantic features output by the third convolutional neural network.

[0160] As an embodiment of the present application, the fusion module 630 can include:

[0161] The feature fusion unit is configured to input the image features and the semantic features into a neural network established in advance to obtain multi-modal document features output by the neural network.

[0162] The neural network is configured to fuse the image features and the semantic features, and the multi-modal document features M are calculated based on the following formula:

[0163] M = A * V + (1-A) * S

[0164] wherein V represents the visual features, S represents the semantic features, A = f(V, S), A is a weighting weight obtained by training the neural network based on the visual feature samples and the semantic feature samples, and f is a mapping function corresponding to the neural network.

[0165] As an embodiment of the present application, the recognition module 640 can include:

[0166] The target detection unit is configured to perform target detection on the document to be recognized based on the multi-modal document features to determine the element positions and categories of the elements in the document to be recognized.

[0167] As an embodiment of the present application, as shown in Figure 7 The apparatus can further include:

[0168] The adjustment module 650 is configured to, after determining the element positions and categories of the elements in the document to be recognized, adjust the element positions and categories of the elements based on the structural relationship between the document elements determined in advance to obtain the final element positions and categories of the elements.

[0169] As an embodiment of the present application, the adjustment module 650 can include:

[0170] An output sub-unit is configured to input the multi-modal document features, the element positions and categories of the elements into a pre-established relationship modeling model, so that the relationship modeling model adjusts the element positions and categories of the elements based on the structural relationships between the document elements and the multi-modal document features, and outputs the final element positions and categories of the elements.

[0171] The embodiment of the present application further provides an electronic device, which comprises Figure 8 As shown in the figure, the electronic device comprises a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 complete mutual communication through the communication bus 804,

[0172] The memory 803 is configured to store a computer program.

[0173] The processor 801 is configured to execute the program stored in the memory 803, and implement the steps of the document layout recognition method in any of the above embodiments.

[0174] It can be seen that in the scheme provided by the embodiment of the present application, the electronic device can acquire a to-be-recognized document, extract visual features and semantic features of the to-be-recognized document, wherein the visual features identify visual characteristics on the overall layout of the image corresponding to the to-be-recognized document, and the semantic features at least include character-level features and text line-level features, the image features and the semantic features are fused to obtain multi-modal document features, and the element positions and categories of the elements in the to-be-recognized document are recognized based on the multi-modal document features. Through the above scheme, the electronic device can acquire the semantic features at the text line level and the semantic features at the character level, and the visual features capable of identifying the overall layout characteristics of the image corresponding to the to-be-recognized document. Since the semantic features at the character level can extract text-level elements such as references and formulas embedded in the text, and the visual features can recognize visual-rich elements such as graphs and tables, the multi-modal document features obtained by fusing the visual features, the semantic features at the text line level and the semantic features at the character level are used for recognition, which can obtain elements such as graphs and tables based on the visual features, and elements such as references and formulas at the character level within the text line, so that the recognition result of the document layout is more comprehensive, and the accuracy of the document layout recognition result is greatly improved.

[0175] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0176] The communication interface is used for communication between the above electronic device and other devices.

[0177] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the above-mentioned processor.

[0178] The above-mentioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0179] In another embodiment provided by the application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the document layout recognition method in any of the above embodiments are implemented.

[0180] In another embodiment provided by the application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute the document layout recognition method in any of the above embodiments.

[0181] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and loaded into the computer for execution. The computer readable medium includes computer storage media and communication media. The computer storage media includes any tangible or physical medium for storing or transmitting the program. The computer storage media can be a volatile (e.g., RAM) or non-volatile (e.g., ROM, disk, or CD) storage medium. The communication media typically include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The computer readable media does not include carrier waves or other transient signals.

[0182] It should be noted that, in the specification, the relative terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In addition, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0183] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the apparatus, electronic device, computer readable storage medium and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0184] The above merely describes the preferred embodiments of the present application, but is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of recognizing a document layout, characterized by, The method comprises: acquiring a to-be-recognized document; extracting visual features and semantic features of the to-be-recognized document, wherein the visual features identify visual characteristics on overall layout of an image corresponding to the to-be-recognized document, the semantic features at least include character-level features and text line-level features, the visual characteristics are visual characteristics formed by graphs, arrangement and positional relationship of the graphs in the image corresponding to the to-be-recognized document, the character-level features represent characters themselves and relationship features between the characters in a text line, and the text line-level features represent the text line itself and relationship features between text lines; fusing the visual features and the semantic features to obtain multi-modal document features; based on the multi-modal document features, recognizing element positions and categories of elements in the to-be-recognized document; wherein the extracting the visual features and the semantic features of the to-be-recognized document comprises: obtaining text content and text positions included in the to-be-recognized document, wherein the text content at least includes character-level text content and text line-level text content; filling the text content into the to-be-recognized document according to the text positions to obtain a semantic feature map; extracting visual features of the to-be-recognized document and determining semantic features of the to-be-recognized document based on the semantic feature map.

2. The method of claim 1, wherein, The step of obtaining the text content and the text positions included in the to-be-recognized document comprises: when judging that the to-be-recognized document is a document in a file format, converting the to-be-recognized document into an image as a to-be-recognized image; performing file analysis on the to-be-recognized document to obtain the text content and the text positions included in the to-be-recognized document; when judging that the to-be-recognized document is a document in an image format, taking the to-be-recognized document as a to-be-recognized image; performing image recognition on the to-be-recognized document to obtain the text content and the text positions included in the to-be-recognized document.

3. The method of claim 2, wherein, The step of extracting the visual features of the to-be-recognized document and determining the semantic features of the to-be-recognized document based on the semantic feature map comprises: inputting the to-be-recognized image and the semantic feature map into a first convolutional neural network established in advance to obtain visual features and semantic features output by the first convolutional neural network; or inputting the to-be-recognized image into a second convolutional neural network established in advance to obtain visual features output by the second convolutional neural network, and inputting the semantic feature map into a third convolutional neural network established in advance to obtain semantic features output by the third convolutional neural network.

4. The method of claim 1, wherein, The step of fusing the visual features and the semantic features to obtain multi-modal document features comprises: inputting the visual features and the semantic features into a neural network established in advance to obtain multi-modal document features output by the neural network, wherein the neural network is used for feature fusion of visual features and semantic features, and the neural network calculates the multi-modal document features M based on the following formula: ; Wherein, V is the visual feature, S is the semantic feature, A=f(V, S), A is the weighted weight obtained by training the neural network through the visual feature sample and the semantic feature sample, and f is a mapping function corresponding to the neural network.

5. The method according to any one of claims 1 to 4, characterized in that, The step of identifying the element position and the category of each element in the to-be-identified document based on the multi-modal document feature comprises: The step of identifying the element position and the category of each element in the to-be-identified document based on the multi-modal document feature comprises:

6. The method of claim 5, wherein, After the step of determining the element position and the category of each element in the to-be-identified document, the method further comprises: The step of adjusting the element position and the category of each element based on the pre-determined structural relationship between document elements to obtain the final element position and the category of each element comprises:

7. The method of claim 6, wherein, The step of adjusting the element position and the category of each element based on the pre-determined structural relationship between document elements to obtain the final element position and the category of each element comprises: The step of adjusting the element position and the category of each element based on the pre-determined structural relationship between document elements to obtain the final element position and the category of each element comprises:

8. A document layout recognition apparatus, characterized by comprising: The step of adjusting the element position and the category of each element based on the pre-determined structural relationship between document elements to obtain the final element position and the category of each element comprises: The device comprises: An acquisition module configured to acquire a to-be-identified document; An extraction module configured to extract visual features and semantic features of the to-be-identified document, wherein the visual features represent visual characteristics on an overall layout of an image corresponding to the to-be-identified document, and the semantic features comprise at least character-level features and text line-level features, the visual characteristics are visual characteristics formed by images, arrangement and positional relationship of the images in the image corresponding to the to-be-identified document, the character-level features represent features of characters themselves and relationship between the characters in a text line, and the text line-level features represent features of text lines themselves and relationship between the text lines. A fusion module configured to fuse the visual features and the semantic features to obtain multi-modal document features. An identification module configured to identify element positions and categories of each element in the to-be-identified document based on the multi-modal document features. The extraction module is specifically configured to: obtain text content and text positions included in the to-be-identified document, wherein the text content comprises at least character-level text content and text line-level text content; fill the text content into the to-be-identified document according to the text positions to obtain a semantic feature map; 9. An electronic device, comprising: extract visual features of the to-be-identified document and determine semantic features of the to-be-identified document based on the semantic feature map. The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method steps in any one of claims 1-7. Wherein, V is the visual feature, S is the semantic feature, A=f(V, S), A is the weighted weight obtained by training the neural network through the visual feature sample and the semantic feature sample, and f is a mapping function corresponding to the neural network.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-7.

Images