Document image processing method and device, equipment and storage medium
By extracting the layout and semantic features in the document image, generating a document tree and serializing it, the document content processing problem that the existing technology cannot meet the diverse application scenarios is solved, and the sequential relationship display and application of document content is realized.
Patent Information
- Application Number
- CN202411347182.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-07
AI Technical Summary
The prior art cannot meet the needs of use of document content in diverse application scenarios, especially in document retrieval scenarios, and it is difficult to effectively process the sequential relationship of document content in document images.
By obtaining the location and content information of the processing objects in the document image, extracting the layout feature vector and semantic feature vectors, generating a document tree, using the tree structure to represent the sequential relationship between the processing objects, and converting it into linearity through serialization processing Structure, easy to use in different application scenarios.
It realizes effective display and application of the order relationship of processing objects in document images, supports document content processing in diverse scenarios, and meets the needs of diversified application.
Smart Images
Figure CN119274198A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a document image processing method, device, equipment and storage medium. Background Art
[0002] Document images refer to images containing text, pictures and other contents, such as images in PDF format. Currently, document image processing includes extracting document contents such as text, tables, and illustrations from document images by means of layout analysis and text recognition, and then displaying the extracted document contents in the reading order of the document images.
[0003] However, the above-mentioned method of processing document images can no longer meet the usage requirements of document content in various application scenarios. Therefore, how to process document images is a technical problem that urgently needs to be solved. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a document image processing method, device, equipment and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a document image processing method, the method comprising:
[0006] Acquire a target document image, and extract position information and content information of a processing object on the target document image;
[0007] Acquire a layout feature vector of the processing object based on the position information of the processing object, and acquire a semantic feature vector of the processing object based on the content information of the processing object;
[0008] Based on the layout feature vectors and semantic feature vectors of each processing object on the target document image, a document tree corresponding to the target document image is generated; wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0009] In an optional implementation, generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image includes:
[0010] Determining a sequential relationship between the processing objects on the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image;
[0011] The processing objects are respectively used as nodes of a tree structure, and a document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
[0012] In an optional implementation manner, before constructing the document tree corresponding to the target document image based on the sequential relationship between the processing objects, the method further includes:
[0013] Determine, based on the layout feature vector and the semantic feature vector of each processing object on the target document image, the category information corresponding to each processing object on the target document image; wherein the category information is used to characterize the category to which the corresponding processing object belongs in the document content of the target document image;
[0014] Accordingly, constructing a document tree corresponding to the target document image based on the sequential relationship between the processing objects includes:
[0015] Based on the sequential relationship between the processing objects and the category information corresponding to the processing objects, a document tree corresponding to the target document image is constructed; wherein the nodes corresponding to the processing objects with different category information are at different levels of the document tree.
[0016] In an optional implementation manner, after generating the document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image, the method further includes:
[0017] The document tree is serialized by pre-order traversal of the tree to obtain a result sequence of the document tree; wherein the result sequence is used to describe the sequential relationship between the processing objects on the target document image based on a linear structure.
[0018] In an optional embodiment, the result sequence includes at least one of a hierarchy separator, a parallel relationship separator and a subordinate relationship separator, wherein the hierarchy separator is used to identify that the corresponding different nodes are at different hierarchies in the tree structure, the parallel relationship separator is used to identify that the corresponding nodes are at the same hierarchical level in the tree structure, and the subordinate relationship separator is used to identify that the corresponding nodes are at a hierarchical level with a subordinate relationship in the tree structure.
[0019] In an optional implementation, the result sequence includes a category index separator, and the category index separator is used to identify the category information and index information of the same node.
[0020] In an optional implementation manner, after serializing the document tree to obtain the result sequence of the document tree, the method further includes:
[0021] The result sequence of the document tree is de-sequenced and reconstructed through pre-order traversal of the tree to generate a tree structure document corresponding to the target document image.
[0022] In an optional implementation manner, before generating the document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image, the method further includes:
[0023] Performing image segmentation processing on the target document image to obtain multiple image blocks;
[0024] Extracting visual feature vectors of the multiple image blocks respectively;
[0025] Accordingly, generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image includes:
[0026] Based on the layout feature vectors and semantic feature vectors of each processing object on the target document image and the visual feature vectors respectively corresponding to the plurality of image blocks, a document tree corresponding to the target document image is generated.
[0027] In an optional implementation, generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image and the visual feature vectors corresponding to the plurality of image blocks respectively includes:
[0028] Inputting the layout feature vector and semantic feature vector of each processing object on the target document image and the visual feature vectors corresponding to the plurality of image blocks into the decoder of the target reading order model, and obtaining the order relationship between the processing objects on the target document image by autoregressive decoding;
[0029] A document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
[0030] In an optional implementation manner, the processing object on the target document image includes a text line object and / or a non-text line object on the target document image, and the extracting the position information and content information of the processing object on the target document image includes:
[0031] Recognize the target document image using an optical character recognition method to obtain position information and content information of each text line object on the target document image;
[0032] and / or,
[0033] The target document image is detected using a layout analysis model to obtain position information and content information of each non-text object on the target document image.
[0034] In a second aspect, the present disclosure provides a document image processing device, the device comprising:
[0035] A first extraction module, used to obtain a target document image and extract position information and content information of a processing object on the target document image;
[0036] An acquisition module, used to acquire a layout feature vector of the processing object based on the position information of the processing object, and to acquire a semantic feature vector of the processing object based on the content information of the processing object;
[0037] A generation module is used to generate a document tree corresponding to the target document image based on the layout feature vector and semantic feature vector of each processing object on the target document image; wherein the document tree represents the sequential relationship between the processing objects on the target document image in a tree structure.
[0038] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a document image processing method as provided in an embodiment of the present disclosure.
[0039] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the document image processing method provided by the embodiment of the present disclosure.
[0040] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.
[0041] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0042] In the document image processing method provided by the embodiment of the present disclosure, first, a target document image is acquired, and the position information and content information of the processing object on the target document image are extracted. Then, a layout feature vector of the processing object is acquired based on the position information of the processing object, and a semantic feature vector of the processing object is acquired based on the content information of the processing object. Then, based on the layout feature vector and the semantic feature vector of each processing object on the target document image, a document tree corresponding to the target document image is generated, wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0043] It can be seen that the embodiment of the present disclosure generates a document tree corresponding to the target document image through the layout feature vectors and semantic feature vectors of each processing object in the target document image, and can display the sequential relationship between the various processing objects in the target document image through the tree structure, thereby facilitating the application of the document content in the target document image to various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0045] Figure 1 A flowchart of a document image processing method provided by an embodiment of the present disclosure;
[0046] Figure 2 A schematic diagram of a target document image provided by an embodiment of the present disclosure;
[0047] Figure 3 A schematic diagram of another target document image provided by an embodiment of the present disclosure;
[0048] Figure 4 A schematic diagram of a flow chart of another document image processing method provided by an embodiment of the present disclosure;
[0049] Figure 5 A schematic diagram of a document image processing process provided by an embodiment of the present disclosure;
[0050] Figure 6 A schematic diagram of the structure of a document image processing device provided by an embodiment of the present disclosure;
[0051] Figure 7 A schematic diagram of the structure of a document image processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0052] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0053] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0054] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0055] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0056] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0057] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0058] At present, the processing method of document images can no longer meet the use requirements of document content in various application scenarios, such as document retrieval scenarios. Therefore, how to process document images is a technical problem that needs to be solved urgently.
[0059] To this end, an embodiment of the present disclosure provides a document image processing method. Specifically, a target document image is acquired, and position information and content information of a processing object on the target document image are extracted. Then, a layout feature vector of the processing object is acquired based on the position information of the processing object, and a semantic feature vector of the processing object is acquired based on the content information of the processing object. Then, based on the layout feature vector and the semantic feature vector of each processing object on the target document image, a document tree corresponding to the target document image is generated, wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0060] It can be seen that the embodiment of the present disclosure generates a document tree corresponding to the target document image through the layout feature vectors and semantic feature vectors of each processing object in the target document image, and can display the sequential relationship between the various processing objects in the target document image through the tree structure, thereby facilitating the application of the document content in the target document image to various scenarios.
[0061] Based on this, an embodiment of the present disclosure provides a document image processing method, which is introduced below in conjunction with a specific embodiment.
[0062] Figure 1 The present invention provides a flowchart of a document image processing method provided by an embodiment of the present invention. The method can be executed by a document image processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 1 As shown, the method includes:
[0063] S101: Acquire a target document image, and extract position information and content information of a processing object on the target document image.
[0064] The document image processing method provided by the embodiments of the present disclosure is applied to a client. Specifically, the client can be deployed on a terminal such as a smart phone, a tablet computer, or a desktop computer.
[0065] The target document image may be any image including a document, and the document may include text, an image, a table, etc. For example, the document may be a paper. The number of the target document images may be one or more, which is specifically set according to the actual situation.
[0066] Regarding the method for acquiring the target document image, specifically, the target document image can be obtained by real-time shooting or by using a network. Regarding the method for acquiring the target document image, the disclosed embodiment does not limit it. For example, the target document image can be an image taken of a certain paper.
[0067] like Figure 2 2 is a schematic diagram of a target document image provided by an embodiment of the present disclosure. The target document image 201 is the front page of a certain paper.
[0068] The processing object in the embodiment of the present disclosure may be a general term for the document content on the target document image, and the processing object may include a text line object and a non-text object. The text line object may be a text unit composed of one or more consecutive characters, and the characters may include at least one of words, letters, numbers, punctuation marks, and spaces, etc. The text line object may be a sentence or a phrase, for example, a text line object is a line of content; non-text objects may include images and / or tables.
[0069] Among them, the position of the processing object of the target document image can include the four vertex coordinates of the area where the processing object is located in the target document image, the content of the processing object can be multiple characters included in the processing object in the target document image, and the number of processing objects can be multiple.
[0070] Specifically, when the target document image is acquired, the recognition algorithm can be used to recognize the target document image to determine the position and content of the text line object and the position and content of the non-text object contained in the target document image. The specific processing logic of the recognition algorithm can be specifically set according to the actual situation and is not limited here.
[0071] In an optional embodiment, when a target document image is acquired, an optical character recognition method is used to recognize characters in the target document image and perform text analysis to determine the position and content of a text line object contained in the target document image, and obtain the corresponding position and content of the text line object as the position information and content information of the text line object on the target document image.
[0072] In another optional implementation, when the target document image is acquired, the target document image is detected using a layout analysis model to determine the display area and content of the non-text object as the location information and content information of the non-text object on the target document image.
[0073] like Figure 3 FIG. 3 is a schematic diagram of another target document image provided by an embodiment of the present disclosure. The target document image includes multiple text line objects, each of which is framed by a rectangular frame to indicate the recognized position and content, as shown in the text line object 301. The target document image also includes non-text objects, as shown in the picture 302, which are framed by a rectangular frame to indicate the recognized position and content.
[0074] S102: Acquire a layout feature vector of the processing object based on the position information of the processing object, and acquire a semantic feature vector of the processing object based on the content information of the processing object.
[0075] The layout feature vector of the processing object is used to describe the layout features of the processing object. The layout feature vector can be a multi-dimensional vector, and each dimension can represent a feature of the processing object. The layout feature vector can include spatial position features, such as the coordinates of text lines, pictures, and tables in the target document image; the layout feature vector can also include geometric shape features, such as the aspect ratio of the picture; the layout feature vector can also include structural relationship features, that is, the arrangement of each processing object in the target document image, relative positions, etc., such as the line spacing between text lines, etc.
[0076] In an optional implementation, feature extraction is performed on the position information of the processing object to obtain a layout feature vector of the processing object.
[0077] Specifically, the position information of the processing object may be input into the model, and after being processed by the model, a layout feature vector of the processing object may be obtained.
[0078] In the embodiment of the present disclosure, the semantic feature vector of the processing object is used to describe the semantic information of the processing object. The semantic feature vector may include the meaning, context, logic and other features corresponding to the processing object.
[0079] In an optional implementation, feature extraction is performed on the content information of the processing object to obtain a semantic feature vector of the processing object. Specifically, the text content contained in the content information of the processing object can be segmented to obtain multiple words, and then the multiple words are encoded to obtain multiple word features, and the mean of the multiple word features is determined to obtain the semantic feature of the processing object.
[0080] In one application scenario, a word segmenter can be used to segment the text content contained in the content information of the processing object to obtain multiple words. The word here refers to the smallest unit (token) in the word segmentation process, which can be a word, letter, number, punctuation mark, etc. Then, an encoder can be used to encode each word to obtain the corresponding word feature, and a mean pooling layer can be used to mean pool the multiple word features, and the mean of the output multiple word features is determined as the semantic feature vector of the processing object.
[0081] S103: Generate a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image.
[0082] The document tree represents the sequential relationship between the processing objects on the target document image in a tree structure.
[0083] The document tree is a data structure that represents each processing object on the target document image in a tree structure. The order relationship between each processing object on the target document image can be reflected through the parent-child relationship between each node on the document tree.
[0084] In an optional embodiment, after obtaining the layout feature vector and semantic feature vector of each processing object on the target document image, the sequential relationship between the processing objects on the target document image is determined based on the layout feature vector and the semantic feature vector. Then, each processing object is used as a node of a tree structure, and a document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
[0085] Specifically, based on the layout feature vector and semantic feature vector of the processing object, the up-down, left-right, and arrangement order of each processing object on the target document image may be determined.
[0086] There is a corresponding relationship between the processing object and the node of the tree structure. Each processing object on the target document image is taken as a node of the tree structure. That is to say, the node of the tree structure contains the information of the corresponding processing object, which may specifically include the location information, content information, etc. of the processing object.
[0087] Regarding constructing a document tree corresponding to the target document image based on the sequential relationship between the processing objects, specifically, the nodes of the tree structure can be determined in sequence according to the sequence between the processing objects, and then the document tree can be constructed to represent the hierarchical structure of the target document image.
[0088] In an optional implementation, in order to facilitate the acquisition of processing objects under a certain category on the target document image, the disclosed embodiment can also determine the category information corresponding to each processing object on the target document image before constructing the document tree corresponding to the target document image. The category information is used to characterize the category to which the corresponding processing object belongs in the document content of the target document image. Processing objects of different categories are in different hierarchical structures. The categories may include article title, attached title, text, author, abstract, references, text-related pictures / tables, etc.
[0089] Specifically, based on the layout feature vector and semantic feature vector of each processing object on the target document image, the category information corresponding to each processing object on the target document image is determined. That is, each processing object in the target document image is classified into categories, for example, the processing objects are classified into article title categories, text categories, etc. The category information corresponding to the processing object can be represented by setting a field. For example, the article title category can be represented by the "title" field.
[0090] Then, based on the sequential relationship between the processing objects on the target document image and the category information corresponding to each processing object, a document tree corresponding to the target document image is constructed. That is to say, each node of the document tree contains the location information, content information and category information of the corresponding processing object.
[0091] Since processing objects of different categories are in different hierarchical structures on the target document image, the nodes corresponding to processing objects with different category information are in different levels of the document tree. For example, the node of the text category is below the node of the article title category.
[0092] In practical applications, since the tree structure requires more space during the storage process and the linear structure occupies less space, in order to reduce the storage space occupied, after constructing the document tree corresponding to the target document image, the embodiment of the present disclosure can serialize the document tree into a linear structure, that is, store the sequential relationship between the processing objects on the target document image in the linear structure.
[0093] Specifically, after generating a document tree corresponding to the target document image based on the layout feature vector and semantic feature vector of each processing object on the target document image, the document tree is serialized through a pre-order traversal of the tree to obtain a result sequence of the document tree, which is a linear structure that describes the sequential relationship between the processing objects on the target document image.
[0094] The pre-order traversal of a tree is a way of traversing the tree structure from top to bottom and from left to right.
[0095] In an optional implementation, in order to express the sequential relationship between the processing objects on the target document image in a linear structure, a separator may also be set in the result sequence in the embodiment of the present disclosure.
[0096] Specifically, the result sequence may include at least one of a level separator, a parallel relationship separator, and a subordinate relationship separator. The level separator can be represented by [PAIR_SEP], and the level separator is used to identify that the corresponding different nodes are at different levels in the tree structure, that is, when the document tree is serialized according to the pre-order traversal of the tree, if the currently traversed node and the previous node are at different levels in the document tree, a level separator can be set between the node indexes corresponding to the two nodes in the result sequence. In other words, a level separator is set between processing objects belonging to different categories in the target document image. For example, when a node belonging to the title category and a node belonging to the author category are serialized into two node indexes in the result sequence, a level separator can be set between the two node indexes.
[0097] The node index corresponds to the node in the document tree and to the processing object in the target document image. The node index can be set to [IDX_k], where k = 1, 2, 3, ..., n, n represents the total number of processing objects, and [IDX_1] represents the first text line. The document content of the corresponding processing object can be obtained from the target document image through the node index.
[0098] The parallel relationship separator can be represented by [SEP], and the parallel relationship separator is used to identify that the corresponding nodes are at the same level in the tree structure, that is, the nodes corresponding to the node indexes on both sides of the parallel relationship separator are at the same level in the tree structure, that is, the processing objects corresponding to the node indexes on both sides of the parallel relationship separator are document contents belonging to the same category in the target document image. That is, when the document tree is serialized according to the pre-order traversal of the tree, if the currently traversed node and the previous node are at the same level in the document tree, a parallel relationship separator can be set between the node indexes corresponding to the two nodes in the result sequence. For example, when two nodes belonging to the text category are serialized into two node indexes in the result sequence, a level separator can be set between the two node indexes, and the two processing objects have a front-to-back order relationship.
[0099] The subordination separator can be represented by [V_SEP], and is used to identify that the corresponding node is at a level with a subordination in the tree structure, that is, when the document tree is serialized according to the pre-order traversal of the tree, if the level to which the currently traversed node belongs has a subordination with the level to which the previous node belongs, a subordination separator can be set between the node indexes corresponding to the two nodes in the result sequence. In other words, the processing objects corresponding to the current node index and the previous node index are document contents belonging to two different categories in the target document image, and the two different categories have a subordination. For example, a subordination separator can be set between the node index belonging to the body category and the node index belonging to the chapter category.
[0100] In practical applications, in order to quickly find the processing object of the corresponding category in the linear structure, in the embodiment of the present disclosure, a category index separator can also be set in the result sequence. The category index separator can be represented by [K_SEP], and the category index separator is used to identify the category information and index information of the same node. That is, when the document tree is serialized, the index separator is set based on the category information of the currently traversed node, and the category information and index information of the current node are on both sides of the index separator.
[0101] Specifically, the category information of the node can be represented by a category separator, which can be represented by [CLA_XX], where "XX" is the field corresponding to the category, for example, the title category separator is [CLA_title], the author category separator is [CLA_author], and so on.
[0102] The index information of the node is used to indicate the position of the current node on the document tree. The index information can be represented by a node index, through which the document content of the corresponding processing object can be obtained from the target document image.
[0103] For example, as shown below, a result sequence is provided in the embodiment of the present disclosure:
[0104] '[CLA_title]','[K_SEP]','[IDX_1]','[PAIR_SEP]','[CLA_author]','[K_SEP]','[IDX_2]','[CLA_sec]','[K_SE P]','[IDX_3]','[V_SEP]','[CLA_fpara]','[K_SEP]','[IDX_4]','[SEP]','[CLA_fpara]','[K_SEP]','[IDX_5]';
[0105] It can be seen from the above content that the level separator [PAIR_SEP] in the result sequence is used to identify that the node corresponding to the node index [IDX_1] and the node corresponding to the node index [IDX_2] are nodes belonging to different levels in the tree structure, that is, the processing objects corresponding to the two nodes belong to different categories in the target document image; the parallel relationship separator [SEP] is used to identify that the node corresponding to the node index [IDX_5] and the node corresponding to the node index [IDX_4] are at the same level in the tree structure, that is, they correspond to the same category on the target document image; the subordinate relationship separator [V_SEP] is used to identify that the level to which the current node index [IDX_4] belongs has a subordinate relationship with the level to which the previous node index [IDX_3] belongs; that is, [CLA_fpara] and [CLA_sec] have a subordinate relationship; the category index separator [K_SEP] is used to identify the category information and index information of the current node, that is, the category index separator is flanked by the category separator and the node index.
[0106] In practical applications, after obtaining a result sequence based on serialization of the document tree, the embodiment of the present disclosure can deserialize the result sequence based on the result sequence to obtain a tree structure document corresponding to the target document image to display the hierarchical relationship between the document contents in the target document image.
[0107] Specifically, based on the layout feature vectors and semantic feature vectors of each processing object on the target document image, a document tree corresponding to the target document image is generated, and the document tree is serialized to obtain a result sequence of the document tree, and then the result sequence is deserialized and reconstructed through a pre-order traversal of the tree to generate a tree structure document corresponding to the target document image. That is, the document content in the target document image is displayed in the form of a tree structure, wherein the tree structure document is filled with the document content of the processing object corresponding to the node index in the target document image according to the category information and index information contained in the result sequence as well as the hierarchical separator, parallel relationship separator and subordinate relationship separator. For example, the tree structure document can be a JSON document, an XML document, etc.
[0108] For example, the following is a diagram of a tree structure document:
[0109] "Title": "XXXX"
[0110] "Author": "XXXX"
[0111] "Chapter": "XXXX"
[0112] "Main text": "XXXX"
[0113] "Paragraph 1": "XXXX"
[0114] "Paragraph 2": "XXXX";
[0115] In this tree-structured document, the title has the highest level, paragraph 1 and paragraph 2 are at the same level, there is a subordinate relationship between the text and the paragraphs, and there is a subordinate relationship between the chapter and the text.
[0116] The document image processing method provided by the embodiment of the present disclosure specifically obtains a target document image, and extracts position information and content information of a processing object on the target document image, then obtains a layout feature vector of the processing object based on the position information of the processing object, and obtains a semantic feature vector of the processing object based on the content information of the processing object, and then generates a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image, wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0117] It can be seen that the embodiment of the present disclosure generates a document tree corresponding to the target document image through the layout feature vectors and semantic feature vectors of each processing object in the target document image, and can display the sequential relationship between the various processing objects in the target document image through the tree structure, thereby facilitating the application of the document content in the target document image to various scenarios.
[0118] In order to facilitate understanding of the embodiment of the present disclosure, the embodiment of the present disclosure also provides a flowchart of another document image processing method, referring to Figure 4 .
[0119] S401: Acquire a target document image, and extract position information and content information of a processing object on the target document image.
[0120] For the content introduction of S401, you can refer to the previous content for understanding, and will not repeat it here.
[0121] S402: Acquire a layout feature vector of the processing object based on the position information of the processing object, and acquire a semantic feature vector of the processing object based on the content information of the processing object.
[0122] In an optional implementation, the location information and content information of the processing object of the target document image can be input into the encoder of the target reading order model, and after feature extraction by the encoder of the target reading order model, the layout feature vector and semantic feature vector of the processing object can be obtained.
[0123] S403: Perform image segmentation processing on the target document image to obtain multiple image blocks.
[0124] Before performing image segmentation on the target document image, the size of the target document image can be adjusted, and then the image segmentation process can be performed on the target document image after the size adjustment. For example, the size can be adjusted to 224*224, and the target document image can be segmented according to a 32x32 window to obtain 49 image blocks.
[0125] S404: Based on the multiple image blocks, respectively extract visual feature vectors of the multiple image blocks.
[0126] The visual feature vector is used to describe the visual features of the target document image, that is, the features obtained by extracting the features of the target document image from the visual dimension.
[0127] Optionally, each image block may be processed by an encoder to obtain a corresponding block visual feature vector and a block position feature vector, and then the block visual feature vector and the block position feature vector of each image block are added to obtain the visual feature vector of each image block.
[0128] S405: Generate a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image and the visual feature vectors corresponding to the multiple image blocks.
[0129] After the visual feature vectors of the multiple image blocks are obtained, a document tree corresponding to the target document image is generated based on the layout feature vectors of each processing object on the target document image, the semantic feature vectors and the visual feature vectors corresponding to the multiple image blocks.
[0130] In an optional embodiment, the layout feature vectors and semantic feature vectors of each processing object on the target document image and the visual feature vectors corresponding to multiple image blocks are input into the decoder of the target reading order model, and the sequential relationship between the processing objects on the target document image is obtained through autoregressive decoding. Then, the document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
[0131] The target reading order model is used to predict the order relationship between the processing objects on the target document image. The decoder performs autoregressive decoding processing on the input layout feature vector, semantic feature vector and visual feature vector according to the reading order to obtain the order relationship between the processing objects on the target document image. The reading order can refer to the order followed when reading text and images, for example, reading a document in the order from top to bottom and from left to right.
[0132] In order to facilitate understanding of the above-mentioned embodiments, Figure 5 A schematic diagram of processing a target document image provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown in the figure, the specific document image processing process in the target reading order model for the target document image is shown. The target reading order model in the figure includes an encoder and a decoder.
[0133] The processing process of the encoder includes: after the target document image is recognized to obtain the location information and content information of the processing object, the layout feature vector and semantic feature vector of the processing object can be determined through the encoding layer. In the figure, seg_i represents the layout feature vector of the processing object, Ti represents the semantic feature vector of the processing object, i=1,2,3,…,n, n represents the total number of processing objects.
[0134] The processing process of the encoder also includes: segmenting the target document image to obtain multiple image blocks, using the image block encoding module to extract the corresponding block visual feature vector and block position feature vector for each image block and adding them to obtain the visual feature vectors of multiple images, where vj represents the block visual feature vector of the image block, patch_j represents the block position feature vector of the image block, j=1,2,3,…,m, and m represents the total number of image blocks;
[0135] Based on the content of the above embodiment, the layout feature vectors, semantic feature vectors of each processing object on the target document image, and visual feature vectors of multiple image blocks are input into the decoder.
[0136] The decoder performs autoregressive decoding processing according to the reading order to obtain the sequential relationship between the processing objects on the target document image, and constructs a document tree corresponding to the target document image based on the sequential relationship between the processing objects.
[0137] The encoder may also determine the category information corresponding to each processing object on the target document image according to the layout feature vector and the semantic feature vector of each processing object on the target document image.
[0138] Furthermore, the encoder may also serialize the document tree corresponding to the target document image into a result sequence in a pre-order traversal manner. The embodiment of the present disclosure may set the above corresponding processing tasks in the encoder, and the above processing tasks may be processed in parallel to improve processing efficiency.
[0139] In order to implement the above embodiments, the present disclosure also proposes a document image processing device. Figure 6 The structure diagram of a document image processing device provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 6 As shown, the device comprises:
[0140] The first extraction module 601 is used to obtain a target document image and extract the position information and content information of the processing object on the target document image;
[0141] An acquisition module 602 is used to acquire a layout feature vector of the processing object based on the position information of the processing object, and to acquire a semantic feature vector of the processing object based on the content information of the processing object;
[0142] The generation module 603 is used to generate a document tree corresponding to the target document image based on the layout feature vector and semantic feature vector of each processing object on the target document image; wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0143] In an optional implementation manner, the generating module includes:
[0144] A determination submodule, configured to determine a sequential relationship between the processing objects on the target document image based on a layout feature vector and a semantic feature vector of each processing object on the target document image;
[0145] The first construction submodule is used to use the processing objects as nodes of a tree structure, and to construct a document tree corresponding to the target document image based on the sequence relationship between the processing objects.
[0146] In an optional implementation, the device further includes:
[0147] A determination module, configured to determine the category information corresponding to each processing object on the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image; wherein the category information is used to characterize the category to which the corresponding processing object belongs in the document content of the target document image;
[0148] Accordingly, the first construction submodule is specifically used for:
[0149] Based on the sequential relationship between the processing objects and the category information corresponding to the processing objects, a document tree corresponding to the target document image is constructed; wherein the nodes corresponding to the processing objects with different category information are at different levels of the document tree.
[0150] In an optional implementation, the device further includes:
[0151] The processing module is used to serialize the document tree through pre-order traversal of the tree to obtain a result sequence of the document tree; wherein the result sequence is used to describe the sequential relationship between the processing objects on the target document image based on a linear structure.
[0152] In an optional embodiment, the result sequence includes at least one of a hierarchy separator, a parallel relationship separator and a subordinate relationship separator, wherein the hierarchy separator is used to identify that the corresponding different nodes are at different hierarchies in the tree structure, the parallel relationship separator is used to identify that the corresponding nodes are at the same hierarchical level in the tree structure, and the subordinate relationship separator is used to identify that the corresponding nodes are at a hierarchical level with a subordinate relationship in the tree structure.
[0153] In an optional implementation, the result sequence includes a category index separator, and the category index separator is used to identify the category information and index information of the same node.
[0154] In an optional implementation, the device further includes:
[0155] The reconstruction module is used to perform a reverse sequence reconstruction on the result sequence of the document tree through a pre-order traversal of the tree to generate a tree structure document corresponding to the target document image.
[0156] In an optional implementation, the device further includes:
[0157] A segmentation module, used for performing image segmentation processing on the target document image to obtain multiple image blocks;
[0158] A second extraction module, used to extract visual feature vectors of the multiple image blocks respectively;
[0159] Accordingly, the generating module is specifically used for:
[0160] Based on the layout feature vectors and semantic feature vectors of each processing object on the target document image and the visual feature vectors respectively corresponding to the plurality of image blocks, a document tree corresponding to the target document image is generated.
[0161] In an optional implementation manner, the generating module includes:
[0162] An input submodule, for inputting the layout feature vector and the semantic feature vector of each processing object on the target document image and the visual feature vectors respectively corresponding to the plurality of image blocks into a decoder of the target reading order model, and obtaining the order relationship between the processing objects on the target document image by autoregressive decoding;
[0163] The second construction submodule is used to construct a document tree corresponding to the target document image based on the sequential relationship between the processing objects.
[0164] In an optional implementation, the processing object on the target document image includes a text line object and / or a non-text line object on the target document image, and the first extraction module includes:
[0165] A recognition submodule, used to recognize the target document image using an optical character recognition method to obtain position information and content information of each text line object on the target document image;
[0166] The detection submodule is used to detect the target document image using a layout analysis model to obtain position information and content information of each non-text object on the target document image.
[0167] In the document image processing device provided by the embodiment of the present disclosure, first, a target document image is acquired, and the position information and content information of the processing object on the target document image are extracted. Then, a layout feature vector of the processing object is acquired based on the position information of the processing object, and a semantic feature vector of the processing object is acquired based on the content information of the processing object. Then, based on the layout feature vector and the semantic feature vector of each processing object on the target document image, a document tree corresponding to the target document image is generated, wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
[0168] It can be seen that the embodiment of the present disclosure generates a document tree corresponding to the target document image through the layout feature vectors and semantic feature vectors of each processing object in the target document image, and can display the sequential relationship between the various processing objects in the target document image through the tree structure, thereby facilitating the application of the document content in the target document image to various scenarios.
[0169] The document image processing device provided in the embodiments of the present disclosure can execute the document image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
[0170] In order to implement the above embodiments, the present disclosure further proposes a computer program product, including a computer program / instruction, which implements the document image processing method in the above embodiments when executed by a processor.
[0171] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the document image processing method described in the embodiments of the present disclosure.
[0172] In addition, the present disclosure also provides a document image processing device, see Figure 7 As shown, it may include:
[0173] Processor 701, memory 702, input device 703 and output device 704. The number of processors 701 in the document image processing device can be one or more. Figure 7 In some embodiments of the present disclosure, the processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means, wherein: Figure 7 The example of connecting through bus is taken in the following.
[0174] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing of the document image processing device by running the software programs and modules stored in the memory 702. The memory 702 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 702 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 703 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the document image processing device.
[0175] Specifically in this embodiment, the processor 701 will load the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 701 will run the applications stored in the memory 702, thereby realizing the various functions of the above-mentioned document image processing device.
[0176] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0177] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A document image processing method, characterized in that: include: Acquire a target document image, and extract position information and content information of a processing object on the target document image; Acquire a layout feature vector of the processing object based on the position information of the processing object, and acquire a semantic feature vector of the processing object based on the content information of the processing object; Based on the layout feature vectors and semantic feature vectors of each processing object on the target document image, a document tree corresponding to the target document image is generated; wherein the document tree represents the sequential relationship between each processing object on the target document image in a tree structure.
2. The method according to claim 1, characterized in that: The step of generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image comprises: Determining a sequential relationship between the processing objects on the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image; The processing objects are respectively used as nodes of a tree structure, and a document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
3. The method according to claim 2, characterized in that Before constructing the document tree corresponding to the target document image based on the sequential relationship between the processing objects, the method further includes: Determine, based on the layout feature vector and the semantic feature vector of each processing object on the target document image, the category information corresponding to each processing object on the target document image; wherein the category information is used to characterize the category to which the corresponding processing object belongs in the document content of the target document image; Accordingly, constructing a document tree corresponding to the target document image based on the sequential relationship between the processing objects includes: Based on the sequential relationship between the processing objects and the category information corresponding to the processing objects, a document tree corresponding to the target document image is constructed; wherein the nodes corresponding to the processing objects with different category information are at different levels of the document tree.
4. The method according to any one of claims 1 to 3, characterized in that After generating the document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image, the method further includes: The document tree is serialized by pre-order traversal of the tree to obtain a result sequence of the document tree; wherein the result sequence is used to describe the sequential relationship between the processing objects on the target document image based on a linear structure.
5. The method according to claim 4, characterized in that The result sequence includes at least one of a level separator, a parallel relationship separator and a subordinate relationship separator. The level separator is used to identify that the corresponding different nodes are at different levels in the tree structure, the parallel relationship separator is used to identify that the corresponding nodes are at the same level in the tree structure, and the subordinate relationship separator is used to identify that the corresponding nodes are at a level with a subordinate relationship in the tree structure.
6. The method according to claim 5, characterized in that The result sequence includes a category index separator, and the category index separator is used to identify the category information and index information of the same node.
7. The method according to claim 4, characterized in that After the document tree is serialized to obtain the result sequence of the document tree, the method further includes: The result sequence of the document tree is de-sequenced and reconstructed through pre-order traversal of the tree to generate a tree structure document corresponding to the target document image.
8. The method according to claim 1, characterized in that: Before generating a document tree corresponding to the target document image based on the layout feature vectors and semantic feature vectors of each processing object on the target document image, the method further includes: Performing image segmentation processing on the target document image to obtain multiple image blocks; Extracting visual feature vectors of the multiple image blocks respectively; Accordingly, generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image includes: Based on the layout feature vectors and semantic feature vectors of each processing object on the target document image and the visual feature vectors respectively corresponding to the plurality of image blocks, a document tree corresponding to the target document image is generated.
9. The method according to claim 8, characterized in that The step of generating a document tree corresponding to the target document image based on the layout feature vector and the semantic feature vector of each processing object on the target document image and the visual feature vectors respectively corresponding to the plurality of image blocks comprises: Inputting the layout feature vector and semantic feature vector of each processing object on the target document image and the visual feature vectors corresponding to the plurality of image blocks into the decoder of the target reading order model, and obtaining the order relationship between the processing objects on the target document image by autoregressive decoding; A document tree corresponding to the target document image is constructed based on the sequential relationship between the processing objects.
10. The method according to claim 1, characterized in that The processing object on the target document image includes a text line object and / or a non-text line object on the target document image, and extracting position information and content information of the processing object on the target document image includes: Recognize the target document image using an optical character recognition method to obtain position information and content information of each text line object on the target document image; and / or, The target document image is detected using a layout analysis model to obtain position information and content information of each non-text object on the target document image.
11. A document image processing device, characterized in that: The device comprises: A first extraction module, used to obtain a target document image and extract position information and content information of a processing object on the target document image; An acquisition module, used to acquire a layout feature vector of the processing object based on the position information of the processing object, and to acquire a semantic feature vector of the processing object based on the content information of the processing object; A generation module is used to generate a document tree corresponding to the target document image based on the layout feature vector and semantic feature vector of each processing object on the target document image; wherein the document tree represents the sequential relationship between the processing objects on the target document image in a tree structure.
12. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the document image processing method described in any one of claims 1-10.
13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the document image processing method described in any one of claims 1 to 10.
14. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.