Image document understanding method and device, equipment and storage medium
Through transformer-based vision encoder and language decoder, the image documents are understood, and the problems of strong dependence, insufficient information utilization, high computing cost and poor scalability in the prior art are solved, thereby achieving a more efficient and stable image document understanding effect.
Patent Information
- Application Number
- CN202411420153.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-02-28
AI Technical Summary
The existing image document understanding methods have problems such as strong dependence, insufficient information utilization, high computing cost and poor scalability, and it is difficult to deal with complex and diverse data and tasks.
Using transformer-based vision encoder and language decoder, we understand image documents and generate analytical sequences, extract visual features through block embedding modules and multiple stage models, and make language predictions through basic decoders.
It realizes the full use of cross-modal information, reduces calculation costs, improves understanding ability and scalability, avoids error accumulation, and makes the model performance more stable and reliable.
Smart Images
Figure CN119540975A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for understanding an image document. Background Art
[0002] In real life, there are many scenarios where image documents need to be understood. For example, in the field of enterprise management, a large number of contracts, reports, invoices, etc. may be saved in the form of images or PDF files.
[0003] Traditionally, employees need to read these documents and manually extract key information for archiving. With the development of technology, visual document understanding has become the mainstream method for image document extraction. At present, visual document understanding is mainly divided into two stages: the first stage, optical character recognition based on deep learning, obtains text information and location information in image documents; the second stage, relationship extraction methods based on deep learning (SDMGR or NER, etc.) extract the relationship information between entities and attributes and understand the text information.
[0004] However, existing methods are highly dependent. Errors in the visual recognition stage will directly affect the accuracy of subsequent semantic understanding, and errors in any subtask will reduce the overall understanding effect; existing methods do not make sufficient use of information, and the two-stage processing cannot fully utilize the relevant information across modalities; existing methods have high computational costs, and both high-quality optical character recognition results and high-quality document understanding results require expensive training calculations; existing methods have poor scalability. When task requirements or data formats change, each stage needs to be redesigned separately, and lacks universality.
[0005] Therefore, the existing methods for understanding image documents have relatively independent two-stage processing, insufficient information transmission, limited understanding ability and scalability, and are difficult to cope with complex and diverse data and tasks. Summary of the invention
[0006] To solve the above problems, the embodiments of the present application provide a method, apparatus, device and storage medium for understanding image documents, which construct a visual encoder and language decoder based on transformer, understand image documents and generate parsing sequences, thereby improving the ability to understand image documents.
[0007] The present application embodiment adopts the following technical solutions:
[0008] In a first aspect, a method for understanding an image document is provided, the method comprising:
[0009] The image is continuously downsampled based on the visual encoder to obtain output features;
[0010] The output features are predicted based on the language decoder to obtain the parsed sequence of the image document;
[0011] The visual encoder is built based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer at the last layer.
[0012] Language decoders are built upon four basic decoders.
[0013] In a second aspect, a device for understanding an image document is provided, the device comprising:
[0014] An extraction unit, used for continuously downsampling the image based on the visual encoder to obtain output features;
[0015] A prediction unit, used to predict the output features based on the language decoder to obtain a parsed sequence of the image document;
[0016] Among them, the visual encoder is built based on a block embedding module and a four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer located at the last layer; the language decoder is built based on four basic decoders.
[0017] In a third aspect, a method for understanding an image document is provided, the method comprising:
[0018] A visual encoder is constructed based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer at the last layer.
[0019] Building a language decoder based on four basic decoders;
[0020] The image is continuously downsampled based on the visual encoder to obtain output features, and the output features are predicted based on the language decoder to obtain the parsed sequence of the image document.
[0021] In a fourth aspect, a device for understanding an image document is provided, the device comprising:
[0022] A visual encoder construction unit, used to construct a visual encoder based on a block embedding module and four stage models; wherein the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer at the last layer;
[0023] A language decoder building unit, used to build a language decoder based on four basic decoders;
[0024] The understanding unit is used to continuously downsample the image based on the visual encoder to obtain output features, and predict the output features based on the language decoder to obtain a parsed sequence of the image document.
[0025] In a fifth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for understanding image documents when executing the computer program.
[0026] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above-mentioned method for understanding image documents when instructed by a processor.
[0027] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0028] The image document understanding method provided by the present application is based on the visual encoder to continuously downsample the image to obtain the output features; the output features are predicted based on the language decoder to obtain the parsing sequence of the image document; the visual encoder is constructed based on the block embedding module and the four-stage model; the first two stage models include two and four local transformer modules respectively, and the latter two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer located at the last layer; the language decoder is constructed based on four basic decoders. The image document understanding method provided by the present application is composed of a visual encoder and a language decoder. The visual encoder and the language decoder are both implemented based on transformers, and the final output generates the parsing sequence of the image document. The method provided by the present application can make full use of the relevant information between cross-modalities and jointly learn the visual, language and structural information in one model. The method provided by the present application can be optimized end-to-end, and the knowledge between different tasks can be transferred to each other, so that the model effect is better and the model performance is more stable and reliable. The method provided by the present application has a simple model structure, and the computational cost is greatly reduced compared with the two-stage image document understanding method. The method provided by the present application does not have a weak coupling problem, and the errors will not accumulate, so that the understanding ability of the image document is stronger. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0030] Figure 1 A schematic diagram showing a flow chart of a method for understanding an image document according to an embodiment of the present application;
[0031] Figure 2 A schematic diagram showing the architecture of a visual encoder and a language decoder according to an embodiment of the present application is shown;
[0032] Figure 3 A diagram showing the effect of implementing a method for understanding an image document according to an embodiment of the present application;
[0033] Figure 4 A schematic diagram showing the architecture of a block embedding module according to an embodiment of the present application is shown;
[0034] Figure 5 A schematic diagram of the architecture of a local transformer model / global transformer model according to an embodiment of the present application is shown;
[0035] Figure 6A schematic diagram showing an implementation method of a feature fusion layer according to an embodiment of the present application is shown;
[0036] Figure 7 A schematic diagram showing the architecture of a basic decoder according to an embodiment of the present application is shown;
[0037] Figure 8 A schematic diagram showing the structure of an apparatus for understanding an image document according to an embodiment of the present application;
[0038] Fig. 9 A schematic flow chart showing a method for understanding an image document according to another embodiment of the present application;
[0039] Fig.10 A schematic diagram showing the structure of an apparatus for understanding an image document according to another embodiment of the present application;
[0040] Fig.11 A schematic structural diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0042] Figure 1 The flowchart of the method for understanding an image document proposed in one embodiment of the present application is shown. Figure 1 As shown, the method includes steps S110 to S120:
[0043] Step S110, continuously downsampling the image based on the visual encoder to obtain output features.
[0044] Step S120: predicting the output features based on the language decoder to obtain a parsed sequence of the image document.
[0045] Figure 2 A schematic diagram showing the architecture of a visual encoder and a language decoder of an image document understanding method proposed in one embodiment of the present application is shown, referring to Figure 2 The architecture of the image document is described in detail.
[0046] The visual encoder can be built based on a block embedding module and a four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer at the last layer.
[0047] Reference Figure 2 As shown, the visual encoder is built based on a block embedding module, a first-stage model, a second-stage model, a third-stage model, and a fourth-stage model.
[0048] The block embedding module is used to divide the image into blocks and fuse the position information of the blocks in the image to obtain the downsampled feature map.
[0049] The first stage model can be constructed by two local transformer modules and a feature fusion layer, and the second stage model can be constructed by four local transformer modules and a feature fusion layer. The local transformer module is used to extract local detail information in the image, such as the strokes and colors of the text in the image document.
[0050] The third stage model can be constructed by four global transformer modules and a feature fusion layer, and the fourth stage model can be constructed by two global transformer modules and a feature fusion layer. The global transformer module is used to extract global features in the image.
[0051] The feature fusion layer is located at the last layer of the model in each stage and is used for downsampling to reduce the resolution and reduce the amount of computation.
[0052] Language decoders are built upon four basic decoders.
[0053] Reference Figure 2 As shown in Figure 1, the language decoder is constructed by four basic decoders to correspond to the visual encoder. The basic decoder can be a standard decoder based on transformer or a decoder based on a commonly used deconvolution module.
[0054] The image is used as the input of the block embedding module. The output of the block embedding module is used as the input of the first stage model. The output features of the previous stage model are used as the input of the next stage model. The output features of the first stage model to the fourth stage model are used as the input of the basic decoder four to the basic decoder one respectively; the output of the previous basic decoder is used as the input of the next basic decoder. After the basic decoder four, the parsed sequence of the image document is output.
[0055] Taking the consumption receipt as an example, the consumption receipt image is used as input, and the image is continuously downsampled by the visual encoder to obtain the output features; the output features are predicted by the language decoder to obtain the parsed sequence of the consumption receipt. Figure 3 The implementation effect diagram of the method for understanding image documents shown in the figure, the parsing sequence of consumption receipts may at least include: <type> receipt (class <answer>Receipt), <Restaurant Name> Xiamen Qingxiang Food Square (restaurant <answer>Xiamen Qingxiang Food Square), <account> 128 (account <answer>128) etc.
[0056] from Figure 1 It can be seen from the method shown that the image document understanding method provided by the present application is based on the visual encoder to continuously downsample the image to obtain the output features; the output features are predicted based on the language decoder to obtain the parsing sequence of the image document; the visual encoder is constructed based on the block embedding module and the four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer located at the last layer; the language decoder is constructed based on four basic decoders. The image document understanding method provided by the present application is composed of a visual encoder and a language decoder. Both the visual encoder and the language decoder are implemented based on transformers, and finally the parsing sequence of the generated image document is output. The method provided by the present application can make full use of the relevant information between cross-modalities and jointly learn the visual, language and structural information in one model. The method provided by the present application can be optimized end-to-end, and the knowledge between different tasks can be transferred to each other, so that the model effect is better and the model performance is more stable and reliable. The method provided in this application has a simple model structure and greatly reduces the computational cost compared to the two-stage image document understanding method. The method provided in this application does not have a weak coupling problem, and errors will not accumulate, making the image document understanding ability stronger.
[0057] In some optional embodiments, in the above method, the block embedding module is constructed based on a convolution layer with a convolution kernel size of 3×3, a batch normalization layer, and a position encoding module; the image is divided into blocks after two convolutions and batch normalization, and after each block is fused with position information, a feature map that is downsampled 4 times is obtained.
[0058] Figure 4 The schematic diagram of the block embedding module of the image document understanding method proposed in one embodiment of the present application is shown. Figure 4 As shown, the block embedding module includes a convolution layer (Conv) with a convolution kernel size of 3×3, a batch normalization layer (BN), and a position encoding module (Position Embedding).
[0059] In this embodiment, the input block embedding module is a 3-channel RGB image, the height of the image is H, and the width of the image is W. Therefore, the input image can be expressed as [H×W×3]. After two 3×3 convolutions and combined with batch normalization, the input image is divided into patches, and combined with the position information of each patch in the image, a feature map of downsampled 4 times [H / 4×W / 4×D] is obtained, where D represents the channel dimension.
[0060] In some optional embodiments, in the above method, each local transformer module is constructed based on an adaptive multi-head self-attention mechanism module and a multi-layer perceptron, and the adaptive multi-head self-attention mechanism module and the multi-layer perceptron are respectively preceded by a layer normalization layer; for each local transformer module, the starting input passes through the previous normalization layer and the adaptive multi-head self-attention mechanism module, and then is residually connected with the starting input to obtain an intermediate result, and the intermediate result passes through the next normalization layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain the output; the output of the previous local transformer module is used as the input of the next local transformer module; the output of the second local transformer module of the first-stage model is used as the input of the feature fusion layer of the first-stage model; the output of the fourth local transformer model of the second-stage model is used as the input of the feature fusion layer of the second-stage model.
[0061] In some optional embodiments, in the above method, each global transformer model is constructed based on a standard multi-head self-attention mechanism module and a multi-layer perceptron, and the standard multi-head self-attention mechanism module and the multi-layer perceptron are respectively preceded by a layer normalization layer; for each global transformer module, the starting input passes through the previous normalization layer and the standard multi-head self-attention mechanism module, and then is residually connected with the starting input to obtain an intermediate result, and the intermediate result passes through the next normalization layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain the output; the output of the previous global transformer module is used as the input of the next global transformer module; the output of the fourth global transformer module of the third-stage model is used as the input of the feature fusion layer of the third-stage model; the output of the second global transformer module of the fourth-stage model is used as the input of the feature fusion layer of the fourth-stage model.
[0062] Figure 5 The schematic diagram of the architecture of the local transformer model / global transformer model of the image document understanding method proposed in one embodiment of the present application is shown. The overall architecture of the local transformer model is basically the same as that of the global transformer model, the main difference being that the local transformer model is constructed using an adaptive multi-head self-attention mechanism (Local-MSA), while the global transformer model is constructed using a standard multi-head self-attention mechanism (MSA).
[0063] Taking the global transformer model as an example, the initial input passes through the layer normalization layer (LN) and the standard multi-head self-attention mechanism, and then performs a residual connection with the initial input to obtain an intermediate result; on this basis, it passes through the layer normalization layer and the multi-layer perceptron (MLP), and then performs a residual connection with the intermediate result to output the global feature map. The activation function of the multi-layer perceptron uses GELU (Gaussian Error Linear Units).
[0064] The adaptive multi-head self-attention mechanism is combined with a moving window to "mask" the part outside the window and only focus on the neighborhood of the element. Therefore, the local transformer model is that the initial input is processed by the layer normalization layer (LN) and then divided into multiple non-overlapping local windows of a set size; local multi-head self-attention calculation is performed on each local window, and the calculation result is residually connected with the initial input, and then the residual connection is made with the calculation result of the layer normalization layer and the multi-layer perceptron to output the local feature map.
[0065] The window size can be 10×10, that is, the input can be evenly divided into multiple local windows in a non-overlapping manner using a 10×10 window. Local multi-head self-attention calculation is performed in units of local windows. The local multi-head self-attention mechanism can split the number of channels of the query matrix Q, key-value matrix K, and value matrix V into several groups, each group performs self-attention calculation independently, and the results of each group are horizontally spliced. When performing local multi-head self-attention calculations in units of local windows, the input can be passed through a linear layer to expand the feature dimension by 3 times and split into a query matrix Q, a key-value matrix K, and a value matrix V; the feature inner product of the query matrix Q and the key-value matrix K is calculated, and then divided by the square root of D to obtain a relationship matrix; the relative position in the local window has a total of 19 values in the range of [-10+1,10-1], a 19×19 size learnable variable is set, and the attention relationship matrix is obtained by adding the two-dimensional relative position encoding and the relationship matrix; the attention relationship matrix is normalized exponentially calculated in the last dimension to obtain a local attention relationship graph; the local attention relationship graph is threaded with the value matrix V to obtain the calculation result of the local multi-head self-attention.
[0066] In some optional embodiments, in the above method, the feature fusion layer is implemented as follows: the input is divided into multiple regions according to 2×2, and the regional blocks at the same position in each region are extracted for channel stitching; after stretching the regional blocks after channel stitching, layer normalization is performed; and after a convolution layer with a convolution kernel size of 1×1, an output with a channel expanded to 2 times is obtained.
[0067] The feature fusion layer is located at the last layer of the model in each stage. Through the feature fusion layer, the output with the channel expanded to 2 times is obtained. Figure 6 The following is a schematic diagram showing an implementation method of the feature fusion layer of the image document understanding method proposed in one embodiment of the present application. Figure 6 For example, the specific implementation of the image fusion layer can be: divide the input into 4 regions with a step size of 2, and each region includes region blocks with serial numbers 1, 2, 3, and 4. Extract according to the serial numbers of the region blocks, that is, extract the region blocks with the same serial number, and convert them into four tensors after extraction. Channel splicing is performed on the region blocks with the same serial number in the depth direction, and the channel-spliced region blocks are stretched in the width direction. After the layer normalization layer is processed, the number of channels is finally expanded to 2 times by using the convolution layer with a convolution kernel of 1×1.
[0068] In some optional embodiments, in the above method, each basic decoder is constructed based on a masked multi-head self-attention sublayer, a multi-head self-attention sublayer and a feedforward fully connected sublayer, and each sublayer is respectively connected to a normalization layer and a residual connection; the output of the previous basic decoder is used as the input of the next basic decoder; the outputs of the first stage model to the fourth stage model are respectively used as the input of the multi-head self-attention sublayer from the fourth basic decoder to the first basic decoder; after passing through the fourth basic decoder, the parsed sequence of the image document is output.
[0069] Figure 7 The schematic diagram of the architecture of the basic decoder of the image document understanding method proposed in one embodiment of the present application is shown. Figure 7 As shown, in this embodiment, each basic decoder includes a masked multi-head self-attention sublayer, a multi-head self-attention sublayer and a feed-forward fully connected sublayer, and each sublayer includes a normalization layer and a residual connection.
[0070] The masked multi-head self-attention sublayer is used to ensure that the prediction of a specific position can only depend on the known output of the previous position, that is, each word in the sequence is not affected by the image of future tags. The multi-head self-attention sublayer is used to establish interaction with the visual encoder, that is, the output of the corresponding stage of the visual encoder is used as the key matrix K and the value matrix V, and the output of the basic decoder masked multi-head self-attention sublayer is used as the query matrix Q. The feed-forward fully connected sublayer is used to perform standardized residual output.
[0071] Starting from the lowest level basic decoder, basic decoder one, the input first passes through the embedding layer, which is similar to the process of the encoder. After embedding, the input passes through the position encoding layer to produce the position embeddings of the corresponding sequence, which are fed into the masked multi-head self-attention sublayer of basic decoder one to accurately calculate the attention scores associated with the language decoder input.
[0072] For the multi-head self-attention sublayer of the latter basic decoder, the query matrix Q comes from the output of the previous basic decoder, and the key-value matrix K and value matrix V come from the output of the corresponding stage model of the visual encoder. This allows each position in the language decoder to pay attention to all positions in the input sequence, effectively integrating the information in the visual encoder with the information in the language decoder.
[0073] Each basic decoder includes a feed-forward fully connected sublayer. The processed output of the feed-forward fully connected sublayer is normalized by merging it with the input of the feed-forward fully connected sublayer and further normalizing the residual output.
[0074] The output of the basic decoder 4 is converted into a prediction sequence through a linear layer as a classifier, and then input into the Softmax layer to generate the probability of words, and finally outputs the parsed sequence of the image document.
[0075] The method for understanding an image document provided by the present application is based on an encoder-decoder structure, and proposes a visual encoder architecture including a block embedding module, a two-stage model including local transformer modules, and a two-stage model including global transformer modules, which is used to continuously downsample the image to obtain the output features of the image; a language decoder including four basic decoders is used to predict the output features, thereby outputting a parsed sequence after understanding the image document.
[0076] The visual encoder and language decoder proposed in this application have a simple model structure and consume less computing power. Compared with the two-stage image document understanding method, the computational cost is greatly reduced. They make full use of the relevant information between cross-modalities and jointly learn visual, language and structural information in one model. They perform end-to-end optimization, and knowledge between different tasks can be transferred to each other. There is no weak coupling problem, and errors will not accumulate. The method proposed in this application makes the model more effective, the model performance more stable and reliable, and the ability to understand image documents is stronger.
[0077] Figure 8 The schematic diagram of the structure of the image document understanding device proposed in one embodiment of the present application is shown. Figure 8 As shown, the device 800 includes:
[0078] An extraction unit 810 is used to continuously downsample the image based on a visual encoder to obtain output features;
[0079] A prediction unit 820, configured to predict the output features based on the language decoder to obtain a parsed sequence of the image document;
[0080] Among them, the visual encoder is built based on a block embedding module and a four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer located at the last layer; the language decoder is built based on four basic decoders.
[0081] It should be noted that the above-mentioned image document understanding device 800 can implement the above-mentioned image document understanding methods one by one, which will not be described in detail.
[0082] Fig. 9 A schematic diagram showing a flow chart of a method for understanding an image document proposed in another embodiment of the present application is shown. Fig. 9 As shown, the method may include the following steps S910 to S930.
[0083] Step S910, constructing a visual encoder based on the block embedding module and the four-stage model; wherein the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer at the last layer;
[0084] Step S920, constructing a language decoder based on four basic decoders;
[0085] Step S930, continuously downsampling the image based on the visual encoder to obtain output features, and predicting the output features based on the language decoder to obtain a parsed sequence of the image document.
[0086] In some optional embodiments, in the above method, a block embedding module is constructed based on a convolution layer with a convolution kernel size of 3×3, a batch normalization layer, and a position encoding module; the block embedding module is used to divide the image into blocks after two convolutions and batch normalization, and after each block is fused with position information, a feature map that is downsampled 4 times is obtained.
[0087] In some optional embodiments, in the above method, each local transformer module is constructed based on an adaptive multi-head self-attention mechanism module and a multi-layer perceptron, and the adaptive multi-head self-attention mechanism module and the multi-layer perceptron are respectively preceded by a layer normalization layer; for each local transformer module, the starting input passes through the previous normalization layer and the adaptive multi-head self-attention mechanism module, and then is residually connected with the starting input to obtain an intermediate result, and the intermediate result passes through the next normalization layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain the output; the output of the previous local transformer module is used as the input of the next local transformer module; the output of the second local transformer module of the first-stage model is used as the input of the feature fusion layer of the first-stage model; the output of the fourth local transformer module of the second-stage model is used as the input of the feature fusion layer of the second-stage model.
[0088] In some optional embodiments, in the above method, each global transformer module is constructed based on a standard multi-head self-attention mechanism module and a multi-layer perceptron, and the standard multi-head self-attention mechanism module and the multi-layer perceptron are respectively preceded by a layer normalization layer; for each global transformer module, the starting input passes through the previous normalization layer and the standard multi-head self-attention mechanism module, and then is residually connected with the starting input to obtain an intermediate result, and the intermediate result passes through the next normalization layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain the output; the output of the previous global transformer module is used as the input of the next global transformer module; the output of the fourth global transformer module of the third-stage model is used as the input of the feature fusion layer of the third-stage model; the output of the second global transformer module of the fourth-stage model is used as the input of the feature fusion layer of the fourth-stage model.
[0089] In some optional embodiments, in the above method, the feature fusion layer is constructed to implement the following method: divide the input into multiple regions according to 2×2, and take the regional blocks at the same position in each region for channel stitching; after stretching the regional blocks after channel stitching, perform layer normalization processing; and obtain the output feature with the channel expanded to 2 times through a convolution layer with a convolution kernel size of 1×1.
[0090] In some optional embodiments, in the above method, each basic decoder is constructed based on a masked multi-head self-attention sublayer, a multi-head self-attention sublayer and a feedforward fully connected sublayer, and each sublayer is respectively connected to a normalization layer and a residual connection; the output of the previous basic decoder is used as the input of the next basic decoder; the outputs of the first stage model to the fourth stage model are respectively used as the input of the multi-head self-attention sublayer from the fourth basic decoder to the first basic decoder; after passing through the fourth basic decoder, the parsed sequence of the image document is output.
[0091] Fig.10 A schematic diagram of the structure of an image document understanding device proposed in another embodiment of the present application is shown. Fig.10 As shown, the device 1000 includes:
[0092] A visual encoder construction unit 1010 is used to construct a visual encoder based on a block embedding module and four stage models; wherein the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer at the last layer;
[0093] A language decoder construction unit 1020, for constructing a language decoder based on four basic decoders;
[0094] The understanding unit 1030 is used to continuously downsample the image based on the visual encoder to obtain output features, and predict the output features based on the language decoder to obtain a parsed sequence of the image document.
[0095] It should be noted that the above-mentioned image document understanding device 1000 can implement the above-mentioned image document understanding methods one by one, which will not be described in detail.
[0096] Fig.11 A schematic diagram showing the structure of a computer device according to an embodiment of the present application is shown. Fig.11 As shown, the internal structure of the computer device may include a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, the functions or steps of the method for understanding an image document are implemented.
[0097] In one embodiment, the computer device provided by the present application includes a memory and a processor, the memory stores a database and a computer program that can be run on the processor, and the processor implements the following steps when executing the computer program:
[0098] The image is continuously downsampled based on the visual encoder to obtain output features;
[0099] The output features are predicted based on the language decoder to obtain the parsed sequence of the image document;
[0100] Among them, the visual encoder is built based on a block embedding module and a four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer located at the last layer; the language decoder is built based on four basic decoders.
[0101] or,
[0102] A visual encoder is constructed based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer at the last layer.
[0103] Building a language decoder based on four basic decoders;
[0104] The image is continuously downsampled based on the visual encoder to obtain output features, and the output features are predicted based on the language decoder to obtain the parsed sequence of the image document.
[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0106] The image is continuously downsampled based on the visual encoder to obtain output features;
[0107] The output features are predicted based on the language decoder to obtain the parsed sequence of the image document;
[0108] Among them, the visual encoder is built based on a block embedding module and a four-stage model; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer located at the last layer; the language decoder is built based on four basic decoders.
[0109] or,
[0110] A visual encoder is constructed based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively. Each stage model also includes a feature fusion layer at the last layer.
[0111] Building a language decoder based on four basic decoders;
[0112] The image is continuously downsampled based on the visual encoder to obtain output features, and the output features are predicted based on the language decoder to obtain the parsed sequence of the image document.
[0113] It should be noted that the above functions or steps that can be implemented by the computer device or computer-readable storage medium can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0115] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.< / answer> < / answer> < / answer>
Claims
1. A method for understanding an image document, characterized in that: The method comprises: The image is continuously downsampled based on the visual encoder to obtain output features; Predicting the output features based on a language decoder to obtain a parsed sequence of the image document; The visual encoder is constructed based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer at the last layer; The language decoder is built on four basic decoders.
2. The method for understanding an image document according to claim 1, characterized in that: The block embedding module is constructed based on a convolutional layer with a convolution kernel size of 3×3, a batch normalization layer, and a position encoding module; The image is divided into blocks after two convolutions and batch normalization, and each block is fused with position information to obtain a feature map that is downsampled 4 times.
3. The method for understanding an image document according to claim 1, characterized in that: Each of the local transformer modules is constructed based on an adaptive multi-head self-attention mechanism module and a multi-layer perceptron, and each of the adaptive multi-head self-attention mechanism module and the multi-layer perceptron has a layer normalization layer before each of them; For each local transformer module, the initial input is subjected to the normalization layer of the previous layer and the adaptive multi-head self-attention mechanism module, and then is residually connected with the initial input to obtain an intermediate result. The intermediate result is subjected to the normalization layer of the next layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain an output; The output of the previous local transformer module is used as the input of the next local transformer module; The output of the second local transformer module of the first stage model is used as the input of the feature fusion layer of the first stage model; The output of the fourth local transformer module of the second-stage model is used as the input of the feature fusion layer of the second-stage model.
4. The method for understanding an image document according to claim 1, characterized in that: Each of the global transformer modules is constructed based on a standard multi-head self-attention mechanism module and a multi-layer perceptron, and each of the standard multi-head self-attention mechanism module and the multi-layer perceptron has a layer normalization layer before each of them; For each of the global transformer modules, the starting input is subjected to the normalization layer of the previous layer and the standard multi-head self-attention mechanism module, and then is residually connected with the starting input to obtain an intermediate result. The intermediate result is subjected to the normalization layer of the next layer and the multi-layer perceptron, and then is residually connected with the intermediate result to obtain an output. The output of the previous global transformer module is used as the input of the next global transformer module; The output of the fourth global transformer module of the third stage model is used as the input of the feature fusion layer of the third stage model; The output of the second global transformer module of the fourth stage model is used as the input of the feature fusion layer of the fourth stage model.
5. The method for understanding an image document according to claim 1, characterized in that: The feature fusion layer is implemented as follows: The input is divided into multiple regions according to 2×2, and the region blocks at the same position in each region are extracted for channel splicing; After stretching the region block after channel splicing, perform layer normalization processing; After a convolution layer with a convolution kernel size of 1×1, the output feature with the channel expanded by 2 times is obtained.
6. A device for understanding an image document, characterized in that: The device comprises: An extraction unit, used for continuously downsampling the image based on the visual encoder to obtain output features; A prediction unit, configured to predict the output features based on a language decoder to obtain a parsed sequence of the image document; Among them, the visual encoder is constructed based on a block embedding module and four stage models; the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer located at the last layer; the language decoder is constructed based on four basic decoders.
7. A method for understanding an image document, characterized in that: A visual encoder is constructed based on a block embedding module and four stage models; wherein the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer located at the last layer; Building a language decoder based on four basic decoders; The image is continuously downsampled based on the visual encoder to obtain output features, and the output features are predicted based on the language decoder to obtain a parsed sequence of the image document.
8. An apparatus for understanding an image document, characterized in that: The device comprises: A visual encoder construction unit, used to construct a visual encoder based on a block embedding module and four stage models; wherein the first two stage models include two and four local transformer modules respectively, and the last two stage models include two and four global transformer modules respectively, and each stage model also includes a feature fusion layer located at the last layer; A language decoder building unit, used to build a language decoder based on four basic decoders; The understanding unit is used to continuously downsample the image based on the visual encoder to obtain output features, and predict the output features based on the language decoder to obtain a parsed sequence of the image document.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for understanding an image document as described in any one of claims 1 to 5 or 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is instructed by a processor, it implements the steps of the method for understanding an image document as described in any one of claims 1 to 5 or 7.