Document image analysis method and device, electronic equipment and storage medium
By using a joint decoder to perform joint spatial and content decoding of document images, the accuracy and efficiency issues of existing document image parsing methods under multi-task conditions are resolved, and efficient and accurate unified processing of text instances is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-03-03
AI Technical Summary
Existing document image parsing methods have low accuracy and efficiency under multi-task requirements, and dedicated models lead to pattern isolation and increased complexity.
A joint decoder is used to perform joint decoding of spatial and content information. Through multi-head self-attention and cross-attention operations, a joint sequence of target text instances is generated. Combined with Bézier curve fitting, the spatial and content information of text instances is processed in a unified manner.
It improves the efficiency and accuracy of document image parsing, simplifies model complexity, reduces resource requirements, and is suitable for unified processing of multiple tasks.
Smart Images
Figure CN121600532A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a document image parsing method, apparatus, electronic device, and storage medium. Background Technology
[0002] Document image analysis refers to the technology of automatically identifying and extracting text, structural, and layout information from scanned documents, photographs, or digital images. It typically includes sub-tasks such as text detection, character recognition, layout analysis, and image segmentation. Document image analysis aims to extract structured information from document images, which includes identifying and parsing text, tables, graphics, and other visual elements in text-rich images.
[0003] With the rapid growth of document image data, research on document image parsing has surged in recent years. Current document image parsing methods typically employ specialized models, such as table recognition models for table recognition and text detection models for text detection. However, while these specialized models exhibit high performance in their respective tasks, their workflows often become more complex under multi-task requirements. Furthermore, the discrete nature of these specialized models leads to pattern isolation, limiting in-depth understanding. Therefore, the accuracy and efficiency of document image parsing using specialized models are relatively low. Summary of the Invention
[0004] The purpose of this application is to provide a document image parsing method, apparatus, electronic device, and storage medium that can solve the problems of low accuracy and efficiency in document image parsing.
[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a document image parsing method, the method comprising: The target document image is encoded using an encoder to obtain the target global feature map information of the target document image; Using a joint decoder, spatial and content joint decoding is performed based on the target global feature map information to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
[0006] Optionally, the step of performing joint spatial and content decoding based on the target global feature map information to obtain the target joint sequence for each target text instance in the target document image includes: Based on the target global feature map information, multi-head self-attention and cross-attention operations are performed to obtain the target spatial information of each target text instance in the target document image; Based on the target spatial information and the target global feature map information, a text generation operation is performed to obtain the target content information of each target text instance in the target document image; The target spatial information and target content information of the target text instance are integrated to obtain the target joint sequence of the target text instance.
[0007] Optionally, the step of performing multi-head self-attention and cross-attention operations based on the target global feature map information to obtain the target spatial information of each target text instance in the target document image includes: Multi-head self-attention and cross-attention operations are performed based on the target global feature map information to obtain candidate spatial information for each target text instance in the target document image; Based on the candidate spatial information, a Bézier curve fitting operation is performed to obtain the target spatial information of each target text instance in the target document image.
[0008] Optionally, the joint decoder is trained in the following manner: Acquire sample data, which includes sample document images and a real joint sequence of each real text instance in the sample document images, wherein the real joint sequence includes the real spatial information and real content information of the real text instance; The encoder is used to encode the sample document image to obtain the sample global feature map information of the sample document image. The initial decoder is used to perform joint spatial and content decoding based on the sample global feature map information to obtain the prediction joint sequence of each predicted text instance in the sample document image. The prediction joint sequence includes the prediction spatial information and prediction content information of the predicted text instance. Calculate the joint loss value based on the real joint sequence and the predicted joint sequence. Once the joint loss value meets the set conditions, the training is considered complete, and the initial decoder that has been trained is used as the joint decoder.
[0009] Optionally, calculating the joint loss value based on the true joint sequence and the predicted joint sequence includes: Calculate the spatial loss value based on the real spatial information and the predicted spatial information, calculate the content loss value based on the real content information and the predicted content information, and calculate the regularization loss value. The joint loss value is obtained by weighted summation of the spatial loss value, the content loss value, and the regularization loss value.
[0010] Optionally, using the trained initial decoder as the joint decoder includes: performing lightweight processing on the trained initial decoder, and using the lightweight initial decoder as the joint decoder.
[0011] Optionally, the initial decoder is a lightweight decoder.
[0012] Secondly, embodiments of this application provide a document image parsing apparatus, the apparatus comprising: The encoding module is used to encode the target document image using an encoder to obtain the target global feature map information of the target document image; The decoding module is used to perform joint decoding of space and content based on the target global feature map information using a joint decoder to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
[0013] Optionally, the decoding module includes: The first processing module is used to perform multi-head self-attention and cross-attention operations based on the target global feature map information to obtain the target spatial information of each target text instance in the target document image. The second processing module is used to perform text generation operations based on the target spatial information and the target global feature map information to obtain the target content information of each target text instance in the target document image. The third processing module is used to integrate the target spatial information and the target content information of the target text instance to obtain the target joint sequence of the target text instance.
[0014] Optionally, the first processing module is specifically used for: Multi-head self-attention and cross-attention operations are performed based on the target global feature map information to obtain candidate spatial information for each target text instance in the target document image; Based on the candidate spatial information, a Bézier curve fitting operation is performed to obtain the target spatial information of each target text instance in the target document image.
[0015] Optionally, the joint decoder is trained using the following modules: The acquisition module is used to acquire sample data, which includes sample document images and a real joint sequence of each real text instance in the sample document images. The real joint sequence includes the real spatial information and real content information of the real text instance. The training module is used to encode the sample document image using the encoder to obtain the sample global feature map information of the sample document image, and to use the initial decoder to perform joint spatial and content decoding based on the sample global feature map information to obtain the prediction joint sequence of each predicted text instance in the sample document image. The prediction joint sequence includes the prediction spatial information and prediction content information of the predicted text instance. The determination module is used to calculate the joint loss value based on the real joint sequence and the predicted joint sequence, determine the training is complete after the joint loss value meets the set conditions, and use the initial decoder that has been trained as the joint decoder.
[0016] Optionally, the determining module is specifically used for: Calculate the spatial loss value based on the real spatial information and the predicted spatial information, calculate the content loss value based on the real content information and the predicted content information, and calculate the regularization loss value. The joint loss value is obtained by weighted summation of the spatial loss value, the content loss value, and the regularization loss value.
[0017] Optionally, the determining module is specifically used to perform lightweight processing on the trained initial decoder, and use the lightweight initial decoder as the joint decoder.
[0018] Optionally, the initial decoder is a lightweight decoder.
[0019] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the document image parsing method as described in any of the preceding claims.
[0020] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the document image parsing method as described in any of the preceding claims.
[0021] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the document image parsing method as described in any of the preceding claims.
[0022] In this embodiment, an encoder is used to encode the target document image to obtain target global feature map information of the target document image. A joint decoder is then used to perform joint spatial and content decoding based on the target global feature map information to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes target spatial information and target content information of the target text instance. Therefore, in this embodiment, the joint decoder can simultaneously perform joint spatial and content decoding to obtain target spatial information and target content information for each target text instance in the target document image. Furthermore, the joint decoding strengthens the correlation between spatial and content information, achieving unified processing of multiple tasks, thus improving the efficiency and accuracy of document image parsing.
[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a flowchart of a document image parsing method according to an embodiment of this application; Figure 2 This is a flowchart of a joint decoder training method according to an embodiment of this application; Figure 3 This is a schematic diagram of a unified framework for document image parsing according to an embodiment of this application; Figure 4 This is a structural block diagram of a document image parsing device according to an embodiment of this application; Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application; Figure 6 This is a structural block diagram of a readable storage medium according to an embodiment of this application. Detailed Implementation
[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0028] Document image parsing tasks can be categorized into: serialized text detection and recognition; table structure and content recognition; and visual entity extraction and localization. However, existing methods are all tailored to specific tasks, with few unified frameworks to handle these tasks. Therefore, this application proposes a unified framework, which refers to a unified technology platform or system architecture that integrates multiple document image parsing tasks, supports end-to-end processing, and collaboratively completes multiple tasks through a unified model or module, thereby improving system consistency, scalability, and engineering practicality.
[0029] Handling this diversity and maintaining superior performance within a unified framework presents several challenges. First, using task-specific headers, adapters, and formulas can hinder model generality. Second, addressing cross-dependencies between tasks is crucial, such as text detection within table recognition. Third, a unified representation of tasks needs to consider key elements (e.g., words, points, lines, cells) and various relationship types (e.g., adjacency between characters, connection between key values, and alignment of table cells). By integrating multiple tasks (e.g., text detection, key information extraction, and table recognition) into a single system, model complexity is simplified, processing efficiency is improved, and model generality is enhanced. This not only reduces development and maintenance costs but also allows the model to adapt more flexibly to different application scenarios, providing consistent and efficient solutions.
[0030] Document image parsing technology plays a crucial role in multimodal tasks, with its primary goal being to accurately and efficiently extract key-value pairs from complex document images. Addressing the issue that document image parsing requires specific architectures for specific tasks due to the diverse objectives and inconsistent patterns, and considering the problems of complex structures, low inference efficiency, high hardware resource consumption, and poor task adaptability in existing document image parsing technologies, this application proposes a single-stage visual localization text parsing scheme that is structurally simple, computationally efficient, resource-friendly, and possesses strong generalization capabilities. This scheme is used to parse document images across different contexts, simultaneously performing joint prediction of spatial layout and text content through a single joint decoder, achieving unified and real-time processing of multiple tasks such as text detection, key information extraction, and table recognition in document images.
[0031] In this embodiment, the multi-stage model is simplified into a single-stage model to improve processing efficiency and application operability. To this end, a joint decoder architecture is designed, and a joint representation and sequence generation strategy is introduced. Through end-to-end training methods and model optimization techniques, the joint decoder can simultaneously predict the spatial and content information of text instances in one forward propagation (i.e., one step).
[0032] Reference Figure 1 The diagram shows a flowchart of a document image parsing method according to an embodiment of this application.
[0033] like Figure 1 As shown, the document image parsing method may include the following steps: Step 101: Encode the target document image using an encoder to obtain the target global feature map information of the target document image.
[0034] In this embodiment, the target document image refers to the document image that needs to be parsed. The target document image may include, but is not limited to, scanned contracts, forms, book pages, etc. Users can upload the target document image to the system in electronic format.
[0035] For example, to further improve the accuracy of data processing, preprocessing can be performed on the target document image to improve its quality. Preprocessing may include, but is not limited to, operations such as grayscale conversion and binarization.
[0036] In this embodiment, the target document image is input into a pre-trained encoder, which encodes the target document image to obtain the target global feature map information. The encoder then outputs the target global feature map information. The target global feature map information may include spatial information and contextual semantic information of the target document image.
[0037] For example, the encoder can be a lightweight image encoder, such as a lightweight convolutional neural network. To further improve the accuracy of the encoder, document image samples can be collected to train the encoder.
[0038] Step 102: Using a joint decoder, perform joint spatial and content decoding based on the target global feature map information to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
[0039] In this embodiment, a joint encoder is pre-trained. After encoding the target document image using the encoder to obtain the target global feature map information of the target document image, the target global feature map information is input into the joint decoder. The joint decoder performs joint spatial and content decoding based on the target global feature map information to obtain the target joint sequence of each target text instance in the target document image. The joint decoder outputs the target joint sequence of each target text instance in the target document image.
[0040] The target joint sequence includes target spatial information and target content information of the target text instance. The target spatial information may include the coordinates of multiple target vertices in the target polygon contour of the target text instance, and the target content information may include multiple target text characters of the target text instance.
[0041] For example, the joint decoder is based on the Transformer model architecture, leveraging its powerful sequence modeling capabilities and self-attention mechanism to simultaneously handle the spatial layout and content generation of text instances. The joint decoder receives target global feature map information from the encoder and outputs a joint target sequence containing target spatial information and corresponding target content information. ,in( ) represents the first polygonal outline of the target in the target spatial information. i The coordinates of the target vertex n This indicates the number of vertices in the target polygonal outline. This indicates the first in the target content information. j One target text character, m This indicates the number of target text characters in the target content information. In each target joint sequence, target spatial information (such as target vertex coordinates) and target content information (such as target text characters) are closely combined to form a complete text instance representation.
[0042] In multi-stage processing methods, the processes of generating spatial information and generating content information are separate, which can lead to error accumulation between the two stages. To avoid this problem and further simplify the processing flow, a target joint sequence form is designed, which encodes the target spatial information and target content information of the target text instance into a single sequence. This target joint sequence allows the joint decoder to generate the complete target text instance in the same step, without the need for separate processing stages.
[0043] In one optional implementation, the process of performing joint spatial and content decoding based on the target global feature map information to obtain the target joint sequence of each target text instance in the target document image may include: performing multi-head self-attention and cross-attention operations based on the target global feature map information to obtain target spatial information of each target text instance in the target document image; performing text generation operations based on the target spatial information and the target global feature map information to obtain target content information of each target text instance in the target document image; and integrating the target spatial information and the target content information of the target text instance to obtain the target joint sequence of the target text instance.
[0044] To enable the joint decoder to perceive the spatial information of the document image, the input target global feature map information is first positionally encoded to generate feature map information containing spatial coordinate information. A multi-head self-attention mechanism is used to process the feature map information containing spatial coordinate information, capturing the spatial relationships within the feature map information and selecting the initial vertex coordinates of the polygonal contour. A cross-attention mechanism is then used, with the initial vertex coordinates as the query matrix and the semantic features (text character features) as the key / value matrix, to accurately select vertex coordinates strongly related to the content as the target spatial information, eliminating irrelevant spatial noise points. Based on the target spatial information, semantic features corresponding to the target spatial information are extracted from the target global feature map information. The joint decoder then generates target content information character by character, ensuring a strict correspondence between the target content information and the target spatial information through attention weight backtracking during the generation process.
[0045] The corresponding formula for the cross-attention mechanism is shown in Formula 1:
[0046] Where Q represents the query matrix, K represents the key matrix, and V represents the value matrix. This represents the dimension of the key vector. express function.
[0047] In one optional implementation, the process of performing multi-head self-attention and cross-attention operations based on the target global feature map information to obtain target spatial information for each target text instance in the target document image may include: performing multi-head self-attention and cross-attention operations based on the target global feature map information to obtain candidate spatial information for each target text instance in the target document image; and performing Bézier curve fitting operations based on the candidate spatial information to obtain target spatial information for each target text instance in the target document image.
[0048] Spatial information obtained from multi-head self-attention and cross-attention operations is used as candidate spatial information (candidate vertex coordinates). Bézier curve fitting is then performed based on the candidate spatial information to reduce the number of vertices and improve representation efficiency.
[0049] The formula for Bézier curve fitting is Formula 2:
[0050] in, This represents a candidate vertex (control point). Denotes Bessel basis functions, This indicates the fitting result. n Indicates the number of vertices. For parameters.
[0051] After receiving the target global feature map information F, the joint decoder first performs position encoding on F to obtain... Then, the target spatial information (i.e., the vertex coordinates of the polygon outline) is generated. Then, the corresponding target content information is generated. Finally, the target joint sequence is output. .
[0052] In this embodiment, a joint decoder can be used to simultaneously perform joint decoding of space and content, obtaining target spatial information and target content information for each target text instance in the target document image. Furthermore, joint decoding strengthens the correlation between space and content, achieving unified processing of multiple tasks, thus improving the efficiency and accuracy of document image parsing.
[0053] In this embodiment, to effectively train the joint decoder, an end-to-end training strategy can be adopted, simultaneously optimizing the generation of spatial and content information through a joint loss function. This strategy can balance the joint decoder's learning of spatial layout and text content during training, helping the joint decoder maintain high accuracy when generating joint sequences. Through end-to-end training, the model can simultaneously learn how to generate joint sequences, including the prediction of spatial and content information, in a single forward propagation, thereby simplifying the training process and improving efficiency.
[0054] Reference Figure 2 The diagram shows a flowchart of a joint decoder training method according to an embodiment of this application.
[0055] like Figure 2 As shown, the joint decoder training method may include the following steps: Step 201: Obtain sample data, which includes sample document images and a real joint sequence of each real text instance in the sample document images. The real joint sequence includes the real spatial information and real content information of the real text instance.
[0056] In this embodiment, the sample document image can be a synthetic document image generated by randomly generating text layout, font, color, background noise, etc., or it can be a real document image obtained from a corresponding dataset. The sample document image is labeled to obtain the true joint sequence of each real text instance in the sample document image.
[0057] The real joint sequence includes the real spatial information and real content information of the real text instance. The real spatial information may include the coordinates of multiple real vertices in the real polygonal contour of the real text instance, and the real content information may include multiple real text characters of the real text instance.
[0058] Step 202: Encode the sample document image using the encoder to obtain the sample global feature map information of the sample document image. Then, use the initial decoder to perform joint spatial and content decoding based on the sample global feature map information to obtain the prediction joint sequence for each predicted text instance in the sample document image. The prediction joint sequence includes the prediction spatial information and prediction content information of the predicted text instance.
[0059] In this embodiment, a sample document image is input into the pre-trained encoder. The encoder encodes the sample document image to obtain sample global feature map information, and then the encoder outputs the sample global feature map information. The sample global feature map information may include spatial information and contextual semantic information of the sample document image.
[0060] In this embodiment, an initial decoder is constructed, which refers to the joint decoder to be trained. The initial decoder is based on the Transformer model architecture.
[0061] After encoding the sample document image using an encoder to obtain the sample global feature map information of the sample document image, the sample global feature map information is input into the initial decoder. The initial decoder performs joint spatial and content decoding based on the sample global feature map information to obtain the joint prediction sequence of each predicted text instance in the sample document image. The initial decoder outputs the joint prediction sequence of each predicted text instance in the sample document image.
[0062] The joint prediction sequence includes the prediction spatial information and prediction content information of the predicted text instance. The prediction spatial information may include the coordinates of multiple predicted vertices in the predicted polygonal contour of the predicted text instance, and the prediction content information may include multiple predicted text characters of the predicted text instance.
[0063] For the specific processing procedures inside the initial decoder, please refer to the relevant description of step 102 above. This embodiment will not discuss it in detail here.
[0064] Step 203: Calculate the joint loss value based on the real joint sequence and the predicted joint sequence. After the joint loss value meets the set conditions, determine that the training is complete, and use the initial decoder that has been trained as the joint decoder.
[0065] The joint loss value is calculated based on the true joint sequence and the predicted joint sequence. If the joint loss value does not meet the set conditions, the adjustable parameters (such as weights and other hyperparameters) of the initial decoder are iteratively adjusted and training continues. Once the joint loss value meets the set conditions, training is considered complete, and the trained initial decoder is used as the joint decoder.
[0066] The preset conditions can be set according to actual needs, and this embodiment does not impose any restrictions on them. For example, the preset conditions can be that the joint loss value is less than a set threshold, or that the joint loss value converges, etc.
[0067] In one optional implementation, the process of calculating the joint loss value based on the real joint sequence and the predicted joint sequence may include: calculating a spatial loss value based on the real spatial information and the predicted spatial information, calculating a content loss value based on the real content information and the predicted content information, and calculating a regularization loss value; and performing a weighted summation of the spatial loss value, the content loss value, and the regularization loss value to obtain the joint loss value.
[0068] This application employs a joint loss function, comprising spatial loss, content loss, and regularization loss. The spatial loss measures the error between the generated predicted spatial information and the true spatial information, and can be calculated using L1 loss or L2 loss to optimize the accuracy of the polygon contour. The content loss measures the difference between the generated predicted content information and the true content information, and can be calculated using cross-entropy loss or CTC (Connectionist Temporal Classification Loss) loss to optimize the accuracy of text generation. This design allows the model to balance the accuracy of spatial layout and the accuracy of text content recognition during training. The regularization loss constrains the model parameter size, prevents overfitting, and ensures that the model's generalization ability is not reduced due to overfitting to noise in the training data during training.
[0069] The formula for calculating the joint loss value can be Equation 3:
[0070] in, It is the joint loss value. It is the spatial loss value. It is the content loss value. It is the regularization loss value. These are the weighting coefficients for the spatial loss value. These are the weighting coefficients for the content loss value. It is the weighting coefficient of the regularization loss value.
[0071] The spatial loss is calculated using the intersection-union ratio (IU) loss of the polygon contour and the vertex coordinate regression loss. The formula for calculating the spatial loss is Equation 4:
[0072] in, It is the intersection-union ratio loss value. It is the regression loss value of vertex coordinates.
[0073] The intersection-union ratio (CURRR) loss value is used to measure the intersection-union ratio between the predicted polygon contour and the true polygon contour. The formula for calculating the CURRR loss value can be expressed as Formula 5:
[0074] in, It predicts polygon outlines. It is a true polygonal outline. Indicates area calculation, Indicates intersection, It represents the union of sets.
[0075] The vertex coordinate regression loss value is used to optimize the vertex coordinates of the polygon contour. The formula for calculating the vertex coordinate regression loss value can be expressed as Formula 6:
[0076] in, It is the first prediction of polygon contours i Predicted vertex coordinates It is the first of the true polygonal contours i The coordinates of the real vertex.
[0077] Content loss can be calculated using either CTC loss or cross-entropy loss, which are suitable for different scenarios. The formula for calculating content loss is Equation 7:
[0078] in, These are real text characters. It predicts text characters. It is the first i The predicted text characters are The probability of.
[0079] In one alternative implementation, data augmentation and transfer learning techniques can also be used to train the decoder. In this implementation, a large number of synthetic and real document images can be used for pre-training, and transfer learning and data augmentation techniques can be used to improve the model's generalization ability across different tasks. First, synthetic document image generation is performed: a large number of synthetic document images are generated by randomly generating text layouts, fonts, colors, and background noise; then data augmentation is performed: including random cropping, rotation, scaling, and color adjustment; next, transfer learning is performed: the model is pre-trained on the synthetic document images, and then fine-tuned on the pre-trained model using a task-specific dataset (real document images).
[0080] The formula for transfer learning is Formula 8:
[0081] in, It is the result of transfer learning. It is a pre-trained model. It is a task-specific dataset.
[0082] In one alternative implementation, a lightweight design can also be implemented.
[0083] For example, using the trained initial decoder as the joint decoder includes: performing lightweight processing on the trained initial decoder, and using the lightweight initial decoder as the joint decoder. Lightweight processing may include, but is not limited to, pruning, quantization, model distillation, etc.
[0084] For example, the initial decoder is a lightweight decoder.
[0085] Model lightweighting: Introduce lightweight model architectures (such as MobileNet or EfficientNet) to reduce the consumption of computing resources, and further compress the model size and improve inference speed through techniques such as pruning, quantization and model distillation.
[0086] Pruning: L1 norm pruning based on weights removes redundant weights. The specific formula for pruning is Formula 9:
[0087] in, This is the weight matrix after pruning. It is the original weight matrix. It's a pruning mask. It represents the Hadamardi (or Hadama) stack.
[0088] Quantization: INT8 (8-bit integer) quantization is used to reduce model storage and computational overhead. The quantization formula is Equation 10:
[0089] in, It is the quantized weight matrix. Represents the quantization function. This indicates the precision of the target quantification.
[0090] Model distillation: Through knowledge distillation, knowledge is transferred from a large model (teacher model) to a small model (student model). The processing formula for model distillation is Formula 11:
[0091] in, It is a teacher model. It is a student model. It is a distillation function.
[0092] In one alternative implementation, real-time optimization can also be performed.
[0093] Inference optimizations are performed for mobile devices and edge computing environments to ensure that the model can achieve real-time parsing in resource-constrained environments.
[0094] Heterogeneous computing: Utilizing the heterogeneous computing capabilities of CPUs (Central Processing Units) and GPUs (Graphics Processing Units) / NPUs (Neural Processing Units) to optimize task allocation. The processing formula for heterogeneous computing is Equation 12:
[0095] in, It is the output result of heterogeneous computing. It is a heterogeneous computation function. It is the input document image. It is a heterogeneous computing strategy.
[0096] Dynamic scheduling: The inference strategy of the model is dynamically adjusted based on the real-time load of the equipment. The processing formula for dynamic scheduling is Formula 13:
[0097] in, It is the result of dynamic scheduling. It is a dynamic scheduling function. It is a static computation strategy. It is a real-time load.
[0098] Reference Figure 3 The diagram illustrates a unified framework for document image parsing according to an embodiment of this application.
[0099] like Figure 3 As shown, in the document image parsing process, the target document image is input into the encoder, and the encoder encodes the target document image to obtain the target global feature map information of the target document image. The target global feature map information is input into the joint decoder, which contains multiple decoders for table recognition, text recognition, and key information extraction, respectively. The joint decoder performs joint decoding of space and content based on the target global feature map information to obtain the target joint sequence of each target text instance in the target document image.
[0100] The embodiments of this application achieve efficient text detection and content recognition through a joint representation and sequence generation strategy, which is suitable for mobile devices and edge computing environments.
[0101] In this embodiment, the parsing results can be displayed: the parsing results are presented to the user in a visual manner, including polygon outlines and text characters. The generated polygon outlines can be highlighted on the target document image to help users intuitively understand the location of the text. The text characters can be displayed in a list format for easy viewing and editing by the user.
[0102] In this embodiment, data export is possible: users can choose to export the parsing results in various formats for further processing or archiving. The exported data includes the polygonal outline and target content information of each target text instance, making it convenient for users to use in other applications.
[0103] The embodiments of this application have the following beneficial effects: Simplified model complexity: By merging the two-stage generation process into a single stage, the complexity of the model is reduced, and the computational overhead of the model is decreased; Improved efficiency and consistency: Through the joint representation and sequence generation strategy, the model can complete all tasks in a single decoding step, improving real-time processing capabilities; Reduced resource requirements: Through lightweight design, the dependence on computing resources is reduced, enabling the model to run effectively on resource-constrained devices; Enhanced accuracy and robustness: The joint sequence ensures the consistency of spatial information and content information, improving the accuracy and robustness of the model.
[0104] It should be noted that the document image parsing method provided in this application embodiment can be executed by a document image parsing device, or a control module within that document image parsing device for executing the document image parsing loading method. This application embodiment uses the execution of the document image parsing loading method by a document image parsing device as an example to illustrate the document image parsing method provided in this application embodiment.
[0105] Reference Figure 4 The diagram shows a structural block diagram of a document image parsing apparatus according to an embodiment of this application.
[0106] like Figure 4 As shown, the document image parsing device may include the following modules: Encoding module 401 is used to encode the target document image using an encoder to obtain the target global feature map information of the target document image; The decoding module 402 is used to perform joint decoding of space and content based on the target global feature map information using a joint decoder to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
[0107] Optionally, the decoding module 402 includes: The first processing module is used to perform multi-head self-attention and cross-attention operations based on the target global feature map information to obtain the target spatial information of each target text instance in the target document image. The second processing module is used to perform text generation operations based on the target spatial information and the target global feature map information to obtain the target content information of each target text instance in the target document image. The third processing module is used to integrate the target spatial information and the target content information of the target text instance to obtain the target joint sequence of the target text instance.
[0108] Optionally, the first processing module is specifically used for: Multi-head self-attention and cross-attention operations are performed based on the target global feature map information to obtain candidate spatial information for each target text instance in the target document image; Based on the candidate spatial information, a Bézier curve fitting operation is performed to obtain the target spatial information of each target text instance in the target document image.
[0109] Optionally, the joint decoder is trained using the following modules: The acquisition module is used to acquire sample data, which includes sample document images and a real joint sequence of each real text instance in the sample document images. The real joint sequence includes the real spatial information and real content information of the real text instance. The training module is used to encode the sample document image using the encoder to obtain the sample global feature map information of the sample document image, and to use the initial decoder to perform joint spatial and content decoding based on the sample global feature map information to obtain the prediction joint sequence of each predicted text instance in the sample document image. The prediction joint sequence includes the prediction spatial information and prediction content information of the predicted text instance. The determination module is used to calculate the joint loss value based on the real joint sequence and the predicted joint sequence, determine the training is complete after the joint loss value meets the set conditions, and use the initial decoder that has been trained as the joint decoder.
[0110] Optionally, the determining module is specifically used for: Calculate the spatial loss value based on the real spatial information and the predicted spatial information, calculate the content loss value based on the real content information and the predicted content information, and calculate the regularization loss value. The joint loss value is obtained by weighted summation of the spatial loss value, the content loss value, and the regularization loss value.
[0111] Optionally, the determining module is specifically used to perform lightweight processing on the trained initial decoder, and use the lightweight initial decoder as the joint decoder.
[0112] Optionally, the initial decoder is a lightweight decoder.
[0113] In this embodiment, a joint decoder can be used to simultaneously perform joint decoding of space and content, obtaining target spatial information and target content information for each target text instance in the target document image. Furthermore, joint decoding strengthens the correlation between space and content, achieving unified processing of multiple tasks, thus improving the efficiency and accuracy of document image parsing.
[0114] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0115] The document image parsing device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0116] The document image parsing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0117] The document image parsing device provided in this application embodiment can implement the various processes implemented by the document image parsing device in any of the above-mentioned method embodiments. To avoid repetition, these processes will not be described again here.
[0118] Optionally, embodiments of this application also provide an electronic device.
[0119] Reference Figure 5 The diagram shows a structural block diagram of an electronic device according to an embodiment of this application.
[0120] like Figure 5As shown, the electronic device 11 includes a processor 111, a memory 112, and a program or instruction 1121 stored in the memory and executable on the processor. When the program or instruction 1121 is executed by the processor 111, it implements the various processes of any of the above-described document image parsing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0121] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0122] Optionally, embodiments of this application also provide a readable storage medium.
[0123] Reference Figure 6 The diagram shows a structural block diagram of a readable storage medium according to an embodiment of this application.
[0124] like Figure 6 As shown, a program or instruction 211 is stored on the readable storage medium 21. When the program or instruction 211 is executed by the processor, it implements the various processes of any of the above document image parsing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0125] The processor may be the processor in the electronic device described in the above embodiments.
[0126] Optionally, this application embodiment also provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the various processes of the above-described document image parsing method embodiments, and achieving the same technical effect, so to avoid repetition, it will not be described again here.
[0127] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0128] The processors mentioned above may include, but are not limited to: Central Processing Unit (CPU), Network Processor (NP), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0129] The aforementioned readable storage media include computer-readable storage media, which may include, but are not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronically erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.
[0130] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.
[0131] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.
[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. The functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0135] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0136] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A document image parsing method, characterized in that, The method includes: The target document image is encoded using an encoder to obtain the target global feature map information of the target document image; Using a joint decoder, spatial and content joint decoding is performed based on the target global feature map information to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
2. The method according to claim 1, characterized in that, The step of performing joint spatial and content decoding based on the target global feature map information to obtain the target joint sequence for each target text instance in the target document image includes: Based on the target global feature map information, multi-head self-attention and cross-attention operations are performed to obtain the target spatial information of each target text instance in the target document image; Based on the target spatial information and the target global feature map information, a text generation operation is performed to obtain the target content information of each target text instance in the target document image; The target spatial information and target content information of the target text instance are integrated to obtain the target joint sequence of the target text instance.
3. The method according to claim 2, characterized in that, The step of performing multi-head self-attention and cross-attention operations based on the target global feature map information to obtain the target spatial information of each target text instance in the target document image includes: Multi-head self-attention and cross-attention operations are performed based on the target global feature map information to obtain candidate spatial information for each target text instance in the target document image; Based on the candidate spatial information, a Bézier curve fitting operation is performed to obtain the target spatial information of each target text instance in the target document image.
4. The method according to claim 1, characterized in that, The joint decoder is trained in the following manner: Acquire sample data, which includes sample document images and a real joint sequence of each real text instance in the sample document images, wherein the real joint sequence includes the real spatial information and real content information of the real text instance; The encoder is used to encode the sample document image to obtain the sample global feature map information of the sample document image. The initial decoder is used to perform joint spatial and content decoding based on the sample global feature map information to obtain the prediction joint sequence of each predicted text instance in the sample document image. The prediction joint sequence includes the prediction spatial information and prediction content information of the predicted text instance. Calculate the joint loss value based on the real joint sequence and the predicted joint sequence. Once the joint loss value meets the set conditions, the training is considered complete, and the initial decoder that has been trained is used as the joint decoder.
5. The method according to claim 4, characterized in that, The step of calculating the joint loss value based on the true joint sequence and the predicted joint sequence includes: Calculate the spatial loss value based on the real spatial information and the predicted spatial information, calculate the content loss value based on the real content information and the predicted content information, and calculate the regularization loss value. The joint loss value is obtained by weighted summation of the spatial loss value, the content loss value, and the regularization loss value.
6. The method according to claim 4, characterized in that, The step of using the trained initial decoder as the joint decoder includes: The trained initial decoder is lightweighted, and the lightweight initial decoder is used as the joint decoder.
7. The method according to claim 4, characterized in that, The initial decoder is a lightweight decoder.
8. A document image parsing device, characterized in that, The device includes: The encoding module is used to encode the target document image using an encoder to obtain the target global feature map information of the target document image; The decoding module is used to perform joint decoding of space and content based on the target global feature map information using a joint decoder to obtain a target joint sequence for each target text instance in the target document image. The target joint sequence includes the target spatial information and target content information of the target text instance.
9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the document image parsing method as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the document image parsing method as described in any one of claims 1 to 7.