Bill structured region segmentation method and system, electronic equipment and storage medium
By combining an improved encoder and attention mechanism with an unsupervised pre-trained optimization model, the accuracy and cost issues in complex backgrounds of document structured region segmentation are solved, achieving high-precision document structured region segmentation.
Patent Information
- Application Number
- CN202511729517.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies struggle to segment structured regions of tickets with high accuracy in complex contexts, and deep learning-based segmentation models require a large amount of high-quality labeled data, resulting in high costs and limiting the promotion and application of these models in real-world scenarios.
An improved encoder is used to extract multi-scale semantic feature maps, which are then combined with spatial attention and channel attention to enhance the feature maps. Attention-enhanced feature maps are generated by fusing them through gating units. Unsupervised pre-training and a composite loss function are used to optimize the model, reducing annotation costs and improving the model's generalization ability.
It achieves high-precision segmentation of structured regions of invoices in complex backgrounds, reduces data annotation costs, and improves the model's adaptability and segmentation accuracy across different invoice styles.
Smart Images
Figure CN121505628A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a method, system, electronic device, and storage medium for segmenting structured regions of invoices. Background Technology
[0002] Against the backdrop of accelerated digital transformation in the financial sector, bank drafts, as a crucial carrier of traditional financial transactions, are experiencing a growing demand for electronic processing. In the automated recognition and processing of draft images, accurately identifying and segmenting structured regions containing customer signatures, account information, amounts, transaction numbers, and ID card numbers has become a key technical aspect for ensuring financial data security, preventing information leakage, and achieving compliant processing. While traditional image segmentation methods can segment structured regions of drafts, their performance is limited in the face of complex backgrounds, diverse draft formats, and uneven lighting, making it difficult to achieve high-precision and robust structured region detection.
[0003] With the rapid development of deep learning, segmentation models based on deep learning (such as U-Net and Mask R-CNN) can be used to achieve structured segmentation of tickets. However, this approach not only struggles to effectively focus on structured regions and has poor robustness to complex background interference, but also requires a large amount of high-quality labeled data for training, resulting in high labeling costs and limiting the promotion and application of the models in real-world scenarios. Summary of the Invention
[0004] In view of this, this application provides a method, system, electronic device and storage medium for segmenting structured regions of invoices, in order to solve the problems in the prior art that it is difficult to effectively focus on structured regions, have poor robustness to complex background interference, and require a large amount of high-quality labeled data for training, which results in high labeling costs and limits the promotion and application of the model in real-world scenarios.
[0005] The first aspect of this application provides a method for segmenting structured regions of a bill, the method comprising:
[0006] Acquire a ticket image and preprocess the ticket image to obtain a target ticket image;
[0007] An improved encoder is used to extract multi-scale semantic feature maps from the target ticket image; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network;
[0008] Based on the multi-scale semantic feature map, a spatial attention-enhanced feature map and a channel attention-enhanced feature map are generated;
[0009] The spatial attention enhancement feature map and the channel attention enhancement feature map are fused based on the multi-scale semantic feature map using a pre-constructed gating unit to obtain an attention enhancement feature map;
[0010] The spatial resolution of the attention-enhanced feature map is restored to obtain the target attention-enhanced feature map;
[0011] The structured region segmentation model utilizes the target attention-enhanced feature map to perform structured segmentation on the ticket image, thereby obtaining the structured regions of the ticket image; wherein, the structured region segmentation model is obtained by training a neural network to be trained using sample data containing positive and negative sample pairs and target sample data with categories.
[0012] Optionally, the invoice image is preprocessed to obtain the target invoice image, including:
[0013] The ticket image is subjected to denoising and contrast enhancement processing;
[0014] The ticket image after noise reduction and contrast enhancement is cropped and scaled.
[0015] Incremental data processing is performed on the cropped and scaled ticket image;
[0016] The target ticket image is obtained by masking the data-augmented ticket image.
[0017] Optionally, generating spatial attention-enhanced feature maps and channel attention-enhanced feature maps based on the multi-scale semantic feature maps includes:
[0018] The multi-scale semantic feature map is subjected to global average pooling using the channel attention branch to obtain the corresponding vector, and a channel attention enhanced feature map is generated based on the vector and the multi-scale semantic feature map.
[0019] Max pooling and average pooling operations are performed on the multi-scale semantic feature map using a spatial attention branch to obtain max pooling and average pooling results. A spatial attention-enhanced feature map is then generated using spatial attention weights and the multi-scale semantic feature map. The spatial attention weights are generated based on the max pooling and average pooling results.
[0020] Optionally, the gating unit includes a fully connected network containing a hidden layer;
[0021] The spatial attention enhancement feature map and the channel attention enhancement feature map are fused based on the multi-scale semantic feature map using a pre-constructed gating unit to obtain an attention enhancement feature map, including:
[0022] The spatial attention-enhanced feature map is processed using the fully connected layer to obtain a first gating value;
[0023] The channel attention enhancement feature map is processed using the fully connected layer to obtain a second gating value;
[0024] First attention weight coefficient and second attention weight coefficient are generated based on the first gating value and the second gating value, respectively;
[0025] The multi-scale semantic feature maps are weighted using the first gate value and its first attention weight coefficient, and the second gate value and its second attention weight coefficient, respectively, to obtain the first weighted result and the second weighted result;
[0026] The first weighted result and the second weighted result are fused to obtain the attention-enhanced feature map.
[0027] Optionally, a structured region segmentation model is obtained by training the neural network to be trained using sample data containing positive and negative sample pairs and target sample data with categories, including:
[0028] Acquire sample data and target sample data, wherein the sample data includes multiple sample pairs, which are either positive or negative sample pairs; the two positive samples in the positive sample pair are obtained by performing two different enhancement operations on the same historical ticket image; the negative sample pair includes two different historical ticket images; the target sample data includes multiple target samples, each carrying a corresponding category and its true value;
[0029] Each of the sample pairs and each of the target samples are input into the neural network to be trained, so that the neural network to be trained can be trained using the input sample pairs and target samples to obtain a structured region segmentation model.
[0030] The process by which the neural network to be trained is performed using the input sample pairs and the target sample is as follows:
[0031] Extract the embedding feature maps of the two samples within the sample pair, and construct a corresponding contrastive loss function using the embedding feature maps of the two samples, so as to perform contrastive pre-training using the contrastive loss function;
[0032] Extract the historical attention enhancement feature map of the target sample, and use the pre-trained neural network to make a prediction based on the historical attention enhancement feature map to obtain the predicted value of the target sample in the corresponding category;
[0033] The cross-entropy loss function and the Dice coefficient loss function are constructed using the predicted and true values of the target sample in the corresponding category, respectively.
[0034] The cross-entropy loss function and the Dice coefficient loss function are integrated to obtain a composite loss function. The parameters of the pre-trained neural network are then adjusted using the composite loss function to obtain a structured region segmentation model.
[0035] Optionally, the method further includes:
[0036] The structured region segmentation model was optimized using gradient descent with the Adam optimizer.
[0037] Optionally, the method includes:
[0038] The structured region is subjected to dilation and erosion morphological operations to obtain the target structured region of the ticket image.
[0039] A second aspect of this application provides a structured region segmentation system for invoices, the system comprising:
[0040] The data preprocessing module is used to acquire the ticket image and preprocess the ticket image to obtain the target ticket image;
[0041] A feature extraction module is used to extract multi-scale semantic feature maps from the target ticket image using an improved encoder; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network.
[0042] A spatial-channel dual-domain dynamic attention enhancement module is used to generate spatial attention enhancement feature maps and channel attention enhancement feature maps based on the multi-scale semantic feature maps; and to fuse the spatial attention enhancement feature maps and the channel attention enhancement feature maps based on the multi-scale semantic feature maps using a pre-built gating unit to obtain an attention enhancement feature map.
[0043] A resolution restoration module is used to restore the spatial resolution of the attention-enhanced feature map to obtain a target attention-enhanced feature map.
[0044] The structured segmentation module is used to perform structured segmentation of the ticket image using the target attention-enhanced feature map through the structured region segmentation model to obtain the structured regions of the ticket image; wherein, the structured region segmentation model is obtained by the contrastive representation learning and unsupervised pre-training module using sample data containing positive and negative sample pairs and target sample data with categories to train the neural network to be trained.
[0045] A third aspect of this application provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the ticket structured region segmentation method provided in the first aspect of this application.
[0046] The fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for performing the bill structured region segmentation method provided in the first aspect of this application.
[0047] This application provides a method, system, electronic device, and storage medium for structured region segmentation of invoices. The method pre-trains a neural network using unlabeled positive and negative sample pairs, enabling the trained model to have good initialization parameters. This not only reduces the cost of data labeling but also improves the model's generalization ability on invoices. Furthermore, training the model using labeled target sample data can further improve the segmentation accuracy. After obtaining the trained structured region segmentation model, the invoice image to be segmented is acquired. A channel-spatial dual-domain dynamic attention mechanism is introduced to generate corresponding spatial attention enhancement feature maps and channel attention enhancement feature maps based on the multi-scale semantic feature maps of the invoice image. Subsequently, a gating unit fuses the spatial and channel attention enhancement feature maps based on the multi-scale semantic feature maps to obtain a target attention enhancement feature map that determines the degree of influence of each domain on the final output, thereby achieving accurate localization and feature enhancement of the structured region. Finally, the trained structured region segmentation model is used to perform structured segmentation of the invoice image based on the target attention enhancement feature map to obtain the corresponding structured regions. This solves the problems of existing technologies, such as difficulty in effectively focusing on structured regions and poor robustness to complex background interference. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0049] Figure 1 A flowchart illustrating a method for segmenting structured regions of a bill, provided as an embodiment of this application;
[0050] Figure 2 A schematic diagram of a structured region segmentation system for invoices provided in an embodiment of the present invention;
[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0054] To better understand the technical solution provided in this application, the technical terms used in this application are explained below:
[0055] U-Net is a convolutional neural network (CNN) architecture commonly used for image segmentation tasks. The U-Net network is characterized by its symmetrical structure, consisting of an encoder and a decoder. The encoder extracts features through convolutional and pooling layers, progressively reducing the image resolution, while the decoder progressively restores the image resolution and uses skip connections to combine low-level features from the encoder with high-level features from the decoder, effectively preserving detailed information in the image. However, when applied to invoice images, although it provides pixel-level accuracy, it is still insufficient in handling complex background interference and diverse invoice styles. This invention improves upon the U-Net architecture by introducing a channel-space dual-domain dynamic attention mechanism between the encoder and decoder, enhancing its ability to handle complex backgrounds and blurred boundaries, thereby improving the segmentation accuracy in complex invoice images.
[0056] Channel attention mechanism: This is a technique that enhances network performance by dynamically adjusting the weights of different channels. In convolutional neural networks, each channel of the feature map represents a different image feature. Through the channel attention mechanism, the network can automatically adjust the weights of each channel during training, making the network pay more attention to the channels most important to the current task (such as segmentation), thereby improving segmentation accuracy. In this invention, by incorporating a channel attention branch that includes the channel attention mechanism, key features in the ticket image are enhanced, thereby improving the model's ability to extract key features from the ticket image subsequently. This allows the model to effectively identify meaningful regions from a lot of noise, thus improving the segmentation effect.
[0057] Spatial attention mechanism: By assigning different weights to different spatial regions of an image, the model can focus on the most important parts of the image. In this invention, the spatial attention branch, which includes a spatial attention mechanism, helps the subsequent model to better identify structured regions in the ticket image, maintaining high segmentation accuracy even when the image is blurred or damaged.
[0058] Dual-domain dynamic attention mechanism: The dual-domain dynamic attention mechanism combines channel attention and spatial attention, and simultaneously weights feature maps in both channel and spatial dimensions. The model can adaptively enhance the response weights of key feature channels and focus on semantically meaningful local regions in the image. This can effectively reduce the interference of background noise and optimize the segmentation results, especially when the ticket image is complex and noisy.
[0059] Contrastive representation pre-training is an unsupervised learning method that learns effective feature representations by comparing the similarities and differences between different samples. This invention introduces a contrastive representation learning strategy, utilizing unlabeled sample data for pre-training to optimize the model's feature representation capabilities. This innovative method improves the model's generalization ability when processing diverse ticket images, effectively addresses the problem of scarce labeled data, and enhances the model's adaptability to unseen ticket styles.
[0060] Composite Loss Function: This function combines cross-entropy loss and Dice coefficient loss to optimize pixel-level segmentation. The cross-entropy loss function improves segmentation accuracy, while the Dice coefficient loss function optimizes the overlap of segmented regions, ensuring higher model stability when handling imbalanced class data. This composite loss function improves accuracy while reducing false negatives and false positives, further optimizing the segmentation results.
[0061] See Figure 1 The diagram illustrates a flowchart of a method for segmenting structured regions of a bill according to an embodiment of this application. This method specifically includes the following steps:
[0062] S101: Acquire the ticket image and preprocess it to obtain the target ticket image.
[0063] In the specific execution step S101, the ticket image of the ticket that needs to be structured segmented is obtained, and the ticket image is denoised and contrast enhanced; the ticket image after denoising and contrast enhancement is cropped and scaled; the ticket image after cropping and scaling is incrementally processed; and the ticket image after data enhancement is masked to obtain the target ticket image.
[0064] In some embodiments, Gaussian filtering or median filtering can be used to remove noise from the ticket image, while histogram equalization or contrast stretching can be used to enhance the contrast of the ticket image, thereby completing the denoising and contrast enhancement processing of the ticket image.
[0065] It should be noted that by performing noise reduction and contrast enhancement processing on the ticket image, the lighting differences in the ticket image can be eliminated, making the details in the ticket image more prominent.
[0066] In some embodiments, the denoised and contrast-enhanced ticket image can be cropped into a specified size according to the input requirements of the neural network, and the cropped ticket image can be adjusted to a uniform size (e.g., 512*512) so that it can be subsequently input into the corresponding neural network.
[0067] It should be noted that incremental data processing includes rotation, scaling, translation, cropping, and so on.
[0068] In some embodiments, random rotation, scaling, translation, and cropping operations can be performed on the cropped and scaled ticket images to not only enhance the diversity of data but also increase the generalization ability of the model, enabling it to maintain high accuracy when faced with different types of ticket images.
[0069] In some embodiments, pixel-level masks of structured regions (such as signatures, amounts, and account numbers) in the data-augmented ticket image can be manually annotated to facilitate subsequent supervised training.
[0070] S102: Extract multi-scale semantic feature maps from the target ticket image using an improved encoder; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network.
[0071] In the embodiments of this application, an improved U-Net architecture is adopted, that is, ResNet-34 is used as the backbone network in the encoder to obtain an improved encoder.
[0072] In the specific execution of step S102, after obtaining the target ticket image, an improved encoder can be used to extract multi-scale semantic feature maps from the target ticket image to retain feature maps at different levels (C1~C5) for subsequent skip connections and attention enhancement, so as to obtain the corresponding attention enhancement feature maps.
[0073] S103: Generate spatial attention-enhanced feature maps and channel attention-enhanced feature maps based on multi-scale semantic feature maps.
[0074] In the specific execution step S103, the channel attention branch is used to generate a channel attention-enhanced feature map based on the multi-scale semantic feature map, and the spatial attention branch is used to generate a spatial attention-enhanced feature map based on the multi-scale semantic feature map.
[0075] Optionally, the process of generating spatial attention-enhanced feature maps and channel attention-enhanced feature maps based on multi-scale semantic feature maps can be as follows: Global average pooling is performed on the multi-scale semantic feature map using the channel attention branch to obtain the corresponding vector, and channel attention-enhanced feature maps are generated based on the vector and the multi-scale semantic feature map; max pooling and average pooling operations are performed on the multi-scale semantic feature map using the spatial attention branch to obtain the max pooling and average pooling results, and spatial attention-enhanced feature maps are generated using the spatial attention weights and the multi-scale semantic feature map; wherein the spatial attention weights are generated based on the max pooling and average pooling results.
[0076] In some embodiments, a multi-scale semantic feature map is input into a channel attention branch, and the multi-channel attention branch performs global average pooling on the input multi-scale semantic feature map to obtain a vector. This vector is then passed through a multilayer perceptron to generate channel attention weights, and the channel attention weights are multiplied by the multi-scale semantic feature map to achieve weighting in the channel dimension, resulting in a channel attention-enhanced feature map.
[0077] It should be noted that by using the channel attention branch to calculate the global information of the multi-scale semantic feature map in each channel, the importance weight of each channel can be adaptively adjusted to highlight the feature map channels that are more important for recognizing structured regions.
[0078] In some embodiments, a multi-scale semantic feature map is input into a spatial attention branch, which performs max pooling and average pooling operations on the multi-scale semantic feature map to obtain max pooling and average pooling results. The max pooling and average pooling results are concatenated, and the concatenated result is passed through one or more convolutional layers to generate spatial attention weights. The sigmoid function is used to map the spatial attention weights to the (0, 1) interval, and finally the mapping result is applied to the multi-scale semantic feature map to obtain a spatial attention-enhanced feature map.
[0079] It should be noted that by using the spatial attention branch to process the multi-scale semantic feature map to obtain the spatial attention enhanced feature map, the subsequent generation of the feature map for the input structured segmentation model can enable the model to adaptively pay attention to the key information regions in the ticket image (such as the regions corresponding to categories such as amount, barcode, signature, and serial number).
[0080] S104: Using a pre-built gating unit, the spatial attention enhancement feature map and the channel attention enhancement feature map are fused based on the multi-scale semantic feature map to obtain the attention enhancement feature map.
[0081] In the specific execution step S104, a gating unit can be pre-constructed, and the gating unit can be used to generate a first gating value and its first attention weight coefficient and a second gating value and its second attention weight coefficient based on the spatial attention enhancement feature map and the channel attention enhancement feature map. Attention enhancement feature map can be generated using the multi-scale semantic feature map, the first gating value and its first attention weight coefficient and the second gating value and its second attention weight coefficient.
[0082] Optionally, the gating unit includes a fully connected network with hidden layers. The process of fusing the spatial attention enhancement feature map and the channel attention enhancement feature map based on the multi-scale semantic feature map using the pre-built gating unit to obtain the attention enhancement feature map can be as follows: the spatial attention enhancement feature map is processed using the fully connected layer to obtain a first gating value; the channel attention enhancement feature map is processed using the fully connected layer to obtain a second gating value; a first attention weight coefficient and a second attention weight coefficient are generated based on the first gating value and the second gating value, respectively; the multi-scale semantic feature map is weighted using the first gating value and its first attention weight coefficient and the second gating value and its second attention weight coefficient, respectively, to obtain a first weighted result and a second weighted result; the first weighted result and the second weighted result are fused to obtain the attention enhancement feature map.
[0083] In practical applications, in order to effectively integrate the attention information in these two domains (spatial attention enhancement feature map and channel attention enhancement feature map), this application designs a gating unit. The main purpose of this unit is to adaptively determine the degree of influence of each domain on the final output, i.e., the corresponding attention weight coefficient, based on the input spatial attention enhancement feature map and channel attention enhancement feature map.
[0084] In some embodiments, the gating unit includes a fully connected network with a small number of hidden layers. The fully connected layers are used to process the spatial attention enhancement feature map to obtain a first gating value, and the channel attention enhancement feature map is processed to obtain a second gating value. The gating values determine the importance ratio of attention in their respective domains. Then, the sigmoid function is used to map the corresponding gating values to the (0, 1) interval to ensure that they can be used as weighting coefficients, i.e., to obtain the corresponding first attention weight coefficient and second attention weight coefficient. The multi-scale semantic feature map is multiplied by its first attention weight and second attention weight, and then multiplied by its respective gating value. Finally, they are added together to obtain the final attention enhancement feature map. The calculation method of the attention enhancement feature map is as shown in formula (1).
[0085] (1)
[0086] In formula (1), Represents multi-scale semantic feature maps. and These are channel attention (first attention weight coefficient) and spatial attention weight (second attention weight coefficient). and These are the first gating value and the second gating value, respectively.
[0087] S105: Restore the spatial resolution of the attention-enhanced feature map to obtain the target attention-enhanced feature map.
[0088] In the specific execution step S105, the attention enhancement feature map can be convolved and upsampled (deconvolved) by the encoder, and the low-level features obtained by upsampling by the encoder can be fused with the high-level features obtained by convolution by skip connections to ensure that the detailed information in the image is effectively restored, so as to restore the corresponding spatial resolution and obtain the target attention enhancement feature map.
[0089] S106: The structured region segmentation model is used to perform structured segmentation of the ticket image using the target attention enhancement feature map to obtain the structured region of the ticket image; wherein, the structured region segmentation model is obtained by training the neural network to be trained using sample data containing positive and negative sample pairs and target sample data with categories.
[0090] In this embodiment, corresponding sample data and target sample data can be obtained, and the sample data and target sample data can be used to train the neural network to be trained to obtain a structured region segmentation model; wherein, the sample data includes multiple sample pairs, which are positive sample pairs or negative sample pairs; the two positive samples of the positive sample pair are obtained by performing two different enhancement operations on the same historical ticket image; the negative sample pair includes two different historical ticket images; the target sample data includes multiple target samples, and the target samples carry the corresponding category and its true value.
[0091] Optionally, the process of training a neural network containing positive and negative sample pairs and target sample data with categories to obtain a structured region segmentation model can be as follows: Input each sample pair and each target sample into the neural network to be trained, allowing the neural network to be trained using the input sample pairs and target samples to obtain the structured region segmentation model. The process of training the neural network using the input sample pairs and target samples is as follows: Extract the embedding feature maps of the two samples within a sample pair, and construct a corresponding contrastive loss function using the embedding feature maps of the two samples for contrastive pre-training; extract the historical attention enhancement feature map of the target sample, and use the contrastively pre-trained neural network to predict the target sample's predicted value in the corresponding category based on the historical attention enhancement feature map; construct a cross-entropy loss function and a Dice coefficient loss function using the predicted value and the true value of the target sample in the corresponding category, respectively; integrate the cross-entropy loss function and the Dice coefficient loss function to obtain a composite loss function, and use the composite loss function to adjust the parameters of the contrastively pre-trained neural network to obtain the structured region segmentation model.
[0092] The process of extracting the historical attention enhancement feature map of the target sample can be referred to the process of extracting the target attention enhancement feature map of the ticket image provided in the embodiments of this application, and is not limited in this embodiment.
[0093] It should be noted that the target sample category can be amount, barcode, signature, serial number, etc., but this embodiment of the application does not limit it.
[0094] In some embodiments, this application employs a contrastive representation learning method to perform a first-stage contrastive pre-training on the neural network to be trained, enabling the neural network to perform contrastive learning on unlabeled sample data. This maximizes the similarity between positive and negative sample pairs to learn the corresponding feature representations, that is, maximizing the similarity between similar samples (positive sample pairs) and minimizing the distance between different samples (negative sample pairs). This helps the model learn useful image feature representations from a large amount of unlabeled data, thereby improving the model's performance without a large amount of labeled data.
[0095] In this embodiment, multiple historical invoice images can be acquired. For each historical invoice image, two different enhancement operations (such as rotation + cropping, color jitter + blurring) are performed, and the historical invoice images obtained by performing the two different enhancement operations constitute a positive sample. Any two historical invoice images constitute a negative sample pair. Given a sample pair... (Positive sample pair or negative sample pair), extract the embedding features of the two samples in the sample pair, and use the embedding features of the two samples to construct the corresponding contrastive loss function, so as to make the distance between similar samples smaller and the distance between dissimilar samples larger. The constructed contrastive loss function is shown in formula (2).
[0096] (2)
[0097] In formula (2), and These are samples and samples Embedding features, It is the temperature coefficient. It refers to the batch size. For comparison loss functions.
[0098] In some embodiments, after completing the first stage of contrastive pre-training, labeled target sample data can be used to fine-tune the parameters of the contrastive pre-trained neural network, further optimizing the network parameters of the neural network and improving the segmentation accuracy of structured regions.
[0099] It should be noted that, in order to optimize the segmentation effect, this application designs a composite loss function, which combines the cross-entropy function and the Dice coefficient loss function to jointly improve the segmentation accuracy and overlap of the model.
[0100] In this embodiment, the cross loss is used to handle pixel-level classification tasks. The network can be trained by minimizing the difference between the predicted value and the true value of the predicted label. Specifically, the pre-trained neural network is used to predict the predicted value of the target sample in the corresponding category based on the historical attention enhancement feature map of the target sample. The cross entropy loss function is constructed using the predicted value and the true value of the target sample in the corresponding category. The cross entropy loss function is shown in formula (3).
[0101] (3)
[0102] In formula (3), The cross-entropy loss function; It is the true value of target sample i in class c. It is the predicted value of target sample i in category c, where C is the number of categories and N is the number of samples.
[0103] In some embodiments, it has been found that the Dice coefficient is used to measure the overlap between the segmentation result and the true label, and is particularly suitable for handling small regions or class imbalance tasks. Therefore, in order to better optimize the overlap of the segmented regions, the corresponding Dice coefficient loss function can be constructed by using the predicted value and the true value of the target sample in the corresponding category. The Dice coefficient loss function is shown in formula (4). The cross-entropy loss function and the Dice coefficient loss function are integrated to obtain the composite loss function, which is shown in formula (5).
[0104] (4)
[0105] In formula (4), The loss function is the Dice coefficient. It is the true value of target sample i in class c. is the predicted value of target sample i in category c, and N is the number of target samples.
[0106] (5)
[0107] in, Let cross-entropy be the loss function. The loss function is the Dice coefficient. This is a composite loss function, and the weighting coefficients are used to balance the effects of the two types of losses.
[0108] Furthermore, after obtaining the structured region segmentation model, the Adam optimizer can be used to perform gradient descent optimization on the model. This allows the model to adaptively adjust the learning rate of each parameter to meet the update requirements of different parameters, thereby accelerating convergence and preventing oscillations during training. Simultaneously, by gradually reducing the learning rate during training, the model is ensured to be more stable as it approaches the optimal solution, avoiding overfitting.
[0109] In the specific execution step S106, the target attention enhancement feature map can be input into the structured region segmentation model so that the structured region segmentation model can use the target attention enhancement feature map to perform structured segmentation of the ticket image and obtain the structured region of the ticket image.
[0110] In this embodiment, after obtaining the structured region of the ticket image, dilation and erosion morphological operations can be further performed on the structured region to refine the segmentation result, remove isolated noise points within the structured segmentation region, fill in missing areas, and ensure the integrity of the structured region. Finally, connected component analysis can be used to clean the results, removing small regions that do not belong to the structured region, ensuring the consistency and accuracy of the final segmentation result.
[0111] This application provides a method for structured region segmentation of invoices. It pre-trains a neural network using unlabeled positive and negative sample pairs, enabling the trained model to have good initialization parameters. This not only reduces the cost of data labeling but also improves the model's generalization ability on invoices. Furthermore, it can train the model using labeled target sample data, further enhancing the segmentation accuracy. After obtaining the trained structured region segmentation model, the invoice image to be segmented is acquired. A channel-space dual-domain dynamic attention mechanism is introduced to generate corresponding spatial attention enhancement feature maps and channel attention enhancement feature maps based on the multi-scale semantic feature maps of the invoice image. Subsequently, a gating unit fuses the spatial and channel attention enhancement feature maps based on the multi-scale semantic feature maps to obtain a target attention enhancement feature map that determines the degree of influence of each domain on the final output, thereby achieving accurate localization and feature enhancement of the structured region. Finally, the trained structured region segmentation model is used to perform structured segmentation of the invoice image based on the target attention enhancement feature map, obtaining the corresponding structured regions. This solves the problems of existing technologies, such as difficulty in effectively focusing on structured regions and poor robustness to complex background interference.
[0112] Based on the structured region segmentation method for invoices provided in the embodiments of this application above, correspondingly, the embodiments of this application provide a structured segmentation system for invoices, such as... Figure 2 As shown, the system includes:
[0113] The data preprocessing module 21 is used to acquire the ticket image and preprocess the ticket image to obtain the target ticket image;
[0114] Feature extraction module 22 is used to extract multi-scale semantic feature maps from the target ticket image using an improved encoder; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network;
[0115] The spatial-channel dual-domain dynamic attention enhancement module 23 is used to generate spatial attention enhancement feature maps and channel attention enhancement feature maps based on multi-scale semantic feature maps; the spatial attention enhancement feature maps and channel attention enhancement feature maps are fused based on multi-scale semantic feature maps using a pre-built gating unit to obtain the attention enhancement feature map;
[0116] The resolution restoration module 24 is used to restore the spatial resolution of the attention-enhanced feature map to obtain the target attention-enhanced feature map.
[0117] The structured segmentation module 25 is used to perform structured segmentation of the ticket image using the target attention enhancement feature map through the structured region segmentation model to obtain the structured region of the ticket image; wherein, the structured region segmentation model is obtained by the contrastive representation learning and unsupervised pre-training module using sample data containing positive and negative sample pairs and target sample data with categories to train the neural network to be trained.
[0118] Optionally, a data preprocessing module is used to acquire the ticket image and preprocess it to obtain the target ticket image. Specifically, this module is used for:
[0119] The ticket image is subjected to noise reduction and contrast enhancement processing;
[0120] The ticket image after noise reduction and contrast enhancement is cropped and scaled.
[0121] Incremental data processing is performed on the cropped and scaled ticket image;
[0122] The target ticket image is obtained by masking the data-augmented ticket image.
[0123] Optionally, a spatial-channel dual-domain dynamic attention enhancement module is provided to generate spatial attention-enhanced feature maps and channel attention-enhanced feature maps based on multi-scale semantic feature maps. Specifically, this module is used for:
[0124] The channel attention branch is used to perform global average pooling on the multi-scale semantic feature map to obtain the corresponding vector, and the channel attention enhanced feature map is generated based on the vector and the multi-scale semantic feature map.
[0125] The spatial attention branch is used to perform max pooling and average pooling operations on the multi-scale semantic feature map to obtain the max pooling result and the average pooling result. The spatial attention weights and the multi-scale semantic feature map are then used to generate a spatial attention-enhanced feature map. The spatial attention weights are generated based on the max pooling result and the average pooling result.
[0126] Optionally, the gating unit includes a fully connected network with hidden layers. It utilizes a pre-built gating unit to fuse spatial attention enhancement feature maps and channel attention enhancement feature maps based on multi-scale semantic feature maps, resulting in a spatial-channel dual-domain dynamic attention enhancement module for the attention enhancement feature map. Specifically, this module is used for:
[0127] Optionally, the neural network to be trained is trained using sample data containing positive and negative sample pairs and target sample data with categories, resulting in a contrastive representation learning and unsupervised pre-training module for the structured region segmentation model, specifically used for:
[0128] Acquire sample data and target sample data. The sample data includes multiple sample pairs, which can be positive or negative. The two positive samples in a positive sample pair are obtained by performing two different enhancement operations on the same historical ticket image. The negative sample pair includes two different historical ticket images. The target sample data includes multiple target samples, each carrying its corresponding category and its true value.
[0129] Each sample pair and each target sample are input into the neural network to be trained, so that the neural network to be trained can be trained using the input sample pairs and target samples to obtain a structured region segmentation model.
[0130] The process of training the neural network using input sample pairs and target samples is as follows:
[0131] Extract the embedding feature maps of the two samples within a sample pair, and construct the corresponding contrastive loss function using the embedding feature maps of the two samples, so as to perform contrastive pre-training using the contrastive loss function;
[0132] Extract the historical attention enhancement feature map of the target sample, and use the pre-trained neural network to make predictions based on the historical attention enhancement feature map to obtain the predicted value of the target sample in the corresponding category;
[0133] The cross-entropy loss function and the Dice coefficient loss function are constructed using the predicted and true values of the target sample in the corresponding category, respectively.
[0134] By integrating the cross-entropy loss function and the Dice coefficient loss function, a composite loss function is obtained. The parameters of the pre-trained neural network are then adjusted using the composite loss function to obtain a structured region segmentation model.
[0135] Optionally, the contrastive representation learning and unsupervised pre-training module provided in this application embodiment is further used for:
[0136] The Adam optimizer is used to optimize the structured region segmentation model using gradient descent.
[0137] Optionally, the structured segmentation module provided in this application embodiment is further used for:
[0138] The target structured region of the bill image is obtained by performing dilation and erosion morphological operations on the structured region.
[0139] This application also provides a storage medium storing program instructions, which, when loaded and executed by a processor, implement any of the above-described embodiments of the document structured region segmentation method.
[0140] This application also provides an electronic device, such as Figure 3 As shown, the device includes a processor 301 and a memory 302, which are connected via a bus. The memory stores program instructions. The processor calls the program instructions in the memory to execute any of the above-described embodiments of the ticket structured region segmentation method.
[0141] The processor mentioned in this article can be the terminal's CPU, an integrated MCU within the terminal, or a combination of a CPU and an MCU. Furthermore, the processor contains a kernel that retrieves the corresponding program from memory; one or more kernels can be configured.
[0142] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. Units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0144] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0146] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for segmenting structured regions of invoices, characterized in that, The method includes: Acquire a ticket image and preprocess the ticket image to obtain a target ticket image; An improved encoder is used to extract multi-scale semantic feature maps from the target ticket image; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network; Based on the multi-scale semantic feature map, a spatial attention-enhanced feature map and a channel attention-enhanced feature map are generated; The spatial attention enhancement feature map and the channel attention enhancement feature map are fused based on the multi-scale semantic feature map using a pre-constructed gating unit to obtain an attention enhancement feature map; The spatial resolution of the attention-enhanced feature map is restored to obtain the target attention-enhanced feature map; The structured region segmentation model utilizes the target attention-enhanced feature map to perform structured segmentation on the ticket image, thereby obtaining the structured regions of the ticket image; wherein, the structured region segmentation model is obtained by training a neural network to be trained using sample data containing positive and negative sample pairs and target sample data with categories.
2. The method according to claim 1, characterized in that, The document image is preprocessed to obtain the target document image, including: The ticket image is subjected to denoising and contrast enhancement processing; The ticket image after noise reduction and contrast enhancement is cropped and scaled. Incremental data processing is performed on the cropped and scaled ticket image; The target ticket image is obtained by masking the data-augmented ticket image.
3. The method according to claim 1, wherein the feature map comprises generating a spatial attention-enhanced feature map and a channel attention-enhanced feature map based on the multi-scale semantic feature map, including: The multi-scale semantic feature map is subjected to global average pooling using the channel attention branch to obtain the corresponding vector, and a channel attention enhanced feature map is generated based on the vector and the multi-scale semantic feature map. Max pooling and average pooling operations are performed on the multi-scale semantic feature map using a spatial attention branch to obtain max pooling and average pooling results. A spatial attention-enhanced feature map is then generated using spatial attention weights and the multi-scale semantic feature map. The spatial attention weights are generated based on the max pooling and average pooling results.
4. The method according to claim 1, characterized in that, The gating unit includes a fully connected network containing a hidden layer; The spatial attention enhancement feature map and the channel attention enhancement feature map are fused based on the multi-scale semantic feature map using a pre-constructed gating unit to obtain an attention enhancement feature map, including: The spatial attention-enhanced feature map is processed using the fully connected layer to obtain a first gating value; The channel attention enhancement feature map is processed using the fully connected layer to obtain a second gating value; First attention weight coefficient and second attention weight coefficient are generated based on the first gating value and the second gating value, respectively; The multi-scale semantic feature maps are weighted using the first gate value and its first attention weight coefficient, and the second gate value and its second attention weight coefficient, respectively, to obtain the first weighted result and the second weighted result; The first weighted result and the second weighted result are fused to obtain the attention-enhanced feature map.
5. The method according to claim 1, characterized in that, A structured region segmentation model is obtained by training a neural network using sample data containing positive and negative sample pairs and target sample data with categories, including: Acquire sample data and target sample data, wherein the sample data includes multiple sample pairs, which are either positive or negative sample pairs; the two positive samples in the positive sample pair are obtained by performing two different enhancement operations on the same historical ticket image; the negative sample pair includes two different historical ticket images; the target sample data includes multiple target samples, each carrying a corresponding category and its true value; Each of the sample pairs and each of the target samples are input into the neural network to be trained, so that the neural network to be trained can be trained using the input sample pairs and target samples to obtain a structured region segmentation model. The process by which the neural network to be trained is performed using the input sample pairs and the target sample is as follows: Extract the embedding feature maps of the two samples within the sample pair, and construct a corresponding contrastive loss function using the embedding feature maps of the two samples, so as to perform contrastive pre-training using the contrastive loss function; Extract the historical attention enhancement feature map of the target sample, and use the pre-trained neural network to make a prediction based on the historical attention enhancement feature map to obtain the predicted value of the target sample in the corresponding category; The cross-entropy loss function and the Dice coefficient loss function are constructed using the predicted and true values of the target sample in the corresponding category, respectively. The cross-entropy loss function and the Dice coefficient loss function are integrated to obtain a composite loss function. The parameters of the pre-trained neural network are then adjusted using the composite loss function to obtain a structured region segmentation model.
6. The method according to claim 5, characterized in that, The method further includes: The structured region segmentation model was optimized using gradient descent with the Adam optimizer.
7. The method according to claim 1, characterized in that the method comprises: The structured region is subjected to dilation and erosion morphological operations to obtain the target structured region of the ticket image.
8. A structured region segmentation system for invoices, characterized in that, The system includes: The data preprocessing module is used to acquire the ticket image and preprocess the ticket image to obtain the target ticket image; A feature extraction module is used to extract multi-scale semantic feature maps from the target ticket image using an improved encoder; wherein the improved encoder is an encoder that uses ResNet-34 as the backbone network. A spatial-channel dual-domain dynamic attention enhancement module is used to generate spatial attention enhancement feature maps and channel attention enhancement feature maps based on the multi-scale semantic feature maps; and to fuse the spatial attention enhancement feature maps and the channel attention enhancement feature maps based on the multi-scale semantic feature maps using a pre-built gating unit to obtain an attention enhancement feature map. A resolution restoration module is used to restore the spatial resolution of the attention-enhanced feature map to obtain a target attention-enhanced feature map. The structured segmentation module is used to perform structured segmentation of the ticket image using the target attention-enhanced feature map through the structured region segmentation model to obtain the structured regions of the ticket image; wherein, the structured region segmentation model is obtained by the contrastive representation learning and unsupervised pre-training module using sample data containing positive and negative sample pairs and target sample data with categories to train the neural network to be trained.
9. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the bill structured region segmentation method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the document structured region segmentation method as described in any one of claims 1-7.