Value-added tax invoice image content segmentation method and system based on YOLOv8-Seg

Abstract

The invention discloses a value-added tax invoice image content segmentation method and system based on YOLOv8-Seg, a small target detection layer is introduced to enable a model to more accurately capture and position a small target in an invoice, and the recognition and detection capability of a complex invoice is effectively improved. A C2fDCN module is introduced, and deformable convolution is fused, so that the perception capability of the model on targets with different sizes and forms in the image is enhanced, and the model can better adapt to various changes in the image. A global attention GAM module is introduced, the feature extraction capability of the model is enhanced, SIoU loss is introduced as a bounding box loss function, the problem of low accuracy in CIoU calculation is solved, and the target positioning precision and stability of the model are improved. By optimizing the subsequent process, the accurate cutting and recognition of the invoice text object are realized, and finally, the character recognition model is constructed based on the character recognition open source framework, so that an accurate and reliable data basis is provided for processing and analyzing the invoice.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a method and system for segmenting value-added tax invoice image content based on YOLOv8-Seg. Background Art

[0002] At present, the financial staff of most enterprises are still using traditional methods, and the processing methods of invoices mainly include paper and automated office. In the early days, various industries mainly used paper invoices for storage, but this method would lead to heavy maintenance work and waste of resources in the later stage. With the continuous development of information technology, automated office came into being, but traditional automated office still only saves invoices in the form of pictures. It is difficult for financial staff to extract information directly from invoice images. Faced with a large number of invoice images, they still need to manually identify and retrieve the required information. This method increases the repetitive work of financial staff, significantly reduces work efficiency, and increases the labor costs of enterprises.

[0003] With the development of deep learning, invoice recognition methods based on deep learning have gradually become mainstream, such as template matching-based methods and key information extraction technologies. However, the invoice structure is complex, contains a lot of similar information, and most of the processes are relatively complex, making it difficult to promote.

[0004] There are several core problems in the existing VAT invoice recognition and processing methods. First, the traditional invoice processing method relies on manual recognition, resulting in low efficiency in information extraction. Financial personnel need to check the invoice images one by one to extract the required information from them, which is time-consuming and error-prone. Second, existing methods such as template matching have poor adaptability and are difficult to handle invoices of various formats, which reduces the recognition rate. In addition, manual intervention leads to a lot of repetitive work, which increases the manpower costs of enterprises. Invoices contain a lot of similar information, making it difficult to distinguish and extract key information when processing. At the same time, although deep learning-based recognition methods are gradually emerging, these methods usually require a large amount of labeled data for training, and the cost of data acquisition and labeling is high, and the generalization ability of the model is limited when facing new types of invoices. Finally, many automated processing methods fail to achieve real-time recognition and processing, resulting in delays in the invoice review and reimbursement process for enterprises. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art and propose a value-added tax invoice content segmentation method and system based on YOLOv8-Seg.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention proposes a method for content segmentation of a value-added tax invoice image based on YOLOv8-Seg, comprising:

[0008] Get VAT invoice image data;

[0009] Build a VAT invoice image content detection model based on the YOLOv8-Seg model;

[0010] Inputting the VAT invoice image data into the VAT invoice image content segmentation model to obtain the VAT invoice image;

[0011] Process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

[0012] Preferably, the VAT invoice image content detection model is constructed based on the YOLOv8-Seg model, specifically:

[0013] Based on the YOLOv8-Seg model, the P2 layer features are introduced for small target detection to obtain the YOLOv8-Seg-P2 model;

[0014] Based on the YOLOv8-Seg-P2 model, the C2f_DCN module is used to replace the C2f module in Backbone to obtain the YOLOv8-Seg-P2-C2f_DCN model;

[0015] Based on the YOLOv8-Seg-P2-C2f_DCN model, the global attention module GAM is introduced into the Neck network to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM model;

[0016] Based on the YOLOv8-Seg-P2-C2f_DCN-GAM model, the SIoU loss function is used to replace the original CIoU loss function to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM-SIoU model, which is the VAT invoice image content segmentation model.

[0017] Preferably, the C2f_DCN module is specifically: the C2f_DCN module adopts a DCN_Bottleneck structure, and the DCN_Bottleneck structure includes two layers of deformable convolutional layers, which are connected in sequence and then superimposed, specifically:

[0018] First, the input feature map is processed by the convolution operation to calculate the position offset; the calculated position offset is superimposed with the original position of the convolution kernel to obtain the final sampling position of the convolution kernel on the input feature map; sampling is performed on the input feature map according to the superimposed position to obtain the sampled feature map; the sampled feature map is element-wise multiplied with the convolution kernel and summed to obtain the final convolution result; the final convolution result is the output feature map, which is expressed as follows:

[0019]

[0020] Among them, p0 is the center position of the sample in the input; p k is the point in the enumerated sampling / offset point set K; Δp k The corresponding offset, x(p0+p k +Δp k ) means taking out the pixel value at that position, w k Represents the convolution kernel relative to (p0+p k +Δp k )The weight value of the pixel at the position.

[0021] Preferably, the specific implementation of the global attention GAM module is as follows:

[0022] Given a feature map F1, perform global average pooling and global maximum pooling operations on the input feature map F1 to capture global channel-level information; process global information through a shared multi-layer perceptron MLP structure, and generate attention weights for each channel through an activation function; multiply the generated channel weights by the input feature map F1 channel by channel to obtain the corrected feature map F2; perform global average pooling and global maximum pooling on the corrected feature map F2 to generate two single-channel feature maps; fuse the features of the two single-channel feature maps through convolution operations, and apply activation functions to generate a spatial attention map; multiply the generated spatial attention map by element-by-element with the input intermediate feature map F2 to obtain the final corrected feature map F3;

[0023]

[0024] Among them: F1 is the input feature map; M c and M s are channel and spatial attention feature maps respectively, is the matrix dot product.

[0025] Preferably, the SIoU loss function includes distance loss, shape loss and intersection-over-union loss;

[0026] The SIoU loss function is as follows:

[0027]

[0028] Among them, L IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss;

[0029] The distance loss between the predicted box and the true box is as follows:

[0030]

[0031] Among them, t is the variable parameter, ρ x is the normalized squared difference between the horizontal coordinates of the center point of the predicted box and the center point of the true box ρ y Represents the normalized squared difference between the horizontal and vertical coordinates of the center point of the predicted box and the center point of the real box γ is the adjustment parameter, is the x-axis coordinate of the center of the real frame, b cx is the x-axis coordinate of the center of the prediction box, is the y-axis coordinate of the center of the real frame, is the y-axis coordinate of the center of the prediction box, c h is the height difference between the center point of the real box and the predicted box, c w is the width difference between the center point of the real box and the predicted box, γ = 2-Λ, Λ is the angle loss σ is the distance between the center point of the real box and the predicted box, c h is the height difference between the center point of the real box and the predicted box;

[0032] The shape loss is as follows:

[0033]

[0034] Among them, t is the variable parameter, ω w is the normalized difference between the predicted box width and the true box width, ω h is the normalized difference between the predicted box height and the true box height, θ is used to control the degree of attention paid to shape loss, w represents the width of the prediction box, and h represents the height of the prediction box; w gt is the height of the real frame, h gt is the actual box width.

[0035] Preferably, the VAT invoice image is processed to obtain accurate VAT invoice content, specifically:

[0036] Extract the content information of the VAT invoice image and crop it to obtain the images of each field;

[0037] The cropped images of each field are sent to the text recognition model for recognition to obtain the text content, and the text content is associated with its semantics in the invoice to obtain accurate VAT invoice content.

[0038] Preferably, the content information of the VAT invoice image is extracted and cropped to obtain images of various fields, specifically:

[0039] The Sklansky algorithm is used to solve the convex hull point set of the VAT invoice image;

[0040] The convex hull point set is processed using the rotating caliper algorithm to generate the minimum area circumscribed rectangle;

[0041] The target is accurately cropped according to the minimum bounding rectangle to obtain images of each field.

[0042] The present invention proposes a VAT invoice image content segmentation system based on YOLOv8-Seg, comprising:

[0043] A first data acquisition module, the first data acquisition module is used to acquire value-added tax invoice image data;

[0044] A model building module, wherein the model building module is used to build a value-added tax invoice image content detection model based on a YOLOv8-Seg model;

[0045] A second data acquisition module, the second data acquisition module is used to input the value-added tax invoice image data into the value-added tax invoice image content segmentation model to obtain the value-added tax invoice image;

[0046] The data processing module is used to process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

[0047] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for content segmentation of a value-added tax invoice image based on YOLOv8-Seg are implemented.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for content segmentation of a value-added tax invoice image based on YOLOv8-Seg.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention proposes a method for content segmentation of VAT invoices based on YOLOv8-Seg. The introduction of a small target detection layer enables the model to more accurately capture and locate small targets in the invoice, effectively improving the recognition and detection capabilities of complex invoices. The C2f_DCN module is introduced, and deformable convolution is integrated to enhance the model's perception of targets of different sizes and shapes in the image, so that it can better adapt to various changes in the image. The global attention GAM module is introduced to enhance the model's feature extraction capability, and the SIoU loss is introduced as the bounding box loss function to solve the problem of low accuracy in CIoU calculation, thereby improving the accuracy and stability of the model's target positioning. By optimizing the subsequent process, accurate cropping and recognition of invoice text objects are achieved. Finally, an SVRT text recognition model is constructed based on the Baidu PaddleOCR text recognition open source framework, providing an accurate and reliable data basis for further processing and analysis of invoices.

[0051] The present invention proposes a VAT invoice content segmentation system based on YOLOv8-Seg, which obtains accurate VAT invoice content and realizes VAT invoice image content segmentation by dividing the system into a first data acquisition module, a model building module, a second data acquisition module and a data processing module. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 This is a flow chart of the VAT invoice content segmentation method based on YOLOv8-Seg of the present invention.

[0054] Figure 2 It is a schematic diagram of the improved YOLOv8 model structure of the present invention.

[0055] Figure 3 It is a schematic diagram of the C2f_DCN structure of the present invention.

[0056] Figure 4 Schematic diagram of the network structure of the global attention mechanism GAM module unit of the present invention.

[0057] Figure 5 It is a schematic diagram of the content recognition result of the present invention.

[0058] Figure 6It is a schematic diagram of the subsequent cropping results of the content recognition results of the present invention.

[0059] Figure 7 It is a schematic diagram of the text recognition result of the present invention.

[0060] Figure 8 This is a diagram of the VAT invoice content segmentation system based on YOLOv8-Seg of the present invention.

[0061] Fig. 9 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0065] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0066] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0067] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0069] Example 1

[0070] The present invention proposes a method for content segmentation of value-added tax invoice images based on YOLOv8-Seg, such as Figure 1 As shown, the following steps are included:

[0071] S1. Obtaining VAT invoice image data;

[0072] S2. Build a VAT invoice image content detection model based on the YOLOv8-Seg model;

[0073] The VAT invoice image content detection model is constructed based on the YOLOv8-Seg model, specifically:

[0074] Based on the YOLOv8-Seg model, the P2 layer features are introduced for small target detection to obtain the YOLOv8-Seg-P2 model;

[0075] Based on the YOLOv8-Seg-P2 model, the C2f_DCN module is used to replace the C2f module in Backbone to obtain the YOLOv8-Seg-P2-C2f_DCN model;

[0076] Based on the YOLOv8-Seg-P2-C2f_DCN model, the global attention module GAM is introduced into the Neck network to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM model;

[0077] Based on the YOLOv8-Seg-P2-C2f_DCN-GAM model, the SIoU loss function is used to replace the original CIoU loss function to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM-SIoU model, which is the VAT invoice image content segmentation model.

[0078] The C2f_DCN module is specifically: The C2f_DCN module adopts a DCN_Bottleneck structure, which includes two layers of deformable convolutional layers, which are connected in sequence and then superimposed. Specifically:

[0079] First, the input feature map is processed by the convolution operation to calculate the position offset; the calculated position offset is superimposed with the original position of the convolution kernel to obtain the final sampling position of the convolution kernel on the input feature map; sampling is performed on the input feature map according to the superimposed position to obtain the sampled feature map; the sampled feature map is element-wise multiplied with the convolution kernel and summed to obtain the final convolution result; the final convolution result is the output feature map, which is expressed as follows:

[0080]

[0081] Among them, p0 is the center position of the sample in the input; p k is the point in the enumerated sampling / offset point set K; Δp k The corresponding offset, x(p0+p k +Δp k ) means taking out the pixel value at that position, w k Represents the convolution kernel relative to (p0+p k +Δp k )The weight value of the pixel at the position.

[0082] The specific implementation of the global attention GAM module is as follows:

[0083] Given a feature map F1, perform global average pooling and global maximum pooling operations on the input feature map F1 to capture global channel-level information; process global information through a shared multi-layer perceptron MLP structure, and generate attention weights for each channel through an activation function; multiply the generated channel weights by the input feature map F1 channel by channel to obtain the corrected feature map F2; perform global average pooling and global maximum pooling on the corrected feature map F2 to generate two single-channel feature maps; fuse the features of the two single-channel feature maps through convolution operations, and apply activation functions to generate a spatial attention map; multiply the generated spatial attention map by element-by-element with the input intermediate feature map F2 to obtain the final corrected feature map F3;

[0084]

[0085] Among them: F1 is the input feature map; M c and M s are channel and spatial attention feature maps respectively, is the matrix dot product.

[0086] The SIoU loss function includes distance loss, shape loss and intersection-over-union loss;

[0087] The SIoU loss function is as follows:

[0088]

[0089] Among them, L IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss;

[0090] The distance loss between the predicted box and the true box is as follows:

[0091]

[0092] Among them, t is the variable parameter, ρ x is the normalized squared difference between the horizontal coordinates of the center point of the predicted box and the center point of the true box ρ y Represents the normalized squared difference between the horizontal and vertical coordinates of the center point of the predicted box and the center point of the real box γ is the adjustment parameter, is the x-axis coordinate of the center of the real frame, is the x-axis coordinate of the center of the prediction box, is the y-axis coordinate of the center of the real frame, is the y-axis coordinate of the center of the prediction box, c h is the height difference between the center point of the real box and the predicted box, c w is the width difference between the center point of the real box and the predicted box, γ = 2-Λ, Λ is the angle loss σ is the distance between the center point of the real box and the predicted box, c h is the height difference between the center point of the real box and the predicted box;

[0093] The shape loss is as follows:

[0094]

[0095] Among them, t is the variable parameter, ω w is the normalized difference between the predicted box width and the true box width, ω h is the normalized difference between the predicted box height and the true box height, θ is used to control the degree of attention paid to shape loss, w represents the width of the prediction box, and h represents the height of the prediction box; w gt is the height of the real frame, h gt is the actual box width.

[0096] S3, inputting the value-added tax invoice image data into the value-added tax invoice image content segmentation model to obtain the value-added tax invoice image;

[0097] S4. Process the VAT invoice image to obtain accurate VAT invoice content and achieve VAT invoice image content segmentation.

[0098] The VAT invoice image is processed to obtain accurate VAT invoice content, specifically:

[0099] Extract the content information of the VAT invoice image and crop it to obtain the images of each field;

[0100] The cropped images of each field are sent to the text recognition model for recognition to obtain the text content, and the text content is associated with its semantics in the invoice to obtain accurate VAT invoice content.

[0101] The content information of the VAT invoice image is extracted and cropped to obtain images of each field, specifically:

[0102] The Sklansky algorithm is used to solve the convex hull point set of the VAT invoice image;

[0103] The convex hull point set is processed using the rotating caliper algorithm to generate the minimum area circumscribed rectangle;

[0104] The target is accurately cropped according to the minimum bounding rectangle to obtain images of each field.

[0105] The following is a detailed description of this method:

[0106] Step 1: collect and annotate VAT invoice images in natural scenes to generate a data set;

[0107] Step 1.1, use a camera to collect the actual VAT invoice image;

[0108] Step 1.2: Use LabelImg to label the dataset and obtain the location information and semantic information of each field.

[0109] Step 2: Use data enhancement method to expand the VAT invoice image dataset obtained in step 1, and divide the dataset into training set, test set and validation set according to the proportion;

[0110] Step 2.1, perform random rotation and affine transformation on the data set obtained in step 1;

[0111] Step 2.2, adjusting the annotation box of each transformed image to ensure that the annotation box matches the transformed image;

[0112] Step 2.3, perform color gamut transformation on the randomly transformed image to adjust its hue, saturation, and brightness;

[0113] In step 2.4, the expanded data set is divided into training set, validation set, and test set in a ratio of 7:2:1.

[0114] Step 3, based on the VAT invoice image dataset obtained in step 2, the YOLOv8 model is improved to obtain an improved YOLOv8-Seg model, and the improved YOLOv8-Seg model is trained;

[0115] The improved YOLOv8-Seg model is as follows:

[0116] Based on the YOLOv8-Seg model, the P2 layer features are introduced for small target detection. The P2 layer features can capture more detailed information and improve the detection ability of small targets (such as small text or borders) in VAT invoices, thus obtaining the YOLOv8-Seg-P2 model;

[0117] Based on the YOLOv8-Seg-P2 model, the C2f_DCN module is used to replace the C2f module in Backbone. The C2f_DCN module combines the high efficiency of C2f and the adaptive feature extraction capability of the deformable convolutional network (DCN), enhancing the model's feature expression capability, especially for the processing of complex structures of VAT invoices, thus obtaining the YOLOv8-Seg-P2-C2f_DCN model;

[0118] Based on the YOLOv8-Seg-P2-C2f_DCN model, the global attention module (GAM) is introduced into the Neck network. The GAM module further improves the recognition accuracy of the model for key content in the VAT invoice image by introducing global context information, especially improving the segmentation effect of the invoice content under complex backgrounds, thus obtaining the YOLOv8-Seg-P2-C2f_DCN-GAM model;

[0119] Based on the YOLOv8-Seg-P2-C2f_DCN-GAM model, the SIoU loss function is used to replace the original CIoU loss function. SIoU (Spherical IoU) can more accurately calculate the difference between bounding boxes, further improve the bounding box regression accuracy of the invoice content, and finally obtain the YOLOv8-Seg-P2-C2f_DCN-GAM-SIoU model, which is the VAT invoice image content segmentation model.

[0120] like Figure 2The Backbone layer shown is the basic part of the model, which is used to extract image features. It consists of a series of convolutional layers, pooling layers, and residual modules. Neck is the middle part of the model, which is responsible for fusing and processing the features extracted by Backbone to generate a more powerful feature representation. It consists of a series of upsampling layers, convolutional layers, and splicing layers. Head is the output part of the model, which is responsible for converting the features generated by Neck into segmentation results. It consists of a series of convolutional layers and upsampling layers. The Head part shown in the figure uses a decoder structure to fuse feature maps of different scales and finally generate a segmentation result of the same size as the input image. In this embodiment, the P2 layer features are introduced as a small target detection layer. The resolution of the P2 layer detection head is 160×160, which is equivalent to performing two downsampling operations in the Backbone network, retaining richer underlying feature information. This method fuses the top-down and bottom-up P2 layer features in the Neck network through the Concat operation, and combines them with the output of the same scale features in the Backbone network, so that the P2 layer detection head is more efficient in processing subtle content. In Backbone, the deformable convolution Dconv is replaced by a new convolution operation, which can effectively handle deformation and distortion in the image, thereby improving the robustness of the model. The GAM attention mechanism module is introduced at the neck, which can effectively capture the global and local features in the image, thereby improving the segmentation accuracy of the model.

[0121] like Figure 3 The C2f_DCN shown includes multiple DCN_Bottleneck modules, a Concat module, and a Conv module. DCN_Bottleneck is a residual module that uses deformable convolution to extract image features. In this embodiment, the input feature map is first processed by a convolution operation to calculate the position offset, and then the calculated position offset is superimposed with the original position of the convolution kernel to obtain the final sampling position of the convolution kernel on the input feature map. Next, sampling is performed on the input feature map according to the superimposed position to obtain the sampled feature map. Finally, the sampled feature map is element-by-element multiplied by the convolution kernel, and a sum operation is performed to obtain the final convolution result. The output feature map is generated by the convolution operation between the kernel w and the input x, which can be expressed as y=w·x, and its value can be expressed as:

[0122]

[0123] Among them, p0 is the center position of the sample in the input; p k is the point in the enumerated sampling / offset point set K; Δp kThe C2f_DCN module adopts the DCN_Bottleneck structure, which contains two layers of deformable convolutional layers, which are connected in sequence and then superimposed.

[0124] like Figure 4 As shown, in this embodiment, the GAM attention mechanism can amplify cross-dimensional interactions while reducing diffuse information, and simultaneously obtain features of three dimensions, thereby avoiding information loss and improving the accuracy of detection. Given a feature map F1, after two independent attention submodules, the original feature map is first corrected by the channel attention mechanism module to obtain the intermediate state F2, and then corrected again by the spatial attention mechanism module to obtain the final feature map F3.

[0125]

[0126] Among them: F1 is the input feature map; M c and M s are channel and spatial attention feature maps respectively, is the matrix dot product. After channel attention, the feature obtained is F2, and the final output feature is F3. Using GAM can effectively improve the model's sensitivity to the target and make it focus on more effective features.

[0127] In this embodiment, CIoU performs well in most cases, but there may still be some limitations in dealing with extreme cases. For example, when the invoice size is small or the invoice is seriously misplaced in the image, the instability of the CIoU loss function in these cases affects the accuracy of the detection results. To address this problem, SIoU is used as the regression loss function of the detection box. SIoU takes into account the vector angle between the required regressions and redefines the penalty indicator. The SIoU loss function consists of distance, shape, and IoU.

[0128] The SIoU loss function is as follows:

[0129]

[0130] Among them, L IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss;

[0131] The distance loss between the predicted box and the true box is as follows:

[0132]

[0133] Among them, t is the variable parameter, ρ x is the normalized squared difference between the horizontal coordinates of the center point of the predicted box and the center point of the true box ρ yRepresents the normalized squared difference between the horizontal and vertical coordinates of the center point of the predicted box and the center point of the real box γ is the adjustment parameter, is the x-axis coordinate of the center of the real frame, b cx is the x-axis coordinate of the center of the prediction box, is the y-axis coordinate of the center of the real frame, is the y-axis coordinate of the center of the prediction box, c h is the height difference between the center point of the real box and the predicted box, c w is the width difference between the center point of the real box and the predicted box, γ = 2-Λ, Λ is the angle loss σ is the distance between the center point of the real box and the predicted box, c h is the height difference between the center point of the real box and the predicted box;

[0134] The shape loss is as follows:

[0135]

[0136] Among them, t is the variable parameter, ω w is the normalized difference between the predicted box width and the true box width, ω h is the normalized difference between the predicted box height and the true box height, θ is used to control the degree of attention paid to shape loss, w represents the width of the prediction box, and h represents the height of the prediction box; w gt is the height of the real frame, h gt is the actual box width.

[0137] Step 3.1, based on the improved YOLOv8-Seg model, set its hyperparameters and perform training iterations;

[0138] Step 3.2, the optimal model is obtained based on the performance evaluation index;

[0139] In step 3.3, the optimal model is used to detect the images in the test set, and the text object mask with the highest confidence is output to complete the detection of the invoice image.

[0140] Step 4: Use the trained model to extract the content information of the invoice image and crop it;

[0141] Step 4.1, using the Sklansky algorithm to solve the convex hull point set of the target image;

[0142] Step 4.2, using the convex hull of the field boundary, further use the rotating caliper algorithm to generate the minimum area circumscribed rectangle;

[0143] Step 4.3, using the obtained minimum bounding rectangle, including the center point, width, height and rotation angle, to accurately crop the target and obtain each field image.

[0144] Step 5: Based on the cropped image, the image is fed into a text recognition model for recognition, and the text content of the field is associated with its semantics in the invoice.

[0145] Step 5.1: Build the SVRT text recognition model based on Baidu PaddleOCR text recognition open source framework;

[0146] Step 5.2: For the cropped field image obtained in step 4, use the SVTR text recognition model to recognize it and obtain the text content in the specific image;

[0147] Step 5.3, for step 5.2, post-process the recognition results, such as deleting some interference items, merging the entries of the detailed items, and associating the text content of the field with its semantics in the invoice.

[0148] The method proposed in the present invention combines instance segmentation with text recognition, and by optimizing the algorithm, the recognition process can be further simplified and the recognition efficiency can be improved. By introducing the P2 layer features as the small target detection layer, the feature fusion ability of the network is enhanced, the feature information at different scales is better utilized, the detection accuracy of small-sized text is improved, and more key positioning information is provided, thereby reducing the error of target positioning. The C2f_DCN module is introduced to enhance the model's perception of targets of different sizes and shapes in the image, and can better adapt to various changes in invoice images, so that the model has stronger adaptability and generalization ability when processing complex invoice scenes. The global attention GAM module is introduced in the Neck network to enhance the feature extraction ability of the model, and the SIoU loss is introduced as the bounding box loss function to solve the problem of low accuracy in CIoU calculation, and improve the accuracy and stability of the model for target positioning; based on the Baidu PaddleOCR text recognition open source framework, the SVRT text recognition model is constructed; the recognition process is effectively simplified and the recognition efficiency is increased.

[0149] like Figure 5 As shown in the figure, this is the result of instance segmentation of the original invoice image. Each key information area on the invoice (such as name, address, invoice number, amount, etc.) will be surrounded by a bounding box. Each bounding box will be marked with a category label, such as "name", "address", "invoice number", "amount", etc. Each bounding box will display a confidence score, indicating the degree of recognition of the model for this bounding box.

[0150] like Figure 6As shown in the figure, the segmentation mask is obtained after instance segmentation, each field area is identified, and each field is cropped separately. Each field contains digital and text information and has a file name, such as "HeJiShuiE.jpg", "KaiPiaoRen.jpg", etc.

[0151] like Figure 7 As shown in the figure, Baidu PaddleOCR text recognition open source framework is used to recognize the invoice field for the cropped content. The recognition result includes the file name, the recognized text content and the recognition confidence. For example, for the field of the file name "HeJiShuiE_.jpg", the recognized text content is "7.86" and the confidence is 0.9936657. It takes a total of 0.037 seconds to recognize 22 cropped images.

[0152] Example 2

[0153] The present invention proposes a VAT invoice image content segmentation system based on YOLOv8-Seg, such as Figure 8 As shown, including:

[0154] A first data acquisition module, the first data acquisition module is used to acquire value-added tax invoice image data;

[0155] A model building module, wherein the model building module is used to build a value-added tax invoice image content detection model based on a YOLOv8-Seg model;

[0156] A second data acquisition module, the second data acquisition module is used to input the value-added tax invoice image data into the value-added tax invoice image content segmentation model to obtain the value-added tax invoice image;

[0157] The data processing module is used to process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

[0158] Example 3

[0159] See also Fig. 9 As shown, the present invention also provides an electronic device 100 for a VAT invoice content segmentation method based on YOLOv8-Seg; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0160] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the VAT invoice content segmentation method based on YOLOv8-Seg described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0161] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0162] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for segmenting the content of a value-added tax invoice based on YOLOv8-Seg, and the processor 102 can execute the plurality of instructions to implement:

[0163] Get VAT invoice image data;

[0164] Build a VAT invoice image content detection model based on the YOLOv8-Seg model;

[0165] Inputting the VAT invoice image data into the VAT invoice image content segmentation model to obtain the VAT invoice image;

[0166] Process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

[0167] Example 4

[0168] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0169] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for content segmentation of value-added tax invoice images based on YOLOv8-Seg, characterized in that: include: Get VAT invoice image data; Build a VAT invoice image content detection model based on the YOLOv8-Seg model; Inputting the VAT invoice image data into the VAT invoice image content segmentation model to obtain the VAT invoice image; Process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

2. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 1, characterized in that: The VAT invoice image content detection model is constructed based on the YOLOv8-Seg model, specifically: Based on the YOLOv8-Seg model, the P2 layer features are introduced for small target detection to obtain the YOLOv8-Seg-P2 model; Based on the YOLOv8-Seg-P2 model, the C2f_DCN module is used to replace the C2f module in Backbone to obtain the YOLOv8-Seg-P2-C2f_DCN model; Based on the YOLOv8-Seg-P2-C2f_DCN model, the global attention module GAM is introduced into the Neck network to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM model; Based on the YOLOv8-Seg-P2-C2f_DCN-GAM model, the SIoU loss function is used to replace the original CIoU loss function to obtain the YOLOv8-Seg-P2-C2f_DCN-GAM-SIoU model, which is the VAT invoice image content segmentation model.

3. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 2, characterized in that: The C2f_DCN module is specifically: The C2f_DCN module adopts a DCN_Bottleneck structure, which includes two layers of deformable convolutional layers, which are connected in sequence and then superimposed. Specifically: First, the input feature map is processed by the convolution operation to calculate the position offset; the calculated position offset is superimposed with the original position of the convolution kernel to obtain the final sampling position of the convolution kernel on the input feature map; sampling is performed on the input feature map according to the superimposed position to obtain the sampled feature map; the sampled feature map is element-wise multiplied with the convolution kernel and summed to obtain the final convolution result; the final convolution result is the output feature map, which is expressed as follows: Among them, p0 is the center position of the sample in the input; p k is the point in the enumerated sampling / offset point set K; Δp k The corresponding offset is x(p0+p k +Δp k ) means taking out the pixel value at that position, w k Represents the convolution kernel relative to (p0+p k +Δp k )The weight value of the pixel at the position.

4. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 2, characterized in that: The specific implementation of the global attention GAM module is as follows: Given a feature map F1, perform global average pooling and global maximum pooling operations on the input feature map F1 to capture global channel-level information; process global information through a shared multi-layer perceptron MLP structure, and generate attention weights for each channel through an activation function; multiply the generated channel weights by the input feature map F1 channel by channel to obtain the corrected feature map F2; perform global average pooling and global maximum pooling on the corrected feature map F2 to generate two single-channel feature maps; fuse the features of the two single-channel feature maps through convolution operations, and apply activation functions to generate a spatial attention map; multiply the generated spatial attention map by element-by-element with the input intermediate feature map F2 to obtain the final corrected feature map F3; Among them: F1 is the input feature map; M c and M s are channel and spatial attention feature maps respectively, is the matrix dot product.

5. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 2, characterized in that: The SIoU loss function includes distance loss, shape loss and intersection-over-union loss; The SIoU loss function is as follows: Among them, L IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss; The distance loss between the predicted box and the true box is as follows: Among them, t is the variable parameter, ρ x is the normalized squared difference between the horizontal coordinates of the center point of the predicted box and the center point of the true box ρ y Represents the normalized squared difference between the horizontal and vertical coordinates of the center point of the predicted box and the center point of the real box γ is the adjustment parameter, is the x-axis coordinate of the center of the real frame, is the x-axis coordinate of the center of the prediction box, is the y-axis coordinate of the center of the real frame, is the y-axis coordinate of the center of the prediction box, c h is the height difference between the center point of the real box and the predicted box, c w is the width difference between the center point of the real box and the predicted box, γ = 2-Λ, Λ is the angle loss σ is the distance between the center point of the real box and the predicted box, c h is the height difference between the center point of the real box and the predicted box; The shape loss is as follows: Among them, t is the variable parameter, ω w is the normalized difference between the predicted box width and the true box width, ω h is the normalized difference between the predicted box height and the true box height, θ is used to control the degree of attention paid to shape loss, w represents the width of the prediction box, and h represents the height of the prediction box; w gt is the height of the real frame, h gt is the actual box width.

6. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 1, characterized in that: The VAT invoice image is processed to obtain accurate VAT invoice content, specifically: Extract the content information of the VAT invoice image and crop it to obtain the images of each field; The cropped images of each field are sent to the text recognition model for recognition to obtain the text content, and the text content is associated with its semantics in the invoice to obtain accurate VAT invoice content.

7. The method for content segmentation of value-added tax invoice images based on YOLOv8-Seg according to claim 6, characterized in that: The content information of the VAT invoice image is extracted and cropped to obtain images of each field, specifically: The Sklansky algorithm is used to solve the convex hull point set of the VAT invoice image; The convex hull point set is processed using the rotating caliper algorithm to generate the minimum area circumscribed rectangle; The target is accurately cropped according to the minimum bounding rectangle to obtain images of each field.

8. A VAT invoice image content segmentation system based on YOLOv8-Seg, characterized in that: include: A first data acquisition module, the first data acquisition module is used to acquire value-added tax invoice image data; A model building module, wherein the model building module is used to build a value-added tax invoice image content detection model based on a YOLOv8-Seg model; A second data acquisition module, the second data acquisition module is used to input the value-added tax invoice image data into the value-added tax invoice image content segmentation model to obtain the value-added tax invoice image; The data processing module is used to process the VAT invoice image, obtain accurate VAT invoice content, and realize VAT invoice image content segmentation.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for content segmentation of a value-added tax invoice image based on YOLOv8-Seg as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for content segmentation of a value-added tax invoice image based on YOLOv8-Seg as described in any one of claims 1 to 7 are implemented.

Images