A method, apparatus, device, medium and product for classifying a bill image

Abstract

This invention discloses a method, apparatus, device, medium, and product for classifying invoice images. It can be applied to the financial technology field. The method includes: acquiring an image of an invoice to be processed, and segmenting the image to obtain an invoice title image block, an invoice wireframe image block, and an invoice layout image block; inputting the invoice title image block into a pre-constructed invoice classification model to obtain a first classification probability, inputting the invoice wireframe image block into the invoice classification model to obtain a second classification probability, and inputting the invoice layout image block into the invoice classification model to obtain a third classification probability; determining a first weight parameter, a second weight parameter, and a third weight parameter of the invoice classification model; and determining the invoice category of the invoice image to be processed based on the first, second, and third classification probabilities, and the first, second, and third weight parameters.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and in particular to a method, apparatus, device, medium and product for classifying invoice images. Background Technology

[0002] As a core payment and settlement voucher and accounting basis in financial activities, negotiable instruments cover a variety of types, including checks, drafts, commercial acceptance drafts, bank acceptance drafts, and invoices. Their processing efficiency and accuracy are directly related to the operating costs, risk control level, and customer service quality of financial institutions.

[0003] In existing financial sector methods for classifying bills, deep learning models are commonly used for prediction. However, directly inputting the entire image into the model can lead to the neglect of key features and make it difficult to accurately extract the core features of the bill image, resulting in low accuracy in bill image classification. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for classifying invoice images, in order to improve the accuracy of invoice image classification.

[0005] According to one aspect of the present invention, a method for classifying invoice images is provided, the method comprising:

[0006] The image of the invoice to be processed is obtained, and the image is segmented to obtain the invoice title image block, the invoice wireframe image block, and the invoice layout image block respectively.

[0007] The image block of the bill title is input into the first network branch of the pre-built bill classification model for image processing to obtain the first classification probability output by the first network branch. The image block of the bill wireframe is input into the second network branch of the bill classification model for image processing to obtain the second classification probability output by the second network branch. The image block of the bill layout is input into the third network branch of the bill classification model for image processing to obtain the third classification probability output by the third network branch.

[0008] Determine the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model.

[0009] Based on the first classification probability, the second classification probability, and the third classification probability, and using the first weight parameter, the document category of the document image to be processed is determined.

[0010] According to another aspect of the present invention, a ticket image classification apparatus is provided, the apparatus comprising:

[0011] The invoice image segmentation module acquires the invoice image to be processed and performs image segmentation on the invoice image to obtain the invoice title image block, the invoice wireframe image block, and the invoice layout image block respectively;

[0012] The classification probability acquisition module inputs the ticket title image block into the first network branch of the pre-built ticket classification model for image processing to obtain the first classification probability output by the first network branch; inputs the ticket wireframe image block into the second network branch of the ticket classification model for image processing to obtain the second classification probability output by the second network branch; and inputs the ticket layout image block into the third network branch of the ticket classification model for image processing to obtain the third classification probability output by the third network branch.

[0013] The weight parameter determination module determines the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model.

[0014] The bill category confirmation module determines the bill category of the bill image to be processed based on the first classification probability, the second classification probability, and the third classification probability, and on the first weight parameter, the second weight parameter, and the third weight parameter.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the ticket image classification method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the ticket image classification method of any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the ticket image classification method of any embodiment of the present invention.

[0021] The technical solution of this invention involves acquiring an image of a ticket to be processed and segmenting it to obtain a ticket title image block, a ticket wireframe image block, and a ticket layout image block. The ticket title image block is input into a first network branch of a pre-constructed ticket classification model for image processing to obtain a first classification probability output by the first network branch. Similarly, the ticket wireframe image block is input into a second network branch of the same model for image processing to obtain a second classification probability output by the second network branch, and the ticket layout image block is input into a third network branch for image processing to obtain a third classification probability output by the third network branch. A first weight parameter corresponding to the first network branch, a second weight parameter corresponding to the second network branch, and a third weight parameter corresponding to the third network branch of the ticket classification model are determined. Based on the first, second, and third classification probabilities, and using the first, second, and third weight parameters, the ticket category of the ticket image to be processed is determined. This technical solution extracts the image blocks of the invoice title, the invoice wireframe, and the invoice layout separately. It then uses the three network branches of the invoice classification model to output the classification probability, and combines the corresponding weight parameters for comprehensive calculation. This can accurately integrate the classification value of different structural features of the invoice, effectively improve the recognition accuracy of the invoice category, reduce the error caused by judging the invoice type by a single feature, and achieve more reliable invoice classification results.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A This is a flowchart of a ticket image classification method provided in Embodiment 1 of the present invention;

[0025] Figure 1B This is a model structure diagram of a bill classification model provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a ticket image classification method provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a flowchart of a ticket image classification method provided in Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a ticket image classification device according to Embodiment 4 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the ticket image classification method of this invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1A This is a flowchart of a bill image classification method provided in Embodiment 1 of the present invention. This embodiment is applicable to the classification of different types of bill images in financial scenarios. The method can be executed by a bill image classification device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes:

[0034] S101. Obtain the image of the invoice to be processed, and perform image segmentation on the image of the invoice to be processed to obtain the invoice title image block, the invoice wireframe image block and the invoice layout image block respectively.

[0035] S102. Input the bill title image block into the first network branch of the pre-constructed bill classification model for image processing to obtain the first classification probability output by the first network branch. Input the bill wireframe image block into the second network branch of the bill classification model for image processing to obtain the second classification probability output by the second network branch. Input the bill layout image block into the third network branch of the bill classification model for image processing to obtain the third classification probability output by the third network branch.

[0036] S103. Determine the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model.

[0037] S104. Based on the first classification probability, the second classification probability, and the third classification probability, and based on the first weight parameter, the second weight parameter, and the third weight parameter, determine the ticket category of the ticket image to be processed.

[0038] The image of the invoice to be processed can be an image of an invoice that needs to be classified, such as a value-added tax invoice, a general invoice, or a receipt. The invoice title image block can be an image area containing the invoice name, the invoice wireframe image block can be an image area of ​​an invoice consisting of borders and dividing lines, and the invoice layout image block can be an image area containing the complete layout of the invoice.

[0039] For example, the title portion, wireframe structure, and entire layout content of the ticket can be located by scanning the image of the ticket to be processed. These areas can then be cropped to obtain three independent image blocks: the ticket title image block, the ticket wireframe image block, and the ticket layout image block.

[0040] The pre-built ticket classification model can be a neural network model pre-trained by relevant technicians to classify ticket images. For example, the ticket classification model can be a multi-branch feature convolutional neural network model structure. The first network branch of the ticket classification model can be a functional branch in the pre-built ticket classification model that specifically processes ticket title image blocks. The first classification probability can be the probability of the ticket category corresponding to the ticket title image block output by the first network branch. The second network branch of the ticket classification model can be a functional branch in the pre-built ticket classification model that specifically processes ticket wireframe image blocks, and the second classification probability can be the probability of the ticket category corresponding to the ticket wireframe image block output by the second network branch. The third network branch of the ticket classification model can be a functional branch in the pre-built ticket classification model that specifically processes ticket layout image blocks, and the third classification probability can be the probability of the ticket category corresponding to the ticket layout image block output by the third network branch.

[0041] In an optional embodiment, the training process of the multi-branch feature convolutional neural network model is as follows: Historical ticket images corresponding to historical tickets are acquired; the historical ticket images are preprocessed to extract historical ticket title image blocks, historical ticket wireframe image blocks, and historical ticket layout image blocks; these image blocks are used as sample data, and labels for the real ticket categories are generated for the sample data; the sample data and their corresponding labels are input into a preset multi-branch feature convolutional neural network model; the first network branch of the model performs feature extraction and classification on the historical ticket title image blocks to obtain a first predicted classification. The probability is calculated by extracting and classifying features from historical ticket wireframe image blocks through the second network branch to obtain the second predicted classification probability, and by extracting and classifying features from historical ticket layout image blocks through the third network branch to obtain the third predicted classification probability. The first, second, and third predicted classification probabilities are then fused to obtain the final predicted ticket category output by the model. Based on the actual ticket category and the final predicted ticket category in the sample data, the model loss value is calculated. The network parameters of the multi-branch feature convolutional neural network model are iteratively updated based on the loss value until the preset model training termination condition is met, thus obtaining the pre-constructed ticket classification model.

[0042] The labels for the actual document categories can be pre-labeled by relevant technical personnel. Examples include invoices, receipts, drafts, and checks. Different types of document categories can be labeled using numerical codes or vector representations; this embodiment does not impose any restrictions on this. The model training termination condition can be pre-set by relevant technical personnel. For example, the termination condition could be that the number of model training iterations reaches a set iteration threshold, or that the change in the model loss value for consecutive preset rounds is less than a set threshold. This embodiment does not impose any restrictions on this.

[0043] For example, if the invoice classification model is a multi-branch feature convolutional neural network model structure, the first network branch can have a convolutional layer and a fully connected layer. The convolutional layer is used to extract local features of the data, and the fully connected layer is used to output the probability of invoice classification. Inputting the invoice title image patch into the first network branch of the invoice classification model yields the first classification probability corresponding to the invoice title image patch. For example, inputting the invoice title image patch into the first network branch extracts the features of the invoice title image patch, and outputs the first classification probability as: VAT special invoice 0.85, ordinary invoice 0.1, receipt 0.05. The second network branch can have a convolutional layer and a fully connected layer. Inputting the invoice wireframe image patch into the second network branch of the invoice classification model yields the invoice classification result corresponding to the invoice wireframe image patch. For example, inputting the invoice wireframe image patch into the second network branch extracts the features of the invoice wireframe image patch, and outputs the second classification probability as: VAT special invoice 0.9, ordinary invoice 0.05, receipt 0.05. The third network branch can consist of a convolutional layer and a fully connected layer. Inputting a block of the invoice image into the third network branch of the invoice classification model yields the corresponding invoice classification result. For example, when a block of the invoice image is input into the third network branch, its features are extracted, and the output third classification probability is: VAT special invoice 0.8, ordinary invoice 0.1, receipt 0.1.

[0044] Furthermore, to more accurately illustrate the network structure of the bill classification model and increase the accuracy of bill classification, in an optional embodiment, the first, second, and third network branches of the bill classification model have the same network structure. These three branches respectively include a feature extraction module, a channel attention module, a spatial attention module, and a classification module. For example, the model structure of the bill classification model is as follows: Figure 1B As shown, the first, second, and third network structure branches of the ticket classification model can be composed of three convolutional layers, one pooling layer, a channel attention layer, a spatial attention layer, a fully connected layer, and a softmax layer.

[0045] The above technical solution extracts multi-dimensional features from document images in parallel through three network branches with the same structure. Combined with channel attention and spatial attention modules, it can accurately capture differences in key fields and core regions. Ultimately, it achieves high-precision and robust document classification results.

[0046] Furthermore, in order to further capture the core discriminative information of the invoice title, strengthen key features, and improve the classification prediction accuracy of the invoice title image patch for invoice type, in an optional embodiment, the invoice title image patch is input into the first network branch of a pre-built invoice classification model for image processing to obtain the first classification probability output by the first network branch, including:

[0047] Step a1: Input the bill title image block into the feature extraction module in the first network branch for feature extraction to obtain the bill title feature map.

[0048] Step a2: Input the ticket title feature map into the channel attention module to enhance the channel features and obtain the channel enhanced feature map.

[0049] Step a3: Input the channel enhancement feature map into the spatial attention module to enhance the region features and obtain the region enhancement feature map.

[0050] Step a4: Input the region enhancement feature map into the classification module to predict the classification probability and obtain the first classification probability.

[0051] Among them, the document title feature map can be a fixed-size feature map obtained by performing convolution operation on the document title image block by the feature extraction module.

[0052] For example, the feature extraction module can be a module consisting of three convolutional layers and one pooling layer. For instance, if the feature extraction module consists of three convolutional layers and one pooling layer, since the network structures of the first, second, and third network branches of the ticket classification model are the same, taking the input of the ticket title image block into the first network branch as an example, the function and principle of the convolutional layers (convolutional layer 1, convolutional layer 2, convolutional layer 3) are as follows: The input ticket title image block is input into the convolutional layer and undergoes the following convolution operation with a 5x5 convolutional kernel:

[0053]

[0054] in, and They represent the first The first layer The first feature map and the second The first layer Each feature map; It is a connection and The convolution kernel; It is a bias term; It is a non-linear activation function.

[0055] If convolutional layer 1 has 8 kernels, convolutional layer 2 has 16 kernels, and convolutional layer 3 has 32 kernels, then inputting a 20×20 image block of the ticket title into convolutional layer 1 yields a 16×16×8 feature map. This 16×16×8 feature map is then input into convolutional layer 2 to obtain a 12×12×16 feature map, and finally into convolutional layer 3 to obtain an 8×8×32 feature map. Inputting this 8×8×32 feature map into a pooling layer results in an 8×8×32 feature map of the ticket title. The pooling layer does not alter the specific shape features; it extracts significant features to prevent overfitting.

[0056] The channel enhancement feature map can be a channel feature map of the same size obtained by assigning channel weights to the ticket title feature map using the channel attention module. The channel attention module can be a module composed of channel attention layers.

[0057] For example, the channel attention module can compress the spatial dimension using average pooling and max pooling to generate channel weights, as shown in the following formula:

[0058]

[0059] Where Mc(X) represents the final output fused global feature, i.e., the channel-enhanced feature map; σ() represents the activation function, which can be the ReLU activation function; MLP() represents a multilayer perceptron; Avgpool(X) represents global average pooling on the input feature map X to obtain the global average feature; and StdPool(X) represents global standard deviation pooling on the input feature map X to obtain the global standard deviation feature. The feature map X is the ticket title feature map output by the feature extraction module.

[0060] The region-enhanced feature map can be a spatially weighted feature map of the same size obtained by assigning spatial region weights to the channel-enhanced feature map using the spatial attention module. The spatial attention module can be a module composed of spatial attention layers.

[0061] For example, the spatial attention module can receive channel-weighted features, generate spatial weights, and highlight key regions:

[0062]

[0063] M s (X´) represents the final output pooled and fused features, i.e., the region augmentation feature map, f 3×3This represents a 3×3 convolution operation. MaxPool(X´) represents the max pooling process, AvgPool(X´) represents the average pooling process, and StdPool(X´) represents the standard pooling process. It's equivalent to performing a 3×3 convolution operation on the max pooling, average pooling, and standard pooling processes. Here, X´ is the channel-enhanced feature map output by the channel attention module.

[0064] The classification module can be a module consisting of a fully connected layer and a softmax layer.

[0065] For example, a fully connected layer can be a layer into which the enhanced feature map is input:

[0066]

[0067] Among them, X f This is the output of the f-th layer. Represents the bias term. X represents the weight matrix of the f-th layer. f-1 This represents the output of the previous layer. The fully connected layer flattens the feature map into neuron vectors. Then with the weight matrix Multiply element by element and add a bias term. Finally, a nonlinear activation function is applied. Output the first Layer neuron vectors .

[0068] The softmax layer can be the neuron vector output by the fully connected layer. The input is fed into the softmax layer, which can then predict each category. The probability distribution is given, and the softmax layer is defined as follows:

[0069]

[0070] in, This indicates the total number of categories of invoices. For example, if the pre-defined invoice categories include VAT invoices, general invoices, and receipts, then the value of C is 3. The first neuron vector represents the first neuron vector. Each element. Perform an exponential operation on the score corresponding to category c in the output of layer f. The softmax layer sums the probabilities of all categories (C categories in total, from c=0 to c=C-1). In other words, the softmax layer calculates the probability of each category within the predefined invoice categories and outputs the results. For example, if the predefined invoice categories include VAT invoices, general invoices, and receipts, the first category probability output by the softmax layer would be: VAT invoice 0.8, general invoice 0.1, and receipt 0.1.

[0071] The above technical solution inputs the bill title image block into the first network branch, and after the key features are enhanced by the feature extraction module, the channel attention module and the spatial attention module, the classification module performs classification probability prediction, which strengthens the judgment of core information by the bill title image block and effectively improves the prediction accuracy of the type classification of the bill title image block.

[0072] Similarly, based on the same method used to determine the first classification probability of the document title image block as described above, the document wireframe image block is input into the second network branch of the document classification model for image processing to obtain the second classification probability output by the second network branch. Likewise, the document layout image block is input into the third network branch of the document classification model for image processing to obtain the third classification probability output by the third network branch. The branch structures of the second and third network branches are the same as those of the first network branch, and the image block processing methods are also the same; therefore, this embodiment will not elaborate further.

[0073] The first weight parameter can be the weight coefficient corresponding to the first classification probability output by the first network branch, the second weight parameter can be the weight coefficient corresponding to the second classification probability output by the second network branch, and the third weight parameter can be the weight coefficient corresponding to the third classification probability output by the third network branch.

[0074] For example, the first weight parameter, the second weight parameter, and the third weight parameter can be manually set by relevant technical personnel according to actual needs. For instance, the first weight parameter in the first network branch is set to 0.3, the second weight parameter in the second network branch is set to 0.3, and the third weight parameter in the third network branch is set to 0.4.

[0075] The document category of the document image to be processed can be the specific document category of the document image. The first classification probability, the second classification probability, and the third classification probability each include the probability corresponding to different document types. For example, the first classification probability includes the classification probability 'a' for document type A, the classification probability 'b' for document type B, and the classification probability 'c' for document type C.

[0076] For example, the classification probability of different bill categories can be obtained by multiplying the first classification probability by the first weight coefficient, the classification probability of different bill categories can be obtained by multiplying the second classification probability by the second weight coefficient, and the classification probability of different bill categories can be obtained by multiplying the third classification probability by the third weight coefficient. The classification probabilities of the same bill category among the three classification probabilities are added together, and the bill category with the highest classification probability is the bill category of the bill image to be processed.

[0077] For example, if the first classification probability includes a probability of 0.85 for VAT special invoices, 0.1 for ordinary invoices, and 0.05 for receipts; the second classification probability includes a probability of 0.9 for VAT special invoices, 0.05 for ordinary invoices, and 0.05 for receipts; and the third classification probability includes a probability of 0.8 for VAT special invoices, 0.1 for ordinary invoices, and 0.1 for receipts. Assuming the first weight parameter is 0.3, the second weight parameter is 0.3, and the third weight parameter is 0.4, then the probability corresponding to a VAT special invoice is: 0.85×0.3 + 0.9×0.3 + 0.8×0.4 = 0.845. The probability corresponding to an ordinary invoice is 0.1×0.3 + 0.05×0.3 + 0.1×0.4 = 0.085. The probability corresponding to the receipt is 0.05×0.3+0.05×0.3+0.1×0.4=0.07. Therefore, the receipt type of the image to be processed is a value-added tax (VAT) special invoice.

[0078] Furthermore, to improve the accuracy of generating weight parameters for different branches, this embodiment also provides a dynamic generation method for weight parameters. In an optional embodiment, the above method further includes:

[0079] Obtain the first historical classification probability corresponding to the first network branch, the second historical classification probability corresponding to the second network branch, and the third historical classification probability corresponding to the third network branch of the bill classification model under the historical time period, and obtain the historical bill categories output by the bill classification model;

[0080] Based on the first historical classification probability and historical bill category, determine the first prediction accuracy of the first network branch, and based on the second historical classification probability and historical bill category, determine the second prediction accuracy of the second network branch, and based on the third historical classification probability and historical bill category, determine the third prediction accuracy of the third network branch.

[0081] Based on the first prediction accuracy, the second prediction accuracy, and the third prediction accuracy, update the first weight parameter corresponding to the first network branch, the second weight parameter corresponding to the second network branch, and the third weight parameter corresponding to the third network branch.

[0082] Among them, the first historical classification probability can be the probability distribution of the first network branch of the bill classification model predicting the historical bill sample as a category, the second historical classification probability can be the probability distribution of the second network branch predicting the same historical bill sample as a category, the third historical classification probability can be the probability distribution of the third network branch predicting the same historical bill sample as a category, and the historical bill category can be the true category of the historical bill sample.

[0083] The first prediction accuracy rate can be the proportion of the number of historical bill samples whose prediction results of the first network branch are consistent with the historical bill category to the total number of historical bill samples included in the statistics; the second prediction accuracy rate can be the proportion of the number of historical bill samples whose prediction results of the second network branch are consistent with the historical bill category to the total number of historical bill samples included in the statistics; and the third prediction accuracy rate can be the proportion of the number of historical bill samples whose prediction results of the third network branch are consistent with the historical bill category to the total number of historical bill samples included in the statistics.

[0084] For example, if the first prediction accuracy is 0.9, the second prediction accuracy is 0.8, and the third prediction accuracy is 0.7, then the first weight parameter can be updated to: 0.9 / (0.9+0.8+0.7)=0.375, the second weight parameter can be updated to: 0.8 / (0.9+0.8+0.7)=0.33, and the third weight parameter can be updated to: 0.7 / (0.9+0.8+0.7)=0.29.

[0085] The above technical solution obtains the historical classification probabilities and corresponding historical bill categories of the three network branches of the bill classification model, calculates the prediction accuracy of each branch, and dynamically updates the corresponding weight parameters accordingly. This achieves precise adaptation and optimization of the contribution of each network branch, thereby improving the overall accuracy, stability and adaptability of bill classification and effectively adapting to the classification needs of bill images in different scenarios. The technical solution of this invention involves acquiring an image of a ticket to be processed and segmenting it to obtain a ticket title image block, a ticket wireframe image block, and a ticket layout image block. The ticket title image block is input into a first network branch of a pre-constructed ticket classification model for image processing to obtain a first classification probability output by the first network branch. Similarly, the ticket wireframe image block is input into a second network branch of the same model for image processing to obtain a second classification probability output by the second network branch, and the ticket layout image block is input into a third network branch for image processing to obtain a third classification probability output by the third network branch. A first weight parameter corresponding to the first network branch, a second weight parameter corresponding to the second network branch, and a third weight parameter corresponding to the third network branch of the ticket classification model are determined. Based on the first, second, and third classification probabilities, and using the first, second, and third weight parameters, the ticket category of the ticket image to be processed is determined. This technical solution extracts the image blocks of the invoice title, the invoice wireframe, and the invoice layout separately. It then uses the three network branches of the invoice classification model to output the classification probability, and combines the corresponding weight parameters for comprehensive calculation. This can accurately integrate the classification value of different structural features of the invoice, effectively improve the recognition accuracy of the invoice category, reduce the error caused by judging the invoice type by a single feature, and achieve more reliable invoice classification results.

[0086] Example 2

[0087] Figure 2 This is a flowchart of a ticket image classification method provided in Embodiment 2 of the present invention. This embodiment optimizes and improves upon the above-mentioned technical solutions. The step "acquiring the ticket image to be processed and performing image segmentation on the ticket image to obtain ticket title image blocks, ticket wireframe image blocks, and ticket layout image blocks" is refined to "performing grayscale processing on the ticket image to obtain a grayscale ticket image; performing binarization processing on the grayscale ticket image to obtain a black and white ticket image; performing denoising processing on the black and white ticket image to obtain a denoised ticket image; performing image region segmentation on the denoised ticket image to obtain several image sub-regions; and performing region image block recognition on each image sub-region to obtain ticket title image blocks, ticket wireframe image blocks, and ticket layout image blocks." This improves the image region segmentation method for the ticket image to be processed.

[0088] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments. For example... Figure 2 As shown, the method includes the following specific steps:

[0089] S201. Perform grayscale processing on the image of the ticket to be processed to obtain a grayscale ticket image.

[0090] S202. Binarize the grayscale ticket image to obtain a black and white ticket image.

[0091] S203. Denoise the black and white ticket image to obtain a denoised ticket image.

[0092] S204. Perform image region segmentation on the denoised ticket image to obtain several image sub-regions.

[0093] S205. Perform region image block recognition on each image sub-region to obtain the ticket title image block, ticket wireframe image block, and ticket layout image block.

[0094] S206. Input the bill title image block into the first network branch of the pre-constructed bill classification model for image processing to obtain the first classification probability output by the first network branch. Input the bill wireframe image block into the second network branch of the bill classification model for image processing to obtain the second classification probability output by the second network branch. Input the bill layout image block into the third network branch of the bill classification model for image processing to obtain the third classification probability output by the third network branch.

[0095] S207. Determine the first weight parameter corresponding to the first network branch of the bill classification model, determine the second weight parameter corresponding to the second network branch of the bill classification model, and determine the third weight parameter corresponding to the third network branch of the bill classification model.

[0096] S208. Based on the first classification probability, the second classification probability, and the third classification probability, and based on the first weight parameter, the second weight parameter, and the third weight parameter, determine the ticket category of the ticket image to be processed.

[0097] Among them, the grayscale ticket image can be a grayscale ticket image obtained by grayscale processing of the ticket image.

[0098] For example, a weighted average method can be used to convert the ticket image to grayscale to obtain grayscale ticket image data. The color of each pixel in the ticket image is composed of red channel values ​​(R), green channel values ​​(G), and blue channel values ​​(B). The weighted average method extracts the R, G, and B values ​​from the ticket image and assigns weights to each. These weights can be preset by relevant technical personnel according to actual needs. If the weight of R is 0.3, the weight of G is 0.59, and the weight of B is 0.11, then the grayscale value of pixel A (50, 40, 200) = 50 × 0.3 + 40 × 0.59 + 200 × 0.11 = 60.6. Grayscale conversion of each pixel in the ticket image yields the grayscale ticket image.

[0099] The black and white ticket image can be a ticket image data obtained by processing a grayscale ticket image.

[0100] For example, an adaptive thresholding method can be used to convert grayscale ticket images. Specifically, a sliding window can be used to traverse each pixel of the grayscale ticket image, and the local threshold of the region can be calculated based on the average grayscale value of the pixels within the window. Then, the grayscale value of the current pixel is compared with the corresponding local threshold. If the pixel grayscale value is lower than the local threshold, it is set to black; if it is higher than or equal to the local threshold, it is set to white. Finally, a clear black and white ticket image data is output.

[0101] To further accurately segment the grayscale invoice image and retain key visual information of the invoice to the greatest extent possible, thus providing a clear black and white invoice image for subsequent processing, in one optional embodiment, the grayscale invoice image is binarized to obtain a black and white invoice image, including:

[0102] Step b1: Analyze the grayscale values ​​of the image pixels in the grayscale ticket image to obtain the first grayscale extreme value and the second grayscale extreme value, and determine the grayscale threshold based on the first grayscale extreme value and the second grayscale extreme value.

[0103] Step b2: Determine the first grayscale ratio and the first grayscale mean based on the first grayscale extreme value and the grayscale threshold.

[0104] Step b3: Determine the second grayscale ratio and the second grayscale mean based on the second grayscale extreme value and the grayscale threshold.

[0105] Step b4: Determine the inter-class variance based on the first grayscale ratio, the second grayscale ratio, the first grayscale mean, and the second grayscale mean.

[0106] Step b5: Based on the inter-class variance and the grayscale threshold, generate a black and white ticket image corresponding to the grayscale ticket image.

[0107] The first grayscale extreme value can be the minimum value among the grayscale values, and the second grayscale extreme value can be the maximum value among the grayscale values. The grayscale threshold can be any grayscale value preset by relevant technical personnel between the first and second grayscale extreme values ​​according to actual needs.

[0108] For example, the first grayscale extreme value is 4, the second grayscale extreme value is 200, and the grayscale threshold can be set to 100.

[0109] Wherein, the first grayscale ratio can be the first grayscale mean, which can be the number of image pixels from the first grayscale extreme value to the grayscale threshold divided by the number of image pixels in the grayscale ticket image. The first grayscale mean can be the sum of the grayscale values ​​corresponding to the image pixels from the first grayscale extreme value to the grayscale threshold divided by the number of image pixels from the first grayscale extreme value to the grayscale threshold.

[0110] For example, if a grayscale ticket image contains 7 pixels with corresponding grayscale values ​​of: pixel A: 4, pixel B: 10, pixel C: 15, pixel D: 20, pixel E: 50, pixel F: 60, and pixel G: 80, then the first grayscale extreme value is 4. If the grayscale threshold is set to 30, then the image pixels from the first grayscale extreme value to the grayscale threshold are pixels A, B, C, and D, and the first grayscale ratio is 4 / 7 = 0.57. The first grayscale mean is the sum of the grayscale values ​​of pixels A, B, C, and D divided by 4, i.e., (4 + 10 + 15 + 20) / 4 = 12.25.

[0111] The second grayscale ratio can be the second grayscale mean, which can be the number of image pixels from the second grayscale extreme value to the grayscale threshold divided by the number of image pixels in the grayscale ticket image. The second grayscale mean can be the sum of the grayscale values ​​corresponding to the image pixels from the first grayscale extreme value to the grayscale threshold divided by the number of image pixels from the second grayscale extreme value to the grayscale threshold.

[0112] For example, if a grayscale ticket image contains 7 pixels with corresponding grayscale values ​​of: pixel A: 4, pixel B: 10, pixel C: 15, pixel D: 20, pixel E: 50, pixel F: 60, and pixel G: 80, then the second grayscale extreme value is 80. If the grayscale threshold is set to 30, then the image pixels from the grayscale threshold to the second grayscale extreme value are pixels E, F, and G, and the second grayscale ratio is 3 / 7 = 0.43. The second grayscale mean is the sum of the grayscale values ​​of pixels E, F, and G divided by 3, i.e., (50 + 60 + 80) / 3 = 63.33.

[0113] Among them, inter-class variance can be used as a parameter to measure the degree of brightness difference between different gray values.

[0114] For example, inter-class variance ,in The first grayscale mean value, The second grayscale mean. The first grayscale ratio, Let be the second grayscale ratio. For example, if the first grayscale ratio is 0.4, the first grayscale mean is 10, the second grayscale ratio is 0.6, and the second grayscale mean is 60, then the inter-class variance is (10-60). 2 ×0.4×0.6=600.

[0115] Among them, a black and white ticket image can be an image composed of pixels in a grayscale ticket image that have been processed to become black or white pixels.

[0116] For example, it is necessary to calculate the inter-class variance using the grayscale threshold of all pixels, find the maximum inter-class variance, and then determine the maximum inter-class variance based on the grayscale threshold corresponding to the maximum inter-class variance. Pixels with values ​​less than or equal to the grayscale threshold are set to black, and pixels with values ​​greater than the grayscale threshold are set to white, thus converting the grayscale ticket image to a black and white ticket image. If a grayscale ticket image contains 5 pixels with grayscale values ​​of: pixel A: 10, pixel B: 15, pixel C: 20, pixel D: 50, and pixel E: 60. When the grayscale threshold is 10, the inter-class variance is 110; when the grayscale threshold is 15, the inter-class variance is 228; when the grayscale threshold is 20, the inter-class variance is 384; when the grayscale threshold is 50, the inter-class variance is 210; and when the grayscale threshold is 60, the inter-class variance is 0. That is, when the grayscale threshold is 20, the inter-class variance is the maximum inter-class variance. In a grayscale image, pixels with a value of 20 or less are set to black, and those with a value greater than 20 are set to white. That is, pixels A, B, and C are black, and pixels D and E are white.

[0117] The above technical solution analyzes the grayscale values ​​of image pixels in a grayscale ticket image to determine the first and second grayscale extreme values ​​and the corresponding grayscale thresholds. It further calculates the ratio and mean of the two grayscale classes and obtains the inter-class variance. Based on the inter-class variance and the grayscale threshold, it achieves accurate conversion from a grayscale ticket image to a black and white ticket image, effectively highlighting the key information in the ticket image and improving the clarity and recognizability of the ticket image.

[0118] Among them, the denoised ticket image can be the image data obtained by removing noise from a black and white ticket image.

[0119] For example, iterate through each pixel of the black and white ticket image data (except for edge pixels), take each pixel as the center, obtain the gray values ​​of the nine pixels around the current pixel position, take the median of these nine pixel gray values ​​as the new gray value of the current pixel, and finally obtain the denoised black and white ticket image.

[0120] The image sub-regions can be defined as several sub-regions into which the image is segmented by projecting the denoised black-and-white document image horizontally and vertically using projection analysis. Projection analysis is an image segmentation technique that performs pixel distribution statistics on the denoised black-and-white document image along both horizontal and vertical directions. This method yields a projection curve with row / column coordinates as the horizontal axis and the cumulative number of black pixels in the corresponding direction as the vertical axis. The valley points are the local extrema with the minimum cumulative number of black pixels on the projection curve.

[0121] For example, a projection analysis method is applied to the denoised black and white ticket image, performing horizontal and vertical projection operations: the cumulative number of black pixels is counted row by row horizontally, and the cumulative number of black pixels is counted column by column vertically, resulting in a horizontal projection curve with row coordinates as the horizontal axis and the cumulative number of black pixels in the horizontal direction as the vertical axis, and a vertical projection curve with column coordinates as the horizontal axis and the cumulative number of black pixels in the vertical direction as the vertical axis. The local extreme point with the fewest cumulative number of black pixels in the two projection curves is the valley point. For example, the blank area between the ticket title and the page layout on the horizontal projection curve corresponds to the valley point, and the blank area between the line frame and the core content on the vertical projection curve corresponds to the valley point. Using these valley points, the image is cut along the horizontal and vertical directions, ultimately dividing the denoised black and white ticket image into several sub-regions. Among them, the area covering the complete layout of the ticket can be used as the ticket page image block, the top 1 / 5 of the image can be used as the ticket title image block, and the continuous straight line structure containing rectangular borders or separator lines can be used as the ticket line frame image block.

[0122] In this embodiment, by sequentially performing grayscale processing, binarization processing, noise reduction processing, region segmentation, and specific image block recognition on the invoice image to be processed, the image quality is gradually optimized, the image structure is split, and three core image blocks—invoice title, wireframe, and layout—are accurately extracted. This effectively removes redundant interference information and clarifies the division of key image regions, providing structured and high-quality image data support for subsequent core tasks such as invoice classification and information extraction, thereby improving the accuracy and efficiency of the overall invoice processing flow.

[0123] Example 3

[0124] Figure 3 This is a flowchart of a ticket image classification method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment provides a preferred example.

[0125] S301. Perform image block recognition on the acquired invoice image to obtain invoice title image block, invoice wireframe image block, and invoice layout image block.

[0126] The process involves acquiring the image of the invoice to be processed and performing grayscale processing to obtain a grayscale invoice image. The grayscale invoice image is then binarized to obtain a black and white invoice image. The black and white invoice image is then denoised to obtain a denoised invoice image. Finally, the denoised invoice image is segmented into several sub-regions. Each sub-region is then used for region image block recognition to obtain invoice title image blocks, invoice wireframe image blocks, and invoice layout image blocks.

[0127] S302. Input the bill title image block into the feature extraction module in the first network branch for feature extraction to obtain the bill title feature map.

[0128] S303. Input the ticket title feature map into the channel attention module to enhance the channel features and obtain the channel enhanced feature map.

[0129] S304. Input the channel enhancement feature map into the spatial attention module to enhance the region features and obtain the region enhancement feature map.

[0130] S305. Input the region enhancement feature map into the classification module to predict the classification probability and obtain the first classification probability.

[0131] S306. Input the bill wireframe image block into the feature extraction module in the second network branch for feature extraction to obtain the bill wireframe feature map.

[0132] S307. Input the bill wireframe feature map into the channel attention module to enhance the channel features and obtain the channel enhanced feature map.

[0133] S308. Input the channel enhancement feature map into the spatial attention module to enhance the region features and obtain the region enhancement feature map.

[0134] S309. Input the region enhancement feature map into the classification module to predict the classification probability and obtain the second classification probability.

[0135] S310. Input the image block of the bill layout into the feature extraction module in the third network branch for feature extraction to obtain the feature map of the bill layout.

[0136] S311. Input the ticket layout feature map into the channel attention module to enhance the channel features and obtain the channel enhanced feature map.

[0137] S312. Input the channel enhancement feature map into the spatial attention module to enhance the region features and obtain the region enhancement feature map.

[0138] S313. Input the region enhancement feature map into the classification module to predict the classification probability and obtain the third classification probability.

[0139] S314. Determine the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model.

[0140] S315. Based on the first classification probability, the second classification probability, and the third classification probability, and based on the first weight parameter, the second weight parameter, and the third weight parameter, determine the ticket category of the ticket image to be processed.

[0141] Example 4

[0142] Figure 4 This is a schematic diagram of a document image classification device provided in Embodiment 4 of the present invention. The document image classification device provided in this embodiment is applicable to document classification scenarios in the field of document processing. This document image classification device can be implemented in hardware and / or software, and can be applied to document classification methods. Specifically, it can be configured in a controller, such as... Figure 4 As shown, the device includes: a ticket image segmentation module 401, a classification probability acquisition module 402, a weight parameter determination module 403, and a ticket category confirmation module 404. Wherein:

[0143] The invoice image segmentation module 401 is used to acquire the invoice image to be processed and to segment the invoice image to be processed to obtain the invoice title image block, the invoice wireframe image block and the invoice layout image block respectively.

[0144] The classification probability acquisition module 402 is used to input the ticket title image block into the first network branch of the pre-built ticket classification model for image processing to obtain the first classification probability output by the first network branch; input the ticket wireframe image block into the second network branch of the ticket classification model for image processing to obtain the second classification probability output by the second network branch; and input the ticket layout image block into the third network branch of the ticket classification model for image processing to obtain the third classification probability output by the third network branch.

[0145] The weight parameter determination module 403 is used to determine the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model.

[0146] The bill category confirmation module 404 is used to determine the bill category of the bill image to be processed based on the first classification probability, the second classification probability and the third classification probability, and the first weight parameter, the second weight parameter and the third weight parameter.

[0147] The technical solution of this invention involves acquiring an image of a ticket to be processed and segmenting it to obtain a ticket title image block, a ticket wireframe image block, and a ticket layout image block. The ticket title image block is input into a first network branch of a pre-constructed ticket classification model for image processing to obtain a first classification probability output by the first network branch. Similarly, the ticket wireframe image block is input into a second network branch of the same model for image processing to obtain a second classification probability output by the second network branch, and the ticket layout image block is input into a third network branch for image processing to obtain a third classification probability output by the third network branch. A first weight parameter corresponding to the first network branch, a second weight parameter corresponding to the second network branch, and a third weight parameter corresponding to the third network branch of the ticket classification model are determined. Based on the first, second, and third classification probabilities, and using the first, second, and third weight parameters, the ticket category of the ticket image to be processed is determined. This technical solution extracts the image blocks of the invoice title, the invoice wireframe, and the invoice layout separately. It then uses the three network branches of the invoice classification model to output the classification probability, and combines the corresponding weight parameters for comprehensive calculation. This can accurately integrate the classification value of different structural features of the invoice, effectively improve the recognition accuracy of the invoice category, reduce the error caused by judging the invoice type by a single feature, and achieve more reliable invoice classification results.

[0148] Optionally, the first, second, and third network branches of the bill classification model have the same network structure; the first, second, and third network branches respectively include a feature extraction module, a channel attention module, a spatial attention module, and a classification module.

[0149] Optional, the classification probability acquisition module 402 is specifically used for:

[0150] The image block of the invoice title is input into the feature extraction module in the first network branch for feature extraction, resulting in the invoice title feature map.

[0151] The ticket title feature map is input into the channel attention module for channel feature enhancement, resulting in a channel enhanced feature map.

[0152] The channel-enhanced feature map is input into the spatial attention module to enhance regional features, resulting in a region-enhanced feature map.

[0153] The region-enhanced feature map is input into the classification module to predict the classification probability, and the first classification probability is obtained.

[0154] Optionally, the ticket image segmentation module 401 includes:

[0155] The grayscale ticket image acquisition unit performs grayscale processing on the ticket image to be processed to obtain a grayscale ticket image.

[0156] The black and white ticket image acquisition unit performs binarization processing on the grayscale ticket image to obtain a black and white ticket image.

[0157] The denoised ticket image acquisition unit performs denoising processing on the black and white ticket image to obtain a denoised ticket image.

[0158] The image sub-region acquisition unit performs image region segmentation on the denoised ticket image to obtain several image sub-regions.

[0159] The ticket image segmentation unit performs region image block recognition on each image sub-region to obtain the ticket title image block, the ticket wireframe image block, and the ticket layout image block.

[0160] Optionally, the black and white receipt image acquisition unit is specifically used for:

[0161] Grayscale values ​​are analyzed for the image pixels in the grayscale ticket image to obtain the first grayscale extreme value and the second grayscale extreme value, and the grayscale threshold is determined based on the first grayscale extreme value and the second grayscale extreme value.

[0162] The first grayscale ratio and the first grayscale mean are determined based on the first grayscale extreme value and the grayscale threshold.

[0163] The second grayscale ratio and the second grayscale mean are determined based on the second grayscale extreme value and the grayscale threshold.

[0164] The inter-class variance is determined based on the first grayscale ratio, the second grayscale ratio, the first grayscale mean, and the second grayscale mean.

[0165] Based on the inter-class variance and a grayscale threshold, a black-and-white image corresponding to the grayscale image is generated.

[0166] Optionally, the device further includes:

[0167] The bill classification model output acquisition module acquires the first historical classification probability corresponding to the first network branch, the second historical classification probability corresponding to the second network branch, and the third historical classification probability corresponding to the third network branch of the bill classification model under historical time periods, as well as the historical bill categories output by the bill classification model.

[0168] The prediction accuracy determination module determines the first prediction accuracy of the first network branch based on the first historical classification probability and the historical bill category, the second prediction accuracy of the second network branch based on the second historical classification probability and the historical bill category, and the third prediction accuracy of the third network branch based on the third historical classification probability and the historical bill category.

[0169] The weight parameter determination module updates the first weight parameter corresponding to the first network branch, the second weight parameter corresponding to the second network branch, and the third weight parameter corresponding to the third network branch based on the first prediction accuracy, the second prediction accuracy, and the third prediction accuracy.

[0170] The document image classification device provided in this embodiment of the invention can execute the document image classification method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0171] Example 5

[0172] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0173] like Figure 5 As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded from storage unit 58 into the RAM 53. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0174] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as the ticket image classification method.

[0176] In some embodiments, the document image classification method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the document image classification method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured as the document image classification method by any other suitable means (e.g., by means of firmware).

[0177] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0179] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0181] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0182] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0183] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0184] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for classifying invoice images, characterized in that, include: The image of the invoice to be processed is obtained, and the image of the invoice to be processed is segmented to obtain the invoice title image block, the invoice wireframe image block and the invoice layout image block respectively; The document title image block is input into the first network branch of the pre-constructed document classification model for image processing to obtain the first classification probability output by the first network branch; the document wireframe image block is input into the second network branch of the document classification model for image processing to obtain the second classification probability output by the second network branch; and the document layout image block is input into the third network branch of the document classification model for image processing to obtain the third classification probability output by the third network branch. Determine the first weight parameter corresponding to the first network branch of the bill classification model, determine the second weight parameter corresponding to the second network branch of the bill classification model, and determine the third weight parameter corresponding to the third network branch of the bill classification model. Based on the first classification probability, the second classification probability, and the third classification probability, and using the first weight parameter, the second weight parameter, and the third weight parameter, the ticket category of the ticket image to be processed is determined.

2. The method according to claim 1, characterized in that, The first, second, and third network branches of the bill classification model have the same network structure; the first, second, and third network branches respectively include a feature extraction module, a channel attention module, a spatial attention module, and a classification module.

3. The method according to claim 2, characterized in that, The step of inputting the bill title image block into the first network branch of a pre-built bill classification model for image processing to obtain the first classification probability output by the first network branch includes: The invoice title image block is input into the feature extraction module in the first network branch for feature extraction to obtain the invoice title feature map; The ticket title feature map is input into the channel attention module for channel feature enhancement to obtain the channel enhanced feature map; The channel-enhanced feature map is input into the spatial attention module to perform region feature enhancement, resulting in a region-enhanced feature map. The enhanced feature map of the region is input into the classification module to predict the classification probability and obtain the first classification probability.

4. The method according to claim 1, characterized in that, The step of segmenting the image of the invoice to be processed to obtain an invoice title image block, an invoice wireframe image block, and an invoice layout image block includes: The image of the ticket to be processed is subjected to grayscale processing to obtain a grayscale ticket image; The grayscale ticket image is binarized to obtain a black and white ticket image; The black and white ticket image is denoised to obtain a denoised ticket image; The denoised ticket image is segmented into several image sub-regions. Region image block recognition is performed on each of the image sub-regions to obtain the ticket title image block, the ticket wireframe image block, and the ticket layout image block.

5. The method according to claim 4, characterized in that, The step of binarizing the grayscale ticket image to obtain a black and white ticket image includes: The grayscale values ​​of the image pixels in the grayscale ticket image are analyzed to obtain the first grayscale extreme value and the second grayscale extreme value, and the grayscale threshold is determined based on the first grayscale extreme value and the second grayscale extreme value. Based on the first grayscale extreme value and the grayscale threshold, determine the first grayscale ratio and the first grayscale mean; and, The second grayscale ratio and the second grayscale mean are determined based on the second grayscale extreme value and the grayscale threshold. The inter-class variance is determined based on the first grayscale ratio, the second grayscale ratio, the first grayscale mean, and the second grayscale mean. Based on the inter-class variance and the grayscale threshold, a black-and-white ticket image corresponding to the grayscale ticket image is generated.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the first historical classification probability corresponding to the first network branch, the second historical classification probability corresponding to the second network branch, and the third historical classification probability corresponding to the third network branch of the bill classification model under the historical time period, and obtain the historical bill categories output by the bill classification model; Based on the first historical classification probability and the historical bill category, a first prediction accuracy of the first network branch is determined; based on the second historical classification probability and the historical bill category, a second prediction accuracy of the second network branch is determined; and based on the third historical classification probability and the historical bill category, a third prediction accuracy of the third network branch is determined. Based on the first prediction accuracy, the second prediction accuracy, and the third prediction accuracy, update the first weight parameter corresponding to the first network branch, the second weight parameter corresponding to the second network branch, and the third weight parameter corresponding to the third network branch.

7. A ticket image classification device, characterized in that, include: The invoice image segmentation module acquires the invoice image to be processed and performs image segmentation on the invoice image to be processed, obtaining invoice title image block, invoice wireframe image block and invoice layout image block respectively; The classification probability acquisition module inputs the ticket title image block into the first network branch of the pre-constructed ticket classification model for image processing to obtain the first classification probability output by the first network branch; inputs the ticket wireframe image block into the second network branch of the ticket classification model for image processing to obtain the second classification probability output by the second network branch; and inputs the ticket layout image block into the third network branch of the ticket classification model for image processing to obtain the third classification probability output by the third network branch. The weight parameter determination module determines the first weight parameter corresponding to the first network branch of the bill classification model, the second weight parameter corresponding to the second network branch of the bill classification model, and the third weight parameter corresponding to the third network branch of the bill classification model. The bill category confirmation module determines the bill category of the bill image to be processed based on the first classification probability, the second classification probability, and the third classification probability, and on the first weight parameter, the second weight parameter, and the third weight parameter.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the ticket image classification method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the ticket image classification method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the ticket image classification method according to any one of claims 1-6.

Images