Image recognition-based financial document classification method

Abstract

This invention belongs to the field of image recognition, specifically a financial invoice classification method based on image recognition. It employs an adaptive attention fusion mechanism to dynamically balance the contribution weights of overall layout and local details. A graph convolutional network is used to construct a relational topology between the support set and the query set, utilizing multi-round message propagation to achieve mutual enhancement of features of similar invoices and mutual inhibition of features of dissimilar invoices, significantly improving feature discrimination under small sample conditions. A joint distance and directional similarity metric is proposed, considering both the magnitude and direction differences of feature vectors. Based on optimal transport theory, invoices are modeled as local feature distributions, and structural similarity is finely matched by solving the bulldozer distance. This invention incorporates incremental learning and new category detection mechanisms, supporting the dynamic expansion of invoice types without retraining the model, and outputting classification results with confidence for manual review or automated decision-making.

Description

Technical Field

[0001] This invention belongs to the field of image recognition, specifically a method for classifying financial documents based on image recognition. Background Technology

[0002] Automatic classification of financial documents is a crucial step in the digital and intelligent transformation of corporate finance. Businesses generate a large number of documents in their daily operations, including but not limited to: VAT invoices, VAT general invoices, electronic invoices, fixed-amount invoices, train tickets, taxi receipts, bank statements, and expense reimbursement forms. Each type of document has a different layout, key information areas, and anti-counterfeiting features.

[0003] Traditional invoice classification processes rely on manual verification or simple OCR keyword matching, resulting in low efficiency, high cost, and susceptibility to errors. With the development of image processing and deep learning technologies, invoice classification methods based on convolutional neural networks have gradually become mainstream. However, existing invoice image classification methods struggle to extract discriminative features that fuse global and local data under small sample conditions, lack modeling of relationships between invoices, rely on a single distance metric, cannot support incremental learning, and exhibit poor robustness to geometric deformation and lighting changes. Summary of the Invention

[0004] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention provides a financial invoice classification method based on image recognition. In the feature extraction stage, an adaptive attention fusion mechanism combining global and local branches is employed to dynamically balance the contribution weights of the overall layout and local details. A graph convolutional network is used to construct a relational topology between the support set and the query set, utilizing multi-round message propagation to achieve mutual enhancement of features of similar invoices and mutual inhibition of features of dissimilar invoices, significantly improving feature discrimination under small sample conditions. A joint distance and directional similarity metric is proposed, considering both the magnitude and directional differences of feature vectors. Based on optimal transmission theory, invoices are modeled as local feature distributions, and structural similarity is finely matched by solving the bulldozer distance. This scheme incorporates incremental learning and new category detection mechanisms, supporting the dynamic expansion of invoice types without retraining the model, and outputting classification results with confidence for manual review or automatic decision-making. Finally, in the preprocessing stage, affine or perspective transformations are used for geometric correction, combined with adaptive Gaussian threshold binarization, effectively resisting interference from tilt, perspective distortion, and uneven lighting. The above mechanisms work together to form a complete technical loop from image input to classification output, which significantly improves the accuracy, robustness and scalability of ticket classification under conditions of a small number of samples.

[0005] The financial document classification method based on image recognition provided by this invention specifically includes the following steps:

[0006] Step S1: Obtain the financial invoice image to be classified. The invoice image includes a support set consisting of invoices with labeled categories and the invoice to be queried. The invoice image is preprocessed and global feature map and local region feature map are extracted.

[0007] The preprocessing includes:

[0008] Affine transformation is used to perform geometric correction on the ticket image;

[0009] An adaptive Gaussian thresholding method is used to binarize the corrected image, separating the foreground text from the background noise to obtain a binarized image.

[0010] The global feature map is obtained by extracting the entire binarized image through a convolutional neural network, and the local region feature map is obtained by dividing the binarized image into a grid and extracting each grid region through a convolutional neural network with shared weights.

[0011] Affine transformation is used to perform geometric correction on the ticket image, and an adaptive Gaussian thresholding method is used to binarize the corrected image. The specific implementation includes:

[0012] Geometric correction: The coordinates of the four corner points in the ticket image are detected, the affine transformation matrix is ​​calculated based on the four corner points, the original ticket image is remapped using the transformation matrix, and the corrected ticket image is obtained by bilinear interpolation; wherein, the affine transformation matrix is ​​a 2x3 matrix containing six degrees of freedom parameters, which is obtained by solving through three pairs of non-collinear matching points;

[0013] The adaptive Gaussian thresholding method involves constructing a grayscale image of the corrected ticket image and performing adaptive Gaussian thresholding binarization on the grayscale image. Specifically, for each pixel, a square neighborhood window of a preset size is taken centered on that pixel. The two-dimensional Gaussian kernel weight of each pixel within the window is calculated, and the weights of all pixels within the window are normalized. The grayscale values ​​of the pixels within the window are weighted and summed using the normalized weights to obtain the Gaussian weighted local mean of that pixel. A preset constant offset is subtracted from the Gaussian weighted local mean to obtain the adaptive threshold for that pixel. The original pixel grayscale value is compared with the adaptive threshold. If the original pixel grayscale value is less than the adaptive threshold, the pixel is marked as foreground; otherwise, it is marked as background, generating a binarized image. The foreground corresponds to the text area and table line area in the ticket, and the background corresponds to the blank area in the ticket.

[0014] Step S2: Input the global feature map and the local region feature map into the global-local channel attention module respectively to generate a global attention-enhanced feature map and a local attention-enhanced feature map respectively, and then dynamically fuse them through an adaptive fusion mechanism to output the fused ticket feature representation;

[0015] The global-local channel attention module specifically includes:

[0016] The global branch is used to apply the channel attention mechanism to the global feature map of the entire ticket image to generate a global attention-enhanced feature map;

[0017] The local branch divides the ticket image into multiple non-overlapping spatial regions. The channel attention mechanism is applied to the feature map of each region. Then, all the enhanced regions are stitched together according to their spatial positions to obtain the local attention-enhanced feature map.

[0018] The adaptive fusion mechanism uses a learnable fusion network to dynamically generate a set of channel-level weights based on the channel concatenation results of the global attention-enhanced feature map and the local attention-enhanced feature map. The channel-level weights are then used to weight and sum the global attention-enhanced feature map and the local attention-enhanced feature map to obtain the ticket feature representation.

[0019] The channel attention mechanism is implemented in the following way:

[0020] Global average pooling is performed on the input feature map to obtain the channel description vector; then, one-dimensional convolution is used to perform local cross-channel interaction on the channel description vector, with the convolution kernel size adapting to the number of channels; finally, the convolution output is mapped to the 0 to 1 interval through the Sigmoid function as the attention weight for each channel, and the weight is multiplied with the original feature map channel by channel to obtain the output feature map.

[0021] Step S3: Using the feature representations of labeled category tickets and the ticket to be queried in the support set as nodes, construct a graph structure, and perform multi-layer message propagation through a graph convolutional network to update the feature representation of each node, thereby obtaining graph-enhanced features;

[0022] The message propagation process of a graph convolutional network updates node features using the following formula: ;

[0023] in, Let represent the ticket characteristics of all nodes in the l-th layer. To add the adjacency matrix after adding self-loops, for The degree matrix, Let be the learnable weight matrix of the l-th layer. It is a non-linear activation function;

[0024] Step S4: Calculate the structural distribution difference cost between the query ticket and each category in the support set using a joint distance-direction similarity metric. The calculation of the structural distribution difference cost specifically includes: decomposing the graph augmentation features of each ticket into a set of local feature vectors, modeling each ticket as a probability distribution of local features, and finding the minimum transmission cost to transfer the local feature distribution of the query ticket to the category distribution of the support set by solving the optimal transmission problem. The single-point transmission cost in the optimal transmission problem is given by the joint distance-direction similarity metric, which simultaneously measures the magnitude and direction differences between two local feature vectors. The minimum transmission cost is the structural distribution difference cost.

[0025] The joint distance-direction similarity metric is defined as: ;

[0026] Where x and y are two graph enhancement features to be compared. The term represents the squared Euclidean distance, measuring the difference in magnitude. The cosine similarity term measures directional differences. To balance the hyperparameters, the contribution ratios of the amplitude and direction terms are dynamically adjusted;

[0027] The cost of structural distribution differences is calculated using the bulldozer distance in the optimal transport problem:

[0028] Suppose that the graph augmented feature decomposition of the query ticket Q is composed of m local feature vectors {q1,q2,…,qm}, forming a discrete probability distribution, with each local feature vector having a quality of 1 / m; suppose that the graph augmented feature decomposition of the k-th class ticket Ck in the support set is composed of n local feature vectors {c1,c2,…,cn}, forming another discrete probability distribution, with each local feature vector having a quality of 1 / n; then the structural distribution difference cost between the query ticket Q and the class Ck is... Defined as the optimal solution to the following linear programming problem: ;

[0029] in, This represents the quality of the transfer from the i-th local feature vector of the query document to the j-th local feature vector of category Ck. For transportation matrix; For the joint distance-direction similarity metric defined above, as per unit mass from Transferred to The transmission cost, This represents the minimum total transmission cost; the minimum total transmission cost obtained by solving this linear programming problem is the structural distribution difference cost.

[0030] The optimal solution to a linear programming problem satisfies the following constraints: ;

[0031] The first constraint ensures that the elements in the transportation matrix are non-negative; the second constraint ensures that all local feature vectors of the ticket to be queried are exported; and the third constraint ensures that all local feature vectors of category Ck are received.

[0032] Step S5: Classify the invoices to be queried according to the calculated structural distribution difference cost. The structural distribution difference cost within different thresholds corresponds to different categories of the invoices to be queried.

[0033] When the cost of the structural distribution difference exceeds the preset new category determination threshold, the invoice to be queried is marked as a potential new category and manual review is triggered.

[0034] Once manually confirmed as a new category, the graph augmentation features and category label of the ticket are directly added to the support set without retraining the feature extraction network or graph convolutional network.

[0035] The beneficial results achieved by the present invention using the above solution are as follows:

[0036] (1) In the feature extraction stage, an adaptive attention fusion mechanism combining global branches and local branches is adopted to dynamically balance the contribution weights of the overall layout and local details; the support set and query set are constructed into a relational topology through graph convolutional network, and multi-round message propagation is used to achieve mutual enhancement of similar bill features and mutual inhibition of dissimilar bill features, which significantly improves the feature discrimination under small sample conditions.

[0037] (2) A joint distance and direction similarity metric is proposed, which takes into account both the magnitude and direction differences of the feature vectors. Based on the optimal transmission theory, the ticket is modeled as a local feature distribution, and the structural similarity is finely matched by solving the bulldozer distance.

[0038] (3) This solution has built-in incremental learning and new category detection mechanism, which supports the dynamic expansion of ticket types without retraining the model, and outputs classification results with confidence for manual review or automatic decision-making.

[0039] (4) In the preprocessing stage, affine or perspective transformation is used for geometric correction, combined with adaptive Gaussian threshold binarization, which effectively resists interference such as tilt, perspective distortion and uneven lighting. Attached Figure Description

[0040] Figure 1 A flowchart of the image recognition-based financial document classification method provided by the present invention.

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0043] Example 1, see Figure 1 The image recognition-based financial document classification method specifically includes the following steps:

[0044] Step S1: Obtain the financial invoice image to be classified. The invoice image includes a support set consisting of invoices with labeled categories and the invoice to be queried. The invoice image is preprocessed and global feature map and local region feature map are extracted.

[0045] Step S2: Input the global feature map and the local region feature map into the global-local channel attention module respectively to generate a global attention-enhanced feature map and a local attention-enhanced feature map respectively, and then dynamically fuse them through an adaptive fusion mechanism to output the fused ticket feature representation;

[0046] Step S3: Using the feature representations of labeled category tickets and the ticket to be queried in the support set as nodes, construct a graph structure, and perform multi-layer message propagation through a graph convolutional network to update the feature representation of each node, thereby obtaining graph-enhanced features;

[0047] Step S4: Calculate the structural distribution difference cost between the query ticket and each category in the support set using a joint distance-direction similarity metric. The calculation of the structural distribution difference cost specifically includes: decomposing the graph augmentation features of each ticket into a set of local feature vectors, modeling each ticket as a probability distribution of local features, and finding the minimum transmission cost to transfer the local feature distribution of the query ticket to the category distribution of the support set by solving the optimal transmission problem. The single-point transmission cost in the optimal transmission problem is given by the joint distance-direction similarity metric, which simultaneously measures the magnitude and direction differences between two local feature vectors. The minimum transmission cost is the structural distribution difference cost.

[0048] Step S5: Classify the query tickets according to the calculated structural distribution difference cost. The structural distribution difference cost within different thresholds corresponds to different categories of query tickets.

[0049] Example 2: This example is based on the above example. The preprocessing in step S1 specifically includes the following steps:

[0050] The preprocessing includes:

[0051] Affine transformation is used to perform geometric correction on the ticket image;

[0052] An adaptive Gaussian thresholding method is used to binarize the corrected image, separating the foreground text from the background noise to obtain a binarized image.

[0053] The global feature map is obtained by extracting the entire binarized image through a convolutional neural network, and the local region feature map is obtained by dividing the binarized image into a grid and extracting each grid region through a convolutional neural network with shared weights.

[0054] Affine transformation is used to perform geometric correction on the ticket image, and an adaptive Gaussian thresholding method is used to binarize the corrected image. The specific implementation includes:

[0055] Geometric correction: The coordinates of the four corner points in the ticket image are detected, the affine transformation matrix is ​​calculated based on the four corner points, the original ticket image is remapped using the transformation matrix, and the corrected ticket image is obtained by bilinear interpolation; wherein, the affine transformation matrix is ​​a 2x3 matrix containing six degrees of freedom parameters, which is obtained by solving through three pairs of non-collinear matching points;

[0056] The adaptive Gaussian thresholding method involves constructing a grayscale image of the corrected ticket image and performing adaptive Gaussian thresholding binarization on the grayscale image. Specifically, for each pixel, a square neighborhood window of a preset size is taken centered on that pixel. The two-dimensional Gaussian kernel weight of each pixel within the window is calculated, and the weights of all pixels within the window are normalized. The grayscale values ​​of the pixels within the window are weighted and summed using the normalized weights to obtain the Gaussian weighted local mean of that pixel. A preset constant offset is subtracted from the Gaussian weighted local mean to obtain the adaptive threshold for that pixel. The original pixel grayscale value is compared with the adaptive threshold. If the original pixel grayscale value is less than the adaptive threshold, the pixel is marked as foreground; otherwise, it is marked as background, generating a binarized image. The foreground corresponds to the text area and table line area in the ticket, and the background corresponds to the blank area in the ticket.

[0057] Example 3, this example is based on the above example, in step S2:

[0058] The global-local channel attention module specifically includes:

[0059] The global branch is used to apply the channel attention mechanism to the global feature map of the entire ticket image to generate a global attention-enhanced feature map;

[0060] The local branch divides the ticket image into multiple non-overlapping spatial regions. The channel attention mechanism is applied to the feature map of each region. Then, all the enhanced regions are stitched together according to their spatial positions to obtain the local attention-enhanced feature map.

[0061] The adaptive fusion mechanism uses a learnable fusion network to dynamically generate a set of channel-level weights based on the channel concatenation results of the global attention-enhanced feature map and the local attention-enhanced feature map. The channel-level weights are then used to weight and sum the global attention-enhanced feature map and the local attention-enhanced feature map to obtain the ticket feature representation.

[0062] The channel attention mechanism is implemented in the following way:

[0063] Global average pooling is performed on the input feature map to obtain the channel description vector; then, a one-dimensional convolution is used to perform local cross-channel interaction on the channel description vector, with the convolution kernel size adapting to the number of channels; finally, the convolution output is mapped to the 0 to 1 interval through the Sigmoid function as the attention weight for each channel, and the weight is multiplied with the original feature map channel by channel to obtain the output feature map.

[0064] Example 4: This example is based on the above examples. In step S3:

[0065] The message propagation process of a graph convolutional network updates node features using the following formula: ;

[0066] in, Let represent the ticket characteristics of all nodes in the l-th layer. To add the adjacency matrix after adding self-loops, for The degree matrix, Let be the learnable weight matrix of the l-th layer. It is a non-linear activation function.

[0067] Example 5, this example is based on the above examples, in step S4:

[0068] The joint distance-direction similarity metric is defined as: ;

[0069] Where x and y are two graph enhancement features to be compared. The term represents the squared Euclidean distance, measuring the difference in magnitude. The cosine similarity term measures directional differences. To balance the hyperparameters, the contribution ratios of the amplitude and direction terms are dynamically adjusted;

[0070] The cost of structural distribution differences is calculated using the bulldozer distance in the optimal transport problem:

[0071] Suppose that the graph augmented feature decomposition of the query ticket Q is composed of m local feature vectors {q1,q2,…,qm}, forming a discrete probability distribution, with each local feature vector having a quality of 1 / m; suppose that the graph augmented feature decomposition of the k-th class ticket Ck in the support set is composed of n local feature vectors {c1,c2,…,cn}, forming another discrete probability distribution, with each local feature vector having a quality of 1 / n; then the structural distribution difference cost between the query ticket Q and the class Ck is... Defined as the optimal solution to the following linear programming problem: ;

[0072] in, This represents the quality of the transfer from the i-th local feature vector of the query document to the j-th local feature vector of category Ck. For transportation matrix; For the joint distance-direction similarity metric defined above, as per unit mass from Transferred to The transmission cost, This represents the minimum total transmission cost; the minimum total transmission cost obtained by solving this linear programming problem is the structural distribution difference cost.

[0073] The optimal solution to a linear programming problem satisfies the following constraints: ;

[0074] The first constraint ensures that the elements in the transportation matrix are non-negative; the second constraint ensures that all local feature vectors of the ticket to be queried are exported; and the third constraint ensures that all local feature vectors of category Ck are received.

[0075] Example 6: This example is based on the above examples. In step S5:

[0076] When the cost of the structural distribution difference exceeds the preset new category determination threshold, the invoice to be queried is marked as a potential new category and manual review is triggered.

[0077] Once manually confirmed as a new category, the graph augmentation features and category label of the ticket are directly added to the support set without retraining the feature extraction network or graph convolutional network.

[0078] Example 7: The above solution is applied to the automatic classification of VAT invoices in the enterprise's financial shared service center. The data collection is as follows:

[0079] Supported configuration sets:

[0080] Value-added tax (VAT) special invoices, numbered S1_1 to S1_5, sample source: historical scanned copies, characteristic description: red invoice supervision seal, complete information of buyer and seller, and clear tax amount column;

[0081] Value-added tax (VAT) general invoices, numbered S2_1 to S2_5, sample source: historical scanned copies, characteristic description: similar in format to special VAT invoices but without the word "special VAT";

[0082] Electronic invoices, numbered S3_1 to S3_5, sample source: electronic PDF converted to image, characteristic description: no traditional red seal, prominent QR code area;

[0083] Fixed-amount invoices, numbered S4_1 to S4_5, sampled from cashier collection, characterized by: fixed amount preset, no details column, and small size;

[0084] Train tickets, numbered S5_1 to S5_5, sample source: reimbursement slip, characteristics: blue or red magnetic card ticket, including origin, destination and seat number;

[0085] Taxi receipts, numbered S6_1 to S6_5, sampled from passenger records, characterized as roll-type printing, including pick-up and drop-off times and mileage;

[0086] Bank receipts, numbered S7_1 to S7_5, sample source: downloaded from online banking, features include counterparty, amount, transaction number, and electronic seal;

[0087] Configuration of the set of tickets to be queried:

[0088] The search result for invoice number Q1 is a VAT special invoice; the image is clear and scanned from the front.

[0089] The query number is Q2, a special VAT invoice, which is slightly tilted (approximately 5 degrees).

[0090] The electronic invoice with serial number Q3 was scanned after printing, but the stamp area is blurry.

[0091] The query result for invoice number Q4, a fixed-amount invoice, shows a shadow when scanned after being folded.

[0092] Search for ticket number Q5; the image is clear.

[0093] Query number Q6, taxi receipt, the writing is partially worn;

[0094] The query result is Q7, a bank receipt with a clear image.

[0095] The query result for invoice number Q8, a special VAT invoice, shows uneven lighting (dark on the left and bright on the right).

[0096] The query result for invoice number Q9 is an electronic invoice; the screenshot taken on my phone and printed has low resolution.

[0097] The search result for invoice number Q10 is a fixed-amount invoice with a clear image.

[0098] Example 8: The above solution is applied to the automatic classification of VAT invoices in the enterprise's financial shared service center. The implementation of the solution is as follows:

[0099] Step S1: Image preprocessing and feature extraction:

[0100] Geometric correction:

[0101] Perform the following operations for each query ticket and support ticket:

[0102] Taking the query of invoice Q2 (a general VAT invoice, tilted at approximately 5 degrees) as an example:

[0103] First, the coordinates of the four corner points in the ticket image are detected. A contour approximation method is used, performing Canny edge detection on the grayscale image to extract all contours. The contour with the largest area and approximately a quadrilateral is selected, and the Douglas-Peucker algorithm is used to approximate the four vertices. The detection results are: top left corner (120, 80), top right corner (1850, 150), bottom right corner (1880, 2650), and bottom left corner (100, 2580).

[0104] The target width and height are estimated based on the distances between the four corner points: the width is estimated to be approximately 1760 pixels, the average of the horizontal distances on the left and right sides, and the height is estimated to be approximately 2500 pixels, the average of the vertical distances on the top and bottom sides. The target rectangle is set to a width of 1760 pixels and a height of 2500 pixels.

[0105] Calculate the affine transformation matrix. Select three pairs of matching points (top left, top right, and bottom left) to solve for the six-degree-of-freedom affine transformation parameters, and obtain the transformation matrix M_affine.

[0106] The original image is remapped using a transformation matrix, and pixel values ​​are obtained by bilinear interpolation for non-integer coordinates. The corrected Q2 document is a standard rectangle, and the tilt is completely eliminated.

[0107] For invoices with severe perspective distortion (such as Q4 fixed-amount invoices that are trapezoidally distorted due to folding during scanning), this embodiment automatically selects perspective transformation (homography transformation) for correction. It uses the four corner points to solve the eight-degree-of-freedom perspective transformation matrix to map the original arbitrary quadrilateral region into a standard rectangle.

[0108] Adaptive Gaussian thresholding binarization:

[0109] Taking the corrected Q2 invoice as an example, perform the following operations:

[0110] The corrected color image is converted to a grayscale image with pixel values ​​ranging from 0 to 255.

[0111] Set the neighborhood window size to 31×31 pixels and the Gaussian kernel standard deviation sigma to 31 / 6, which is approximately 5.17.

[0112] For each pixel in the grayscale image (taking coordinates (500, 800) as an example), a 31×31 neighborhood window is extracted centered on that point. The two-dimensional Gaussian kernel weights of the 961 pixels within the window are calculated and normalized so that the sum of all weights is 1.

[0113] The grayscale values ​​of the pixels within the window are summed using normalized weights, resulting in a Gaussian weighted local mean of 142.6 for that pixel.

[0114] If the preset constant offset C is set to 10, then the adaptive threshold is 142.6 minus 10 equals 132.6.

[0115] The original pixel grayscale value is 88, which is less than the threshold of 132.6, so this pixel is marked as the foreground (text area).

[0116] Traverse all pixels in the image to generate a binary image, where the foreground pixel value is 1 (white, representing text and table lines) and the background pixel value is 0 (black, representing blank areas).

[0117] For Q3 e-invoices (scanned after printing, with a blurred stamp area), the adaptive Gaussian threshold can dynamically adjust the threshold according to the local lighting conditions, so that the stamp area, although blurred, can still be correctly extracted as the foreground; while the traditional fixed threshold (such as the global threshold of 128) will misjudge the blurred stamp area as the background.

[0118] Global and local feature extraction:

[0119] Global feature extraction: The binarized image of the entire ticket (1760×2500 pixels) is input into a convolutional neural network (ResNet-12 is used as the backbone network in this embodiment). After feature extraction of four residual blocks, a 2048-dimensional global feature vector is output.

[0120] Local Feature Extraction: The binarized image is divided into a regular 16×16 pixel grid, resulting in approximately 17,160 grid regions of size 110×156. Each grid region is processed by a lightweight convolutional network with shared weights to extract a 256-dimensional local feature vector. All local feature vectors are then organized spatially into a feature map with dimensions of 110×156×256.

[0121] Step S2: Global-Local Channel Attention Fusion:

[0122] The execution process of S2 is described using the Q1 VAT special invoice as an example:

[0123] Global branch handling:

[0124] Input the global feature map F (size C×H×W, in this embodiment C=2048, H=14, W=20) output in step S1 into the global branch.

[0125] Global average pooling is performed on the feature map F to obtain a 2048-dimensional channel description vector.

[0126] Calculate the adaptive one-dimensional convolution kernel size: k=5 (calculated using the adaptive formula based on the number of channels 2048).

[0127] One-dimensional convolution with k=5 is used to perform local cross-channel interaction on the channel description vectors to capture the dependencies between channels.

[0128] The convolution output is mapped to the 0-1 range using the Sigmoid function, resulting in 2048 channel weights.

[0129] The original feature map F is multiplied with the channel weights channel by channel to output a global attention-enhanced feature map.

[0130] Local branch processing:

[0131] The feature map F of the ticket image is spatially divided into 6×8 non-overlapping regions (each region contains approximately 2-3 original feature units).

[0132] The same ECA channel attention mechanism as the global branch was applied to each region block to obtain 48 enhanced region block features.

[0133] All region blocks are stitched together according to their original spatial positions to restore the size to C×H×W, resulting in a local attention-enhanced feature map.

[0134] Adaptive fusion:

[0135] By splicing along the channel dimension, a cascaded feature map with a size of 4096×H×W is obtained.

[0136] Global average pooling is performed on the cascaded feature maps to obtain a 4096-dimensional vector.

[0137] The 4096-dimensional vector is reduced to 4096 dimensions using a 1×1 convolutional layer (in this actual embodiment, grouped convolution is used to reduce the number of parameters).

[0138] The output is mapped to the 0-1 interval using the Sigmoid function, then split into two 2048-dimensional weight vectors, and the final fused features are calculated.

[0139] For VAT special invoices (Q1), the learned global weight is relatively high (approximately 0.72), while the local weight is relatively low (approximately 0.28), indicating that this type of invoice mainly relies on overall layout features for identification. For fixed-amount invoices (Q4), the local weight is relatively high (approximately 0.65), while the global weight is relatively low (approximately 0.35), because the fixed-amount invoice has a simple layout, but the key difference between different denominations lies in the printed amount number area, requiring more attention to local details.

[0140] Step S3: Graph Propagation Module:

[0141] Graph structure construction:

[0142] The 45 samples—35 labeled tickets from the support set and 10 tickets to be queried from the query set—are used as graph nodes.

[0143] The cosine similarity between each pair of nodes is calculated as the initial edge weight. For node pairs with a similarity below the threshold of 0.3, no connection is established to reduce computational cost.

[0144] Add a self-loop to each node to ensure that the node's own characteristics are preserved during propagation.

[0145] Construct an adjacency matrix A (45×45) and calculate the degree matrix D.

[0146] Multi-round graph convolutional propagation:

[0147] Initialize the node feature matrix H(0), where the features of each node are the fused features output in step S2.

[0148] First round of propagation (l=0):

[0149] Calculate the normalized adjacency matrix, multiply the normalized adjacency matrix by H(0), and update the feature of each node to a weighted average of its own and its neighboring node features.

[0150] Multiply by the learnable weight matrix W(0) (2048×2048), and then pass through the ReLU activation function.

[0151] The first layer output H(1) is obtained.

[0152] Second round of propagation (l=1): Repeat the above process, using W(1) as the weight matrix.

[0153] Third round of propagation (l=2): After the last round of propagation, output H(3) as a graph enhancement feature.

[0154] Dissemination effect:

[0155] Before propagation, the initial features of the five VAT invoices supporting the cluster already showed some similarity (average cosine similarity 0.68), but there were some intra-class differences (for example, slight differences in the position of the seal on invoices from different years). After three rounds of graph propagation, the features of these five nodes reinforced each other, the average similarity increased to 0.85, and the intra-class differences were significantly reduced.

[0156] Meanwhile, the similarity between query node Q1 (VAT special invoice) and the five supporting nodes of special invoices averaged 0.62 before propagation, which increased to 0.78 after propagation. However, the similarity between query node Q2 (VAT general invoice) and the supporting nodes of special invoices did not change much before and after propagation (0.51→0.53), effectively achieving the effect of mutual reinforcement of similar invoice features and mutual inhibition of dissimilar invoice features.

[0157] Step S4: DDS-EMD Joint Distribution Measurement:

[0158] Decomposition graph enhancement features are local feature distributions:

[0159] Taking query node Q1 (VAT invoice) as an example, its graph augmentation features (2048 dimensions) are mapped to 128 local feature vectors through a projection matrix, each with a dimension of 256. These 128 local feature vectors constitute a discrete probability distribution, and the quality of each vector is assigned a value of 1 / 128.

[0160] For each type of ticket in the support set, the graph augmentation features of all its support samples are also decomposed into local feature distributions, and the mean of the local feature distributions of all support samples in that type is taken as the prototype distribution of that type.

[0161] Calculate the DDS distance matrix:

[0162] Taking Q1 and the VAT special invoice category as an example, calculate 128×128=16384 DDS distance values ​​between the 128 local feature vectors of Q1 and the 128 local feature vectors of the prototype distribution of the special invoice category.

[0163] For one distance calculation instance: take the i-th local feature vector qi of Q1 and the j-th prototype local feature vector cj of the special invoice category. Set the balancing hyperparameter γ=0.8.

[0164] The squared Euclidean distance term is calculated to be 0.0342.

[0165] The cosine similarity term is calculated to be 0.942.

[0166] The calculated DDS value is 0.7570.

[0167] Repeated calculations yielded 16,384 DDS values, which formed the cost matrix.

[0168] Solve the optimal transmission problem (bulldozer distance):

[0169] The aforementioned cost matrix and supply and demand constraints are input into the linear programming solver (this embodiment uses the Sinkhorn algorithm for fast approximate solution). The constraints include: all elements in the transportation matrix T are non-negative; the sum of each row (local features of Q1) equals 1 / 128 (all transferred out); the sum of each column (local features of the special invoice category prototype) equals 1 / 128 (all received).

[0170] The minimum total transmission cost EMD(Q1, special invoice) is calculated to be 0.4832.

[0171] Similarly, calculate the EMD values ​​for Q1 and the other six types of bills:

[0172] Q1 vs VAT general invoice: EMD=0.7823;

[0173] Q1 vs e-invoice: EMD=0.8756;

[0174] Q1 vs. fixed amount invoice: EMD=0.9234;

[0175] Q1 vs train ticket: EMD=0.9567;

[0176] Q1 vs. taxi receipt: EMD=0.9812;

[0177] Q1 vs. bank statement: EMD=0.9123;

[0178] The minimum EMD value is 0.4832, corresponding to the VAT special invoice category.

[0179] Step S5: Classification Output and Incremental Learning

[0180] Classification Decision:

[0181] For invoice Q1, the minimum structural distribution difference cost is 0.4832, corresponding to the category of VAT special invoice. The classification threshold range is set as follows:

[0182] Cost less than 0.60: High confidence normal classification;

[0183] Costs between 0.60 and 0.75: medium confidence level classification, manual sampling recommended;

[0184] Cost greater than 0.75: Low confidence level or new category candidate;

[0185] The cost of Q1, 0.4832, is less than 0.60, therefore the category "Value-Added Tax Invoice" is directly output, and the confidence score is calculated using Softmax normalization.

[0186] Output: Category = VAT invoice, Confidence level = 63.3%, Recommendation = Automatic posting.

[0187] New category detection:

[0188] The query result for invoice Q9 is a low-resolution screenshot printout of an e-invoice. Its EMD value compared to all known categories is calculated, with the lowest EMD value being 0.8123 (corresponding to the e-invoice category), which exceeds the preset new category threshold of 0.75. The system marks this invoice as a "potential new category (low-quality e-invoice variant)," triggering a manual review interface.

[0189] Financial auditors confirmed that the invoice was indeed a low-resolution version of an e-invoice (the invoice itself was still classified as an e-invoice), but the poor image quality resulted in low classification confidence. The system added the image enhancement features and the category label "e-invoice" to the support set, increasing the number of e-invoice samples in the support set from 5 to 6. Subsequent processing of similar low-quality invoices significantly improved classification confidence.

[0190] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0191] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0192] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A financial document classification method based on image recognition, characterized in that, Includes the following steps: Step S1: Obtain the financial invoice image to be classified. The invoice image includes a support set consisting of invoices with labeled categories and the invoice to be queried. The invoice image is preprocessed and global feature map and local region feature map are extracted. Step S2: Input the global feature map and the local region feature map into the global-local channel attention module respectively to generate a global attention-enhanced feature map and a local attention-enhanced feature map respectively, and then dynamically fuse them through an adaptive fusion mechanism to output the fused ticket feature representation; Step S3: Using the feature representations of labeled category tickets and the ticket to be queried in the support set as nodes, construct a graph structure, and perform multi-layer message propagation through a graph convolutional network to update the feature representation of each node, thereby obtaining graph-enhanced features; Step S4: Calculate the structural distribution difference cost between the query ticket and each category in the support set using a joint distance-direction similarity metric. The calculation of the structural distribution difference cost specifically includes: decomposing the graph enhancement features of each ticket into a set of local feature vectors, modeling each ticket as a probability distribution of local features, and finding the minimum transmission cost to transfer the local feature distribution of the query ticket to the category distribution of the support set by solving the optimal transmission problem. The single-point transmission cost in the optimal transmission problem is given by the joint distance-direction similarity metric, and the minimum transmission cost is the structural distribution difference cost. Step S5: Classify the query tickets according to the calculated structural distribution difference cost. The structural distribution difference cost within different thresholds corresponds to different categories of query tickets.

2. The financial document classification method based on image recognition according to claim 1, characterized in that, The preprocessing in step S1 includes: Affine transformation is used to perform geometric correction on the ticket image; An adaptive Gaussian thresholding method is used to binarize the corrected image, separating the foreground text from the background noise to obtain a binarized image. The global feature map is obtained by extracting the entire binarized image through a convolutional neural network, and the local region feature map is obtained by dividing the binarized image into a grid and extracting each grid region through a convolutional neural network with shared weights.

3. The financial document classification method based on image recognition according to claim 2, characterized in that, The global-local channel attention module in step S2 specifically includes: The global branch is used to apply the channel attention mechanism to the global feature map of the entire ticket image to generate a global attention-enhanced feature map; The local branch divides the ticket image into multiple non-overlapping spatial regions. The channel attention mechanism is applied to the feature map of each region. Then, all the enhanced regions are stitched together according to their spatial positions to obtain the local attention-enhanced feature map. The adaptive fusion mechanism uses a learnable fusion network to dynamically generate a set of channel-level weights based on the channel concatenation results of the global attention-enhanced feature map and the local attention-enhanced feature map. The channel-level weights are then used to weight and sum the global attention-enhanced feature map and the local attention-enhanced feature map to obtain the ticket feature representation.

4. The financial document classification method based on image recognition according to claim 3, characterized in that, The channel attention mechanism is implemented in the following way: Global average pooling is performed on the input feature map to obtain the channel description vector; then, a one-dimensional convolution is used to perform local cross-channel interaction on the channel description vector, with the convolution kernel size adapting to the number of channels; finally, the convolution output is mapped to the 0 to 1 interval through the Sigmoid function as the attention weight for each channel, and the weight is multiplied with the original feature map channel by channel to obtain the output feature map.

5. The financial document classification method based on image recognition according to claim 1, characterized in that, The message propagation process of the graph convolutional network in step S3 updates node features using the following formula: ； in, Let represent the ticket characteristics of all nodes in the l-th layer. To add the adjacency matrix after adding self-loops, for The degree matrix, Let be the learnable weight matrix of the l-th layer. It is a non-linear activation function.

6. The financial document classification method based on image recognition according to claim 5, characterized in that, The joint distance-direction similarity metric in step S4 is defined as follows: ； Where x and y are two graph enhancement features to be compared. The term represents the squared Euclidean distance, measuring the difference in magnitude. The cosine similarity term measures directional differences. To balance the hyperparameters, the contribution ratios of the amplitude and direction terms are dynamically adjusted.

7. The financial document classification method based on image recognition according to claim 6, characterized in that, The structural distribution difference cost in step S4 is calculated using the bulldozer distance in the optimal transport problem: Suppose that the graph augmented feature decomposition of the query ticket Q is composed of m local feature vectors {q1,q2,…,qm}, forming a discrete probability distribution, with each local feature vector having a quality of 1 / m; suppose that the graph augmented feature decomposition of the k-th class ticket Ck in the support set is composed of n local feature vectors {c1,c2,…,cn}, forming another discrete probability distribution, with each local feature vector having a quality of 1 / n; then the structural distribution difference cost between the query ticket Q and the class Ck is... Defined as the optimal solution to the following linear programming problem: ； in, This represents the quality of the transfer from the i-th local feature vector of the query document to the j-th local feature vector of category Ck. For transportation matrix; For the joint distance-direction similarity metric defined above, as per unit mass from Transferred to The transmission cost, This represents the minimum total transmission cost; the minimum total transmission cost obtained by solving this linear programming problem is the structural distribution difference cost.

8. The financial document classification method based on image recognition according to claim 7, characterized in that, The optimal solution to the linear programming problem satisfies the following constraints: ； The first constraint ensures that the elements in the transportation matrix are non-negative, the second constraint ensures that all local feature vectors of the ticket to be queried are exported, and the third constraint ensures that all local feature vectors of category Ck are received.

9. The financial document classification method based on image recognition according to claim 8, characterized in that, Step S5 is followed by incremental learning steps: When the cost of the structural distribution difference exceeds the preset new category determination threshold, the invoice to be queried is marked as a potential new category and manual review is triggered. Once manually confirmed as a new category, the graph augmentation features and category label of the ticket are directly added to the support set without retraining the feature extraction network or graph convolutional network.

10. The financial document classification method based on image recognition according to claim 2, characterized in that, Affine transformation is used to perform geometric correction on the ticket image, and an adaptive Gaussian thresholding method is used to binarize the corrected image. The specific implementation includes: Geometric correction: The coordinates of the four corner points in the ticket image are detected, the affine transformation matrix is ​​calculated based on the four corner points, the original ticket image is remapped using the transformation matrix, and the corrected ticket image is obtained by bilinear interpolation; wherein, the affine transformation matrix is ​​a 2x3 matrix containing six degrees of freedom parameters, which is obtained by solving through three pairs of non-collinear matching points; The adaptive Gaussian thresholding method involves constructing a grayscale image of the corrected ticket image and performing adaptive Gaussian thresholding binarization on the grayscale image. Specifically, for each pixel, a square neighborhood window of a preset size is taken centered on that pixel. The two-dimensional Gaussian kernel weight of each pixel within the window is calculated, and the weights of all pixels within the window are normalized. The grayscale values ​​of the pixels within the window are weighted and summed using the normalized weights to obtain the Gaussian weighted local mean of that pixel. A preset constant offset is subtracted from the Gaussian weighted local mean to obtain the adaptive threshold for that pixel. The original pixel grayscale value is compared with the adaptive threshold. If the original pixel grayscale value is less than the adaptive threshold, the pixel is marked as foreground; otherwise, it is marked as background, generating a binarized image. The foreground corresponds to the text area and table line area in the ticket, and the background corresponds to the blank area in the ticket.

Images