Bill identification method, apparatus and device, medium and product

Abstract

The invention provides a bill recognition method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence. The method comprises the following steps: taking a pre-trained bill recognition model and a to-be-recognized bill image, wherein the pre-trained bill recognition model comprises a multi-scale feature extraction module, a self-attention mechanism module, a self-adaptive anchor point frame module and a detection head; inputting the bill image into a feature extraction module to obtain a basic feature map; inputting the basic feature map into a self-attention mechanism module to obtain a target feature map; inputting the target feature map into a self-adaptive anchor point frame module to obtain a self-adaptive anchor point frame; and inputting the target feature map and the adaptive anchor box into a detection head to obtain a bill recognition result. The multi-scale feature extraction module can capture features of targets of different sizes, the self-attention mechanism module can pay attention to key information, the adaptive anchor point frame module can generate more appropriate anchor points according to features of bill images, and the combination of the three modules can improve the understanding ability of the model for complex scenes, thereby improving the accuracy of key information identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a bill identification method, device, equipment, medium and product. BACKGROUND

[0002] In a modern business system, bills are the core carriers of fund circulation and business vouchers, and their processing efficiency and accuracy are directly related to the operating costs of an institution. The current bill types are diverse, including traditional vouchers such as checks and bills of exchange, and new bills such as value-added tax bills and electronic travel orders. These bills often carry key structured information such as amounts, dates, account numbers, and seals. Therefore, one of the core needs is to use intelligent means to quickly and accurately extract key information from bills.

[0003] Currently, existing financial bill identification methods mainly rely on deep learning target detection models. Deep learning-driven target detection technology has gradually become the mainstream solution for bill identification. The process includes: first, generating candidate regions from the original bill image through a region proposal network, and then using a classification head and a regression head to predict the region class and coordinate frame position, respectively. The deep learning-based target detection model has limited multi-scale feature extraction capability when dealing with images with complex backgrounds and many small targets such as bills, which can easily result in missed detection and false detection, thus reducing the accuracy of key information identification. SUMMARY

[0004] The embodiments of the present application provide a bill identification method, device, equipment, medium and product to solve the technical problem of low accuracy of bill key information identification caused by the prior art.

[0005] In a first aspect, the embodiments of the present application provide a bill identification method, comprising:

[0006] obtaining a pre-trained bill identification model and a to-be-identified bill image; wherein the pre-trained bill identification model comprises a multi-scale feature extraction module, a self-attention mechanism module, a self-adaptive anchor frame module, and a detection head;

[0007] inputting the to-be-identified bill image into the multi-scale feature extraction module to obtain a basic feature map fused with full-scale information;

[0008] inputting the basic feature map into the self-attention mechanism module to obtain a target feature map containing global dependency relationships;

[0009] inputting the target feature map into the self-adaptive anchor frame module to obtain a self-adaptive anchor frame suitable for the to-be-identified bill image;

[0010] inputting the target feature map and the self-adaptive anchor frame into the detection head to obtain a bill identification result.

[0011] In a possible design, the multi-scale feature extraction module includes a backbone network and a feature pyramid network.

[0012] The inputting of the to-be-identified bill image into the multi-scale feature extraction module includes:

[0013] The inputting of the to-be-identified bill image into the backbone network includes feature extraction at multiple different depth stages, to obtain standardized feature maps at multiple different depth levels.

[0014] The inputting of the standardized feature maps at multiple different depth levels into the feature pyramid network includes feature fusion of the standardized feature maps at multiple different depth levels along a top-down path, to obtain multi-scale feature maps.

[0015] The multi-scale feature maps are subjected to a spatial attention mechanism, to obtain the basic feature maps fused with full-scale information.

[0016] In a possible design, the inputting of the basic feature maps into the self-attention mechanism module includes:

[0017] The inputting of the basic feature maps into the self-attention mechanism module includes feature map unfolding processing, to obtain unfolded feature maps.

[0018] The unfolded feature maps are mapped into a query matrix, a key matrix, and a value matrix.

[0019] According to the query matrix and the key matrix, a similarity matrix between feature points is determined.

[0020] According to the similarity matrix, similarity normalization processing is performed, to obtain an attention weight matrix.

[0021] According to the attention weight matrix, weighted summation is performed on the value matrix, to obtain the target feature maps containing global dependency relationships.

[0022] In a possible design, the inputting of the target feature maps into the adaptive anchor box module includes:

[0023] The inputting of the target feature maps into the adaptive anchor box module includes anchor box initialization operations, to generate a group of initial anchor boxes at each position of the target feature maps.

[0024] According to the target feature maps, prediction processing is performed through four parallel branches, to obtain a predicted intersection over union (IoU) improvement value and a predicted size confidence.

[0025] According to each initial anchor box on the target feature map, the predicted intersection-over-union value, the predicted size confidence, a preset intersection-over-union threshold and a preset size confidence threshold, an anchor box adjustment operation is performed to obtain an adaptive anchor box suitable for the to-be-recognized bill image after updating the coordinate position and / or the size information.

[0026] In a possible design, the obtaining of the pre-trained bill recognition model includes:

[0027] A bill image sample set, a real target data set and an initial recognition neural network model are obtained, wherein the real target data set includes real anchor boxes and real sample bill information.

[0028] The bill image sample set is input into the initial recognition neural network model for model training, to obtain a predicted anchor box and a sample bill recognition result.

[0029] According to the predicted anchor box, the real anchor box, the sample bill recognition result, the real sample bill information and a preset loss function, a parameter adjustment operation is performed on the initial recognition neural network model, to obtain a pre-trained bill recognition model.

[0030] In a possible design, the preset loss function includes a classification loss function and a bounding box loss function.

[0031] The parameter adjustment operation on the initial recognition neural network model according to the predicted anchor box, the real anchor box, the sample bill recognition result, the real sample bill information and the preset loss function to obtain the pre-trained bill recognition model includes:

[0032] According to the sample bill recognition result and the real sample bill information, a category difference is calculated through the classification loss function, to obtain a classification loss difference value.

[0033] According to the predicted anchor box, the real anchor box and the bounding box loss function, a regression error is calculated, to obtain a bounding box regression difference value.

[0034] According to the classification loss difference value, the bounding box regression difference value and a preset balance factor, a model error value is calculated.

[0035] According to the model error value, a parameter adjustment operation is performed on the initial identification neural network model to obtain an adjusted identification neural network model, and the step of inputting the bill image sample set into the initial identification neural network model for model training to obtain a predicted anchor point frame and a sample bill identification result is returned until the finally obtained model error value is within a preset error range, and the identification neural network model obtained after the last parameter adjustment is taken as a pre-trained bill identification model.

[0036] In a possible design, before the step of inputting the to-be-identified bill image into the multi-scale feature extraction module, the method further includes:

[0037] performing an anomaly detection operation on the to-be-identified bill image, and when it is detected that the to-be-identified bill image is an abnormal bill, performing a repair operation on the to-be-identified bill image by using a pre-trained repair network to obtain a repaired bill image;

[0038] taking the repaired bill image as the to-be-identified bill image.

[0039] In a possible design, the types of abnormal bills include a first type of abnormal bill and a second type of abnormal bill, and the pre-trained repair network includes a local repair network and a context reasoning repair network.

[0040] The step of performing the repair operation on the to-be-identified bill image by using the pre-trained repair network to obtain the repaired bill image includes:

[0041] when it is detected that the to-be-identified bill image is the first type of abnormal bill, performing a local repair operation on the to-be-identified bill image by using the local repair network to obtain the repaired bill image; or

[0042] when it is detected that the to-be-identified bill image is the second type of abnormal bill, performing a context repair operation on the to-be-identified bill image by using the context reasoning repair network to obtain the repaired bill image.

[0043] In a second aspect, an embodiment of the present application provides a bill identification apparatus, including:

[0044] a obtaining module configured to obtain a pre-trained bill identification model and a to-be-identified bill image; the pre-trained bill identification model includes a multi-scale feature extraction module, a self-attention mechanism module, a self-adaptive anchor point frame module, and a detection head.

[0045] a basic feature map obtaining module configured to input the to-be-identified bill image into the multi-scale feature extraction module to obtain a basic feature map fused with full-scale information.

[0046] a target feature map acquisition module, configured to input the base feature map into the self-attention mechanism module to obtain a target feature map containing global dependency relationships;

[0047] an adaptive anchor box acquisition module, configured to input the target feature map into the adaptive anchor box module to obtain an adaptive anchor box suitable for the to-be-recognized bill image;

[0048] a bill recognition result acquisition module, configured to input the target feature map and the adaptive anchor box into the detection head to obtain a bill recognition result.

[0049] In a third aspect, an electronic device is provided, including a processor and a memory connected with the processor in communication;

[0050] The memory stores computer-executable instructions;

[0051] The processor executes the computer-executable instructions stored in the memory to implement the bill recognition method provided in the first aspect.

[0052] In a fourth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the bill recognition method provided in the first aspect.

[0053] In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the computer program is used to implement the bill recognition method provided in the first aspect.

[0054] The present application provides a bill recognition method, device, equipment, medium and product, including: obtaining a pre-trained bill recognition model and a to-be-recognized bill image; inputting the to-be-recognized bill image into a multi-scale feature extraction module to obtain a base feature map fused with full-scale information; inputting the base feature map into a self-attention mechanism module to obtain a target feature map containing global dependency relationships; inputting the target feature map into an adaptive anchor box module to obtain an adaptive anchor box suitable for the to-be-recognized bill image; and inputting the target feature map and the adaptive anchor box into a detection head to obtain a bill recognition result. Through the above method, the following technical effects are achieved: the multi-scale feature extraction module can capture the features of different sizes of targets on the bill, the self-attention mechanism module can focus on key information and capture the relationships of remote areas in the image, and the adaptive anchor box module can generate more suitable anchor points according to the characteristics of the bill image. The combination of the multi-scale feature extraction module, the self-attention mechanism module and the adaptive anchor box module can improve the understanding ability of the model for complex scenes, thereby improving the accuracy of key information recognition. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0056] Figure 1 Flowchart of the bill recognition method provided for the embodiments of the application Figure 1

[0057] Figure 2 Flowchart of the bill recognition method provided for the embodiments of the application Figure 2

[0058] Figure 3 Flowchart of the bill recognition method provided for the embodiments of the application Figure 3

[0059] Figure 4 Flowchart of the bill recognition method provided for the embodiments of the application Figure 4

[0060] Figure 5 Flowchart of the bill recognition method provided for the embodiments of the application Figure 5

[0061] Figure 6 Flowchart of the bill recognition method provided for the embodiments of the application Figure 6

[0062] Figure 7 Flowchart of the bill recognition method provided for the embodiments of the application Figure 7

[0063] Figure 8 Structure diagram of the bill recognition device provided for the embodiments of the application

[0064] Figure 9 Structure diagram of the electronic device provided for the embodiments of the application

[0065] Explanation of reference signs:

[0066] 910 - processor; 920 - memory; 930 - communication component; 940 - bus.

[0067] The specific embodiments of the application have been shown by the above drawings, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] ​​​​​​​The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to any embodiment of the application, unless specified otherwise. Accordingly, when the description of the exemplary embodiments has been referenced, modifications and variations from the described embodiments will be readily apparent to those of ordinary skill in the art, i.e., anyone working in the field, without departing from the scope of the application. It is intended that any embodiment not specifically described fall within the scope of the application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort fall within the scope of the present application.

[0069] In the embodiments of the present application, the terms "first", "second", and the like are used to distinguish between similar or identical items or components having substantially the same function and effect. Those skilled in the art will understand that the terms "first", "second", and the like do not limit the quantity and execution order, and the terms "first", "second", and the like do not necessarily mean different. It should be noted that the words "exemplary" or "for example" in the embodiments of the present application are used to indicate an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of the words "exemplary" or "for example" is intended to present the relevant concept in a specific manner. In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more.

[0070] It should be noted that the ticket identification method provided in the embodiments of the present application is only an example, and the ticket identification method can further include more or less content.

[0071] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of the relevant data need to comply with relevant laws, regulations, and standards, and provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0072] In order to clearly understand the technical solutions of the present application, the solutions of the prior art are described in detail.

[0073] In the prior art, mainly relying on a target detection model of deep learning, a target detection technology driven by deep learning gradually becomes a mainstream scheme for bill recognition. The flow thereof includes: first, generating a candidate region from an original bill image through a region proposal network, and then respectively predicting a region category and a coordinate frame position by using a classification head and a regression head. The target detection model based on deep learning has limited extraction capability for multi-scale features when processing a bill image with a complex background and many small targets, and is prone to missing detection and false detection, so that the accuracy of key information recognition is low.

[0074] To sum up, how to design a kind of bill key information recognition method capable of solving the technical problem of low accuracy caused by the prior art is a problem that needs to be solved by the present application.

[0075] Therefore, in view of the above technical problems existing in the prior art, the present application provides a bill recognition method, device, equipment, medium and product, which aims to improve the accuracy of bill key information recognition.

[0076] The application scenario of the bill recognition method, device, equipment, medium and product provided by the present application will be introduced below. The following application scenarios are only examples, and the purpose is to help those skilled in the art to understand the technical content of the present application, but it does not mean that the present application cannot be used for other devices, systems, environments or scenarios.

[0077] 1) Bank check automatic seal verification and amount verification scene: through the bill recognition method provided by the present application, the handwritten amount, the amount in capital letters, the date of issue and the seal position on the check can be accurately recognized, the seal library is automatically compared to ensure the authenticity of the seal, and the misjudgment caused by illegible handwriting or blurred seal during manual verification is avoided.

[0078] 2) Enterprise financial bill batch input and reimbursement audit scene: through the bill recognition method provided by the present application, key information such as bill code, number, amount and tax amount can be quickly extracted, enterprise account books are automatically matched and bill authenticity is verified, and manual input errors are reduced.

[0079] Figure 1 Flowchart of the bill recognition method provided by the present application Figure 1 Therefore, the bill recognition method provided by the present application includes the following steps:

[0080] S101, acquiring a pre-trained bill recognition model and a to-be-recognized bill image.

[0081] In the present application, the pre-trained bill recognition model includes a multi-scale feature extraction module, a self-attention mechanism module, a self-adaptive anchor frame module and a detection head.

[0082] Optionally, the image of the bill to be recognized is obtained by scanning or shooting the existing bill. Here, the specific way of obtaining the image of the bill to be recognized is not limited.

[0083] S102, input the image of the bill to be recognized into the multi-scale feature extraction module to obtain a basic feature map fused with full-scale information.

[0084] In this embodiment, the multi-scale feature extraction module aims to extract features of the bill to be recognized from different scales to cope with various scale changes of the target object in the bill to be recognized. The multi-scale feature extraction module enhances the detection ability of the model by generating feature maps of different resolutions. The basic feature map is a high-dimensional feature map.

[0085] S103, input the basic feature map into the self-attention mechanism module to obtain a target feature map containing global dependency.

[0086] In this embodiment, the self-attention mechanism is a method of capturing global dependency of input data. The self-attention mechanism module generates global attention distribution by calculating the similarity between feature points in the feature map, thereby enhancing the model's ability to perceive global information.

[0087] The target feature map is a feature map after global context information enhancement of the basic feature map.

[0088] S104, input the target feature map into the adaptive anchor box module to obtain an adaptive anchor box suitable for the bill to be recognized.

[0089] In this embodiment, the anchor box is a key component in the target detection task, which provides a set of predefined boxes for the model to predict the position and size of the target. The adaptive anchor box module dynamically adjusts the size and shape of the anchor box to better adapt to specific feature maps and target objects, thereby improving detection accuracy.

[0090] S105, input the target feature map and the adaptive anchor box into the detection head to obtain the bill recognition result.

[0091] In this embodiment, the pre-trained bill recognition model can be a trained YOLO5 neural network model. YOLO5 is a deep learning model for target detection, which includes a multi-scale feature extraction module, a self-attention mechanism module, an adaptive anchor box module, and a detection head.

[0092] The detection head first performs feature mapping and extraction on the target feature map to generate class prediction scores and boundary box regression parameters corresponding to the adaptive anchor box; then, high-confidence candidate boxes are selected according to the class prediction scores, and the adaptive anchor box is adjusted in coordinates and size in combination with the boundary box regression parameters; then, redundant candidate boxes are removed through non-maximum suppression (NMS) to retain the optimal detection box; finally, the class label and position information corresponding to the detection box are associated to output a bill recognition result containing bill key information and coordinates thereof.

[0093] The multi-scale feature extraction module can capture the features of targets of different sizes on the bill, the self-attention mechanism module can focus on key information and capture the relationship of remote areas in the image, and the adaptive anchor box module can generate more suitable anchors according to the characteristics of the bill image. The combination of the multi-scale feature extraction module, the self-attention mechanism module and the adaptive anchor box module can improve the understanding ability of the model for complex scenes, thereby improving the accuracy of key information recognition.

[0094] The application provides a bill recognition method, including: obtaining a pre-trained bill recognition model and a to-be-recognized bill image; inputting the to-be-recognized bill image into a multi-scale feature extraction module to obtain a basic feature map fused with full-scale information; inputting the basic feature map into a self-attention mechanism module to obtain a target feature map containing global dependency relationships; inputting the target feature map and the adaptive anchor box into a detection head to obtain a bill recognition result. Through the above method, the following technical effects are achieved: the multi-scale feature extraction module can capture the features of targets of different sizes on the bill, the self-attention mechanism module can focus on key information and capture the relationship of remote areas in the image, and the adaptive anchor box module can generate more suitable anchors according to the characteristics of the bill image. The combination of the multi-scale feature extraction module, the self-attention mechanism module and the adaptive anchor box module can improve the understanding ability of the model for complex scenes, thereby improving the accuracy of key information recognition.

[0095] Figure 2 Flowchart of the bill recognition method provided by the embodiment of the application Figure 2 In the bill recognition method provided in the embodiment, the multi-scale feature extraction module includes a backbone network and a feature pyramid network, and S102 includes the following steps:

[0096] S201, inputting the to-be-recognized bill image into the backbone network to obtain standardized feature maps of multiple different depth levels through feature extraction at multiple different depth stages.

[0097] In this embodiment, a pre-trained convolutional neural network is used as the backbone network, and cross-layer connections are added based on the original structure of the backbone network. Optionally, the backbone network uses an improved ResNet-50.

[0098] The image of the bill to be identified is input into the backbone network. The input backbone network, where C is the number of channels, H and W are the height and width, respectively, and after passing through different convolutional layers of the backbone network, feature maps F2, F3, F4, and F5 are extracted at four stages: conv2_x, conv3_x, conv4_x, and conv5_x. Among them, , , , The resolutions of these feature maps decrease from high to low, capturing information at different scales.

[0099] Mathematically, it is represented as follows:

[0100]

[0101] where i = 2, 3, 4, 5, Conv i is the convolution operation at the i-th layer of the backbone network, F i is the feature map corresponding to the i-th layer, and I is the image of the bill to be identified.

[0102] A financial bill image, for example, an office desk photo containing a value-added tax bill, is input into the backbone network, which outputs corresponding feature maps at four different depth stages: conv2_x, conv3_x, conv4_x, and conv5_x. The shallow feature map, such as the feature map output by conv2_x, has high resolution and contains rich detailed information, such as the edges and strokes of the bill; the deep feature map, such as the feature map output by conv5_x, has low resolution but strong semantic information, such as the perception that "this is a bill".

[0103] F2, F3, F4, and F5 are unified to have a channel number of 256 through 1x1 convolution, obtaining multiple standardized feature maps at different depth levels , , and .

[0104] S202, input the multiple standardized feature maps at different depth levels into the feature pyramid network, and perform feature fusion along the top-down path to obtain multi-scale feature maps.

[0105] In this embodiment, after obtaining the standardized feature maps of multiple different depth levels, a feature pyramid network (FPN) is constructed. The FPN transmits high-level semantic information to low-resolution feature maps through a top-down path and a lateral connection, and combines spatial detail information of low-level features, finally outputting multi-scale feature maps. The specific process is as follows:

[0106] Top-down path:

[0107]

[0108]

[0109] wherein, is a bilinear interpolation up-sampling operation, is an adaptive weight coefficient, The calculation method of is as follows:

[0110]

[0111] wherein, W i is a weight matrix, is a sigmoid activation function, and [;] is a channel dimension splicing operation. The weight matrix W i is a learnable parameter that can be automatically optimized through neural network training.

[0112] Finally, the multi-scale feature map is obtained.

[0113] S203, a spatial attention mechanism is applied to the multi-scale feature map to obtain a basic feature map fused with full-scale information.

[0114] In this embodiment, a spatial attention mechanism is applied to each layer feature map P i in the multi-scale feature map by using a spatial attention module:

[0115]

[0116] wherein, is a 1x1 convolution operation, is an average pooling operation, is a maximum pooling operation.

[0117]

[0118] After obtaining , , and , the , 、 and Bilinear interpolation is performed to unify to H / 4xW / 4 resolution, and then depth concatenation is performed to obtain the basic feature map fused with full-scale information .

[0119] The backbone network extracts features from low level to high level layer by layer, and the feature pyramid network fuses features of different levels through the top-down path and horizontal connection, so that the obtained basic feature map contains both shallow high-resolution detail information and deep high semantic information, which can enhance the adaptability to multi-scale targets and reduce missed detection and false detection in complex scenes.

[0120] Figure 3 The flowchart of the bill recognition method provided by the embodiment of the application is shown Figure 3 Therefore, in the bill recognition method provided by the embodiment, S103 includes the following steps:

[0121] S301, input the basic feature map into the self-attention mechanism module for feature map unfolding processing to obtain an unfolded feature map.

[0122] In the embodiment, in order to calculate self-attention, each feature point in the basic feature map needs to be unfolded and mapped into a new space for similarity calculation. The basic feature map wherein C is the number of channels, H and W are the height and width of the basic feature map respectively, the basic feature map is input into the self-attention mechanism module for feature map unfolding processing to obtain an unfolded feature map wherein is the total number of feature points.

[0123] S302, map the unfolded feature map into a query matrix, a key matrix and a value matrix.

[0124] In the embodiment, linear transformation is applied, and optionally, 1x1 convolution is used to map the unfolded feature map into three new spaces to obtain a query matrix, a key matrix and a value matrix respectively, and the calculation formula is as follows:

[0125]

[0126] wherein Q is the query matrix, W Q is a weight matrix with learnable parameters in each attention mechanism, used to generate the query matrix, F flat is the unfolded feature map.

[0127]

[0128] wherein K is the key matrix, W KF is a weight matrix with learnable parameters in each attention mechanism for generating the key matrix. flat is the unfolded feature map.

[0129]

[0130] where V is the value matrix, W V F is a weight matrix with learnable parameters in each attention mechanism for generating the value matrix. flat is the unfolded feature map.

[0131] The shapes of the obtained Q, K and V are where d is the dimension after mapping.

[0132] S303, determining a similarity matrix between feature points according to the query matrix and the key matrix.

[0133] In this embodiment, the dot product between the query matrix and the key matrix is calculated to obtain the similarity matrix between the feature points. The similarity matrix is used to measure the relationship between each feature point and other feature points.

[0134]

[0135] Here, the similarity matrix A is a matrix, where A ij represents the similarity between the i-th feature point and the j-th feature point.

[0136] S304, similarity normalization processing is performed according to the similarity matrix to obtain an attention weight matrix.

[0137] In this embodiment, in order to convert the similarity matrix into an attention weight matrix that can be used for weighted summation, a Softmax operation needs to be performed on the similarity matrix. The role of Softmax is to normalize the similarity of each feature point into a probability distribution:

[0138]

[0139] where, is the attention weight matrix, is a scaling factor used to prevent the gradient vanishing problem caused by too large similarity values.

[0140] S305, weighted summation is performed on the value matrix according to the attention weight matrix to obtain a target feature map containing global dependency relationships.

[0141] In this embodiment, the attention weight matrix W attn is used to perform weighted summation on the value matrix V to output the feature map O:

[0142]

[0143] where each element of O is a weighted sum based on global context information, thus capturing global dependencies.

[0144] The feature map O is transformed back to the shape of the original feature map to obtain a target feature map that contains global dependencies:

[0145]

[0146] where F out is the target feature map that retains both local information of the input feature map and enhanced modeling capability of global dependencies through the global attention mechanism.

[0147] The base feature map is input into the self-attention mechanism module to capture long-distance context information by calculating the correlation weight of any two pixel points in the feature map, which can break through the limitation of the local receptive field of the convolution operation, enhance the perception of non-continuous distribution key information, and improve the robustness and semantic consistency of target recognition under complex layout.

[0148] Figure 4 Flowchart of the bill recognition method provided by the embodiment of the application Figure 4 If yes, in the bill recognition method provided by the embodiment, S104 includes the following steps:

[0149] S401, input the target feature map into the adaptive anchor box module for anchor box initialization operation to generate a group of initial anchor boxes at each position of the target feature map.

[0150] In the embodiment, the target feature map contains context information. A group of initial anchor boxes is generated at each position of the target feature map, and these initial anchor boxes are usually based on a group of predefined sizes and scales. For example, assuming that K initial anchor boxes are generated at each position of the target feature map F out , the set of initial anchor boxes can be represented as where i=1, 2, …, H, j=1, 2, …, W, and k=1, 2, …, K. represents the kth anchor box generated at the position (i, j) of the target feature map, x i,j k and y i,j k are the center coordinates of the anchor box, w k and h k are the width and height of the anchor box, respectively.

[0151] S402, according to the target feature map, perform prediction processing through four parallel branches to obtain a predicted intersection-over-union improvement value and a predicted size confidence.

[0152] In this embodiment, the target feature map is input, and prediction processing is performed through four parallel branches, and the prediction position offset and the prediction size scaling factor are also obtained. The calculation formula of the prediction position offset is as follows:

[0153] In this embodiment, the target feature map is input, and prediction processing is performed through four parallel branches, and the prediction position offset and the prediction size scaling factor are also obtained. The calculation formula of the prediction position offset is as follows:

[0154]

[0155] The calculation formula of the prediction size scaling factor is as follows:

[0156]

[0157] wherein, [s w ,s y ] is the prediction size scaling factor, is the prediction position offset, tanh() is an activation function, is a 1x1 convolution, ReLu() is an activation function, F out is a target feature map, is a 3-layer 3x3 convolution stack.

[0158] S403, according to each initial anchor box on the target feature map, the prediction intersection over union (IoU) value, the prediction size confidence, the preset intersection over union threshold and the preset size confidence threshold, the anchor box adjustment operation is performed to obtain the adaptive anchor box suitable for the to-be-identified bill image after updating the coordinate position and / or updating the size information.

[0159] In this embodiment, the anchor box adjustment operation is adjusted according to the preset threshold. The preset intersection over union threshold is 0.1, and the coordinate position update condition is: when the prediction IoU improvement value ΔIoU>0.1, the coordinate position is updated.

[0160]

[0161]

[0162]

[0163] wherein, ΔIoU is the prediction IoU improvement value, W iou is a learnable parameter, F roi is a feature vector, which is usually extracted from the target feature map through RoIPooling or RoI Align, and are updated horizontal and vertical coordinates respectively, x and y are horizontal and vertical coordinates of the current center point of the anchor box, Δx and Δy are regression offset of the horizontal and vertical coordinates respectively, w is the width of the anchor box, and h is the height of the anchor box.

[0164] The preset size confidence threshold is 0.5, and the size information updating condition is that when the predicted size confidence c is greater than 0.5, the size information is updated.

[0165]

[0166]

[0167]

[0168] wherein c is the predicted size confidence, W c is a learnable parameter, s w is a scale parameter for adjusting the width, s h is a scale parameter for adjusting the height, w and k are the current width and height of the anchor box respectively, and are the adjusted width and height of the anchor box respectively.

[0169] The coordinate position is updated by dynamically judging whether the predicted intersection over union improvement value exceeds the preset intersection over union threshold, and the size information is updated according to whether the predicted size confidence exceeds the preset size confidence threshold, so that the anchor box can be adaptively optimized according to the actual distribution and scale characteristics of the key information in the bill image, the matching accuracy of various targets in bills of different types and different formats is improved, and the missed detection and false detection caused by too large difference between the fixed anchor box and the actual target are reduced.

[0170] Figure 5 is a flowchart of a bill identification method provided by an embodiment of the present application Figure 5 In the bill identification method provided by the embodiment, the obtaining of the pre-trained bill identification model in S101 includes the following steps:

[0171] S501, obtaining a bill image sample set, a real target data set and an initial identification neural network model.

[0172] In the embodiment, the real target data set includes real anchor boxes and real sample bill information. The initial identification neural network model can be an untrained YOLO5 neural network model.

[0173] Optionally, the bill image sample set and the real target data set are obtained by scanning or photographing the existing bills. Here, the specific way of obtaining the bill image sample set and the real target data set is not limited.

[0174] S502, input the bill image sample set into the initial recognition neural network model for model training to obtain a predicted anchor point frame and a sample bill recognition result.

[0175] In this embodiment, the bill image sample set is input into the initial recognition neural network model for model training to obtain a predicted anchor point frame and a sample bill recognition result output by the initial recognition neural network model.

[0176] S503, performing parameter adjustment operation on the initial recognition neural network model according to the predicted anchor point frame, the real anchor point frame, the sample bill recognition result, the real sample bill information and a preset loss function to obtain a pre-trained bill recognition model.

[0177] In the model training process, the predicted anchor point frame needs to be matched with the real anchor point frame. Intersection over Union (IoU) is usually used to measure the overlap between the predicted anchor point frame and the real anchor point frame. Given a predicted anchor point frame and a real anchor point frame The IoU calculation formula is as follows:

[0178]

[0179] Where ∩ represents the intersection area of the predicted anchor point frame and the real anchor point frame, and ∪ represents the union area of the predicted anchor point frame and the real anchor point frame.

[0180] Positive sample selection: select the anchor point frame with IoU greater than a certain threshold with the real anchor point frame as the positive sample, for example, the threshold is 0.5.

[0181] Negative sample selection: the anchor point frame with IoU less than a certain low threshold with the predicted anchor point frame is selected as the negative sample for background class classification training, for example, the low threshold is 0.4. The selection of positive samples and negative samples is a common means of model training.

[0182] At the same time, the sample bill recognition result and the real sample bill information also need to be matched, the value of the preset loss function is calculated, and then the model parameters are updated according to the gradient of the loss function through the back propagation algorithm to optimize the network weight. Repeat the above steps, iterate multiple epochs on the training set, until the model converges or reaches the preset stopping condition. Finally, save the model after parameter adjustment to obtain the pre-trained bill recognition model. Optionally, the preset loss function is a cross-entropy loss function or an IoU loss function.

[0183] By comparing the size difference between the predicted anchor box and the real anchor box, the sample bill recognition result and the content difference between the real sample bill information, the loss function is used to quantify the model error and optimize the parameters in the reverse direction. The model can gradually learn the feature distribution and positioning rules of the key information of the bill, improve the recognition accuracy and generalization ability of the bill with complex background and various styles, and provide a high-performance model basis for subsequent practical scene applications.

[0184] Figure 6 Flowchart of the bill recognition method provided by the embodiment of the application Figure 6 Therefore, in the bill recognition method provided by the embodiment, the preset loss function includes a classification loss function and a bounding box loss function, and S503 includes the following steps:

[0185] S601, according to the sample bill recognition result and the real sample bill information, the classification loss function is used to calculate the category difference, and the classification loss difference value is obtained.

[0186] In this embodiment, the adaptive anchor box module is usually accompanied by a loss function, which is used to optimize the classification and bounding box regression at the same time. The preset loss function includes a classification loss function and a bounding box loss function.

[0187] The classification loss function is usually used to measure the difference between the predicted category and the real category:

[0188]

[0189] Wherein, L cls is the classification loss difference value, c is the category index, C is the total number of categories, y c is the one-hot encoding of the real sample bill information, p c is the predicted probability that the sample bill recognition result belongs to the category c.

[0190] S602, according to the predicted anchor box, the real anchor box and the bounding box loss function, the regression error is calculated, and the bounding box regression difference value is obtained.

[0191] In this embodiment, the bounding box loss function is usually used to measure the regression error between the predicted anchor box and the real anchor box:

[0192]

[0193] Wherein, L reg is the bounding box regression difference value, k is the anchor box index, K is the total number of anchor boxes to be regressed, is the Smooth L1 loss function, is the jth regression parameter of the ith predicted anchor box, and G is the jth regression target value corresponding to the real anchor box.

[0194] S603, calculate a model error value according to the classification loss difference value, the bounding box regression difference value and a preset balance factor.

[0195] In this embodiment, the model error value is a weighted sum of the classification loss difference value and the bounding box regression difference value:

[0196]

[0197] wherein L is the model error value, L cls is the classification loss difference value, L reg is the bounding box regression difference value, and λ is the preset balance factor for adjusting the weights of the classification loss and the bounding box regression loss.

[0198] S604, perform parameter adjustment operation on the initial recognition neural network model according to the model error value to obtain an adjusted recognition neural network model, and return to the step of inputting the bill image sample set into the initial recognition neural network model for model training to obtain the predicted anchor box and the sample bill recognition result until the finally obtained model error value is within a preset error range, and the recognition neural network model obtained after the last parameter adjustment is taken as the pre-trained bill recognition model.

[0199] In this embodiment, the initial recognition neural network model is adjusted according to the model error value L to obtain an adjusted recognition neural network model, and the process returns to S502 until the finally obtained model error value is within a preset error range or the model converges, and the recognition neural network model obtained after the last parameter adjustment is taken as the pre-trained bill recognition model.

[0200] By dynamically adjusting the loss weights of the classification loss difference value and the bounding box regression difference value through the preset balance factor, the error of one task in the classification task and the regression task is avoided to dominate the model optimization direction, so that the model can learn the discriminative features of the bill categories and the accurate positioning features of the key information at the same time, and the stability and balance of the overall recognition performance are improved.

[0201] Figure 7 Flowchart of the bill recognition method provided by the embodiments of the present application Figure 7 Therefore, in the bill recognition method provided by the embodiments of the present application, before S102, the following steps are further included:

[0202] S701, perform anomaly detection operation on the to-be-recognized bill image, and when it is detected that the to-be-recognized bill image is an abnormal bill, perform repair operation on the to-be-recognized bill image through the pre-trained repair network to obtain a repaired bill image.

[0203] In the embodiment, the abnormal detection operation is performed on the to-be-recognized bill image. When it is detected that the dirt area of the to-be-recognized bill image is greater than a preset threshold, it is determined that the to-be-recognized bill image is an abnormal bill. The pre-trained repair network is used to perform the repair operation on the to-be-recognized bill image, and the repaired bill image is obtained.

[0204] S702, the repaired bill image is taken as a to-be-recognized bill image.

[0205] In the embodiment, the repaired bill image is taken as a to-be-recognized bill image, and S102 is executed.

[0206] The to-be-recognized bill image is recognized and repaired in advance, the interference of abnormal factors on subsequent recognition tasks is reduced, the integrity and clarity of bill key information are improved, high-quality image input is provided for accurately extracting bill content, and recognition errors caused by image quality problems are reduced.

[0207] On the basis of the above-mentioned embodiments, the embodiment provides a bill recognition method. In the embodiment, the type of the abnormal bill includes a first type of abnormal bill and a second type of abnormal bill, the pre-trained repair network includes a local repair network and a context reasoning repair network, and S801 or S802 is selected and executed according to the type of the abnormal bill.

[0208] S801, when it is detected that the to-be-recognized bill image is the first type of abnormal bill, the local repair operation is performed on the to-be-recognized bill image by using the local repair network, and the repaired bill image is obtained.

[0209] In the embodiment, the first area threshold T1 is 5%, the second area threshold T2 is 20%, and T2=0.2*H*W. When the dirt area of the to-be-recognized bill image is less than T1, the to-be-recognized bill image is considered to be the first type of abnormal bill. When the dirt area of the to-be-recognized bill image is greater than or equal to T1 and less than T2, the to-be-recognized bill image is considered to be the second type of abnormal bill.

[0210] When it is detected that the to-be-recognized bill image is the first type of abnormal bill, the local repair operation is performed on the to-be-recognized bill image by using the local repair network. The local repair network is represented as follows:

[0211]

[0212] Wherein, L is the repaired bill image, F damaged is a to-be-recognized bill image of the first type of abnormal bill, is a 1-layer 3*3 convolution, and ReLu() is an activation function.

[0213] S802, when it is detected that the to-be-identified bill image is a second-class abnormal bill type, performing a context repair operation on the to-be-identified bill image through a context reasoning repair network to obtain a repaired bill image.

[0214] In this embodiment, when it is detected that the to-be-identified bill image is a second-class abnormal bill type, a context reasoning repair network is used to perform a context repair operation on the to-be-identified bill image, and the context reasoning repair network is represented as:

[0215]

[0216] wherein, is the repaired bill image, F is the to-be-identified bill image of the second-class abnormal bill type, is a dilated convolution operation, d is a dilated rate, and a is an adaptive mixing parameter, and the calculation formula of a is as follows:

[0217]

[0218] wherein, area is the area of the to-be-identified bill image, and T2 is a second area threshold.

[0219] When the area of the stain is greater than or equal to T2, it is considered that the to-be-identified bill image is a third-class abnormal bill type, a manual review flag is triggered, and the to-be-identified bill image is sent to a manual review end.

[0220] For a first-class abnormal bill type, a local repair network is used to accurately repair local details, and for a second-class abnormal bill type, a context reasoning repair network is used to fill in missing content by using global semantic association, so that different abnormal types are repaired in different ways, the repair efficiency and the repair accuracy are improved, and the repair effect of bill images of various abnormal types is ensured.

[0221] Figure 8 FIG. 1 is a structural schematic diagram of a bill identification device provided by an embodiment of the present application. As shown in FIG. 1, in this embodiment, the bill identification device can be located in an electronic device. The bill identification device includes: Figure 8

[0222] The acquisition module 901 is configured to acquire a pre-trained bill identification model and a to-be-identified bill image, wherein the pre-trained bill identification model includes a multi-scale feature extraction module, a self-attention mechanism module, an adaptive anchor box module, and a detection head.

[0223] The basic feature map acquisition module 902 is configured to input the to-be-identified bill image into the multi-scale feature extraction module to obtain a basic feature map fused with full-scale information.

[0224] The target feature map acquisition module 903 is configured to input the basic feature map into the self-attention mechanism module to obtain a target feature map containing global dependency relationships.​

[0225] The adaptive anchor box obtaining module 904 is configured to input the target feature map into an adaptive anchor box module to obtain an adaptive anchor box suitable for the to-be-identified bill image.

[0226] The bill identification result obtaining module 905 is configured to input the target feature map and the adaptive anchor box into a detection head to obtain a bill identification result.

[0227] The bill identification device provided in this embodiment can perform Figure 1 The technical solutions of the bill identification method embodiment shown in FIG. 8 are similar to those of the bill identification device embodiment shown in FIG. 7, and thus the implementation principles and technical effects are similar, which will not be repeated here. Figure 1 The technical solutions of the bill identification method embodiment shown in FIG. 8 are similar to those of the bill identification device embodiment shown in FIG. 7, and thus the implementation principles and technical effects are similar, which will not be repeated here.

[0228] Meanwhile, the bill identification device provided in this embodiment is further refined on the basis of the bill identification device provided in the previous embodiment.

[0229] Optionally, in this embodiment, the multi-scale feature extraction module includes a backbone network and a feature pyramid network, and the basic feature map obtaining module 902 is further configured to:

[0230] input the to-be-identified bill image into the backbone network to obtain a plurality of standardized feature maps of different depth levels through feature extraction at a plurality of different depth stages; input the plurality of standardized feature maps of different depth levels into the feature pyramid network to perform feature fusion on the plurality of standardized feature maps of different depth levels along a top-down path to obtain a multi-scale feature map; and apply a spatial attention mechanism to the multi-scale feature map to obtain a basic feature map fused with full-scale information.

[0231] Optionally, in this embodiment, the target feature map obtaining module 903 is further configured to:

[0232] input the basic feature map into a self-attention mechanism module for feature map unfolding processing to obtain an unfolded feature map; map the unfolded feature map into a query matrix, a key matrix and a value matrix; determine a similarity matrix between feature points according to the query matrix and the key matrix; perform similarity normalization processing on the similarity matrix to obtain an attention weight matrix; and perform weighted summation on the value matrix according to the attention weight matrix to obtain a target feature map containing global dependency relationships.

[0233] Optionally, in this embodiment, the adaptive anchor box obtaining module 904 is further configured to:

[0234] The target feature map is input into the adaptive anchor box module for anchor box initialization operation to generate a set of initial anchor boxes at each position of the target feature map; the target feature map is subjected to prediction processing through four parallel branches to obtain a predicted intersection over union improvement value and a predicted size confidence; and the anchor box adjustment operation is performed on each initial anchor box on the target feature map, the predicted intersection over union improvement value, the predicted size confidence, a preset intersection over union threshold and a preset size confidence threshold to obtain an adaptive anchor box suitable for the to-be-identified bill image after updating the coordinate position and / or updating the size information.

[0235] Optionally, in the embodiment, the obtaining module 901 is further configured to:

[0236] obtain a bill image sample set, a real target data set and an initial identification neural network model; the real target data set includes real anchor boxes and real sample bill information; input the bill image sample set into the initial identification neural network model for model training to obtain predicted anchor boxes and sample bill identification results; and perform parameter adjustment operation on the initial identification neural network model according to the predicted anchor boxes, the real anchor boxes, the sample bill identification results, the real sample bill information and a preset loss function to obtain a pre-trained bill identification model.

[0237] Optionally, in the embodiment, the preset loss function includes a classification loss function and a bounding box loss function, and the obtaining module 901 is further configured to:

[0238] perform category difference calculation through the classification loss function according to the sample bill identification results and the real sample bill information to obtain a classification loss difference value; perform regression error calculation through the bounding box loss function according to the predicted anchor boxes, the real anchor boxes and the bounding box loss function to obtain a bounding box regression difference value; calculate a model error value according to the classification loss difference value, the bounding box regression difference value and a preset balance factor; perform parameter adjustment operation on the initial identification neural network model according to the model error value to obtain an adjusted identification neural network model, and return to the step of inputting the bill image sample set into the initial identification neural network model for model training to obtain the predicted anchor boxes and the sample bill identification results until the finally obtained model error value is within a preset error range, and the identification neural network model obtained after the last parameter adjustment is taken as the pre-trained bill identification model.

[0239] Optionally, in the embodiment, the method further includes:

[0240] an anomaly detection module configured to perform anomaly detection operation on the to-be-identified bill image, and when it is detected that the to-be-identified bill image is an abnormal bill, perform repair operation on the to-be-identified bill image through the pre-trained repair network to obtain a repaired bill image; and take the repaired bill image as the to-be-identified bill image.

[0241] Optionally, in the embodiment, the type of the abnormal bill includes a first type of abnormal bill and a second type of abnormal bill, the pre-trained repair network includes a local repair network and a context reasoning repair network, and the abnormality detection module is further configured to:

[0242] When it is detected that the to-be-recognized bill image is of the first type of abnormal bill, the local repair network is used to perform a local repair operation on the to-be-recognized bill image to obtain a repaired bill image; or when it is detected that the to-be-recognized bill image is of the second type of abnormal bill, the context reasoning repair network is used to perform a context repair operation on the to-be-recognized bill image to obtain a repaired bill image.

[0243] Figure 9 A structural schematic diagram of an electronic device is provided for the embodiment of the present application. The electronic device is intended for various electronic devices that can execute the bill recognition method, such as microcomputers, single-chip microcomputers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0244] As shown in Figure 9 the electronic device includes at least one processor 910 and a memory 920. The electronic device further includes a communication component 930. The processor 910, the memory 920, and the communication component 930 are connected through a bus 940.

[0245] In the specific implementation process, the at least one processor 910 executes the computer execution instructions stored in the memory 920, so that the at least one processor 910 executes the bill recognition method performed by the electronic device as described above.

[0246] The specific implementation process of the processor 910 can be referred to the bill recognition method embodiments described above, which has similar implementation principles and technical effects, and will not be described here again in the embodiment.

[0247] In the above embodiments, it should be understood that the processor 910 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor 910 can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0248] The memory 920 can include a high-speed RAM memory and can also include a non-volatile storage NVM, such as at least one disk memory.

[0249] The bus 940 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 940 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 940 in the drawings of the present application does not limit to only one bus or one type of bus.

[0250] The functions implemented by the electronic device and the master device described above are introduced for the scheme provided by the embodiments of the present application. It can be understood that the electronic device or the master device contains the hardware structure and / or software module corresponding to the execution of each function in order to implement the above functions. The units and algorithm steps of each example described in combination with the embodiments disclosed in the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solution of the embodiments of the present application.

[0251] The present application also provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the bill recognition method is realized.

[0252] The above computer readable storage medium, the readable storage medium can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0253] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. The readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). The processor and the readable storage medium can also be present as discrete components in an electronic device or host device.

[0254] The memory 920 is a non-transitory computer readable storage medium provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the bill identification method provided by the present application.

[0255] The memory 920 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules of the bill identification method in the embodiments of the present application (for example, the obtaining module 901, the base feature map obtaining module 902, the target feature map obtaining module 903, the adaptive anchor box obtaining module 904 and the bill identification result obtaining module 905 shown in the figure). The processor 910 executes various functional applications and data processing by running the non-transitory software programs, instructions and modules stored in the memory 920, that is, implements the bill identification method in the method embodiments. Figure 8

[0256] Meanwhile, the present embodiment also provides a computer program product, which includes a computer program, and the computer program is used to implement the bill identification method of the above-mentioned embodiments when executed by the processor.

[0257] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0258] ​It should be understood that the device embodiments described above are merely illustrative, and the device of the present application can also be implemented in other manners. For example, the division of the units / modules in the above embodiments is merely a logical function division, and the actual implementation can be in another manner. For example, two or more units / modules can be combined, or some features can be ignored or not executed.

[0259] It should be understood that the device embodiments described above are merely illustrative, and the device of the present application can also be implemented in other manners. For example, the division of the units / modules in the above embodiments is merely a logical function division, and the actual implementation can be in another manner. For example, two or more units / modules can be combined, or some features can be ignored or not executed.

[0260] In addition, unless specifically stated, each functional unit / module in each embodiment of the present application can be integrated in one unit / module, or each unit / module can exist physically, or two or more units / modules can be integrated together. The integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0261] The integrated unit / module, if implemented in the form of hardware, can be a digital circuit and an analog circuit. The physical implementation of the hardware structure includes but is not limited to transistors and memristors. Unless specifically stated, the processor can be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, and an ASIC. Unless specifically stated, the storage unit can be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), and a hybrid memory cube (HMC).

[0262] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0263] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily, and in order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0264] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0265] It should be understood that the present application is not limited to the precise construction that has been described and illustrated herein and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the application is indicated by the appended claims.

Claims

1. A method for recognizing invoices, characterized in that, include: Obtain a pre-trained invoice recognition model and an image of the invoice to be recognized; wherein, the pre-trained invoice recognition model includes a multi-scale feature extraction module, a self-attention mechanism module, an adaptive anchor box module, and a detection head; The image of the ticket to be identified is input into the multi-scale feature extraction module to obtain a basic feature map that incorporates full-scale information. The basic feature map is input into the self-attention mechanism module to obtain a target feature map containing global dependencies; The target feature map is input into the adaptive anchor box module to obtain an adaptive anchor box suitable for the image of the ticket to be identified. The target feature map and the adaptive anchor box are input into the detection head to obtain the ticket recognition result.

2. The method according to claim 1, characterized in that, The multi-scale feature extraction module includes a backbone network and a feature pyramid network; The step of inputting the image of the ticket to be identified into the multi-scale feature extraction module to obtain a basic feature map fused with full-scale information includes: The image of the ticket to be identified is input into the backbone network, and after feature extraction at multiple different depth stages, standardized feature maps at multiple different depth levels are obtained. The standardized feature maps at different depths are input into the feature pyramid network, and the standardized feature maps at different depths are fused along a top-down path to obtain a multi-scale feature map. A spatial attention mechanism is applied to the multi-scale feature map to obtain a basic feature map that incorporates full-scale information.

3. The method according to claim 1, characterized in that, The step of inputting the basic feature map into the self-attention mechanism module to obtain a target feature map containing global dependencies includes: The basic feature map is input into the self-attention mechanism module for feature map unpacking to obtain the unpacked feature map. The expanded feature map is mapped into a query matrix, a key matrix, and a value matrix; Based on the query matrix and the key matrix, a similarity matrix between feature points is determined; The similarity matrix is ​​normalized to obtain the attention weight matrix. The value matrix is ​​weighted and summed according to the attention weight matrix to obtain a target feature map containing global dependencies.

4. The method according to claim 1, characterized in that, The step of inputting the target feature map into the adaptive anchor box module to obtain an adaptive anchor box suitable for the document image to be recognized includes: The target feature map is input into the adaptive anchor box module to perform anchor box initialization operation, so as to generate a set of initial anchor boxes at each position of the target feature map; Based on the target feature map, prediction processing is performed through four parallel branches to obtain the predicted intersection-union ratio improvement value and the prediction size confidence. An anchor box adjustment operation is performed based on each initial anchor box on the target feature map, the predicted intersection-union ratio (IU) boost value, the predicted size confidence, the preset IU threshold, and the preset size confidence threshold to obtain an adaptive anchor box suitable for the image of the ticket to be identified after updating the coordinate position and / or updating the size information.

5. The method according to claim 1, characterized in that, The process of obtaining the pre-trained ticket recognition model includes: Acquire a sample set of invoice images, a real target dataset, and an initial recognition neural network model; wherein, the real target dataset includes real anchor boxes and real sample invoice information; The sample set of invoice images is input into the initial recognition neural network model for model training to obtain the predicted anchor boxes and the sample invoice recognition results. Based on the predicted anchor box, the real anchor box, the sample ticket recognition result, the real sample ticket information, and the preset loss function, the parameters of the initial recognition neural network model are adjusted to obtain a pre-trained ticket recognition model.

6. The method according to claim 5, characterized in that, The preset loss function includes a classification loss function and a bounding box loss function; The step of adjusting the parameters of the initial recognition neural network model based on the predicted anchor box, the ground truth anchor box, the sample ticket recognition result, the real sample ticket information, and a preset loss function to obtain a pre-trained ticket recognition model includes: Based on the sample ticket recognition results and the real sample ticket information, the classification loss function is used to calculate the category difference and obtain the classification loss difference value. The regression error is calculated based on the predicted anchor box, the true anchor box, and the bounding box loss function to obtain the bounding box regression difference value. The model error value is calculated based on the classification loss difference value, the bounding box regression difference value, and the preset balance factor. Based on the model error value, the parameters of the initial recognition neural network model are adjusted to obtain the adjusted recognition neural network model. Then, the process of inputting the ticket image sample set into the initial recognition neural network model for model training to obtain the predicted anchor box and the sample ticket recognition result is repeated until the final model error value is within the preset error range. The recognition neural network model obtained after the last parameter adjustment is used as the pre-trained ticket recognition model.

7. The method according to any one of claims 1 to 6, characterized in that, Before inputting the image of the ticket to be identified into the multi-scale feature extraction module, the method further includes: An anomaly detection operation is performed on the image of the ticket to be identified. When the image of the ticket to be identified is detected as an abnormal ticket, the image of the ticket to be identified is repaired by a pre-trained repair network to obtain a repaired ticket image. The repaired ticket image is used as the ticket image to be identified.

8. The method according to claim 7, characterized in that, The types of abnormal tickets include a type I abnormal ticket and a type II abnormal ticket; the pre-trained repair network includes a local repair network and a context-reasoning repair network. When the image of the invoice to be identified is detected as an abnormal invoice, a pre-trained repair network is used to repair the image of the invoice to be identified, resulting in a repaired invoice image, including: When the image of the invoice to be identified is detected to be of the aforementioned abnormal invoice type, the local repair network is used to perform a local repair operation on the image of the invoice to be identified, resulting in a repaired invoice image; or When the image of the ticket to be identified is detected to be of the second type of abnormal ticket, the context inference repair network is used to perform context repair operation on the image of the ticket to be identified to obtain the repaired ticket image.

9. A ticket recognition device, characterized in that, include: The acquisition module is used to acquire a pre-trained invoice recognition model and an image of the invoice to be recognized; wherein, the pre-trained invoice recognition model includes a multi-scale feature extraction module, a self-attention mechanism module, an adaptive anchor box module, and a detection head; The basic feature map acquisition module is used to input the image of the ticket to be identified into the multi-scale feature extraction module to obtain a basic feature map that integrates full-scale information. The target feature map acquisition module is used to input the basic feature map into the self-attention mechanism module to obtain a target feature map containing global dependencies; An adaptive anchor box acquisition module is used to input the target feature map into the adaptive anchor box module to obtain an adaptive anchor box suitable for the document image to be identified. The invoice recognition result acquisition module is used to input the target feature map and the adaptive anchor point box into the detection head to obtain the invoice recognition result.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the ticket recognition method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the ticket recognition method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program, which, when executed by a processor, is used to implement the ticket recognition method as described in any one of claims 1 to 8.

Images