A sales bill recognition method and system

Abstract

The present application discloses a sales bill recognition method and system, the recognition method comprising: preprocessing the sales bill image to obtain the preprocessed image, the preprocessing including tilt correction; segmenting the preprocessed image to obtain at least one phrase-level image sample in the image; extracting multi-scale features in the phrase-level image sample; identifying the bill information in the phrase-level image sample based on the multi-scale features; synthesizing the recognition results of all phrase-level image samples, and using the synthesized recognition results as the sales bill recognition results. The present application segments the image into at least one phrase-level image sample, and extracts the features of the phrase-level image sample at multiple scales as the basis for bill information extraction, thereby overcoming the recognition difficulties caused by the diversity of image scales and the differences in acquisition conditions, and improving the accuracy of bill information extraction.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to a sales invoice recognition method and system. Background Art

[0002] Tobacco sales companies often need to summarize and check daily sales bills, which often requires a lot of manpower. Manual checking is prone to accidental errors. Therefore, image recognition methods are currently used to extract information on bills. However, different regions have different management departments, and different customers use different management software, resulting in large differences in the format and content of sales bills. In addition, sales bills have a large variety of scales (such as zoom ratios) and collection conditions during image acquisition. Existing bill recognition methods cannot be applied to all scales and collection conditions, resulting in low accuracy in extracting bill information. Summary of the invention

[0003] The present application provides a sales receipt recognition method and system, which segments an image into at least one phrase-level image sample and extracts features of the phrase-level image sample at multiple scales as the basis for extracting receipt information. This overcomes the recognition difficulties caused by the diversity of image scales and differences in acquisition conditions, and improves the accuracy of receipt information extraction.

[0004] The present application provides a sales bill identification method, comprising:

[0005] Preprocessing the image of the sales receipt to obtain a preprocessed image, wherein the preprocessing includes tilt correction;

[0006] Segmenting the preprocessed image to obtain at least one phrase-level image sample in the image;

[0007] Extracting multi-scale features from phrase-level image samples;

[0008] Identify bill information in phrase-level image samples based on multi-scale features;

[0009] The recognition results of all phrase-level image samples are integrated and the integrated recognition results are used as the recognition results of the sales invoice.

[0010] Preferably, segmenting the preprocessed image to obtain at least one phrase-level image sample in the image specifically includes:

[0011] Find the X-direction gradient and Y-direction gradient of the preprocessed image, determine the edge of the text block based on the X-direction gradient and Y-direction gradient, and realize the positioning of the text block;

[0012] Each text block is segmented into at least one phrase-level image sample through a vertical projection operation.

[0013] Preferably, after the text block is located, each text block is subjected to dilation and erosion operations to detect the text area in the text block;

[0014] and

[0015] Each text region is segmented into at least one phrase-level image sample through a vertical projection operation.

[0016] Preferably, a first multi-branch convolution module is used to extract multi-scale features in phrase-level image samples. The first multi-branch convolution module includes multiple convolution branches and a maximum pooling layer. Different convolution branches include convolution kernels of the same or different numbers and sizes. The output data of each convolution branch is input into the maximum pooling layer, and the output result of the maximum pooling layer is used as the multi-scale feature in the phrase-level image sample.

[0017] Preferably, a first multi-branch convolution module and at least one second multi-branch convolution module are used to extract multi-scale features in phrase-level image samples, and the second multi-branch convolution module has the same structure as the first multi-branch convolution module;

[0018] The input data of each second multi-branch convolution module is the output data of the first multi-branch convolution module or the second multi-branch convolution module before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module is used as the multi-scale feature in the phrase-level image sample.

[0019] Preferably, identifying the bill information in the phrase-level image sample based on the multi-scale features specifically includes:

[0020] Use the global average pooling method to process multi-scale features and obtain the original feature map;

[0021] The original feature map is input into the spatial channel squeezing and excitation network to obtain the comprehensive attention feature map;

[0022] The original feature map and the comprehensive attention feature map are added and classified through the fully connected layer to obtain the recognition result of the phrase-level image sample as the ticket information in the phrase-level image sample.

[0023] The present application also provides a sales bill recognition system, including a preprocessing module, an image segmentation module, a multi-scale feature extraction module, a bill information recognition module, and a comprehensive module;

[0024] The preprocessing module is used to preprocess the image of the sales receipt to obtain a preprocessed image, wherein the preprocessing includes tilt correction;

[0025] The image segmentation module is used to segment the preprocessed image to obtain at least one phrase-level image sample in the image;

[0026] The multi-scale feature extraction module is used to extract multi-scale features from phrase-level image samples;

[0027] The bill information recognition module is used to recognize the bill information in the phrase-level image samples based on multi-scale features;

[0028] The comprehensive module is used to integrate the recognition results of all phrase-level image samples and use the integrated recognition results as the recognition results of the sales invoice.

[0029] Preferably, the multi-scale feature extraction module includes a first multi-branch convolution module, which includes multiple convolution branches and a maximum pooling layer. Different convolution branches include convolution kernels of the same or different numbers and sizes, and the maximum pooling layer is connected to the output ends of all convolution branches.

[0030] Preferably, the multi-scale feature extraction module further includes at least one second multi-branch convolution module, and the structure of the second multi-branch convolution module is the same as that of the first multi-branch convolution module;

[0031] The input data of each second multi-branch convolution module is the output data of the first multi-branch convolution module or the second multi-branch convolution module before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module is used as the output data of the multi-scale feature extraction module.

[0032] Preferably, the bill information recognition module includes an original feature map acquisition module, a comprehensive attention feature map acquisition module and a feature map synthesis module;

[0033] The original feature map acquisition module is used to process multi-scale features using the global average pooling method to obtain the original feature map;

[0034] The comprehensive attention feature map acquisition module is used to input the original feature map into the spatial channel squeezing and excitation network to obtain the comprehensive attention feature map;

[0035] The feature map synthesis module is used to add the original feature map and the comprehensive attention feature map, and classify them through the fully connected layer to obtain the recognition result of the phrase-level image sample as the ticket information in the phrase-level image sample.

[0036] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0038] Figure 1A flow chart of a sales note identification method provided for this application;

[0039] Figure 2 A flowchart of a preferred embodiment of image segmentation provided by the present application;

[0040] Figure 3 A flowchart for identifying bill information in phrase-level image samples based on multi-scale features provided in this application;

[0041] Figure 4 A structural diagram of the sales receipt identification system provided for this application;

[0042] Figure 5 A structural diagram of an embodiment of the first multi-branch convolution module provided in the present application;

[0043] Figure 6 This is a structural diagram of the comprehensive attention feature map acquisition module provided in this application. DETAILED DESCRIPTION

[0044] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0048] The present application provides a sales receipt recognition method and system, which segments an image into at least one phrase-level image sample and extracts features of the phrase-level image sample at multiple scales as the basis for extracting receipt information. This solves the recognition difficulties caused by the diversity of image scales and differences in acquisition conditions, and improves the accuracy of receipt information extraction.

[0049] Embodiment 1

[0050] like Figure 1 As shown, the sales bill identification method includes:

[0051] S110: Preprocess the image of the sales receipt to obtain a preprocessed image.

[0052] As an embodiment, the pre-processing includes tilt correction.

[0053] When shooting a sales receipt, there may be a deviation between the sales receipt and the coordinate system of the camera. Therefore, before recognizing the receipt information, the image needs to be tilted. Specifically, the four vertex corners of the receipt on the image are extracted, and the length, width and tilt angle of the receipt on the image are determined according to the positions of the four vertex corners. Then, the receipt is rotated and aligned so that its coordinate system coincides with the coordinate system of the image, thereby obtaining a corrected image as the pre-processed image.

[0054] In a preferred embodiment, preprocessing the image further includes using a Gaussian difference filter to perform noise filtering on the corrected image, and performing binarization processing on the noise filtered image to obtain a preprocessed image.

[0055] S120: Segment the preprocessed image to obtain at least one phrase-level image sample in the image.

[0056] Segmenting the preprocessed image to obtain at least one phrase-level image sample in the image, specifically including:

[0057] S1201: Find the X-direction gradient and the Y-direction gradient of the preprocessed image, determine the edge of the text block according to the X-direction gradient and the Y-direction gradient, locate the text block, and thereby identify the text portion (including at least one character) on the image. It should be noted that the text block also includes the blank area around the text portion.

[0058] Specifically, the convolution operation is performed on each pixel on the preprocessed image through two convolution kernels of the Sobel operator. Among the two convolution kernels, one has the largest response to the vertical edge (i.e., the gradient in the Y direction), and the other has the largest response to the horizontal edge (i.e., the gradient in the X direction). The maximum value of the two convolution results is used as the output of the pixel on the edge of the text block, thereby obtaining the edge of the text block.

[0059] S1203: Segment each text block into at least one phrase-level image sample by vertical projection operation. Segment two parts of text whose distance is greater than a threshold value from each other by vertical projection operation, and each part is used as a phrase-level image sample.

[0060] As mentioned above, the text block contains not only the text part but also the blank area around the text part. In order to more accurately identify the text information, preferably, between S1201 and S1203, S1202 is also included: dilation and erosion operations are performed on each text block to detect the text area in the text block, and the blank area around the text part is removed from the text area. In this preferred embodiment, in S1203, each text area is segmented into at least one phrase-level image sample by vertical projection operation, such as Figure 2 shown.

[0061] S130: Extract multi-scale features from phrase-level image samples.

[0062] Specifically, as an embodiment, a first multi-branch convolution module is used to extract multi-scale features in phrase-level image samples. Figure 5 The first multi-branch convolution module includes multiple convolution branches and a maximum pooling layer. Different convolution branches include convolution kernels of the same or different numbers and sizes. The output data of each convolution branch is input into the maximum pooling layer, and the output result of the maximum pooling layer is used as the multi-scale feature in the phrase-level image sample.

[0063] As another embodiment, a first multi-branch convolution module and at least one second multi-branch convolution module are used to extract multi-scale features in phrase-level image samples, and the second multi-branch convolution module has the same structure as the first multi-branch convolution module. The input data of each second multi-branch convolution module is the output data of the first multi-branch convolution module or the second multi-branch convolution module before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module is used as the multi-scale feature in the phrase-level image sample.

[0064] In this preferred embodiment, the multi-scale feature extraction process integrates the branching idea of ​​the initial sequence model and the short-circuit mechanism of the residual network, and performs multi-scale and multi-level feature extraction on the same phrase-level image samples, overcoming the differences caused by multi-scale and diverse acquisition conditions, making the feature extraction more comprehensive and accurate.

[0065] S140: Identify bill information in phrase-level image samples based on multi-scale features.

[0066] Specifically, as an example, Figure 3 As shown in FIG. 1 , the bill information in the phrase-level image sample is identified based on multi-scale features, specifically including:

[0067] S1401: Use the global average pooling method to process the multi-scale features to obtain the original feature map.

[0068] S1402: Input the original feature map into the Spatial and Channel Squeeze & Excitation (SCSE) network to obtain a comprehensive attention feature map.

[0069] Specifically, the SCSE network includes a spatial attention mechanism and a channel attention mechanism.

[0070] The spatial attention mechanism is used to obtain a spatial attention feature map. Specifically, as an embodiment, the dimension of the original feature map is converted from [C, H, W] to a feature map of [C*H*W, 1, 1], where C represents the number of channels of the original feature map, H represents the height of the original feature map, and W represents the width of the original feature map. Then, two 1×1 convolution kernels are used to process the feature map to obtain a C-dimensional vector. The C-dimensional vector is then normalized using the sigmoid function to obtain the corresponding mask. Finally, a spatial attention feature map calibrated by information is obtained by channel multiplication.

[0071] The channel attention mechanism is used to obtain a channel attention feature map. Specifically, as an embodiment, a 1×1 convolution operation is performed on the original feature map, and its dimension is converted from [C, H, W] to a single channel feature map of [1, H, W]. The single channel feature map is then activated with a sigmoid function and applied to the original feature map to complete the calibration of the channel information and obtain a channel attention feature map.

[0072] The SCSE network also combines the channel attention feature map and the spatial attention feature map to form a comprehensive attention feature map.

[0073] S1403: Add the original feature map and the comprehensive attention feature map, and classify them through the fully connected layer to obtain the recognition result of the phrase-level image sample as the bill information in the phrase-level image sample.

[0074] S150: Integrate the recognition results of all phrase-level image samples, the integrated recognition result includes all the bill information on the sales bill, and use the integrated recognition result as the recognition result of the sales bill.

[0075] Embodiment 2

[0076] Based on the above identification method, the present application also provides a sales bill identification system. Figure 4 As shown, the sales receipt recognition system includes a preprocessing module 410, an image segmentation module 420, a multi-scale feature extraction module 430, a receipt information recognition module 440 and a synthesis module 450.

[0077] The preprocessing module 410 is used to preprocess the image of the sales receipt to obtain a preprocessed image.

[0078] As an embodiment, the preprocessing module 410 includes a tilt correction module, which is used to perform tilt correction on the image of the sales receipt to obtain a corrected image, and use the corrected image as the preprocessed image.

[0079] As another embodiment, the preprocessing module 410 further includes a filtering module and a binarization module. The filtering module is used to perform noise filtering on the corrected image using a Gaussian difference filter to obtain a noise filtered image. The binarization module is used to perform binarization on the noise filtered image to obtain a preprocessed image.

[0080] The image segmentation module 420 is used to segment the preprocessed image to obtain at least one phrase-level image sample in the image.

[0081] As an embodiment, the image segmentation module 420 includes a text block positioning module 4201 and a vertical projection module 4203. The text block positioning module is used to find the X-direction gradient and the Y-direction gradient of the preprocessed image, determine the edge of the text block according to the X-direction gradient and the Y-direction gradient, and realize the positioning of the text block. The vertical projection module 4203 is used to segment each text block into at least one phrase-level image sample through a vertical projection operation.

[0082] As a preferred embodiment, the image segmentation module 420 further includes a text region acquisition module 4202, which is used to perform dilation and erosion operations on each text block to detect the text region in the text block. In this preferred embodiment, the vertical projection module 4203 is used to segment each text region into at least one phrase-level image sample through a vertical projection operation.

[0083] The multi-scale feature extraction module 430 is used to extract multi-scale features from phrase-level image samples.

[0084] As an embodiment, the multi-scale feature extraction module 430 includes a first multi-branch convolution module 4301. The first multi-branch convolution module 4301 includes multiple convolution branches and a maximum pooling layer, different convolution branches include the same or different convolution kernels in number and size, and each convolution branch extracts different features in phrase-level image samples for different image scales. The maximum pooling layer is connected to the output ends of all convolution branches.

[0085] Figure 5 An embodiment of the first multi-branch convolution module is shown. Figure 5As shown, the first multi-branch convolution module consists of four branches: branch 0 includes 1 1*1 basic convolution; branch 1 includes 1 1*1 basic convolution and 1 3*1 basic convolution connected in series; branch 2 includes 1 1*1 basic convolution and 2 3*1 basic convolutions connected in series; branch 3 includes an average pooling layer and 1 1*1 basic convolution connected in series.

[0086] As another example, Figure 4 As shown, the multi-scale feature extraction module 430 includes a first multi-branch convolution module 4301 and at least one second multi-branch convolution module 4302, and the second multi-branch convolution module 4302 has the same structure as the first multi-branch convolution module 4301. The input data of each second multi-branch convolution module 4302 is the output data of the first multi-branch convolution module 4301 or the second multi-branch convolution module 4302 before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module 4302 is used as the output data of the multi-scale feature extraction module.

[0087] The bill information recognition module 440 is used to recognize the bill information in the phrase-level image samples based on the multi-scale features.

[0088] As an example, Figure 4 As shown, the bill information recognition module 440 includes an original feature map acquisition module 4401, a comprehensive attention feature map acquisition module 4402 and a feature map synthesis module 4403.

[0089] The original feature map acquisition module 4401 is used to process the multi-scale features using the global average pooling method to obtain the original feature map.

[0090] The comprehensive attention feature map acquisition module 4402 is used to input the original feature map into the spatial channel squeezing and excitation network to obtain the comprehensive attention feature map.

[0091] like Figure 6 As shown, the comprehensive attention feature map acquisition module 4402 includes a spatial attention feature map acquisition module 44021, a channel attention feature map acquisition module 44022 and a stitching module 44023.

[0092] The spatial attention feature map acquisition module 44021 is used to convert the dimension of the original feature map from [C, H, W] to a feature map of [C*H*W, 1, 1], and then use two 1×1 convolution kernels to process the feature map to obtain a C-dimensional vector, and then use the sigmoid function to normalize the C-dimensional vector to obtain the corresponding mask. Finally, the spatial attention feature map calibrated by the information is obtained through channel multiplication.

[0093] The channel attention feature map acquisition module 44022 is used to perform a 1×1 convolution operation on the original feature map and convert its dimension from [C, H, W] to a single-channel feature map of [1, H, W]. The single-channel feature map is then activated using a sigmoid function and applied to the original feature map to complete the calibration of the channel information and obtain a channel attention feature map.

[0094] The stitching module 44023 is used to stitch the channel attention feature map and the spatial attention feature map to form a comprehensive attention feature map.

[0095] The feature map synthesis module 4403 is used to add the original feature map and the comprehensive attention feature map, and classify them through the fully connected layer to obtain the recognition result of the phrase-level image sample as the bill information in the phrase-level image sample.

[0096] The integration module 450 is used to integrate the recognition results of all phrase-level image samples as the recognition result of the sales bill.

[0097] In view of the diversity of content, format and scale of sales invoices and the differences in collection conditions, this application uses a multi-branch convolution module to collect multi-scale image information, establishes more nonlinear transformations, and makes the convolution block have a stronger learning feature ability. It also introduces the attention mechanism of the SCSE network, which can focus on more critical information in the current task, significantly improve the accuracy of invoice recognition, and achieve better results in most practical application scenarios.

[0098] Although some specific embodiments of the present application have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A sales bill recognition method, characterized in that: include: Preprocessing the image of the sales receipt to obtain a preprocessed image, wherein the preprocessing includes tilt correction; Segmenting the preprocessed image to obtain at least one phrase-level image sample in the image; extracting multi-scale features from the phrase-level image samples; identifying the bill information in the phrase-level image sample according to the multi-scale features; Combining the recognition results of all phrase-level image samples, and using the combined recognition results as the recognition results of the sales receipt; The step of identifying the bill information in the phrase-level image sample according to the multi-scale features specifically includes: The multi-scale features are processed using a global average pooling method to obtain an original feature map; Inputting the original feature map into a spatial channel squeezing and excitation network to obtain a comprehensive attention feature map; Adding the original feature map and the comprehensive attention feature map, and performing classification through a fully connected layer to obtain a recognition result of the phrase-level image sample as the bill information in the phrase-level image sample; Wherein, a first multi-branch convolution module and at least one second multi-branch convolution module are used to extract multi-scale features in the phrase-level image sample, and the second multi-branch convolution module has the same structure as the first multi-branch convolution module; The first multi-branch convolution module includes multiple convolution branches and a maximum pooling layer, different convolution branches include convolution kernels of the same or different numbers and sizes, and output data of each convolution branch is input into the maximum pooling layer; The input data of each second multi-branch convolution module is the output data of the first multi-branch convolution module before the second multi-branch convolution module or the second multi-branch convolution module before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module is used as the multi-scale feature in the phrase-level image sample.

2. The sales bill recognition method according to claim 1, characterized in that: Segmenting the preprocessed image to obtain at least one phrase-level image sample in the image specifically includes: Finding the X-direction gradient and the Y-direction gradient of the preprocessed image, determining the edge of the text block according to the X-direction gradient and the Y-direction gradient, and realizing the positioning of the text block; Each of the text blocks is segmented into at least one phrase-level image sample through a vertical projection operation.

3. The sales bill recognition method according to claim 2, characterized in that: After the text blocks are located, dilation and erosion operations are performed on each of the text blocks to detect the text area in the text blocks; and Each of the text regions is segmented into at least one phrase-level image sample by a vertical projection operation.

4. A sales receipt recognition system, characterized in that: It includes pre-processing module, image segmentation module, multi-scale feature extraction module, bill information recognition module and comprehensive module; The preprocessing module is used to preprocess the image of the sales receipt to obtain a preprocessed image, wherein the preprocessing includes tilt correction; The image segmentation module is used to segment the preprocessed image to obtain at least one phrase-level image sample in the image; The multi-scale feature extraction module is used to extract multi-scale features in the phrase-level image samples; The bill information recognition module is used to recognize the bill information in the phrase-level image sample based on the multi-scale features; The synthesis module is used to synthesize the recognition results of all phrase-level image samples and use the synthesized recognition results as the recognition results of the sales bill; The bill information recognition module includes an original feature map acquisition module, a comprehensive attention feature map acquisition module and a feature map synthesis module; The original feature map acquisition module is used to process the multi-scale features using a global average pooling method to obtain an original feature map; The comprehensive attention feature map acquisition module is used to input the original feature map into the spatial channel squeezing and excitation network to obtain a comprehensive attention feature map; The feature map synthesis module is used to add the original feature map and the comprehensive attention feature map, and classify them through a fully connected layer to obtain the recognition result of the phrase-level image sample as the bill information in the phrase-level image sample; The multi-scale feature extraction module includes a first multi-branch convolution module and at least one second multi-branch convolution module, and the second multi-branch convolution module has the same structure as the first multi-branch convolution module; The first multi-branch convolution module includes multiple convolution branches and a maximum pooling layer, different convolution branches include convolution kernels of the same or different numbers and sizes, and the maximum pooling layer is connected to the output ends of all convolution branches; The input data of each second multi-branch convolution module is the output data of the first multi-branch convolution module before the second multi-branch convolution module or the second multi-branch convolution module before the second multi-branch convolution module, and the pooling result of the maximum pooling layer in the last second multi-branch convolution module is used as the output data of the multi-scale feature extraction module.

Images