A method for optical character recognition of shopping receipts

Abstract

The present invention discloses an optical character recognition method for a shopping receipt image in a natural scene, the method comprising using a text detection module to perform region detection on an original image to obtain a receipt text image, and using a text recognition module to perform text recognition on the receipt text image. The text detection module is provided with at least two tasks, namely, salient object detection and text detection, and the text recognition module is provided with an attention module based on a Transformer. The technical solution disclosed in the present invention reduces overfitting of data and improves detection speed by establishing a multi-task model, decouples the alignment process and the prediction process by using an attention module based on a Transformer, accelerates the reasoning speed and improves the text recognition accuracy.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an optical character recognition method for shopping receipts. Background Art

[0002] Optical character recognition (OCR) technology is a computer-based conversion technology that uses a standard encoding scheme to convert images of text into digital form as machine-editable text. This technology eliminates the need to manually type documents into computer systems and is widely used in multiple industries, such as banks, shopping malls, hotels, logistics, etc. According to the recognition scene, OCR technology can be roughly divided into dedicated OCR for recognizing specific scenes and general OCR for recognizing multiple scenes. According to the content of the recognized image, the scene can be divided into simple scenes with clear and fixed patterns and more complex natural scenes. The difficulty of natural scene text recognition is extremely high. The reasons include: the image background is extremely rich, and it often faces problems such as low brightness, low contrast, uneven lighting, perspective deformation and incomplete occlusion. In addition, the layout of the text may be distorted, wrinkled, and reversed. The text may also have problems such as various fonts, different font sizes, weights and colors.

[0003] The OCR technology of shopping receipts is particularly important for shopping malls. In order to improve user stickiness, shopping malls usually carry out various preferential activities based on user consumption, such as returning points to users that can be used to deduct cash. This requires accurate knowledge of user consumption information. However, due to the electronicization of receipts of various brands in shopping malls and other commercial reasons, this information is generally difficult to obtain directly. Summary of the invention

[0004] Based on this, the purpose of the present invention is to achieve fast and high-precision recognition of bill text information in natural scenes by establishing a multi-task text detection model and a text recognition model based on the Transformer attention module, so as to solve the problem of high error rate of bill recognition in natural scenes in the prior art.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for optical character recognition of a shopping receipt, the method comprising:

[0007] S1: Use a text detection module to perform region detection on the original image to obtain a bill text image; the original image includes a bill background and a bill text, and the bill text image includes bill text information and bill barcode information; the region detection method includes performing data enhancement on the original image, selecting a first deep neural network to extract a bill feature image, and establishing at least two tasks of significant object detection and text detection, wherein the significant object detection is used to obtain a bill outline, and the text detection is used to obtain a bill text image within the bill outline;

[0008] S2: using a text recognition module to perform text recognition on the bill text image obtained in step S1 to obtain text content; the text recognition module includes an STN correction network, a feature extraction network, a Transformer-based attention module and a Transformer decoder; the Transformer-based attention module includes a Transformer attention weight and an attention mechanism;

[0009] The method for performing text recognition on a bill text image includes:

[0010] Input a bill text image and correct the image shooting problem through the STN correction network, wherein the image shooting problem includes distortion, bending and perspective problems;

[0011] Selecting a second deep neural network to compress the bill text graph into a first sequence, wherein the first sequence includes a plurality of vectors;

[0012] Sending the first sequence to the Transformer-based attention module, and outputting a second sequence after learning attention weights and updating module parameters;

[0013] The second sequence is decoded by a Transformer decoder to obtain a third sequence, and the third sequence is outputted through probability mapping as the predicted text content of the input bill text graph.

[0014] Specifically, the text detection uses a progressive aggregation method.

[0015] Preferably, the first deep neural network includes a residual neural network component, and deformable convolution is selected.

[0016] Preferably, the length of the first sequence is one quarter of the width of the input bill text image.

[0017] Specifically, the dimension of each vector in the first sequence is the number of channels output by the last convolutional network layer of the second deep neural network.

[0018] Specifically, the third sequence includes several prediction vectors, the dimension of each prediction vector is equal to the number of corresponding character categories, and the softmax function is used to map the value of the prediction vector to the probability space, obtain the probability value of the corresponding character, and select the character with the largest probability value to form the predicted text content.

[0019] Specifically, the training set samples of the text detection module and the text recognition module are artificially synthesized, and the steps of artificially synthesizing the training set samples include randomly generating text margins, selecting data from a bill background library, a bill corpus and a bill font library, writing the corpus data into the background data according to the specifications of the font data, and simulating various image losses in natural scenes.

[0020] Furthermore, the image loss includes blur, curvature, brightness loss, color loss, sharpness loss, temporal lines and random noise.

[0021] Furthermore, the bill background library includes the bill background of the original image, the bill corpus includes the bill general corpus and the bill annotated corpus, and the bill font library includes the commonly used fonts for bills.

[0022] Specifically, the method for performing data enhancement on the original image includes random scale transformation, random horizontal flipping, random rotation and random cropping operations.

[0023] The above technical solution disclosed in the present invention has the following beneficial effects:

[0024] 1) Construction of a multi-task detection model to remove text information other than shopping receipts from the original image, reduce overfitting of the computing network, and provide a basis for subsequent applications and image correction of shopping receipts;

[0025] 2) Enhance the data of the original image to improve the robustness and accuracy of text detection;

[0026] 3) The feature extraction network uses deformable convolution to improve the ability to process complex image and text information;

[0027] 4) Based on the Transformer attention module, the alignment process is decoupled from the prediction process to improve the accuracy of text recognition;

[0028] 5) Artificially synthesize training set samples to train the text detection module and text recognition module to improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural diagram of a shopping receipt text detection module of the present invention;

[0030] Figure 2 It is a structural diagram of a shopping receipt text recognition module of the present invention;

[0031] Figure 3 It is a flow chart of the artificial synthesis of training set samples of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the specific implementation of the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the embodiments described herein are only used to explain the present invention, but are not used to limit the scope of the present invention.

[0033] Please see attached Figure 1 and Figure 2 , Figure 1 This is a structural diagram of the shopping receipt text detection module of the present invention. Figure 2 This is a structural diagram of the shopping receipt text recognition module of the present invention, which consists of Figure 1 and Figure 2 It can be seen that:

[0034] The present invention discloses a method for optical character recognition of a shopping receipt, the method comprising:

[0035] S1: Use a text detection module to perform region detection on the original image to obtain a bill text image; the original image includes a bill background and a bill text, and the bill text image includes bill text information and bill barcode information; the region detection method includes performing data enhancement on the original image, selecting a first deep neural network to extract a bill feature image, and establishing at least two tasks of significant object detection and text detection, wherein the significant object detection is used to obtain a bill outline, and the text detection is used to obtain a bill text image within the bill outline;

[0036] Data augmentation is performed on the original image of the shopping receipt. To ensure batch training of the model, the same data augmentation method is used for the same batch of data. The data is augmented with different categories of data with a certain probability, including random scale transformation, random horizontal flipping, random rotation, and random cropping operations. A batch of data may use zero or more of the above operations.

[0037] The original image with strong data is extracted through the first deep neural network to obtain a feature map. The first deep neural network is composed of residual neural network components. The overall structure of the network is similar to the feature pyramid network. The first convolution module of each downsampling in the network is replaced by a deformable convolution. The shape of the convolution kernel is fixed to a rectangle, which makes the convolution limited in its ability to sense complex shapes. Although the receptive field corresponding to each pixel of the deep feature map gradually increases by superimposing convolution layers, this receptive field is still a fixed rectangle. The deformable convolution network adds a layer of convolution to the original convolution position, which can further adjust the horizontal and vertical directions based on the original convolution kernel. This adjustment is completed by adding an offset to the convolution kernel position. The offset added by the deformable convolution unit is the output of the additional convolution layer. This information is obtained through learning.

[0038] In order to distinguish different text images with close distances, the enhanced original image is passed through the feature extraction network to obtain several feature maps, one of which is used to detect the bill contour, and the rest are used to detect and generate the bill text map. These feature maps for generating bill text maps correspond to different sizes. During training, these feature maps and the original binary images of bills of different sizes are used to calculate the loss function, and the parameters of the model are updated by gradient descent. During inference, the idea of ​​breadth-first is used to take the feature map with the smallest scale as the benchmark, and the output bill text map is obtained by progressive aggregation. The small-sized feature map is used to distinguish different texts, and the large-sized feature map is used to learn the contour of the bill text map. Finally, the bill text map is output by the progressive aggregation method.

[0039] S2: Use the text recognition module to perform text recognition on the bill text image obtained in step S1 to obtain the text content. The text recognition module of the embodiment of the present invention includes an STN correction network, a feature extraction network, a Transformer-based attention module and a Transformer decoder; the Transformer-based attention module includes a Transformer attention weight and an attention mechanism.

[0040] First, the bill text image obtained in step S1 is corrected by the STN correction network. The STN correction network can learn the parameters required for image transformation without additional supervision information. The correction of the network is completed by two sets of control points. The network has a priori knowledge of the position of the control points in the input image, which are two sets of coordinates at a fixed distance from the upper and lower boundaries. At the same time, the input image can learn the actual control points on the current bill text image through the prediction network composed of convolutional layers and fully connected layers. Using these two sets of control points, a mapping can be calculated, which can be applied to the input image to correct the bill text image. The image shooting problem is corrected by the STN correction network, which includes distortion, bending and perspective problems.

[0041] The rectified bill text image is input into the feature extraction network. Specifically, the second deep neural network is used to compress the bill text image into a one-dimensional first sequence. The length of the first sequence obtained after compression is one-fourth of the width of the input bill text image, and the dimension of each vector in the first sequence is the number of channels output by the last layer of the convolutional network.

[0042] The first sequence is input into the Transformer weight branch to learn the attention weights, the first sequence is input into the Transformer attention module, the keys and values ​​of the attention mechanism are updated, and after learning the attention weights and updating the module parameters, the second sequence is output.

[0043] The second sequence is decoded by the Transformer decoder to obtain the third sequence, and the third sequence is output through probability mapping to predict the text content of the input bill text image. Specifically, each element in the third sequence is a prediction vector with the same dimension as the number of character categories. The value of each prediction vector is mapped to the probability space through the softmax function to obtain the probability value of each dimension corresponding to a certain character, and the character with the largest probability is selected to form the predicted text content.

[0044] Please see attached Figure 3 , Figure 3 is a flow chart of artificially synthesizing training set samples in an embodiment of the present invention. In particular, the training set samples of the text detection module and the text recognition module are artificially synthesized. Figure 3 It can be seen that the steps of artificially synthesizing training set samples include randomly generating text margins, selecting data from the bill background library, bill corpus and bill font library, writing the corpus data into the background data according to the specifications of the font data, and simulating various image losses in natural scenes. Among them, image loss includes blur, curvature, brightness loss, color loss, sharpness loss, lines at any time and random noise. The bill background library includes the bill background of the original image, the bill corpus includes the bill general corpus and the bill annotation corpus, and the bill font library includes the commonly used fonts for bills. Using artificially synthesized training sets for model training enables the model to learn more common text combinations and more complex environments, improving the generalization ability of the model.

[0045] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for optical character recognition of a shopping receipt, characterized in that: The method includes: S1: Use a text detection module to perform region detection on the original image to obtain a bill text image; the original image includes a bill background and a bill text, and the bill text image includes bill text information and bill barcode information; the region detection method includes performing data enhancement on the original image, selecting a first deep neural network to extract a bill feature image, and establishing at least two tasks of significant object detection and text detection, wherein the significant object detection is used to obtain a bill outline, and the text detection is used to obtain a bill text image within the bill outline; S2: using a text recognition module to perform text recognition on the bill text image obtained in step S1 to obtain text content; the text recognition module includes an STN correction network, a feature extraction network, a Transformer-based attention module and a Transformer decoder; the Transformer-based attention module includes a Transformer attention weight and an attention mechanism; Among them, the method for text recognition on the bill text image includes: inputting the bill text image, correcting the image shooting problems through the STN correction network, and the image shooting problems include distortion, bending and perspective problems; selecting a second deep neural network to compress the bill text image into a first sequence, and the first sequence includes a number of vectors; sending the first sequence to the Transformer-based attention module, and outputting the second sequence after learning the attention weight and updating the module parameters; the second sequence is decoded by the Transformer decoder to obtain a third sequence, and the third sequence is output through probability mapping. The predicted text content of the input bill text image.

2. The method for optical character recognition according to claim 1, characterized in that: The text detection adopts a progressive aggregation method.

3. The method for optical character recognition according to claim 1, characterized in that: The first deep neural network includes a residual neural network component, and deformable convolution is selected.

4. The method for optical character recognition according to claim 1, characterized in that: The length of the first sequence is one quarter of the width of the input bill text image.

5. The method for optical character recognition according to claim 1, characterized in that: The dimension of each vector in the first sequence is the number of channels output by the last convolutional network layer of the second deep neural network.

6. The method for optical character recognition according to claim 1, characterized in that: The third sequence includes a number of prediction vectors, the dimension of each prediction vector is equal to the number of corresponding character categories, and the softmax function is used to map the value of the prediction vector to the probability space, obtain the probability value of the corresponding character, and select the character with the largest probability value to form the predicted text content.

7. The method for optical character recognition according to claim 1, characterized in that: The training set samples of the text detection module and the text recognition module are artificially synthesized. The steps of artificially synthesizing the training set samples include randomly generating text margins, selecting data from a bill background library, a bill corpus and a bill font library, writing the corpus data into the background data according to the specifications of the font data, and simulating various image losses in natural scenes.

8. The method for optical character recognition according to claim 7, characterized in that: The image loss includes blur, curvature, brightness loss, color loss, sharpness loss, temporal lines and random noise.

9. The method for optical character recognition according to claim 7, characterized in that: The bill background library includes the bill background of the original image, the bill corpus includes the bill general corpus and the bill annotated corpus, and the bill font library includes the commonly used fonts for bills.

10. The method for optical character recognition according to claim 1, characterized in that: The method for performing data enhancement on the original image includes random scale transformation, random horizontal flipping, random rotation and random cropping operations.

Images