Bill detection method, device, equipment and storage medium based on deep learning

Through deep learning technology, combined with convolutional neural network and spatial transformation network, the problem of incomplete text correction in bill detection is solved, and the accuracy and efficiency of text recognition and bill detection are improved.

CN110807455BActive Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910884908.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-19
Publication Date
2024-11-08
Estimated Expiration
2039-09-19

AI Technical Summary

Technical Problem

The existing bill detection system cannot take into account the entire bill during the text correction process, resulting in low accuracy and low efficiency in text recognition.

Method used

The deep learning-based method is adopted to perform angle correction through convolutional neural networks, and the spatial transformation network (STN) is used to perform detailed correction of text information, extract and adjust text features, and improve the accuracy of text recognition and the efficiency of bill detection.

Benefits of technology

It improves the accuracy and efficiency of bill text detection, ensures the accuracy of text recognition and the efficiency of bill text detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110807455B_ABST
    Figure CN110807455B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence, and discloses a bill detection method, device, equipment and storage medium based on deep learning. The present invention inputs the text extracted after identifying the bill into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection. The method of the present invention includes: obtaining a bill image to be detected; normalizing the bill image to obtain a normalized image; performing angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image; extracting required text information from the adjusted image according to a preset network model to obtain a text image with text information; correcting the text information in the text image according to a spatial transformation network STN model to obtain an output image with corrected text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a bill detection method, device, equipment and storage medium based on deep learning. Background Art

[0002] A bill is a voucher made in accordance with the prescribed form according to the law and showing the obligation to pay money. Major companies now use bills as statistical information for settlement expenses, so this information needs to be stored in the database. When the corresponding bill data is entered, the bill detection system can correctly extract the text information in the bill. Its importance is self-evident.

[0003] The existing bill detection often pre-processes the bill before detection, detects the bill after the pre-processing, and corrects the text in the bill after detection.

[0004] However, in the process of bill text correction, the text on the entire bill cannot be taken into account, which leads to low accuracy of text recognition and low efficiency of bill detection. Summary of the invention

[0005] The present invention provides a bill detection method, device, equipment and storage medium based on deep learning, which are used to solve the problem of low accuracy of Chinese character recognition during bill text detection, thereby improving the accuracy and efficiency of bill text detection.

[0006] The first aspect of an embodiment of the present invention provides a bill detection method based on deep learning, including: obtaining a bill image to be detected; normalizing the bill image to obtain a normalized image; performing angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image; extracting required text information from the adjusted image according to a preset network model to obtain a text image with the text information; correcting the text information in the text image according to a spatial transformation network (STN) model to obtain an output image, wherein the output image has corrected text.

[0007] Optionally, in a first implementation method of the first aspect of the embodiment of the present application, a first text coordinate of the text information is obtained; a transformation parameter is obtained, the transformation parameter maps the coordinate relationship between the first text coordinate and the preset text coordinate, and the preset text coordinate limits the size, direction and position of the corrected text; based on the preset text coordinate and the transformation parameter, a second text coordinate is calculated, and the coordinate position of the second text coordinate corresponds to that of the first text coordinate; based on the first text coordinate and the second text coordinate, the output image with the corrected text is obtained.

[0008] Optionally, in a second implementation method of the first aspect of the embodiment of the present invention, a first pixel value at the first text coordinates is obtained; a second pixel value at the second text coordinates is obtained; the first pixel value is filled into the second pixel value according to a preset rule, and the output image with the corrected text is obtained.

[0009] Optionally, in a third implementation manner of the first aspect of the embodiment of the present invention, pixel values ​​on the bill image are obtained; the pixel values ​​are normalized to obtain a normalized image.

[0010] Optionally, in a fourth implementation method of the first aspect of the embodiment of the present invention, the target position coordinates and target coordinate order of the four edges of the normalized image are obtained; the rotation angle between the target position coordinates and the preset standard coordinates are obtained, and the preset standard coordinates are the position coordinates of the four edges of the standard bill image; the rotation direction between the target coordinate order and the preset standard order is obtained, and the preset standard order is obtained in a clockwise direction starting from the upper left corner of the standard bill image; the normalized image is rotated and adjusted according to the rotation angle and the rotation direction to obtain the adjusted image.

[0011] Optionally, in a fifth implementation method of the first aspect of the embodiment of the present invention, text features of the text information are extracted from the adjusted image; the text features are cyclically connected to obtain a text box; the text box is output to obtain a text line, and a text image with the text line is obtained.

[0012] Optionally, in a sixth implementation of the first aspect of the embodiment of the present invention, the loss value of the adjusted image is calculated according to a root mean square (RMS) function; and the loss value is optimized using an optimizer to optimize the adjusted image.

[0013] The second aspect of the present invention provides a deep learning-based bill detection device, comprising: an acquisition unit, used to acquire a bill image to be detected; a processing unit, used to normalize the bill image to obtain a normalized image; an adjustment unit, used to perform angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image; an extraction unit, used to extract required text information from the adjusted image according to a preset network model to obtain a text image with the text information; a correction unit, used to correct the text information in the text image according to a spatial transformation network (STN) model to obtain an output image with corrected text.

[0014] Optionally, in a first implementation method of the second aspect of the embodiment of the present application, the correction unit includes: a first acquisition module, used to acquire the first text coordinates of the text information; a second acquisition module, used to acquire transformation parameters, the transformation parameters mapping the coordinate relationship between the first text coordinates and the preset text coordinates, the preset text coordinates limiting the size, direction and position of the corrected text; a calculation module, used to calculate the second text coordinates according to the preset text coordinates and the transformation parameters, the second text coordinates and the coordinate positions of the first text coordinates corresponding to each other; an output module, used to obtain the output image with the corrected text according to the first text coordinates and the second text coordinates.

[0015] Optionally, in a second implementation method of the second aspect of the embodiment of the present application, the output module is specifically used to: obtain a first pixel value on the first text coordinates; obtain a second pixel value on the second text coordinates; fill the first pixel value into the second pixel value according to a preset rule, and obtain the output image with corrected text.

[0016] Optionally, in a third implementation of the second aspect of the embodiment of the present application, the processing unit is specifically used to: obtain pixel values ​​on the bill image; normalize the pixel values ​​to obtain a normalized image.

[0017] Optionally, in a fourth implementation method of the second aspect of the embodiment of the present application, the adjustment unit is specifically used to: obtain the target position coordinates and target coordinate order of the four edges of the normalized image; obtain the rotation angle between the target position coordinates and the preset standard coordinates, the preset standard coordinates are the position coordinates of the four edges of the standard bill image; obtain the rotation direction between the target coordinate order and the preset standard order, the preset standard order is to obtain them in a clockwise direction starting from the upper left corner of the standard bill image; rotate and adjust the normalized image according to the rotation angle and the rotation direction to obtain the adjusted image.

[0018] Optionally, in the fifth implementation method of the second aspect of the embodiment of the present application, the extraction unit is specifically used to: extract text features of the text information in the adjusted picture; cyclically connect the text features to obtain a text box; output the text box to obtain a text line, and obtain a text picture with the text line.

[0019] Optionally, in a sixth implementation method of the second aspect of the embodiment of the present application, the deep learning-based bill detection device also includes: a calculation unit, used to calculate the loss value of the adjusted image according to a root mean square RMS function; and an optimization unit, used to optimize the loss value using an optimizer to optimize the adjusted image.

[0020] The third aspect of the present invention provides a deep learning-based bill detection device, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a line; the at least one processor calls the instructions in the memory so that the deep learning-based bill detection device executes the method described in the first aspect above.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0022] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0023] The present invention inputs the text extracted after bill recognition into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of an embodiment of a bill detection method based on deep learning in an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of another embodiment of the bill detection method based on deep learning in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of an embodiment of a bill detection device based on deep learning in an embodiment of the present invention;

[0027] Figure 4 Schematic diagram of another embodiment of a bill detection device based on deep learning in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of an embodiment of a bill detection device based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention provides a bill detection method, device, equipment and storage medium based on deep learning, which are used to solve the problem of low accuracy of Chinese character recognition during bill text detection, thereby improving the accuracy and efficiency of bill text detection.

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] For ease of understanding, the specific process in the embodiment of the present invention is described below. Figure 1 , an embodiment of the bill detection method based on deep learning in an embodiment of the present invention includes:

[0033] 101. Obtain the image of the bill that needs to be tested.

[0034] The server obtains the bill image to be detected. The server obtains the bill image to be processed. The format of the bill image is not limited here. The size of the bill image is a regular size (24CM in length and 14CM in width). The size of the bill image is not limited here.

[0035] It should be noted that the characters that can be recognized in the bill image include: printed characters, such as Chinese printed characters (simplified and traditional), English printed characters, and digital printed characters; handwritten characters, such as Chinese handwritten characters (simplified and traditional), English handwritten characters, and digital handwritten characters; symbols, such as identification codes, barcodes, CheckBox, ╳, ▇, ○, ●; customizable symbols and customizable characters, such as Japanese, Korean, Mongolian, Tibetan, Uyghur, Kazakh, Kirgiz, and Arabic. The recognized characters are not limited here.

[0036] It is understandable that the execution subject of the present invention can be a deep learning-based bill detection device or server, or other bill detection devices with a deep learning network model, which is not specifically limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0037] 102. Normalize the bill image to obtain a normalized image.

[0038] The server normalizes the bill image and obtains the normalized image. The server obtains the pixel values ​​on the bill image; the server normalizes the pixel values ​​and obtains the normalized image.

[0039] The server normalizes the acquired bill images. Specifically, the bill images are normalized, that is, pixel normalization. Normalization does not change the contrast of the image, and at the same time ensures that all pixel values ​​of the normalized images are within the range of [0, 1]. The formula used is as follows:

[0040]

[0041] Where: a' is the normalized pixel value of the bill image, and a is the original pixel value of the bill image.

[0042] It is understandable that the normalization of bill images is mainly for the convenience of data processing. The data is mapped to the range of 0 to 1 for processing, and the processed images or data are then input into the network model, making server calculations more convenient and faster.

[0043] 103. According to the convolutional neural network model, the normalized image is angle corrected to obtain an adjusted image.

[0044] The server corrects the angle of the normalized image according to the convolutional neural network model and obtains the adjusted image. The server obtains the target position coordinates and target coordinate order of the four edges of the normalized image; the server obtains the rotation angle between the target position coordinates and the preset standard coordinates, and the preset standard coordinates are the position coordinates of the four edges of the standard bill image; the server obtains the rotation direction between the target coordinate order and the preset standard order, and the preset standard order is obtained in a clockwise order starting from the upper left corner of the standard bill image; the server rotates and adjusts the normalized image according to the rotation angle and rotation direction to obtain the adjusted image.

[0045] The server performs angle correction by obtaining the coordinates and coordinate order of the four edges of the bill image. The position coordinates and coordinate order of the four edges of the bill are obtained using a convolutional neural network (CNN). CNN is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. CNN has the ability to represent learning and can classify input information according to its hierarchical structure in a translation-invariant manner. Therefore, CNN can make the input image and the topological structure of the network more consistent. CNN includes VGGNet structure, GoogleNet structure and ResNet structure. The results obtained by the server using the CNN network model are more accurate.

[0046] To further explain, the server uses the target position coordinates and the preset standard coordinates to calculate the rotation matrix, and then calculates the Euler angle to determine the angle that the image needs to be rotated; the server uses the target coordinate order and the preset standard order to calculate the direction in which the image needs to be rotated, and determines whether the rotation direction is clockwise or counterclockwise.

[0047] 104. According to the preset network model, the required text information is extracted from the adjusted image to obtain a text image with the text information.

[0048] The server extracts the required text information from the adjusted image according to the preset network model and obtains a text image with text information. The server extracts the text features of the text information from the adjusted image; the server connects the text features in a loop to obtain a text box; the server outputs the text box to obtain a text line and obtains a text image with a text line. The preset network model here is scene text detection (detecting text in natural image with connectionist text proposal network, CTPN). The CTPN network model mainly includes text detection and text recognition. The main function of text detection is to find the text area from the image and separate the text area from the original image; the main function of text recognition is to perform text recognition on the separated image. The main process of text recognition is: preprocessing, image denoising (filtering algorithm), image enhancement, and scaling, the purpose is to remove the background or noise, highlight the text part, and scale the image to a size suitable for processing; feature extraction, the features commonly extracted are: edge features, stroke features, and structural features; recognition, the commonly used recognition methods are: classifiers and random forests.

[0049] It should be noted that CTPN is used to extract text information in this application. The CTPN model is a prior art in this field. This application does not improve it and does not use it as the invention point of this application.

[0050] It is understandable that after the server extracts text information using the CTPN model, the displayed text information is erroneous and does not match the actual text, so further text correction is required.

[0051] 105. According to the spatial transformer network STN model, the text information is corrected in the text image, and an output image is obtained, and the output image has the corrected text.

[0052] The server corrects the text information in the text image according to the spatial transform network (STN) model and obtains an output image with the corrected text.

[0053] The server here uses the STN network model, which performs spatial transformation on the input image, corrects the input image, and outputs a new image. For example, a tilted image is input into the STN network model, and the STN network model adjusts the image to the standard direction and enlarges the image to fill the entire screen. Therefore, the server uses the STN network model to correct the text information. The STN network model consists of three parts: the input module is a network structure that uses the elements of the transformation matrix as output. The transformation matrix can be in any form, for example, the affine transformation matrix is ​​a transformation matrix with a parameter of 6. The network structure can be a variety of network structures terminated by a regression layer, and a fully connected structure or a convolutional structure can be used; the affine module generates a rule for making specific transformations to the input module based on the parameters of the input module. The formula for the affine transformation used here is:

[0054]

[0055] The affine module normalizes the input horizontal and vertical coordinate ranges and the output horizontal and vertical coordinate ranges to [-1, +1] at the same time. The image is cropped, translated, rotated, stretched and distorted to form the output feature map; the acquisition module, after obtaining the corresponding rules and transformation matrix, needs to sample the original feature map to obtain a new feature map, which is the transformed image.

[0056] The present invention inputs the text extracted after bill recognition into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection.

[0057] See also Figure 2Another embodiment of the bill detection method based on deep learning in the embodiment of the present invention includes:

[0058] 201. Obtain the image of the bill to be inspected.

[0059] The server obtains the bill image to be detected. The server obtains the bill image to be processed. The format of the bill image is not limited here. The size of the bill image is the regular size (24CM in length and 14CM in width). The size of the bill image is not limited here.

[0060] It should be noted that the characters that can be recognized in the bill image include: printed characters, such as Chinese printed characters (simplified and traditional), English printed characters, and digital printed characters; handwritten characters, such as Chinese handwritten characters (simplified and traditional), English handwritten characters, and digital handwritten characters; symbols, such as identification codes, barcodes, CheckBox, ╳, ▇, ○, ●; customizable symbols and customizable characters, such as Japanese, Korean, Mongolian, Tibetan, Uyghur, Kazakh, Kirgiz, and Arabic. The recognized characters are not limited here.

[0061] It is understandable that the execution subject of the present invention may be a server or a server, or may be a device having a deep learning network model, which is not specifically limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0062] 202. Normalize the bill image to obtain a normalized image.

[0063] The server normalizes the bill image and obtains the normalized image. The server obtains the pixel values ​​on the bill image; the server normalizes the pixel values ​​and obtains the normalized image.

[0064] The server normalizes the acquired bill images. Specifically, the bill images are normalized, that is, pixel normalization. Normalization does not change the contrast of the image, and at the same time ensures that all pixel values ​​of the normalized images are within the range of [0, 1]. The formula used is as follows:

[0065]

[0066] Where: a' is the normalized pixel value of the bill image, and a is the original pixel value of the bill image.

[0067] It is understandable that the normalization of bill images is mainly for the convenience of data processing. The data is mapped to the range of 0 to 1 for processing, and the processed images or data are then input into the network model, making server calculations more convenient and faster.

[0068] 203. According to the convolutional neural network model, the normalized image is angle-corrected to obtain an adjusted image.

[0069] The server corrects the angle of the normalized image according to the convolutional neural network model and obtains the adjusted image. Specifically, the server obtains the target position coordinates and target coordinate order of the four edges of the normalized image; the server obtains the rotation angle between the target position coordinates and the preset standard coordinates, and the preset standard coordinates are the position coordinates of the four edges of the standard bill image; the server obtains the rotation direction between the target coordinate order and the preset standard order, and the order of obtaining the preset standard order is to obtain it clockwise from the upper left corner of the standard bill image; the server rotates and adjusts the normalized image according to the rotation angle and rotation direction to obtain the adjusted image.

[0070] The server performs angle correction by obtaining the coordinates and coordinate order of the four edges of the bill image. The position coordinates and coordinate order of the four edges of the bill are obtained using a convolutional neural network (CNN). CNN is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. CNN has the ability to represent learning and can classify input information according to its hierarchical structure in a translation-invariant manner. Therefore, CNN can make the input image and the topological structure of the network more consistent. CNN includes VGGNet structure, GoogleNet structure and ResNet structure. The results obtained by the server using the CNN network model are more accurate.

[0071] To further explain, the server uses the target position coordinates and the preset standard coordinates to calculate the rotation matrix, and then calculates the Euler angle to determine the angle that the image needs to be rotated; the server uses the target coordinate order and the preset standard order to calculate the direction in which the image needs to be rotated, and determines whether the rotation direction is clockwise or counterclockwise.

[0072] 204. Calculate the loss value of the adjusted image according to the root mean square RMS function.

[0073] The server calculates the loss value of the adjusted image based on the root mean square RMS function.

[0074] Specifically, the server calculates the loss value of the adjusted image according to the root mean square RMS function, and the formula used is as follows:

[0075]

[0076] Where RMS means root mean square, min means sign function, y means target position coordinate, and y' means preset standard coordinate.

[0077] It is understandable that when the server adjusts the adjusted image, the predicted value of the image coordinates will deviate from the true value to a certain extent. The loss function is used to estimate the degree of inconsistency between the predicted value and the true value. It is a non-negative real-valued function. The smaller the loss function, the better the robustness.

[0078] 205. Use the optimizer to optimize the loss value and optimize the adjusted image.

[0079] The server uses the optimizer to optimize the loss value and optimize the adjusted image.

[0080] After the server obtains the loss function based on forward propagation, it uses the optimizer to optimize the model parameters, thereby optimizing the adjusted image and making the adjusted image more accurate.

[0081] It should be noted that the commonly used optimizers are: batch gradient descent (BGD) optimizer, the BGD optimizer uses the existing parameters to generate an estimated output y' for each input in the training set, and then compares it with the actual output y, counts all errors, and averages them to obtain the average error, which is used as the basis for updating the parameters; stochastic gradient descent (SGD) optimizer, the SGD optimizer randomly extracts a batch of samples and uses them as the basis to update the data parameters for optimization. When the server uses the SGD optimizer to optimize a large data set, it can also converge and optimize at a faster speed; adaptive moment estimation (Adam) optimizer, the Adam optimizer uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of each parameter. The advantage of the server using the Adam optimizer is that after bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable. The optimizer used is not limited here.

[0082] 206. According to the preset network model, the required text information is extracted from the adjusted image to obtain a text image with the text information.

[0083] The server extracts the required text information from the adjusted image according to the preset network model and obtains a text image with text information. The server extracts the text features of the text information from the adjusted image; the server connects the text features in a loop to obtain a text box; the server outputs the text box to obtain a text line and obtains a text image with a text line. The preset network model here is scene text detection (detecting text in natural image with connectionist text proposal network, CTPN). The CTPN network model mainly includes text detection and text recognition. The main function of text detection is to find the text area from the image and separate the text area from the original image; the main function of text recognition is to perform text recognition on the separated image. The main process of text recognition is: preprocessing, image denoising (filtering algorithm), image enhancement, and scaling, the purpose is to remove the background or noise, highlight the text part, and scale the image to a size suitable for processing; feature extraction, the features commonly extracted are: edge features, stroke features, and structural features; recognition, the commonly used recognition methods are: classifiers and random forests.

[0084] It should be noted that the CTPN network model is used to extract text information in the present application. The CTPN network model is the prior art in this field. The present application does not improve it and does not use it as the invention point of the present application.

[0085] It is understandable that after the server extracts text information using the CTPN network model, the displayed text information is erroneous and does not match the actual text, so further text correction is required.

[0086] 207. According to the spatial transformer network STN model, the text information is corrected in the text image, and an output image is obtained, and the output image has the corrected text.

[0087] The server corrects the text information in the text image according to the spatial transformation network STN model, and obtains an output image with the corrected text. Specifically, the server obtains the first text coordinates of the text information; the server obtains the transformation parameters, and the transformation parameters map the coordinate relationship between the first text coordinates and the preset text coordinates, and the preset text coordinates limit the size, direction and position of the corrected text; the server calculates the second text coordinates according to the preset text coordinates and the transformation parameters, and the coordinate positions of the second text coordinates and the first text coordinates correspond to each other; the server obtains the output image with the corrected text according to the first text coordinates and the second text coordinates.

[0088] It should be noted that the server corrects the text by filling the pixel value at the first text coordinate with the pixel value at the second text coordinate. Specifically, the server obtains the first pixel value at the first text coordinate; the server obtains the second pixel value at the second text coordinate; the server fills the first pixel value with the second pixel value according to a preset rule, and obtains an output image with the corrected text.

[0089] It should be noted that the preset text coordinates limit the size, direction and position of the corrected text, and the transformation parameters map the coordinate relationship between the first text coordinates and the preset text coordinates. The role of the transformation parameters is to translate, rotate and scale the adjusted image to transform the adjusted image into the required state. In linear algebra, translation, rotation and scaling can all be achieved using matrix operations, and the transformation parameters in this application are generally in the form of matrices.

[0090] The coordinate positions of the second text coordinates and the first text coordinates correspond to each other. The server calculates the second text coordinates (the position where the text is to be placed) according to the transformation parameters and the preset text coordinates, and multiplies the transformation parameters by the obtained first text coordinates to generate the position coordinates of the ideal bill image. Specifically, the following formula is used:

[0091]

[0092] Where: is the i-th preset coordinate, is the i-th second character coordinate (corresponding to the preset character coordinate), and θ is the preset transformation parameter (generally a matrix).

[0093] It is understandable that after the position coordinates of the ideal adjusted image are calculated, the pixel values ​​on it are not the same as the pixel values ​​on the actual first text coordinates. Therefore, it is necessary to adopt a pixel value filling method to obtain the pixel values ​​on the first text coordinates and fill them into the corresponding pixel values ​​of the second text coordinates according to the preset rules to complete the pixel value filling and generate an output image with corrected text. In the filling process, matrix operations are not required. The preset rules here often use bilinear interpolation to calculate the filling coordinates through this algorithm. Specifically, the following formula is used:

[0094]

[0095] Where: n and m will traverse all coordinates of the first character, U nm is the pixel value at the coordinate of the first character of the ith character, V i is the pixel value at the second text coordinate, max is the sign function, is the i-th second text coordinate.

[0096] For example, the preset text coordinates are (2, 2) and the transformation parameters are

[0097] The calculated coordinates of the second character are (1.6, 2.4) according to the formula:

[0098]

[0099] According to the filling rule of bilinear difference, the pixel values ​​of the four coordinate points (1, 2), (1, 3), (2, 2), and (2, 3) around the first text coordinate (1.6, 2.4) are used to fill the pixel value at (1.6, 2.4) in the second text coordinate. The pixel value to be filled is obtained by the formula, V = U 21 (1-0.6)(1-0.4)+U 22 (1-0.4)(1-0.4)+U 31 (1-0.6)(1-0.6)+U 32 (1-0.4)(1-0.6)It should be noted that the first subscript of U in the above formula is the ordinate, and the second subscript is the abscissa. In this way, the pixel values ​​at the corresponding coordinates are filled and the text information is corrected.

[0100] The present invention inputs the text extracted after bill recognition into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection.

[0101] See also Figure 3 , an embodiment of a bill detection device based on deep learning in an embodiment of the present invention includes:

[0102] The acquisition unit 301 is used to acquire the bill image to be detected;

[0103] The processing unit 302 is used to normalize the bill image to obtain a normalized image;

[0104] An adjustment unit 303 is used to perform angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image;

[0105] An extraction unit 304 is used to extract required text information from the adjusted image according to a preset network model, and obtain a text image with the text information;

[0106] The correction unit 305 is used to correct the text information in the text image according to the spatial transformer network STN model, and obtain an output image with the corrected text.

[0107] In an embodiment of the present invention, an acquisition unit 301 acquires a bill image to be detected; a processing unit 302 normalizes the bill image to obtain a normalized image; an adjustment unit 303 performs angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image; an extraction unit 304 extracts required text information from the adjusted image according to a preset network model to obtain a text image with the text information; a correction unit 305 corrects the text information in the text image according to a spatial transformation network (STN) model to obtain an output image with corrected text.

[0108] The present invention inputs the text extracted after bill recognition into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection.

[0109] See also Figure 4 Another embodiment of the bill detection device based on deep learning in the embodiment of the present invention includes:

[0110] An acquisition unit 301 is used to acquire a bill image to be detected;

[0111] The processing unit 302 is used to normalize the bill image to obtain a normalized image;

[0112] An adjustment unit 303 is used to perform angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image;

[0113] An extraction unit 304 is used to extract required text information from the adjusted image according to a preset network model, and obtain a text image with the text information;

[0114] The correction unit 305 is used to correct the text information in the text image according to the spatial transformer network STN model, and obtain an output image with the corrected text.

[0115] Optionally, the correction unit 305 includes:

[0116] A first acquisition module 3051 is used to acquire the first character coordinates of the text information;

[0117] A second acquisition module 3052 is used to acquire a transformation parameter, wherein the transformation parameter maps the coordinate relationship between the first character coordinate and the preset character coordinate, wherein the preset character coordinate defines the size, direction and position of the corrected character;

[0118] A calculation module 3053, used to calculate a second character coordinate according to the preset character coordinate and the transformation parameter, wherein the coordinate position of the second character coordinate corresponds to the coordinate position of the first character coordinate;

[0119] The output module 3054 is used to obtain the output image with the corrected text according to the first text coordinates and the second text coordinates.

[0120] Optionally, the output module 3054 is specifically used for:

[0121] Obtaining a first pixel value at the first character coordinates;

[0122] Obtain a second pixel value at the second text coordinates;

[0123] The first pixel value is filled into the second pixel value according to a preset rule to obtain the output image with the corrected text.

[0124] Optionally, the processing unit 302 is specifically configured to:

[0125] Obtaining pixel values ​​on the bill image;

[0126] The pixel values ​​are normalized to obtain a normalized image.

[0127] Optionally, the adjusting unit 303 is specifically configured to:

[0128] Obtaining target position coordinates and target coordinate order of four edges of the normalized image;

[0129] Obtaining a rotation angle between the target position coordinates and the preset standard coordinates, wherein the preset standard coordinates are the position coordinates of the four edges of the standard bill image;

[0130] Obtaining the rotation direction between the target coordinate sequence and a preset standard sequence, wherein the preset standard sequence is obtained clockwise from the upper left corner of the standard bill image;

[0131] The normalized image is rotationally adjusted according to the rotation angle and the rotation direction to obtain an adjusted image.

[0132] Optionally, the extraction unit 304 is specifically used for:

[0133] Extracting text features of the text information from the adjusted image;

[0134] Circularly connect the text features to obtain a text box;

[0135] The text box is output to obtain a text line, and a text image with the text line is acquired.

[0136] Optionally, the bill detection device based on deep learning also includes:

[0137] A calculation unit 306, configured to calculate a loss value of the adjusted image according to a root mean square RMS function;

[0138] The optimization unit 307 is used to optimize the loss value using an optimizer to optimize the adjusted image.

[0139] In an embodiment of the present invention, an acquisition unit 301 is used to acquire a bill image to be detected; a processing unit 302 is used to normalize the bill image to acquire a normalized image; an adjustment unit 303 is used to perform angle correction on the normalized image according to a convolutional neural network model to acquire an adjusted image; an extraction unit 304 is used to extract required text information from the adjusted image according to a preset network model to acquire a text image with the text information; a correction unit 305 is used to correct the text information in the text image according to a spatial transformation network STN model to acquire an output image with corrected text; a calculation unit 306 is used to calculate a loss value of the adjusted image according to a root mean square RMS function; an optimization unit 307 is used to optimize the loss value using an optimizer to optimize the adjusted image.

[0140] The present invention inputs the text extracted after bill recognition into the STN network, and performs detailed correction on the text, thereby improving the accuracy of text recognition and the efficiency of bill detection.

[0141] above Figure 3 to Figure 4 The device for detecting bills based on deep learning in an embodiment of the present invention is described in detail from the perspective of modular functional entities. The device for detecting bills based on deep learning in an embodiment of the present invention is described in detail from the perspective of hardware processing.

[0142] Figure 5: is a structural diagram of a deep learning-based bill detection device provided by an embodiment of the present invention. The deep learning-based bill detection device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 501 (for example, one or more processors) and a memory 509, and one or more storage media 508 (for example, one or more massive storage devices) storing application programs 507 or data 506. Among them, the memory 509 and the storage medium 508 can be short-term storage or persistent storage. The program stored in the storage medium 508 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the deep learning-based bill detection device. Furthermore, the processor 501 can be configured to communicate with the storage medium 508 to execute a series of instruction operations in the storage medium 508 on the deep learning-based bill detection device 500.

[0143] The deep learning-based bill detection device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems 505, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the deep learning-based bill detection device shown in the figure does not constitute a limitation of the deep learning-based bill detection device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0144] Combine the following Figure 5 A detailed introduction to the various components of the bill detection equipment based on deep learning:

[0145] The processor 501 is the control center of the bill detection device based on deep learning, and can be processed according to the bill detection method based on deep learning. The processor 501 uses various interfaces and lines to connect the various parts of the entire bill detection device based on deep learning, and solves the problem of low accuracy of Chinese character recognition during bill text detection by running or executing software programs and / or modules stored in the memory 509, and calling data stored in the memory 509, thereby improving the accuracy and efficiency of bill text detection. The storage medium 508 and the memory 509 are both carriers for storing data. In the embodiment of the present invention, the storage medium 508 can refer to an internal memory with a small storage capacity but a fast speed, and the memory 509 can be an external memory with a large storage capacity but a slow storage speed.

[0146] The memory 509 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the deep learning-based ticket detection device 500 by running the software programs and modules stored in the memory 509. The memory 509 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created based on the use of the deep learning-based ticket detection device, etc. In addition, the memory 509 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The deep learning-based ticket detection program and the received data stream provided in the embodiment of the present invention are stored in the memory, and when needed, the processor 501 calls from the memory 509.

[0147] When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0149] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bill detection method based on deep learning, characterized in that: include: Get the bill image that needs to be tested; Normalizing the bill image to obtain a normalized image; According to the convolutional neural network model, the normalized image is angle-corrected to obtain an adjusted image; According to the preset network model, the required text information is extracted from the adjusted image, and a text image with the text information is obtained; According to the spatial transformer network (STN) model, the text information is corrected in the text image to obtain an output image, wherein the output image has the corrected text; The step of performing angle correction on the normalized image according to the convolutional neural network model to obtain the adjusted image includes: Obtaining target position coordinates and target coordinate order of four edges of the normalized image; Obtaining a rotation angle between the target position coordinates and the preset standard coordinates, wherein the preset standard coordinates are the position coordinates of the four edges of the standard bill image; Obtaining the rotation direction between the target coordinate sequence and a preset standard sequence, wherein the preset standard sequence is obtained clockwise from the upper left corner of the standard bill image; According to the rotation angle and the rotation direction, the normalized image is rotated and adjusted to obtain an adjusted image; After performing angle correction on the normalized image according to the convolutional neural network model and obtaining the adjusted image, extracting required text information from the adjusted image according to the preset network model and before obtaining the text image with the text information, the method further includes: Calculate the loss value of the preset standard coordinates and the target position coordinates in the adjusted image according to the root mean square RMS function; The loss value is optimized by using an optimizer to optimize the adjusted image.

2. The bill detection method according to claim 1, characterized in that: The step of correcting the text information in the text picture according to the spatial transformer network (STN) model to obtain an output picture, wherein the output picture has the corrected text, comprises: Obtaining the coordinates of the first character of the text information; Acquire transformation parameters, wherein the transformation parameters map the coordinate relationship between the first character coordinates and the preset character coordinates, wherein the preset character coordinates define the size, direction and position of the corrected character; Calculating second character coordinates according to the preset character coordinates and the transformation parameters, wherein the coordinate positions of the second character coordinates correspond to the coordinate positions of the first character coordinates; The output image with the corrected text is obtained according to the first text coordinates and the second text coordinates.

3. The bill detection method according to claim 2, characterized in that: The step of obtaining the output image with the corrected text according to the first text coordinates and the second text coordinates includes: Obtaining a first pixel value at the first character coordinates; Obtain a second pixel value at the second text coordinates; The first pixel value is filled into the second pixel value according to a preset rule to obtain the output image with the corrected text.

4. The bill detection method according to claim 1, characterized in that: The step of normalizing the bill image to obtain the normalized image includes: Obtaining pixel values ​​on the bill image; The pixel values ​​are normalized to obtain a normalized image.

5. The bill detection method according to claim 1, characterized in that: The extracting required text information from the adjusted picture according to the preset network model to obtain a text picture with the text information comprises: Extracting text features of the text information from the adjusted image; Circularly connect the text features to obtain a text box; The text box is output to obtain a text line, and a text image with the text line is acquired.

6. A bill detection device based on deep learning, characterized in that: include: An acquisition unit, used to acquire a bill image to be detected; A processing unit, configured to normalize the bill image to obtain a normalized image; An adjustment unit, configured to perform angle correction on the normalized image according to a convolutional neural network model to obtain an adjusted image; An extraction unit, configured to extract required text information from the adjusted image according to a preset network model, and obtain a text image with the text information; A correction unit, used for correcting the text information in the text picture according to a spatial transformer network (STN) model, and obtaining an output picture, wherein the output picture has corrected text; The adjustment unit is specifically used to: obtain the target position coordinates and target coordinate order of the four edges of the normalized image; obtain the rotation angle between the target position coordinates and the preset standard coordinates, the preset standard coordinates are the position coordinates of the four edges of the standard bill image; obtain the rotation direction between the target coordinate order and the preset standard order, the preset standard order is to obtain in a clockwise direction starting from the upper left corner of the standard bill image; according to the rotation angle and the rotation direction, the normalized image is rotated and adjusted to obtain the adjusted image; A calculation unit is used to calculate the loss value of the preset standard coordinates and the target position coordinates in the adjusted image according to a root mean square RMS function, and an optimization unit is used to optimize the loss value using an optimizer to optimize the adjusted image.

7. The bill detection device according to claim 6, characterized in that: The correction unit includes: a first acquisition module, used to obtain the first text coordinates of the text information; a second acquisition module, used to obtain transformation parameters, the transformation parameters map the coordinate relationship between the first text coordinates and the preset text coordinates, the preset text coordinates limit the size, direction and position of the corrected text; a calculation module, used to calculate the second text coordinates according to the preset text coordinates and the transformation parameters, the second text coordinates and the coordinate position of the first text coordinates correspond to each other; an output module, used to obtain the output image with the corrected text according to the first text coordinates and the second text coordinates.

8. The bill detection device according to claim 6, characterized in that: The extraction unit is specifically used for: extracting text features of the text information in the adjusted image; cyclically connecting the text features to obtain a text box; outputting the text box to obtain a text line, and acquiring a text image with the text line.

9. A bill detection device based on deep learning, characterized in that: include: A memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the deep learning-based ticket detection device executes the method as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.