Text area recognition method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and is applied to the field of smart government affairs. It provides a text area recognition method, device, equipment and storage medium for improving the recognition accuracy of key text areas in business bills. The text area recognition method includes: performing regional segmentation and direction adjustment on the business bill image to be processed to obtain a preprocessed business bill image; performing text distribution type classification on the preprocessed business bill image through an image classification network based on an attention mechanism to obtain target text distribution type information; performing text box detection on the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box; performing optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information. In addition, the present invention also relates to blockchain technology, and the business bill image to be processed can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image detection, and in particular to a text area recognition method, device, equipment and storage medium. Background Art

[0002] With the rapid development of technologies such as computer vision, deep learning has been applied in more and more fields. However, compared with text recognition in other scenarios, the layout of business bill images is fixed, and its text distribution is very important prior information. Making full use of this information can make the accuracy of business bill text recognition much higher than that of other irregular scene text recognition. Therefore, the recognition technology of business bill images using deep learning has been continuously developing. However, they all have some common problems: they only support a few types of business bills, and their generalization is very limited; due to insufficient experimental data collection methods, they do not support complex scenes or the accuracy is not high after the scene changes. As a result, the complexity of its recognition leads to low recognition accuracy in the key text area of ​​the business bill. Summary of the invention

[0003] The present invention provides a text area recognition method, device, equipment and storage medium, which are used to improve the recognition accuracy of key text areas of business bills.

[0004] A first aspect of the present invention provides a method for identifying a text region, comprising:

[0005] Acquire a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a preprocessed business bill image;

[0006] Using a preset image classification network based on an attention mechanism, the preprocessed business bill image is classified into text distribution types to obtain target text distribution type information;

[0007] By using a preset text detection network, based on the target text distribution type information, text box detection is performed on the preprocessed business bill image to obtain a target text box;

[0008] Optical character recognition is performed on the target text frame through a preset end-to-end text recognition network to obtain text area recognition information.

[0009] Optionally, in a first implementation of the first aspect of the present invention, the pre-processed business document image is classified into text distribution types by using a preset image classification network based on an attention mechanism to obtain target text distribution type information, including:

[0010] Performing matrix vector convolution processing on the preprocessed business bill image through a preset image classification network based on an attention mechanism to obtain image features after convolution processing;

[0011] Acquiring a channel activation value based on the image features after the convolution processing, and recalibrating the original features of the image features after the convolution processing based on the channel activation value to obtain a text distribution type;

[0012] The location information of the text distribution is determined based on the text distribution type to obtain target text distribution type information.

[0013] Optionally, in a second implementation of the first aspect of the present invention, the performing text box detection on the preprocessed business document image based on the target text distribution type information through a preset text detection network to obtain a target text box includes:

[0014] By using a preset text detection network, based on the target text distribution type information, a feature map is extracted from the preprocessed business bill image to obtain a business bill feature map;

[0015] Performing probability map prediction and threshold map prediction on the business bill feature map respectively to obtain a probability map and a threshold map;

[0016] An approximate binary image is calculated based on the probability image and the threshold image, and rotation correction is performed on the approximate binary image to obtain a target text box.

[0017] Optionally, in a third implementation of the first aspect of the present invention, the extracting a feature map of the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a business bill feature map includes:

[0018] By using a preset text detection network, based on the target text distribution type information, feature extraction is performed on the preprocessed business bill image to obtain text distribution image features;

[0019] Upsampling and fusing the text distribution image features to obtain fused image features;

[0020] The fused image features are connected to obtain a business bill feature map.

[0021] Optionally, in a fourth implementation of the first aspect of the present invention, performing optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information includes:

[0022] Through a preset end-to-end text recognition network, convolution processing is performed on the target text box to obtain a text feature map;

[0023] Extracting text sequence features from the text feature graph to obtain a text sequence;

[0024] The text sequence is transcribed character by character to obtain text region identification information.

[0025] Optionally, in a fifth implementation of the first aspect of the present invention, the step of acquiring the business document image to be processed, performing region segmentation and direction adjustment on the business document image to be processed to obtain a preprocessed business document image includes:

[0026] Acquire a business bill image to be processed, and perform image region segmentation on the business bill image to be processed based on a preset image segmentation algorithm to obtain a segmented image;

[0027] By using a preset computer vision software library and based on a preset image processing function, the segmented image is processed to obtain a processed initial image;

[0028] Performing pixel value discrimination on the processed initial image and performing image deletion to obtain a processed candidate image;

[0029] The processed candidate image is rotated in direction to obtain a pre-processed business bill image.

[0030] Optionally, in a sixth implementation of the first aspect of the present invention, the acquiring of the business document image to be processed, and performing image region segmentation on the business document image to be processed based on a preset image segmentation algorithm to obtain a segmented image include:

[0031] Obtain the business bill image to be processed, and determine the threshold based on the preset maximum inter-class variance algorithm;

[0032] By using a preset threshold segmentation algorithm, the business document image to be processed is subjected to image binarization segmentation based on the threshold to obtain a segmented image.

[0033] A second aspect of the present invention provides a text region recognition device, comprising:

[0034] A segmentation and adjustment module is used to obtain a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a pre-processed business bill image;

[0035] A classification module, used to classify the text distribution type of the preprocessed business bill image through a preset image classification network based on an attention mechanism, so as to obtain target text distribution type information;

[0036] A detection module, configured to perform text box detection on the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box;

[0037] The recognition module is used to perform optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information.

[0038] Optionally, in a first implementation of the second aspect of the present invention, the classification module includes:

[0039] A convolution processing unit, used to perform matrix vector convolution processing on the preprocessed business bill image through a preset image classification network based on an attention mechanism, so as to obtain image features after convolution processing;

[0040] A recalibration unit, used for obtaining a channel activation value based on the image features after the convolution processing, and recalibrating the original features of the image features after the convolution processing based on the channel activation value to obtain a text distribution type;

[0041] The determination unit is used to determine the position information of the text distribution based on the text distribution type to obtain the target text distribution type information.

[0042] Optionally, in a second implementation of the second aspect of the present invention, the detection module includes:

[0043] An extraction unit, configured to extract a feature map of the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a business bill feature map;

[0044] A prediction unit, used to perform probability map prediction and threshold map prediction on the business bill feature map to obtain a probability map and a threshold map;

[0045] A calculation correction unit is used to calculate an approximate binary image based on the probability map and the threshold map, and perform rotation correction on the approximate binary image to obtain a target text box.

[0046] Optionally, in a third implementation of the second aspect of the present invention, the extraction unit is specifically used to:

[0047] By using a preset text detection network, based on the target text distribution type information, feature extraction is performed on the preprocessed business bill image to obtain text distribution image features;

[0048] Upsampling and fusing the text distribution image features to obtain fused image features;

[0049] The fused image features are connected to obtain a business bill feature map.

[0050] Optionally, in a fourth implementation of the second aspect of the present invention, the identification module is specifically used to:

[0051] Through a preset end-to-end text recognition network, convolution processing is performed on the target text box to obtain a text feature map;

[0052] Extracting text sequence features from the text feature graph to obtain a text sequence;

[0053] The text sequence is transcribed character by character to obtain text region identification information.

[0054] Optionally, in a fifth implementation manner of the second aspect of the present invention, the segmentation adjustment module includes:

[0055] A segmentation unit, used to obtain a business bill image to be processed, and perform image region segmentation on the business bill image to be processed based on a preset image segmentation algorithm to obtain a segmented image;

[0056] An image processing unit, used to perform image processing on the segmented image based on a preset image processing function through a preset computer vision software library to obtain a processed initial image;

[0057] A deleting unit, used for determining pixel values ​​of the processed initial image and deleting the image to obtain a processed candidate image;

[0058] The rotation unit is used to rotate the processed candidate image to obtain a pre-processed business document image.

[0059] Optionally, in a sixth implementation manner of the second aspect of the present invention, the segmentation unit is specifically configured to:

[0060] Obtain the business bill image to be processed, and determine the threshold based on the preset maximum inter-class variance algorithm;

[0061] By using a preset threshold segmentation algorithm, the business document image to be processed is subjected to image binarization segmentation based on the threshold to obtain a segmented image.

[0062] A third aspect of the present invention provides a text area recognition device, comprising: a memory and at least one processor, wherein the memory stores a computer program; the at least one processor calls the computer program in the memory so that the text area recognition device executes the above-mentioned text area recognition method.

[0063] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned text region recognition method.

[0064] In the technical solution provided by the present invention, a business bill image to be processed is obtained, and the business bill image to be processed is segmented and oriented to obtain a pre-processed business bill image; a text distribution type classification is performed on the pre-processed business bill image through a preset image classification network based on an attention mechanism to obtain target text distribution type information; a text box detection is performed on the pre-processed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box; an optical character recognition is performed on the target text box through a preset end-to-end text recognition network to obtain text area recognition information. In the embodiment of the present invention, through the regional segmentation of the business bill image to be processed, the text distribution type classification of the image classification network based on the attention mechanism, the text box detection of the text detection network, and the optical character recognition based on the end-to-end text recognition network, the business bill text and recognition under complex backgrounds or after scene changes, as well as the reading of indefinite text, are realized, the complexity of identifying a wide variety of business bill text boxes is reduced, and the generalization degree is high, thereby improving the recognition accuracy of key text areas of business bills. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of an embodiment of a method for identifying a text area in an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of another embodiment of a method for identifying a text area in an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of an embodiment of a device for identifying a text area in an embodiment of the present invention;

[0068] Figure 4 It is a schematic diagram of another embodiment of a device for identifying a text area in an embodiment of the present invention;

[0069] Figure 5 The figure is a schematic diagram of an embodiment of a device for recognizing a text area in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention provide a method, device, equipment and storage medium for identifying a text area, which improves the recognition accuracy of key text areas of business documents.

[0071] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] Due to the wide variety of business bills, their recognition is complicated. In addition, the quality of original images varies. In the actual recognition process, there will be problems such as wear, deformity, wrinkles, character overlap, tilt, occlusion, uneven lighting and complex background, resulting in low recognition accuracy of key text areas in business bills. Therefore, the present invention provides a text area recognition method to improve the recognition accuracy of key text areas in business bills.

[0075] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the text area recognition method in the embodiment of the present invention includes:

[0076] 101. Obtain a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a pre-processed business bill image.

[0077] It is understandable that the execution subject of the present invention may be a text area recognition device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0078] The server can be a stand-alone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0079] Among them, the number of business document images to be processed can be one or more, that is, the present invention provides a text area recognition method that can perform text area recognition on one business document image to be processed, and can also perform parallel text area recognition on more than one business document images to be processed.

[0080] Among them, in a feasible implementation method, when the number of business bill images to be processed is more than one, the business bill images to be processed may include more than one bill image of different business bill types, and may also include more than one bill image of the same business bill type.

[0081] After receiving the original business bill image sent by the mobile terminal or uploaded by the recognition system, the server performs image preprocessing on the original business bill image to obtain the business bill image to be processed.

[0082] In a feasible implementation, after receiving the original business bill image sent by the mobile terminal or uploaded by the recognition system, the server performs permission verification and security detection on the original business bill image to ensure the security of the original business bill image, and obtains the business bill image after initial detection; calls a preset bill comparison template (the bill comparison template can be a template corresponding to all business bill types, or a set unified standardized template), and based on the preset comparison requirements, compares and analyzes the business bill image after initial detection to obtain the reviewed business bill image, wherein the preset comparison requirements include whether the image is worn, whether the image is complete, whether the image characters are overlapping, tilted, blocked, etc.; performs image preprocessing on the reviewed business bill image to obtain the business bill image to be processed, wherein the implementation method of the image preprocessing can be various existing technical means of image preprocessing, or can be a set technical means of image preprocessing.

[0083] In a feasible implementation method, after obtaining the business bill image to be processed, the server matches the target bill area template corresponding to the business bill image to be processed; based on the preset area recognition algorithm and the preset bill area template, the business bill image to be processed is identified and segmented to obtain the area image to be identified, and the area image to be identified is detected according to the preset area image direction to obtain the area image to be identified that is different from the preset area image direction, thereby obtaining the area image to be adjusted; the direction of the area image to be adjusted is adjusted according to the preset area image direction to obtain the preprocessed business bill image.

[0084] It should be noted that, in a feasible implementation, the server may call the preset computer vision software library OpenCV to binarize the business invoice image to be processed based on a preset threshold() function to achieve regional segmentation of the business invoice image to be processed.

[0085] 102. Through the preset image classification network based on the attention mechanism, the pre-processed business bill image is classified into text distribution type to obtain the target text distribution type information.

[0086] The server calls a preset image classification network based on the attention mechanism, extracts text distribution features from the preprocessed business bill image, determines the type of the business bill based on the text distribution features, and determines the location information of the text distribution based on the type of the business bill, so as to classify the text distribution type of the preprocessed business bill image, thereby obtaining target text distribution type information. The specific network structure of the image classification network based on the attention mechanism is not limited here.

[0087] By using an image classification network based on the attention mechanism to classify text distribution types, it is possible to ignore irrelevant information and focus on key information, thereby improving the accuracy of text distribution type classification and the quality of target text distribution type information.

[0088] 103. Through a preset text detection network, based on target text distribution type information, text box detection is performed on the preprocessed business document image to obtain a target text box.

[0089] The server calls a preset text detection network, and based on the target text distribution type information, extracts a feature map of the preprocessed business bill image to obtain a feature map, and performs text box recognition and extraction on the feature map, thereby obtaining a target text box, wherein, specifically, the implementation method of performing text box recognition and extraction on the feature map can be to extract text box features from the feature map and output a quasi-binary map or a probability map, thereby obtaining a target text box. The structure of the text detection network in the text area recognition method of the present invention is not limited.

[0090] 104. Through a preset end-to-end text recognition network, optical character recognition is performed on the target text box to obtain text area recognition information.

[0091] The server calls a preset end-to-end text recognition network, extracts a feature map from the target text box to obtain a text feature map, extracts text sequence features from the text feature map to obtain text sequence features, performs character recognition and character conversion on the text sequence features to obtain text region recognition information. The structure of the end-to-end text recognition network in the text region recognition method of the present invention is not limited.

[0092] It should be noted that the text region recognition method provided by the present invention can be applied to the recognition of text regions of business document images of government affairs systems in the field of smart government affairs.

[0093] In the embodiment of the present invention, through the regional segmentation of the business bill image to be processed, the classification of text distribution types based on the image classification network of the attention mechanism, the text box detection of the text detection network, and the optical character recognition based on the end-to-end text recognition network, the business bill text and recognition under complex backgrounds or after scene changes, as well as the reading of irregular texts are realized, and the complexity of identifying a wide variety of business bill text boxes is reduced. The generalization degree is high, thereby improving the recognition accuracy of key text areas of business bills. This solution can be applied to the field of smart government affairs, thereby promoting the construction of smart cities.

[0094] See also Figure 2 Another embodiment of the text area recognition method in the embodiment of the present invention includes:

[0095] 201. Obtain a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a pre-processed business bill image.

[0096] Specifically, the server obtains a business invoice image to be processed, and based on a preset image segmentation algorithm, performs image region segmentation on the business invoice image to be processed to obtain a segmented image; through a preset computer vision software library, based on a preset image processing function, performs image processing on the segmented image to obtain a processed initial image; performs pixel value discrimination on the processed initial image and performs image deletion to obtain a processed candidate image; and performs direction rotation on the processed candidate image to obtain a pre-processed business invoice image.

[0097] After the server obtains the business bill image to be processed, it performs image region segmentation on the business bill image to be processed based on a preset image segmentation algorithm to obtain a segmented image, wherein the image segmentation algorithm in the embodiment of the present invention is preferably a threshold segmentation algorithm. Specifically, the server obtains the business bill image to be processed, and determines the threshold based on the preset maximum inter-class variance algorithm; the preset threshold segmentation algorithm performs image binarization segmentation on the business bill image to be processed based on the threshold to obtain a segmented image. The server divides the business bill image to be processed into foreground and background according to the grayscale characteristics of the image through the preset maximum inter-class variance algorithm, and determines the threshold of regional segmentation through the foreground and background, so that the segmentation with the largest inter-class variance means the minimum probability of misclassification; after the threshold is determined, the threshold is compared with the grayscale value of the pixel point one by one, and the segmentation result directly gives the image area, that is, the preset threshold segmentation algorithm performs image binarization segmentation on the business bill image to be processed based on the threshold to obtain a segmented image, wherein the threshold segmentation algorithm is specifically as follows: T represents the threshold, f(i, j) represents the business bill image to be processed, g(i, j) represents the segmented image, g(i, j) = 1 represents the image element of the business bill (foreground), and g(i, j) = 0 represents the image element of the background. By performing image binary segmentation on the business bill image to be processed based on the threshold, the accuracy of the segmentation of the business bill image region to be processed is improved, and the subsequent recognition of the text region of the business bill image to be processed is facilitated.

[0098] The server calls the preset computer vision software library OpenCV, and performs image processing on the segmented image based on the preset image processing function to obtain the processed initial image. Specifically, decontamination is performed based on the inpaint() function, black / white edge removal is performed based on the floodFill() function, automatic grayscale is performed based on the cvtColor() function, color filtering is performed based on the colorFilter() function, and image resizing is performed based on the resize() function. The above-mentioned decontamination, black / white edge removal, automatic grayscale, color filtering and image resizing can be processed according to a preset processing sequence, for example, decontamination is performed first, then black / white edge removal, then automatic grayscale, then color filtering, and finally image resizing; or the processing can be performed in parallel, for example, decontamination, black / white edge removal, automatic grayscale, color filtering and image resizing are performed in parallel.

[0099] The server obtains the pixel value of the processed initial image, and determines whether the pixel value of the processed initial image is a preset blank pixel value to determine whether the processed initial image is blank. If so, the corresponding processed initial image is deleted, if not, no processing is performed, thereby realizing pixel value discrimination of the processed initial image and image deletion to obtain a processed candidate image; detecting whether the direction of the processed candidate image is a preset direction, if not, calling the computer vision software library OpenCV, and rotating the processed candidate image based on the rotate() function to obtain a preprocessed business invoice image, and if so, determining the corresponding processed candidate image as the preprocessed business invoice image.

[0100] By acquiring a business bill image to be processed, the image region of the business bill image to be processed is segmented based on a preset image segmentation algorithm to obtain a segmented image; through a preset computer vision software library, based on a preset image processing function, image processing is performed on the segmented image to obtain a processed initial image; pixel value discrimination is performed on the processed initial image and image deletion is performed to obtain a processed candidate image; the processed candidate image is rotated in direction to obtain a pre-processed business bill image, thereby unifying and ensuring the quality of the pre-processed business bill image, and facilitating the subsequent recognition of the text region of the pre-processed business bill image.

[0101] 202. Through a preset image classification network based on an attention mechanism, matrix vector convolution processing is performed on the preprocessed business invoice image to obtain image features after convolution processing.

[0102] Preferably, the image classification network based on the attention mechanism in the embodiment of the present invention is Squeeze-and-Excitation Networks (SENet). SENet can automatically obtain the importance of each feature channel by learning, and then enhance the useful features and suppress the features that are not very useful for the current task according to this importance, thereby solving the existing problems: the filters learned by each channel of the general convolutional neural network (CNN) operate on the local receptive field, so each feature map cannot use the context information of other feature maps, and the receptive field size at the lower level of the network is very small, which makes the situation more serious. SENet mainly includes two operations: compression (or decompression) Squeeze and excitation Excitation, which can be applied to any mapping.

[0103] The server performs matrix-vector conversion on the pre-processed business bill image through a preset image classification network based on the attention mechanism to obtain a matrix vector; the server calls a preset convolution operation function to operate on the matrix vector to obtain the image features after convolution processing, wherein the convolution operation function is as follows: After the convolution operation, the matrix dimension of the original input changes, and its output (i.e., the image feature vector after convolution) is U∈H*W*C, which is represented by the vector U=[u1,u2,…,u C ], the convolution kernel is V = [v1,v2,…,v C ],v C represents the Cth convolution kernel, X represents the matrix vector, and is represented by a vector as X∈H'*W'*C', X=[x1,x2,…,x C' ], H'*W'*C' is a matrix vector, H' and W' represent the original data matrix dimensions of the preprocessed business bill image, C' represents the number of feature channels, and * represents the convolution operation. represents a 2D convolution kernel with s channels, x s Indicates the input value of the current channel.

[0104] 203. Obtain channel activation values ​​based on the image features after convolution processing, and recalibrate the original features of the image features after convolution processing based on the channel activation values ​​to obtain the text distribution type.

[0105] The execution process of the server obtaining the channel activation value based on the image features after convolution processing includes: performing global average pooling on the image features after convolution processing to implement the Squeeze operation to obtain the channel features. The Squeeze operation converts each two-dimensional feature channel into a real number. This real number has a global receptive field to some extent, which represents the global distribution of the response on the feature channel; performing dimensionality reduction processing on the channel features and performing activation function processing to implement the Excitation operation to obtain the channel activation value.

[0106] Specifically, the server performs global average pooling on the image features after convolution processing to obtain the channel features, including: performing global average pooling on the image features after convolution processing by a preset calculation formula of channel statistics information to obtain the channel features, wherein the calculation formula of the channel statistics information is specifically as follows: Among them, z C Represents channel-based statistical information, which is formally formed by shrinking U through its spatial dimension H*W. The dimension after pooling is 1*1*C, that is, the channel feature, F sq Indicates the Squeeze operation, u C It represents the image features after convolution processing, and H and W represent the spatial dimensions.

[0107] The server performs dimensionality reduction processing on the channel features and performs activation function processing to obtain a channel activation value, including: calculating the channel activation value based on the channel features through a preset activation value calculation formula, wherein the specific activation value calculation formula is as follows: S = F ex (z,W)=σ(g(z,W))=σ(W2ReLU(W1z), S represents the channel activation value, F ex represents the activation of the Excitation operation, z represents the channel feature, W represents the width of the aforementioned data matrix, and σ represents the sigmoid operation, that is, g represents the gate selection mechanism in the form of sigmoid. It should be noted that, in order to reduce the complexity of the model and improve the generalization ability, a structure containing two fully connected layers (FC) is used here, where the first FC layer plays a role in dimensionality reduction. The dimensionality reduction coefficient is r, which is a hyperparameter. Then, the ReLU activation function is used, that is, f(x) = max(0,x). Therefore, the two sub-matrices split The final FC layer restores the original dimension.

[0108] The server multiplies the channel activation value and the original feature of the image feature after convolution processing to achieve recalibration of the original feature and obtain the calibration value. The text distribution type is determined based on the calibration value. That is, SENet multiplies the learned activation value of each channel (sigmoid activation, value 0~1, that is, channel activation value) by the original feature on U to complete the recalibration of the original feature in the channel dimension F scale Operation, to obtain the calibration value of the original feature on the C channel. Specifically, the channel activation value and the image feature after convolution processing can be calculated through a preset calibration value calculation formula to obtain the calibration value of the original feature on the C channel, and the text distribution type can be determined based on the calibration value, wherein the calibration value calculation formula is as follows: represents the calibration value of the original feature on the C channel, F scale Represents the recalibration operation of the original features in the channel dimension, u C represents the image features after convolution processing, S C Indicates the channel activation value. The calibration value can be determined as the text distribution type, or the corresponding text distribution type can be matched by the calibration value. The channel activation value is obtained based on the image features after convolution processing, and the original features of the image features after convolution processing are recalibrated based on the channel activation value to obtain the text distribution type, thereby improving the efficiency and accuracy of text distribution type recognition.

[0109] 204. Determine the location information of the text distribution based on the text distribution type to obtain target text distribution type information.

[0110] After obtaining the text distribution type through SENet's compression Squeeze operation and activation Excitation operation, the server can search and match from the preset database through the text distribution type to obtain the location information of the text distribution of the business document corresponding to the text distribution type, thereby obtaining the target text distribution type information, which is the location information of the text distribution.

[0111] 205. Using a preset text detection network, based on target text distribution type information, text box detection is performed on the preprocessed business document image to obtain a target text box.

[0112] Specifically, the server uses a preset text detection network to extract feature maps of the preprocessed business invoice image based on target text distribution type information to obtain a business invoice feature map; performs probability map prediction and threshold map prediction on the business invoice feature map to obtain a probability map and a threshold map; calculates an approximate binary map based on the probability map and the threshold map, and performs rotation correction on the approximate binary map to obtain a target text box.

[0113] Preferably, the text detection network in the embodiment of the present invention is Differentiable BinarizationNet (DBNet), which can perform a binarization process in a segmentation network and can adaptively set a binarization threshold, which not only simplifies post-processing but also improves the performance of text detection.

[0114] The server calls DBNet to extract feature maps of the preprocessed business invoice image based on the target text distribution type information to obtain a business invoice feature map. Specifically, the server uses a preset text detection network to extract feature maps of the preprocessed business invoice image based on the target text distribution type information to obtain text distribution image features; upsamples and fuses the text distribution image features to obtain fused image features; and concatenates the fused image features to obtain a business invoice feature map.

[0115] The server predicts a probability map through the business bill feature map, and predicts a threshold map based on the business bill feature map, and calculates an approximate binary map based on the probability map and the threshold map, thereby obtaining an initial text box. The server calls a preset text direction classifier to detect the direction of the initial text box to obtain a text box to be rotated, wherein the text direction classifier is mainly used in scenes where the image is not 0 degrees. In this scene, it is necessary to perform a rotation operation on the text line detected in the image (i.e., the text box to be rotated); perform an affine transformation (rotation correction) on the text box to be rotated to obtain a rotated text box; through the text direction classifier, classify the text angle of the rotated text box according to the preset angle to obtain the target text box, wherein the preset angles include 0 degrees, 45 degrees, 90 degrees, 180 degrees and 270 degrees.

[0116] Through the preset text detection network, based on the target text distribution type information, the feature map of the preprocessed business invoice image is extracted, the probability map prediction and threshold map prediction are performed on the business invoice feature map respectively, the approximate binary map is calculated based on the probability map and the threshold map, and the approximate binary map is rotationally corrected, thereby ensuring the standardization and quality of the target text box, facilitating the subsequent text recognition based on the end-to-end text recognition network, and improving the convenience and accuracy of the recognition of key text areas in the business invoice.

[0117] 206. Perform optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information.

[0118] Specifically, the server performs convolution processing on the target text box through a preset end-to-end text recognition network to obtain a text feature map; extracts text sequence features from the text feature map to obtain a text sequence; and transcribes the text sequence to obtain text area recognition information.

[0119] Preferably, the end-to-end text recognition network in the embodiment of the present invention is a Convolutional Recurrent Neural Network (CRNN), which includes a convolutional layer, a recurrent network layer and a transcription layer, wherein the convolutional layer is an ordinary convolutional neural network (CNN) for extracting feature maps of the input image, and the recurrent network layer is a deep bidirectional long short-term memory (LSTM) network.

[0120] The server calls the convolution layer to perform convolution processing on the target text box to obtain a text feature map; calls the recurrent network layer to extract text sequence features from the text feature map to obtain a text sequence; calls the transcription layer to perform normalized exponential softmax processing on the text sequence and output the corresponding characters to obtain text region recognition information. The text region recognition information includes the text corresponding to the position information of each text distribution of the business bill, which can also be understood as the bill details related to the text region recognition information, such as the bill number, date, final payment amount, as well as the buyer and seller information and the related description of the bill.

[0121] By performing optical character recognition on the target text box through CRNN, the adaptability, precision and efficiency of identifying key text areas in business documents are improved, and the recognition accuracy of key text areas in business documents is improved.

[0122] It should be noted that the text region recognition method provided by the present invention can be applied to the recognition of text regions of business document images of government affairs systems in the field of smart government affairs.

[0123] In the embodiment of the present invention, through the regional segmentation of the business bill image to be processed, the classification of text distribution types based on the image classification network of the attention mechanism, the text box detection of the text detection network, and the optical character recognition based on the end-to-end text recognition network, the business bill text and recognition under complex backgrounds or after scene changes, as well as the reading of irregular texts are realized, and the complexity of identifying a wide variety of business bill text boxes is reduced. The generalization degree is high, thereby improving the recognition accuracy of key text areas of business bills. This solution can be applied to the field of smart government affairs, thereby promoting the construction of smart cities.

[0124] The above describes the text region recognition method in the embodiment of the present invention. The following describes the text region recognition device in the embodiment of the present invention. Figure 3 , an embodiment of a text area recognition device in an embodiment of the present invention includes:

[0125] The segmentation and adjustment module 301 is used to obtain the business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain the pre-processed business bill image;

[0126] The classification module 302 is used to classify the text distribution type of the preprocessed business bill image through a preset image classification network based on the attention mechanism to obtain target text distribution type information;

[0127] The detection module 303 is used to perform text box detection on the pre-processed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box;

[0128] The recognition module 304 is used to perform optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information.

[0129] The functional implementation of each module in the above-mentioned text region recognition device corresponds to each step in the above-mentioned text region recognition method embodiment, and its functions and implementation processes are not described one by one here.

[0130] In the embodiment of the present invention, through the regional segmentation of the business bill image to be processed, the classification of text distribution types based on the image classification network of the attention mechanism, the text box detection of the text detection network, and the optical character recognition based on the end-to-end text recognition network, the business bill text and recognition under complex backgrounds or after scene changes, as well as the reading of irregular texts are realized, and the complexity of identifying a wide variety of business bill text boxes is reduced. The generalization degree is high, thereby improving the recognition accuracy of key text areas of business bills. This solution can be applied to the field of smart government affairs, thereby promoting the construction of smart cities.

[0131] See also Figure 4 Another embodiment of the text area recognition device in the embodiment of the present invention includes:

[0132] The segmentation and adjustment module 301 is used to obtain the business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain the pre-processed business bill image;

[0133] The classification module 302 is used to classify the text distribution type of the preprocessed business bill image through a preset image classification network based on the attention mechanism to obtain target text distribution type information;

[0134] The classification module 302 specifically includes:

[0135] The convolution processing unit 3021 is used to perform matrix vector convolution processing on the pre-processed business bill image through a preset image classification network based on the attention mechanism to obtain the image features after the convolution processing;

[0136] A recalibration unit 3022 is used to obtain a channel activation value based on the image features after the convolution processing, and recalibrate the original features of the image features after the convolution processing based on the channel activation value to obtain a text distribution type;

[0137] A determination unit 3023, configured to determine the position information of the text distribution based on the text distribution type, and obtain the target text distribution type information;

[0138] The detection module 303 is used to perform text box detection on the pre-processed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box;

[0139] The recognition module 304 is used to perform optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information.

[0140] Optionally, the detection module 303 includes:

[0141] The extraction unit 3031 is used to extract a feature map of the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a business bill feature map;

[0142] The prediction unit 3032 is used to perform probability map prediction and threshold map prediction on the business bill feature map to obtain a probability map and a threshold map;

[0143] The calculation correction unit 3033 is used to calculate an approximate binary image based on the probability image and the threshold image, and perform rotation correction on the approximate binary image to obtain a target text box.

[0144] Optionally, the extraction unit 3031 may also be specifically used for:

[0145] Through the preset text detection network, based on the target text distribution type information, feature extraction is performed on the preprocessed business bill image to obtain text distribution image features;

[0146] Upsample and fuse the text distribution image features to obtain fused image features;

[0147] The fused image features are connected to obtain the business bill feature map.

[0148] Optionally, the identification module 304 may also be specifically configured to:

[0149] Through the preset end-to-end text recognition network, the target text box is convolved to obtain a text feature map;

[0150] Extract text sequence features from the text feature graph to obtain a text sequence;

[0151] The text sequence is transcribed character by character to obtain text region identification information.

[0152] Optionally, the segmentation adjustment module 301 includes:

[0153] The segmentation unit 3011 is used to obtain the business bill image to be processed, and perform image region segmentation on the business bill image to be processed based on a preset image segmentation algorithm to obtain a segmented image;

[0154] The image processing unit 3012 is used to perform image processing on the segmented image based on a preset image processing function through a preset computer vision software library to obtain a processed initial image;

[0155] A deleting unit 3013 is used to determine the pixel values ​​of the processed initial image and delete the image to obtain a processed candidate image;

[0156] The rotation unit 3014 is used to rotate the processed candidate image to obtain a pre-processed business document image.

[0157] Optionally, the segmentation unit 3011 may also be specifically used for:

[0158] Obtain the business bill image to be processed, and determine the threshold based on the preset maximum inter-class variance algorithm;

[0159] Through a preset threshold segmentation algorithm, the business bill image to be processed is subjected to image binarization segmentation based on the threshold value to obtain a segmented image.

[0160] The functional implementation of each module and each unit in the above-mentioned text region recognition device corresponds to each step in the above-mentioned text region recognition method embodiment, and their functions and implementation processes are not repeated here one by one.

[0161] In the embodiment of the present invention, through the regional segmentation of the business bill image to be processed, the classification of text distribution types based on the image classification network of the attention mechanism, the text box detection of the text detection network, and the optical character recognition based on the end-to-end text recognition network, the business bill text and recognition under complex backgrounds or after scene changes, as well as the reading of irregular texts are realized, and the complexity of identifying a wide variety of business bill text boxes is reduced. The generalization degree is high, thereby improving the recognition accuracy of key text areas of business bills. This solution can be applied to the field of smart government affairs, thereby promoting the construction of smart cities.

[0162] above Figure 3 and Figure 4 The text region recognition device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The text region recognition device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0163] Figure 5 1 is a schematic diagram of the structure of a text area recognition device provided by an embodiment of the present invention. The text area recognition device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of computer program operations in the text area recognition device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of computer program operations in the storage medium 530 on the text area recognition device 500.

[0164] The text area recognition device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The illustrated text region recognition device structure does not constitute a limitation on the text region recognition device, and may include more or fewer components than illustrated, or may combine certain components, or may have different component arrangements.

[0165] The present invention also provides a text region recognition device, comprising: a memory and at least one processor, wherein a computer program is stored in the memory, and the memory and the at least one processor are interconnected via a line; the at least one processor calls the computer program in the memory so that the text region recognition device performs the steps in the above-mentioned text region recognition method. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is run on a computer, the computer performs the steps in the text region recognition method.

[0166] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0167] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several computer programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0170] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying a text region, characterized in that: The text area recognition method comprises: Acquire a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a pre-processed business bill image; Using a preset image classification network based on an attention mechanism, the preprocessed business bill image is classified into text distribution types to obtain target text distribution type information; The method of performing text distribution type classification on the preprocessed business bill image through a preset image classification network based on an attention mechanism to obtain target text distribution type information includes: Using a preset image classification network based on an attention mechanism, the preprocessed business bill image is classified into a text distribution type to obtain a text distribution type; searching and matching is performed from a preset database using the text distribution type to obtain text distribution position information of the business bill corresponding to the text distribution type; and the text distribution position information is determined as target text distribution type information; By using a preset text detection network, based on the target text distribution type information, text box detection is performed on the preprocessed business bill image to obtain a target text box; The method of performing text box detection on the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box includes: By using a preset text detection network, based on the target text distribution type information, feature extraction is performed on the preprocessed business bill image to obtain text distribution image features; Upsampling and fusing the text distribution image features to obtain fused image features; Connecting the fused image features to obtain a business bill feature map; Performing probability map prediction and threshold map prediction on the business bill feature map respectively to obtain a probability map and a threshold map; Calculating an approximate binary image based on the probability image and the threshold image, and performing rotation correction on the approximate binary image to obtain a target text box; Optical character recognition is performed on the target text frame through a preset end-to-end text recognition network to obtain text area recognition information.

2. The text area recognition method according to claim 1, characterized in that: The pre-processed business bill image is classified into text distribution types through a preset image classification network based on the attention mechanism to obtain text distribution types, including: Through a preset image classification network based on an attention mechanism, matrix vector convolution processing is performed on the preprocessed business bill image to obtain image features after convolution processing; A channel activation value is obtained based on the image features after the convolution processing, and based on the channel activation value, the original features of the image features after the convolution processing are recalibrated to obtain the text distribution type.

3. The text area recognition method according to claim 1, characterized in that: The method of performing optical character recognition on the target text frame through a preset end-to-end text recognition network to obtain text area recognition information includes: Through a preset end-to-end text recognition network, convolution processing is performed on the target text box to obtain a text feature map; Extracting text sequence features from the text feature graph to obtain a text sequence; The text sequence is transcribed character by character to obtain text region identification information.

4. The method for identifying a text area according to any one of claims 1 to 3, characterized in that: The step of acquiring the business bill image to be processed, segmenting the business bill image to be processed and adjusting its direction to obtain a pre-processed business bill image includes: Acquire a business bill image to be processed, and perform image region segmentation on the business bill image to be processed based on a preset image segmentation algorithm to obtain a segmented image; By using a preset computer vision software library and based on a preset image processing function, the segmented image is processed to obtain a processed initial image; Performing pixel value discrimination on the processed initial image and performing image deletion to obtain a processed candidate image; The processed candidate image is rotated in direction to obtain a pre-processed business bill image.

5. The text area recognition method according to claim 4, characterized in that: The step of acquiring a business document image to be processed and performing image region segmentation on the business document image to be processed based on a preset image segmentation algorithm to obtain a segmented image includes: Obtain the business bill image to be processed, and determine the threshold based on the preset maximum inter-class variance algorithm; By using a preset threshold segmentation algorithm, the business document image to be processed is subjected to image binarization segmentation based on the threshold to obtain a segmented image.

6. A text area recognition device, characterized in that: The text area recognition device comprises: A segmentation and adjustment module is used to obtain a business bill image to be processed, perform region segmentation and direction adjustment on the business bill image to be processed, and obtain a pre-processed business bill image; A classification module is used to classify the text distribution type of the preprocessed business bill image through a preset image classification network based on an attention mechanism to obtain target text distribution type information; wherein, the text distribution type classification of the preprocessed business bill image through a preset image classification network based on an attention mechanism to obtain target text distribution type information includes: classifying the text distribution type of the preprocessed business bill image through a preset image classification network based on an attention mechanism to obtain the text distribution type; searching and matching from a preset database through the text distribution type to obtain the position information of the text distribution of the business bill corresponding to the text distribution type; determining the position information of the text distribution as the target text distribution type information; A detection module, used to perform text box detection on the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box; wherein, the text box detection on the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain a target text box includes: extracting features of the preprocessed business bill image based on the target text distribution type information through a preset text detection network to obtain text distribution image features; upsampling and fusing the text distribution image features to obtain fused image features; connecting the fused image features to obtain a business bill feature map; performing probability map prediction and threshold map prediction on the business bill feature map to obtain a probability map and a threshold map; calculating an approximate binary map based on the probability map and the threshold map, and performing rotation correction on the approximate binary map to obtain a target text box; The recognition module is used to perform optical character recognition on the target text box through a preset end-to-end text recognition network to obtain text area recognition information.

7. A text area recognition device, characterized in that: The text region recognition device comprises: a memory and at least one processor, wherein a computer program is stored in the memory; The at least one processor calls the computer program in the memory to enable the text region recognition device to perform the text region recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for recognizing a text area according to any one of claims 1 to 5 is implemented.

Images