Optical character recognition method based on improved OCR network

Abstract

The application discloses an optical character recognition method based on an improved OCR network, and comprises the following steps: obtaining basic data, and dividing a training set and a test set; sending data in the training set into the improved OCR network for training, and obtaining an optimal network through multiple iterations until a loss value is minimum; inputting data in the test set into the optimal network obtained through training to obtain prediction information; then, using NMS and setting a threshold to screen out detection frames meeting conditions from the prediction information; finally, drawing the finally obtained detection frames on an original image, and marking text recognition information in a top-left corner region of the detection frames. The application can realize accurate recognition of optical characters, and meet the requirement of improving accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical character recognition, and particularly discloses an optical character recognition method based on an improved OCR network. BACKGROUND

[0002] Optical character recognition (OCR) technology has a rich application scenario, such as license plate recognition, bank card information recognition, and identity card information recognition. The application of OCR technology can greatly reduce the labor cost and improve the efficiency. However, the recognition accuracy is still low when the current OCR technology faces difficult data such as irregular shape text, background interference, and perspective transformation. In recent years, a part of research work has adopted a graph convolutional network (GCN) capable of processing local vision and semantic context to optimize the OCR processing result and improve the recognition accuracy for the above difficult data. However, the current OCR model based on GCN does not explicitly optimize the generalization performance of the model, and the structure of the OCR model based on GCN is still artificially designed, which has the problems of suboptimality and low efficiency.

[0003] In summary of the above discussion, it has high practical application value to invent a method for improving the feature expression capability of the OCR model based on GCN. SUMMARY

[0004] The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides an optical character recognition method based on an improved OCR network, which can realize accurate recognition of optical characters and meet the requirement of improving the accuracy.

[0005] To achieve the above object, the technical scheme provided by the application is: an optical character recognition method based on an improved OCR network, the improved OCR network being an improvement on a feature extraction module, a feature enhancement module and a prediction module of a traditional OCR network; the improvement on the feature extraction module is: using an adjacency matrix prediction module (ADJ module) to replace a text direction offset prediction module (TDO module), the ADJ module being a series of convolution operations, predicting connection information between nodes through a specific loss function and outputting adjacency matrix information, and further adding a feature map module (FEA module), the FEA module also being a series of convolution operations, used for extracting image feature information and outputting feature map information, the improved feature extraction module outputting text position information, adjacency matrix information, feature map information and text box position information; the improvement on the feature enhancement module is: establishing a graph convolution network after the feature map, dividing the feature map into nodes with a size of 16x16, flattening the nodes, taking out the 16x16 size matrix one by one in a row-first order and arranging them into a one-dimensional vector, the size of the vector being 256, and the adjacency matrix information output by the preceding ADJ module also being input into the graph convolution network to strengthen the extraction of the correlation information between nodes, the graph convolution network outputting a series of node features; the improvement on the prediction module is: constructing a decoder to sample and decode the enhanced node features and the text position features output by the feature extraction module, used for replacing the character prediction channel to perform character prediction and obtain a final result;

[0006] The specific implementation of the optical character recognition method comprises the following steps:

[0007] 1) Obtain basic data, including images with English text and perform labeling, mark the text box and corresponding text, so that the labeled data has text box and text information, then divide the data into a training set and a test set, respectively used for network training and testing;

[0008] 2) send the data in the training set into the improved OCR network for training, during the training, obtain the feature information of the image through the feature extraction module, input the extracted feature information into the subsequent prediction module for integration and output, and obtain the text prediction result of the image; in the back propagation, calculate the total loss value through the joint loss of the feature extraction module and the feature enhancement module, wherein the feature extraction module is to calculate the distance error of the center point position and the text box position, the feature enhancement module is to calculate the mean square error of the adjacency matrix and the connection time sequence classification loss of the final text recognition, and the total error is obtained by weighted summation; after multiple iterations until the loss value is minimum, the optimal network is obtained;

[0009] 3) inputting data in the test set into the trained optimal network to obtain prediction information, then screening detection boxes meeting the conditions from the prediction information using NMS and setting a threshold, drawing the finally obtained detection boxes on the original image, and marking text recognition information in the upper left corner region of the detection boxes.

[0010] Further, the step 1) comprises the following steps:

[0011] 1.1) image loading preprocessing: using the CTW-1500 dataset, the CTW-1500 dataset is preprocessed, and images without labels are removed;

[0012] 1.2) Image label processing: For the label of the CTW-1500 dataset, it is divided into text location prediction label, text box prediction label, text character classification map prediction label and text direction offset prediction label. The text location prediction label and the text box prediction label are retained, and the text direction offset prediction label is processed and converted into an adjacency matrix label. The image is divided into multiple 16x16 matrices, each matrix is taken as a node, and the connection information between the nodes is obtained by decomposing the vector information of the text direction offset prediction label. Edges are added between the nodes through the connection information between the nodes. The text direction offset prediction label is a three-dimensional matrix with a size of (H, W, 3). The first dimension has a size of H, indicating the image height. The second dimension has a size of W, indicating the image width. The third dimension has a size of 3, i.e. the channel size is 3. The information contained in the three channels is as follows: the first channel records the horizontal direction information of the direction vector. For each pixel point, greater than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the right, and less than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the left. The second channel records the vertical direction information of the direction vector. For each pixel point, greater than 0 indicates that the vertical component of the position direction vector of the pixel point is upward, and less than 0 indicates that the horizontal component of the position direction vector of the pixel point is downward. The third channel indicates the relative height of the character. When processing the text direction offset prediction label, the information used is the horizontal direction information of the direction vector and the vertical direction information of the direction vector, corresponding to the first two channels in the third dimension. For the pixels in the node range, the average direction vector is calculated, and then the position of the next node connected by the node is determined according to the obtained direction vector. The specific operation is as follows: according to the component size, the current node and the up, down, left and right nodes are connected. If the horizontal component is greater than 0, the node to the right of the current node is connected. If the horizontal component is less than 0, the node to the left of the current node is connected. If the vertical component is greater than 0, the node above the current node is connected. If the vertical component is less than 0, the node below the current node is connected. After judging the connection relationship of the up, down, left and right nodes, the nodes in the diagonal direction also need to be connected. According to the direction of the average vector of the current node, the current node and the nodes below left, above left, below right and above right are connected. The connection form is in the form of an adjacency matrix. The size of the adjacency matrix is (num, num), and num is the number of nodes, which is calculated as num = H / / 16 x W / / 16. When the corresponding coordinate (i, j) of the adjacency matrix is 1, it indicates that the i-th node has an edge connected to the j-th node. The calculation of the adjacency matrix label is as follows: the entire three-dimensional matrix corresponding to the text direction offset prediction label needs to be traversed.

[0013] First, get the coordinates (x, y) of each pixel point, where x is the horizontal coordinate and y is the vertical coordinate. Calculate the magnitude of the horizontal and vertical components at that point. If the absolute value of the direction vector corresponding to the coordinate point is less than a preset threshold, it is not processed. If it is greater than the threshold, calculate the adjacency matrix label. The magnitude of the horizontal and vertical components is calculated as follows:

[0014] x _direct =D map [x,y,0]

[0015] y _direct =D map [x,y,1]

[0016] Where x _direct Indicates the magnitude of the horizontal component, y _direct Indicates the magnitude of the vertical component, D map Represents the text direction offset prediction label;

[0017] Then, read the node number corresponding to the pixel:

[0018] pos=x / / 16+y / / 16×W / / 16

[0019] Where, pos is the node number;

[0020] Finally, process the horizontally adjacent nodes, vertically adjacent nodes, and the merged vector direction adjacent nodes, set the adjacency matrix values ​​of the corresponding positions to 1, and record the adjacency matrix as A;

[0021] Processing horizontally connected nodes:

[0022] x _sub =1if x _direct >0else x _sub =-1

[0023] A[pos,pos+x _sub ]=1

[0024] Where x _sub Indicates the horizontal node offset;

[0025] Processing vertical adjacent nodes:

[0026] y _sub =-1if y _direct >0else y _sub =1

[0027] A[pos,pos+y _sub ×W / / 16]=1

[0028] Where y _subIndicates the vertical node offset;

[0029] Processing vector direction adjacent nodes:

[0030] A[pos,pos+x _sub +y _sub ×W / / 16]=1

[0031] 1.3) Divide the dataset: After the image labeling is completed, the image and label files are divided into training and test sets in proportion.

[0032] Furthermore, in step 2), the text position prediction label, text box prediction label, and adjacency matrix label of the training set play different roles in the entire network and are used to train the network. The image information of the training set is input into the improved OCR network, features are extracted in the feature extraction module, and the features are input into the graph neural network to enhance and extract text information. Finally, character prediction is made, which includes the following steps:

[0033] The images of the training set are input into the feature extraction module. The feature extraction module includes a deep residual network Resnet50, a feature extraction pyramid FPN, and four branch modules. The four branch modules include a text centerline prediction module, namely the TCL module, a text edge offset prediction module, namely the TBO module, a FEA module, and an ADJ module. Among them, the TCL module is used to locate the position of the text in the image, and its label is the position of the text center point in the image. The TBO module is used to predict the position of the text box, and its label is the position and offset angle parameters of the four vertices of the text box. The FEA module uses a series of convolutions to extract image information as the input of the graph neural network. The ADJ module learns the connection information between nodes in a supervised manner and uses a series of convolutions to extract image information. A sigmoid layer needs to be added later to limit the output size between 0 and 1. After the image information passes through Resnet50 and FPN, the deep feature information of the image is obtained, and then the deep feature information passes through the four branch modules to obtain four differentiated image feature information;

[0034] The feature enhancement module uses a feature map and an adjacency matrix to construct a graph neural network, and the feature map and the adjacency matrix correspond to the outputs of the FEA module and the ADJ module respectively, wherein the output of the ADJ module needs to be converted into an adjacency matrix, and the specific conversion method is consistent with the processing in image labeling, that is, an adjacency matrix is converted, which is denoted as B, and each image has a corresponding unique graph structure, and using a specific graph structure can improve the accuracy of the OCR network. The graph neural network uses two layers of GCN to aggregate the features of adjacent nodes, and the output dimension is 64. The feature map information is enhanced: first, the feature map is divided into multiple nodes, each node is a 16x16 matrix, and the size of the feature map is (H, W, 3), H is the height of the image, and W is the width of the image. First, perform one round of convolution on the feature map to change the size of the third dimension to 1, and obtain a matrix with a size of (H, W, 1). Then, for each node divided, the node is flattened, and the 16x16 matrix is taken out one by one in row-major order and arranged into a one-dimensional vector with a size of 256. These vectors are input into the graph convolution, and the input adjacency matrix needs to be processed. A parameter is set as a threshold, and all values greater than the threshold are recorded as 1, otherwise as 0. Then, the graph neural network is converted into an undirected graph, that is, the connected nodes are changed to bidirectional connection at the adjacency matrix level. The graph neural network uses two layers of GCN, and the dimension of the first layer is 128 and the dimension of the second layer is 64. After the graph neural network, a series of graph node vectors are obtained.

[0035] The node features predicted by the graph neural network are first sampled and then input into the Decoder for decoding. TCL_map is the output of the TCL module and is a matrix with a size of (H, W, 1). The values in the matrix are between 0 and 1, representing the probability that a pixel point is text. The valid character corresponding node is obtained from the matrix. For coordinate point (x, y), if the value of the corresponding position of TCL_map is greater than a given threshold, then the node corresponding to the coordinate is determined to be a valid character node. The corresponding node number is calculated as follows:

[0036] pos=x / / 16+y / / 16×W / / 16

[0037] The dimension number of the input to the Decoder is reduced by obtaining the valid node position. The Decoder is composed of RNN, fully connected layer and softmax. The data input to the Decoder is the node feature obtained after sampling, and finally the character alignment is performed by CTC to obtain the output text information.

[0038] The loss function of the improved OCR network is improved, loss weighting is performed on the TCL module, the TDO module, the ADJ module and the Decoder, wherein the Dice loss is used for calculating the TCL module and the ADJ module, and the loss of the TCL module and the loss of the ADJ module are respectively denoted as Tcl_Loss and Adj_Loss; the loss calculation in the traditional OCR network is used for the TDO module, the boundary is finely controlled, and the loss of the TDO module is denoted as Tdo_Loss; the CTC loss is used for the Decoder to perform loss calculation on data, so that the output data is aligned, and the loss of the Decoder is denoted as CTC_Loss; and the final loss is Loss, and finally the loss function is:

[0039] Loss = lambda 1 * Tcl_Loss + lambda 2 * Tdo_Loss + lambda 3 * Adj_Loss + lambda 4 * CTC_Loss

[0040] In the formula, lambda 1, lambda 2, lambda 3 and lambda 4 are hyperparameters.

[0041] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0042] 1. The present application adds a graph convolution network in the feature enhancement module, trains the relationship between the sampling node and the adjacent node, thereby enhancing important features and suppressing invalid features, can efficiently extract text features from the input image, and overcomes the data difficulty problems such as background interference and perspective transformation.

[0043] 2. The present application provides a method for constructing an adjacency matrix by predicting a text direction offset label, which can be effectively applied to the prediction of a graph neural network, reduce the calculation cost, and construct an adjacency matrix based on direction information. At the same time, the method makes each image have a corresponding unique graph structure, and using a specific graph structure can improve the precision of the OCR network.

[0044] 3. The present application proposes an ADJ module and an FEA module, which respectively acquire the features of the image through a series of convolution operations. The ADJ module acquires the connection features between nodes through multi-layer convolution, and the FEA module acquires the global features of the image through multi-layer convolution. The construction of the two modules can mine multi-scale features and strengthen the attention to the spatial connection features between texts.

[0045] 4. The method for sampling the graph nodes through text position information in the present application can effectively connect the graph convolution network in the feature enhancement module and the Decoder of the prediction module, sample and decode the enhanced node features and the text position features output by the feature extraction module, and is used for replacing the character prediction channel to perform character prediction, thereby significantly improving the character prediction ability. BRIEF DESCRIPTION OF DRAWINGS

[0046] Fig. 1 The overall architecture diagram of the method of the present application; in the figure, GCN is a graph convolution module, Decoder is a graph node decoder module, gather represents a sampling operation, image represents an input image, and text represents an output predicted text.

[0047] Fig. 2 The structural diagram of the FEA module and the ADJ module implemented in the present application; in the figure, FPN_map is the output of the FPN, Feature_map is the output of the FEA module, Adj_map is the output of the ADJ module, Conv2d is a convolution operation, 1x1 and 3x3 appearing represent the convolution kernel size, and simgoid is a function module that maps data to the range of (0, 1).

[0048] Fig. 3 The adjacency matrix label information generation graph implemented in the present application; in the figure, D map The input text direction offset prediction label, three channels are respectively for horizontal direction judgment, vertical direction judgment and diagonal direction judgment, Pos is the calculation of the current position corresponding node number, the calculated node number is recorded as pos, Connect is a connection node operation, the corresponding connection is a one-way connection, which is embodied as a directed graph. DETAILED DESCRIPTION

[0049] The present application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0050] As Figs. 1 to 3As shown, the embodiment discloses an optical character recognition method based on an improved OCR network, which is improved in feature extraction module, feature enhancement module and prediction module of the traditional OCR network; the improvement in the feature extraction module is: the text direction offset prediction module (TDO module) is replaced by the adjacency matrix prediction module (ADJ module), the ADJ module is a series of convolution operations, the connection information between nodes is predicted through a specific loss function, and the adjacency matrix information is output, in addition, the feature map module (FEA module) is added, the FEA module is also a series of convolution operations, which is used for extracting image feature information and outputting feature map information, the improved feature extraction module will output text position information, adjacency matrix information, feature map information and text box position information; the improvement in the feature enhancement module is: a graph convolution network is established after the feature map, the feature map is divided into nodes with a size of 16*16, the nodes are flattened, and the 16*16 size matrix is taken out one by one in row priority order and arranged into a one-dimensional vector, the size of the vector is 256, the adjacency matrix information output by the ADJ module in front is also input into the graph convolution network to strengthen the correlation information extraction between nodes, and the graph convolution network will output a series of node features; the improvement in the prediction module is: a decoder is constructed, the enhanced node features and the text position features output by the feature extraction module are sampled and decoded, which is used for replacing the character prediction channel to predict characters, and the final result is obtained;

[0051] The specific implementation of the optical character recognition method comprises the following steps:

[0052] 1) Obtain basic data, including images with English characters and perform labeling, mark the text box and corresponding text, so that the labeled data has text box and text information, then divide the data into training set and test set for network training and testing, comprising the following steps:

[0053] 1.1) Image loading preprocessing: adopt CTW-1500 dataset, and perform preprocessing on the CTW-1500 dataset, and remove the images without labels;

[0054] 1.2) Image label processing: For the label of the CTW-1500 dataset, it is divided into text location prediction label, text box prediction label, text character classification map prediction label and text direction offset prediction label, the text location prediction label and the text box prediction label are reserved, and then the text direction offset prediction label is processed and converted into an adjacency matrix label, the image is divided into multiple 16x16 matrices, each matrix is taken as a node, the connection information between the nodes is obtained by decomposing the vector information of the text direction offset prediction label, and edges are added between the nodes through the connection information between the nodes, wherein the text direction offset prediction label is a three-dimensional matrix with a size of (H, W, 3), the matrix has three dimensions, the size of the first dimension is H, indicating the image height, the size of the second dimension is W, indicating the image width, and the size of the third dimension is 3, that is, the channel size is 3, and the information contained in the three channels is as follows: the first channel records the horizontal direction information of the direction vector, for each pixel point, greater than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the right, and less than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the left, the second channel records the vertical direction information of the direction vector, for each pixel point, greater than 0 indicates that the vertical component of the position direction vector of the pixel point is upward, and less than 0 indicates that the horizontal component of the position direction vector of the pixel point is downward, and the third channel indicates the relative height of the character; when processing the text direction offset prediction label, the information used is the horizontal direction information of the direction vector and the vertical direction information of the direction vector, corresponding to the first two channels in the third dimension; for the pixels in the node range, the average direction vector is calculated, and then the position of the next node connected by the node is judged according to the obtained direction vector, and the specific operation is as follows: the current node and the up, down, left and right nodes are connected according to the component size, if the horizontal component is greater than 0, the node to the right of the current node is connected, and if the horizontal component is less than 0, the node to the left of the current node is connected, if the vertical component is greater than 0, the node above the current node is connected, and if the vertical component is less than 0, the node below the current node is connected; after judging the connection relationship of the up, down, left and right nodes, the nodes in the diagonal direction also need to be connected, the current node and the left lower, left upper, right lower and right upper nodes are connected according to the direction of the average vector of the current node, and the connection form is in the form of an adjacency matrix, the size of the adjacency matrix is (num, num), num is the number of nodes, and is calculated as num = H / / 16xW / / 16, when the corresponding coordinates (i, j) of the adjacency matrix are 1, it indicates that the i-th node has an edge connected to the j-th node, wherein the adjacency matrix label is calculated as follows, and the entire three-dimensional matrix corresponding to the text direction offset prediction label needs to be traversed:

[0055] First, the coordinate point (x, y) of each pixel point is acquired, x is the horizontal coordinate, y is the vertical coordinate, the size of the horizontal and vertical component at the point is calculated, if the absolute value of the direction vector of the corresponding coordinate point is less than a preset threshold, it is not processed, if it is greater than the threshold, the calculation of the adjacency matrix label is performed, and the size of the horizontal and vertical components is calculated as follows:

[0056] x _direct =D map [x,y,0]

[0057] y _direct =D map [x,y,1]

[0058] In the formula, x _direct represents the size of the horizontal component, y _direct represents the size of the vertical component, and D map represents the text direction offset prediction label.

[0059] Then, the node number corresponding to the pixel point is read:

[0060] pos=x / / 16+y / / 16×W / / 16

[0061] In the formula, pos is the node number;

[0062] Finally, the horizontal adjacent node, the vertical adjacent node and the combined vector direction adjacent node are processed, the adjacency matrix value of the corresponding position is set to 1, and the adjacency matrix is denoted as A.

[0063] Process the horizontal adjacent node:

[0064] x _sub =1if x _direct >0else x _sub =-1

[0065] A[pos,pos+x _sub ]=1

[0066] In the formula, x _sub represents the horizontal node offset;

[0067] Process the vertical adjacent node:

[0068] y _sub =-1if y _direct >0else y _sub =1

[0069] A[pos,pos+y _sub ×W / / 16]=1

[0070] In the formula, y _subrepresents a vertical node offset;

[0071] Processing vector direction adjacent nodes:

[0072] A[pos,pos+x _sub +y _sub xW / / 16] = 1

[0073] 1.3) Dividing the data set: after the image label processing is completed, the image and the label file are divided into a training set and a test set in proportion.

[0074] 2) The data in the training set is sent into the improved OCR network for training. When training, the feature information of the image is obtained through the feature extraction module, the extracted feature information is input into the subsequent prediction module for integration and output, and the text prediction result of the image is obtained. In the back propagation, the total loss value is calculated through the joint loss of the feature extraction module and the feature enhancement module, wherein the feature extraction module is to calculate the distance error of the center point position and the text box position, the feature enhancement module is to calculate the mean square error of the adjacency matrix and the connection time sequence classification loss of the final text recognition, and the total error is obtained by weighted summation. After multiple iterations until the loss value is minimum, the optimal network is obtained.

[0075] The image in the training set is input into the improved feature extraction module, which includes a deep residual network Resnet50, a feature extraction pyramid FPN, and four branch modules. First, the input image is subjected to feature extraction by the deep residual network Resnet50 and the feature extraction pyramid FPN. After the image passes through the Resnet built in the application, a 7-layer feature matrix is output. The first layer is the original matrix, denoted as c0, the second layer is the matrix after one convolution, denoted as c1, and the last five layers are the feature matrices obtained after 1, 2, 3, 4, and 5 times of feature extraction, respectively, denoted as c2, c3, c4, c5, and c6. The obtained 7-layer matrix enters the FPN and is divided into two parts. The first three layers of feature matrix are subjected to convolution, addition, and pooling operations to obtain the upper feature map F_up. The last five layers are subjected to convolution, addition, and pooling operations to obtain the lower feature map F_down. Finally, the upper feature map and the lower feature map are added to obtain the feature map FPN_map. The four branch modules include a text center line prediction module (TCL module), a text border offset prediction module (TBO module), an FEA module, and an ADJ module. The TCL module is used to locate the position of the text in the image, and its label is the position of the text center point in the image. The TBO module is used to predict the position of the text box, and its label is the position of the four vertices of the text box and the offset angle parameter. The FEA module uses a series of convolutions to extract image information as the input of the graph neural network. The ADJ module learns the connection information between nodes under supervision and uses a series of convolutions to extract image information. A sigmoid layer needs to be added at the end to limit the output size to between 0 and 1. Each module is composed of 4 layers of convolution and pooling, with convolution kernel sizes of 1x1 and 3x3 alternating. A sigmoid layer needs to be added at the end of the TCL and ADJ modules to limit the output size to between 0 and 1. Different information needed is extracted from the four branch modules and input into the feature enhancement module.

[0076] The feature enhancement module uses a feature map and an adjacency matrix to construct a graph neural network, and the feature map and the adjacency matrix correspond to the outputs of the FEA module and the ADJ module respectively, wherein the output of the ADJ module needs to be converted into an adjacency matrix, and the specific conversion method is consistent with the processing in image labeling, that is, an adjacency matrix is converted, which is denoted as B, and each image has a corresponding unique graph structure, and using a specific graph structure can improve the accuracy of the OCR network. The graph neural network uses two layers of GCN to aggregate the features of adjacent nodes, and the output dimension is 64. The feature map information is enhanced: first, the feature map is divided into multiple nodes, each node is a 16x16 matrix, and the size of the feature map is (H, W, 3), H is the image height, and W is the image width. First, perform one round of convolution on the feature map to change the size of the third dimension to 1, and obtain a matrix with a size of (H, W, 1). Then, for each node divided, the node is flattened, and the 16x16 matrix is taken out one by one in row-major order and arranged into a one-dimensional vector with a size of 256. Input these vectors into the graph convolution, and the input adjacency matrix needs to be processed. A parameter is set as a threshold, and all values greater than the threshold are recorded as 1, otherwise as 0. Then, the graph neural network is converted into an undirected graph, that is, the connected nodes are changed to bidirectional connection at the adjacency matrix level. The graph neural network uses two layers of GCN, the dimension of the first layer is 128, and the dimension of the second layer is 64. After the graph neural network, a series of graph node vectors are obtained.

[0077] The node features predicted by the graph neural network are first sampled and then input into the Decoder for decoding. TCL_map is the output of the TCL module and is a matrix with a size of (H, W, 1). The values in the matrix are between 0 and 1, representing the probability that a pixel point is text. The valid character corresponding node is obtained from the matrix. For the coordinate point (x, y), if the value of the corresponding position of TCL_map is greater than a given threshold, then the node corresponding to the coordinate is determined to be a valid character node. The corresponding node number is calculated as follows:

[0078] pos=x / / 16+y / / 16×W / / 16

[0079] The dimension number of the input Decoder is reduced by obtaining the valid node number. Finally, a CTC is used for character alignment to obtain the output text information. The sampling method of TCL_map is as follows:

[0080] First, process the values in TCL_map. Set a threshold, and record the values greater than the threshold as 1, and otherwise as 0.

[0081] The matrix is marked with connected components, the meaning of connected components is that there is a path between all points, and the marking method here is: starting from any pixel point of the matrix, marking all pixel points, each connected component has a different serial number. Return the number K of marked connected components and the marked matrix, the number of connected components represents the number of predicted characters;

[0082] The directed graph adjacency matrix is traversed, topological sorting is performed, all nodes are traversed, the corresponding node connected component is recorded, and the node is added to the sorted list, and finally a node sequence is generated.

[0083] After the above processing, a node sequence is generated, and the graph nodes are taken according to the sequence, and the sampled nodes are obtained as the input of the decoder.

[0084] The decoder is composed of a basic RNN, a full connection layer and a softmax, the data input to the decoder is the node feature obtained after sampling, and the final output dimension is (K, 37), 37 is the prediction classification, representing the probability of 37 characters. The output of the decoder is the prediction result.

[0085] The training process uses a weighted loss function, and the loss of the TCL module, the TDO module, the ADJ module and the decoder is weighted, wherein the TCL module and the ADJ module use Dice loss for calculation, and are respectively denoted as Tcl_Loss and Adj_Loss, which represent the loss of the TCL module and the loss of the ADJ module; the TDO module uses the loss calculation in the traditional OCR network, and controls the boundary in detail, and is denoted as Tdo_Loss, which represents the loss of the TDO module; the decoder uses CTC loss to calculate the loss of the data, and aligns the output data, and is denoted as CTC_Loss, which represents the loss of the decoder; the final loss is Loss, and finally the loss function is:

[0086] Loss = λ1 x Tcl_Loss + λ2 x Tdo_Loss + λ3 x Adj_Loss + λ4 x CTC_Loss

[0087] In the formula, λ1, λ2, λ3 and λ4 are hyperparameters, λ1, λ2 and λ3 are initially set to 1, and λ4 is set to 0.5.

[0088] 3) input the data in the test set into the optimal network obtained by training to obtain prediction information, then use NMS and set threshold to screen out detection boxes meeting the conditions from the prediction information, and finally draw the obtained detection boxes on the original image, and mark the text recognition information in the upper left corner region of the detection box.

[0089] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. An optical character recognition method based on an improved OCR network, characterized in that, The improved OCR network is improved on the feature extraction module, the feature enhancement module and the prediction module of the traditional OCR network; the improvement on the feature extraction module is: using an adjacency matrix prediction module, namely an ADJ module, to replace a text direction offset prediction module, namely a TDO module, the ADJ module is a series of convolution operations, the connection information between nodes is predicted through a loss function, and adjacency matrix information is output, in addition, a feature map module, namely a FEA module, is added, the FEA module is also a series of convolution operations, and is used for extracting image feature information and outputting feature map information, the improved feature extraction module will output text position information, adjacency matrix information, feature map information and text box position information; the improvement on the feature enhancement module is: establishing a graph convolution network after the feature map, the feature map is divided into nodes with a size of 16*16, the nodes are flattened, and a 16*16 matrix is taken out one by one in a row-first order and arranged into a one-dimensional vector, the size of the vector is 256, and the adjacency matrix information output by the preceding ADJ module is also input into the graph convolution network to enhance the correlation information extraction between nodes, and the graph convolution network outputs a series of node features; the improvement on the prediction module is: constructing a decoder to sample and decode the enhanced node features and the text position features output by the feature extraction module, so as to replace a character prediction channel to perform character prediction and obtain a final result; The specific implementation of the optical character recognition method comprises the following steps: 1) obtaining basic data, including an image with English text and performing labeling, marking a text box and corresponding text, so that the labeled data has the text box and text information, then dividing the data into a training set and a test set for network training and testing respectively; 2) inputting the data in the training set into the improved OCR network for training, acquiring feature information of the image through the feature extraction module during training, inputting the extracted feature information into a subsequent prediction module for integration and output, and obtaining a text prediction result of the image; in the back propagation, the total loss value is calculated through the joint loss of the feature extraction module and the feature enhancement module, wherein the feature extraction module is to calculate the distance error of the center point position and the text box position, the feature enhancement module is to calculate the mean square error of the adjacency matrix and the connection time sequence classification loss of the final text recognition, and the total error is obtained by weighted summation; after multiple iterations until the loss value is minimum, the optimal network is obtained; 3) inputting the data in the test set into the optimal network obtained through training to obtain prediction information, then using NMS and setting a threshold to screen out detection boxes meeting the conditions from the prediction information, and drawing the finally obtained detection boxes on the original image, and marking the text recognition information in the top left corner region of the detection box. 2.The improved OCR network-based optical character recognition method of claim 1, wherein, The step 1) comprises the following steps: 1.1) image loading preprocessing: using the CTW-1500 dataset, the CTW-1500 dataset is preprocessed, and the images without labels are removed; 1.2) Image label processing: For the label of the CTW-1500 dataset, it is divided into text location prediction label, text box prediction label, text character classification map prediction label and text direction offset prediction label, the text location prediction label and the text box prediction label are reserved, and then the text direction offset prediction label is processed and converted into an adjacency matrix label, the image is divided into multiple 16x16 matrices, each matrix is taken as a node, the connection information between the nodes is obtained by decomposing the vector information of the text direction offset prediction label, and edges are added between the nodes through the connection information between the nodes, wherein the text direction offset prediction label is a three-dimensional matrix with a size of (H, W, 3), the matrix has three dimensions, the size of the first dimension is H, indicating the image height, the size of the second dimension is W, indicating the image width, and the size of the third dimension is 3, that is, the channel size is 3, and the information contained in the three channels is as follows: the first channel records the horizontal direction information of the direction vector, for each pixel point, greater than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the right, less than 0 indicates that the horizontal component of the position direction vector of the pixel point is to the left, the second channel records the vertical direction information of the direction vector, for each pixel point, greater than 0 indicates that the vertical component of the position direction vector of the pixel point is upward, less than 0 indicates that the horizontal component of the position direction vector of the pixel point is downward, and the third channel indicates the relative height of the character; when processing the text direction information, the horizontal direction information of the text direction offset prediction label and the vertical direction information of the direction vector are used, corresponding to the first two channels in the third dimension; for the pixels in the node range, the average direction vector is calculated, and then the position of the next node connected by the node is judged according to the obtained direction vector, and the specific operation is as follows: the current node and the up, down, left and right nodes are connected according to the component size, if the horizontal component is greater than 0, the node to the right of the current node is connected, if the horizontal component is less than 0, the node to the left of the current node is connected, if the vertical component is greater than 0, the node above the current node is connected, and if the vertical component is less than 0, the node below the current node is connected; after judging the connection relationship of the up, down, left and right nodes, the nodes in the diagonal direction also need to be connected, the current node and the left lower node, the left upper node, the right lower node and the right upper node are connected according to the direction of the average vector of the current node, and the connection form is in the form of an adjacency matrix, the size of the adjacency matrix is (num, num), num is the number of nodes, and is calculated as num = H / / 16xW / / 16, when the corresponding coordinate (i, j) of the adjacency matrix is 1, it indicates that the i-th node has an edge connected to the j-th node, wherein the adjacency matrix label is calculated as follows, the three-dimensional matrix corresponding to the text direction offset prediction label needs to be traversed: Firstly, the coordinate point (x, y) of each pixel point is obtained, x is the horizontal coordinate, y is the vertical coordinate, the size of the horizontal and vertical components at the point is calculated, if the absolute value of the direction vector corresponding to the coordinate point is less than a preset threshold, it is not processed, if it is greater than the threshold, the calculation of the adjacency matrix label is carried out, and the size of the horizontal and vertical components is calculated as follows: x _direct = D map [x,y,0] y _direct = D map [x,y,1] In the formula, x _direct represents the size of the horizontal component, y _direct represents the size of the vertical component, D map represents a text direction offset prediction label; Then, the node number corresponding to the pixel point is read: pos=x / / 16+y / / 16×W / / 16 In the formula, pos is the node number; Finally, the horizontal adjacent node, the vertical adjacent node and the vector direction adjacent node after merging are processed, the adjacency matrix value corresponding to the position is set to 1, and the adjacency matrix is denoted as A; Processing horizontal adjacent nodes: x _sub = 1 if x _direct > 0 else x _sub = -1 A[pos,pos+x _sub ] = 1 wherein x _sub represents a horizontal node offset; Processing vertical adjacent nodes: y _sub = -1 if y _direct > 0 else y _sub = 1 A[pos,pos+y _sub xW / / 16] = 1 Where y _sub Indicates the vertical node offset; Processing vector direction adjacent nodes: A[pos, pos+x _sub +y _sub ×W / / 16] = 1 1.3) Divide the data set: after the image label processing is completed, the image and the label file are divided into a training set and a test set according to a proportion. 3.The improved OCR network-based optical character recognition method of claim 2, wherein, In step 2), the text position prediction label, the text box prediction label and the adjacency matrix label of the training set play different roles in the whole network, which are used for training the network. The image information of the training set is input into the improved OCR network, the features are extracted in the feature extraction module, the features are input into the graph neural network for strengthening and extracting the text information, and finally the character prediction is made, including the following steps: The image of the training set is input into the feature extraction module, the feature extraction module includes a deep residual network Resnet50, a feature extraction pyramid FPN and four branch modules, the four branch modules include a text center line prediction module TCL module, a text boundary offset prediction module TBO module, an FEA module and an ADJ module, wherein the TCL module is used to locate the position of the text in the image, and the label of the TCL module is the position of the text center point in the image, the TBO module is used to predict the position of the text box, and the label of the TBO module is the position of the four vertices of the text box and the offset angle parameter, the FEA module uses a series of convolutions to extract image information as the input of the graph neural network, and the ADJ module learns the connection information between nodes supervisedly, uses a series of convolutions to extract image information, and a sigmoid layer needs to be added later to limit the output size to between 0 and 1. After the image information passes through Resnet50 and FPN, the depth feature information of the image is obtained, and then the four differentiated image feature information is obtained after the depth feature information passes through the four branch modules. The feature enhancement module uses the feature map and the adjacency matrix to construct a graph neural network, and the feature map and the adjacency matrix correspond to the outputs of the FEA module and the ADJ module respectively, wherein the output of the ADJ module needs to be converted into an adjacency matrix, and the specific conversion method is consistent with the processing in image labeling, that is, converted into an adjacency matrix, denoted as B, each image has a corresponding graph structure, and the graph neural network uses two layers of GCN to aggregate the features of adjacent nodes, and the output dimension is 64, and the feature map information is enhanced: first, the feature map is divided into multiple nodes, each node is a 16*16 matrix, the size of the feature map is (H, W, 3), H is the image height, and W is the image width, a round of convolution is performed on the feature map first, the size of the third dimension is changed to 1, and a matrix with a size of (H, W, 1) is obtained, then for each node divided, the node is flattened, and the 16*16 matrix is taken out one by one in row-major order and arranged into a one-dimensional vector, the size of the vector is 256, the vectors are input into the graph convolution, the input adjacency matrix needs to be processed, a parameter is set as a threshold, all values greater than the threshold are recorded as 1, otherwise as 0, then the graph neural network is converted into an undirected graph, that is, the connected nodes are changed into bidirectional connection at the level of the adjacency matrix, the graph neural network uses two layers of GCN, the dimension of the first layer is 128, and the dimension of the second layer is 64, and a series of vector sizes of graph nodes are obtained after the graph neural network; The node features predicted by the graph neural network are first sampled and then input into the Decoder for decoding, TCL_map is the output of the TCL module, which is a matrix with a size of (H, W, 1), the values in the matrix are between 0 and 1, indicating the probability that a pixel point is text, the valid character corresponding node is obtained from the matrix, for a coordinate point (x, y), if the value of the position corresponding to TCL_map is greater than a given threshold, then the node corresponding to the coordinate is determined to be a valid character node, and the corresponding node number is calculated: pos=x / / 16+y / / 16×W / / 16 The dimension number of the input Decoder is reduced by obtaining the valid node position; the Decoder is composed of RNN, fully connected layer and softmax, the data input into the Decoder is the node features obtained after sampling, and finally CTC is used for character alignment to obtain the output text information; The loss function of the improved OCR network is improved, and the loss of the TCL module, the TDO module, the ADJ module and the Decoder is weighted. The Dice loss is used for calculating the TCL module and the ADJ module, which are respectively denoted as Tcl_Loss and Adj_Loss, representing the loss of the TCL module and the loss of the ADJ module. The loss calculation in the traditional OCR network is used for the TDO module, and the boundary is finely controlled, which is denoted as Tdo_Loss, representing the loss of the TDO module. The CTC loss is used for the Decoder to calculate the loss of the data and align the output data, which is denoted as CTC_Loss, representing the loss of the Decoder. The final loss is Loss, and the loss function is finally obtained as follows: Loss=λ1×Tcl_Loss+λ2×Tdo_Loss+λ3×Adj_Loss+λ4×CTC_Loss In the formula, λ1, λ2, λ3 and λ4 are hyperparameters.

Images