An end-to-end text recognition method based on dynamic sampling

Abstract

The application discloses an end-to-end character recognition method based on dynamic sampling, comprising the following steps: obtaining a text image to be processed; inputting the text image to be processed into a text detection model to obtain a character region in the text image to be processed; inputting the character region into a dynamic sampling model to obtain a character visual feature corresponding to the character region; and inputting the character visual feature into a recognition model to obtain a recognition result. The application uses a dynamic sampling network to connect a text detection network and a text recognition network, can recognize complex and various characters in various scenes, is suitable for an end-to-end character recognition method of different mechanisms, and has a high accuracy on various disclosed data sets, and has a high practicability and application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and artificial intelligence, and particularly relates to an end-to-end character recognition method based on dynamic sampling. BACKGROUND

[0002] As an important task of computer vision, the purpose of end-to-end character recognition is to extract characters in a picture. With the development of deep learning technology and related hardware, more and more deep learning-based character recognition networks are applied to actual production and life. A general end-to-end character recognition method includes two sub-networks, a detection network and a recognition network. The detection network is responsible for locating the position of the text in the image. The recognition network is responsible for sampling the text region located by the detection network and extracting the character content. Using different sampling methods to sample the text region located by the detection network will greatly affect the final recognition performance. Therefore, connecting the detection network and the recognition network has become an important research topic. Although some methods try to design some sampling operators to sample the text region located by the detection network for the recognition network. However, these sampling operators are static and can only sample on a fixed grid, which is easy to sample the background region and ignore the actual text position, resulting in poor recognition effect of the text with complex background, various shapes and large scale differences. Secondly, these sampling operators need to be used with special detection networks and recognition networks, and have poor universality. SUMMARY

[0003] In order to realize end-to-end character recognition, the application provides an end-to-end character recognition method based on dynamic sampling. The method can dynamically sample features for the recognition network, can greatly improve the recognition accuracy, and has high use value.

[0004] To achieve the above purpose, the application provides an end-to-end character recognition method based on dynamic sampling, comprising the following steps:

[0005] Obtaining a text image to be processed;

[0006] Inputting the text image to be processed into a text detection model to obtain a character region in the text image to be processed;

[0007] Inputting the character region into a dynamic sampling model to obtain a character visual feature corresponding to the character region;

[0008] Inputting the character visual feature into a recognition model to obtain a recognition result.

[0009] Optionally, the text detection model adopts a sparse RCNN model.

[0010] The sparse RCNN model comprises a multi-head attention mechanism, a full connection layer, a dynamic convolution layer, a LayerNorm normalization operation, a Dropout operation and an activation layer.

[0011] Optionally, the text image to be processed is input into the text detection model to obtain the text region in the image to be processed, comprising:

[0012] The text image is feature-extracted by using a Transformer encoder to obtain a feature map;

[0013] Based on the convolution layer, the feature map is pre-detected to obtain a plurality of candidate regions;

[0014] The plurality of candidate regions are feature-extracted by using ROI Align to obtain visual features of the corresponding regions of the feature map;

[0015] Based on the sparse RCNN model, the plurality of candidate regions are corrected by using the visual features to obtain the text region in the image to be processed.

[0016] Optionally, based on the sparse RCNN model, the plurality of candidate regions are corrected by using the visual features to obtain the text region in the image to be processed, comprising:

[0017] The plurality of candidate regions are randomly initialized to obtain a plurality of candidate features, and the plurality of candidate regions and the plurality of candidate features are one-to-one corresponding;

[0018] The plurality of candidate features are added to the plurality of candidate features after the plurality of candidate features pass through the multi-head attention mechanism, and are subjected to LayerNorm normalization, and then pass through the full connection layer to obtain the weight of the convolution kernel;

[0019] Based on the weight of the convolution kernel, the visual features are subjected to convolution operation, LayerNorm normalization and activation operation, and the full connection layer is used to output a plurality of offsets;

[0020] The plurality of candidate regions are corrected by using the plurality of offsets to obtain corrected candidate regions and corrected candidate features;

[0021] The corrected candidate regions are feature-extracted by using ROI Align, and are input into the next layer of the sparse RCNN model until the text region in the image to be processed is obtained.

[0022] Optionally, the dynamic sampling model comprises a low-level attention model, a high-level attention model, a deformable convolution sampling model and an identification conversion model.

[0023] Optionally, inputting the text region into the dynamic sampling model to obtain the text visual feature corresponding to the text region comprises:

[0024] inputting the text region into the low-level attention model and the high-level attention model respectively to obtain the low-level semantic level feature and the high-level semantic level feature;

[0025] performing a splicing operation on the low-level semantic level feature and the high-level semantic level feature, and inputting the spliced feature into the deformable convolution sampling model to extract the dynamic sampling feature;

[0026] inputting the dynamic sampling feature and the correction candidate feature into the recognition conversion model to fuse the dynamic sampling feature and the correction candidate feature, and obtaining the text visual feature corresponding to the text region.

[0027] Optionally, inputting the text region into the low-level attention model and the high-level attention model respectively to obtain the low-level semantic level feature and the high-level semantic level feature comprises:

[0028] dividing the text region to obtain a plurality of small windows and performing attention mechanism calculation to obtain the low-level semantic level feature;

[0029] using an average pooling operation to reduce the text region in the to-be-processed image, and performing global attention mechanism calculation to obtain the high-level semantic level feature.

[0030] Optionally, the deformable convolution sampling model comprises a deformable convolution layer, a convolution layer, a BatchNorm normalization layer and a GELU activation layer.

[0031] Optionally, performing a splicing operation on the low-level semantic level feature and the high-level semantic level feature, and inputting the spliced feature into the deformable convolution sampling model to extract the dynamic sampling feature comprises:

[0032] splicing the low-level semantic level feature and the high-level semantic level feature to obtain a spliced feature;

[0033] the deformable convolution layer uses the spliced feature to predict an offset;

[0034] based on the offset, the deformable convolution layer samples the spliced feature to obtain a sampled feature;

[0035] using the BatchNorm normalization layer and the GELU activation layer to normalize and activate the sampled feature to obtain a normalized and activated feature;

[0036] The normalized and activated features are down-sampled by the convolutional layer to obtain the dynamic sampling features.

[0037] Optionally, the text visual features are input into the recognition model to obtain the recognition result, including:

[0038] The text visual features are converted into a character sequence by using a two-dimensional attention mechanism to obtain the recognition result.

[0039] The present application has the following beneficial effects:

[0040] (1) The present application can dynamically sample the feature map effectively by proposing a dynamic sampling network, and no longer uses a static sampling operator, thereby avoiding sampling to the background area and ignoring the actual text position.

[0041] (2) The present application proposes a sparse RCNN network-based detection network; such a detection network avoids the dense prediction in the previous method, reduces the computational burden, and enables interaction between different texts to enhance the detection accuracy.

[0042] (3) The present application has high recognition accuracy and strong robustness, and is suitable for various end-to-end recognition networks.

[0043] (4) The present application uses a dynamic sampling network to connect the text detection and text recognition networks, can recognize complex and diverse texts in various scenes, is suitable for different mechanism end-to-end text recognition methods, and achieves a high accuracy on various public data sets, has high practicability and application value. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and are not intended to limit the present application. In the drawings:

[0045] Figure 1 A flowchart of an embodiment of the present application, a dynamic sampling-based end-to-end text recognition method, is shown in the figure;

[0046] Figure 2 A flowchart of a text detection network proposed by an embodiment of the present application is shown in the figure;

[0047] Figure 3 A flowchart of a sparse RCNN network proposed by an embodiment of the present application is shown in the figure;

[0048] Figure 4 A flowchart of a dynamic sampling network proposed by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0051] like Figure 1 As shown, this embodiment provides an end-to-end text recognition method based on dynamic sampling, including the following steps:

[0052] S1. Obtain the text image to be processed;

[0053] S2. Construct a text detection network. Input the text image to be processed into the detection network to obtain the text regions in the image to be processed.

[0054] like Figure 2 As shown, the text detection network employs a sparse RCNN network. For the input text image to be processed, the text detection network first uses a Transformer network pre-trained on the ImageNet large-scale classification dataset to extract feature maps from the text image.

[0055] Based on the feature maps, 300 candidate regions are pre-detected using convolutional layers. Then, ROIAlign is used to extract the visual features (ROIs) of the corresponding regions from the feature maps based on the spatial location of each text segment. feat ; ROI based on the visual features of the corresponding region feat The sparse RCNN network continuously corrects 300 candidate regions and interacts with different texts, finally outputting the text regions in the image to be processed.

[0056] like Figure 3 As shown, the sparse RCNN network includes: three multi-head attention mechanisms, six fully connected layers, six dynamic convolutional layers, nine LayerNorm normalization operations, twelve Dropout operations, and six activation layers. For 300 candidate regions, 300 candidate features are first obtained through random initialization; each of the 300 candidate regions corresponds one-to-one with a candidate feature; after passing through multi-head attention, the 300 candidate features are added together and then normalized using LayerNorm; finally, the weights of two convolutional kernels are generated through fully connected layers; these weights are applied to the visual feature ROI. featThe visual features after each convolution operation are normalized by LayerNorm and then activated by an activation layer. Finally, the features after the convolution are output by a fully connected layer to correct the 300 candidate regions, referred to as corrected candidate regions, and update the 300 candidate features, referred to as corrected candidate features. The corrected candidate regions are extracted by ROI Align to obtain new visual features and corrected candidate features, which are input into the sparse RCNN network of the next layer. The embodiment has three layers of sparse RCNN networks.

[0057] S3, constructing a dynamic sampling network, inputting the text region in the image to be processed into the dynamic sampling network, sampling the text of different shapes or sizes, and obtaining the visual features corresponding to the text region;

[0058] The dynamic sampling network uses ROI Align to dynamically extract the visual features of the feature map region as the input of the recognition network according to the text region in the image to be processed output by the text detection network.

[0059] As shown in Figure 4 The dynamic sampling network includes a low-level attention network, a high-level attention network, a deformable convolution sampling network, and a recognition conversion network.

[0060] The low-level attention network is used to extract low-level semantic features.

[0061] The low-level attention network uses a local attention mechanism on a high-resolution feature map to extract low-level semantic features. The low-level attention network divides the input features of the dynamic sampling network into small windows, and calculates the attention mechanism in the small windows to extract features.

[0062] The high-level attention network is used to extract high-level semantic features.

[0063] The high-level attention network uses a global attention mechanism on a low-resolution feature map to extract high-level semantic features. The high-level attention network uses an average pooling operation to reduce the input feature map. The reduced feature map is globally calculated by the attention mechanism to extract features.

[0064] The deformable convolution sampling network uses low-level semantic features and high-level semantic features as input to dynamically sample features.

[0065] The deformable convolution sampling network is constructed by using a deformable convolution network, and the deformable convolution network comprises a deformable convolution layer, a convolution layer, a BatchNorm normalization layer and a GELU activation layer. For the input low-level semantic level feature and high-level semantic level feature, the deformable convolution layer can predict the offset in the height direction and the width direction. According to the offset, the deformable convolution layer can sample the input low-level semantic level feature and high-level semantic level feature, accurately sample the position of the text, and remove the background noise. The BatchNorm normalization layer and the GELU activation layer are used to normalize and activate the sampled features. The convolution layer is used to down-sample the normalized and activated features to obtain dynamic sampling features.

[0066] The recognition conversion network fuses the rectified candidate features output by the sparse RCNN network and the dynamic sampling features through dynamic convolution, so that the gradient of the recognition loss is back propagated to the detection network, and the synergy of the detection network and the recognition network is strengthened.

[0067] S4, constructing a recognition network, inputting the features output by the dynamic sampling network into the recognition network, and obtaining a recognition result.

[0068] The recognition network uses the text visual features output by the dynamic sampling network as input.

[0069] The recognition network uses a two-dimensional attention mechanism to convert the text visual features output by the dynamic sampling network into a character sequence to obtain the final recognition result.

[0070] Training method: the gradient descent algorithm is used for training, the gradient is calculated from the last layer, the parameters of the network are updated layer by layer, and the purpose of training the network is achieved; the deep neural network is trained under random initialization of parameters.

[0071] Training parameter setting: the training data is input into the network for training, the optimization algorithm used is an adaptive gradient descent method (AdamW), the learning rate is 0.000025, the iteration number is 450,000 iterations, and the learning rate updating strategy is that the learning rate is attenuated to 1 / 10 of the original value after 360,000 and 420,000 selected generations.

[0072] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An end-to-end character recognition method based on dynamic sampling, characterized in that, Includes the following steps: Obtain the text image to be processed; The text image to be processed is input into a text detection model to obtain the text regions in the image. The text detection model employs a sparse R-CNN model. Based on this sparse R-CNN model, visual features are used to correct several candidate regions to obtain the text regions in the image to be processed. The candidate regions are randomly initialized to obtain several candidate features, with each candidate region corresponding to one of the candidate features. These candidate features are then added to each other using a multi-head attention mechanism and normalized using LayerNorm. The resulting data is then passed through a fully connected layer to obtain the weights of the convolution kernels. Based on these kernel weights, convolution, LayerNorm normalization, and activation operations are performed on the visual features, and the fully connected layer outputs several offsets. These offsets are used to correct the candidate regions, obtaining corrected candidate regions and corrected candidate features. The corrected candidate regions are then used for feature extraction using ROI Align and input into the next layer of the sparse R-CNN model until the text regions in the image to be processed are obtained. The text region is input into a dynamic sampling model to obtain the corresponding visual features. The dynamic sampling model includes a low-level attention model, a high-level attention model, a deformable convolutional sampling model, and a recognition-conversion model. The text region is input into the low-level attention model and the high-level attention model respectively to obtain low-level semantic features and high-level semantic features. The low-level semantic features and the high-level semantic features are concatenated and input into the deformable convolutional sampling model for feature extraction to obtain dynamic sampling features. The low-level semantic features and the high-level semantic features are concatenated to obtain concatenated features. A deformable convolutional layer uses the concatenated features for prediction to obtain an offset. Based on the offset, the deformable convolutional layer samples the concatenated features to obtain sampling features. The sampled features are then normalized and activated using a BatchNorm normalization layer and a GELU activation layer to obtain normalized and activated features. The normalized and activated features are then downsampled using a convolutional layer to obtain dynamically sampled features. The low-level semantic features and the high-level semantic features are concatenated to obtain concatenated features. A deformable convolutional layer uses the concatenated features to make predictions and obtain offsets. Based on these offsets, the deformable convolutional layer samples the concatenated features to obtain sampled features. The sampled features are then normalized and activated using a BatchNorm normalization layer and a GELU activation layer to obtain normalized and activated features. Finally, the normalized and activated features are downsampled using a convolutional layer to obtain dynamically sampled features. The dynamic sampling features and the correction candidate features are input into the recognition and conversion model for fusion to obtain the visual features of the text region; The visual features of the text are input into the recognition model to obtain the recognition result.

2. The end-to-end character recognition method based on dynamic sampling as described in claim 1, characterized in that, The sparse RCNN model includes a multi-head attention mechanism, fully connected layers, dynamic convolutional layers, LayerNorm normalization, Dropout, and activation layers.

3. The end-to-end character recognition method based on dynamic sampling as described in claim 2, characterized in that, Inputting the text image to be processed into the text detection model to obtain the text regions in the image to be processed includes: The text image is processed using a Transformer encoder to extract features and obtain a feature map. Based on the convolutional layer, the feature map is pre-coarsely detected to obtain several candidate regions; The ROI Align is used to extract features from several candidate regions to obtain the visual features of the regions corresponding to the feature maps. Based on the sparse RCNN model, the visual features are used to correct several candidate regions to obtain the text regions in the image to be processed.

4. The end-to-end character recognition method based on dynamic sampling as described in claim 1, characterized in that, The text region is input into the low-level attention model and the high-level attention model respectively to obtain the low-level semantic features and the high-level semantic features, including: The text region is divided into several small windows, and attention mechanism calculations are performed to obtain the low-level semantic features. The text region in the image to be processed is reduced by using average pooling, and a global attention mechanism is calculated to obtain the high-level semantic features.

5. The end-to-end character recognition method based on dynamic sampling as described in claim 1, characterized in that, The deformable convolutional sampling model includes a deformable convolutional layer, a convolutional layer, a BatchNorm normalization layer, and a GELU activation layer.

6. The end-to-end character recognition method based on dynamic sampling as described in claim 1, characterized in that, Inputting the visual features of the text into the recognition model to obtain the recognition result includes: The visual features of the text are converted into a character sequence using a two-dimensional attention mechanism to obtain the recognition result.

Images