Deformed text recognition and positioning method and device based on improved CTPN algorithm

Abstract

The invention provides a deformed text recognition and positioning method and device based on an improved CTPN algorithm, and the method comprises the steps: obtaining a deformed text image to be recognized and positioned, carrying out the preprocessing, generating a text denoising image, carrying out the processing through a pre-constructed multi-layer convolution feature extraction model, and carrying out the recognition and positioning of the deformed text. Generating a shallow-layer highlighted feature map, a deep-layer highlighted feature map and a deeper-layer highlighted feature map; processing through a pre-constructed feature weight learning model based on a space attention mechanism to generate a deep-layer space weighted feature mapping graph and a deeper-layer space weighted feature mapping graph; and finally, through a pre-constructed feature serialization and deformation text recognition and positioning model, generating a deformation text recognition and positioning information matrix to realize recognition and positioning of the deformation text. According to the technical scheme, the integrity of the deformed text is enhanced, and the recognition accuracy is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text recognition, and in particular relates to a method and a device for deformable text recognition and positioning based on an improved CTPN algorithm. Background Art

[0002] In the field of optical character recognition (OCR), character region localization is a key technical challenge, especially when dealing with small text, text with a large aspect ratio, and text at multiple scales and directions. Traditional character region localization methods include connected component-based, edge-based, texture-based, and learning-based localization. These methods often show low noise resistance and recognition difficulty in the face of complex backgrounds or diverse fonts and sizes. In recent years, learning-based localization methods, especially those using machine learning techniques, have significantly improved the performance of OCR technology. These methods are usually trained with a large amount of data to give the system powerful recognition capabilities. Convolutional neural networks (CNNs) play an important role in this process. As an important part of the field of deep learning, CNNs have become one of the most popular and efficient algorithms in the fields of image, audio recognition, and natural language processing.

[0003] Compared with traditional neural networks, CNN can retain its two-dimensional structure when processing images, and complete the mapping from the input layer to the feature map by selecting a receptive field. This structure greatly reduces the training parameters and improves the efficiency of image processing. The role of the convolution layer is to extract local features and combine them to form higher-level features, while the pooling layer after each convolution layer can condense the calculated features to improve the accuracy of recognition. There are several main methods for text area recognition based on CNN. The application of the SSD (Single Shot MultiBox Detector) method in text detection shows certain advantages over the single-stage YOLO algorithm and the two-stage Fast R-CNN algorithm, but its largest feature map has a large receptive field for small objects, resulting in weak position features. The Textboxes algorithm is based on the SSD framework and uses VGG16 as the backbone feature network, but it still faces the problem of being unable to handle special cases (such as occlusion, overexposure) and multi-directional text. The CTPN (Connectionist Text Proposal Network) algorithm optimizes anchor point selection through differentiation and adopts a text line construction method to effectively solve the problem of long text recognition. However, the complex post-processing process and poor detection effect on certain text types (such as deformed text) are not good. The Seglink algorithm combines the local segment prediction of CTPN and the multi-scale prediction of SSD to simultaneously predict texts of different scales, but the merging method of straight line fitting cannot handle deformed text. Therefore, in order to improve the accuracy and reliability of deformed text detection, it is urgent and necessary to seek a deformed text recognition and positioning method based on the improved CTPN algorithm to provide a new perspective for the development of optical character recognition technology. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a deformable text recognition and positioning method and device based on an improved CTPN algorithm, which realizes accurate recognition of deformable text by constructing a multi-model processing method.

[0005] In a first aspect, a deformable text recognition and positioning method based on an improved CTPN algorithm is provided, the method comprising the following steps:

[0006] Step S1. Obtain a deformed text image to be identified and located, and generate a text denoised image after preprocessing using a binary wavelet transform algorithm;

[0007] Step S2. Processing the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and a deeper salient feature map;

[0008] Step S3. Processing the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map;

[0009] Step S4. Processing the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model based on a channel attention mechanism to generate a deep weighted feature map;

[0010] Step S5. Processing the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map;

[0011] Step S6. The feature fusion map is subjected to a pre-built feature serialization and deformable text recognition and positioning model to generate a deformable text recognition and positioning information matrix, thereby realizing the recognition and positioning of the deformable text.

[0012] As a further improved technical solution of the present invention, the step S1 specifically includes the following contents:

[0013] Obtain the deformed text image to be identified and located and convert it into a grayscale image, and then generate a pre-processed text denoising image after normalization, J-times binary wavelet transform, linear transformation and pre-built joint threshold model processing;

[0014] The pre-processed text denoising image is subjected to linear transformation and binary wavelet reconstruction to generate a text denoising image;

[0015] Wherein, the joint threshold model is:

[0016]

[0017] Wherein, α represents the joint coefficient, and the value range of α is [0,1]. When α is 1 and 0 respectively, the joint threshold model is converted into the soft threshold model and the hard threshold model; j, k represent the horizontal coordinate and the vertical coordinate of the image respectively; d j,k Represents the pixel value at coordinate (j, k); sgn() represents the sign function; || represents the absolute value; represents the updated pixel value; λ j Represents the threshold value and is set to:

[0018]

[0019] Where N is the size of the second transformed image; σ is the noise variance and is set to:

[0020] σ 2 =median|dj,k | / 0.6754

[0021] Among them, median|d j,k | represents the median value of the pixel values ​​in the second transformed image.

[0022] As a further improved technical solution of the present invention, step S2 specifically includes the following contents:

[0023] Input the text denoising image into a pre-built multi-layer convolutional feature extraction model based on a VGG16 backbone network, extract shallow feature maps, deep feature maps and deeper feature maps in the fifth convolutional layer, the seventh convolutional layer and the ninth convolutional layer of the multi-layer convolutional feature extraction model respectively, and generate shallow salient feature maps, deep salient feature maps and deeper salient feature maps through a pre-built self-attention module,

[0024] Among them, the self-attention module includes a first convolution unit, a second convolution unit, a third convolution unit, a horizontal attention unit and a vertical attention unit; the first convolution unit, the second convolution unit and the third convolution unit all include a convolution layer with a convolution kernel of 1×1, a BN normalization layer and a ReLU activation layer; the horizontal attention unit includes a first reorganization unit, a second reorganization unit, a third reorganization unit and a fourth reorganization unit; the vertical attention unit includes a fifth reorganization unit, a sixth reorganization unit, a seventh reorganization unit and an eighth reorganization unit.

[0025] As a further improved technical solution of the present invention, the steps of generating a shallow salient feature map, a deep salient feature map and a deeper salient feature map by the pre-constructed self-attention module include:

[0026] The shallow feature map, the deep feature map and the deeper feature map are respectively processed by the first convolution unit to obtain a shallow feature map to be highlighted, a deep feature map to be highlighted and a deeper feature map to be highlighted;

[0027] The shallow feature to be highlighted image is respectively subjected to the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain the shallow level value item vector V q1 , shallow horizontal key value vector K q1 and the shallow level query vector Q q1 The shallow feature to be highlighted image is obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the shallow vertical value item vector V q2 , shallow vertical key value vector K q2 and the shallow vertical query vector Q q2 ;

[0028] The deep feature to be highlighted image is respectively subjected to the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain the deep level value item vector V s1 , deep level key value vector K s1 , deep horizontal query vector Q s1 The deep feature to be highlighted image is respectively obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the deep vertical value item vector V s2 , deep vertical key value vector K s2 and the deep vertical query vector Q s2 ;

[0029] The deeper feature map is processed by the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain a deeper horizontal value item vector V g1 , the deeper level key value vector K g1 , a deeper level query vector Q g1 The deeper feature map is obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain a deeper vertical value item vector V g2 , a deeper vertical key value vector K g2 and a deeper vertical query vector Q g2 ;

[0030] The shallow horizontal protrusion vector graph F is obtained by the following formula qx , shallow vertical protrusion vector diagram F qy ; Deep horizontal protrusion vector map F sx , deep vertical protrusion vector diagram F sy ; Deeper horizontal highlight vector diagram F gx , deeper vertical protrusion vector diagram F gy ;

[0031]

[0032] Where m = q, s, g, Softmax represents the normalized exponential function; represents matrix multiplication; W a represents the weight matrix;

[0033] The shallow horizontal protrusion vector diagram F qx , the deep horizontal protrusion vector diagram F sx and the deeper level highlight vector diagram F gx The shallow horizontal salient feature map, the deep horizontal salient feature map, and the deeper horizontal salient feature map are generated respectively through the fourth recombination unit and the second convolution unit in sequence; the shallow vertical salient vector map F qy , deep vertical protrusion vector diagram F sy And deeper vertical protrusion vector diagram F syPassing through the eighth recombination unit and the third convolution unit in sequence respectively, a shallow vertical salient feature map, a deep vertical salient feature map, and a deeper vertical salient feature map are obtained;

[0034] The shallow horizontal salient feature map is added to the shallow vertical salient feature map to obtain a shallow salient feature map;

[0035] The deep horizontal salient feature map is added to the deep vertical salient feature map to obtain a deep salient feature map;

[0036] The deeper horizontal salient feature map is added to the deeper vertical salient feature map to obtain a deeper salient feature map.

[0037] As a further improved technical solution of the present invention, step S3 specifically includes the following contents:

[0038] Constructing a feature weight learning model based on the spatial attention mechanism including a fourth convolutional unit, a spatial global average pooling layer, a fully connected layer, and a ReLU activation layer; the deep salient feature map and the deeper salient feature map are respectively subjected to the fourth convolutional unit spatial global average pooling layer, the fully connected layer, and the ReLU activation layer to generate a deep spatial attention weight and a deeper spatial attention weight;

[0039] The deep spatial attention weight is multiplied by the deep salient feature map to obtain a deep spatial weighted feature map; and the deeper spatial attention weight is multiplied by the deeper salient feature map to obtain a deeper spatial weighted feature map.

[0040] As a further improved technical solution of the present invention, step S4 specifically includes the following contents:

[0041] Constructing a multi-channel feature weight learning model based on a channel attention mechanism including a global average pooling layer, a fully connected layer, a ReLU activation layer and an upsampling layer, wherein the deep spatial weighted feature map and the deeper spatial weighted feature map are respectively sequentially subjected to the channel global average pooling layer, the fully connected layer and the ReLU activation layer to generate a deep channel attention weight and a deeper channel attention weight;

[0042] Multiplying the deep channel attention weight with the deep spatial weighted feature map, and adding the deep spatial weighted feature map to obtain a deep channel weighted feature map; multiplying the deeper channel attention weight with the deeper spatial weighted feature map, and adding the deeper spatial weighted feature map to obtain a deeper channel weighted feature map;

[0043] The deeper channel weighted feature map is expanded by an upsampling layer to expand the image size, and then added to the deeper channel weighted feature map to obtain a deeper weighted feature map.

[0044] As a further improved technical solution of the present invention, step S5 specifically includes the following contents:

[0045] A feature fusion optimization model including an upsampling layer, a BatchNorm normalization layer and a fusion layer is constructed. After the deep weighted feature map is processed by the upsampling layer and the BatchNorm normalization layer, a deep normalized feature map is generated. The deep normalized feature map is fused with the shallow salient feature map through the fusion layer to obtain a feature fusion map.

[0046] As a further improved technical solution of the present invention, step S6 specifically includes the following contents:

[0047] Construct a feature serialization and deformable text recognition and positioning model including feature serialization layer, fully connected layer and text line construction and positioning layer

[0048] The feature fusion map is passed through a sliding window of size d×d in a feature serialization layer to obtain a serialized feature vector, and then a text feature vector is generated by locating the minimum range of the deformed text using n rectangular boxes;

[0049] The text feature vector is passed through a fully connected layer to score the deformed text rectangles with different heights and fixed widths to generate a deformed text rectangle evaluation vector, and then passed through the text line construction positioning layer to generate a deformed text recognition positioning information matrix.

[0050] In a second aspect, the present invention further proposes a deformable text recognition and positioning device based on an improved CTPN algorithm, the device comprising:

[0051] An acquisition module is used to acquire the deformed text image to be identified and located, and generate a text denoised image after preprocessing by a binary wavelet transform algorithm;

[0052] A first generation module is used to process the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and a deeper salient feature map;

[0053] The second generation module is used to process the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map.

[0054] A third generation module is used to process the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model of a channel attention mechanism to generate a deep weighted feature map;

[0055] A fourth generating module, used for processing the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map;

[0056] The fifth generation module is used to generate a deformable text recognition and positioning information matrix through the feature fusion map through the pre-built feature serialization and deformable text recognition and positioning model to realize the recognition and positioning of the deformable text.

[0057] In a third aspect, the present invention provides an electronic device, comprising:

[0058] A memory on which application programs and an operating system are stored;

[0059] A processor is used to implement the steps of the identification and positioning method as described in the first aspect when executing the program or instruction.

[0060] 1. The present invention proposes a deformed text recognition and positioning method based on an improved CTPN algorithm. With the help of the translation invariance of the dyadic wavelet transform, the image containing the deformed text can better retain the original information while obtaining an excellent denoising effect, thereby avoiding the loss of some important text feature information; at the same time, based on the constructed self-attention module, the text features are more effectively extracted and highlighted from a global perspective, thereby realizing the maximum recognition of the deformed text features.

[0061] 2. The present invention proposes a deformable text recognition and positioning method based on an improved CTPN algorithm, which constructs a feature weight learning model based on a spatial attention mechanism and a multi-channel feature weight learning model based on a channel attention mechanism, allowing the network to dynamically focus on different areas when processing images and dynamically learn and adjust the importance of each channel, so that the network can better understand the structure and contextual information of the image, and pay more attention to the image channels that are more important to the current task; at the same time, residual connections are introduced to allow the network to learn residuals, thereby retaining the original features to avoid information loss.

[0062] 3. The present invention proposes a deformable text recognition and positioning method based on an improved CTPN algorithm. By constructing a feature fusion mapping diagram, the deep semantic features of the deformable text weighted by space and channel attention are organically integrated with the shallow detail features of the deformable text, which is more conducive to the extraction and recognition of the deformable text information; by constructing a feature serialization and deformable text recognition and positioning model, the deformable text features are fragmented, the precise position of the deformable text is obtained through scoring, and the broken deformable text is spliced ​​together through the text line construction method, thereby enhancing the integrity of the deformable text and effectively improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a deformable text recognition and positioning method based on an improved CTPN algorithm provided in an embodiment of the present application.

[0064] Figure 2 A schematic diagram of the structure of the self-attention module provided in an embodiment of the present application.

[0065] Figure 3 A schematic diagram of the structure of a feature weight learning model based on a spatial attention mechanism provided in an embodiment of the present application.

[0066] Figure 4 A schematic diagram of the structure of a multi-channel feature weight learning model based on a channel attention mechanism provided in an embodiment of the present application.

[0067] Figure 5 A structural diagram of a deformable text recognition and positioning device based on an improved CTPN algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] For ease of understanding, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, the present invention proposes a deformable text recognition and positioning method based on an improved CTPN algorithm, and the method specifically includes the following steps:

[0071] Step S1. Obtain a deformed text image to be identified and located, and generate a text denoised image after preprocessing using a binary wavelet transform algorithm.

[0072] In this embodiment, step S1 specifically includes the following contents:

[0073] S11. Convert the image containing the deformed text into a grayscale image and perform normalization processing to obtain a normalized image, so as to reduce the computational cost of the image and standardize the text image.

[0074] S12, subjecting the normalized image to J dyadic wavelet transforms to obtain a first transformed image. In a specific embodiment, J is 3.

[0075] S13, subjecting the first transformed image to linear transformation to obtain a second transformed image.

[0076] S14, based on the soft threshold model and the hard threshold model, a joint threshold model is constructed, and threshold processing is performed on the second transformed image to obtain a third transformed image; the joint threshold model is:

[0077]

[0078] Wherein, α represents the joint coefficient, and the value range of α is [0,1]. When α is 1 and 0 respectively, the joint threshold model is converted into the soft threshold model and the hard threshold model; j, k represent the horizontal coordinate and the vertical coordinate of the image respectively; d j,k represents the pixel value at the coordinate (j, k); sgn() represents the sign function; ‖ represents the absolute value; represents the updated pixel value; λ j Represents the threshold value and is set to:

[0079]

[0080] Where N is the size of the second transformed image; σ is the noise variance and is set to:

[0081] σ 2 =median|d j,k | / 0.6754

[0082] Among them, median|d j,k | represents the median value of the pixel values ​​in the second transformed image.

[0083] S15. Perform a linear inverse transformation on the third transformed image to obtain a fourth transformed image.

[0084] S16. Reconstruct the fourth transformed image through binary wavelet to obtain a text denoised image.

[0085] With the help of the translation invariance of dyadic wavelet transform, images containing deformed text can better retain the original information while achieving excellent denoising effect, avoiding the loss of some important text feature information.

[0086] Step S2. Processing the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and an even deeper salient feature map.

[0087] In this embodiment, step S2 includes the following contents:

[0088] S21. Based on the backbone network of VGG16, the fifth convolutional layer, the seventh convolutional layer and the ninth convolutional layer are respectively taken to obtain the corresponding shallow feature map, deep feature map and deeper feature map; the size of the shallow feature map is W0×H0×C, the size of the deep feature map is W1×H1×C, and the size of the deeper feature map is W2×H2×C, wherein W0 and H0 respectively represent the width and height of the shallow feature map, W1 and H1 respectively represent the width and height of the deep feature map, W2 and H2 respectively represent the width and height of the deeper feature map, and C represents the number of channels. In a specific embodiment, C is preferably 512.

[0089] S22. Construct a self-attention module based on the shallow feature map, the deep feature map and the deeper feature map, respectively, to obtain a shallow salient feature map, a deep salient feature map and a deeper salient feature map, respectively.

[0090] like Figure 2 As shown, the self-attention module includes a first convolution unit, a second convolution unit, a third convolution unit, a horizontal attention unit and a vertical attention unit; the first convolution unit, the second convolution unit and the third convolution unit all include a convolution layer with a convolution kernel of 1×1, a BN normalization layer and a ReLU activation layer; the horizontal attention unit includes a first reorganization unit, a second reorganization unit, a third reorganization unit and a fourth reorganization unit; the vertical attention unit includes a fifth reorganization unit, a sixth reorganization unit, a seventh reorganization unit and an eighth reorganization unit.

[0091] The first reorganization unit and the third reorganization unit reorganize the image size from W×H×C to W×CH, the second reorganization unit reorganizes the image size from W×H×C to CH×W, and the fourth reorganization unit reorganizes the image size from W×CH to W×H×C; the fifth reorganization unit and the sixth reorganization unit reorganize the image size from W×H×C to H×CW, the seventh reorganization unit reorganizes the image size from W×H×C to CW×H, and the eighth reorganization unit reorganizes the image size from H×CW to W×H×C; wherein W and H represent the width and height of the image, respectively.

[0092] S221, the shallow feature map, the deep feature map and the deeper feature map are respectively processed by the first convolution unit to obtain the shallow feature to be highlighted map, the deep feature to be highlighted map and the deeper feature to be highlighted map.

[0093] S222: The shallow feature to be highlighted image is processed by the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain the shallow level value item vector V q1 , shallow horizontal key value vector K q1 and the shallow level query vector Q q1 The shallow feature to be highlighted image is obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the shallow vertical value item vector V q2 , shallow vertical key value vector K q2 and the shallow vertical query vector Q q2 .

[0094] Similarly, based on the deep feature map, the first convolution unit is used to obtain a deep feature map to be highlighted, and the deep feature map to be highlighted is used to obtain a deep horizontal value item vector V through the first reorganization unit, the second reorganization unit and the third reorganization unit respectively. s1 , deep level key value vector K s1 , deep horizontal query vector Q s1 The deep feature to be highlighted image is obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the deep vertical value item vector V s2 , deep vertical key value vector K s2 and the deep vertical query vector Q s2 Based on the deeper feature map, the deeper feature map to be highlighted is obtained by the first convolution unit, and the deeper feature map to be highlighted is obtained by the first reorganization unit, the second reorganization unit and the third reorganization unit respectively to obtain the deeper horizontal value item vector V g1 , the deeper level key value vector K g1 , a deeper level query vector Q g1 The deeper features to be highlighted are respectively obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the deeper vertical value item vector V g2 , a deeper vertical key value vector K g2 and a deeper vertical query vector Qg2 .

[0095] S223, shallow level value vector V q1 , shallow horizontal key value vector K q1 and the shallow level query vector Q q1 The shallow horizontal protrusion vector F is obtained by the first conversion formula qx , shallow vertical value vector V q2 , shallow vertical key value vector K q2 and the shallow vertical query vector Q q2 The shallow vertical protrusion vector F is obtained by the second conversion formula qy ; The first conversion formula and the second conversion formula are respectively expressed as:

[0096]

[0097] Among them, Softmax represents the normalized exponential function; represents matrix multiplication; W a represents the weight matrix.

[0098] Similarly, based on the deep level value term vector V s1 , deep level key value vector K s1 and the deep horizontal query vector Q s1 Get the deep horizontal protrusion vector map F sx , based on the deep vertical value term vector V s2 , deep vertical key value vector K s2 and the deep vertical query vector Q s2 Get the deep vertical protrusion vector map F sy ;

[0099]

[0100] Based on the deeper level value term vector V g1 , the deeper level key value vector K g1 and a deeper level query vector Q g1 Get a deeper horizontal highlight vector image F gx , based on the deeper vertical value term vector V g2 , a deeper vertical key value vector K g2 and a deeper vertical query vector Q g2 Get a deeper vertical protrusion vector diagram F gy .

[0101]

[0102] In summary, the shallow horizontal protrusion vector graph F can be obtained by the following formula: qx , shallow vertical protrusion vector diagram F qy; Deep horizontal protrusion vector map F sx , deep vertical protrusion vector diagram F sy ; Deeper horizontal highlight vector diagram F gx , deeper vertical protrusion vector diagram F gy ;

[0103]

[0104] Where m = q, s, g, Softmax represents the normalized exponential function; Represents matrix multiplication.

[0105] S224, shallow horizontal protrusion vector diagram F qx After the fourth reorganization unit and the second convolution unit, the shallow horizontal salient feature map is obtained; the shallow vertical salient vector map F qy The shallow vertical salient feature map is obtained by sequentially passing through the eighth recombination unit and the third convolution unit; the shallow horizontal salient feature map is added to the shallow vertical salient feature map to obtain the shallow salient feature map.

[0106] Similarly, based on the deep level salient vector map F sx Obtain the deep horizontal salient feature map based on the deep vertical salient vector map F sy Obtain a deep vertical salient feature map, and then add the two to obtain a deep salient feature map; based on the deeper horizontal salient vector map F gx Obtain a deeper horizontal salient feature map based on a deeper vertical salient vector map F sy A deeper vertical salient feature map is obtained, and then the two are added together to obtain a deeper salient feature map.

[0107] Based on the constructed self-attention module, the shallow feature maps, deep feature maps and deeper feature maps extracted with the help of the VGG16 skeleton can be used to more effectively extract and highlight text features from a global perspective.

[0108] Step S3. Processing the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map;

[0109] like Figure 3As shown, in this embodiment, the feature weight learning model based on the spatial attention mechanism constructed in step S3 includes a fourth convolution unit, a spatial global average pooling layer, a fully connected layer and a ReLU activation layer; the fourth convolution unit includes a convolution layer with a convolution kernel of 1×1, a BatchNorm normalization layer and a ReLU activation layer; the spatial global average pooling layer represents a global average pooling process for all channels of the image. Specifically, it includes the following contents:

[0110] S31, the deep salient feature map and the deeper salient feature map are respectively passed through the fourth convolution unit to obtain a deep spatial information map and a deeper spatial information map.

[0111] S32, the deep spatial information graph and the deeper spatial information graph are respectively subjected to the spatial global average pooling layer and the fully connected layer in sequence to obtain the deep spatial attention weight and the deeper spatial attention weight. The sizes of the deep spatial attention weight and the deeper spatial attention weight are W1×H1×1 and W2×H2×1 respectively.

[0112] S33. With the help of ReLU activation layer, the deep spatial attention weights and the deeper spatial attention weights are normalized and the negative features are removed.

[0113] S34. Multiply the normalized deep spatial attention weights by the deep salient feature map to obtain a deep spatial weighted feature map; multiply the normalized deeper spatial attention weights by the deeper salient feature map to obtain a deeper spatial weighted feature map.

[0114] Constructing a feature weight learning model based on the spatial attention mechanism allows the network to dynamically focus on different areas when processing images, so that the network can better understand the structure and contextual information of the image and pay more attention to the image areas that are more important to the current task.

[0115] Step S4. Process the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model of a channel attention mechanism to generate a deep weighted feature map.

[0116] In this embodiment, if Figure 4 As shown in the figure, the multi-channel feature weight learning model based on the channel attention mechanism includes a channel global average pooling layer, a fully connected layer, a ReLU activation layer and an upsampling layer; the fourth convolution unit includes a convolution layer with a convolution kernel of 1×1, a BatchNorm normalization layer and a ReLU activation layer; the channel global average pooling layer represents the global average pooling processing for the feature map of each channel of the image; the upsampling layer adopts a deconvolution operation.

[0117] S41, the deep spatial weighted feature map and the deeper spatial weighted feature map are respectively passed through the channel global average pooling layer and the fully connected layer in sequence to obtain the deep channel attention weight and the deeper channel attention weight. The size of the deep channel attention weight and the deeper channel attention weight are both 1×1×C.

[0118] S42. With the help of ReLU activation layer, the deep channel attention weights and the deeper channel attention weights are normalized respectively.

[0119] S43. Multiply the normalized deep channel attention weights by the deep spatial weighted feature map, and add the deep spatial weighted feature map to obtain a deep channel weighted feature map; multiply the normalized deeper channel attention weights by the deeper spatial weighted feature map, and add the deeper spatial weighted feature map to obtain a deeper channel weighted feature map.

[0120] S44, the deeper channel weighted feature map is expanded to an image size of W1×H1×C through an upsampling layer, and then added to the deeper channel weighted feature map to obtain a deeper weighted feature map.

[0121] A multi-channel feature weight learning model based on the channel attention mechanism is constructed, which can dynamically learn and adjust the importance of each channel. By adjusting the feature contribution between channels, the network can pay more attention to the image channels that are more important to the current task. At the same time, residual connections are introduced to allow the network to learn residuals, thereby retaining the original features to avoid information loss.

[0122] Step S5. Processing the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map.

[0123] In this embodiment, the feature fusion optimization model includes an upsampling layer, a BatchNorm normalization layer and a fusion layer. The upsampling layer uses a deconvolution operation.

[0124] S51. The deep weighted feature map is expanded to W0×H0×C by the upsampling layer, and then normalized by the BatchNorm normalization layer to obtain a deep normalized feature map. The BatchNorm normalization layer can avoid the network from being difficult to converge or gradient explosion caused by weight update, and can also increase the training speed, accelerate the model convergence speed and prevent the model from overfitting.

[0125] S52. By means of the element-wise operation Eltwise layer, the deep normalized feature map is fused with the shallow salient feature map to obtain a feature fusion map.

[0126] The element-wise operation Eltwise layer can select element sum operation, element dot product operation and element maximum operation. In a preferred specific embodiment, the element dot product operation is used to fuse the deep normalized feature map with the shallow salient feature map.

[0127] By constructing a feature fusion map, the deep semantic features of deformable text weighted by spatial and channel attention are organically integrated with the shallow detailed features of deformable text, which is more conducive to the extraction and recognition of deformable text information.

[0128] Step S6. The feature fusion map is subjected to a pre-built feature serialization and deformable text recognition and positioning model to generate a deformable text recognition and positioning information matrix to realize the recognition and positioning of the deformable text.

[0129] In this embodiment, the specific contents of step S6 include the following:

[0130] S61. Obtain a serialized feature vector for the feature fusion map using a d×d sliding window. In a specific embodiment, d is 3, and the size of the serialized feature vector is 3×3×512.

[0131] S62: for the serialized feature vector, use n rectangular boxes to locate the minimum range of the deformed text, define the recursive form of the recursive network, and obtain the deformed text feature vector; the width and height of the rectangular box are set to w and h respectively; the width and length of the deformed text feature vector are W respectively. out , C out In a specific embodiment, n is 10, w is 16 pixels, h is between 10 and 234 pixels, and C out Take 256.

[0132] S63, the deformed text feature vector is passed through a fully connected layer, and the rectangular boxes of the deformed text with different heights and a fixed width of w are scored to obtain a deformed text rectangular box evaluation vector. The deformed text rectangular box evaluation vector includes the deformed text rectangular box position information composed of the center point and the rectangular box height, the deformed text rectangular box scoring information, and the parameters for adjusting the position of the deformed text rectangular box endpoints. The deformed text rectangular box scoring information includes the score predicted as a deformed text box and the score predicted as a non-deformed text box. In a specific embodiment, the output dimension of the fully connected layer is 512.

[0133] S63. Based on the deformed text rectangular box evaluation vector, a text line construction method is used to sort and merge the vectors to obtain a deformed text recognition and positioning information matrix.

[0134] The text line construction method is to combine adjacent deformed text rectangular boxes at the same horizontal position when the combination rule is met. The combination rule is that the two deformed text rectangular boxes are nearest neighbors to each other and the horizontal distance is less than 60 unit pixels, or the overlapping part of the height of the two deformed text rectangular boxes accounts for more than 70% of the two deformed text rectangular boxes themselves.

[0135] The present invention constructs a feature serialization and deformable text recognition and positioning model, fragments the deformable text features, obtains the precise position of the deformable text through scoring, and splices the broken deformable text together through a text line construction method, thereby enhancing the integrity of the deformable text and effectively improving the recognition accuracy.

[0136] The present application also provides a deformable text recognition and positioning device based on an improved CTPN algorithm. Figure 5 As shown, the device 500 includes:

[0137] The acquisition module 501 is used to acquire the deformed text image to be identified and located, and generate a text denoised image after preprocessing by a binary wavelet transform algorithm;

[0138] A first generating module 502 is used to process the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and a deeper salient feature map;

[0139] The second generation module 503 is used to process the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map.

[0140] A third generation module 504 is used to process the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model of a channel attention mechanism to generate a deep weighted feature map;

[0141] The fourth generating module 505 is used to process the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map;

[0142] The fifth generating module 506 is used to generate a deformable text recognition and positioning information matrix through the pre-built feature serialization and deformable text recognition and positioning model of the feature fusion map to realize the recognition and positioning of the deformable text.

[0143] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores an application and an operating system; the processor is used to implement the steps of the above-mentioned identification and positioning method when executing the program or instructions.

[0144] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned identification and positioning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0146] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0148] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A deformable text recognition and positioning method based on an improved CTPN algorithm is characterized by: The method comprises the following steps: Step S1. Obtain a deformed text image to be identified and located, and generate a text denoised image after preprocessing using a binary wavelet transform algorithm; Step S2. Processing the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and a deeper salient feature map; Step S3. Processing the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map; Step S4. Processing the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model based on a channel attention mechanism to generate a deep weighted feature map; Step S5. Processing the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map; Step S6. The feature fusion map is subjected to a pre-built feature serialization and deformable text recognition and positioning model to generate a deformable text recognition and positioning information matrix, thereby realizing the recognition and positioning of the deformable text.

2. The identification and positioning method according to claim 1, characterized in that: The step S1 specifically includes the following contents: Obtain the deformed text image to be identified and located and convert it into a grayscale image, and then generate a pre-processed text denoising image after normalization, J-times binary wavelet transform, linear transformation and pre-built joint threshold model processing; The pre-processed text denoising image is subjected to linear transformation and binary wavelet reconstruction to generate a text denoising image; Wherein, the joint threshold model is: Wherein, α represents the joint coefficient, and the value range of α is [0,1]. When α is 1 and 0 respectively, the joint threshold model is converted into the soft threshold model and the hard threshold model; j, k represent the horizontal coordinate and the vertical coordinate of the image respectively; d j,k Represents the pixel value at coordinate (j, k); sgn() represents the sign function; || represents the absolute value; represents the updated pixel value; λ j Represents the threshold value and is set to: Where N is the size of the second transformed image; σ is the noise variance and is set to: s 2 =median|d j,k | / 0.6754 Among them, median|d j,k | represents the median value of the pixel values ​​in the second transformed image.

3. The identification and positioning method according to claim 2, characterized in that: The step S2 specifically includes the following contents: Input the text denoising image into a pre-built multi-layer convolutional feature extraction model based on a VGG16 backbone network, extract shallow feature maps, deep feature maps and deeper feature maps in the fifth convolutional layer, the seventh convolutional layer and the ninth convolutional layer of the multi-layer convolutional feature extraction model respectively, and generate shallow salient feature maps, deep salient feature maps and deeper salient feature maps through a pre-built self-attention module, Among them, the self-attention module includes a first convolution unit, a second convolution unit, a third convolution unit, a horizontal attention unit and a vertical attention unit; the first convolution unit, the second convolution unit and the third convolution unit all include a convolution layer with a convolution kernel of 1×1, a BN normalization layer and a ReLU activation layer; the horizontal attention unit includes a first reorganization unit, a second reorganization unit, a third reorganization unit and a fourth reorganization unit; the vertical attention unit includes a fifth reorganization unit, a sixth reorganization unit, a seventh reorganization unit and an eighth reorganization unit.

4. The identification and positioning method according to claim 3, characterized in that: The steps of generating a shallow salient feature map, a deep salient feature map and a deeper salient feature map by the pre-built self-attention module include: The shallow feature map, the deep feature map and the deeper feature map are respectively processed by the first convolution unit to obtain a shallow feature map to be highlighted, a deep feature map to be highlighted and a deeper feature map to be highlighted; The shallow feature to be highlighted image is respectively subjected to the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain the shallow level value item vector V q1 , shallow horizontal key value vector K q1 and the shallow level query vector Q q1 The shallow feature to be highlighted image is respectively obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the shallow vertical value item vector V q2 , shallow vertical key value vector K q2 and the shallow vertical query vector Q q2 ; The deep feature to be highlighted image is respectively subjected to the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain the deep level value item vector V s1 , deep level key value vector K s1 , deep horizontal query vector Q s1 The deep feature to be highlighted image is respectively obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain the deep vertical value item vector V s2 , deep vertical key value vector K s2 and the deep vertical query vector Q s2 ; The deeper feature map is processed by the first reorganization unit, the second reorganization unit and the third reorganization unit to obtain a deeper horizontal value item vector V g1 , the deeper level key value vector K g1 , a deeper level query vector Q g1 The deeper feature map is obtained by the fifth reorganization unit, the sixth reorganization unit and the seventh reorganization unit to obtain a deeper vertical value item vector V g2 , a deeper vertical key value vector K g2 and a deeper vertical query vector Q g2 ; The shallow horizontal protrusion vector graph F is obtained by the following formula qx , shallow vertical protrusion vector diagram F qy , deep horizontal protrusion vector diagram F sx , deep vertical protrusion vector diagram F sy ; Deeper horizontal highlight vector diagram F gx , deeper vertical protrusion vector diagram F gy ; Where m = q, s, g, Softmax represents the normalized exponential function; represents matrix multiplication; W a represents the weight matrix; The shallow horizontal protrusion vector diagram F qx , the deep horizontal protrusion vector diagram F sx and the deeper level highlight vector diagram F gx The shallow horizontal salient feature map, the deep horizontal salient feature map, and the deeper horizontal salient feature map are generated respectively through the fourth recombination unit and the second convolution unit in sequence; the shallow vertical salient vector map F qy , deep vertical protrusion vector diagram F sy And deeper vertical protrusion vector diagram F sy Passing through the eighth recombination unit and the third convolution unit in sequence respectively, a shallow vertical salient feature map, a deep vertical salient feature map, and a deeper vertical salient feature map are obtained; The shallow horizontal salient feature map is added to the shallow vertical salient feature map to obtain a shallow salient feature map; The deep horizontal salient feature map is added to the deep vertical salient feature map to obtain a deep salient feature map; The deeper horizontal salient feature map is added to the deeper vertical salient feature map to obtain a deeper salient feature map.

5. The identification and positioning method according to claim 4, characterized in that: The step S3 specifically includes the following contents: Constructing a feature weight learning model based on the spatial attention mechanism including a fourth convolutional unit, a spatial global average pooling layer, a fully connected layer, and a ReLU activation layer; the deep salient feature map and the deeper salient feature map are respectively subjected to the fourth convolutional unit spatial global average pooling layer, the fully connected layer, and the ReLU activation layer to generate a deep spatial attention weight and a deeper spatial attention weight; The deep spatial attention weight is multiplied by the deep salient feature map to obtain a deep spatial weighted feature map; and the deeper spatial attention weight is multiplied by the deeper salient feature map to obtain a deeper spatial weighted feature map.

6. The identification and positioning method according to claim 5, characterized in that: The step S4 specifically includes the following contents: Constructing a multi-channel feature weight learning model based on a channel attention mechanism including a global average pooling layer, a fully connected layer, a ReLU activation layer and an upsampling layer, wherein the deep spatial weighted feature map and the deeper spatial weighted feature map are respectively sequentially subjected to the channel global average pooling layer, the fully connected layer and the ReLU activation layer to generate a deep channel attention weight and a deeper channel attention weight; Multiplying the deep channel attention weight with the deep spatial weighted feature map, and adding the deep spatial weighted feature map to obtain a deep channel weighted feature map; multiplying the deeper channel attention weight with the deeper spatial weighted feature map, and adding the deeper spatial weighted feature map to obtain a deeper channel weighted feature map; The deeper channel weighted feature map is expanded by an upsampling layer to expand the image size, and then added to the deeper channel weighted feature map to obtain a deeper weighted feature map.

7. The identification and positioning method according to claim 6, characterized in that: The step S5 specifically includes the following contents: A feature fusion optimization model including an upsampling layer, a BatchNorm normalization layer and a fusion layer is constructed. After the deep weighted feature map is processed by the upsampling layer and the BatchNorm normalization layer, a deep normalized feature map is generated. The deep normalized feature map is fused with the shallow salient feature map through the fusion layer to obtain a feature fusion map.

8. The identification and positioning method according to claim 7, characterized in that: The step S6 specifically includes the following contents: Construct a feature serialization and deformable text recognition and positioning model including feature serialization layer, fully connected layer and text line construction and positioning layer The feature fusion map is passed through a sliding window of size d×d in a feature serialization layer to obtain a serialized feature vector, and then a text feature vector is generated by locating the minimum range of the deformed text using n rectangular boxes; The text feature vector is passed through a fully connected layer to score the deformed text rectangles with different heights and fixed widths to generate a deformed text rectangle evaluation vector, and then passed through the text line construction positioning layer to generate a deformed text recognition positioning information matrix.

9. A deformable text recognition and positioning device based on an improved CTPN algorithm, characterized in that: The device comprises: An acquisition module is used to acquire the deformed text image to be identified and located, and generate a text denoised image after preprocessing by a binary wavelet transform algorithm; A first generation module is used to process the text denoising image through a pre-built multi-layer convolutional feature extraction model to generate a shallow salient feature map, a deep salient feature map and a deeper salient feature map; The second generation module is used to process the deep salient feature map and the deeper salient feature map through a pre-built feature weight learning model based on a spatial attention mechanism to generate a deep spatial weighted feature map and a deeper spatial weighted feature map. A third generation module is used to process the deep spatial weighted feature map and the deeper spatial weighted feature map through a pre-built multi-channel feature weight learning model of a channel attention mechanism to generate a deep weighted feature map; A fourth generating module, used for processing the shallow salient feature map and the deep weighted feature map through a pre-built feature fusion optimization model to generate a feature fusion map; The fifth generation module is used to generate a deformable text recognition and positioning information matrix through the feature fusion map through the pre-built feature serialization and deformable text recognition and positioning model to realize the recognition and positioning of the deformable text.

10. An electronic device, characterized in that: include: A memory on which application programs and an operating system are stored; A processor, used to implement the steps of the identification and positioning method as described in any one of claims 1 to 8 when executing the program or instruction.

Images