Scene text detection method and device and storage medium

Abstract

The scene text detection method comprises the following steps: acquiring a to-be-detected image and performing feature extraction on the to-be-detected image; wherein the to-be-detected image comprises a scene text; processing the features obtained through feature extraction; performing scene text core area probability prediction and boundary threshold prediction according to a processing result to obtain a binary image of the to-be-detected image; wherein the processing at least comprises one of attention feature fusion processing and first fusion processing; carrying out combined attention processing; or the processing at least comprises one of feature screening and second fusion processing; carrying out segmentation processing, edge enhancement and third fusion processing; the feature extraction comprises the steps of performing feature extraction on the to-be-detected image through a convolutional neural network, and adjusting the extracted feature dimensions of the plurality of branches to be preset dimensions to obtain shallow features of the plurality of branches.

Description

Technical Field

[0001] This article relates to but is not limited to the field of computer vision, and in particular to a scene text detection method, device and storage medium. Background Art

[0002] Currently, in the field of computer vision, text detection includes document recognition, table recognition, bill recognition, ID card recognition, etc. It is relatively easy to detect the above-mentioned texts, but it is more difficult to detect texts with variable shapes, drastic scale changes, and complex and diverse backgrounds because they have higher requirements in shape description, multi-scale detection, and resistance to background interference. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The present disclosure provides a scene text detection method, including:

[0005] Acquire an image to be detected and perform feature extraction on the image to be detected; wherein the image to be detected contains scene text;

[0006] Processing the features obtained through feature extraction;

[0007] According to the processing results, the scene text core area probability prediction and the boundary threshold prediction are performed to obtain a binary image of the image to be detected;

[0008] The processing includes at least one of the following:

[0009] Attentional feature fusion and first fusion processing;

[0010] joint attention processing;

[0011] Or the processing includes at least one of the following:

[0012] Feature screening and second fusion processing;

[0013] Segmentation, edge enhancement and third fusion processing;

[0014] The feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

[0015] In an exemplary embodiment, the attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, determining the deep features of the branch based on the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

[0016] In an exemplary embodiment, the channel-weighted coding feature of the branch is determined according to the deep feature of the next branch of the branch, the shallow feature of the branch, and a preset CAM module;

[0017] Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch of the branch after upsampling, and the preset SAM module;

[0018] The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

[0019] In an exemplary embodiment, determining the channel-weighted coding feature of the branch according to the deep feature of the lower branch of the branch, the shallow feature of the branch, and a preset CAM module includes:

[0020] The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

[0021] In an exemplary embodiment, determining the weighted decoding features of the spatial information of the branch according to the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and the preset SAM module includes:

[0022] The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

[0023] In an exemplary embodiment, the first fusion process includes: concatenating the deep features of all branches to form a fused deep feature.

[0024] In an exemplary embodiment, the joint attention processing includes: determining the weight features of the first fused features based on the weight vector of each channel of the first fused features and the spatial mask; and performing residual connection based on the first fused features and the weight features of the first fused features.

[0025] In an exemplary embodiment, the weight vector of each channel in the features after the first fusion processing is extracted by the preset CAM module;

[0026] The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

[0027] In an exemplary embodiment, performing residual connection according to the first fused feature and the weight feature of the first fused feature includes:

[0028] Multiplying the first fused feature and the weight feature of the first fused feature by position;

[0029] The multiplication result is added to the feature after the first fusion processing.

[0030] In an exemplary embodiment, the feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch according to the shallow features of the branch and the screening features of the next branch of the branch;

[0031] Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the atrous spatial convolution pooling pyramid module.

[0032] In an exemplary embodiment, determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes:

[0033] Upsampling the screening feature of the next branch of the branch by a preset multiple; adding the upsampled feature to the shallow feature of the branch to obtain a first feature;

[0034] The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature;

[0035] The first feature is integrated with multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature;

[0036] The screening feature of the branch is determined according to the second feature and the third feature.

[0037] In an exemplary embodiment, the second fusion process includes: splicing the screening features of all branches.

[0038] In an exemplary embodiment, the segmentation process includes: extracting a first semantic feature, performing a first upsampling, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

[0039] In an exemplary embodiment, the edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling and normalization processes on the features obtained by the second fusion process in sequence.

[0040] In an exemplary embodiment, the third fusion processing includes: splicing the features obtained by the segmentation processing and the features obtained by the edge enhancement processing; extracting the fourth semantic features, integrating the information between multiple channels, and normalizing the spliced ​​features in sequence.

[0041] The present disclosure provides a scene text detection device, which is characterized by comprising an acquisition module, a feature extraction module, a processing module, and a post-processing module; wherein the acquisition module is configured to acquire an image to be detected;

[0042] The feature extraction module is configured to extract features from the image to be detected;

[0043] The processing module is configured to process the features obtained by feature extraction;

[0044] The post-processing module is configured to perform a probability prediction of the scene text core area and a boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected;

[0045] The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module;

[0046] The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

[0047] The present disclosure provides a scene text detection device, including a memory and a processor.

[0048] The memory is used to store a program for scene text detection;

[0049] The processor is used to read and execute the program for scene text detection and execute the above-mentioned scene text detection method.

[0050] The present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable the computer to execute the above-mentioned scene text detection method.

[0051] Other aspects will be apparent upon reading and understanding the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shape and size of one or more components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0053] Figure 1 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0054] Figure 2 A schematic diagram of a preset CAM module according to at least one embodiment of the present disclosure;

[0055] Figure 3 A schematic diagram of a preset SAM module according to at least one embodiment of the present disclosure;

[0056] Figure 4 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0057] Figure 5 A schematic diagram of a network model of a scenario of at least one embodiment of the present disclosure;

[0058] Figure 6 This is the structure diagram of the convolution module of the convolutional neural network;

[0059] Figure 7 for Figure 5 The structural diagram of the attention feature fusion module of the shown model;

[0060] Figure 8 for Figure 5 Structural diagram of the joint attention module of the shown model;

[0061] Fig. 9 It is the differentiable binarization process diagram;

[0062] Fig.10 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0063] Fig.11 A schematic diagram of a network model of a scenario of at least one embodiment of the present disclosure;

[0064] Fig.12 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0065] Fig.13 A schematic diagram of a network model of a scenario of at least one embodiment of the present disclosure;

[0066] Fig.14for Fig.13 Block diagram of the FF module of the model shown;

[0067] Fig.15 for Fig.13 Block diagram of the segmentation module of the shown model;

[0068] Fig.16 for Fig.13 Structural diagram of the edge enhancement module of the model shown;

[0069] Fig.17 for Fig.13 A block diagram of the third fusion processing module of the illustrated model;

[0070] Fig.18 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0071] Fig.19 The present invention is a schematic diagram of a network model of a scenario according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The embodiments can be implemented in a plurality of different forms. A person of ordinary skill in the art can easily understand the fact that the method and content can be transformed into one or more forms without departing from the purpose and scope of the present disclosure. Therefore, the present disclosure should not be interpreted as being limited to the contents described in the following embodiments. In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0073] The ordinal numbers such as "first", "second", and "third" in the present disclosure are provided to avoid confusion of constituent elements, and are not intended to limit the quantity. The "plurality" in the present disclosure means a quantity of two or more than two.

[0074] Compared with document text, text in natural scenes is more susceptible to factors such as its own variable shape, drastic scale changes, and complex backgrounds. Therefore, the model of scene text detection has higher requirements in terms of shape description, multi-scale detection, and resistance to background interference. Based on this, the present invention uses the segmentation-based DBNet as the basic model framework, and then adds the designed processing method to the DBNet framework in the form of a module, achieving high-precision detection of text in scenes with variable shapes, drastic scale changes, and complex and diverse backgrounds.

[0075] Figure 1 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure, such as Figure 1As shown, the scene text detection method includes the following steps S11-S13:

[0076] S11, acquiring an image to be detected and performing feature extraction on the image to be detected;

[0077] S12, processing the features obtained by feature extraction;

[0078] S13, performing the scene text core area probability prediction and boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0079] In an exemplary embodiment, in step S11, the image to be detected contains scene text. Scene text generally refers to text in a real scene image.

[0080] In step S11, in an exemplary embodiment, the feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

[0081] In an exemplary embodiment, the convolutional neural network (ResNet-18 can be used as the backbone network, and the network is mainly composed of 5 convolution modules, namely conv1, conv2, conv3, conv4 and conv5) can extract the features of conv2, conv3, conv4 and conv5 respectively, and the extracted features can be adjusted to, for example, 256 dimensions through a 1×1 convolution module. It can be considered that conv2, conv3, conv4 and conv5 are respectively used as a branch, and the features output by conv2, conv3, conv4 and conv5 are unified in dimension as shallow features of their respective branches. The preset dimension can be 256 dimensions.

[0082] In an exemplary embodiment, in step S12, the processing includes at least one of the following:

[0083] Attentional feature fusion and first fusion processing;

[0084] joint attention processing;

[0085] Or the processing includes at least one of the following:

[0086] Feature screening and second fusion processing;

[0087] Segmentation processing, edge enhancement and third fusion processing.

[0088] Among them, attention feature fusion uses the attention mechanism to extract different emphasis information in the deep and shallow layers to calibrate and strengthen the input feature vector, thereby performing high-precision text detection in scenes with variable shapes, drastic scale changes, and complex and diverse backgrounds.

[0089] The first fusion process fuses the output features of the attention feature fusion.

[0090] Joint attention processing improves information representation capabilities through global information interaction.

[0091] Feature screening can extract effective information in the process of reverse restoration of features from low resolution to high resolution through multi-scale feature screening.

[0092] The second fusion process fuses the output features of the feature screening.

[0093] The segmentation process can classify the text area and the background into two categories.

[0094] Edge enhancement improves the accuracy of text area boundaries and solves the problem of fuzzy text edge detection and reduced boundary area accuracy caused by the use of segmentation algorithms.

[0095] The third fusion process fuses the output features of the segmentation process and the output features of the edge enhancement process.

[0096] In an exemplary embodiment, the attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, determining the deep features of the branch based on the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

[0097] In an exemplary embodiment, the channel-weighted coding feature of the branch is determined according to the deep feature of the next branch of the branch, the shallow feature of the branch, and a preset CAM module;

[0098] Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch of the branch after upsampling, and the preset SAM module;

[0099] The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

[0100] Among them, the preset CAM module (Channel Attention Module) is a channel attention module. The preset CAM module can make the network pay more attention to the channel characteristics and position information of the target.

[0101] like Figure 2As shown in the figure, the preset CAM module first applies maximum pooling (MaxPooling) and average pooling (AvgPooling) to the input feature map F at each spatial position to obtain two C×1×1 vectors, which are then sent to a shared multi-layer perceptron MLP (i.e., a neural network) containing two fully connected layers (FC). Finally, the pixels are added and fused, and after an activation function, the channel attention map Mc is obtained.

[0102] The preset SAM module (Spatial Attention Model) is a spatial attention module. The preset SAM module can highlight the area of ​​interest, thereby improving the accuracy and efficiency of image segmentation.

[0103] like Figure 3 As shown in the figure, the preset SAM module first applies MaxPooling and AvgPooling on each channel to obtain two 1×H×W feature maps, which are then fused by channel and sent to a standard convolutional layer. After the activation function, the spatial attention map Ms is obtained.

[0104] In an exemplary embodiment, determining the channel-weighted coding feature of the branch according to the deep feature of the lower branch of the branch, the shallow feature of the branch, and a preset CAM module includes:

[0105] The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

[0106] In an exemplary embodiment, determining the weighted decoding features of the spatial information of the branch according to the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and the preset SAM module includes:

[0107] The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

[0108] The purpose of upsampling the deep features of the next branch of the branch is to restore the image size, because in order to obtain the deep features of the branch, pooling is performed after the attention feature fusion process.

[0109] The deepest convolution module here is the convolution module farthest from the input end in the convolutional neural network. For example, if the convolutional neural network contains 5 convolution modules, then each convolution module corresponds to a layer (that is, corresponds to a branch), and the 5 convolution modules are arranged according to the distance from the input end of the convolutional neural network and named C1, C2, C3, C4, and C5 respectively. C1 is the convolution module closest to the input end in the convolutional neural network, and C5 is the convolution module farthest from the input end in the convolutional neural network.

[0110] The upper branch of a branch corresponds to a convolutional module that is closer to the input end of the convolutional neural network than the branch, and the lower branch of a branch corresponds to a convolutional module that is farther from the input end of the convolutional neural network than the branch. For example, the upper branch of the branch corresponding to C3 is the branch corresponding to C2, and the lower branch of the branch corresponding to C3 is the branch corresponding to C4. The upper branch of a branch is adjacent to the branch. The lower branch of a branch is adjacent to the branch.

[0111] The deep features of the branch corresponding to C5 are the features obtained by pooling the output of C5 (i.e., the shallow features of the branch corresponding to C5) (pooling can reduce the height and width by 2 times respectively). The deep features of the next branch corresponding to C4 are the deep features of the branch corresponding to C5.

[0112] The purpose of pooling in the neural convolution module is to reduce the size of the feature map, compress feature information, and improve the computational efficiency of the model.

[0113] Common pooling methods include Max Pooling and Average Pooling, which can sample the feature map obtained by convolution, extract the maximum value or average value in the local area, and output it as the result.

[0114] The pooling layer can perform sampling through a sliding window, so during the pooling process, the size of the feature map can be changed by adjusting the window size and step size, thereby controlling the complexity and number of parameters of the model.

[0115] Because the pooling layer can reduce network parameters and improve computational efficiency, it is widely used in convolutional neural networks. At the same time, the pooling layer operation does reduce the size of the feature map by half, but not all pooling layers are like this. For example, you can set the step size to 1 or use strided convolution instead of pooling to control the size of the output feature map.

[0116] In an exemplary embodiment, the first fusion process includes:

[0117] The deep features of all branches are concatenated to form fused deep features.

[0118] The concatenation operation can be used to directly connect two features. For example, if the dimensions of the two input features X and Y are p and q, the dimension of the output feature Z after concatenating X and Y is p+q.

[0119] In an exemplary embodiment, the joint attention processing includes: determining the weight features of the first fused features based on the weight vector of each channel of the first fused features and the spatial mask; and performing residual connection based on the first fused features and the weight features of the first fused features.

[0120] In an exemplary embodiment, the weight vector of each channel in the features after the first fusion processing is extracted by the preset CAM module;

[0121] The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

[0122] In an exemplary embodiment, performing residual connection according to the first fused feature and the weight feature of the first fused feature includes:

[0123] Multiplying the first fused feature and the weight feature of the first fused feature by position;

[0124] The multiplication result is added to the feature after the first fusion processing.

[0125] In an exemplary embodiment, the feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch according to the shallow features of the branch and the screening features of the next branch of the branch;

[0126] Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the atrous spatial convolution pooling pyramid module.

[0127] In an exemplary embodiment, determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes:

[0128] Upsampling the screening feature of the next branch of the branch by a preset multiple; adding the upsampled feature to the shallow feature of the branch to obtain a first feature;

[0129] The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature;

[0130] The first feature is integrated with multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature;

[0131] The screening feature of the branch is determined according to the second feature and the third feature.

[0132] In an exemplary embodiment, determining the screening feature of the branch according to the second feature and the third feature includes:

[0133] The second feature and the third feature are added together to obtain the screening feature of the branch.

[0134] In an exemplary embodiment, the second fusion process includes: splicing the screening features of all branches.

[0135] The concatenation operation can be used to directly connect two features. For example, if the dimensions of the two input features X and Y are p and q, the dimension of the output feature Z after concatenating X and Y is p+q.

[0136] In an exemplary embodiment, the segmentation process includes: extracting a first semantic feature, performing a first upsampling, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

[0137] Exemplarily, the first semantic feature is extracted by two first preset convolution modules. After the first upsampling, a feature having a resolution size of the original input image can be obtained. The information between multiple channels is integrated by the second preset convolution module.

[0138] Exemplarily, the first preset convolution module may be a 3×3 convolution module, and the second preset convolution module may be a 1×1 convolution module. The normalization process is performed by a Sigmoid function.

[0139] In an exemplary embodiment, the edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling and normalization processes on the features obtained by the second fusion process in sequence.

[0140] Exemplarily, the second semantic feature is extracted by a first preset convolution module, the second upsampling and the third upsampling are respectively performed by a first deconvolution module, and normalization is performed by a Sigmoid function.

[0141] Exemplarily, the first preset convolution module may be a 3×3 convolution module, and the first deconvolution module may be a T-2×2 convolution module.

[0142] In an exemplary embodiment, the third fusion processing includes: splicing the features obtained by the segmentation processing and the features obtained by the edge enhancement processing; extracting the fourth semantic features, integrating the information between multiple channels, and normalizing the spliced ​​features in sequence.

[0143] Exemplarily, the fourth semantic feature is extracted by the first preset convolution module, the information between multiple channels is integrated by the second preset convolution module, and normalization is performed by the Sigmoid function.

[0144] The concatenation operation can be used to directly connect two features. For example, if the dimensions of the two input features X and Y are p and q, the dimension of the output feature Z after concatenating X and Y is p+q.

[0145] Figure 4 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure, Figure 4 It can be seen that the scene text processing method includes the following steps S21-S23.

[0146] S21, acquiring an image to be detected and performing feature extraction on the image to be detected;

[0147] S22, performing attention feature fusion, first fusion processing and joint attention processing on the features obtained by feature extraction in sequence;

[0148] S23, performing the scene text core area probability prediction and boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0149] The network architecture corresponding to the scene text processing method is as follows Figure 5 As shown. Figure 5 In Figure 5 As shown, the image to be detected 100 is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 200, an attention feature fusion module 300, a first fusion processing module 400, a joint attention module 500 and a post-processing module 600. Figure 5 The scene text network model in can be established as follows:

[0150] Constructing a training data set and an initial network model, wherein the training data set includes a plurality of preprocessed images including scene text and corresponding labels;

[0151] The initial network model is trained according to the training data set, and the trained network model is used as a preset scene text network model.

[0152] In an exemplary embodiment, when constructing the training data set, it may include preprocessing the images for constructing the training data set.

[0153] Among them, preprocessing can include image decoding, label encoding, basic data augmentation, channel transformation and normalization.

[0154] The image decoding may be to convert the image to be decoded into a Numpy format.

[0155] Label encoding can be to parse the label information in the txt file and save it in a unified format. The label information includes the location information of the manually marked frame and the text information. The txt file is the training data corresponding to the image, including the label information. The format can be the format of Baidu PPOCR data, including coordinate information and text information.

[0156] Basic data augmentation can include random horizontal flipping, random rotation, random scaling, random cropping, etc.

[0157] Channel transformation can be to transform the format of the image after the above processing from the image data format [H, W, C] (i.e., height, width and number of channels) to the format of training data used by the neural network [C, H, W] (i.e., number of channels, height and width).

[0158] Set the optimization parameters, number of iterations, loss function, and evaluation metrics, and input the preprocessed images into the initial network model training.

[0159] The amad method can be used to optimize the parameters. The number of iterations can be set to 1000. Each iteration will be evaluated once. When the maximum number of iterations is reached, the training will be terminated.

[0160] The loss function L is given by L s , L b and L t It consists of three parts, as follows:

[0161] L=L s +α×L b +β×L t ;

[0162] Among them, L s , L b , L t They represent the training losses of the probability map P, the approximate binary map B, and the threshold map T respectively. s and L b The binary cross entropy loss function is used, and the formula is as follows:

[0163]

[0164] Among them, Sl Represents the core area of ​​the text predicted during training, y i represents the label value of the i-th pixel position, x i Represents the predicted probability value of the position. The loss L of the threshold map T t The L1 loss is used, and the formula is as follows:

[0165]

[0166] R d is the text bounding box area, represents the label of the threshold map T at position j, is the predicted value of the threshold at this location.

[0167] There are three evaluation indicators for text detection, namely precision, recall and the harmonic mean (Hmean) of the first two. The calculation logic of the three indicators is:

[0168] 1. Create a matrix of size [n,m] called iouMat, where n is the number of GT (Ground Truth) boxes and m is the number of detected boxes; n,m are the number of boxes with text marked as "###" removed; "###" indicates unclear text. This is the text calibration method for data.

[0169] 2. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by n to get Recall;

[0170] 3. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by m to get Precision;

[0171] 4. The calculation formula of Hmean indicator is as follows:

[0172] Hmean=2.0×(Precision×Recall / Precision+Recall).

[0173] Save the weights of the trained model, use the trained network model as the preset scene text network model, and use the network model for text detection.

[0174] The feature extraction module 200 may use a convolutional neural network (e.g. Figure 6As shown in the figure, the feature extraction of the image to be detected is performed, and each of the four convolution modules conv2-conv5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolution module, and the channels are uniformly adjusted to 256 dimensions, and the features after the channel unification are recorded as in k (k=1,2,3,4).

[0175] The attention feature fusion module 300 includes an AFFM submodule and a pooling submodule. The AFFM (Attention Feature Fusion Module) submodule contains two inputs and one output. The input is: k (k=1,2,3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoded fusion feature after AFFM feature fusion is out k (k=1,2,3).

[0176] The pooling submodule is configured to reduce the output of the AFFM submodule, for example, reducing the height and width of the output of the AFFM submodule by 2 times respectively.

[0177] The first fusion processing module 400 concatenates the deep features of all branches to form fused deep features.

[0178] The global information interaction is embodied by the joint attention module 500, namely the JAM module (i.e., joint attention module: Joint Attention Module).

[0179] The post-processing module 600 is divided into two branches, upper and lower, for respectively performing text core area probability prediction and boundary threshold prediction.

[0180] In step S21, a convolutional neural network can be used to extract features of the image to be detected. Figure 6 Each of the four convolution modules conv2-conv5 in the above figure is used as a branch. The output features of each branch are passed through a 1×1 convolution module respectively, and the channels are uniformly adjusted to 256 dimensions. The features after channel unification are recorded as in k (k=1,2,3,4).

[0181] In step S22, the AFFM submodule (i.e., Attention Feature Fusion Module) includes two inputs and one output. The inputs are: k (k=1,2,3) is the feature of AFFM from the shallow encoding end (feature extraction stage); outk+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoded fusion feature after AFFM feature fusion is out k (k=1,2,3). AFFM submodules are as follows Figure 7 shown.

[0182] The operating mechanism of the AFFM submodule is as follows:

[0183] Out the deep features of the lower branch k+1 The C×1×1 channel weight vector is obtained through the CAM module and combined with the feature vector in from the encoder. k Multiply each channel to obtain the channel-weighted encoding features;

[0184] The shallow characteristics of the upper branch are in k The 1×H×W spatial information mask is obtained through the SAM module and combined with the out k+1 The features are multiplied position by position to obtain the decoded features weighted by spatial information;

[0185] The weighted encoding features and decoding features are added by position to obtain the fused decoding features out k .

[0186] The internal operation formula of the AFFM sub-model is expressed as:

[0187]

[0188] out4=in4.

[0189] When calculating out3, the lower branch deep features use out4. Similarly, when calculating out2, the lower branch deep features use out3. When calculating out1, the lower branch deep features use out2.

[0190] The core of the AFFM sub-module is to use the attention mechanism to extract different emphasis information of the deep and shallow layers to calibrate and strengthen the input feature vector, that is, use SAM to extract the spatial information of the shallow features to weight the upsampled deep features, and use the CAM module to extract the global information of the deep features to weight the shallow information from the encoding end, and then add and fuse the calibrated features of both sides.

[0191] In step S22, in order to improve the information representation ability of the cascaded features, convolution is often used to fuse the cascaded features. However, conventional convolution operations only fuse information in the local receptive field and ignore global information interaction. Figure 8 As shown in Figure 2, it reflects the global information interaction.

[0192] The processing flow of the JAM module is as follows:

[0193] Use CAM and SAM to model the dependency between the channel and spatial position of the input feature F, use the global information of the feature to obtain the weight of each channel, and use the relationship between the spatial context to highlight the features related to the target and suppress irrelevant background noise;

[0194] The obtained C×1×1 weight vector and 1×H×W spatial mask are expanded to the dimension of the input feature F and then multiplied by position to obtain the weight feature F' of the same dimension as F;

[0195] As the number of network layers increases, the objective function is prone to fall into a local optimal solution, and as the number of layers increases, the problem of gradient vanishing becomes more serious. The residual network is a very effective network for alleviating the gradient vanishing problem and can increase the depth of the network for effective training. Therefore, residual connections are added to the module to ensure the effectiveness of model training. The details of the residual connection implementation are shown in the following formula:

[0196] F″=JA(F)=F+F×F′=F+F×σ{EX[CA(F)]×EX[SA(F)]}.

[0197] In step S23, after the enhanced fusion feature F" is obtained through the above steps, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0198] 1) F" first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu, and the resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0199] 2) Perform a deconvolution operation on the obtained feature map with a convolution kernel of (2,2). The shape of the obtained feature map is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0200] 3) Perform deconvolution operation again, which is the same as the second step, except that the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the original image size;

[0201] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0202] Finally, the probability map P and the threshold map T are transformed into the final binary map B (such as Fig. 9 shown).

[0203] Among them, the formula for differentiable binarization is:

[0204]

[0205] The standard binarization method is not differentiable, which makes the network unable to be trained end-to-end. To solve this problem, a differentiable binarization method is used in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold map T. The threshold at each position of the image is predicted by the network instead of using a fixed value, which can better separate the text background and foreground.

[0206] The disclosed embodiment can improve the accuracy of text detection and pay attention to global information interaction in scenarios with variable shapes, drastic scale changes, and complex and diverse backgrounds through attention feature fusion processing, thereby improving information representation capabilities.

[0207] Fig.10 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure, Fig.10 It can be seen that the scene text processing method includes the following steps S101-S103.

[0208] S101, acquiring an image to be detected and performing feature extraction on the image to be detected;

[0209] S102, performing attention feature fusion and first fusion processing on the features obtained by feature extraction in sequence;

[0210] S103, performing the scene text core area probability prediction and boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0211] The network architecture corresponding to the scene text processing method is as follows Fig.11 As shown. Fig.11 In the embodiment, the image to be detected 100 is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 200, an attention feature fusion module 300, a first fusion processing module 400 and a post-processing module 600. Fig.11 The scene text network model in can be established as follows:

[0212] Constructing a training data set and an initial network model, wherein the training data set includes a plurality of preprocessed images including scene text and corresponding labels;

[0213] The initial network model is trained according to the training data set, and the trained network model is used as a preset scene text network model.

[0214] In an exemplary embodiment, when constructing the training data set, it may include preprocessing the images for constructing the training data set.

[0215] Among them, preprocessing can include image decoding, label encoding, basic data augmentation, channel transformation and normalization.

[0216] The image decoding may be to convert the image to be decoded into a Numpy format.

[0217] Label encoding can be to parse the label information in the txt file and save it in a unified format. The label information includes the location information of the manually marked frame and the text information. The txt file is the training data corresponding to the image, including the label information. The format can be the format of Baidu PPOCR data, including coordinate information and text information.

[0218] Basic data augmentation can include random horizontal flipping, random rotation, random scaling, random cropping, etc.

[0219] Channel transformation can be to transform the format of the image after the above processing from the image data format [H, W, C] (i.e., height, width and number of channels) to the format of training data used by the neural network [C, H, W] (i.e., number of channels, height and width).

[0220] Set the optimization parameters, number of iterations, loss function, and evaluation metrics, and input the preprocessed images into the initial network model training.

[0221] The amad method can be used to optimize the parameters. The number of iterations can be set to 1000. Each iteration will be evaluated once. When the maximum number of iterations is reached, the training will be terminated.

[0222] The loss function L is given by L s , L b and L t It consists of three parts, as follows:

[0223] L=L s +α×L b +β×L t ;

[0224] Among them, L s , L b , L t They represent the training losses of the probability map P, the approximate binary map B, and the threshold map T respectively. s and L b The binary cross entropy loss function is used, and the formula is as follows:

[0225]

[0226] Among them, S l Represents the core area of ​​the text predicted during training, y i represents the label value of the i-th pixel position, x i Represents the predicted probability value of the position. The loss L of the threshold map T t The L1 loss is used, and the formula is as follows:

[0227]

[0228] R d is the text bounding box area, represents the label of the threshold map T at position j, is the predicted value of the threshold at this location.

[0229] There are three evaluation indicators for text detection, namely precision, recall and the harmonic mean (Hmean) of the first two. The calculation logic of the three indicators is:

[0230] 1. Create a matrix of size [n,m] called iouMat, where n is the number of GT (Ground Truth) boxes and m is the number of detected boxes; n,m are the number of boxes with text marked as "###" removed; "###" indicates unclear text. This is the text calibration method for data.

[0231] 2. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by n to get Recall;

[0232] 3. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by m to get Precision;

[0233] 4. The calculation formula of Hmean indicator is as follows:

[0234] Hmean=2.0×(Precision×Recall / Precision+Recall).

[0235] Save the weights of the trained model, use the trained network model as the preset scene text network model, and use the network model for text detection.

[0236] The feature extraction module 200 may use a convolutional neural network (e.g. Figure 6As shown in the figure, the feature extraction of the image to be detected is performed, and each of the four convolution modules conv2-conv5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolution module, and the channels are uniformly adjusted to 256 dimensions, and the features after the channel unification are recorded as in k (k=1,2,3,4).

[0237] The attention feature fusion module 300 includes an AFFM submodule and a pooling submodule. The AFFM (Attention Feature Fusion Module) submodule contains two inputs and one output. The input is: k (k=1,2,3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoded fusion feature after AFFM feature fusion is out k (k=1,2,3).

[0238] The pooling submodule is configured to reduce the output of the AFFM submodule, for example, reducing the height and width of the output of the AFFM submodule by 2 times respectively.

[0239] The first fusion processing module 400 concatenates the deep features of all branches to form fused deep features.

[0240] The post-processing module 600 is divided into two branches, upper and lower, for respectively performing text core area probability prediction and boundary threshold prediction.

[0241] In step S101, a convolutional neural network can be used to extract features of the image to be detected. Figure 6 Each of the four convolution modules conv2-conv5 in the above figure is used as a branch. The output features of each branch are passed through a 1×1 convolution module respectively, and the channels are uniformly adjusted to 256 dimensions. The features after channel unification are recorded as in k (k=1,2,3,4).

[0242] In step S102, the AFFM submodule (ie, Attention Feature Fusion Module) includes two inputs and one output. The inputs are: k (k=1,2,3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoded fusion feature after AFFM feature fusion is outk (k=1,2,3). AFFM submodules are as follows Figure 7 shown.

[0243] The operating mechanism of the AFFM submodule is as follows:

[0244] Out the deep features of the lower branch k+1 The C×1×1 channel weight vector is obtained through the CAM module and combined with the feature vector in from the encoder. k Multiply each channel to obtain the channel-weighted encoding features;

[0245] The shallow characteristics of the upper branch are in k The 1×H×W spatial information mask is obtained through the SAM module and combined with the out k+1 The features are multiplied position by position to obtain the decoded features weighted by spatial information;

[0246] The weighted encoding features and decoding features are added by position to obtain the fused decoding features out k .

[0247] The internal operation formula of the AFFM sub-model is expressed as:

[0248]

[0249] out4=in4.

[0250] When calculating out3, the lower branch deep features use out4. Similarly, when calculating out2, the lower branch deep features use out3. When calculating out1, the lower branch deep features use out2.

[0251] The core of the AFFM sub-module is to use the attention mechanism to extract different emphasis information of the deep and shallow layers to calibrate and strengthen the input feature vector, that is, use SAM to extract the spatial information of the shallow features to weight the upsampled deep features, and use the CAM module to extract the global information of the deep features to weight the shallow information from the encoding end, and then add and fuse the calibrated features of both sides.

[0252] In step S93, after the enhanced fusion feature F is obtained through the above steps, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0253] 1) F first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu to obtain a feature map with shape (batch, 64, 1 / 4W, 1 / 4H);

[0254] 2) Perform a deconvolution operation on the obtained feature map with a convolution kernel of (2,2). The shape of the obtained feature map is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0255] 3) Perform deconvolution operation again, which is the same as the second step, except that the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the original image size;

[0256] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0257] Finally, the probability map P and the threshold map T are transformed into the final binary map B (such as Fig. 9 shown).

[0258] Among them, the formula for differentiable binarization is:

[0259]

[0260] The standard binarization method is not differentiable, which makes it impossible to train the network end-to-end. In order to solve this problem, a differentiable binarization processing method is adopted in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold graph T. The threshold at each position of the image is predicted through the network instead of using a fixed value, which can better separate the text background and foreground. The disclosed embodiment can improve the accuracy of text detection in scenes with variable shapes, drastic scale changes, and complex and diverse backgrounds through attention feature fusion processing.

[0261] Fig.12 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure, such as Fig.12 As shown, the scene text processing method includes the following steps S121-S123.

[0262] S121, obtaining an image to be detected and performing preprocessing and feature extraction on the image to be detected;

[0263] S122, performing feature screening, second fusion processing, segmentation processing, edge enhancement and third fusion processing on the features obtained by feature extraction in sequence;

[0264] S123, performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0265] Fig.13The network architecture corresponding to the scene text processing method is shown. Fig.13 The process of establishing the scene text network model shown can be similar to Figure 5 The process of establishing the scene text network model in . Fig.13 As shown, the image 10 to be detected is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 20, a feature screening module 30, a second fusion processing module 40, a segmentation module 50, an edge enhancement module 60, a third fusion processing module 70 and a post-processing module 80. The convolutional neural network is set to feature extraction. The feature screening module 30 includes an ASPP (Atrous Spatial Pyramid Pooling, hollow spatial convolution pooling pyramid) submodule and an FF (Feature Filter, feature screening) submodule. The ASPP submodule (hollow spatial convolution pooling pyramid module) is set to extract the deepest layer information from the highest dimensional features (i.e., the features output by the deepest convolution module) to use the deepest layer information as the initial internal features of the feature screening submodule (FF). The FF submodule is set to perform feature screening. The second fusion processing module 40 is set to splice the features after feature screening. The segmentation module 50 is set to segment the features spliced ​​by the second fusion processing module. The edge enhancement module 60 is set to perform edge enhancement processing on the features spliced ​​by the second fusion processing module. The third fusion processing module 70 is configured to splice the features after segmentation processing and the features after edge enhancement processing. The post-processing module 80 is configured to perform the scene text core area probability prediction and boundary threshold prediction according to the output of the third fusion processing module to obtain the binary image of the image to be detected.

[0266] In step S121, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolutional modules c2-c5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolutional module, and the channels are uniformly adjusted to 256 dimensions. The features after the channels are unified are recorded as in k (k=1,2,3,4). The convolutional neural network may be ResNet-18.

[0267] In step S122, the structure diagram of the feature screening submodule corresponding to the feature screening is as follows: Fig.14 As shown in FIG. UP indicates that the input feature is upsampled using bilinear interpolation, and the upsampled resolution is twice the original. The 1×1 convolution module is set to change the number of channels of the input feature. σ indicates the sigmoid operation, which is used to normalize the input feature. The input feature in step S72 refers to the feature input to the corresponding module. S k+1Indicates the features after the lower branch feature screening, k = 1, 2, 3. S k Indicates the characteristics of this branch after feature filtering (i.e., filtering characteristics).

[0268] Combination Fig.13 It can be seen that in4 will obtain the initial internal features through ASPP, and the initial internal features will be input into the FF module together with in3 as S4 to obtain S3, in2 and S3 will be input into the FF module to obtain S2, and in1 and S2 will be input into the FF module to obtain S1.

[0269] Taking the process of obtaining S3 as an example, the feature screening process is as follows:

[0270] Upsample S4;

[0271] The up-sampled result of S4 is added to in3, and the number of channels of the added result is changed; after the number of channels is changed, a sigmoid operation is performed to obtain the first value;

[0272] The first value is multiplied by the result of upsampling by S4 to obtain a second value;

[0273] Change the number of channels of in3 and perform sigmoid operation to get the third value;

[0274] The third value is multiplied by in3 to get the fourth value;

[0275] The second value is added to the fourth value to obtain the feature S3.

[0276] In step S122, the second fusion processing module 40 is configured to concatenate S4, S3, S2, and S1 to obtain f.

[0277] In step S122, the segmentation process is performed by Fig.15 The segmentation module shown is used to perform binary classification of text area and background. In the present disclosure, all text areas are predicted as foreground and are not distinguished at the neural network level.

[0278] In the segmentation module, UP represents 4 times upsampling using bilinear interpolation. The input feature f is a fused multi-scale feature. It first passes through two 3×3 convolutions to extract binary semantic information from the fused features, and then upsamples the obtained features to the original input resolution size. It then passes through a 1×1 convolution to integrate information between multiple channels, and converts the output features into a single channel. Finally, the integrated features are normalized using the sigmoid function. The final output h is a probability heat map with pixel values ​​ranging from 0 to 1, with a dimension of H×W×1. In the output prediction map, the pixel values ​​of the background part are close to 0, while the pixel values ​​of all text areas are close to 1.

[0279] In step S122, the edge enhancement process is performed by Fig.16 The edge enhancement module 60 shown is used to detect the boundary of the text area, so that the text and the background can be accurately distinguished on the one hand, and the text belonging to different text areas can be effectively separated on the other hand to avoid adhesion.

[0280] The T-2×2 in the edge enhancement module 60 represents a deconvolution with a convolution kernel of 2×2, which is used to upsample the resolution by 2 times. The width and height of the input feature f are W / 4 and H / 4 respectively, and the output g is a feature map with a dimension of H×W×1. Therefore, the resolution of the output feature is 4 times that of the input, and the edge features of the text area are predicted.

[0281] After being processed by the edge enhancement module 60, the accuracy of the text area boundary can be improved, and the problem of fuzzy edge detection of the text and reduced accuracy of the boundary area caused by the use of a segmentation algorithm can be improved.

[0282] The boundary line of the original text is a manually marked box, which does not need to be recognized and is directly mapped to the image. In the process of manual box drawing, the boundary can be reduced. The reason for reducing the boundary box is to deal with the situation where the detection boxes of the previous and next lines of text overlap too much, causing adhesion. The value can be adjusted. In the existing scheme, the value is generally adjusted to between 0.7-0.8 and 0.3-0.4. The following method can be used to reduce the boundary.

[0283] The boundaries of the text area are represented by Where V m Represents the coordinate points on the boundary, and n represents the number of coordinate points. The boundary line of the original text area is taken as the outer boundary L1, and the boundary line of the text area is reduced inward by 0.7 and 0.3 respectively as the center L2 and inner boundary L3 of the boundary area. The pixel value of the L2 position is set to 1, and the pixel value outside L1 and inside L3 is set to 0, and then the text edge area between L1 and L3 is filled using the linear interpolation method. The boundary reduction formula used in the above process is:

[0284]

[0285] Where A and D represent the area and perimeter of the text area, respectively; r represents the ratio of inward reduction, which is 0.7 and 0.3 for L2 and L3, respectively; and d is the distance that the boundary point coordinates are reduced inward.

[0286] In step S122, the third fusion process is performed as follows Fig.17The third fusion module is shown in the figure. In the structure of the fusion module, g and h represent the output of the edge enhancement module and the output of the segmentation module respectively. They first undergo a cascade operation in the channel dimension, and then further extract semantic features through a 3×3 convolution operation. Then, 1×1 convolution is used to further integrate the features after the initial fusion. In the fusion stage, features of different levels are not used for feature fusion of the attention mechanism. Instead, the features of the same size dimension of the edge enhancement module and the segmentation module are fused after the above operation. Then, the sigmoid function is used to normalize the integrated semantic information, and finally a suitable binarization threshold is selected for post-processing.

[0287] In step S123, after the enhanced fusion feature F" is obtained through the third fusion process, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0288] 1) F" first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu, and the resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0289] 2) Perform a deconvolution operation on the obtained feature map with a convolution kernel of (2,2). The shape of the obtained feature map is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0290] 3) Perform deconvolution operation again, which is the same as the second step, except that the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the original image size;

[0291] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0292] Finally, the probability map P and the threshold map T are transformed into the final binary map B (such as Fig. 9 shown).

[0293] Among them, the formula for differentiable binarization is:

[0294]

[0295] The standard binarization method is not differentiable, which makes the network unable to be trained end-to-end. To solve this problem, a differentiable binarization method is used in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold map T. The threshold at each position of the image is predicted by the network instead of using a fixed value, which can better separate the text background and foreground.

[0296] The disclosed embodiment can extract effective information in the process of reverse restoration of features from low resolution to high resolution through feature screening, segmentation processing and edge enhancement processing, thereby improving the accuracy of text area boundaries.

[0297] Fig.18 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure, such as Fig.18 As shown, the scene text processing method includes the following steps S181-S183.

[0298] S181, obtaining an image to be detected and performing preprocessing and feature extraction on the image to be detected;

[0299] S182, performing feature screening and second fusion processing on the features obtained by feature extraction in sequence;

[0300] S183, performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0301] Fig.19 The network architecture corresponding to the scene text processing method is shown. Fig.19 The process of establishing the scene text network model shown can be similar to Figure 5 The process of establishing the scene text network model in . Fig.19As shown, the image to be detected 10 is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 20, a feature screening module 30, a second fusion processing module 40 and a post-processing module 80. The convolutional neural network is set to feature extraction. The feature screening module 30 includes an ASPP (Atrous Spatial Pyramid Pooling, hollow space convolution pooling pyramid) submodule and an FF (Feature Filter, feature screening) submodule. The ASPP submodule (hollow space convolution pooling pyramid module) is set to extract the deepest layer information from the highest dimensional features (i.e., the features output by the deepest convolution module) to use the deepest layer information as the initial internal features of the feature screening submodule (FF). The FF submodule is set to perform feature screening. The second fusion processing module 40 is set to splice the features after feature screening. The post-processing module 80 is set to perform the scene text core area probability prediction and boundary threshold prediction according to the output of the third fusion processing module to obtain a binary image of the image to be detected.

[0302] In step S181, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolutional modules c2-c5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolutional module, and the channels are uniformly adjusted to 256 dimensions. The features after the channels are unified are recorded as in k (k=1,2,3,4). The convolutional neural network may be ResNet-18.

[0303] In step S182, the structure diagram of the feature screening submodule corresponding to the feature screening is as follows: Fig.14 As shown in FIG. UP indicates that the input feature is upsampled using bilinear interpolation, and the upsampled resolution is twice the original. The 1×1 convolution module is set to change the number of channels of the input feature. σ indicates the sigmoid operation, which is used to normalize the input feature. The input feature in step S72 refers to the feature input to the corresponding module. S k+1 Indicates the features after the lower branch feature screening, k = 1, 2, 3. S k Indicates the characteristics of this branch after feature filtering (i.e., filtering characteristics).

[0304] Combination Fig.13 It can be seen that in4 will obtain the initial internal features through ASPP, and the initial internal features will be input into the FF module together with in3 as S4 to obtain S3, in2 and S3 will be input into the FF module to obtain S2, and in1 and S2 will be input into the FF module to obtain S1.

[0305] Taking the process of obtaining S3 as an example, the feature screening process is as follows:

[0306] Upsample S4;

[0307] The up-sampled result of S4 is added to in3, and the number of channels of the added result is changed; after the number of channels is changed, a sigmoid operation is performed to obtain the first value;

[0308] The first value is multiplied by the result of upsampling by S4 to obtain a second value;

[0309] Change the number of channels of in3 and perform sigmoid operation to get the third value;

[0310] The third value is multiplied by in3 to get the fourth value;

[0311] The second value is added to the fourth value to obtain the feature S3.

[0312] In step S182, the second fusion processing module 40 is configured to concatenate S4, S3, S2, and S1 to obtain f.

[0313] In step S183, after the feature f is obtained through the second fusion process, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0314] 1) f first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu, and the resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0315] 2) Perform a deconvolution operation on the obtained feature map with a convolution kernel of (2,2). The shape of the obtained feature map is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0316] 3) Perform deconvolution operation again, which is the same as the second step, except that the output feature map channel is 1, and the obtained feature map shape is (batch, W, H), which is the original image size;

[0317] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0318] Finally, the probability map P and the threshold map T are transformed into the final binary map B (such as Fig. 9 shown).

[0319] Among them, the formula for differentiable binarization is:

[0320]

[0321] The standard binarization method is not differentiable, which makes it impossible to train the network end-to-end. In order to solve this problem, a differentiable binarization processing method is adopted in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold graph T. The threshold at each position of the image is predicted through the network instead of using a fixed value, which can better separate the text background and foreground. The disclosed embodiment improves the detection capability of multi-scale text through feature screening.

[0322] The present invention provides a scene text detection device, including an acquisition module, a feature extraction module,

[0323] Processing module, post-processing module; wherein,

[0324] The acquisition module is configured to acquire an image to be detected;

[0325] The feature extraction module is configured to extract features from the image to be detected;

[0326] The processing module is configured to process the features obtained by feature extraction;

[0327] The post-processing module is configured to perform a probability prediction of the scene text core area and a boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected;

[0328] The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module;

[0329] The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

[0330] The present disclosure provides a scene text detection device, including a memory and a processor.

[0331] The memory is used to store a program for scene text detection;

[0332] The processor is used to read and execute the program for scene text detection and execute the above-mentioned scene text detection method.

[0333] The present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable the computer to execute the above-mentioned scene text detection method.

[0334] The drawings in this disclosure only relate to the structures involved in this disclosure, and other structures may refer to the general design. In the absence of conflict, the embodiments of this disclosure and the features in the embodiments may be combined with each other to obtain new embodiments.

[0335] The above embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to the contents specifically shown and described herein. Various modifications, substitutions or omissions may be made to the forms and details of the implementation without departing from the scope of the present disclosure.

Claims

1. A scene text detection method, characterized in that: include: Acquire an image to be detected and perform feature extraction on the image to be detected; wherein the image to be detected contains scene text; Processing the features obtained through feature extraction; According to the processing results, the scene text core area probability prediction and the boundary threshold prediction are performed to obtain a binary image of the image to be detected; The processing includes at least one of the following: Attentional feature fusion and first fusion processing; joint attention processing; Or the processing includes at least one of the following: Feature screening and second fusion processing; Segmentation, edge enhancement and third fusion processing; The feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

2. The scene text detection method according to claim 1, characterized in that: The attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, the deep features of the branch are determined according to the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

3. The scene text detection method according to claim 2, characterized in that: Determine the channel-weighted coding features of the branch according to the deep features of the next branch of the branch, the shallow features of the branch, and a preset CAM module; Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch of the branch after upsampling, and the preset SAM module; The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

4. The scene text detection method according to claim 3, characterized in that: Determining the channel-weighted coding features of the branch according to the deep features of the lower branch of the branch, the shallow features of the branch, and a preset CAM module, including: The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

5. The scene text detection method according to claim 3, characterized in that: Determine the weighted decoding features of the spatial information of the branch according to the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and the preset SAM module, including: The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

6. The scene text detection method according to claim 3, characterized in that: The first fusion processing includes: splicing the deep features of all branches to form fused deep features.

7. The scene text detection method according to claim 3, characterized in that: The joint attention processing includes: determining the weight features of the features after the first fusion processing according to the weight vector of each channel of the features after the first fusion processing and the spatial mask; and performing residual connection according to the features after the first fusion processing and the weight features of the features after the first fusion processing.

8. The scene text detection method according to claim 7, characterized in that: Extracting a weight vector of each channel in the features after the first fusion processing through the preset CAM module; The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

9. The scene text detection method according to claim 7, characterized in that: Performing residual connection according to the features after the first fusion processing and the weight features of the features after the first fusion processing includes: Multiplying the first fused feature and the weight feature of the first fused feature by position; The multiplication result is added to the feature after the first fusion processing.

10. The scene text detection method according to claim 1, characterized in that: The feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch according to the shallow features of the branch and the screening features of the next branch of the branch; Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the atrous spatial convolution pooling pyramid module.

11. The scene text detection method according to claim 10, characterized in that: Determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes: Upsampling the screening feature of the next branch of the branch by a preset multiple; adding the upsampled feature to the shallow feature of the branch to obtain a first feature; The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature; The first feature is integrated with multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature; The screening feature of the branch is determined according to the second feature and the third feature.

12. The scene text detection method according to claim 10, characterized in that: The second fusion process includes: splicing the screening features of all branches.

13. The scene text detection method according to claim 1, characterized in that: The segmentation process includes: extracting a first semantic feature, performing a first upsampling, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

14. The scene text detection method according to claim 1, characterized in that: The edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling and normalization processes on the features obtained by the second fusion process in sequence.

15. The scene text detection method according to claim 1, characterized in that: The third fusion process includes: splicing the features obtained by the segmentation process and the features obtained by the edge enhancement process; extracting the fourth semantic features, integrating the information between multiple channels, and normalizing the spliced ​​features in sequence.

16. A scene text detection device, characterized in that: Including acquisition module, feature extraction module, Processing module, post-processing module; wherein, The acquisition module is configured to acquire an image to be detected; The feature extraction module is configured to extract features from the image to be detected; The processing module is configured to process the features obtained by feature extraction; The post-processing module is configured to perform a probability prediction of the scene text core area and a boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected; The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module; The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

17. A scene text detection device, comprising a memory and a processor, characterized in that: The memory is used to store a program for scene text detection; The processor is used to read and execute the program for scene text detection, and to perform the method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions, wherein: The computer executable instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

Images