Scene text detection method and apparatus, and storage medium

Abstract

A scene text detection method, comprising: acquiring an image to be subjected to detection, and performing feature extraction on said image, wherein said image contains scene text; processing features obtained by means of the feature extraction; and on the basis of a processing result, performing core-region probability prediction and boundary threshold prediction on the scene text, so as to obtain a binary image of said image, wherein the processing at least comprises one of the following: attention feature fusion and first fusion processing, and joint attention processing, or the processing at least comprises one of the following: feature screening and second fusion processing, and segmentation processing, edge enhancement and third fusion processing, and the feature extraction comprises: performing feature extraction on said image by means of a convolutional neural network, and adjusting a feature dimension of a plurality of extracted branches to a preset dimension, so as to obtain a shallow feature of the plurality of branches.

Description

Scene text detection method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 29, 2023, with application number 202311102556.7 and invention name “Scene text detection method, device and storage medium”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] This article relates to but is not limited to the field of computer vision, and in particular to a scene text detection method, device, and storage medium. Background Art

[0003] Currently, in the field of computer vision, text detection encompasses a wide range of areas, including document recognition, form recognition, bill recognition, and ID card recognition. While detecting these types of text is relatively straightforward, detecting text in scenes with varying shapes, dramatic scale variations, and complex and diverse backgrounds is more challenging due to the higher requirements for shape description, multi-scale detection, and resistance to background interference.

[0004] Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present disclosure provides a scene text detection method, including:

[0007] Acquire an image to be detected and perform feature extraction on the image to be detected; wherein the image to be detected contains scene text;

[0008] Processing the features obtained through feature extraction;

[0009] Performing the scene text core area probability prediction and boundary threshold prediction according to the processing results to obtain a binary image of the image to be detected;

[0010] The processing includes at least one of the following:

[0011] Attentional feature fusion and first fusion processing;

[0012] joint attention processing;

[0013] Or the processing includes at least one of the following:

[0014] Feature screening and second fusion processing;

[0015] Segmentation processing, edge enhancement and third fusion processing;

[0016] The feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to preset dimensions to obtain shallow features of the multiple branches.

[0017] In an exemplary embodiment, the attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, determining the deep features of the branch based on the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

[0018] In an exemplary embodiment, the channel-weighted coding feature of the branch is determined based on the deep feature of the next branch of the branch, the shallow feature of the branch, and a preset CAM module;

[0019] Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch after upsampling, and the preset SAM module;

[0020] The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

[0021] In an exemplary embodiment, determining the channel-weighted coding feature of the branch according to the deep feature of the lower branch of the branch, the shallow feature of the branch, and a preset CAM module includes:

[0022] The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

[0023] In an exemplary embodiment, determining the weighted decoding features of the spatial information of the branch based on the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and a preset SAM module includes:

[0024] The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

[0025] In an exemplary embodiment, the first fusion process includes: concatenating the deep features of all branches to form a fused deep feature.

[0026] In an exemplary embodiment, the joint attention processing includes: determining the weight features of the first fused features based on the weight vector of each channel of the first fused features and the spatial mask; and performing residual connection based on the first fused features and the weight features of the first fused features.

[0027] In an exemplary embodiment, the weight vector of each channel in the features after the first fusion processing is extracted by the preset CAM module;

[0028] The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

[0029] In an exemplary embodiment, performing residual connection based on the first fused feature and the weight feature of the first fused feature includes:

[0030] Multiplying the first fused feature and the weighted feature of the first fused feature by position;

[0031] The multiplication result is added to the first fusion-processed feature.

[0032] In an exemplary embodiment, the feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch based on the shallow features of the branch and the screening features of the next branch of the branch;

[0033] Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the void space convolution pooling pyramid module.

[0034] In an exemplary embodiment, determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes:

[0035] Upsampling the screening features of the next branch of the branch by a preset multiple; adding the upsampled features to the shallow features of the branch to obtain the first feature;

[0036] The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature;

[0037] The first feature is integrated with the multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature;

[0038] The screening feature of the branch is determined according to the second feature and the third feature.

[0039] In an exemplary embodiment, the second fusion process includes: concatenating the screening features of all branches.

[0040] In an exemplary embodiment, the segmentation process includes: extracting a first semantic feature, performing a first upsampling process, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

[0041] In an exemplary embodiment, the edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling, and normalizing processes on the features obtained by the second fusion process in sequence.

[0042] In an exemplary embodiment, the third fusion processing includes: splicing the features obtained by the segmentation processing and the features obtained by the edge enhancement processing; extracting the fourth semantic feature, integrating information between multiple channels, and normalizing the spliced ​​features in turn.

[0043] The present disclosure provides a scene text detection device, comprising an acquisition module, a feature extraction module,

[0044] Processing module, post-processing module; wherein the acquisition module is configured to acquire an image to be detected;

[0045] The feature extraction module is configured to extract features from the image to be detected;

[0046] The processing module is configured to process the features obtained by feature extraction;

[0047] The post-processing module is configured to perform probability prediction of the scene text core area and boundary threshold prediction according to the processing results to obtain a binary image of the image to be detected;

[0048] The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module;

[0049] The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to preset dimensions to obtain shallow features of the multiple branches.

[0050] The present disclosure provides a scene text detection device, comprising a memory and a processor.

[0051] The memory is used to store a program for scene text detection;

[0052] The processor is configured to read and execute the program for scene text detection and perform the above-mentioned scene text detection method.

[0053] The present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable the computer to execute the above-mentioned scene text detection method.

[0054] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0055] Summary of the Figures

[0056] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure. The shapes and sizes of one or more components in the accompanying drawings do not reflect the actual scale and are intended only to illustrate the contents of the present disclosure.

[0057] FIG1 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0058] FIG2 is a schematic diagram of a preset CAM module according to at least one embodiment of the present disclosure;

[0059] FIG3 is a schematic diagram of a preset SAM module according to at least one embodiment of the present disclosure;

[0060] FIG4 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0061] FIG5 is a schematic diagram of a network model for a scenario according to at least one embodiment of the present disclosure;

[0062] Figure 6 is a structural diagram of the convolutional module of a convolutional neural network;

[0063] FIG7 is a structural diagram of the attention feature fusion module of the model shown in FIG5 ;

[0064] FIG8 is a structural diagram of the joint attention module of the model shown in FIG5 ;

[0065] Figure 9 is a diagram of the differentiable binarization process;

[0066] FIG10 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0067] FIG11 is a schematic diagram of a network model for a scenario according to at least one embodiment of the present disclosure;

[0068] FIG12 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0069] FIG13 is a schematic diagram of a network model for a scenario according to at least one embodiment of the present disclosure;

[0070] FIG14 is a structural diagram of the FF module of the model shown in FIG13 ;

[0071] FIG15 is a structural diagram of the segmentation module of the model shown in FIG13 ;

[0072] FIG16 is a structural diagram of the edge enhancement module of the model shown in FIG13 ;

[0073] FIG17 is a structural diagram of the third fusion processing module of the model shown in FIG13 ;

[0074] FIG18 is a flowchart of a scene text detection method according to at least one embodiment of the present disclosure;

[0075] FIG19 is a schematic diagram of a scenario network model according to at least one embodiment of the present disclosure.

[0076] Details

[0077] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The embodiments can be implemented in a variety of different forms. A person of ordinary skill in the art can easily understand the fact that the method and content can be transformed into one or more forms without departing from the purpose and scope of the present disclosure. Therefore, the present disclosure should not be interpreted as being limited to the contents described in the following embodiments. In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other in any way.

[0078] In the present disclosure, ordinal numbers such as "first," "second," and "third" are provided to avoid confusion among constituent elements, and are not intended to limit the number. In the present disclosure, "plurality" refers to a number of two or more.

[0079] Compared to document text, text in natural scenes is more susceptible to factors such as its own variable shape, dramatic scale changes, and complex backgrounds. Therefore, scene text detection models have higher requirements in terms of shape description, multi-scale detection, and resistance to background interference. Based on this, this paper uses the segmentation-based DBNet as the basic model framework and then integrates the designed processing method into the DBNet framework in a modular form, achieving high-precision detection of text in scenes with variable shape, dramatic scale changes, and complex and diverse backgrounds.

[0080] FIG1 is a flow chart of a scene text detection method according to at least one embodiment of the present disclosure. As shown in FIG1 , the scene text detection method includes the following steps S11 to S13:

[0081] S11, obtaining an image to be detected and performing feature extraction on the image to be detected;

[0082] S12, processing the features obtained by feature extraction;

[0083] S13. Performing probability prediction of the scene text core area and boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0084] In an exemplary embodiment, in step S11, the image to be detected contains scene text. Scene text generally refers to text in a real scene image.

[0085] In step S11, in an exemplary embodiment, the feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to preset dimensions to obtain shallow features of the multiple branches.

[0086] In an exemplary embodiment, the convolutional neural network (ResNet-18 can be used as the backbone network, which is mainly composed of 5 convolution modules, namely conv1, conv2, conv3, conv4 and conv5) can extract the features of conv2, conv3, conv4 and conv5 respectively, and the extracted features can be adjusted to, for example, 256 dimensions through a 1×1 convolution module. Conv2, conv3, conv4 and conv5 can be considered as a branch respectively, and the features output by conv2, conv3, conv4 and conv5 are unified in dimension as the shallow features of their respective branches. The preset dimension can be 256 dimensions.

[0087] In an exemplary embodiment, in step S12, the processing includes at least one of the following:

[0088] Attentional feature fusion and first fusion processing;

[0089] joint attention processing;

[0090] Or the processing includes at least one of the following:

[0091] Feature screening and second fusion processing;

[0092] Segmentation processing, edge enhancement and third fusion processing.

[0093] Among them, attention feature fusion uses the attention mechanism to extract different emphasis information in deep and shallow layers to calibrate and strengthen the input feature vector, thereby performing high-precision text detection in scenes with variable shapes, drastic scale changes, and complex and diverse backgrounds.

[0094] The first fusion process fuses the output features of the attention feature fusion.

[0095] Joint attention processing improves information representation capabilities through global information interaction.

[0096] Feature screening can extract effective information in the process of reverse restoration of features from low resolution to high resolution through multi-scale feature screening.

[0097] The second fusion process fuses the output features of the feature screening.

[0098] The segmentation process can classify the text area and the background into two categories.

[0099] Edge enhancement improves the accuracy of text area boundaries and solves the problem of fuzzy text edge detection and reduced boundary area accuracy caused by the use of segmentation algorithms.

[0100] The third fusion process fuses the output features of the segmentation process and the output features of the edge enhancement process.

[0101] In an exemplary embodiment, the attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, determining the deep features of the branch based on the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

[0102] In an exemplary embodiment, the channel-weighted coding feature of the branch is determined based on the deep feature of the next branch of the branch, the shallow feature of the branch, and a preset CAM module;

[0103] Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch after upsampling, and the preset SAM module;

[0104] The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

[0105] Among them, the preset CAM module (Channel Attention Module) is a channel attention module. The preset CAM module can make the network pay more attention to the channel characteristics and position information of the target.

[0106] As shown in Figure 2, the preset CAM module first applies maximum pooling (MaxPooling) and average pooling (AvgPooling) to the input feature map F at each spatial position to obtain two C×1×1 vectors, which are then sent to a shared multi-layer perceptron MLP (i.e., a neural network) containing two fully connected layers (FC). Finally, the pixels are added and fused, and after an activation function, the channel attention map Mc is obtained.

[0107] The preset SAM module (Spatial Attention Model) is a spatial attention module. The preset SAM module can highlight the area of ​​interest, thereby improving the accuracy and efficiency of image segmentation.

[0108] As shown in Figure 3, the preset SAM module first applies MaxPooling and AvgPooling on each channel to obtain two 1×H×W feature maps, which are then fused by channel and sent to a standard convolutional layer. After the activation function, the spatial attention map Ms is obtained.

[0109] In an exemplary embodiment, determining the channel-weighted coding feature of the branch according to the deep feature of the lower branch of the branch, the shallow feature of the branch, and a preset CAM module includes:

[0110] The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

[0111] In an exemplary embodiment, determining the weighted decoding features of the spatial information of the branch based on the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and a preset SAM module includes:

[0112] The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

[0113] The purpose of upsampling the deep features of the next branch of the branch is to restore the image size, because in order to obtain the deep features of the branch, pooling is performed after the attention feature fusion process.

[0114] The deepest convolutional module here is the convolutional module farthest from the input terminal in the convolutional neural network. For example, if the convolutional neural network contains 5 convolutional modules, then each convolutional module corresponds to a layer (that is, corresponds to a branch). The 5 convolutional modules are arranged according to their distance from the input terminal of the convolutional neural network and named C1, C2, C3, C4, and C5 respectively. C1 is the convolutional module closest to the input terminal in the convolutional neural network, and C5 is the convolutional module farthest from the input terminal in the convolutional neural network.

[0115] The upper branch of a branch corresponds to a convolutional module closer to the input of the convolutional neural network than the branch itself, and the lower branch of a branch corresponds to a convolutional module farther from the input of the convolutional neural network than the branch itself. For example, the upper branch of the branch corresponding to C3 is the branch corresponding to C2, and the lower branch of the branch corresponding to C3 is the branch corresponding to C4. The upper branch of a branch is adjacent to the branch itself. The lower branch of a branch is adjacent to the branch itself.

[0116] The deep features of the branch corresponding to C5 are the features obtained by pooling the output of C5 (i.e., the shallow features of the branch corresponding to C5) (pooling can reduce the height and width by 2 times respectively). The deep features of the next branch corresponding to C4 are the deep features of the branch corresponding to C5.

[0117] The purpose of pooling in the neural convolution module is to reduce the size of the feature map, compress feature information, and improve the computational efficiency of the model.

[0118] Common pooling methods include Max Pooling and Average Pooling, which can sample the feature map obtained by convolution, extract the maximum value or average value in the local area, and output it as the result.

[0119] The pooling layer can be sampled by sliding the window, so during the pooling process, the size of the feature map can be changed by adjusting the window size and step size, thereby controlling the complexity and number of parameters of the model.

[0120] Because pooling layers reduce network parameters and improve computational efficiency, they are widely used in convolutional neural networks. While pooling does reduce the size of feature maps by half, this is not true for all pooling layers. For example, you can control the size of the output feature map by setting the stride to 1 or using strided convolution instead of pooling.

[0121] In an exemplary embodiment, the first fusion process includes:

[0122] The deep features of all branches are concatenated to form the fused deep features.

[0123] Concatenation can be done using the Concat operation, which directly concatenates two features. For example, if the dimensions of the two input features X and Y are p and q, then the dimension of the output feature Z after concatenating X and Y is p+q.

[0124] In an exemplary embodiment, the joint attention processing includes: determining the weight features of the first fused features based on the weight vector of each channel of the first fused features and the spatial mask; and performing residual connection based on the first fused features and the weight features of the first fused features.

[0125] In an exemplary embodiment, the weight vector of each channel in the features after the first fusion processing is extracted by the preset CAM module;

[0126] The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

[0127] In an exemplary embodiment, performing residual connection based on the first fused feature and the weight feature of the first fused feature includes:

[0128] Multiplying the first fused feature and the weighted feature of the first fused feature by position;

[0129] The multiplication result is added to the first fusion-processed feature.

[0130] In an exemplary embodiment, the feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch based on the shallow features of the branch and the screening features of the next branch of the branch;

[0131] Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the void space convolution pooling pyramid module.

[0132] In an exemplary embodiment, determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes:

[0133] Upsampling the screening features of the next branch of the branch by a preset multiple; adding the upsampled features to the shallow features of the branch to obtain the first feature;

[0134] The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature;

[0135] The first feature is integrated with the multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature;

[0136] The screening feature of the branch is determined according to the second feature and the third feature.

[0137] In an exemplary embodiment, determining the screening feature of the branch according to the second feature and the third feature includes:

[0138] The second feature and the third feature are added together to obtain the screening feature of the branch.

[0139] In an exemplary embodiment, the second fusion process includes: concatenating the screening features of all branches.

[0140] Concatenation can be done using the Concat operation, which directly concatenates two features. For example, if the dimensions of the two input features X and Y are p and q, then the dimension of the output feature Z after concatenating X and Y is p+q.

[0141] In an exemplary embodiment, the segmentation process includes: extracting a first semantic feature, performing a first upsampling process, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

[0142] For example, two first preset convolution modules are used to extract the first semantic feature. After the first upsampling, features having a resolution of the original input image can be obtained. The second preset convolution module is used to integrate information between multiple channels.

[0143] For example, the first preset convolution module may be a 3×3 convolution module, and the second preset convolution module may be a 1×1 convolution module. Normalization processing is performed using a Sigmoid function.

[0144] In an exemplary embodiment, the edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling, and normalizing processes on the features obtained by the second fusion process in sequence.

[0145] Exemplarily, the second semantic feature is extracted by a first preset convolution module, the second upsampling and the third upsampling are performed respectively by a first deconvolution module, and normalization is performed by a Sigmoid function.

[0146] Exemplarily, the first preset convolution module may be a 3×3 convolution module, and the first deconvolution module may be a T-2×2 convolution module.

[0147] In an exemplary embodiment, the third fusion processing includes: splicing the features obtained by the segmentation processing and the features obtained by the edge enhancement processing; extracting the fourth semantic feature, integrating information between multiple channels, and normalizing the spliced ​​features in turn.

[0148] Exemplarily, the fourth semantic feature is extracted by a first preset convolution module, the information between multiple channels is integrated by a second preset convolution module, and normalization is performed by a Sigmoid function.

[0149] Concatenation can be done using the Concat operation, which directly concatenates two features. For example, if the dimensions of the two input features X and Y are p and q, then the dimension of the output feature Z after concatenating X and Y is p+q.

[0150] FIG4 is a flow chart of a scene text processing method according to at least one embodiment of the present disclosure. As can be seen from FIG4 , the scene text processing method includes the following steps S21 - S23 .

[0151] S21, obtaining an image to be detected and performing feature extraction on the image to be detected;

[0152] S22, performing attention feature fusion, first fusion processing, and joint attention processing on the features obtained by feature extraction in sequence;

[0153] S23 , performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0154] The network architecture corresponding to the scene text processing method is shown in FIG5 . In FIG5 , as shown in FIG5 , the image to be detected 100 is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 200, an attention feature fusion module 300, a first fusion processing module 400, a joint attention module 500, and a post-processing module 600. The scene text network model in FIG5 can be established as follows:

[0155] Constructing a training data set and an initial network model, wherein the training data set includes a plurality of preprocessed images including scene text and corresponding labels;

[0156] The initial network model is trained according to the training data set, and the trained network model is used as a preset scene text network model.

[0157] In an exemplary embodiment, when constructing the training dataset, it may include pre-processing the images used to construct the training dataset.

[0158] Among them, preprocessing can include image decoding, label encoding, basic data augmentation, channel transformation and normalization.

[0159] The image decoding may be to convert the image to be decoded into Numpy format.

[0160] Label encoding can be done by parsing label information from a txt file and saving it in a unified format. Label information includes the location of manually annotated boxes and text information. The txt file is the training data corresponding to the image, including label information. The format can be Baidu PPOCR data, including coordinate information and text information.

[0161] Basic data augmentation can include random horizontal flipping, random rotation, random scaling, random cropping, etc.

[0162] Channel transformation can be to transform the format of the image after the above processing from the image data format [H, W, C] (i.e., height, width and number of channels) to the format of training data used by the neural network [C, H, W] (i.e., number of channels, height and width).

[0163] Set the optimization parameters, number of iterations, loss function, and evaluation metrics, and input the preprocessed images into the initial network model training.

[0164] The Amad method can be used to optimize parameters. The number of iterations can be set to 1000. Evaluation is performed after each iteration. Training is terminated when the maximum number of iterations is reached.

[0165] The loss function L is composed of L s , L b and L t It consists of three parts, as follows:

[0166] L=L s +α×L b +β×L t ;

[0167] Among them, L s , L b , L t Represents the training loss of probability map P, approximate binary map B, and threshold map T respectively. s and L b The binary cross entropy loss function is used, and the formula is as follows:

[0168] Among them, S l Represents the core area of ​​the text predicted during training, y i Indicates the label value of the i-th pixel position, x i Represents the predicted probability value of the position. The loss L of the threshold map T t The L1 loss is used, and the formula is as follows:

[0169] R d is the text bounding box area, represents the label of the threshold map T at position j, is the predicted value of the threshold at this location.

[0170] There are three evaluation indicators for text detection: precision, recall, and the harmonic mean (Hmean) of the first two. The calculation logic of the three indicators is as follows:

[0171] 1. Create a matrix called iouMat of size [n,m], where n is the number of ground truth (GT) boxes and m is the number of detected boxes. n and m are the number of boxes excluding those marked with "###" text; "###" indicates unclear text. This is how text is marked in the data.

[0172] 2. In iouMat, count the number of IOUs greater than the threshold 0.5 and divide this value by n to get Recall;

[0173] 3. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by m to get Precision;

[0174] 4. The calculation formula of Hmean indicator is as follows:

[0175] Hmean=2.0×(Precision×Recall / Precision+Recall).

[0176] Save the weights of the trained model, use the trained network model as the preset scene text network model, and use the network model for text detection.

[0177] The feature extraction module 200 can use a convolutional neural network (as shown in FIG6 ) to extract features of the image to be detected. Each of the four convolution modules conv2-conv5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolution module, and the channels are uniformly adjusted to 256 dimensions. The features after the channel unification are recorded as in k (k=1,2,3,4).

[0178] The attention feature fusion module 300 includes an AFFM submodule and a pooling submodule. The AFFM (Attention Feature Fusion Module) submodule contains two inputs and one output. k (k=1, 2, 3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1(k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoding fusion feature after AFFM feature fusion is out k (k=1,2,3).

[0179] The pooling submodule is configured to reduce the output of the AFFM submodule, for example, reducing the height and width of the output of the AFFM submodule by 2 times respectively.

[0180] The first fusion processing module 400 concatenates the deep features of all branches to form a fused deep feature.

[0181] Global information interaction is embodied through the joint attention module 500, namely the JAM module (i.e., Joint Attention Module).

[0182] The post-processing module 600 is divided into two branches, upper and lower, for respectively performing text core area probability prediction and boundary threshold prediction.

[0183] In step S21, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolutional modules conv2-conv5 in the convolutional neural network (as shown in FIG6 ) is used as a branch. The output features of each branch are respectively passed through a 1×1 convolutional module, and the channels are uniformly adjusted to 256 dimensions. The features after channel unification are recorded as in k (k=1,2,3,4).

[0184] In step S22, the AFFM submodule (ie, Attention Feature Fusion Module) includes two inputs and one output. k (k=1, 2, 3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoding fusion feature after AFFM feature fusion is out k (k=1,2,3). The AFFM submodule is shown in Figure 7.

[0185] The operating mechanism of the AFFM submodule is as follows:

[0186] Out the deep features of the lower branch k+1 The C×1×1 channel weight vector is obtained through the CAM module and combined with the feature vector from the encoding end k Multiply channel by channel to obtain the channel-weighted encoding features;

[0187] The shallow characteristics of the upper branch are in kThe 1×H×W spatial information mask is obtained through the SAM module and combined with the out k+1 The features are multiplied position by position to obtain the decoding features weighted by spatial information;

[0188] The weighted encoding features and decoding features are added by position to obtain the fused decoding features out k .

[0189] The internal operation formula of the AFFM sub-model is expressed as:

[0190] out4=in4.

[0191] When calculating out3, the deep features of the lower branch use out4. Similarly, when calculating out2, the deep features of the lower branch use out3. When calculating out1, the deep features of the lower branch use out2.

[0192] The core of the AFFM sub-module is to use the attention mechanism to extract different focuses of information from deep and shallow layers to calibrate and strengthen the input feature vector. That is, the spatial information of shallow features extracted by SAM is used to weight the upsampled deep features, and the global information of deep features is extracted by CAM module to weight the shallow information from the encoding end. The calibrated features of both sides are then added and fused.

[0193] In step S22, in order to improve the information representation ability of the cascaded features, convolution is often used to fuse the cascaded features. However, conventional convolution operations only fuse information in the local receptive field and ignore global information interaction. The present disclosure embodies global information interaction through the JAM module (i.e., Joint Attention Module) (as shown in Figure 8).

[0194] The processing flow of the JAM module is as follows:

[0195] Use CAM and SAM to model the dependency between the channel and spatial position of the input feature F, use the global information of the feature to obtain the weight of each channel, and use the relationship between the spatial context to highlight the features related to the target and suppress irrelevant background noise;

[0196] The obtained C×1×1 weight vector and 1×H×W spatial mask are expanded to the dimension of the input feature F and then multiplied by position to obtain the weight feature F' of the same dimension as F;

[0197] As the number of network layers increases, the objective function is prone to falling into local optimal solutions. Furthermore, the vanishing gradient problem becomes more serious as the number of layers increases. Residual networks are very effective in alleviating the vanishing gradient problem and can increase the depth of the network for effective training. Therefore, residual connections are added to the module to ensure the effectiveness of model training. The implementation details of residual connections are shown in the following formula:

[0198] F″=JA(F)=F+F×F′=F+F×σ{EX[CA(F)]×EX[SA(F)]}.

[0199] In step S23, after the enhanced fusion feature F" is obtained through the above steps, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0200] 1) F' first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu. The resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0201] 2) Deconvolution is performed on the obtained feature map with a convolution kernel of (2,2). The resulting feature map shape is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0202] 3) Perform deconvolution again, which is the same as the second step, except that the output feature map channel is 1, and the resulting feature map shape is (batch, W, H), which is the original image size.

[0203] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0204] Finally, the probability map P and the threshold map T are binarized using differentiable binarization to obtain the final binary map B (as shown in Figure 9).

[0205] Among them, the formula for differentiable binarization is:

[0206] Standard binarization methods are non-differentiable, making it impossible to train the network end-to-end. To address this issue, the algorithm employs a differentiable binarization approach. As can be seen from the above formula, differentiable binarization approximates the step function used in standard binarization, making the function globally differentiable and differentiable. The flow of the differentiable binarization algorithm is shown in red arrows in the figure. The key difference lies in the inclusion of a threshold map T. The network predicts the threshold at each position in the image, rather than using a fixed value. This allows for better separation of text background and foreground.

[0207] The disclosed embodiment uses attention feature fusion processing to improve the accuracy of text detection in scenarios with variable shapes, drastic scale changes, and complex and diverse backgrounds, and pays attention to global information interaction, thereby improving information representation capabilities.

[0208] FIG10 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure. As can be seen from FIG10 , the scene text processing method includes the following steps S101 - S103 .

[0209] S101, obtaining an image to be detected and performing feature extraction on the image to be detected;

[0210] S102, performing attention feature fusion and first fusion processing on the features obtained by feature extraction in sequence;

[0211] S103 , performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0212] The network architecture corresponding to the scene text processing method is shown in Figure 11. In Figure 11, the image to be detected 100 is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 200, an attention feature fusion module 300, a first fusion processing module 400, and a post-processing module 600. The scene text network model in Figure 11 can be established as follows:

[0213] Constructing a training data set and an initial network model, wherein the training data set includes a plurality of preprocessed images including scene text and corresponding labels;

[0214] The initial network model is trained according to the training data set, and the trained network model is used as a preset scene text network model.

[0215] In an exemplary embodiment, when constructing the training dataset, it may include pre-processing the images used to construct the training dataset.

[0216] Among them, preprocessing can include image decoding, label encoding, basic data augmentation, channel transformation and normalization.

[0217] The image decoding may be to convert the image to be decoded into Numpy format.

[0218] Label encoding can be done by parsing label information from a txt file and saving it in a unified format. Label information includes the location of manually annotated boxes and text information. The txt file is the training data corresponding to the image, including label information. The format can be Baidu PPOCR data, including coordinate information and text information.

[0219] Basic data augmentation can include random horizontal flipping, random rotation, random scaling, random cropping, etc.

[0220] Channel transformation can be to transform the format of the image after the above processing from the image data format [H, W, C] (i.e., height, width and number of channels) to the format of training data used by the neural network [C, H, W] (i.e., number of channels, height and width).

[0221] Set the optimization parameters, number of iterations, loss function, and evaluation metrics, and input the preprocessed images into the initial network model training.

[0222] The Amad method can be used to optimize parameters. The number of iterations can be set to 1000. Evaluation is performed after each iteration. Training is terminated when the maximum number of iterations is reached.

[0223] The loss function L is composed of L s , L b and L t It consists of three parts, as follows:

[0224] L=L s +α×L b +β×L t ;

[0225] Among them, L s , L b , L t Represents the training loss of probability map P, approximate binary map B, and threshold map T respectively. s and L b The binary cross entropy loss function is used, and the formula is as follows:

[0226] Among them, S l Represents the core area of ​​the text predicted during training, y i Indicates the label value of the i-th pixel position, x i Represents the predicted probability value of the position. The loss L of the threshold map T t The L1 loss is used, and the formula is as follows:

[0227] R d is the text bounding box area, represents the label of the threshold map T at position j, is the predicted value of the threshold at this location.

[0228] There are three evaluation indicators for text detection: precision, recall, and the harmonic mean (Hmean) of the first two. The calculation logic of the three indicators is as follows:

[0229] 1. Create a matrix called iouMat of size [n,m], where n is the number of ground truth (GT) boxes and m is the number of detected boxes. n and m are the number of boxes excluding those marked with "###" text; "###" indicates unclear text. This is how text is marked in the data.

[0230] 2. In iouMat, count the number of IOUs greater than the threshold 0.5 and divide this value by n to get Recall;

[0231] 3. In iouMat, count the number of IOUs greater than the threshold of 0.5, and divide this value by m to get Precision;

[0232] 4. The calculation formula of Hmean indicator is as follows:

[0233] Hmean=2.0×(Precision×Recall / Precision+Recall).

[0234] Save the weights of the trained model, use the trained network model as the preset scene text network model, and use the network model for text detection.

[0235] The feature extraction module 200 can use a convolutional neural network (as shown in FIG6 ) to extract features of the image to be detected. Each of the four convolution modules conv2-conv5 in the convolutional neural network is used as a branch. The output features of each branch are respectively passed through a 1×1 convolution module, and the channels are uniformly adjusted to 256 dimensions. The features after the channel unification are recorded as in k (k=1,2,3,4).

[0236] The attention feature fusion module 300 includes an AFFM submodule and a pooling submodule. The AFFM (Attention Feture Fusion Module) submodule contains two inputs and one output. k (k=1, 2, 3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoding fusion feature after AFFM feature fusion is out k (k=1,2,3).

[0237] The pooling submodule is configured to reduce the output of the AFFM submodule, for example, reducing the height and width of the output of the AFFM submodule by 2 times respectively.

[0238] The first fusion processing module 400 concatenates the deep features of all branches to form a fused deep feature.

[0239] The post-processing module 600 is divided into two branches, upper and lower, for respectively performing text core area probability prediction and boundary threshold prediction.

[0240] In step S101, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolutional modules conv2-conv5 in the convolutional neural network (as shown in FIG6 ) is used as a branch. The output features of each branch are respectively passed through a 1×1 convolutional module, and the channels are uniformly adjusted to 256 dimensions. The features after the channel unification are recorded as in k (k=1,2,3,4).

[0241] In step S102, the AFFM submodule (ie, Attention Feature Fusion Module) includes two inputs and one output. k (k=1, 2, 3) is the feature of AFFM from the shallow encoding end (feature extraction stage); out k+1 (k=1,2,3) is the feature of AFFM from the deep decoding end (attention feature fusion). Output: The output decoding fusion feature after AFFM feature fusion is out k (k=1,2,3). The AFFM submodule is shown in Figure 7.

[0242] The operating mechanism of the AFFM submodule is as follows:

[0243] Out the deep features of the lower branch k+1 The C×1×1 channel weight vector is obtained through the CAM module and combined with the feature vector from the encoding end k Multiply channel by channel to obtain the channel-weighted encoding features;

[0244] The shallow characteristics of the upper branch are in k The 1×H×W spatial information mask is obtained through the SAM module and combined with the out k+1 The features are multiplied position by position to obtain the decoding features weighted by spatial information;

[0245] The weighted encoding features and decoding features are added by position to obtain the fused decoding features out k .

[0246] The internal operation formula of the AFFM sub-model is expressed as:

[0247] out4=in4.

[0248] When calculating out3, the deep features of the lower branch use out4. Similarly, when calculating out2, the deep features of the lower branch use out3. When calculating out1, the deep features of the lower branch use out2.

[0249] The core of the AFFM sub-module is to use the attention mechanism to extract different focuses of information from deep and shallow layers to calibrate and strengthen the input feature vector. That is, the spatial information of shallow features extracted by SAM is used to weight the upsampled deep features, and the global information of deep features is extracted by CAM module to weight the shallow information from the encoding end. The calibrated features of both sides are then added and fused.

[0250] In step S93, after the enhanced fusion feature F is obtained through the above steps, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0251] 1) F first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu. The resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0252] 2) Deconvolution is performed on the obtained feature map with a convolution kernel of (2,2). The resulting feature map shape is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0253] 3) Perform deconvolution again, which is the same as the second step, except that the output feature map channel is 1, and the resulting feature map shape is (batch, W, H), which is the original image size.

[0254] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0255] Finally, the probability map P and the threshold map T are binarized using differentiable binarization to obtain the final binary map B (as shown in Figure 9).

[0256] Among them, the formula for differentiable binarization is:

[0257] The standard binarization method is non-differentiable, which makes it impossible to train the network end-to-end. In order to solve this problem, a differentiable binarization processing method is adopted in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold map T. The threshold at each position of the image is predicted through the network, rather than using a fixed value, which can better separate the text background and foreground. The embodiment of the present disclosure can improve the accuracy of text detection in scenes with variable shapes, drastic scale changes, and complex and diverse backgrounds through attention feature fusion processing.

[0258] FIG12 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure. As shown in FIG12 , the scene text processing method includes the following steps S121 - S123 .

[0259] S121, obtaining an image to be detected and performing preprocessing and feature extraction on the image to be detected;

[0260] S122, performing feature screening, second fusion processing, segmentation processing, edge enhancement and third fusion processing on the features obtained by feature extraction in sequence;

[0261] S123 , performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0262] Figure 13 shows the network architecture corresponding to the scene text processing method. The process of establishing the scene text network model shown in Figure 13 can be similar to the process of establishing the scene text network model in Figure 5. As shown in Figure 13, the image 10 to be detected is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 20, a feature screening module 30, a second fusion processing module 40, a segmentation module 50, an edge enhancement module 60, a third fusion processing module 70 and a post-processing module 80. The convolutional neural network is configured to extract features. The feature screening module 30 includes an ASPP (Atrous Spatial Pyramid Pooling) submodule and an FF (Feature Filter) submodule. The ASPP submodule (Atrous Spatial Convolution Pooling Pyramid module) is configured to extract the deepest layer information from the highest dimensional features (i.e., the features output by the deepest convolution module) to use the deepest layer information as the initial internal features of the feature screening submodule (FF). The FF submodule is configured to perform feature screening. The second fusion processing module 40 is configured to splice the features after feature screening. The segmentation module 50 is configured to perform segmentation processing on the features spliced ​​by the second fusion processing module. The edge enhancement module 60 is configured to perform edge enhancement processing on the features spliced ​​by the second fusion processing module. The third fusion processing module 70 is configured to splice the features after the segmentation processing and the features after the edge enhancement processing. The post-processing module 80 is configured to perform the scene text core area probability prediction and boundary threshold prediction based on the output of the third fusion processing module to obtain a binary image of the image to be detected.

[0263] In step S121, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolution modules c2-c5 in the convolutional neural network is used as a branch. The output features of each branch are passed through a 1×1 convolution module respectively, and the channels are uniformly adjusted to 256 dimensions. The features after channel unification are recorded as in k (k=1,2,3,4). The convolutional neural network can be ResNet-18.

[0264] In step S122, the structure of the feature screening submodule corresponding to feature screening is shown in Figure 14. UP indicates that the input features are upsampled using bilinear interpolation, and the resolution after upsampling is twice the original. The 1×1 convolution module is set to change the number of channels of the input features. σ represents the sigmoid operation, which is used to normalize the input features. The input features in step S72 refer to the features input to the corresponding module. k+1 Indicates the features after filtering of the lower branch features, k = 1, 2, 3. S k Indicates the characteristics of this branch after feature filtering (i.e., filtered characteristics).

[0265] Combined with Figure 13, it can be seen that in4 will obtain the initial internal features through ASPP, and the initial internal features will be input into the FF module together with in3 as S4 to obtain S3, in2 and S3 will be input into the FF module to obtain S2, and in1 and S2 will be input into the FF module to obtain S1.

[0266] Taking the process of obtaining S3 as an example, the feature screening process is as follows:

[0267] Upsample S4;

[0268] Add the up-sampled result of S4 to in3, and change the number of channels of the added result; after changing the number of channels, perform a sigmoid operation to obtain the first value;

[0269] The first value is multiplied by the result of upsampling by S4 to obtain the second value;

[0270] Change the number of channels of in3 and perform sigmoid operation to obtain the third value;

[0271] The third value is multiplied by in3 to get the fourth value;

[0272] The second value and the fourth value are added to obtain the feature S3.

[0273] In step S122 , the second fusion processing module 40 is configured to concatenate S4 , S3 , S2 , and S1 to obtain f.

[0274] In step S122, the segmentation process is performed by the segmentation module shown in Figure 15. The role of the segmentation module is to perform binary classification of text areas and background. In this disclosure, all text areas are predicted as foreground and are not distinguished at the neural network level.

[0275] In the segmentation module, UP stands for 4x upsampling using bilinear interpolation. The input feature f is a fused multi-scale feature. It first undergoes two 3×3 convolutions to extract binary semantic information from the fused features. The resulting features are then upsampled to the original input resolution. A 1×1 convolution is then performed to integrate multi-channel information. The output features are then converted to a single channel. Finally, the integrated features are normalized using the sigmoid function. The final output h is a probability heat map with pixel values ​​ranging from 0 to 1, with dimensions H×W×1. In the output prediction map, the pixel values ​​of the background are close to 0, while the pixel values ​​of all text areas are close to 1.

[0276] In step S122, edge enhancement processing is performed by the edge enhancement module 60 shown in Figure 16. The purpose of edge enhancement is to detect the boundaries of the text area, so that on the one hand, the text and background can be accurately distinguished, and on the other hand, the text belonging to different text areas can be effectively separated to avoid adhesion.

[0277] The T-2×2 in edge enhancement module 60 represents a deconvolution with a 2×2 kernel, which upsamples the image to a factor of 2. The width and height of the input feature f are W / 4 and H / 4, respectively, and the output g is a feature map with dimensions H×W×1. Therefore, the resolution of the output feature is four times that of the input, and the predicted features are edge features of the text region.

[0278] After the processing of the edge enhancement module 60, the accuracy of the text area boundary can be improved, and the problem of fuzzy edge detection of the text and reduced accuracy of the boundary area caused by the use of a segmentation algorithm is improved.

[0279] The original text boundary is a manually annotated box; it is directly mapped to the image without requiring recognition. During the manual framing process, the boundary can be reduced. This reduction is done to address situations where the detection boxes of the previous and next lines of text overlap excessively, causing bridging. The value is adjustable. In existing solutions, the value is generally adjusted to between 0.7-0.8 and 0.3-0.4. The following methods can be used to reduce the boundary.

[0280] The boundaries of the text area are represented as Where V m Represents the coordinate points on the boundary, and n represents the number of coordinate points. The boundary line of the original text area is used as the outer boundary L1. The boundary line of the text area is reduced inward by 0.7 and 0.3, respectively, to form the center L2 and inner boundary L3 of the boundary area. The pixel value at L2 is set to 1, and the pixel values ​​outside L1 and inside L3 are set to 0. Then, linear interpolation is used to fill the text edge area between L1 and L3. The boundary reduction formula used in the above process is:

[0281] Where A and D represent the area and perimeter of the text region, respectively; r represents the ratio of inward reduction, which is 0.7 and 0.3 for L2 and L3, respectively; and d is the distance that the boundary point coordinates are reduced inward.

[0282] In step S122, the third fusion process is performed using the third fusion module shown in Figure 17. In the fusion module structure, g and h represent the output of the edge enhancement module and the output of the segmentation module, respectively. They are first concatenated in the channel dimension, and then further semantic features are extracted through a 3×3 convolution operation. The initially fused features are then further integrated using a 1×1 convolution. During the fusion stage, features from different levels are not fused using the attention mechanism. Instead, features of the same dimensions from the edge enhancement module and the segmentation module are fused after the aforementioned operation. The integrated semantic information is then normalized using a sigmoid function, and finally, an appropriate binarization threshold is selected for post-processing.

[0283] In step S123, after the enhanced fusion feature F" is obtained through the third fusion process, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0284] 1) F' first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu. The resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0285] 2) Deconvolution is performed on the obtained feature map with a convolution kernel of (2,2). The resulting feature map shape is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0286] 3) Perform deconvolution again, which is the same as the second step, except that the output feature map channel is 1, and the resulting feature map shape is (batch, W, H), which is the original image size.

[0287] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0288] Finally, the probability map P and the threshold map T are binarized using differentiable binarization to obtain the final binary map B (as shown in Figure 9).

[0289] Among them, the formula for differentiable binarization is:

[0290] Standard binarization methods are non-differentiable, making it impossible to train the network end-to-end. To address this issue, the algorithm employs a differentiable binarization approach. As can be seen from the above formula, differentiable binarization approximates the step function used in standard binarization, making the function globally differentiable and differentiable. The flow of the differentiable binarization algorithm is shown in red arrows in the figure. The key difference lies in the inclusion of a threshold map T. The network predicts the threshold at each position in the image, rather than using a fixed value. This allows for better separation of text background and foreground.

[0291] The disclosed embodiment can extract effective information in the process of reverse restoration of features from low resolution to high resolution through feature screening, segmentation processing and edge enhancement processing, thereby improving the accuracy of text area boundaries.

[0292] FIG18 is a flowchart of a scene text processing method according to at least one embodiment of the present disclosure. As shown in FIG18 , the scene text processing method includes the following steps S181 - S183 .

[0293] S181, obtaining an image to be detected and performing preprocessing and feature extraction on the image to be detected;

[0294] S182, performing feature screening and second fusion processing on the features obtained by feature extraction in sequence;

[0295] S183 , performing the scene text core area probability prediction and the boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected.

[0296] Figure 19 shows the network architecture corresponding to the scene text processing method. The process of establishing the scene text network model shown in Figure 19 can be similar to the process of establishing the scene text network model in Figure 5. As shown in Figure 19, the image 10 to be detected is input into the scene text network model for text detection. The scene text network model includes a feature extraction module 20, a feature screening module 30, a second fusion processing module 40 and a post-processing module 80. The convolutional neural network is configured to extract features. The feature screening module 30 includes an ASPP (Atrous Spatial Pyramid Pooling) submodule and an FF (Feature Filter) submodule. The ASPP submodule (Atrous Spatial Convolution Pooling Pyramid module) is configured to extract the deepest layer information from the highest dimensional features (i.e., the features output by the deepest convolution module) to use the deepest layer information as the initial internal features of the feature screening submodule (FF). The FF submodule is configured to perform feature screening. The second fusion processing module 40 is configured to splice the features after feature screening. The post-processing module 80 is configured to perform the scene text core area probability prediction and boundary threshold prediction according to the output of the third fusion processing module to obtain a binary image of the image to be detected.

[0297] In step S181, a convolutional neural network can be used to extract features of the image to be detected. Each of the four convolutional modules c2-c5 in the convolutional neural network is used as a branch. The output features of each branch are passed through a 1×1 convolutional module respectively, and the channels are uniformly adjusted to 256 dimensions. The features after channel unification are recorded as in k(k=1,2,3,4). The convolutional neural network can be ResNet-18.

[0298] In step S182, the structure of the feature screening submodule corresponding to feature screening is shown in Figure 14. UP indicates that the input features are upsampled using bilinear interpolation, and the resolution after upsampling is twice the original. The 1×1 convolution module is set to change the number of channels of the input features. σ represents the sigmoid operation, which is used to normalize the input features. The input features in step S72 refer to the features input to the corresponding module. k+1 Indicates the features after filtering of the lower branch features, k = 1, 2, 3. S k Indicates the characteristics of this branch after feature filtering (i.e., filtered characteristics).

[0299] Combined with Figure 13, it can be seen that in4 will obtain the initial internal features through ASPP, and the initial internal features will be input into the FF module together with in3 as S4 to obtain S3, in2 and S3 will be input into the FF module to obtain S2, and in1 and S2 will be input into the FF module to obtain S1.

[0300] Taking the process of obtaining S3 as an example, the feature screening process is as follows:

[0301] Upsample S4;

[0302] Add the up-sampled result of S4 to in3, and change the number of channels of the added result; after changing the number of channels, perform a sigmoid operation to obtain the first value;

[0303] The first value is multiplied by the result of upsampling by S4 to obtain the second value;

[0304] Change the number of channels of in3 and perform sigmoid operation to obtain the third value;

[0305] The third value is multiplied by in3 to get the fourth value;

[0306] The second value and the fourth value are added to obtain the feature S3.

[0307] In step S182 , the second fusion processing module 40 is configured to concatenate S4 , S3 , S2 , and S1 to obtain f.

[0308] In step S183, after the feature f is obtained through the second fusion process, it is divided into two branches, upper and lower, to perform text core area probability prediction and boundary threshold prediction respectively. The generation process of the probability map P and the threshold map T is as follows:

[0309] 1) f first passes through the convolution layer to compress the channel to 1 / 4 of the input, and then passes through BN and relu, and the resulting feature map shape is (batch, 64, 1 / 4W, 1 / 4H);

[0310] 2) Deconvolution is performed on the obtained feature map with a convolution kernel of (2,2). The resulting feature map shape is (batch,256,1 / 2W,1 / 2H), which is half the size of the original image.

[0311] 3) Perform deconvolution again, which is the same as the second step, except that the output feature map channel is 1, and the resulting feature map shape is (batch, W, H), which is the original image size.

[0312] 4) Finally, after the sigmoid function, the probability map P is output, and the adaptive threshold is set to output the threshold map T.

[0313] Finally, the probability map P and the threshold map T are binarized using differentiable binarization to obtain the final binary map B (as shown in Figure 9).

[0314] Among them, the formula for differentiable binarization is:

[0315] The standard binarization method is non-differentiable, which makes it impossible to train the network end-to-end. In order to solve this problem, a differentiable binarization processing method is adopted in the algorithm. As can be seen from the above formula, the differentiable binarization approximates the step function in the standard binarization, making the function globally differentiable and differentiable. The process of the differentiable binarization algorithm is shown by the red arrow in the figure. The biggest difference is that it has a threshold map T. The threshold at each position of the image is predicted through the network, rather than using a fixed value, which can better separate the text background and foreground. The embodiment of the present disclosure improves the detection capability of multi-scale text through feature screening.

[0316] The present disclosure provides a scene text detection device, comprising an acquisition module, a feature extraction module,

[0317] Processing module, post-processing module; wherein,

[0318] The acquisition module is configured to acquire an image to be detected;

[0319] The feature extraction module is configured to extract features from the image to be detected;

[0320] The processing module is configured to process the features obtained by feature extraction;

[0321] The post-processing module is configured to perform probability prediction of the scene text core area and boundary threshold prediction according to the processing results to obtain a binary image of the image to be detected;

[0322] The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module;

[0323] The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to preset dimensions to obtain shallow features of the multiple branches.

[0324] The present disclosure provides a scene text detection device, comprising a memory and a processor.

[0325] The memory is used to store a program for scene text detection;

[0326] The processor is configured to read and execute the program for scene text detection and perform the above-mentioned scene text detection method.

[0327] The present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable the computer to execute the above-mentioned scene text detection method.

[0328] The drawings in this disclosure only relate to the structures involved in this disclosure, and other structures may refer to general designs. In the absence of conflict, the embodiments of this disclosure and the features in the embodiments may be combined with each other to obtain new embodiments.

[0329] The above embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to what is specifically shown and described herein. Various modifications, substitutions, or omissions may be made to the implementation forms and details without departing from the scope of the present disclosure.

Claims

1. A scene text detection method, comprising: Acquire an image to be detected and perform feature extraction on the image to be detected; wherein the image to be detected contains scene text; Processing the features obtained through feature extraction; According to the processing results, the scene text core area probability prediction and the boundary threshold prediction are performed to obtain a binary image of the image to be detected; The processing includes at least one of the following: Attentional feature fusion and first fusion processing; joint attention processing; Or the processing includes at least one of the following: Feature screening and second fusion processing; Segmentation, edge enhancement and third fusion processing; The feature extraction includes: performing feature extraction on the image to be detected through a convolutional neural network, and adjusting the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

2. The scene text detection method according to claim 1, wherein: The attention feature fusion processing includes: for each branch other than the branch corresponding to the deepest convolution module, the deep features of the branch are determined according to the channel-weighted encoding features of the branch and the spatial information-weighted decoding features of the branch.

3. The scene text detection method according to claim 2, wherein: Determine the channel-weighted coding features of the branch according to the deep features of the next branch of the branch, the shallow features of the branch, and a preset CAM module; Determine the decoding features of the branch after the spatial information weighting according to the shallow features of the branch, the deep features of the next branch of the branch after upsampling, and the preset SAM module; The deep features of the branch corresponding to the deepest convolution module are the features obtained by pooling the shallow features of the branch corresponding to the deepest convolution module.

4. The scene text detection method according to claim 3, wherein: Determining the channel-weighted coding features of the branch according to the deep features of the lower branch of the branch, the shallow features of the branch, and a preset CAM module, including: The features of the deep features of the lower branch of the branch are extracted through the preset CAM module, and the extracted features are multiplied channel by channel with the shallow features of the branch to obtain the channel-weighted coding features of the branch.

5. The scene text detection method according to claim 3, wherein: Determine the weighted decoding features of the spatial information of the branch according to the shallow features of the branch, the deep features of the lower branch of the branch after upsampling, and the preset SAM module, including: The shallow features of the branch are extracted by the preset SAM module, and the extracted features are multiplied position by position with the deep features of the lower branch of the branch after upsampling to obtain the decoding features of the branch after spatial information weighting.

6. The scene text detection method according to claim 3, wherein: The first fusion processing includes: splicing the deep features of all branches to form fused deep features.

7. The scene text detection method according to claim 3, wherein: The joint attention processing includes: determining the weight features of the features after the first fusion processing according to the weight vector of each channel of the features after the first fusion processing and the spatial mask; and performing residual connection according to the features after the first fusion processing and the weight features of the features after the first fusion processing.

8. The scene text detection method according to claim 7, wherein: Extracting a weight vector of each channel in the features after the first fusion processing through the preset CAM module; The spatial mask of the features after the first fusion processing is extracted through the preset SAM module.

9. The scene text detection method according to claim 7, wherein: Performing residual connection according to the features after the first fusion processing and the weight features of the features after the first fusion processing includes: Multiplying the first fused feature and the weight feature of the first fused feature by position; The multiplication result is added to the feature after the first fusion processing.

10. The scene text detection method according to claim 1, wherein: The feature screening includes: for each branch other than the branch corresponding to the deepest convolution module, determining the screening feature of the branch according to the shallow features of the branch and the screening features of the next branch of the branch; Among them, the screening features of the branch corresponding to the deepest convolution module are the initial internal features; the initial internal features are the features extracted from the shallow features of the branch corresponding to the deepest convolution module through the atrous spatial convolution pooling pyramid module.

11. The scene text detection method according to claim 10, wherein: Determining the screening feature of the branch according to the shallow feature of the branch and the screening feature of the next branch of the branch includes: Upsampling the screening feature of the next branch of the branch by a preset multiple; adding the upsampled feature to the shallow feature of the branch to obtain a first feature; The shallow feature of the branch is integrated with the information between multiple channels and normalized, and then multiplied with the shallow feature of the branch to obtain the second feature; The first feature is integrated with multi-channel information and normalized, and then multiplied by the upsampled feature to obtain the third feature; The screening feature of the branch is determined according to the second feature and the third feature.

12. The scene text detection method according to claim 10, wherein: The second fusion process includes: splicing the screening features of all branches.

13. The scene text detection method according to claim 1, wherein: The segmentation process includes: extracting a first semantic feature, performing a first upsampling, integrating information between multiple channels, and normalizing the features obtained by the second fusion process in sequence.

14. The scene text detection method according to claim 1, wherein: The edge enhancement process includes: extracting second semantic features, performing second upsampling, third upsampling and normalization processes on the features obtained by the second fusion process in sequence.

15. The scene text detection method according to claim 1, wherein: The third fusion process includes: the features obtained by the segmentation process and the features obtained by the edge enhancement process. The features are spliced; the fourth semantic features are extracted from the spliced ​​features, the information between multiple channels is integrated, and the normalization is performed.

16. A scene text detection device, comprising an acquisition module, a feature extraction module, Processing module, post-processing module; wherein, The acquisition module is configured to acquire an image to be detected; The feature extraction module is configured to extract features from the image to be detected; The processing module is configured to process the features obtained by feature extraction; The post-processing module is configured to perform a probability prediction of the scene text core area and a boundary threshold prediction according to the processing result to obtain a binary image of the image to be detected; The processing module includes at least one of the following modules: an attention feature fusion module and a first fusion processing module; a joint attention processing module; or the processing module includes at least one of the following modules: a feature screening module and a second fusion processing module; a segmentation processing module, an edge enhancement module and a third fusion processing module; The feature extraction module is specifically configured to: perform feature extraction on the image to be detected through a convolutional neural network, and adjust the feature dimensions of the extracted multiple branches to a preset dimension to obtain shallow features of the multiple branches.

17. A scene text detection device, comprising a memory and a processor, The memory is used to store a program for scene text detection; The processor is configured to read and execute the program for scene text detection, and to perform the method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions, wherein: The computer executable instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

Images