Text detection method, device, electronic device and storage medium

Through pre-trained feature extraction network and binarization processing, the detection difficulties caused by text distortion and irregular morphology in the image are solved, and the accurate detection of the image text area is achieved, which improves the efficiency and accuracy of text detection.

CN115063810BActive Publication Date: 2024-12-03LIANREN HEALTHCARE BIG DATA TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210729277.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-24
Publication Date
2024-12-03
Estimated Expiration
2042-06-24

AI Technical Summary

Technical Problem

In the prior art, when detecting text information in an image, the text distortion and irregular shapes lead to low detection accuracy and it is difficult to accurately identify text areas.

Method used

A pre-trained feature extraction network is used to extract the detected images, generate a target feature map, and distinguish the text area from the background area through binarization to determine the text detection results.

Benefits of technology

It improves the accuracy and efficiency of text detection, can accurately locate text areas in the image, and enhances the accuracy of text detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115063810B_ABST
    Figure CN115063810B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a text detection method, device, electronic device and storage medium. The method comprises: obtaining an image to be detected, performing feature extraction on the image to be detected based on a pre-trained feature extraction network, and obtaining a target feature map corresponding to the image to be detected; determining a binary image corresponding to the image to be detected according to the target feature map, wherein the binary image is used to distinguish a text area containing text information in the image to be detected from a background area other than the text area; determining a text detection result of the image to be detected according to the binary image, wherein the text detection result at least includes the location information of the text area. The technical solution of the embodiment of the present invention realizes efficient detection of text in an image and improves the accuracy of text detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of text detection technology, and in particular to a text detection method, device, electronic equipment and storage medium. Background Art

[0002] As one of the main ways of information transmission, images carry a lot of information. The text information in images often also carries important semantic information. Therefore, in many application scenarios, text information in images will be detected.

[0003] Usually, the text information in the image is often not regular text, and there are situations where the background of the area including the text in the image is complex, the text is distorted, and the text shape is irregular. Therefore, when detecting text using traditional computer text detection technology, the accuracy is low, the text area may be missed, and it is difficult to accurately and efficiently identify the text area in the image. Summary of the invention

[0004] The present invention provides a text detection method, device, electronic device and storage medium to achieve accurate text detection.

[0005] According to one aspect of the present invention, there is provided a text detection method, comprising:

[0006] Acquire an image to be detected, and extract features from the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected;

[0007] Determine a binary image corresponding to the image to be detected according to the target feature map, wherein the binary image is used to distinguish a text area containing text information in the image to be detected from a background area other than the text area;

[0008] A text detection result of the image to be detected is determined according to the binary image, wherein the text detection result at least includes position information of the text area.

[0009] According to another aspect of the present invention, there is provided a text detection device, comprising:

[0010] A feature extraction module is used to obtain an image to be detected, and perform feature extraction on the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected;

[0011] An image binarization module, used for determining a binarized image corresponding to the image to be detected according to the target feature map, wherein the binarized image is used to distinguish a text area containing text information from a background area other than the text area in the image to be detected;

[0012] The detection result determination module is used to determine the text detection result of the image to be detected according to the binary image, wherein the text detection result at least includes the position information of the text area.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text detection method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text detection method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to extract features from the image to be detected through a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected. The feature extraction network can efficiently extract features that may contain text in the image to be detected, thereby improving the efficiency of text detection. The binary image corresponding to the image to be detected is determined by the extracted target feature map. The text area containing text information and the background area not containing text area can be clearly determined by binarizing the target feature map, and the text area in the image to be detected can be accurately located. Then, the text detection result of the image to be detected is determined by the binary image, which solves the technical problem in the prior art that it is difficult to accurately and efficiently identify the text area in the image due to text distortion and irregular text morphology. Accurate detection of text areas in images is achieved, and the efficiency and accuracy of text detection are improved.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a text detection method provided by Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of another text detection method provided by Embodiment 2 of the present invention;

[0023] Figure 3 is a flowchart of another text detection method provided by an embodiment of the present invention;

[0024] Figure 4 is a training and detection flow chart of a text detection model provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of the structure of an improved DBNet network provided by an embodiment of the present invention;

[0026] Figure 6 is a structural schematic diagram of a text detection device provided by Embodiment 3 of the present invention;

[0027] Figure 7 It is a schematic diagram of the structure of an electronic device for implementing the text detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 A flowchart of a text detection method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of detecting text in an image and determining the position of the text in the image. The method can be executed by a text detection device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110, obtaining an image to be detected, and performing feature extraction on the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected.

[0033] The image to be detected may be an image that needs to be subjected to text recognition. It is understandable that the image to be detected may include a text area or may not include a text area.

[0034] The feature extraction network may be a network for extracting image features from an image to be detected. The feature extraction network is obtained by training a pre-established artificial intelligence network based on a sample detection image and an expected feature map corresponding to the sample detection image. The pre-established artificial intelligence network may be, for example, at least one of a CNN (Convolutional Neural Network), HRNet, and FPN (Feature Pyramid Networks).

[0035] Optionally, there may be multiple ways to obtain the image to be detected. Optionally, obtaining the image to be detected includes: receiving the image to be detected submitted based on an image submission operation, or receiving the image to be detected uploaded by an image upload operation, or receiving an image obtained by dragging the image to be detected into a preset image detection frame, or, by acting on a preset image capture control to enable an image captured by an image capture device, etc.

[0036] Specifically, after obtaining the input image to be detected, the image to be detected is input into a pre-trained feature extraction network, and features are extracted from the image to be detected through the feature extraction network. The feature extraction network outputs a feature map of the image to be detected after feature extraction is completed, and the output feature map is determined as the target feature map corresponding to the image to be detected.

[0037] S120. Determine a binary image corresponding to the image to be detected according to the target feature map.

[0038] The binary image is used to distinguish between a text area containing text information and a background area other than the text area in the image to be detected. Exemplarily, the binary image may be an image in which the pixel value of the text area containing text information is set to 1, and the pixel value of the background area other than the text area is set to 0.

[0039] Specifically, according to the acquired target feature map, each target feature in the target feature map is determined, and it is identified whether each target feature is a text area. If the target feature in the target feature map is a text area containing text information, the pixel value of the target feature is set to 1; if the target feature in the target feature map is a background area that does not contain text information, the pixel value of the target feature is set to 0, and then each target feature of the target feature map is binarized to generate a binary image corresponding to the image to be detected.

[0040] S130: Determine a text detection result of the image to be detected according to the binarized image.

[0041] Wherein, the text detection result at least includes the position information of the text area. It can be understood that the position information used to characterize the text area can be in various forms, for example. The text position information includes the text detection result or text coordinate data of the text area marked with the image to be detected. Optionally, the text detection result of the image to be detected based on the binarized image includes: determining the text position information corresponding to the text information in the image to be detected based on the binarized image. Specifically, it can be to mark the text position information of the text information in the image to be detected in the image to be detected based on the text area in the binarized image, or it can be to generate text coordinate data corresponding to the text area according to the position of the text area in the binarized image as the text detection result of the image to be detected.

[0042] Specifically, a connected area of ​​a text area including text information is determined through a binary image, and a text box including the text area is expanded according to the connected area. The position information of the text box including the text information in the image to be detected is determined according to the position information of the text box of the text area in the binary image. The text detection result of the image to be detected is determined according to the position information of the text box of the text area and the text box including the text information in the image to be detected.

[0043] The technical solution of the embodiment of the present invention is to extract features from the image to be detected through a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected. The feature extraction network can efficiently extract features that may contain text in the image to be detected, thereby improving the efficiency of text detection. The binary image corresponding to the image to be detected is determined by the extracted target feature map. The text area containing text information and the background area not containing text area can be clearly determined by binarizing the target feature map, and the text area in the image to be detected can be accurately located. Then, the text detection result of the image to be detected is determined by the binary image, which solves the technical problem in the prior art that it is difficult to accurately and efficiently identify the text area in the image due to text distortion and irregular text morphology. Accurate detection of text areas in images is achieved, and the efficiency and accuracy of text detection are improved.

[0044] Embodiment 2

[0045] Figure 2 It is a flowchart of another text detection method provided in Example 2 of the present invention. This embodiment further explains how to extract features of the image to be detected based on a pre-trained feature extraction network and how to determine a binary image corresponding to the image to be detected based on the probability map and the threshold map.

[0046] Optionally, the feature extraction network includes a high-resolution module and an attention mechanism module. Furthermore, the feature extraction network based on the pre-trained feature extraction performs feature extraction on the image to be detected to obtain a target feature map corresponding to the image to be detected, including: extracting features from the image to be detected based on the pre-trained high-resolution module to obtain sub-network feature maps of at least two resolutions; performing weighted processing on each sub-network feature map based on the attention mechanism module, and performing feature fusion on the weighted sub-network feature maps to obtain a target feature map corresponding to the image to be detected.

[0047] Optionally, determining the binary image corresponding to the image to be detected based on the target feature map includes: performing text segmentation on the target feature map based on a pre-trained text segmentation network to obtain a probability map and a threshold map corresponding to the target feature map, respectively; and determining the binary image corresponding to the image to be detected based on the probability map and the threshold map.

[0048] like Figure 2 As shown, the method of this embodiment may specifically include:

[0049] S210: Acquire an image to be detected.

[0050] S220, performing feature extraction on the image to be detected based on the pre-trained high-resolution module to obtain sub-network feature maps of at least two resolutions.

[0051] The high-resolution module may be a neural network module including sub-networks of different resolutions from high to low connected in parallel, for example, including a sub-network of original resolution, a sub-network of 1 / 2 resolution of the original resolution, a sub-network of 1 / 4 resolution of the original resolution, and a sub-network of 1 / 8 resolution of the original resolution.

[0052] Specifically, the acquired image to be detected is input into a pre-trained high-resolution module to extract features of the image to be detected, and the high-resolution module generates sub-network feature maps of at least two resolutions according to the input image to be detected.

[0053] Optionally, the high-resolution module includes at least two sub-extraction networks of different resolutions connected in parallel, and each sub-extraction network is respectively connected to at least one attention mechanism module.

[0054] Specifically, in the high-resolution module, the first stage can be a high-resolution backbone network, which reduces the high-resolution sub-network to a low-resolution sub-extraction network through a convolution layer, and generates multiple resolution sub-extraction networks, and then connects the high-resolution backbone network and the low-resolution sub-extraction network in parallel to generate the second stage; the third stage further reduces the resolution of the low-resolution sub-extraction network in the second stage through a convolution layer, further generates multiple low-resolution sub-extraction networks, and then connects the high-resolution backbone network of the first stage and the low-resolution sub-extraction networks of the second and third stages in parallel again, and then reduces the resolution of the sub-extraction network of each stage from high to low one by one to form more stages, connects the high-resolution backbone network and the low-resolution sub-extraction network in parallel, and repeatedly exchanges information in the parallel multi-resolution sub-extraction networks in the entire high-resolution module to perform repeated multi-feature fusion. In which, each sub-extraction network in the high-resolution module is connected to at least one attention mechanism network.

[0055] Optionally, the high-resolution module based on pre-training can be an HRNet network. Taking the HRNet network as an example, before training the high-resolution module, a large number of sample text images can be collected as sample images to be detected, and the sample images to be detected are subjected to resolution as sample low-resolution images. During the training process, the sample images to be detected are input into the high-resolution module to be trained to obtain the low-resolution image output by the high-resolution module to be trained, and the loss is calculated between the output low-resolution image and the sample low-resolution image through the loss function, and the parameters in the high-resolution module to be trained are adjusted according to the calculated loss. When it is detected that the loss function in the high-resolution module to be trained converges, the training of the high-resolution module to be trained is completed, and the trained high-resolution module is obtained. Among them, the loss function can be a mean square error function.

[0056] S230, based on the attention mechanism module, weighted processing is performed on each sub-network feature map respectively, and feature fusion is performed on the weighted sub-network feature maps to obtain a target feature map corresponding to the image to be detected.

[0057] The attention module may be a SE-Net (Squeeze-and-Excitation Networks) module. In an embodiment of the present invention, the attention module may be used to perform inter-channel feature correction on the sub-network feature maps of different resolutions obtained by the high-resolution module.

[0058] Specifically, the weight of each sub-network feature map is determined respectively through the attention mechanism module, and the sub-network feature maps corresponding to the weight of each sub-network feature map are weighted according to the weight of each sub-network feature map, and then the weighted sub-network feature maps are feature fused to obtain the target feature map corresponding to the image to be detected.

[0059] Optionally, in an optional technical solution of the present invention, the attention mechanism module performs weighted processing on each sub-network feature map respectively, including: for each sub-network feature map output by the sub-extraction network, the attention mechanism modules connected to the sub-extraction networks respectively perform weighted processing on the sub-network feature maps.

[0060] Specifically, in the high-resolution module, for each sub-network feature map output by the sub-extraction network, the weight matrix corresponding to the current sub-network feature map is determined through the attention mechanism module connected to each sub-extraction network, and the weight matrix of the current sub-network feature map is multiplied by the sub-network feature map.

[0061] Optionally, in another embodiment of the present invention, the attention mechanism includes a pooling layer, a first fully connected layer connected to the pooling layer, a first nonlinear activation function layer connected to the first fully connected layer, a second fully connected layer connected to the nonlinear activation function layer, and a second nonlinear activation function layer connected to the second fully connected layer. The pooling layer may be a global average pooling layer. The first nonlinear activation function layer may be a ReLu layer. The second nonlinear activation function layer may be a sigmoid activation function layer.

[0062] Optionally, the attention mechanism-based module performs weighted processing on the sub-network feature map of each resolution respectively, including: performing global average pooling processing on the sub-network feature map through the pooling layer of the attention mechanism to obtain a target tensor; processing the target tensor through a first fully connected layer, a first nonlinear activation function layer, a second fully connected layer and a second nonlinear activation function layer to obtain a weight coefficient matrix corresponding to the sub-network feature map; and performing weighted processing on the sub-network feature map of each resolution based on the weight coefficient matrix.

[0063] Specifically, the attention module first performs global average pooling on the sub-network feature map through the global average pooling layer to obtain the target tensor; the target tensor is processed by the first fully connected layer, the first nonlinear activation function layer, the second fully connected layer and the second nonlinear activation function layer to obtain the weight coefficient matrix corresponding to the sub-network feature map; based on the weight coefficient matrix, the sub-network feature map of each resolution is weighted to obtain each weighted sub-network feature map. Among them, the target tensor has all the features of the sub-network feature map.

[0064] Exemplarily, the attention module first processes each sub-network feature map through a global average pooling layer to generate a target tensor, and then forms a Bottleneck structure through two fully connected layers (Fully Connected) and a first nonlinear activation function layer (ReLu). The dimension of the target tensor is reduced to 1 / 16 of the input through the first fully connected layer (Fully Connected), and then the target tensor is increased back to its original dimension after being activated by the first nonlinear activation function layer (ReLu). Then, the importance of each sub-network feature map is learned according to the second nonlinear activation function layer (sigmoid activation function), and the weight coefficient matrix of each sub-network feature map is determined.

[0065] S240. Determine a binary image corresponding to the image to be detected according to the target feature map.

[0066] Optionally, text segmentation is performed on the target feature map based on a pre-trained text segmentation network to obtain a probability map and a threshold map corresponding to the target feature map, respectively; and a binary image corresponding to the image to be detected is determined based on the probability map and the threshold map.

[0067] Among them, the probability map can be an image generated according to a fixed segmentation threshold in the target feature map text segmentation network, and the probability map can be used to represent the probability area where text exists in the target feature map; the threshold map can be an image generated according to an adaptive segmentation threshold in the target feature map text segmentation network, and the threshold map can be used to represent the threshold range of the area where the text is located in the target feature map.

[0068] The text segmentation network can be obtained by training a pre-established convolutional neural network (CNN) based on a sample feature map and an expected probability map and an expected threshold map corresponding to the sample feature map, or by improving DBnet.

[0069] Optionally, before training the text segmentation network, a large number of sample feature images are first collected as training samples, and the expected threshold map and expected probability map corresponding to the sample feature image segmentation are used as labels for sample facial images. During the training process, the sample feature image is input into the text segmentation network to be trained, and the training threshold map and training probability map after training are obtained. The output training threshold map and training probability map are compared with the expected threshold map and expected probability map through a loss function to calculate the loss, and the parameters in the text segmentation network to be trained are adjusted according to the calculated loss. When it is detected that the loss function in the text segmentation network to be trained converges, the training of the text segmentation network to be trained is completed, and the text segmentation network is obtained. Among them, the loss function of the text segmentation network can be a cross entropy error function.

[0070] Specifically, the target feature map is input into the text segmentation network, and the probability map of the target feature map is output according to the fixed segmentation threshold in the text segmentation network. According to the adaptive segmentation threshold in the text segmentation network, the threshold map of the target feature map is output.

[0071] Optionally, determining the binarized image corresponding to the image to be detected based on the probability map and the threshold map includes: performing differentiable binarization processing on the probability map and the threshold map to obtain the binarized image corresponding to the image to be detected.

[0072] The differentiable binarization may be performed based on the probability map pixel points in the probability map and the threshold map pixel points in the threshold map to perform differentiable binarization calculation.

[0073] Specifically, the probability map pixels in the probability map and the threshold map pixels in the threshold map are determined according to the probability map and the threshold map, and differentiable binarization processing is performed on the probability map pixels in the probability map and the threshold map pixels in the threshold map to generate a binary image corresponding to the image to be detected.

[0074] S250: Determine a text detection result of the image to be detected according to the binarized image.

[0075] The technical solution of the embodiment of the present invention is to extract features of the image to be detected through a pre-trained high-resolution module, and then perform weighted processing through the attention mechanism of each sub-extraction network connection in the high-resolution module, and finally obtain a target feature map corresponding to the image to be detected through feature fusion, and the sub-extraction networks in the high-resolution module are connected in parallel to maintain high resolution from beginning to end, so that the features in the image can be effectively extracted, and the weighting through the attention module can improve the accuracy of the recognition model, and at the same time, the convergence speed of the model can be accelerated, thereby improving the efficiency of text detection, and the extracted target feature map is segmented into a probability map and a threshold map through a text segmentation network, and then the probability map and the threshold map are subjected to differentiable binarization processing, and the binarized image can be further close to the text area in the image through differentiable binarization, and the text area containing text information and the background area not containing the text area can be clearly determined by binarizing the target feature map, and the text area in the image to be detected can be accurately located, and then the text detection result of the image to be detected is determined through the binarized image, which solves the technical problem in the prior art that it is difficult to accurately and efficiently identify the text area in the image due to text distortion and irregular text morphology. It achieves accurate detection of text areas in images and improves the efficiency and accuracy of text detection.

[0076] Figure 3 is a flow chart of another text detection method provided by an embodiment of the present invention, such as Figure 3 As shown, the method includes:

[0077] S310, obtaining a text detection image dataset, wherein the text detection image dataset includes a text detection image training set and a text detection image test set.

[0078] The text detection image training set includes training images; and the text detection image test set includes images to be detected.

[0079] S320, input the image to be detected in the text detection image test set into the high-resolution module based on pre-training, and obtain subnetwork feature maps of the original resolution, 1 / 2 resolution, 1 / 4 resolution and 1 / 8 resolution of the image to be detected respectively.

[0080] Among them, the pre-trained high-resolution module is the HRNet network.

[0081] Optionally, the HRnet network of this embodiment consists of a high-resolution backbone network and three branch networks with different resolutions. The HRnet network initially includes two 3×3 convolutional layers with a step size of 2 to reduce the image size to 1 / 4 of the original size. The HRnet network includes four stages:

[0082] The first stage contains 4 residual units, each of which consists of a bottleneck structure with 64 channels. Finally, 3×3 transition convolutions (with strides of 1 and 2) are used in parallel to reduce the number of channels to C (32) and the resolution becomes 1 / 2 of the original (C = 64). The number of channels in the four stages is C, 2C, 4C, and 8C respectively. Each branch in each stage contains 4 residual units, each of which contains 2 3×3 convolutions. Through transition convolution, each newly added branch network has 1 / 2 resolution and 2 times the number of channels of the previous branch network. In addition to the 3×3 convolution downsampling with a stride of 2, the second, third, and fourth stages also include upsampling with the nearest neighbor strategy to add the three branch networks to the backbone network one by one, and finally form a multi-branch parallel network structure, in which the backbone network maintains high-resolution output throughout the process. Each channel of each resolution network is connected to at least one channel attention mechanism module.

[0083] Specifically, the image to be detected is input into a high-resolution module based on pre-training, and the high-resolution module outputs subnetwork feature maps of the original resolution, 1 / 2 resolution, 1 / 4 resolution and 1 / 8 resolution of the image to be detected.

[0084] S330. Perform inter-channel feature correction on the feature maps of each resolution subnetwork according to the channel attention mechanism module to obtain feature maps of each resolution subnetwork after feature correction.

[0085] Among them, the channel attention mechanism module is used to calculate the weight coefficient matrix of feature correction of feature maps of each resolution subnetwork.

[0086] Specifically, the network structure of the channel attention mechanism module is as follows: first, a global average pooling layer is used to obtain features with the same number of channels as the feature map (1×1×C'), and then two FC fully connected layers (Fully Connected) are used to reduce the dimension to C' / r and increase the dimension to the original C' (r=16), and then a sigmoid activation function is used to activate the feature map (so that the value is between 0 and 1) to obtain a weight coefficient matrix. The weight coefficient matrix is ​​directly multiplied by the original feature map matrix to obtain the corrected subnetwork feature map.

[0087] S340, fusing the feature maps of each resolution subnetwork after feature correction to obtain a fused feature map, that is, a target feature map.

[0088] S350, inputting the fused feature map into the text segmentation network DBhead network, and determining the text detection result according to the output of the DBhead network.

[0089] Specifically, the DBhead network is used to perform image segmentation on the fused feature map, generate a probability map and a threshold map of the fused feature image, and then generate a binary image of the fused feature map through differential binarization based on the probability map and the threshold map of the fused feature image. The DBhead network outputs the binary image of the fused feature map, and the text detection result of the image to be detected is determined based on the binary image of the fused feature map output by the DBhead network.

[0090] In the embodiment of the present invention, the implementation logic of the entire file detection method can be encapsulated into a file detection model. Figure 4 is a training and detection flow chart of a text detection model provided by an embodiment of the present invention, such as Figure 4 As shown, after obtaining the text detection image dataset, the data is initialized in the text detection image dataset, and the model parameters of the text detection model to be trained are initialized. Then, the initialized text detection image dataset and the initialized model parameters are passed through the improved text detection model DBNet network for model training. After the text detection model is trained, the images to be detected in the text detection image test set are input into the trained text detection model, and the text detection results are output.

[0091] Specifically, the file detection model can be obtained by improving the DBNet network. Figure 5 A schematic diagram of the structure of an improved DBNet network provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown in the figure, the improved DBnet includes: original resolution network, 1 / 2 resolution network, 1 / 4 resolution network and 1 / 8 resolution network. Each resolution network is connected to each other in parallel for up and down sampling. Each resolution network is connected to the attention mechanism module. Each resolution network generates a subnetwork feature map of the corresponding resolution, and then performs feature fusion after passing through the attention mechanism module to generate a fused feature map. Then, a probability map and a threshold map are generated based on the fused feature map, and the binary image of the fused feature map is determined, and finally the text detection result is generated.

[0092] Optionally, after inputting the image to be detected in the text detection image test set into the trained text detection model and outputting the text detection result, the method further includes: calculating the loss through a loss function according to the verification image of the image to be detected corresponding to the image to be detected in the text detection image test set and the text detection result, and adjusting the model parameters in the text detection model according to the calculated loss. When it is detected that the loss function in the text detection model converges, the text detection model training is completed and the text detection model is obtained. The loss function of the text detection model can be a cross entropy error function.

[0093] It should be noted that the loss function of the text detection model can be a loss function for each network in the text detection model, for example, the loss function can be set for the feature extraction network and the text detection model respectively. The loss function of the text detection model can also be a loss function used to adjust the entire file detection model.

[0094] Embodiment 3

[0095] Figure 6 Schematic diagram of the structure of a text detection device provided by Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a feature extraction module 410, an image binarization module 420 and a detection result determination module 430, wherein:

[0096] The feature extraction module 410 is used to obtain an image to be detected, and perform feature extraction on the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected;

[0097] An image binarization module 420 is used to determine a binarized image corresponding to the image to be detected according to the target feature map, wherein the binarized image is used to distinguish a text area containing text information from a background area other than the text area in the image to be detected;

[0098] The detection result determination module 430 is used to determine the text detection result of the image to be detected according to the binary image, wherein the text detection result at least includes the position information of the text area.

[0099] The technical solution of the embodiment of the present invention is to extract features from the image to be detected through a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected. The feature extraction network can efficiently extract features that may contain text in the image to be detected, thereby improving the efficiency of text detection. The binary image corresponding to the image to be detected is determined by the extracted target feature map. The text area containing text information and the background area not containing text area can be clearly determined by binarizing the target feature map, and the text area in the image to be detected can be accurately located. Then, the text detection result of the image to be detected is determined by the binary image, which solves the technical problem in the prior art that it is difficult to accurately and efficiently identify the text area in the image due to text distortion and irregular text morphology. Accurate detection of text areas in images is achieved, and the efficiency and accuracy of text detection are improved.

[0100] Optionally, the feature extraction module 410 is specifically used for:

[0101] The method of extracting features of the image to be detected based on the pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected includes:

[0102] Extracting features of the image to be detected based on the pre-trained high-resolution module to obtain sub-network feature maps of at least two resolutions;

[0103] Based on the attention mechanism module, each sub-network feature map is weighted, and the weighted sub-network feature maps are fused to obtain a target feature map corresponding to the image to be detected.

[0104] Optionally, the feature extraction module 410 is further configured to:

[0105] The high-resolution module includes at least two sub-extraction networks of different resolutions connected in parallel, and each sub-extraction network is respectively connected to at least one attention mechanism module;

[0106] The attention mechanism module performs weighted processing on each sub-network feature map, including:

[0107] For the sub-network feature graph output by each sub-extraction network, the attention mechanism modules connected to the sub-extraction network respectively perform weighted processing on the sub-network feature graph.

[0108] Optionally, the feature extraction module 410 is further configured to:

[0109] The attention mechanism includes a pooling layer, a first fully connected layer connected to the pooling layer, a first non-linear activation function layer connected to the first fully connected layer, a second fully connected layer connected to the non-linear activation function layer, and a second non-linear activation function layer connected to the second fully connected layer;

[0110] The attention mechanism-based module performs weighted processing on the sub-network feature graphs of each resolution respectively, including:

[0111] Performing global average pooling processing on the sub-network feature map through the pooling layer of the attention mechanism to obtain a target tensor;

[0112] Processing the target tensor via a first fully connected layer, a first nonlinear activation function layer, a second fully connected layer, and a second nonlinear activation function layer to obtain a weight coefficient matrix corresponding to the sub-network feature graph;

[0113] The sub-network feature maps of each resolution are weighted based on the weight coefficient matrix.

[0114] Optionally, the image binarization module 420 is specifically used for:

[0115] Performing text segmentation on the target feature map based on a pre-trained text segmentation network to obtain a probability map and a threshold map corresponding to the target feature map;

[0116] A binary image corresponding to the image to be detected is determined according to the probability map and the threshold map.

[0117] Optionally, the image binarization module 420 is further configured to:

[0118] The probability map and the threshold map are subjected to differentiable binarization processing to obtain a binarized image corresponding to the image to be detected.

[0119] Optionally, the detection result determination module 430 is specifically used to:

[0120] Determine text position information corresponding to the text information in the image to be detected according to the binary image, wherein the text position information includes a text detection image or text coordinate data of a text area marked with the image to be detected.

[0121] The text detection device provided in the embodiment of the present invention can execute the text detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0122] Embodiment 4

[0123] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a text detection method.

[0127] In some embodiments, the text detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the text detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the text detection method in any other appropriate manner (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0135] Embodiment 5

[0136] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the text detection method steps provided in any embodiment of the present invention are implemented. The method includes:

[0137] Acquire an image to be detected, and extract features from the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected;

[0138] Determine a binary image corresponding to the image to be detected according to the target feature map, wherein the binary image is used to distinguish a text area containing text information in the image to be detected from a background area other than the text area;

[0139] A text detection result of the image to be detected is determined according to the binary image, wherein the text detection result at least includes position information of the text area.

[0140] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0143] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text detection method, It is characterized in that include: Acquire an image to be detected, and extract features from the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected; Determine a binary image corresponding to the image to be detected according to the target feature map, wherein the binary image is used to distinguish a text area containing text information in the image to be detected from a background area other than the text area; Determining a text detection result of the image to be detected according to the binary image, wherein the text detection result at least includes position information of the text area; Wherein, the feature extraction network includes a high-resolution module and an attention mechanism module; The method of extracting features of the image to be detected based on the pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected includes: Extracting features of the image to be detected based on the pre-trained high-resolution module to obtain sub-network feature maps of at least two resolutions; Based on the attention mechanism module, each of the sub-network feature maps is weighted, and the weighted sub-network feature maps are subjected to feature fusion to obtain a target feature map corresponding to the image to be detected; The high-resolution module includes at least two sub-extraction networks with different resolutions connected in parallel, and each of the sub-extraction networks is respectively connected to at least one attention mechanism module; The attention mechanism module performs weighted processing on each sub-network feature map, including: For each sub-network feature graph output by the sub-extraction network, the attention mechanism module connected to the sub-extraction network performs weighted processing on the sub-network feature graph respectively; The attention mechanism includes a pooling layer, a first fully connected layer connected to the pooling layer, a first nonlinear activation function layer connected to the first fully connected layer, a second fully connected layer connected to the nonlinear activation function layer, and a second nonlinear activation function layer connected to the second fully connected layer; The attention mechanism-based module performs weighted processing on the sub-network feature graphs of each resolution, including: Performing global average pooling processing on the sub-network feature map through the pooling layer of the attention mechanism to obtain a target tensor; wherein the target tensor includes all features of the sub-network feature map; Processing the target tensor through the first fully connected layer, the first nonlinear activation function layer, the second fully connected layer, and the second nonlinear activation function layer to obtain a weight coefficient matrix corresponding to the sub-network feature graph; The sub-network feature maps of each resolution are weighted based on the weight coefficient matrix.

2. The method according to claim 1, It is characterized in that The step of determining a binary image corresponding to the image to be detected according to the target feature map includes: Performing text segmentation on the target feature map based on a pre-trained text segmentation network to obtain a probability map and a threshold map corresponding to the target feature map; A binary image corresponding to the image to be detected is determined according to the probability map and the threshold map.

3. The method according to claim 2, It is characterized in that The step of determining a binary image corresponding to the image to be detected according to the probability map and the threshold map includes: The probability map and the threshold map are subjected to differentiable binarization processing to obtain a binarized image corresponding to the image to be detected.

4. The method according to claim 1, It is characterized in that The step of determining the text detection result of the image to be detected according to the binary image comprises: Determine text position information corresponding to the text information in the image to be detected according to the binary image, wherein the text position information includes a text detection image or text coordinate data of a text area marked with the image to be detected.

5. A text detection device, It is characterized in that include: A feature extraction module is used to obtain an image to be detected, and perform feature extraction on the image to be detected based on a pre-trained feature extraction network to obtain a target feature map corresponding to the image to be detected; An image binarization module, used for determining a binarized image corresponding to the image to be detected according to the target feature map, wherein the binarized image is used to distinguish a text area containing text information from a background area other than the text area in the image to be detected; A detection result determination module, used to determine a text detection result of the image to be detected according to the binary image, wherein the text detection result at least includes position information of the text area; Wherein, the feature extraction network includes a high-resolution module and an attention mechanism module; The feature extraction module is specifically used to extract features from the image to be detected based on the pre-trained high-resolution module to obtain sub-network feature maps of at least two resolutions; weighted processing is performed on each of the sub-network feature maps based on the attention mechanism module, and feature fusion is performed on each of the weighted sub-network feature maps to obtain a target feature map corresponding to the image to be detected; The high-resolution module includes at least two sub-extraction networks with different resolutions connected in parallel, and each of the sub-extraction networks is respectively connected to at least one attention mechanism module; The feature extraction module is further configured to perform weighted processing on the sub-network feature graphs output by each sub-extraction network by the attention mechanism modules connected to the sub-extraction networks; The attention mechanism includes a pooling layer, a first fully connected layer connected to the pooling layer, a first nonlinear activation function layer connected to the first fully connected layer, a second fully connected layer connected to the nonlinear activation function layer, and a second nonlinear activation function layer connected to the second fully connected layer; Among them, the feature extraction module is specifically used to perform global average pooling processing on the sub-network feature map through the pooling layer of the attention mechanism to obtain a target tensor; wherein the target tensor includes all features of the sub-network feature map; the target tensor is processed through the first fully connected layer, the first non-linear activation function layer, the second fully connected layer and the second non-linear activation function layer to obtain a weight coefficient matrix corresponding to the sub-network feature map; based on the weight coefficient matrix, the sub-network feature map of each resolution is weighted.

6. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text detection method according to any one of claims 1 to 4 when executed.

Citation Information

Patent Citations

  • Image feature extraction method and device, storage medium and electronic equipment

    CN110298413A

  • Text detection method and device, electronic equipment and storage medium

    CN111723841A

  • Target detection method and device, electronic equipment and storage medium

    CN112990203A

  • Lightweight multi-scale feature fusion real-time image semantic segmentation method and system

    CN114445430A

  • Image processing method and apparatus for tamper proofing

    US20130039588A1