A cost-effective scene text detection method and system

Abstract

The present invention discloses a cost-effective scene text detection method and system, belonging to the field of image font recognition. The present invention firstly proposes an unsupervised domain adaptation method in the pre-training stage, by eliminating the domain difference between virtual data and real data, making the virtual data closer to the real data in terms of features, thereby obtaining a better initial model of transfer learning; secondly, a semi-supervised active learning method is proposed in the fine-tuning stage, which adaptively selects data for annotation according to the amount of annotation information of the real data, and then simultaneously uses the annotated and unannotated real data for training, so as to improve the scene text detection performance as much as possible under the condition of relatively low annotation cost.

Description

Technical Field

[0001] The present invention belongs to the field of image font recognition, and in particular relates to a cost-effective scene text detection method and system. Background Art

[0002] Text detection and recognition for scene images is a research hotspot in the field of computer vision. In recent years, with the development of deep learning, the detection and recognition technology of scene text has made great progress. However, the training of deep learning models requires a large amount of labeled data, and obtaining this labeled data is very expensive and time-consuming. In order to alleviate this problem, many researchers use large-scale automatically labeled virtual data to replace real data for training. At present, the performance of text recognition models trained on virtual and real data sets is comparable, while the performance of text detection models trained on virtual data sets is significantly behind that of text detection models trained on real data sets. In existing text detection work, a transfer learning method is often used, that is, the text detection model is first pre-trained on a large-scale virtual data set, and then fine-tuned on a real target data set for a specific scene.

[0003] During the pre-training phase, the unlabeled real dataset is not utilized, which makes it difficult for the model pre-trained only on the virtual dataset to adapt to the distribution of the real target dataset during migration, resulting in suboptimal transfer learning performance.

[0004] In the fine-tuning stage, existing methods still require a large annotation cost for fine-tuning on the entire real target dataset, and ignore the fact that the amount of information obtained by annotating different images is different, and some annotation information is redundant. Summary of the invention

[0005] The purpose of the present invention is to propose a cost-effective scene text detection method and system, which can achieve considerable performance while effectively saving data annotation costs. First, an unsupervised domain adaptation method is proposed in the pre-training stage. By eliminating the domain differences between virtual data and real data, the virtual data is closer to the real data in terms of features, thereby obtaining a better initial model for transfer learning. Secondly, a semi-supervised active learning method is proposed in the fine-tuning stage. According to the amount of annotation information of the real data, data is adaptively selected for annotation, and then the real data with and without annotations are used for training at the same time, so as to improve the scene text detection performance as much as possible under the condition of low annotation cost.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is as follows:

[0007] A cost-effective scene text detection method comprises the following steps:

[0008] Using labeled virtual data as a source domain and unlabeled real data as a target domain, both the virtual data and the real data are image data, a scene text detection network with an entropy-aware global alignment module and a text region alignment module is pre-trained, wherein the entropy-aware global alignment module aligns feature maps of the source domain and the target domain to reduce domain differences, and the text region alignment module aligns features of text candidate regions whose classification confidences predicted by the classification branch network are greater than a confidence threshold, and a pre-trained model is obtained after training;

[0009] Using the pre-trained model to perform active learning based on uncertainty measurement, a number of images are selected from the real data of the target domain for manual annotation to obtain the manually annotated real data, and a new scene text detection network is retrained using the manually annotated real data;

[0010] Using the pre-trained model and the retrained scene text detection network, the remaining real data in the target domain are enhanced with pseudo-annotation to obtain the real data with enhanced pseudo-annotation.

[0011] Fine-tuning the pre-trained model using the manually annotated real data and the enhanced pseudo-annotated real data;

[0012] The entropy-aware global alignment module and text region alignment module are removed from the fine-tuned pre-trained model to obtain a scene text detection network with updated parameters. The scene text detection network with updated parameters is used to process the scene image to be detected to obtain text detection results of arbitrary shapes.

[0013] The scene text detection network includes a feature extraction module, a region candidate network module, an interest region alignment network module and a detection head network module. The steps of processing the input image data include: using the convolutional neural network of the feature extraction module to extract the feature map of the scene image; using the region candidate network module to select the text candidate region from the feature map; using the interest region alignment network module to obtain the characteristics of the text candidate region according to the feature map and the text candidate region; using the classification branch, regression branch and segmentation branch of the detection head network module to process the characteristics of the text candidate region, the classification branch predicts the confidence score that the text candidate region belongs to the text region, the regression branch predicts the position offset between the text candidate region and the actual text region, the segmentation branch predicts the mask information of the text in the text candidate region, and generates the final arbitrary shape text detection result according to the mask information.

[0014] Furthermore, the pre-training loss is L pre =L base +λ1L1+λ2L2+λ3L3, where L baseis the loss generated by the scene text detection network during pre-training, L1, L2 are the losses generated by the entropy-aware global alignment module during pre-training, L3 is the loss generated by the text area alignment module during pre-training, and λ1, λ2, λ3 are weight parameters.

[0015] Furthermore, the loss of the scene text detection network during pre-training is:

[0016] L base =L rpn +L cls +L reg +L seg

[0017] Among them, L rpn , L cls , L reg and L seg Represent the losses generated by the region candidate network module, classification branch, regression branch, and segmentation branch respectively;

[0018] The loss of retraining the new scene text detection network and the loss of fine-tuning are both equal to L base .

[0019] Furthermore, the entropy-aware global alignment module includes two domain discriminators D1, D2 and two corresponding gradient reversal layers GRL, with a structure of GRL+D1+GRL+D2; the text area alignment module includes a domain discriminator D3 and a corresponding gradient reversal layer GRL, with a structure of GRL+D3; the domain discriminator is used to maximize the feature difference between virtual data and real data, and output a domain discrimination probability map; the gradient reversal layer is used to reverse the sign of the back-propagation gradient of the model when passing through this layer, so as to realize the adversarial learning of the domain discriminator and the feature extraction module.

[0020] Furthermore, when the entropy-aware global alignment module aligns the feature maps of the source domain and the target domain, the loss generated is:

[0021]

[0022] in, is the source domain image, is the target domain image, They are the features of the kth position of the feature map of the convolutional neural network output source domain image and target domain image, respectively, and the log is based on 2.

[0023] Furthermore, the domain discrimination probability map of the D1 output of the entropy-aware global alignment module is represented as d k =D1(G(x k )), using the information entropy function to calculate the entropy h of domain discrimination k =H(d k )=-dk ·log(d k )-(1-d k )·log(1-d k ); then align the entropy of the domain discrimination of the two domains, and the loss is:

[0024]

[0025] in, is the entropy of the domain discrimination of the source domain image, is the domain discriminative entropy of the target domain image.

[0026] Furthermore, the loss caused by the text region alignment module aligning the features of the text candidate regions whose classification confidence predicted by the classification branch network is greater than the confidence threshold is:

[0027]

[0028] in, are the i-th text candidate regions in the source domain image and the target domain image, respectively. They are The characteristics of , log to base 2.

[0029] Furthermore, active learning based on uncertainty measurement refers to using a data selection model to iteratively select data from an unlabeled real data set in the target domain for manual labeling, and the steps include:

[0030] Initially, the data selection model M0 is initialized using the pre-trained model, the unlabeled data pool U0 stores all unlabeled real data images of the target domain, and the manually labeled data pool L0 is empty;

[0031] During one iteration, M0 is used to estimate the image-level uncertainty measurement scores of all images in U0, and K images with the highest image-level uncertainty scores are selected. U0 is updated to U1 after subtracting the K images, and the K images are manually annotated and added to L0 to obtain L1, and M0 is retrained with the K manually annotated images to obtain M1; iterate according to this step until the amount of data in the manually annotated data pool reaches a predetermined threshold, and the iteration terminates.

[0032] Furthermore, the steps of obtaining an image-level uncertainty metric score include:

[0033] First, for each text candidate region x of the unlabeled real data in the target domain, the classification branch of the detection head network predicts the probability p of it being classified as text x , by calculating its entropy as the uncertainty measure score of the text candidate area:

[0034] H(x)=-px ·log(p x )-(1-p x )·log(1-p x )

[0035] Where log is based on 2;

[0036] Then, we average all the text candidate regions in the image to get the image-level uncertainty metric score:

[0037]

[0038] Where N is the number of text candidate regions in the image.

[0039] Furthermore, when enhancing the pseudo-annotation of the remaining real data in the target domain, the intersection over union (IoU) of the pseudo-annotation results of the pre-trained model and the retrained scene text detection network is calculated, and the result with an IoU greater than a specific threshold is taken as the final pseudo-annotation.

[0040] A cost-effective scene text detection system comprises: a scene text detection network, an entropy-aware global alignment module and a text region alignment module; the scene text detection network comprises a feature extraction module, a region candidate network module, an interest region alignment network module and a detection head network module: wherein:

[0041] A feature extraction module, used to extract feature maps of scene images through convolutional neural networks;

[0042] The region candidate network module is used to process the feature map and generate text candidate regions;

[0043] The region of interest alignment network module is used to process the feature map and the text candidate region to obtain the features of the text candidate region;

[0044] The detection head network module includes three branch networks: classification branch, regression branch, and segmentation branch. The classification branch is used to predict the confidence score that the text candidate area belongs to the text area. The regression branch is used to predict the position offset between the text candidate area and the actual text area. The segmentation branch is used to predict the mask information of the text in the text candidate area and generate the final arbitrary shape text detection result based on the mask information.

[0045] An entropy-aware global alignment module is used to align the feature maps of the convolutional neural network outputs of the source domain and the target domain during the pre-training phase to reduce the domain difference, where the source domain is labeled virtual data and the target domain is unlabeled real data;

[0046] The text region alignment module is used to align the features of the text candidate regions whose classification confidence predicted by the classification branch network is greater than the confidence threshold during the pre-training stage.

[0047] The present invention proposes a novel scene text detection framework, which uses an unsupervised domain adaptation algorithm to simultaneously utilize virtual data and unlabeled real data, and a semi-supervised active learning algorithm to reduce the real data labeling cost. For the unsupervised domain adaptation algorithm, two modules, entropy-aware global alignment and text region alignment, are proposed to eliminate the domain differences between virtual data and real data. For the semi-supervised active learning algorithm, an active learning algorithm based on uncertainty measurement and an enhanced pseudo-labeling strategy are proposed to adaptively select real data for labeling and effectively utilize the remaining real data. The present invention is significantly superior to the traditional pre-training and fine-tuning transfer learning strategy on multiple benchmark data sets, and can obtain better scene text detection performance with less labeling cost. The present invention aligns the features of virtual data with real data at the global and local levels, respectively, to give full play to the potential of virtual data. In addition, a semi-supervised active learning method is proposed to allow the model to adaptively select a small amount of real data for labeling, and an enhanced strategy is proposed to give the remaining data reliable pseudo-labeling. Experiments show that the method proposed by the present invention is effective on multiple data sets, significantly superior to the baseline method, and can achieve excellent performance even when only a small amount of real data is manually labeled. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural diagram of a cost-effective scene text detection system of the present invention.

[0049] Figure 2 It is a comparison chart of the F value of the test of the present invention and the traditional benchmark method. DETAILED DESCRIPTION

[0050] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0051] The present invention constructs a cost-effective scene text detection system through a two-stage framework. For ease of understanding and explanation, the present invention uses Mask R-CNN as a basic network to illustrate the technical solution of the present invention, but is not limited to this network, and any scene text detection network can be used. Figure 1As shown, Mask R-CNN includes: (1) Feature extraction module: Use convolutional neural network to extract feature map of scene image; (2) Region candidate network module: Process feature map to generate text candidate region; (3) Region of interest alignment network module: Process feature map and text candidate region to obtain the features of candidate region; (4) Detection head network module: According to the features of candidate region, through the processing of three branch networks of classification, regression and segmentation, it generates text detection results of arbitrary shapes. Among them, the classification branch is used to predict the confidence score that the candidate region belongs to the text region, the regression branch is used to predict the position offset between the candidate region and the actual text region, and the segmentation branch is used to predict the mask information of the text in the candidate region and generate the final text detection result based on the mask information. The loss function of Mask R-CNN can be expressed as:

[0052] L base =L rpn +L cls +L reg +L seg

[0053] Among them, L rpn , L cls , L reg and L seg They represent the loss functions of the region candidate network, classification branch, regression branch, and segmentation branch, respectively, which are consistent with the original Mask R-CNN algorithm and will not be repeated here.

[0054] Based on this, the framework proposed in this paper mainly includes two parts: (1) pre-training based on unsupervised domain adaptation; (2) fine-tuning based on semi-supervised active learning.

[0055] In the pre-training stage, the present invention uses labeled virtual data (image data) as the source domain and unlabeled real data (image data) as the target domain, and designs two unsupervised domain adaptation modules: the entropy-aware global alignment module and the text region alignment module. The present invention implements the domain adaptation algorithm in an adversarial learning manner and introduces three domain discriminators ( Figure 1 D1, D2, D3) and the corresponding gradient reversal layer (GRL). The domain discriminator outputs the domain discrimination probability map; the function of the gradient reversal layer is to reverse the sign of the back propagation gradient of the model when it passes through this layer. In this way, adversarial learning between the domain discriminator and the feature extraction module can be achieved, that is, the domain discriminator is used to maximize the feature difference between virtual data and real data, while the feature extraction module is used to minimize the feature difference between virtual data and real data.

[0056] Entropy-aware global alignment module: First, the feature maps of the convolutional neural network outputs corresponding to the source domain and the target domain are aligned to reduce the domain difference. The loss function is:

[0057]

[0058] in, is the source domain image, is the target domain image, G(x k ) is the feature of the kth position of the feature map output by the convolutional neural network, and the log is based on 2 (the same below).

[0059] Then, considering that the source domain is labeled but the target domain is not, the model’s confidence level for the two domains will be different. Therefore, the present invention proposes to use the entropy of the D1 discrimination result to further reduce the domain difference. Specifically, the domain discrimination probability map output by D1 can be expressed as d k =D1(G(x k )), the entropy of domain discrimination can be calculated using the information entropy function:

[0060] h k =H(d k )=-d k ·log(d k )-(1-d k )·log(1-d k )

[0061] Then align the entropy of the domain discrimination of the two domains, and the loss function is:

[0062]

[0063] in, is the entropy of the domain discrimination of the source domain image, is the domain discriminative entropy of the target domain image.

[0064] Text region alignment module: Since the foreground regions between different domains share more common features than the background regions, the alignment of the foreground regions is more important in the domain adaptation process. The present invention proposes a text region alignment module to strengthen the feature alignment of the text foreground regions. Specifically, the present invention aligns the features of the text candidate regions whose classification confidence predicted by the classification branch in the detection head network is greater than the confidence threshold, and the loss function is:

[0065]

[0066] Among them, x i is the i-th text candidate region in the image, F(x i ) is x i characteristics.

[0067] So the loss function of pre-training is L pre =L base+λ1L1+λ2L2+λ3L3, λ1,λ2,λ3 are weight parameters.

[0068] In the fine-tuning stage, it is time-consuming, laborious and unnecessary to label the real data of the entire target domain and use it for fine-tuning. In general self-training strategies, pre-trained models are often used to predict unlabeled real target data sets and the prediction results are used as pseudo-labels, but such pseudo-labels are often not accurate enough and may mislead model learning. The present invention designs a semi-supervised active learning method in the fine-tuning stage, first using an active learning algorithm based on uncertainty metrics to select real data with low confidence in the prediction of the pre-trained model, and then manually labeling them, while the remaining real data are automatically labeled using an enhanced pseudo-labeling strategy.

[0069] Active learning algorithm based on uncertainty measurement: Active learning is the process of iteratively selecting data from the unlabeled target domain real data set for manual labeling using a data selection model. In order to increase the amount of information brought by the labeled data to the model learning, the present invention selects the data sample with the largest model prediction uncertainty for labeling in each active learning iteration step. Specifically, for each text candidate region x of the unlabeled real data, the classification branch of the detection head network can predict the probability p of it being classified as text. x , by calculating its entropy as the uncertainty measure score of the text candidate area:

[0070] H(x)=-p x ·log(p x )-(1-p x )·log(1-p x )

[0071] The image-level uncertainty metric score is obtained by averaging all the text candidate regions in the image:

[0072]

[0073] Where N is the number of text candidate regions in the image.

[0074] Assume that the entire active learning process is divided into T steps, and the data selection model at the tth step is M t , the unlabeled data pool is U t , the manually labeled data pool is L t. Initially, M0 is initialized with the pre-trained model, U0 stores all unlabeled target domain real data images, and L0 is empty. At t=1, M0 is used to estimate the image-level uncertainty measurement scores of all images in U0, and the K images with the highest image-level uncertainty scores are selected. U0 is updated to U1 after subtracting the K images, and the K images are manually annotated and added to L0 to obtain L1, and M0 is retrained with the K manually annotated images to obtain M1. This method is iterated until the amount of data in the manually annotated data pool reaches a predetermined threshold, and the iteration is terminated.

[0075] Enhanced pseudo-labeling strategy: Due to the different distributions of virtual data and real data, different text detection results are often produced when the models trained with them are used for prediction. The present invention proposes to use the complementarity of virtual data and real data to provide a reliable pseudo-label for the remaining real data. Specifically, the present invention uses a pre-trained model and a model trained with manually labeled data selected from an active learning algorithm, namely a retrained Mask R-CNN network (with a training loss of L base ), pseudo-label the remaining real data, and calculate the intersection over union (IoU) of the pseudo-labeling results of the two models. Only the results with IoU greater than a specific threshold are used as the final pseudo-labeling. The pseudo-labeling is to distinguish it from manual labeling. Manual labeling is real labeling. Pseudo-labeling is to use the predicted detection result of a model for an image as its label. The advantage of pseudo-labeling is that it does not require labeling costs, but the disadvantage is that it may not be accurate.

[0076] Finally, the pre-trained model is fine-tuned using the manually annotated real data selected by active learning and the remaining enhanced pseudo-annotated real data, and the fine-tuning loss L fine =L base Among them, since the pseudo-annotated data is not accurate enough in text positioning, the pseudo-annotated data does not participate in the optimization of the regression and segmentation branches in the detection head network.

[0077] During the testing process, the two proposed domain adaptation modules can be removed, and only the Mask R-CNN network with updated parameters is needed to predict the text detection results.

[0078] experiment:

[0079] The inventors conducted experiments on two virtual datasets and two real datasets to evaluate the performance of the technical method of the present invention. Among them, SynthText is a virtual dataset containing 800,000 images, which automatically adds virtual text on natural images through a heuristic algorithm. VISD is a virtual dataset containing 10,000 images. The semantic rationality of the text and background is considered during the generation process of this dataset, so it looks more realistic. ICDAR2015 is a scene text dataset captured by a real camera, which contains 1,000 training images and 500 test images. ICDAR2013 is a real dataset containing 229 training images and 233 test images. The text in this dataset is generally placed horizontally and is more prominent.

[0080] Figure 2 The comparison results of the proposed method and the baseline method in various dataset migration tasks are shown. The baseline method is pre-trained on all virtual data and fine-tuned on randomly selected real data. Figure 2 It can be seen that no matter what the proportion of manual annotation of the selected real target data set is, the present invention is obviously better than the benchmark method, which proves the effectiveness of the technical method proposed in the present invention.

[0081] Table 1 shows the comparison of the effects of the present invention and other advanced methods on the two migration tasks of VISD to ICDAR2015 and VISD to ICDAR2013. Among them, other comparison methods show the results of pre-training on VISD and fine-tuning on randomly selected ICDAR2015 or ICDAR2013. It can be seen from Table 1 that under the condition of the same manual annotation ratio of the target dataset, the present invention is significantly better than other advanced methods. In the task of VISD to ICDAR2015, the present invention, with only 5% manually annotated target dataset, exceeds the performance of PSENet, PAN, etc. with a 10% manually annotated target dataset. This proves that the present invention can effectively reduce the annotation cost.

[0082] Table 1 Comparison of the present invention with other advanced methods

[0083]

[0084] Table 2 shows the ablation experiment of the present invention on the VISD migration to ICDAR2015 task, where the manual annotation ratio of the target dataset is 10%. It can be seen that all the modules proposed in the present invention are effective in improving the performance.

[0085] Table 2 Ablation experiment of the present invention

[0086]

[0087] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A cost-effective scene text detection method, characterized in that: The following steps are involved: Using labeled virtual data as a source domain and unlabeled real data as a target domain, both the virtual data and the real data are image data, a scene text detection network with an entropy-aware global alignment module and a text region alignment module is pre-trained, wherein the entropy-aware global alignment module aligns feature maps of the source domain and the target domain to reduce domain differences, and the text region alignment module aligns features of text candidate regions whose classification confidences predicted by the classification branch network are greater than a confidence threshold, and a pre-trained model is obtained after training; Using the pre-trained model to perform active learning based on uncertainty measurement, a number of images are selected from the real data of the target domain for manual annotation to obtain the manually annotated real data, and a new scene text detection network is retrained using the manually annotated real data; Using the pre-trained model and the retrained scene text detection network, the remaining real data in the target domain are enhanced with pseudo-annotation to obtain the real data with enhanced pseudo-annotation. Fine-tuning the pre-trained model using the manually annotated real data and the enhanced pseudo-annotated real data; The entropy-aware global alignment module and text region alignment module are removed from the fine-tuned pre-trained model to obtain a scene text detection network with updated parameters. The scene text detection network with updated parameters is used to process the scene image to be detected to obtain text detection results of arbitrary shapes. The scene text detection network includes a feature extraction module, a region candidate network module, an interest region alignment network module and a detection head network module. The steps of processing the input image data include: using the convolutional neural network of the feature extraction module to extract the feature map of the scene image; using the region candidate network module to select the text candidate region from the feature map; using the interest region alignment network module to obtain the characteristics of the text candidate region according to the feature map and the text candidate region; using the classification branch, regression branch and segmentation branch of the detection head network module to process the characteristics of the text candidate region, the classification branch predicts the confidence score that the text candidate region belongs to the text region, the regression branch predicts the position offset between the text candidate region and the actual text region, the segmentation branch predicts the mask information of the text in the text candidate region, and generates the final arbitrary shape text detection result according to the mask information.

2. The method according to claim 1, characterized in that The loss of pre-training is L pre =L base +λ1L1+λ2L2+λ3L3, where L base is the loss generated by the scene text detection network during pre-training, L1, L2 are the losses generated by the entropy-aware global alignment module during pre-training, L3 is the loss generated by the text area alignment module during pre-training, and λ1, λ2, λ3 are weight parameters.

3. The method according to claim 2, characterized in that The loss of the scene text detection network during pre-training is: L base =L rpn +L cls +L reg +L seg Among them, L rpn , L cls , L reg and L seg Represent the losses generated by the region candidate network module, classification branch, regression branch, and segmentation branch respectively; The loss of retraining the new scene text detection network and the loss of fine-tuning are both equal to L base .

4. The method according to claim 2, characterized in that The entropy-aware global alignment module consists of two domain discriminators D1, D2 and two corresponding gradient reversal layers GRL, with the structure of GRL+D1+GRL+D2; The text region alignment module includes a domain discriminator D3 and a corresponding gradient reversal layer GRL, with a structure of GRL+D3. The domain discriminator is used to maximize the feature difference between virtual data and real data, and output a domain discrimination probability map. The gradient reversal layer is used to reverse the sign of the back-propagation gradient of the model when it passes through this layer, thereby realizing adversarial learning between the domain discriminator and the feature extraction module.

5. The method according to claim 4, characterized in that When the entropy-aware global alignment module aligns the feature maps of the source domain and the target domain, the loss generated is: in, is the source domain image, is the target domain image, are the features of the kth position of the feature maps of the convolutional neural network output source domain image and target domain image, respectively, and the log is based on 2; The domain discrimination probability map of the D1 output of the entropy-aware global alignment module is denoted as d k =D1(G(x k )), using the information entropy function to calculate the entropy h of domain discrimination k =H(d k )=-d k ·log(d k )-(1-d k )·log(1-d k ); then align the entropy of the domain discrimination of the two domains, and the loss is: in, is the entropy of the domain discrimination of the source domain image, is the domain discriminative entropy of the target domain image.

6. The method according to claim 4, characterized in that The loss caused by the text region alignment module to align the features of the text candidate regions whose classification confidence predicted by the classification branch network is greater than the confidence threshold is: in, are the i-th text candidate regions in the source domain image and the target domain image, respectively. They are The characteristics of , log to base 2.

7. The method according to claim 1, characterized in that Active learning based on uncertainty measurement refers to using a data selection model to iteratively select data from the unlabeled real data set of the target domain for manual labeling. The steps include: Initially, the data selection model M0 is initialized using the pre-trained model, the unlabeled data pool U0 stores all unlabeled real data images of the target domain, and the manually labeled data pool L0 is empty; During one iteration, M0 is used to estimate the image-level uncertainty measurement scores of all images in U0, and K images with the highest image-level uncertainty scores are selected. U0 is updated to U1 after subtracting the K images, and the K images are manually annotated and added to L0 to obtain L1, and M0 is retrained with the K manually annotated images to obtain M1; iterate according to this step until the amount of data in the manually annotated data pool reaches a predetermined threshold, and the iteration terminates.

8. The method according to claim 7, characterized in that The steps to obtain the image-level uncertainty metric score include: First, for each text candidate region x of the unlabeled real data in the target domain, the classification branch of the detection head network predicts the probability p of it being classified as text x , by calculating its entropy as the uncertainty measure score of the text candidate area: H(x)=-p x ·log(p x )-(1-p x )·log(1-p x ) Where log is based on 2; Then, we average all the text candidate regions in the image to get the image-level uncertainty metric score: Where N is the number of text candidate regions in the image.

9. The method according to claim 1, characterized in that When enhancing the pseudo-annotation of the remaining real data in the target domain, the intersection over union (IoU) of the pseudo-annotation results of the pre-trained model and the retrained scene text detection network is calculated, and the result with an IoU greater than a specific threshold is taken as the final pseudo-annotation.

10. A cost-effective scene text detection system, characterized in that: include: Scene text detection network, entropy-aware global alignment module, and text region alignment module; The scene text detection network includes a feature extraction module, a region candidate network module, an area of ​​interest alignment network module and a detection head network module: A feature extraction module, used to extract feature maps of scene images through convolutional neural networks; The region candidate network module is used to process the feature map and generate text candidate regions; The region of interest alignment network module is used to process the feature map and the text candidate region to obtain the features of the text candidate region; The detection head network module includes three branch networks: classification branch, regression branch, and segmentation branch. The classification branch is used to predict the confidence score that the text candidate area belongs to the text area. The regression branch is used to predict the position offset between the text candidate area and the actual text area. The segmentation branch is used to predict the mask information of the text in the text candidate area and generate the final arbitrary shape text detection result based on the mask information. An entropy-aware global alignment module is used to align the feature maps of the convolutional neural network outputs of the source domain and the target domain during the pre-training phase to reduce the domain difference, where the source domain is labeled virtual data and the target domain is unlabeled real data; The text region alignment module is used to align the features of the text candidate regions whose classification confidence predicted by the classification branch network is greater than the confidence threshold during the pre-training stage.

Images