A hybrid supervised text detection method based on prompt segmentation task

Abstract

The application relates to a hybrid supervision text detection method based on a prompt segmentation task, comprising the following steps of S1, constructing a hybrid supervision text detection model TextSAM comprising an image encoder, a prompt encoder and a mask decoder, the TextSAM adopts a hybrid supervision training mode, a training data set of the hybrid supervision training mode contains a part of a full supervision data set and a part of a weak supervision data set, at this time, full supervision training is carried out; at this time, weak supervision training is carried out. An image is input into the trained TextSAM, and the output is a prediction label of the image. Experiments prove that under the hybrid supervision training mode, only 25% of full supervision data and 75% of weak supervision data are used, and the TextSAM can achieve a performance close to that of 100% full supervision data, thereby reducing the dependence on labeled data.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a hybrid supervised text detection method based on cue segmentation tasks. Background Technology

[0002] In today's world, the way humans interact with machines is undergoing a major transformation. Among these transformations, endowing computers with the ability to understand and perceive the real world is particularly important. Text, as a tool created by humans, carries rich semantic information and is closely integrated with real-world scenarios. Therefore, understanding and processing text within a scene is crucial for computers to truly understand and perceive their surroundings. Utilizing computer technology to detect and recognize text in images and videos, capturing key information, is extremely important for research in the field of Computer Vision (CV).

[0003] Optical Character Recognition (OCR) technology is widely used to extract text information from various images and has been applied in many fields. For example, in the field of smart office, automated spreadsheet processing programs can detect and recognize spreadsheet content and match it according to preset patterns; in the retail and logistics fields, recognizing text information on product labels and packages helps systems organize product and package information, improving the operational efficiency of retail and logistics systems; and in the field of human-computer interaction, OCR technology can help visually impaired people recognize and read aloud text they cannot see. Traditional OCR technology mainly processes document images with simple backgrounds and has strict limitations on the shooting angle, lighting conditions, and image quality of the input image. However, in scene text images, the image itself may have problems such as blurriness, tilt, perspective, and uneven lighting, and the shape, size, and aspect ratio of the text region in the image also vary. These factors make it difficult for traditional OCR technology to achieve good results for complex scene text images. With the development of high-performance computing technology, deep learning technology has gradually replaced traditional OCR technology based on manually designed features and classifiers.

[0004] Accurate prediction of text regions in scene images is crucial for subsequent text recognition. Therefore, natural scene text detection technology has become a research hotspot in the field of computer vision, with broad application potential and practical value.

[0005] In recent years, scene text detection algorithms have made significant progress. These detectors can detect text of arbitrary shapes. However, these methods typically require large-scale polygon-level annotated data during training. According to research by Zhao et al., annotating the contour of a curved text takes approximately three times longer than annotating a quadrilateral (13 seconds vs. 4 seconds). Therefore, obtaining such a fully supervised annotated dataset is expensive and time-consuming, hindering the acquisition of large-scale scene text datasets. In the field of scene text detection, commonly used curved text datasets such as Total-Text and CTW1500 contain 1555 and 1500 fully annotated images, respectively. Compared to datasets in object detection and segmentation (such as COCO), these curved scene text datasets are relatively small. When constructing the SA-1B dataset, which contains over 11 million images and over 1 billion segmentation mask labels, Meta AI employed an interactive segmentation annotation tool. This tool generates hints by having professional annotators click on foreground / background points to improve the model output. This approach improves the efficiency of annotating individual targets and accelerates the overall annotation process, making it essential for annotating complex targets with diverse shapes.

[0006] The cornerstone of interactive segmentation annotation tools is a cue-supported segmentation network. To build a similar solution in the field of scene text detection, this chapter explores how to apply the cue-based segmentation method, Segment Anything Model (SAM), to scene text detection. Among various interaction methods, this paper selects text region foreground points and text bounding boxes as cues, constructing a cue-based hybrid supervised scene text detection model, TextSAM. To reduce the dependence on fully supervised data during model training, this chapter employs various weakly supervised annotation methods (such as text region foreground points or bounding boxes) to study hybrid supervised training for TextSAM. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, the technical problem this invention aims to solve is that traditional methods suffer from: 1) contour-based methods suffer from significant noise, resulting in low efficiency in optimizing along rough text contours; 2) fully supervised methods rely on a large number of fully supervised labels, which are difficult to obtain; and current hybrid supervised models cannot meet the requirements.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a hybrid supervised text detection method based on cue segmentation task, comprising the following steps:

[0009] S1: Construct a hybrid supervised text detection model, TextSAM, which consists of three parts: an image encoder, a cue encoder, and a mask decoder.

[0010] An image encoder is used to extract image features. It consists of a visual Transformer and is responsible for extracting image features.

[0011] The cue encoder is used to encode different forms of cues into the same space, and its input is the cue position code and multiple learnable embeddings.

[0012] The mask decoder is used to fuse image features and cue codes to generate the corresponding segmentation mask prediction.

[0013] S2: TextSAM employs a hybrid supervised training model, and its training dataset includes a portion of fully supervised datasets. and some weakly supervised datasets Among them, the fully supervised dataset Includes images and the fully supervised set of labels y s y s Contains the outlines of all text regions; weakly supervised dataset Includes images And weakly labeled tags y w .

[0014] Hybrid supervised training has two branches: fully supervised training and weakly supervised training. This is achieved by defining a set of fully supervised input data. and weakly supervised datasets By adjusting the data ratio, a corresponding proportion of mixed-supervised training can be completed. When At that time, full-supervised training will be conducted; when At that time, weak supervision training will be conducted.

[0015] S3: Input an image into the trained TextSAM, and the output is the predicted label for that image.

[0016] Furthermore, the hybrid supervised training in S2 includes the following steps:

[0017] 1) Fully supervised training set Weakly supervised training set The number of training rounds is N; the maximum number of iterations in each iteration for fully supervised training and weakly supervised training are S1 and S2, respectively. This represents the nth image and its corresponding label in the fully supervised training set. This represents the nth image and its corresponding weak label in a weakly supervised dataset.

[0018] Two TextSAMs with identical structures are used, one as a student network. Another as a teacher network

[0019] 2) Student Network initialization;

[0020] 3) Teacher Network initialization,

[0021] 4) Initialize loss Loss under full supervision and loss under weak supervision, respectively;

[0022] 5) Obtain The images and their corresponding labels in the data are grouped into a single batch;

[0023] 6) Random selection One target in the image is used as the training target;

[0024] 7) Obtain hints for the student model for the randomly selected training objective in 6) and enhance those hints;

[0025] 8) The student model performs forward propagation to predict the labels of the randomly selected targets in 6).

[0026] 9) Calculate the loss between the true label and the predicted label of the randomly selected target in 6), and add this loss to the fully supervised loss. superior;

[0027] 10) Determine if the fully supervised training has reached the maximum number of iterations S1. If it has reached S1, then execute 11); otherwise, return to 6) to select a new training target.

[0028] 11) Obtain The images and their corresponding weak labels are grouped into a batch;

[0029] 12) Random selection One target in the image is used as the training target;

[0030] 13) Obtain hints for the student model for the randomly selected training objective in 12) and enhance those hints;

[0031] 14) Obtain prompts from the teacher model for the randomly selected training objective in 12);

[0032] 15) The student model forward propagates to predict the labels of the randomly selected targets in 12);

[0033] 16) The teacher model forward propagates to predict the labels of the randomly selected targets in 12);

[0034] 17) The teacher model generates pseudo-labels for the training targets randomly selected in 12), and uses these pseudo-labels as their corresponding weak labels;

[0035] 18) Calculate the loss between the weak label and the predicted label of the randomly selected target in 12), and add this loss to the weak supervision loss. superior;

[0036] 19) Determine if the weakly supervised training has reached the maximum number of iterations S2. If it has reached S2, then execute 20); otherwise, return to 12) and select a new training target.

[0037] 20) Calculate the total loss Then based on the current batch The parameters of the student model are updated using backpropagation with gradient descent. For the total loss of fully supervised training, The total loss for weakly supervised training is μ, where μ is the balance factor.

[0038] 21) Update the parameters of the teacher network using EMA;

[0039] 22) Determine if the maximum number of iterations N has been reached. If the maximum number of iterations has been reached, then the current teacher model and student model are the optimal teacher model and optimal student model, otherwise return 4).

[0040] Furthermore, in 7) and 13), the method for obtaining the prompt is: the prompt is a foreground point prompt or a bounding box prompt;

[0041] Random selection program randomly selects images The selected contour of a target region is used to simultaneously generate hints and mask labels, resulting in hint P and ground truth segmentation map labels.

[0042] The function will be prompted to retrieve the outline of the selected target area. A foreground point tooltip can be generated by calculating the center point of the text region, or a bounding box tooltip can be generated by calculating the smallest bounding rectangle in the text region.

[0043] Furthermore, in 7) and 13), the method for enhancing the prompts is as follows:

[0044] Enhancements to foreground point indicators: Center point offset is considered a disturbance;

[0045] Data augmentation for bounding box hints: Set two types of perturbations: center point offset and scale variation.

[0046] Furthermore, the total loss of the fully supervised training The calculation method is as follows:

[0047] DICE loss and Focal loss are for the mask image output, while the mean squared error loss (MSE) is for the intersection-over-union prediction.

[0048] DICE losses It is used to measure the difference in distribution between the model's mask output and the real mask.

[0049]

[0050] in These are actual text region mask labels. This is a probability graph of the prediction.

[0051] Focal loss Calculated from the predicted probability map and the true mask:

[0052]

[0053] Where α is the positive and negative sample balance factor, γ is the easy and difficult sample balance factor, and H and W represent the width and height of the probability plot, respectively. For the mask of pixels h and w, Let h and w be the model's predicted probability values.

[0054] The mean squared error loss (MSE) measures the difference in distribution between the model's predicted cross-union ratio (CUI) output and the true CUI.

[0055]

[0056] in It is the intersection-union ratio (IoU) of the binary image of the predicted probability map and the true mask. It is the predicted crossover ratio.

[0057] Ultimately, this resulted in an overall loss.

[0058]

[0059] Where λ is the balance factor.

[0060] Furthermore, the weak supervision loss The calculation method is as follows:

[0061]

[0062] in, For teacher model intersection-over-comparison prediction, For student model intersection-over-union prediction, For the output of the student model, This is a pseudo-tag.

[0063] Furthermore, the step of updating the teacher network parameters using EMA in step 21) is as follows:

[0064] θ t ←αθ t +(1-α)θ s

[0065] Where θ t For the network parameters of the teacher model, θ s Let α be the network parameter of the student network.

[0066] Compared with the prior art, the present invention has at least the following advantages:

[0067] 1. During interactive annotation, TextSAM uses human input as prompts and then outputs contour-level pseudo-labels. By using weaker annotation forms, such as bounding boxes or foreground points, annotation time can be significantly reduced. Compared to contour annotation (approximately 13 seconds per unit), bounding box annotation (approximately 4 seconds per unit) and foreground point annotation (approximately 2 seconds per unit) can significantly reduce annotation time costs.

[0068] 2. The method of this invention utilizes foreground points or bounding boxes of text regions as cues and can be fine-tuned on real datasets. Experimental results show that, combined with cues, TextSAM significantly outperforms existing scene text detection models. When using bounding boxes as cues, TextSAM leads the TIoU-F metrics on the Total-Text, CTW1500, and ICDAR2015 datasets by 13.5%, 7.5%, and 14.0%, respectively, demonstrating TextSAM's significant potential in interactive annotation work. In the hybrid supervised training mode, using only 25% fully supervised data and 75% weakly supervised data, TextSAM achieves performance close to that of 100% fully supervised data, proving that the hybrid supervised training mode proposed in this invention can effectively reduce the model's dependence on fully supervised data during training.

[0069] 3. This paper sets up a training task based on the cue segmentation task, using the foreground points and bounding boxes of the text region as cues to avoid the influence of noise from the contour annotation.

[0070] 4. To reduce the model's dependence on fully supervised data, this paper combines a hybrid supervised training mode, using a mixture of partially weakly supervised labeled data and fully supervised labeled data for training. Furthermore, based on the Segment AnythingModel (SAM) cue segmentation model, the segmentation capabilities of SAM are transferred to scene text detection tasks, improving segmentation performance. Experiments demonstrate that TextSAM can achieve performance similar to that of fully supervised training even with a low proportion of fully supervised data. Attached Figure Description

[0071] Figure 1Ablation experiment with threshold τ. The dashed line represents the corresponding baseline.

[0072] Figure 2 Equilibrium factor μ ablation experiment. The dashed line represents the corresponding baseline.

[0073] Figure 3 TextSAM network structure diagram.

[0074] Figure 4 A schematic diagram of a text training mode in a hybrid supervised learning scenario.

[0075] Figure 5 The basic steps of fully supervised training are: ① Randomly select a target from the labels; ② Generate the corresponding mask image label.

[0076] Figure 6 TextSAM inference process. Detailed Implementation

[0077] The present invention will now be described in further detail.

[0078] See Figure 1-6 A hybrid supervised text detection model, TextSAM, is constructed, which consists of three parts: an image encoder, a cue encoder, and a mask decoder.

[0079] An image encoder is used to extract image features. It consists of a visual Transformer (ViT) and is responsible for extracting image features.

[0080] The cue encoder is used to encode different forms of cues into the same space, and its input is the cue position code and multiple learnable embeddings.

[0081] The mask decoder is used to fuse image features and cue codes to generate the corresponding segmentation mask prediction.

[0082] 1. For image encoders:

[0083] Image I is input to the image encoder to obtain image features f.

[0084] 2. For the prompt encoder:

[0085] To meet the needs of interactive scene annotation, TextSAM selected the foreground point and bounding box of the text region as the text cue P. Meanwhile, because the mask output of text is unambiguous, TextSAM restricts a cue to having only one corresponding mask output.

[0086] For a foreground point, its positional encoding embedding is first obtained through positional encoding. This positional encoding embedding is then added to a learnable embedding to obtain the foreground point's embedding encoding. For a bounding box, its embedding encoding is composed of the embedding encodings of the top-left and bottom-right corner points of the bounding box. The cue-based embedding encoding is then concatenated with learnable variables representing the mask image prediction and the intersection-over-union (IoU) prediction to obtain the sparse embedding.

[0087] 3. For mask generators:

[0088] After the image features output by the image encoder and the sparse embedding generated by the cue encoder are fed into the mask generator, the mask generator will generate the corresponding mask value output. Intersection over Union (IOU) prediction output This process can be simplified to

[0089] A hybrid supervised text detection method based on cue segmentation task includes the following steps:

[0090] S1: Construct a hybrid supervised text detection model, TextSAM, which consists of three parts: an image encoder, a cue encoder, and a mask decoder.

[0091] An image encoder is used to extract image features. It consists of a visual Transformer and is responsible for extracting image features.

[0092] The cue encoder is used to encode different forms of cues into the same space, and its input is the cue position code and multiple learnable embeddings.

[0093] The mask decoder is used to fuse image features and cue codes to generate the corresponding segmentation mask prediction.

[0094] S2: TextSAM employs a hybrid supervised training model, and its training dataset includes a portion of fully supervised datasets. and some weakly supervised datasets Among them, the fully supervised dataset Includes images and fully supervised label set Contains the outlines of all text regions; weakly supervised dataset Includes images And weak tags For example, the bounding box of all text areas, the foreground point, etc.

[0095] Hybrid supervised training has two branches: fully supervised training and weakly supervised training. This is achieved by defining a set of fully supervised input data. and weakly supervised datasets By adjusting the data ratio, a corresponding proportion of mixed-supervised training can be completed. When At that time, full-supervised training will be conducted; when At that time, weak supervision training will be conducted.

[0096] S3: Input an image into the trained TextSAM, and the output is the predicted label for that image.

[0097] Specifically, the hybrid supervised training in S2 includes the following steps:

[0098] 1) Fully supervised training set Weakly supervised training set The number of training rounds is N; the maximum number of iterations in each iteration for fully supervised training and weakly supervised training are S1 and S2, respectively. This represents the nth image and its corresponding label in the fully supervised training set. This represents the nth image and its corresponding weak label in a weakly supervised dataset.

[0099] Two TextSAMs with identical structures are used, one as a student network. Another as a teacher network

[0100] 2) Student Network initialization;

[0101] 3) Teacher Network initialization,

[0102] 4) Initialize loss Loss under full supervision and loss under weak supervision, respectively;

[0103] 5) Obtain The images and their corresponding labels in the data are grouped into a single batch;

[0104] 6) Each image contains multiple targets, each marked with a corresponding bounding box, and the targets are randomly selected. One target in the image is used as the training target;

[0105] 7) Obtain hints for the student model for the randomly selected training objective in 6) and enhance those hints;

[0106] 8) The student model performs forward propagation to predict the labels of the randomly selected targets in 6).

[0107] 9) Calculate the loss between the true label and the predicted label of the randomly selected target in 6), and add this loss to the fully supervised loss. superior;

[0108] 10) Determine if the fully supervised training has reached the maximum number of iterations S1. If it has reached S1, then execute 11); otherwise, return to 6) to select a new training target.

[0109] 11) Obtain The images and their corresponding weak labels are grouped into a batch;

[0110] 12) Random selection One target in the image is used as the training target;

[0111] 13) Obtain hints for the student model for the randomly selected training objective in 12) and enhance those hints;

[0112] 14) Obtain prompts from the teacher model for the randomly selected training objective in 12);

[0113] 15) The student model forward propagates to predict the labels of the randomly selected targets in 12);

[0114] 16) The teacher model forward propagates to predict the labels of the randomly selected targets in 12);

[0115] 17) The teacher model generates pseudo-labels for the training targets randomly selected in 12), and uses these pseudo-labels as their corresponding weak labels;

[0116] 18) Calculate the loss between the weak label and the predicted label of the randomly selected target in 12), and add this loss to the weak supervision loss. superior.

[0117] 19) Determine if the weakly supervised training has reached the maximum number of iterations S2. If it has reached S2, then execute 20); otherwise, return to 12) and select a new training target.

[0118] 20) Calculate the total loss Then based on the current batch The parameters of the student model are updated using backpropagation with gradient descent. Total losses for fully supervised training The total loss for weakly supervised training is μ, where μ is the balance factor.

[0119] 21) Update the parameters of the teacher network using EMA;

[0120] 22) Determine if the maximum number of iterations N has been reached. If the maximum number of iterations has been reached, then the current teacher model and student model are the optimal teacher model and optimal student model, otherwise return 4).

[0121] Branch 1, the process of fully supervised training:

[0122] S2-1-0: Initialize model parameters.

[0123] S2-1-1: Using images The TextSAM is input in pairs with the label y. During training, the random selection process will randomly select... The selected contour of a target region is used to generate a hint P, and the selected contour is also used to generate a mask label to obtain the true segmentation map label.

[0124] S2-1-2: Selecting the outline of the target area will prompt you to obtain the function. Foreground point cues are generated by calculating the center point of the text region, and bounding box cues are generated by calculating the smallest bounding rectangle in the text region.

[0125] S2-3: To obtain more diverse prompts, a prompt enhancement function is used. Additional data augmentation will be applied to foreground and bounding box cues. Foreground cues, the augmentation involves a perturbation of the center point offset; for bounding box cues, the augmentation uses both center point offset and scale variation as perturbations. Preferably, the center point offset augmentation is limited to 5 pixels, and the scale variation is limited to 2%.

[0126] S2-1-4: Handled prompt and images It will simultaneously send the corresponding mask output to TextSAM. Intersection over Union (IOU) prediction output

[0127] S2-1-5: Calculate the loss for the current batch DICE loss and Focal loss are for the mask image output, while the mean squared error loss (MSE) is for the intersection-over-union prediction.

[0128] DICE losses It is used to measure the difference in distribution between the model's mask output and the real mask.

[0129]

[0130] in These are actual text region mask labels. This is a probability graph of the prediction.

[0131] Focal loss Calculated from the predicted probability map and the true mask:

[0132]

[0133] Where α is the positive and negative sample balance factor, γ is the easy and difficult sample balance factor, and H and W represent the width and height of the probability plot, respectively. These are actual text region mask labels. It is a probability graph for prediction, so it corresponds to... It is the mask corresponding to pixels h and w. For the model prediction probability values ​​corresponding to pixels h and w, α is preferably set to 0.25 and γ is preferably set to 2.

[0134] The mean squared error loss (MSE) measures the difference in distribution between the model's predicted cross-union ratio (CUI) output and the true CUI.

[0135]

[0136] in It is the intersection-union ratio (IoU) of the binary image of the predicted probability map and the true mask. This is the predicted intersection-union ratio. During loss calculation in each iteration, It will output the mask based on the model's prediction. With real labels Calculated in real time according to the formula

[0137] Ultimately, this resulted in an overall loss.

[0138]

[0139] Where λ is the balance factor, which is set to 20.

[0140] Branch 2, the process of weakly supervised training:

[0141] The weakly supervised training process constructed in this invention is based on exponential average moving average and a teacher-student model. Hybrid training simultaneously performs fully supervised and weakly supervised learning during the training process. In the weakly supervised learning process, pseudo-labels are generated for unlabeled images, and these generated pseudo-labels, along with existing fully supervised labels, are used to train the detector in each iteration. This end-to-end approach avoids complex multi-stage training schemes and generates a flywheel effect during training, resulting in increasingly better performance for both the teacher and student models. This optimization process mainly includes the following two steps:

[0142] S2-2-1: Optimization of the student model: Results of the teacher model output This will serve as a pseudo-label to guide students' model learning, among which For teacher model segmentation mask image prediction, For teacher model intersection-over-comparison prediction, For student model intersection-union prediction. The supervision labels needed for the student model are obtained after binarization. To align the output of the student model With pseudo-tags This invention selects the DICE loss; for the intersection-union ratio (IoU) prediction alignment between the student and teacher models, this invention uses the consistency alignment loss (MSE). Overall optimization loss. as follows:

[0143]

[0144] in, The supervision labels needed for the student model are obtained after binarization.

[0145] S2-2-2: Optimization of the teacher model: network parameters θ of the teacher model t The network parameters θ of the student network s Updated using the exponential moving average (EMA): θ t ←αθ t +(1-α)θ s After each batch of training, the parameters of the teacher model are updated based on the parameters of the student model. α is set to 0.999.

[0146] 2. Actual interactive segmentation process

[0147] Import the original SAM model and initialize TextSAM.

[0148] Pre-training was performed on a synthetic text dataset.

[0149] Complete the training and fine-tuning on the corresponding dataset.

[0150] Human input uses weaker annotation forms, such as bounding boxes or foreground points, and inputs them into TextSAM as cues along with the image.

[0151] Image features are acquired via an image encoder. If the prompt is a foreground point, its positional embedding is first obtained through positional encoding. This positional embedding is then added to a learnable embedding to obtain the foreground point's embedding encoding. If it's a bounding box, its embedding encoding is composed of the embedding encodings of the top-left and bottom-right corner points of the bounding box. The resulting embedding encoding is concatenated with learnable variables representing the mask image prediction and the intersection-over-union (IoU) prediction to obtain a sparse embedding. The image features and prompt are then input into a mask generator, which outputs the corresponding mask value and IoU prediction.

[0152] To obtain a more accurate segmentation image.

[0153] 2. Experimental Results and Analysis

[0154] (1) Experimental Environment

[0155] Based on the high-performance graphics computing server provided by the laboratory, the specific experimental environment for this experiment is shown in Table 1:

[0156] Table 1 Experimental Environment

[0157]

[0158] (2) Experimental details

[0159] During the training phase, the AdamW optimizer and a linear learning rate Warming Up strategy were employed, along with a phased learning rate decay strategy. After Warming Up (250 iterations), the learning rate was 1e-5, decreasing by a factor of 10 at iterations 6666 and 8666, respectively. For the fine-tuning task, 10,000 iterations were set, while for the pre-training task, 100,000 iterations were set. The batch size for all experiments was set to 4. In the experiments, the ViT-Tiny-based SAM model was used as the base model, and the official pre-trained model was loaded during pre-training on the synthetic text dataset. Image preprocessing involved random cropping, random flipping, and random rotation, followed by resizing to 1024×1024. Training and inference were performed on an RTX 3090 GPU. After obtaining the inference results, the Total-Text, CTW1500, and ICDAR2015 datasets were evaluated using the evaluation scripts provided by TIoU. The evaluation will yield evaluation metrics on the test set, including the harmonic mean F-value and the harmonic mean TIoU (TIoU-F).

[0160] (3) Comparative experiments and results analysis

[0161] ① The model's generalization ability on real datasets

[0162] To verify the model's generalization ability on real datasets, this experiment first pre-trained TextSAM on the SynthText synthetic text dataset, and then directly performed inference on real datasets. The experimental results are shown in Table 2. Without fine-tuning on real datasets, when using boundaries as cues, the proposed method outperforms previous conventional scene text detection algorithms on the F-metric. On the curved datasets Total-Text and CTW1500, it surpasses the previous best detector TextBPN++ by 6.4% and 1.0%, respectively. When using text region foreground points as cues, the model's generalization performance differs significantly from that when using bounding boxes. When only foreground points are cued, the text in the SynthText dataset is quadrilateral, while the text shapes in real datasets are diverse, which limits the model's actual performance on real datasets. Furthermore, due to differences in text scale, text spacing, and text size between the model and real data, simply indicating the presence of "text" in the region is far from sufficient.

[0163] Table 2 shows the generalization ability of the model on the real dataset after training on the synthetic dataset.

[0164]

[0165]

[0166] ②Total-Text

[0167] To further verify the generalization ability of the cue model on real-world datasets, after pre-training the model on a synthetic text dataset, this invention further fine-tuned the model on a real-world scene text dataset. First, the Total-Text dataset (a curved text dataset) was used, and the experimental results are shown in Table 3. With foreground points as cues, TextSAM achieved F-values ​​and TIoU-F scores of 91.6% and 68.0% respectively on 10% fully supervised data and 90% weakly supervised data. These results surpass those of the generally best-performing scene text method, TextBPN++. On 100% fully supervised data, using bounding boxes as cues, TextSAM achieved an F-value of 99.6% and a TIoU-F score of 79.8%. Introducing scene text region cues significantly improved the model's detection performance. On 25% fully supervised data and 75% weakly supervised data, with bounding boxes as cues, TextSAM achieved F-values ​​and TIoU-F scores close to those of fully supervised training, differing by only 0.2% and 0.4% respectively. This demonstrates that the hybrid training supervision mode proposed in this invention can achieve performance comparable to training with 100% fully supervised data with a lower proportion of fully supervised data. When foreground points are used as cues, although fully supervised training gives TextSAM a significant advantage in localization performance (a 2.0% higher harmonic mean F-score), the hybrid supervised training of TextSAM is superior in the precision of predicting text contours. The 0.5% improvement in the TIoU-F-score indicates that hybrid supervised training maintains the accuracy of the model's output contours while reducing the amount of fully supervised data used, and effectively mitigates the overfitting problem on the test set.

[0168] Table 3. Detection results of the Total-Text dataset.

[0169]

[0170]

[0171] ③CTW1500

[0172] TextSAM also achieved good results on another widely used curved text dataset, CTW1500, as shown in Table 4. It's worth noting that when the fully supervised data ratio is 100%, TextSAM did not achieve a significantly better TIoU-F value. Although it maintained the best result on the harmonic mean F-index, the TIoU-F value on 100% fully supervised data was 1.6% lower (with foreground cue) and 0.8% lower (with bounding box cue) than on 50% fully supervised data. This indicates that under this training condition, the model experienced some overfitting when trained on 100% fully supervised data, thus affecting its generalization ability on the test set. Using mixed supervised training can alleviate this problem to some extent, thereby improving the model's performance on the TIoU-F index.

[0173] Table 4. Detection results of the CTW1500 dataset.

[0174]

[0175] ④ICDAR2015

[0176] ICDAR2015 is renowned for its small, dense, and multi-directional text regions, making it a highly challenging dataset. Because its annotation format is quadrilateral, which is similar in form to bounding box cueing, and because cueing models tend to classify cue regions as positive samples, the harmonic mean F fails to evaluate the performance of TextSAM. Specifically, as shown in Table 5, the harmonic mean F for bounding box cueing is consistently above 99.5%. Therefore, this invention focuses more on the TIoU-F metric. The results show that while TextSAM is designed for detecting text regions of arbitrary shapes, it also performs well in detecting quadrilateral text regions and handles small text regions effectively.

[0177] Table 5 shows the detection results from the ICDAR2015 dataset. The best results are highlighted in bold.

[0178]

[0179] (2) Ablation experiment and results analysis

[0180] ① Benefits of pre-training on the SynthText dataset

[0181] As shown in Table 6, on both the Total-Text and CTW1500 datasets, both using former sights as cues and bounding boxes as cues result in stable and significant performance improvements. On the Total-Text dataset, directly loading the pre-trained model provided by Zhang et al. for fine-tuning on Total-Text results in F-scores and TIoU-F-scores that are 2.9% and 4.2% lower (for former sights as cues), and 0.4% and 1.3% lower (for bounding boxes as cues), respectively, compared to pre-training on synthetic text datasets and then fine-tuning. This indicates that pre-training on synthetic text datasets is highly beneficial. Therefore, in the fine-tuning experiments on real datasets in this invention, weights pre-trained on synthetic text datasets are always loaded.

[0182] Table 6 shows the experimental results regarding the impact of pre-training on model performance.

[0183]

[0184] ② The impact of pseudo-label binarization threshold τ on model performance

[0185] The mask image output by the teacher model is first binarized into a binary image before being used to guide the student model. In the binarization operation, the output value of the predicted probability map is compared with a threshold τ. Pixels greater than the threshold are considered foreground points, and pixels less than the threshold are considered background points. This can be represented as... The threshold τ plays a decisive role in the generation of the final binary image. To explore the impact of the threshold τ on the model, this invention sets the proportions of fully supervised data and weakly supervised data to 50% and 50%, respectively, and sets the balance factor μ to 1.0, then conducts ablation experiments with the threshold τ. The experimental results are as follows: Figure 1 As shown, Figure 1 The left side shows the Tiou-F metric, and the right side shows the F metric. The top and bottom represent the use of foreground points or bounding boxes as cues, respectively. Different datasets use different shapes to represent these cues in the graph. Ablation experiments conducted on the Total-Text and CTW1500 datasets show that the threshold τ is crucial for training the teacher-student network; a suitable threshold τ can lead to better performance for TextSAM. The quality of pseudo-labels greatly affects the model's final learning performance. In practice, a higher τ means the pseudo-labels contain less background, but it can also lead to misclassification of the foreground region, resulting in improper handling of boundary regions. A lower τ means the pseudo-labels contain a more complete foreground region, but it can also lead to misclassification of the background region, resulting in misjudging background pixels.

[0186] from Figure 1Observations show that both excessively high and excessively low thresholds negatively impact TextSAM performance. On the two arbitrary-shape scene text datasets, Total-Text and CTW1500, the accuracy trends are similar regardless of whether foreground points or bounding boxes are used as cues. As the threshold τ increases from 0.1 to 0.3, TextSAM's performance gradually improves; however, when the threshold τ exceeds 0.3, it actually reduces model performance, even falling below the baseline. Compared to the baseline, in terms of the TIoU-F metric, bounding boxes as cues brought performance improvements of up to 1.3% and 1.6% on both datasets, while foreground points as cues brought improvements of 1.0% and 1.0% respectively. Based on these findings, this invention sets the threshold τ to 0.3 by default in the comparative experiments.

[0187] ③ The impact of the loss balance factor μ on model performance

[0188] To balance the fully supervised and weakly supervised losses across batches, this invention sets a loss balancing factor μ. To investigate the impact of μ on the model, the proportions of fully supervised and weakly supervised data were set to 50% and 50%, respectively, and the pseudo-label binarization threshold τ was set to 0.3. Then, an ablation experiment was conducted on the loss balancing factor μ. The experimental results are as follows: Figure 2 As shown, Figure 2 The left side shows the Tiou-F index, the right side shows the F index, and the top and bottom indicate whether foreground points or bounding boxes are used as cues, respectively. Different datasets are represented by different shapes in the graph. Observe... Figure 2 This shows that the model's performance fluctuates under different μ settings. Appropriate parameter settings are beneficial for hybrid supervised learning. Excessively high μ can lead to weak supervised loss. The proportion will be high, and the model will focus excessively on maintaining the same distribution as the teacher's model's output, while neglecting the loss of fully supervised learning. Insufficient learning leads to a decline in model performance, as demonstrated in the Total-Text experiments. When the balance factor μ is set to 2.0, the model's performance decreases by 1.7% (foreground points as cues) and 2.9% (bounding boxes as cues) compared to the baseline. As the balance factor μ slowly increases from 0.25 to 1.0, the model shows some performance improvement on both scene text datasets using foreground points or bounding boxes. When the hyperparameter μ changes from 1.0 to 1.25, the model using foreground points as cues on Total-Text may move from a relatively good local minimum or plateau to a worse region, resulting in a significant performance drop. Gradually increasing μ helps the model escape this unfavorable region and begin moving towards a better local minimum or plateau, thus slightly improving performance. However, overall, the model's performance is lower than the baseline, indicating that the hyperparameter μ should not exceed 1.0. Based on the above assessments, this invention sets the balance factor μ to 1.0 by default in the comparative experiments.

[0189] ④ Validation of the effectiveness of the hybrid supervision model

[0190] This invention proposes a hybrid supervised text detection model, TextSAM, based on cue segmentation, and employs the classic momentum teacher model. The teacher model is updated from the student model using an exponential average moving strategy, thus maintaining its stability. To verify the model's contribution to performance improvement, a series of experiments were conducted. Specific experimental results are shown in Tables 7 and 8. On two datasets containing curved text, under different proportions of fully supervised and weakly supervised data, and with various cue types, the presence of the teacher model consistently resulted in varying TIoU-F improvements. With 25% fully supervised data and 75% weakly supervised data, TextSAM improved the TIoU-F values ​​by 1.3% and 1.6% on the Total-Text and CTW1500 datasets, respectively, reaching 79.5% and 77.7%. Compared to the performance on 100% fully supervised data, the differences were 0.3% and -0.8%, respectively. These experimental data validate the effectiveness and reliability of the hybrid supervised training model.

[0191] Table 7 shows the experimental results on Total-Text regarding the effectiveness of hybrid supervised training.

[0192]

[0193] Table 8 shows the experimental results on CTW1500 for verifying the effectiveness of hybrid supervised training.

[0194]

[0195] ⑤ Time consumption analysis

[0196] Table 9 analyzes the time consumption of the TextSAM model for inference on Total-Text. Due to the specific nature of the cue segmentation method, only one encoding is required when processing an image. Therefore, the efficiency of the cue encoder and mask is also crucial in the complete inference of an image. TextSAM, using ViT-Tiny as the image encoder, can complete the entire inference in approximately 910ms on a modern CPU and complete cue encoding and mask decoding in 65ms, which is sufficient for interactive annotation requirements. However, using a larger image encoder, such as ViT-H, requires deployment on a modern GPU to meet the inference speed requirements of interactive annotation.

[0197] Table 9 shows the inference speed analysis of each module of the model on Total-Text.

[0198]

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A hybrid supervised text detection method based on cue segmentation task, characterized in that: Includes the following steps: S1: Construct a hybrid supervised text detection model, TextSAM, which consists of three parts: an image encoder, a cue encoder, and a mask decoder. An image encoder is used to extract image features. It consists of a visual Transformer and is responsible for extracting image features. A cue encoder is used to encode different forms of cues into the same space, and its input is the cue's positional code and multiple learnable embeddings; The mask decoder is used to fuse image features and cue codes to generate the corresponding segmentation mask prediction; S2: TextSAM employs a hybrid supervised training model, and its training dataset includes a portion of fully supervised datasets. and some weakly supervised datasets The fully supervised dataset Includes images and fully supervised label set , Contains the outlines of all text regions; weakly supervised dataset Includes images And weak tags ; Hybrid supervised training has two branches: fully supervised training and weakly supervised training; it is achieved by defining a set of fully supervised input data. and weakly supervised datasets By adjusting the data ratio, a corresponding proportion of mixed supervised training can be completed; when When, that is, fully supervised training is conducted; when At that time, weak supervision training should be carried out; The hybrid supervised training includes the following steps: 1) Fully supervised training set Weakly supervised training set The number of training rounds N; the maximum number of iterations per iteration for fully supervised training and weakly supervised training are respectively , This represents the nth image and its corresponding label in the fully supervised training set. This represents the nth image and its corresponding weak label in a weakly supervised dataset. Two TextSAMs with identical structures are used, one as a student network. Another is a teacher network ; 2) Student Network initialization; 3) Teacher Network initialization, ; 4) Initialize loss , Loss under full supervision and loss under weak supervision, respectively; 5) Obtain The images and their corresponding labels in the data are grouped into a single batch; 6) Random selection One target in the image is used as the training target; 7) Obtain hints for the student model for the randomly selected training objective in 6) and enhance those hints; 8) The student model performs forward propagation to predict the labels of the randomly selected targets in 6). 9) Calculate the loss between the true label and the predicted label of the randomly selected target in 6), and add this loss to the fully supervised loss. ; 10) Determine if the fully supervised training has reached the maximum number of iterations. If you arrive If yes, then execute step 11); otherwise, return to step 6) and select a new training target. 11) Obtain The images and their corresponding weak labels are grouped into a batch; 12) Random selection One target in the image is used as the training target; 13) Obtain hints for the student model for the randomly selected training objective in 12) and enhance those hints; 14) Obtain hints from the teacher model for the randomly selected training objective in 12); 15) The student model performs forward propagation to predict the labels of the randomly selected targets in 12); 16) The teacher model performs forward propagation to predict the labels of the randomly selected targets in 12); 17) The teacher model generates pseudo-labels for the training targets randomly selected in 12), and uses these pseudo-labels as their corresponding weak labels; 18) Calculate the loss between the weak label and the predicted label of the randomly selected target in 12), and add this loss to the weak supervision loss. ; 19) Determine if the weakly supervised training has reached the maximum number of iterations. If you arrive If yes, then execute step 20); otherwise, return to step 12 and select a new training target. 20) Calculate the total loss = + µ Then, based on the current batch The parameters of the student model are updated using backpropagation with gradient descent. , For the total loss of fully supervised training, The total loss for weakly supervised training is given by μ, which is the balance factor. 21) Update the parameters of the teacher network using EMA; 22) Determine if the maximum number of iterations N has been reached. If the maximum number of iterations has been reached, then the current teacher model and student model are the optimal teacher model and optimal student model, otherwise return to 4); S3: Input an image into the trained TextSAM, and the output is the predicted label for that image.

2. The hybrid supervised text detection method based on cue segmentation task as described in claim 1, characterized in that: In 7) and 13), the method for obtaining the prompt is as follows: Random selection program randomly selects images The outline of a target region is selected, and the selected outline will be used to generate hints and mask labels simultaneously. and true segmentation map labels ; The function will be prompted to retrieve the outline of the selected target area. Calculate the center point of the text region to generate a foreground point tooltip, or calculate the smallest bounding rectangle in the text region as the bounding box tooltip.

3. The hybrid supervised text detection method based on cue segmentation task as described in claim 2, characterized in that: In 7) and 13), the method for enhancing the prompts is as follows: Enhancements to foreground point indicators: Center point offset is considered a disturbance; Data augmentation for bounding box hints: Set two types of perturbations: center point offset and scale variation.

4. The hybrid supervised text detection method based on cue segmentation task as described in claim 3, characterized in that: The total loss of fully supervised training The calculation method is as follows: The DICE loss and Focal loss are for the mask image output, while the mean squared error loss (MSE) is for the intersection-over-union prediction. DICE losses It is used to measure the difference in distribution between the model's mask output and the real mask; in These are actual text region mask labels. It is a probability graph of the prediction; Focal loss Calculated from the predicted probability map and the true mask: in It is a positive and negative sample balance factor. It is a balance factor between easy and difficult samples. and These represent the width and height of the probability plot, respectively. For the mask of pixels h and w, Predict the probability values ​​for the model at pixels h and w; The mean squared error loss (MSE) measures the difference in distribution between the model's predicted cross-union ratio (CUI) output and the actual CUI. in It is the first The intersection-union ratio (IoU) of the binary map of the predicted probability map of each sample with the true mask. It is the first Crossover union ratio predicted for each sample; Ultimately, this resulted in an overall loss. : in It is a balancing factor.

5. The hybrid supervised text detection method based on cue segmentation task as described in claim 4, characterized in that: The weak supervision loss The calculation method is as follows: in, For the first Cross-match and prediction of individual teacher models For the first Cross-polation and comparison of predictions for each sample of student models For the output of the student model, This is a pseudo-tag.

6. The hybrid supervised text detection method based on cue segmentation task as described in claim 5, characterized in that: The steps in 21) for updating the parameters of the teacher network using EMA are as follows: in For the network parameters of the teacher model, For the network parameters of the student network, is a coefficient.

Images