Unmanned aerial vehicle visual angle text detection model based on boundary adaptation

Abstract

The invention relates to the technical field of unmanned aerial vehicle visual angle text detection, and discloses an unmanned aerial vehicle visual angle text detection model based on boundary adaptation, firstly, ResNet50 is used as a backbone network, a mixed text attention mechanism is proposed, and the mixed text attention mechanism is introduced into a feature extraction module to enhance the perception ability of a text area; in addition, the invention provides a spatial feature fusion module which is used for adaptively fusing text features of different scales, so that the adaptability of the model is enhanced. And finally, a local feature extractor is introduced into a Transform of a text detail boundary iterative optimization module, so that accurate optimization and positioning of the text boundary under complex background interference are realized, and complex post-processing steps are avoided. A large number of experiments on a challenging text detection data set and a text detection data set based on the unmanned aerial vehicle verify high robustness and advanced performance of the method provided by the invention, and a solid foundation is laid for practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of drone perspective text detection, and in particular to a drone perspective text detection model based on boundary adaptation. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in daily life and professional fields, including military reconnaissance, environmental monitoring, logistics, urban planning, and disaster relief. Text detection and recognition in drone images are of great significance for enhancing the environmental perception capabilities of drones. This is because text usually contains important geographic location information, facility names, directional signs, and advertising content. Extracting this text information can significantly enhance the environmental perception capabilities of drones in complex environments, thereby better understanding the surrounding environment and improving the accuracy of autonomous decision-making and path planning. In addition, for the development of smart cities, it helps to analyze the distribution of commercial layouts and community street advertisements. However, aerial images taken from the perspective of drones have challenges such as complex background information, diverse text shapes and directions, small-scale text, and occlusion, making the text detection task particularly difficult.

[0003] Existing text detection techniques mainly target images captured in natural scenes. However, images captured by drones present unique challenges and requirements. First, drones usually capture images from an aerial perspective, resulting in text appearing in various shapes and directions. Second, drones often capture images at higher altitudes, resulting in small text sizes and significant background interference. In addition, due to the dynamic nature of drone flight, the same text area may be captured from multiple angles. Finally, due to lighting changes caused by different time of day and weather conditions, shadows and highlights may appear in the image, further increasing the complexity of text detection. Therefore, it is crucial to develop a text detection model specifically for drone perspectives. By introducing boundary adaptation technology, the model's detection capabilities in complex scenes have been effectively enhanced to meet the needs of practical applications.

[0004] Currently popular text detection methods can be roughly divided into two categories: regression-based methods and segmentation-based methods. Regression-based methods are used to predict the boundary coordinates of text boxes and do not require additional post-processing operations, thus improving computational efficiency and showing good performance in various text formats. However, these methods show instability when dealing with small-scale and dense text, and it is difficult to achieve satisfactory results when faced with text containing complex background information.

[0005] For segmentation-based text detection methods, such as Pan and DBNet, they utilize pixel-level text region masks and generally provide higher accuracy than regression-based methods. Due to the ability to capture the geometric structure of text, such methods can achieve better results when dealing with text of various shapes and orientations. In addition, segmentation methods can divide text regions into blocks, making them perform better when dealing with dense text and complex backgrounds. However, this method usually requires complex post-processing steps to assign pixel groups to text regions, which requires a lot of labeled data and computing resources.

[0006] These methods usually rely on convolutional neural networks (CNNs), but CNNs often ignore the long-range dependencies and global spatial relationships between texts, making them sensitive to noisy regions in the text. However, in the task of text detection from a drone's perspective, texts may appear in various scales, orientations, and shapes, and will be affected by complex background interference. Therefore, global features and long-range dependencies are crucial for accurate text detection. In addition, commonly used CNN backbone networks, such as ResNet and VGG, provide coarse-grained high-resolution features, which are useful for large-scale text detection but not conducive to detecting small-scale text instances.

[0007] Therefore, it is necessary to provide a drone perspective text detection model based on boundary adaptation to solve the above technical problems. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a drone perspective text detection model based on boundary adaptation.

[0009] The drone perspective text detection model based on boundary adaptation provided by the present invention includes the following components:

[0010] A hybrid text attention mechanism (HTAM) to enhance the perception of text regions in the feature extraction stage;

[0011] A spatial feature fusion module (SFFM) is used to adaptively fuse text features of different scales and effectively integrate the output features of low-level semantics and high-level semantics, thereby enhancing the representation ability of features, enriching semantic information, and ultimately improving the model's understanding of image content and perception of text of different scales;

[0012] A Text Detail Transformer (TDT), including a Local Feature Extractor (LFE), is used for an iterative refinement process to optimize text boundaries.

[0013] Preferably, the hybrid text attention mechanism (HTAM) is divided into two parts: a channel attention mechanism and a spatial attention mechanism to reduce detection omissions caused by perspective changes, lighting shadows and occlusions.

[0014] Preferably, the spatial feature fusion module (SFFM) realizes the fusion of high-level and low-level features through weighted feature maps, thereby enhancing the model's detection capability for texts of different scales.

[0015] Preferably, the text detail transformer (TDT) improves the model's ability to extract local information from feature maps by introducing a local feature extractor (LFE) in the Transformer block, thereby optimizing the refinement of text boundaries.

[0016] Preferably, the channel attention mechanism introduces an efficient channel attention aggregation mechanism (A-ECA): by performing a maximum pooling operation on the input feature map, detailed texture features related to the text boundary are extracted, thereby enhancing the perception of the text boundary; at the same time, average pooling is used to capture the overall information of the image area, which facilitates the understanding of the overall image structure and background, which are contextual features related to the target; therefore, these two processing methods are applied to the input feature map at the same time; after processing, the two generated feature maps are connected, and the spatial information of the feature map is aggregated by merging the average pooling features with the maximum pooling features.

[0017] Preferably, the channel attention mechanism also introduces a local cross-channel interaction strategy and an adaptive one-dimensional convolution structure to achieve more comprehensive cross-channel information exchange. Through network learning, different weights corresponding to different channels on the feature map are obtained, thereby providing more accurate attention information along the channel dimension.

[0018] The present invention proposes a method for drone perspective text detection using the above-mentioned drone perspective text detection model based on boundary adaptation, and the method comprises the following steps:

[0019] Extract image features using hybrid text attention mechanism (HTAM);

[0020] Multi-scale features are fused through the spatial feature fusion module (SFFM);

[0021] A Text Detail Transformer (TDT) is used to iteratively optimize text boundaries to improve the detection accuracy.

[0022] Compared with related technologies, the drone perspective text detection model based on boundary adaptation provided by the present invention has the following beneficial effects:

[0023] This paper innovatively creates a dataset for text detection from drone perspective.

[0024] The hybrid text attention mechanism (HTAM) proposed in this paper targets the text features from the perspective of drones, and improves the accuracy of the model under complex backgrounds, diverse perspectives and low contrast conditions; this method improves the training convergence and the robustness of the model, and significantly improves the text detection performance from the perspective of drones.

[0025] Aiming at the common small-scale text problem under the perspective of drones, this paper proposes an innovative spatial feature fusion module (SFFM). By integrating feature maps from different receptive fields, the method of this paper shows significant effectiveness on a drone-perspective text detection dataset containing a large number of small-scale text instances.

[0026] By introducing a local feature extractor (LFE) into the Transformer-based iterative boundary optimization module, the present invention enhances its ability to extract local information from feature maps. This improvement reduces the interference of complex backgrounds in drone-view images and significantly improves the effect of text boundary optimization.

[0027] Extensive experiments on private drone datasets and public datasets demonstrate that the proposed method achieves advanced levels in performance and efficiency and shows good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The invention uses the text detail boundary iterative optimization module to refine the rough text boundary. (a) The backbone network and the boundary proposal module generate the rough text boundary. (b) The text detail boundary iterative optimization module is used to iteratively refine the rough text boundary, thereby obtaining a fine text boundary map.

[0029] Figure 2 This is a framework diagram of the drone perspective text detection model based on boundary adaptation proposed in the present invention;

[0030] Figure 3 is the backbone network diagram for feature extraction;

[0031] Figure 4 It is the structural diagram of HTAM;

[0032] Figure 5 It is the structural diagram of SFFM;

[0033] Figure 6 This is the encoder structure diagram of TDT. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0035] This paper proposes the DADNet framework for text detection from drone perspective, which fully exploits the advantages of Transformer and CNN. Specifically, our model consists of three main components: feature extraction backbone network, coarse bounding box generation module, and boundary refinement in boundary iterative optimization module. DADNet improves model performance in the following three aspects:

[0036] (1) A hybrid text attention mechanism (HTAM) is introduced, which combines the spatial and channel attention mechanisms to give the model position perception and feature selection capabilities. This mechanism reduces detection misses caused by perspective changes, lighting shadows, and occlusions, and enhances the model’s robustness and ability to perceive text features.

[0037] (2) To address the problem of text information of different scales in drone-view images, a spatial feature fusion module (SFFM) is designed. This module integrates the features of low-level language output and high-level semantic output, and realizes the fusion of high-level and low-level features. The weighted features calculated by this module are similar to the spatial attention mechanism, which promotes the selection and integration of features.

[0038] (3) A text detail transformer (TDT) is designed in the boundary iterative optimization module. By introducing a local feature extractor (LFE) in the original Transformer block, the boundary information of the text region is further enhanced. Specifically, by using LFE to process the feature map to obtain local information, better text region boundary features can be obtained, which has a positive impact on the boundary optimization of text features. The results are shown in the following example. Figure 1 shown.

[0039] To verify the effectiveness of DADNet, we collected images from drone datasets such as VisDrone2019 and UAVid, screened the text data from the drone perspective, annotated them, and created a drone text dataset for drone perspective text detection. In addition, we further evaluated the performance of the model on challenging text detection datasets such as Total-Text and CTW1500. Our model achieved state-of-the-art results on these datasets.

[0040] The UAV perspective text detection model framework based on boundary adaptation proposed in this paper is as follows Figure 2 As shown in the figure, ResNet50 is used as the backbone network for feature extraction. In order to enhance the model's sensitivity to text features, a high-order attention mechanism (HTAM) is introduced. Figure 3Our backbone network is illustrated. A spatial feature fusion module is adopted based on a multi-layer feature fusion strategy to maintain spatial resolution and make full use of information at multiple levels. The extracted features are then passed to a boundary proposal module, which generates a rough text boundary proposal for each text region. Each boundary proposal consists of N control points, representing a potential text instance. Using this acquired prior information, the text boundary detail iteration module refines the text boundary. Finally, an accurate text boundary is obtained.

[0041] The following will specifically explain the hybrid text attention mechanism (HTAM), spatial feature fusion module (SFFM) and text detail transformer (TDT):

[0042] 1. Hybrid Text Attention Mechanism (HTAM):

[0043] The application of attention mechanism in deep learning is of great significance. In computer vision tasks, by allowing the model to assign different attention weights to different regions of the image, the model can better focus on the region of interest, thereby improving performance and generalization. This leads to improved performance and adaptability.

[0044] The use of attention mechanism in text detection from drone perspective shows positive effects for the following reasons: First, the input data of this task usually contains complex natural scenes with a lot of interference, such as complex background, occlusion, noise, changes in natural lighting, and text at different angles. By adopting the attention mechanism, the model is able to selectively focus on important image areas related to text and ignore other parts of the image. This improves the accuracy and robustness of detection. In addition, the text from the drone perspective may have significant changes in shape, size, font, and spatial arrangement of characters. The attention mechanism can help the model adapt to these changes and better focus on and understand the text area.

[0045] like Figure 4As shown in the figure, the hybrid text attention mechanism (HTAM) can be divided into two parts: channel attention mechanism and spatial attention mechanism. For the channel attention mechanism, an efficient channel attention aggregation mechanism (A-ECA) is introduced. By performing a maximum pooling operation on the input feature map, detailed texture features related to the text boundary are extracted, thereby enhancing the perception of the text boundary. At the same time, average pooling is used to capture the overall information of the image region, which is convenient for understanding the overall image structure and background, which are contextual features related to the target. Therefore, these two processing methods are applied to the input feature map at the same time. After processing, the two generated feature maps are connected to aggregate the spatial information of the feature map by merging the average pooling features with the maximum pooling features. In addition, a local cross-channel interaction strategy and an adaptive one-dimensional convolutional structure are introduced to achieve more comprehensive cross-channel information exchange. Through network learning, different weights corresponding to different channels on the feature map are obtained, thereby providing more precise attention information along the channel dimension.

[0046] The specific implementation involves inputting feature maps Decomposed along the channel dimension into and Then the decomposed feature maps are sent to the average pooling layer F avg (X a ) and the maximum pooling layer F max (X m ). For the average pooling layer, the average value within the pooling window is calculated to smooth the feature map and highlight the global information. In contrast, the maximum pooling layer selects the maximum value within the pooling window, effectively extracting edges and other significant features. Subsequently, the feature concatenation operation merges the feature maps obtained from different pooling operations along the channel dimension, thereby integrating multiple feature representations and enhancing the feature expression ability of the model. The following is And the formulaic representation of the feature map concatenation process:

[0047]

[0048]

[0049] X concat =F avg (X a )||F max (X m )

[0050] In Formula 1, H and W represent the height and width of the feature map, respectively, and X a (h,w) and X m(h,w) represents the pixel value of the position (h,w) in the feature map, and || represents the connection operation. Finally, the local channel interaction strategy and the adaptive one-dimensional convolution structure are introduced. Through the one-dimensional convolution, the model realizes comprehensive channel information exchange, thereby enhancing the feature correlation between channels. The calculation formula of A-ECA is as follows:

[0051] y c =σ[C1D k (F avg (X a )||F max (X m ))]×X

[0052] In Formula 2, σ represents the sigmoid function, F avg (X a ) represents average pooling, F max (X m ) represents maximum pooling, and C1D represents a one-dimensional convolution with k parameters. The spatial attention mechanism (SAM) introduces the relevant parts into the bottleneck attention module to reduce the spatial information loss caused by the pooling layer. In order to further preserve the feature map, the pooling layer is removed. To prevent a significant increase in parameters, we introduce depthwise separable convolution, which divides the standard convolution into two steps: depthwise convolution and pointwise convolution, thereby reducing the computational complexity of the model and making the network more lightweight. This is particularly beneficial for complex models such as attention mechanisms and helps reduce the risk of overfitting. First, a 3×3 depthwise convolution is applied to capture the feature map y c The spatial information in the image is retained while maintaining the relationship between channels, further enhancing the extraction of local features of the image. The processed feature map is subjected to point-by-point convolution for channel scaling. By reducing the number of channels, the computational complexity of the model is reduced, a lightweight feature representation is introduced, and features are further refined. Subsequently, another 3×3 depthwise convolution is applied to the feature map with reduced channels to further introduce spatial relationships. Finally, the feature map is restored using point-by-point convolution. This convolution layer corresponds to the second convolution layer and restores the channels. SAM achieves dynamic attention allocation in the image space through channel reduction and restoration and depthwise separable convolution. This operation helps the model pay more attention to the text area in the image, thereby improving the performance of the model. The calculation formula of SAM is as follows:

[0053] w s =BatchNorm(((y c *DW1)*PW1)*DW2)*PW2)×y c

[0054] In formula 3, Represents the output of channel convolution, where DW represents depthwise convolution and PW represents pointwise convolution.

[0055] The hybrid text attention mechanism combines the channel attention mechanism and the spatial attention mechanism. First, the channel attention mechanism dynamically allocates attention between different channels, emphasizing specific channel information that is important for the task, thereby enhancing the model's abstraction and representation capabilities. This helps to better capture the key features in the text detection task and ultimately improves model performance. Second, the spatial attention mechanism enables the model to focus on specific areas of the input tensor, thereby processing relevant features more intensively. Through the spatial attention mechanism, the model can effectively locate and identify text regions, thereby improving the accuracy and robustness of text detection. By making full use of the advantages of channel-level and spatial-level information, the hybrid text attention mechanism enables the model to better understand and process complex text images, thereby improving the performance and effectiveness of text detection tasks. Figure 6 A comparison of the effect of using a hybrid text attention mechanism and a baseline heatmap is shown.

[0056] 2. Spatial Feature Fusion Module (SFFM):

[0057] like Figure 5 As shown in the figure, when the feature maps are input from the backbone network to the multi-level feature fusion structure, the feature maps with smaller receptive fields are located at the bottom of the pyramid structure. In the process of backward fusion, these feature maps with smaller receptive fields are finally fused. In addition, when the number of input channels is small, the feature information with smaller receptive fields contributes relatively less to the fused feature information, resulting in less processing of feature information with smaller receptive fields and reducing their influence in the final output. However, the feature maps with larger receptive fields are located at the top of the pyramid. In the process of backward fusion, the proportion of feature maps with larger receptive fields gradually decreases. This causes the model to pay insufficient attention to feature information with smaller receptive fields, thereby missing the detection of small-scale text in some images and failing to fully detect large-scale text in other images. Therefore, a spatial feature fusion module (SFFM) is introduced into the original multi-level feature fusion structure, aiming to appropriately fuse feature maps with smaller and larger receptive fields.

[0058] SFFM is mainly aimed at effectively integrating the output features of low-level semantics and high-level semantics, thereby enhancing the representation ability of features, enriching semantic information, and ultimately improving the model's understanding of image content and perception of texts of different scales. Due to the differences in attributes such as the number of channels and feature scale, it is impossible to apply a simple weighting operation to the two sets of features. Therefore, the SFFM feature fusion module is used to fuse the two parts of features. Figure 5 As shown, the SFFM module accepts two inputs: one is the low-level feature Another is the advanced features The two parts of features are first concatenated and then processed by simple convolution operations and activation functions. Next, the spatial attention mechanism is used to capture the correlation between different regions, so as to effectively distinguish the text area and background information in complex natural scenes. This module can better handle the changes in the position and layout of text in the image, and finally obtain the final feature representation that integrates different levels of information. The calculation formula is as follows:

[0059] S=BatchNorm(Relu(Conv(concate([X sp ,X cp ]))))

[0060] F=(S×SpatialAttentioN(s))+S

[0061] In Formula 4, concat represents the concatenation operation; Conv represents the 3×3 convolution operation; Spatial attention refers to the spatial attention module, such as Figure 5 As shown. The spatial attention mechanism in SFFM can more effectively capture the correlation between different regions in the input feature map, thereby enhancing the model's perception of local structure. Based on the results of multi-layer feature fusion, the high-level information is first fused with the middle-level features in the original network, and then fused with the low-level features and finally output. This design ensures that the SFFM module only involves a small number of convolution operations and simple element-by-element multiplication and addition operations without introducing additional computational overhead. Therefore, the model can extract text features of different scales more comprehensively and accurately when processing text detection tasks from the perspective of drones. This approach not only solves the performance problem, but also avoids additional computational costs.

[0062] 3. Text Detail Transformer (TDT):

[0063] In previous studies, control points are mainly located at the edges of text instances. For example, in DPText-DETR, points are sampled along the outer boundary of each text region. However, such features sampled at the edge usually contain many background attributes, making it difficult to focus only on the text. In the subsequent TEXTBPN++, a boundary transformer module is proposed, which iteratively predicts the offset of each vertex pointing to the text boundary based on the learned coarse boundary proposal. For each coarse boundary represented by a closed polygon, the multi-head attention mechanism in the boundary transformer compares global similarities to associate distant objects, but is relatively weak in capturing text boundaries and local structures. This weakness results in poor adaptation to text boundaries when the coarse boundary proposal is optimized in the boundary iterative optimization module. To address these challenges, we propose the Text Detail Transformer (TDT), which leverages the powerful local feature extraction capability of convolutional neural networks (CNNs) to construct a local feature extractor (LFE) and integrates it into the transformer. Its detailed architecture is shown in Figure 2. Figure 6 As shown in Figure 1. First, the input features are split along the channel dimension, and then the split components are fed into the LFE and global feature extractors respectively. Here, the high-frequency mixer consists of a group convolution operation and a 1×1 convolution operation, while the global feature extractor is implemented through a multi-head attention mechanism. In terms of technical implementation, for the input feature map It is decomposed along the channel dimension into and The formulas for local feature extractor and global feature extractor are as follows:

[0064] Y h =Conv(Relu(BatchNorm(GConv(X h ))))

[0065] Y l =MSA(AvgPooling(X l ))

[0066] In Formula 5, GConv represents 3×3 grouped convolution, Conv represents 1×1 convolution, and MSA refers to the multi-head self-attention mechanism in the transformer.

[0067] The boundary iterative optimization module adopts an encoder-decoder structure, where the encoder consists of three layers, our proposed Text Detail Transformer (TDT) has residual connections, and the decoder is a simple multi-layer perceptron (MLP), such as Figure 6 As shown. Each encoder layer can be expressed as:

[0068] X′=X×TD TransBlock(X)

[0069] In Formula 6, The feature matrix representing the boundary proposal. Each Text Detail Transformer (TDT) has a standard architecture consisting of a parallel local feature extractor, a global feature extractor, and a multi-layer perceptron network (MLP).

[0070] The main advantage of the proposed TDT is its flexibility in optimizing text boundaries, which enables it to better handle fuzzy or irregular text boundaries, thereby improving the accuracy and robustness of text detection and recognition. This fusion method effectively integrates global and local information, exploits long-range dependencies and local details, overcomes the limitations of a single method, and provides stronger feature representation capabilities for text processing tasks. It can better refine boundary information during text boundary optimization and reduce the interference of background noise. Therefore, the final boundary contour can fit the text area more accurately.

[0071] The present invention will be further described below through the following experiment:

[0072] (1) Dataset

[0073] Total-Text: This dataset contains 1,255 training images and 300 test images, providing word-level polygon annotations for texts in different orientations and irregular shapes. The dataset covers a variety of scenes, including outdoor scenery and buildings.

[0074] CTW1500: This dataset includes 1,500 natural scene images, of which 1,000 are used for training and 500 are used for testing. This dataset mainly features curved text and covers a variety of scenes such as outdoor scenery and city streets.

[0075] Drone-text: This drone-perspective text dataset contains 2,000 images, of which 1,600 are used for training and 400 for testing. The dataset is derived from data captured by drones, covering text data from drone datasets such as Visdrone2019 and UAVid, showing various text images under different urban backgrounds, perspectives, and lighting conditions. These images include not only ground text, but also text on stores and billboards. The dataset is annotated using PPOCRLabel and converted to the text annotation format of Icdar2015. Each line in the annotation file represents a text object, and the first eight numbers are coordinate information (x1, y1, x2, y2, x3, y3, x4, y4), forming a polygon represented by four clockwise points. Among them, Unprocessed-img refers to the original image in the dataset, Annotated-img refers to the annotated image, and Label refers to the annotation file of the original data.

[0076] (2) Implementation details

[0077] In the experiment, ResNet50 is used as the backbone network. The size of the input image is set to 640×640, and the model is trained in 660 cycles, and the batch size is set to 12. The initial learning rate is set to 0.001, and no pre-trained model is used, and it decays by 0.9 every 50 cycles. Adam is selected as the optimizer. Data augmentation techniques such as random rotation, random flipping, and random cropping are also applied. The experimental environment consists of Python 3.8 and PyTorch 1.7.0 framework. The training is performed on an NVIDIA RTXA6000 GPU with 48GB of memory, and the CPU used is Intel(R) Xeon(R) Gold 6226R@2.9GHz.

[0078] (3) Evaluation indicators

[0079] "R", "P" and "F" stand for recall, precision and F-measure respectively. The performance of the algorithm is evaluated by precision, recall and F-measure, which are calculated as follows:

[0080]

[0081]

[0082]

[0083] Among them, TP (True Positive) represents the true situation, FP (False Positive) represents the false positive situation, and FN (False Negative) represents the false negative situation.

[0084] Table 1: Ablation experiments of DADNet on the Total-Text dataset: hybrid text attention mechanism, spatial feature fusion module and local feature extractor. “HTAM” stands for hybrid text attention mechanism, “SFFM” stands for spatial feature fusion module and “LFE” stands for local feature extractor. “P”, “R” and “F” correspond to precision, recall and F-measure respectively.

[0085]

[0086] Table 2: Ablation experiments of DADNet on the Drone-text dataset: hybrid text attention mechanism, spatial feature fusion module and local feature extractor. “HTAM” stands for hybrid text attention mechanism, “SFFM” stands for spatial feature fusion module and “LFE” stands for local feature extractor. “P”, “R” and “F” correspond to precision, recall and F-measure respectively.

[0087]

[0088] (4) Ablation studies

[0089] We conducted ablation studies on the Total-Text and Drone-text datasets to further verify the superior performance of the text detection method from the drone perspective, as well as the effectiveness of HTAM, SFFM, and LFE. The detailed experimental results are shown in Tables 1 and 2.

[0090] As shown in Tables 1 and 2, the introduction of HTAM significantly improves the performance of both datasets. Specifically, HTAM improves the F-measure performance by 1.3% on the Total-Text dataset and by 0.6% on the Drone-text dataset. On this basis, combined with SFFM, the module is able to capture and appropriately fuse features at different levels, improving the detection of text information at various scales without significantly increasing the computational overhead. On the Total-Text dataset, SFFM contributes to a 1.6% improvement in F-measure. On the Drone-text dataset, F-measure is improved by 2.0%. Finally, the introduction of LFE integrates all modules and achieves state-of-the-art results on both datasets. On the Total-Text dataset, the precision is improved by 2.7%, the recall is improved by 0.7%, and the F-measure reaches 88.8%, marking a 1.7% improvement. On the Drone-text dataset, the precision is increased by 5.1%, the recall is increased by 5.6%, and the F-measure reaches 76.3%, marking a 5.4% improvement.

[0091] The main function of LFE is to iteratively optimize the text boundary and reduce the interference of complex background. As shown in Table 1-2, LFE improves the performance on both Total-Text and Drone-Text datasets. When LFE is introduced alone, the F-measure of both datasets is improved by 0.7%, indicating its positive impact on boundary optimization. After the introduction of HTAM and SFFM, the results with and without LFE are compared. The Total-Text dataset is improved by 0.1%, while the Drone-Text dataset is improved by 3.6%. The Total-Text dataset is mainly composed of conventional text detection tasks. After the introduction of HTAM and SFFM, the detection accuracy has been improved. Since most of the text in this dataset is horizontal and not affected by background and angle interference, boundary optimization without LFE still achieves good results. In contrast, the Drone-Text dataset contains multiple perspectives and complex backgrounds. After HTAM and SFFM improve the text detection ability, LFE significantly refines the rough boundaries of the detected complex text, bringing significant performance improvement.

[0092] F-measure statistics of the baseline model and DADNet from epoch 300 to epoch 660 on the Total-Text dataset. Evaluation was performed every 5 epochs, generating a total of 72 data points. Subsequently, a frequency distribution histogram was created to analyze the evaluation results of the model. It can be seen that the F-measure value of the model is significantly improved after integrating the various modules.

[0093] (5) Comparisons with Previous Methods

[0094] To verify the versatility of the method, we compare it with other state-of-the-art methods on three standard datasets. These datasets include two curve text benchmarks (Total-Text and CTW1500) and a dataset for detecting text from an aerial perspective. The visual results of DADNet on different datasets and its detailed numerical results are provided in Tables 3-5.

[0095] Table 3: Comparison of DADNet with other methods on the Total-Text dataset

[0096]

[0097] Total-Text: The Total-Text dataset is widely used in the field of arbitrary shape text detection. It contains a variety of text types, including multi-directional, horizontal, and curved text lines, and is ideal for verifying the ability of our method to detect arbitrary shape text. When testing on this dataset, the images are resized to be in the range of (640, 1024) while maintaining their aspect ratio. The thresholds thd and this are set to 0.3 and 0.85, respectively. The results of our method on the Total-Text dataset are shown in the last column of Table 3. By comparison, it can be clearly seen that our method performs better than the current mainstream models. Without using additional datasets, the F-measure reaches 88.8%, which is the highest performance obtained without using external data. For example, compared with the current best performing MixNet (F-measure is 87.8%), our method exceeds it by 1.0%. Table 3 provides detailed numerical results.

[0098] Table 4: Comparison of DADNet with other methods on the CTW1500 dataset

[0099]

[0100] CTW1500: Compared with the word-level annotations of Total-Text, the curved text instances in CTW1500 are annotated with polygons containing 14 vertices, which poses a greater challenge. Our method exploits SFFM and TDT to better capture long-range and local features to handle such cases. When testing on this dataset, the images are also resized to be in the range of (640,1024) while maintaining their aspect ratio, and the thresholds thd and this are set to 0.3 and 0.855, respectively. The results are shown in Table 4. It is obvious that our model achieves 88.3% accuracy, 83.9% recall, and 86.0% F-measure. Both precision and F-measure reach the best level, while recall exceeds most recent models. Compared with the previous best-performing method TextFuseNet, DADNet improves by 3.3% and 0.6% in accuracy and F-measure, respectively. These results show that our DADNet has satisfactory performance.

[0101] Table 5: Comparison of DADNet and other methods on the Drone-Text dataset.

[0102]

[0103]

[0104] Drone-text: To demonstrate the diversity of the method in drone-perspective scenarios, the model is trained using the Drone-text dataset. This dataset contains a variety of complex Chinese scenes, including text information such as store signs, billboards, buildings, road signs, and traffic signs. Due to the aerial perspective of drone imagery, the dataset contains rich text information and the same text instances from different perspectives. In addition, some text regions may be partially occluded or in shadow, resulting in reduced model perceptibility, especially for small-scale text instances. In addition, the wide field of view provided by drones also increases the sensitivity to complex background interference. Therefore, this dataset effectively simulates the diverse scenes encountered in drone-perspective settings, which poses significant challenges for text detection.

[0105] We selected mainstream models including FAST, MixNet, DBNet++, TextPMs and TextBPN++ to compare with our model. FAST designs an ultra-simple kernel function for modeling text of arbitrary shapes. Meanwhile, the TextNet network is designed for text detection. MixNet designs a novel text detection backbone network FSNet and adopts a central transformer block to exploit the 1D manifold constraint of scene text. TextPMs proposes an innovative probability map-based segmentation detection method to achieve accurate text instance detection. DBNet++ introduces an adaptive scale fusion module for scale-robust feature fusion and integrates the binarization process into the DB module in the segmentation network to optimize the segmentation network and DB module, thereby producing more accurate results. TextBPN++ systematically proposes a unified coarse-to-fine framework for the detection of text of arbitrary shapes, which enables accurate and efficient text boundary localization without post-processing.

[0106] To verify the effectiveness of the model, we used FAST, MixNet, DBNet++, TextPMs, and TextBPN++ algorithms for detection on the Drone-text dataset. The detailed detection results are analyzed in Table 5. Our model achieved 80.0% precision, 73.0% recall, and 76.3% F-measure in text detection, which represents the most advanced performance of text detection from the perspective of drones. Compared with MixNet, the F-measure value of our model is 4.4% higher, which performs significantly better in drone-based text detection.

[0107] Compared with related technologies, the drone perspective text detection model based on boundary adaptation provided by the present invention has the following beneficial effects:

[0108] This paper annotates a drone-based text detection dataset and proposes a novel text detection model for drone-based perspective. Our model makes three key improvements to the arbitrary shape text detection framework: first, the hybrid text attention mechanism enhances the perception of text regions; second, the scale feature fusion module optimizes the processing of text features at different scales; third, the text detection transformer reduces the interference of complex background by integrating local features, achieving more accurate text boundary localization without complex post-processing. Extensive experiments show that our method using the ResNet50 backbone network performs well on public datasets and drone-based text detection datasets.

[0109] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A drone-view text detection model based on boundary adaptation, characterized in that: It includes the following components: A hybrid text attention mechanism to enhance the perception of text regions in the feature extraction stage; A spatial feature fusion module is used to adaptively fuse text features of different scales and effectively integrate the output features of low-level semantics and high-level semantics, thereby enhancing the representation ability of features, enriching semantic information, and ultimately improving the model's understanding of image content and perception of texts of different scales; A text detail transformer, including a local feature extractor, is used to optimize the iterative refinement process of text boundaries.

2. The drone perspective text detection model based on boundary adaptation according to claim 1 is characterized in that: The hybrid text attention mechanism is divided into two parts: channel attention mechanism and spatial attention mechanism to reduce detection omissions caused by perspective changes, lighting shadows and occlusions.

3. The drone perspective text detection model based on boundary adaptation according to claim 1 is characterized in that: The spatial feature fusion module realizes the fusion of high-level and low-level features through weighted feature maps, thereby enhancing the model's ability to detect texts of different scales.

4. The drone perspective text detection model based on boundary adaptation according to claim 1 is characterized in that: The text detail transformer improves the model’s ability to extract local information from feature maps by introducing a local feature extractor in the Transformer block, thereby optimizing the refinement of text boundaries.

5. The boundary adaptation-based drone perspective text detection model according to claim 2 is characterized in that: The channel attention mechanism introduces an efficient channel attention aggregation mechanism: by performing a maximum pooling operation on the input feature map, detailed texture features related to the text boundary are extracted, thereby enhancing the perception of the text boundary; at the same time, average pooling is used to capture the overall information of the image area, which facilitates the understanding of the overall image structure and background, which are all contextual features related to the target; Therefore, these two processing methods are applied to the input feature map simultaneously; after processing, the two generated feature maps are concatenated to aggregate the spatial information of the feature maps by merging the average pooling features with the maximum pooling features.

6. The drone perspective text detection model based on boundary adaptation according to claim 1 is characterized in that: The channel attention mechanism also introduces a local cross-channel interaction strategy and an adaptive one-dimensional convolution structure to achieve more comprehensive cross-channel information exchange. Through network learning, different weights corresponding to different channels on the feature map are obtained, thereby providing more accurate attention information along the channel dimension.

7. A method for drone perspective text detection using the drone perspective text detection model based on boundary adaptation according to claims 1-6, the method comprising the following steps: Extract image features using hybrid text attention mechanism; Fusion of multi-scale features through spatial feature fusion module; Use the Text Detail Transformer to iteratively optimize text boundaries to improve detection accuracy.

Images