Remote sensing image directed target identification method, apparatus and device, and storage medium

Abstract

The embodiment of the invention relates to the field of remote sensing image target detection and recognition, and discloses a remote sensing image directed target recognition method and device, equipment and a storage medium. According to the method, multi-scale feature extraction, feature selection, feature alignment and feature fusion are carried out in sequence to obtain a target feature map, and rotation bounding box information is obtained based on prediction of the target feature map and comprises the width and height of a rotation bounding box, vertex offset and the type of the rotation bounding box. The vertex offset comprises offset generated when two adjacent vertexes of the rotating bounding box slide along edges corresponding to the two vertexes on the horizontal rectangular frame, and the type of the rotating bounding box comprises whether the rotating bounding box is a horizontal rectangular frame or not; and reconstructing based on the rotation bounding box information to obtain a rotation bounding box. According to the method, the problem that the fine-grained recognition capability of similar targets is insufficient is solved, and the capability of perceiving the shape and direction change of the directed target and the prediction precision of the fine-grained target and the rotation bounding box in directed target recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image target detection and recognition, and in particular to a remote sensing image directional target recognition method, apparatus, device and storage medium. Background Art

[0002] At present, the methods for remote sensing image target detection and recognition mainly include:

[0003] 1) Target Recognition Methods Based on Artificially Designed Features: Targeted recognition methods must be designed based on the characteristics of the target. Because the grayscale and texture of targets like ships and aircraft are highly similar to those in the parking area, and the background is more complex, their detection relies primarily on features such as corners and outlines. Consequently, target detection algorithms based on artificially designed features suffer from low speed, are susceptible to interference from image quality and the target's background, have poor adaptability to complex scenarios, and lack the ability to accurately and finely identify targets.

[0004] 2) Deep Learning-Based Object Recognition Methods: Existing deep learning-based object recognition methods primarily focus on detecting and identifying broad categories of targets. They are not yet well suited for practical applications in fine-grained object detection tasks, such as distinguishing between ships and aircraft. Because objects all belong to the same broad category, objects of different subcategories share high similarities in shape, grayscale, and texture, differing only in certain components. This presents significant challenges for high-precision fine-grained object recognition, and existing methods are insufficient for fine-grained recognition of similar objects. Summary of the Invention

[0005] The purpose of the present invention is to at least provide a method, device, equipment and storage medium for remote sensing image directional target recognition, which can at least solve the problem of insufficient fine-grained recognition ability of similar targets, and at least achieve the effect of improving the ability to perceive the shape and direction changes of directional targets in directional target recognition and the prediction accuracy of fine-grained targets and rotated bounding boxes.

[0006] To solve the above technical problems, at least one embodiment of the present application provides a method for identifying directional targets in remote sensing images, comprising:

[0007] Acquire target remote sensing images;

[0008] performing multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image in sequence to obtain a target feature map, and predicting rotational bounding box information based on the target feature map, wherein the rotational bounding box information includes width and height of the rotational bounding box, vertex offsets, and a rotational bounding box type, wherein the vertex offsets include offsets generated by sliding two adjacent vertices of the rotational bounding box along edges corresponding to the two vertices on a horizontal rectangular box, and the rotational bounding box type includes whether it is a horizontal rectangular box;

[0009] A rotated bounding box is reconstructed based on the rotated bounding box information.

[0010] At least one embodiment of the present application further provides a remote sensing image directional target recognition device, comprising:

[0011] Image acquisition module, used to acquire target remote sensing images;

[0012] The target recognition module is configured to sequentially perform multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image to obtain a target feature map, predict rotated bounding box information based on the target feature map, and reconstruct a rotated bounding box based on the rotated bounding box information; the rotated bounding box information includes the width and height of the rotated bounding box, the vertex offset, and the type of the rotated bounding box; the vertex offset includes the offset generated by two adjacent vertices of the rotated bounding box sliding along the edges corresponding to the two vertices on the horizontal rectangular box; and the type of the rotated bounding box includes whether it is a horizontal rectangular box.

[0013] At least one embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned remote sensing image directional target recognition method.

[0014] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned method for identifying directed targets in remote sensing images when executed by a processor.

[0015] The embodiment of the present application provides a method for identifying directional targets in remote sensing images. The method sequentially performs multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image. This method can significantly improve the ability to perceive the shape and direction changes of directional targets in directional target recognition. The target feature map obtained can predict the rotation bounding box information including the width and height of the rotation bounding box, the vertex offset, and the rotation bounding box type, thereby improving the prediction accuracy of fine-grained targets and rotation bounding boxes. Since the vertex offset is the offset generated by the sliding of two adjacent vertices of the rotation bounding box along the corresponding two sides of the horizontal rectangular box, and the rotation bounding box type indicates whether it is a horizontal rectangular box, the rotation bounding box is represented based on the sliding vertex, so that the reconstructed rotation bounding box representing the directional target in the remote sensing image is accurate and reliable.

[0016] In some optional embodiments, the remote sensing image directed target recognition method further includes:

[0017] Constructing and training a directed target recognition model of a single-stage anchor-free convolutional neural network architecture, wherein the input of the directed target recognition model includes a remote sensing image and the output of the directed target recognition model includes a rotated bounding box;

[0018] The directed target recognition model is used to perform multi-scale feature extraction, feature selection, feature alignment and feature fusion on the target remote sensing image in sequence to obtain a target feature map, predict the rotation bounding box information based on the target feature map, and reconstruct the rotation bounding box based on the rotation bounding box information.

[0019] Directed target recognition is performed through a single-stage anchor-free convolutional neural network architecture directed target recognition model, which can effectively reduce the amount of computation, improve the detection speed, and well meet the prediction accuracy requirements of actual tasks in fine-grained target recognition tasks.

[0020] In some optional embodiments, the directed target recognition model includes:

[0021] A multi-scale feature extraction module is used to extract multi-scale features from the target remote sensing image in sequence to obtain first feature maps of different scales;

[0022] a feature alignment and fusion module, configured to perform feature selection processing on the first feature maps of different scales to obtain second feature maps of different scales, perform feature alignment processing on the second feature maps of different scales to obtain third feature maps of different scales, and perform feature fusion on the second feature map and the third feature map to obtain a target feature map;

[0023] A target information prediction module, configured to predict the rotation bounding box information based on the target feature map;

[0024] The prediction result output module is used to reconstruct a rotated bounding box based on the rotated bounding box information.

[0025] By combining multi-scale feature extraction, feature alignment and fusion, the accuracy of the rotation bounding box prediction of the oriented target recognition model can be effectively improved, and the ability of the convolutional neural network to distinguish and identify similar targets can be enhanced.

[0026] In some optional embodiments, the target information prediction module includes:

[0027] A deformable convolution module, configured to adaptively adjust and correct the receptive field of the target feature map;

[0028] A convolutional layer, configured to extract features from the feature map obtained by the deformable convolution module;

[0029] The information prediction layer is used to predict the rotation bounding box information based on the feature map obtained by the convolutional layer.

[0030] By combining the deformable convolution module and the convolution layer, adaptive adjustment and correction of the receptive field of the target feature map are achieved, which can further enhance the perception of changes in target direction and shape. The deformable convolution has a flexible field of view and improves the perception of rotation and scale changes of oriented targets.

[0031] In some optional embodiments, the multi-scale feature extraction module adopts a multi-scale convolutional neural network as a backbone network, and the multi-scale feature extraction module is used to perform multi-scale feature extraction on the target remote sensing image in sequence based on the backbone network to obtain first feature maps of different scales, and the resolution of the first feature maps of different scales decreases step by step, and the channel data increases step by step.

[0032] By introducing the idea of sliding vertices to represent the rotated bounding box and feature selection, feature alignment and feature fusion into the single-stage anchor-free convolutional neural network architecture, it is possible to improve the spatial and semantic perception ability of the convolutional neural network for directional targets, improve the accuracy of the single-stage anchor-free convolutional neural network architecture for rotating bounding box prediction, and enhance the single-stage anchor-free convolutional neural network's ability to distinguish and identify similar targets.

[0033] In some optional embodiments, training the directed target recognition model includes:

[0034] Obtain remote sensing image target detection datasets HRSC2016, FGSD2021, DOSR, and MAR20;

[0035] The four remote sensing image target detection datasets are divided into training set, validation set and test set respectively;

[0036] The directed target recognition model is trained based on a training set, a validation set, and a test set to obtain a directed target capable of recognizing multiple categories and models.

[0037] The directed target recognition model of this application is trained using the open source target detection datasets HRSC2016, FGSD2021, DOSR and MAR20, which can achieve accurate detection of multiple models of various directed targets, including cars, aircraft, ships, etc.

[0038] In some optional embodiments, the type of the rotated bounding box is determined based on a hyperparameter, where the hyperparameter is determined according to an area ratio between the rotated bounding box and its minimum circumscribed horizontal rectangular box:

[0039] When the area ratio is higher than a threshold, the hyperparameter is a first value, and the first value is used to indicate that the type of the rotated bounding box is a horizontal rectangular box;

[0040] When the area ratio is not higher than a threshold, the hyperparameter is a second value, and the second value is used to indicate that the type of the rotated bounding box is not a horizontal rectangular box.

[0041] The rotating bounding box representation based on sliding vertices can convert angle predictions into distance (length) predictions. At the same time, the hyperparameters of the bounding box type are used to constrain and correct extreme points, which can effectively reduce the instability of prediction results when directly predicting angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0043] Figure 1 This is a flowchart of a method for identifying directional targets in remote sensing images provided by an embodiment of the present application;

[0044] Figure 2 Schematic diagram of a rotating bounding box representation method based on sliding vertices provided in an embodiment of the present application;

[0045] Figure 3 Schematic diagram of a convolutional neural network architecture based on a feature alignment network pyramid provided in an embodiment of the present application;

[0046] Figure 4 Schematic diagram comparing the FPN and FaPN feature fusion methods provided in the embodiments of the present application;

[0047] Figure 5 This is a schematic diagram comparing the accuracy of different training methods provided in the embodiments of the present application when quantizing the number of training times;

[0048] Figure 6 This is a schematic diagram of the change trend of the mean average precision (mAP) of different methods under different IoUs provided in the embodiments of the present application;

[0049] Figure 7 1 is a diagram showing a visual comparison of the detection results of the three methods provided in the embodiments of the present application on the HRSC2016 and DOSR datasets;

[0050] Figure 8 This is the fine-grained ship target recognition result of the method provided by the embodiment of the present application on the FGSD2021 dataset;

[0051] Figure 9 This is the fine-grained ship target recognition result of the method provided by the embodiment of the present application on the FGSD2021 dataset;

[0052] Figure 10 This is the fine-grained ship target recognition result of the method provided by the embodiment of the present application on the DOSR dataset;

[0053] Figure 11 This is the fine-grained aircraft target recognition result of the method provided in the embodiment of the present application on the dataset MAR20;

[0054] Figure 12 This is a schematic diagram of a remote sensing image directional target recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0056] To facilitate understanding of the embodiments of the present application, relevant content regarding remote sensing image target recognition is first introduced here.

[0057] With the rapid development of high-resolution remote sensing Earth observation systems, obtaining high-resolution remote sensing imagery of key regions around the world is no longer difficult, providing ample data support for intelligent interpretation of remote sensing imagery. Target detection and recognition in remote sensing imagery is a key task in remote sensing image interpretation, with significant research and application value in areas such as land and resources surveys, land use planning, geospatial intelligence acquisition, and maritime target monitoring.

[0058] 1) Traditional target recognition methods based on manually designed features: Targeted recognition methods need to be designed based on the characteristics of the target. For example, when identifying an aircraft, image segmentation or visual saliency detection algorithms are used to extract the target from relatively empty locations such as the apron. The target is then confirmed by combining features such as corner points, invariant moments, and edge contours.

[0059] 2) Object Recognition Methods Based on Deep Learning: In 2012, the first convolutional neural network (AlexNet) was developed for large-scale image classification. Its image classification accuracy significantly outperformed traditional methods using hand-crafted features. Since then, deep learning-based object detection and recognition methods have developed rapidly. Researchers have applied natural scene object detection algorithms such as Faster RCNN, RetinaNet, and CenterNet to remote sensing imagery object detection, significantly improving the speed, accuracy, and generalization of remote sensing imagery object detection and gradually making intelligent remote sensing image interpretation possible. However, these general object detection algorithms use horizontal rectangular boxes to represent object boundaries, which cannot accurately describe the boundaries of objects with large aspect ratios and arbitrary orientations, such as ships. Especially when such objects are densely distributed, the horizontal rectangular boxes of the objects will have significant overlap and interference, which can easily lead to missed detections. To predict the rotated bounding boxes of objects, researchers have proposed a series of algorithms, such as Gliding Vertex, R3Det, and S2A-Net, based on general object detection algorithms. These methods have significantly improved the detection accuracy of objects such as ships, bridges, and vehicles.

[0060] Remote sensing images are characterized by large size, complex backgrounds, and large variations in image quality and resolution. It is difficult to quickly and accurately detect and identify important typical targets such as cars, ships, and aircraft in remote sensing images. This is especially true for target interpretation tasks that require distinguishing specific types of ships and aircraft. This still mainly relies on manual interpretation, which requires extensive expert knowledge and experience and makes it difficult to rapidly process and analyze massive amounts of imagery.

[0061] 1) Traditional object recognition methods based on artificially designed features rely heavily on features such as corners and outlines when detecting and identifying ships, as the grayscale and texture of ships are highly similar to those of objects like docks, and the background is more complex. Consequently, object detection algorithms based on artificially designed features suffer from low speed, are susceptible to interference from image quality and the target's background, have poor adaptability to complex scenarios, and lack the ability to perform high-precision and refined target recognition.

[0062] 2) Existing deep learning-based object recognition methods primarily focus on detecting and identifying broad categories of targets. They are not yet well suited for practical applications involving fine-grained targets, such as those needed to distinguish specific models of ships and aircraft. This is primarily because fine-grained targets all belong to the same broad category, while targets of different subcategories share high similarities in shape, grayscale, and texture, differing only in certain components. This presents significant challenges for high-precision fine-grained object recognition. Existing object detection algorithms are not yet capable of distinguishing similar targets, and research progress in this area has been relatively slow.

[0063] In order to solve the technical problem that the above-mentioned existing target detection algorithms have insufficient ability to identify similar targets, the present invention proposes a method for identifying directed targets in remote sensing images. The implementation details of the method for identifying directed targets in remote sensing images of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0064] Example 1:

[0065] The remote sensing image directional target recognition method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 Shown, including:

[0066] Step 101: Acquire a target remote sensing image.

[0067] In step 102, multi-scale feature extraction, feature selection, feature alignment, and feature fusion are sequentially performed on the target remote sensing image to obtain a target feature map, and the rotation bounding box information is predicted based on the target feature map.

[0068] Specifically, the rotation bounding box information includes the width and height of the rotation bounding box, the vertex offset, and the rotation bounding box type. The vertex offset includes the offset generated by the sliding of two adjacent vertices of the rotation bounding box along the edges corresponding to the two vertices on the horizontal rectangular box. The rotation bounding box type includes whether it is a horizontal rectangular box.

[0069] In a specific implementation, the rotated bounding box information may also include a center point category probability and a center point offset. The center point category probability refers to the probability that a directional target represented by the center point belongs to different target types or to different models of a given target type. The center point offset refers to the offset of the center point position.

[0070] Step 103: reconstruct a rotated bounding box based on the rotated bounding box information.

[0071] The embodiment of the present application provides a method for identifying directional targets in remote sensing images. The method sequentially performs multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image. This method can significantly improve the ability to perceive the shape and direction changes of directional targets in directional target recognition. The target feature map obtained can predict the rotation bounding box information including the width and height of the rotation bounding box, the vertex offset, and the rotation bounding box type, thereby improving the prediction accuracy of fine-grained targets and rotation bounding boxes. Since the vertex offset is the offset generated by the sliding of two adjacent vertices of the rotation bounding box along the corresponding two sides of the horizontal rectangular box, and the rotation bounding box type indicates whether it is a horizontal rectangular box, the rotation bounding box is represented based on the sliding vertex, so that the reconstructed rotation bounding box representing the directional target in the remote sensing image is accurate and reliable.

[0072] In some embodiments, the remote sensing image directional target recognition method further includes:

[0073] Step 100: construct and train a directed target recognition model of a single-stage anchor-free convolutional neural network architecture, wherein the input of the directed target recognition model includes remote sensing images, and the output of the directed target recognition model includes a rotated bounding box.

[0074] Specifically, the directed target recognition model is used to perform multi-scale feature extraction, feature selection, feature alignment and feature fusion on the target remote sensing image in sequence to obtain a target feature map, and predict the rotated bounding box information based on the target feature map, and reconstruct the rotated bounding box based on the rotated bounding box information.

[0075] Directed target recognition is performed through a single-stage anchor-free convolutional neural network architecture directed target recognition model, which can effectively reduce the amount of computation, improve the detection speed, and well meet the prediction accuracy requirements of actual tasks in fine-grained target recognition tasks.

[0076] In some embodiments, the directed target recognition model includes:

[0077] A multi-scale feature extraction module is used to extract multi-scale features of the target remote sensing image in sequence to obtain first feature maps of different scales;

[0078] A feature alignment and fusion module is used to perform feature selection processing on the first feature maps of different scales to obtain second feature maps of different scales, perform feature alignment processing on the second feature maps of different scales to obtain third feature maps of different scales, and perform feature fusion on the second feature maps and the third feature maps to obtain a target feature map;

[0079] The target information prediction module is used to predict the rotation bounding box information based on the target feature map;

[0080] The prediction result output module is used to reconstruct the rotated bounding box based on the rotated bounding box information.

[0081] By combining multi-scale feature extraction, feature alignment and fusion, the accuracy of the rotation bounding box prediction of the oriented target recognition model can be effectively improved, and the ability of the convolutional neural network to distinguish and identify similar targets can be enhanced.

[0082] In the specific implementation, the multi-scale feature extraction module uses a multi-scale convolutional neural network as the backbone network. In order to improve the convolutional neural network's ability to identify similar categories of targets in remote sensing images and improve the recognition accuracy of the convolutional neural network-based target detection algorithm for fine-grained directed targets such as ships and aircraft, remote sensing image directed fine-grained target detection and recognition is performed based on sliding vertices. The idea of predicting the target as a point is adopted, and the offset of the target's rotation bounding box vertices relative to the horizontal rectangular box vertices is predicted to determine the boundary of the target's rotation bounding box; at the same time, through a single-stage anchor-free convolutional neural network architecture, the prediction accuracy of fine-grained targets and rotation boxes is improved by enhancing the convolutional neural network's ability to perceive changes in target shape and direction.

[0083] Most target detection algorithms rely on the target's tilt angle to calculate the target's rotation bounding box. However, slight changes in the angle will cause the rotation bounding box to deviate dramatically from the true position. At the same time, the periodic change of the angle leads to discontinuous loss in the training process. To overcome the problems caused by angle prediction and training, this embodiment uses the idea of sliding vertices to represent the target's rotation bounding box. The rotation bounding box representation method based on sliding vertices is as follows: Figure 2 shown.

[0084] like Figure 2 As shown in (a), the top, right, bottom and left vertices of the target's rotation bounding box OBB are o1, o2, o3 and o4 respectively, and the four vertices of the minimum circumscribed horizontal rectangular box HBB of the rotation bounding box OBB are h1, h2, h3 and h4 respectively. From the positional relationship between the rotation bounding box OBB and the horizontal rectangular box HBB, it can be seen that when the direction and size of the rotation bounding box OBB change, the size of HBB also changes, and the four vertices of OBB slide on the four sides of HBB. Since the rectangular OBB has central symmetry, when the positions of the four vertices of the horizontal rectangular box HBB are known, it is only necessary to predict the horizontal offset g1 of vertex o1 relative to vertex h1 and the vertical offset g2 of vertex o2 relative to vertex h2 to determine the exact position of the rotation bounding box OBB. However, there is a special case, such as Figure 2As shown in (b) in the figure, when g1 reaches a minimum value (close to 0) or a maximum value (close to the width of the horizontal rectangular box HBB), the target's rotated bounding box approaches a horizontal rectangular box. Therefore, the target bounding box type hyperparameter can be introduced to determine whether the target's rotated bounding box is a horizontal box (approximately a horizontal box).

[0085] In some embodiments, the type of the rotated bounding box is determined based on a hyperparameter, which is determined according to the area ratio between the rotated bounding box and its minimum circumscribed horizontal rectangular box: when the area ratio is higher than a threshold, the hyperparameter is a first value, and the first value is used to characterize that the type of the rotated bounding box is a horizontal rectangular box; when the area ratio is not higher than the threshold, the hyperparameter is a second value, and the second value is used to characterize that the type of the rotated bounding box is not a horizontal rectangular box.

[0086] In one example, the hyperparameter α is determined using formula (1), the threshold is 0.95, the first value is 1, and the second value is 0.

[0087]

[0088] Where S OBB represents the area of the rotated bounding box OBB, S HBB Represents the area of the horizontal rectangular box HBB.

[0089] In summary, the target's rotated bounding box can be effectively reconstructed based on the center (x, y) of the target horizontal rectangular box (HBB), its width and height (w, h), the vertex offsets (g1, g2) of the OBB, and the target bounding box type hyperparameter α. The sliding vertex-based rotated bounding box representation converts angle predictions into distance (length) predictions. Furthermore, the bounding box type hyperparameters are used to constrain and correct extreme points, effectively reducing the instability of predictions that can occur when directly predicting angles.

[0090] By introducing the idea of sliding vertices to represent the rotated bounding box and feature selection, feature alignment and feature fusion into the single-stage anchor-free convolutional neural network architecture, it is possible to improve the spatial and semantic perception ability of the convolutional neural network for directional targets, improve the accuracy of the single-stage anchor-free convolutional neural network architecture for rotating bounding box prediction, and enhance the single-stage anchor-free convolutional neural network's ability to distinguish and identify similar targets.

[0091] The Gliding Vertex algorithm is based on the two-stage Faster RCNN algorithm. This algorithm has a large amount of computation and slow detection speed. Its accuracy in fine-grained target recognition tasks cannot meet the needs of actual tasks.

[0092] In one example, to solve the problem of the Gliding Vertex algorithm, the idea of sliding vertices to represent the target's rotating bounding box was introduced into the CenterNet architecture of the anchor-free target detection algorithm. At the same time, in order to improve the spatial and semantic perception ability of the convolutional neural network for directional targets, the following methods were designed: Figure 3 The convolutional neural network architecture CenterGV, shown here, is based on a feature alignment network pyramid to improve the accuracy of anchor-free object detection algorithms for predicting rotated bounding boxes and enhance the convolutional neural network's ability to distinguish and identify similar objects. This network architecture primarily includes multi-scale feature extraction, feature alignment and fusion (FAF), and target information prediction.

[0093] Remote sensing images feature a wide variety of objects, significant size variations, and complex backgrounds. The receptive field of a single-scale feature map cannot effectively cover objects of varying sizes. Therefore, using high-performance convolutional neural networks to extract and preserve information at different scales is crucial for remote sensing imagery object detection. To extract multi-scale information from images, the multi-scale feature extraction module in this example uses the multi-scale convolutional neural network Res2Net50 as its backbone. This network has demonstrated excellent performance in tasks such as remote sensing image segmentation and object detection. Res2Net consists of several residual modules. Within each residual block, hierarchical residual connections are constructed, replacing the common single 3×3 convolution kernel. This significantly increases the network's receptive field and enables the representation of multi-scale features at a finer granularity. Based on the resolution of the feature maps output by the residual modules, the Res2Net architecture can be divided into five stages, with the feature maps output by these five stages denoted as C1, C2, C3, C4, and C5, respectively. Due to the downsampling mechanism of convolutional neural networks, the multi-scale feature maps C1, C2, C3, C4, and C5 extracted from remote sensing images using the backbone network have progressively decreasing resolution and increasing channel counts. Taking an input image of 800×800×3 pixels as an example (where 3 represents the RGB channels), the sizes of the C1, C2, C3, C4, and C5 feature maps are 400×400×64 pixels, 200×200×128 pixels, 100×100×256 pixels, 50×50×512 pixels, and 25×25×2048 pixels, respectively. Low-level feature maps have high resolution and contain more positional and detailed information, but they also have a smaller receptive field, less semantic information, and more noise. High-level feature maps have low resolution and poorer perception of spatial detail, but they have a larger receptive field and stronger semantic information. Fusion of features at different scales is an important means to improve target detection performance. In this example, four feature maps, C2, C3, C4, and C5, are used for subsequent feature fusion and target detection tasks.

[0094] In order to effectively fuse the spatial and semantic information of feature maps with different resolutions, the FPN structure needs to first upsample the low-resolution feature map to the same size as the high-resolution feature map, and then add the elements at the corresponding positions of the feature map. The feature fusion method of FPN is as follows: Figure 4 As shown in (a) in . This method is simple and intuitive, but there is a phenomenon of spatial information misalignment in feature maps of different scales. The method of direct upsampling and then summing will destroy the continuity of the spatial information of the feature map to a certain extent, aggravate the information misalignment phenomenon, and damage the prediction of the target boundary and spatial position. In particular, the direction of the target in the remote sensing image is random, and the convolutional neural network's ability to perceive the target's spatial information is even more important for the accurate prediction of the boundary of a directional target. Based on the above considerations, in this example, the feature alignment and feature fusion ideas in the Feature-aligned Pyramid Network (FaPN) in the dense image prediction task are introduced into the remote sensing image fine-grained target detection task, in order to enhance the convolutional neural network's ability to perceive changes in target direction and shape, and alleviate the spatial information misalignment phenomenon between feature maps of different scales. Based on the FPN feature fusion method, FaPN introduces a feature selection module (Feature Selection Module, FSM) and a feature alignment module (Feature Alignment Module, FAM). The feature fusion method of FaPN is as follows: Figure 4 As shown in (b) in .

[0095] The feature alignment and fusion module includes:

[0096] a. Feature selection

[0097] Because feature maps of different scales have not only differences in resolution, but also differences in the number of channels. The FPN structure uses 1×1 convolution to reduce the channels of the low-resolution feature map to the same as the high-resolution feature map, which cannot selectively retain important information and suppress or discard useless information. Therefore, in the feature fusion process in this example, the feature selection module FSM is used to replace the 1×1 convolution, which can be used to enhance the important information of the feature map and accurately allocate the necessary spatial details. The core idea of ​​FSM is to use the channel attention mechanism to learn the weights of different channels of the feature map, realize the weighting of the feature map, and then add it to the original feature map through the residual connection, and finally use 1×1 convolution for further screening. The FSM implementation can be expressed as:

[0098]

[0099] u=σ(Conv 1×1 (z)) (3)

[0100] z=[z1,z2,z3,…,z D] (4)

[0101]

[0102] Where C i is the input feature map (first feature map) (i∈{2,3,4}), C i =[c1,c2,c3,…,c D ], D is the number of channels of the input feature map; is the FSM output feature map (second feature map), D′ is the number of channels of the output feature map; u is the weight of different channels of the input feature map; Conv 1×1 (·) is a convolution layer with a kernel size of 1×1; σ(·) is a sigmoid activation function; H i and W i Represents the feature map C i height and width.

[0103] b. Feature alignment

[0104] Feature alignment mainly uses deformable convolution to learn the transformation offset of pixels in feature maps of different scales, that is, to adjust the spatial feature distribution of low-resolution feature maps according to the spatial details provided by high-resolution feature maps, to achieve alignment of spatial information and semantic information of feature maps of two scales, and to alleviate the damage of feature fusion method of element-by-element addition to the prediction of target boundary information and spatial information. In the feature alignment process, first C i+1 Feature map after upsampling Feature map after FSM filtering Splicing, using the convolution layer with a convolution kernel size of 3×3 to learn the offset D of the spatial information between the two group feature maps i+1 ; Then, according to the offset D i+1 , using deformable convolution to transform the feature map Aligning spatial features to feature maps The feature alignment implementation process can be expressed as:

[0105]

[0106] Where, is the feature map after feature alignment processing, is the result after upsampling the low-resolution feature map, DConv 3×3 (·) Deformable convolution with kernel size of 3×3, D i+1 It is the spatial information offset between feature maps of different scales; is the high-resolution feature map after feature selection, Conv 3×3(·) represents a convolution layer with a convolution kernel size of 3×3, Concat(·) concatenates the feature maps according to the channel dimension, and Upsampling(·) is the upsampling function.

[0107] c. Feature fusion

[0108] Feature map after feature alignment The high-resolution feature map after feature selection is added and fused, and then the fused result is processed using a convolution layer with a convolution kernel size of 3×3 to eliminate the interference of noise and the feature aliasing effect during upsampling. The feature fusion process can be expressed as:

[0109]

[0110] Where, P i Represents the final fusion result of adjacent low-level feature maps and high-level feature maps. Feature maps C5, C4, and C3 are aligned and fused with the high-resolution feature map of the previous layer in sequence to obtain the final high-resolution feature map P2 for target information prediction, whose size is 1 / 4 of the input image.

[0111] In some embodiments, the target information prediction module includes:

[0112] Deformable convolution module, used to adaptively adjust and correct the receptive field of the target feature map;

[0113] Convolutional layer, used to extract features from the feature map obtained by the deformable convolution module;

[0114] The information prediction layer is used to predict the rotation bounding box information based on the feature map obtained by the convolutional layer.

[0115] By combining the deformable convolution module and the convolution layer, adaptive adjustment and correction of the receptive field of the target feature map are achieved, which can further enhance the perception of changes in target direction and shape. The deformable convolution has a flexible field of view and improves the perception of rotation and scale changes of oriented targets.

[0116] Continuing with the previous example, based on the feature map P2, we adopt the idea of predicting the target as a point and construct a detection head to predict the relevant information of the target's rotation bounding box. The detection head consists of 5 prediction branches, which predict the class probability of the target's center point respectively. Center point offset Width and height of the rotated bounding box Sliding Vertex Offset and rotated bounding box types Where W and H are the width and height of the input image, s is the downsampling factor of the input image (for example, s is 4), and K represents the category of the object to be detected. Each prediction branch consists of a deformable convolution module (for example, a 3×3 kernel size and 256 kernels) and a convolutional layer. This enables adaptive adjustment and correction of the receptive field of the feature map, further enhancing the prediction branch's ability to perceive changes in object orientation and shape.

[0117] In one example, the loss function L used to train the directed target recognition model is composed of the category probability loss L of the target center point. Q , the position offset loss of the center point L O , width and height loss L of the rotated bounding box B , the offset loss L of the sliding vertex G And the type loss L of the rotation bounding box A It consists of five parts.

[0118] The category probability loss L of the target center point Q The calculation method of is the focal loss function, that is:

[0119]

[0120] Where Y xyk ∈Q is the true value of the label, is the predicted value, (c x ,c y ) represents the position of the target center point on the original remote sensing image, δ is the standard deviation of the target rotation bounding box size, N is the number of targets, β and γ are parameters that control the degree of target contribution, and the same settings as the CenterNet algorithm are used, which are set to 2 and 4 respectively.

[0121] Rotation bounding box size loss L B for:

[0122]

[0123] Where, b=[b w ,b h ] represents the actual width and height of the horizontal rectangular box HBB, b∈B; represents the predicted value of the network,

[0124] Target center point offset loss L O for

[0125]

[0126] Where, Represents the true value of the center point offset, o∈O; Represents the predicted value of the target center point offset, s is the step size of feature map downsampling, which is 4.

[0127] Sliding vertex offset loss L G for:

[0128]

[0129] In the formula, g = [d1, d2] is the true value, is the network prediction value,

[0130] The hyperparameter α of the rotation bounding box type takes a value of 0 or 1, so it can be regarded as a two-classification problem. The loss L of the rotation bounding box type is A The cross entropy function is used for calculation, namely:

[0131]

[0132] Where α is the true value of the rotation bounding box type parameter, α∈A; is the predicted value of the network,

[0133] The total loss L can be expressed as:

[0134] L=λ1L Q +λ2L B +λ3L O +λ4L G +λ5L A (16)

[0135] Where λ1, λ2, λ3, λ4, and λ5 are set to 1.0, 0.5, 1.0, 1.0, 1.0, and 1.0 by default.

[0136] In some embodiments, training a directed object recognition model includes:

[0137] Obtain remote sensing image target detection datasets HRSC2016, FGSD2021, DOSR, and MAR20;

[0138] Divide the four remote sensing image target detection datasets into training sets, validation sets, and test sets respectively; and

[0139] The directed target recognition model is trained based on the training set, validation set and test set to obtain a directed target that can recognize multiple categories and models.

[0140] By using the open source target detection datasets HRSC2016, FGSD2021, DOSR and MAR20, the directed target recognition model is trained, which can achieve accurate detection of multiple models of various directed targets, including cars, aircraft, ships, etc.

[0141] Example 2:

[0142] In order to verify the performance of the method in the aforementioned embodiment in the remote sensing image directional target detection task and fine-grained target recognition task, this embodiment uses the open source remote sensing image target detection datasets HRSC2016, FGSD2021, DOSR and MAR20 for experimental verification.

[0143] 1) Experimental data

[0144] The HRSC2016 dataset contains 1,061 images, ranging in size from 300 × 300 pixels to 1,500 × 900 pixels. The training set, validation set, and test set contain 436, 181, and 444 images, respectively. During the experiments, the training set was used for model training, and the images in the test set were used for model performance evaluation.

[0145] The FGSD2021 dataset covers multiple ports worldwide and annotates 20 categories of ships. The dataset contains 636 images, including 424 for training and 212 for testing, with a resolution of approximately 1 meter. Because the dataset includes many large images covering the entire port, the images are sliced into 800×800 pixel slices (with a 200 pixel overlap between slices) for training and prediction.

[0146] The DOSR dataset contains 1,066 images, 6,127 ship objects, and 20 ship categories. The image size ranges from 600 to 1,300 pixels, and the resolution ranges from 0.5 to 2.5 meters. The number of images in the training set, validation set, and test set is 532, 223, and 320, respectively. The distribution of classes in the DOSR dataset is severely unbalanced, with some classes having fewer than 20 training samples.

[0147] The MAR20 dataset, comprising 60 airports, 3,842 high-resolution remote sensing images, 20 military aircraft models (designated A1-A20), and 22,341 aircraft targets, is currently the largest dataset for military aircraft recognition using remote sensing imagery. The different aircraft models in this dataset exhibit significant similarities in appearance, posing a significant challenge to high-precision, fine-grained aircraft recognition.

[0148] 2) Experimental setup and accuracy evaluation

[0149] When training the network model, methods such as rotating and scaling images are used to enhance the diversity of training data and improve the generalization of the network model. The training parameters are set as follows: the batch size is 6, the maximum number of training times is 300; the Adam optimizer is used, the initial learning rate is set to 0.0005, and the learning rate is adjusted in an exponential decay manner, and the weight decay factor is set to 0.0005; the experimental environment uses the Pytorch deep learning framework, and the relevant experiments are completed on two RTX2080TI graphics cards with 11GB of video memory. The design of the network model of this embodiment is based on the Res2Net50 network. Therefore, when training the network model, the Res2Net50 pre-trained weights are loaded to initialize the network model parameters to speed up the convergence speed of model training. In order to evaluate the performance of the method of this embodiment in fine-grained target recognition, the average precision AP (Average Precision) is used as the accuracy evaluation index of a single category target, and the average precision mean mAP (Mean Average Precision) is used as the accuracy evaluation index of multiple categories. Among them, AP50 and mAP50 represent the average precision and mean average precision when the intersection over union (IoU) of the target prediction box and the true box is 50%, respectively. The larger the IoU, the higher the overlap between the target prediction box and its true box, and the more accurate the target position information prediction: an IoU of 0 means that the target prediction box does not intersect with the true box, and an IoU of 1 means that the target prediction box completely overlaps with its true box.

[0150] 3) Experimental analysis

[0151] In order to verify the feasibility and effectiveness of the sliding vertex representation and the convolutional neural network architecture based on feature alignment in the task of directional fine-grained target recognition, a single-class target detection verification test was conducted on the HRSC2016 dataset, and the fine-grained multi-class target recognition effect evaluation was also conducted on the DOSR dataset. First, the convolutional neural network architecture using the traditional feature pyramid is recorded as the baseline architecture (Baseline), and the method of directly predicting the tilt angle of the target rotation box on the baseline architecture is recorded as Baseline+Angle; the sliding vertex representation method introduced on the baseline architecture is recorded as Baseline+GV, which is the model innovatively adopted by the applicant in the process of designing the recognition model; the method of the present invention cleverly combines the sliding vertex representation method with the convolutional neural network architecture based on feature alignment to construct a single-stage anchor-free convolutional neural network architecture based on sliding vertices. Under the same training conditions, the above three methods are compared with the number of training times on the HRSC2016 dataset and the DOSR dataset. Figure 5As shown in the figure, (a) shows the change of accuracy with the number of training times on the HRSC2016 dataset, and (b) shows the change of accuracy with the number of training times on the DOSR dataset.

[0152] from Figure 5 It can be seen that the three methods exhibit similar accuracy curves on both datasets. That is, as the number of training iterations increases, the prediction accuracy of the model on the test set gradually improves. The proposed method achieves the highest accuracy, followed by the Baseline+GV method, and the Baseline+Angle method achieves the lowest. However, due to the different approaches to improving the network models of the three methods, their accuracy also differs significantly: the Baseline+Angle method has the lowest accuracy and exhibits significant oscillation during training. This is due to the periodicity and critical jumps of angles, which make it difficult to converge the model directly predicting the angle of the target's rotated box, resulting in inaccurate angle predictions. During the prediction process, slight errors in the angle can cause the predicted target box to deviate significantly from its true position. The Baseline+GV method, on the other hand, effectively alleviates this oscillation during model training, significantly accelerating model convergence and significantly improving final prediction performance. On the single-category ship recognition task on the HRSC2016 dataset, the sliding vertex representation improves the mean average precision (AP50) by 1.12% over the angle representation. On the multi-category ship recognition task on the DOSR dataset, the sliding vertex representation improves the mean average precision (mAP50) by 7.81%. It can be seen that, under the same network architecture, improving the prediction method of the target rotation box has a decisive improvement in the detection accuracy of directional targets, especially for the detection task of fine-grained multi-class targets. The final method adopted by this invention has a smoother convergence curve, higher target detection accuracy and fine-grained recognition accuracy than the Baseline+GV method: on the HRSC2016 dataset, the final method adopted by this invention improves the mean average precision (mAP50) of the Baseline+GV method by 1.56%; on the DOSR dataset, the final method adopted by this invention improves the mean average precision (mAP50) of the Baseline+GV method by 13.38%. This is due to further improvements to the network architecture: a multi-scale feature fusion method based on feature alignment is used to align the spatial and semantic information of the feature map, and deformable convolution is used to achieve effective perception of the diverse changes in target orientation, thereby achieving more accurate prediction of targets with large aspect ratios.

[0153] In order to further compare the differences in the prediction accuracy of the target rotation bounding box among different methods, the average precision mAP change trend of different methods under different IoU is shown as follows: Figure 6As shown in the figure, (a) is the average precision mAP change trend of different methods under different IoU on the HRSC2016 dataset, and (b) is the average precision mAP change trend of different methods under different IoU on the DOSR dataset.

[0154] from Figure 6 As can be seen, as the IoU threshold increases, that is, when the evaluation criteria become more stringent, the accuracy of the three methods on both datasets shows a significant downward trend. However, overall, the performance achieved by the method finally adopted by the present invention is consistently significantly better than the other two methods: For the single-class object detection task on the HRSC2016 dataset, when the IoU threshold is 0.5, the mAPs of the method, Baseline+GV, and Baseline+Angle are 90.44%, 89.82%, and 87.47%, respectively, with little difference in accuracy. However, when the IoU threshold is 0.75, the mAPs of the three methods drop to 79.99%, 66.45%, and 42.87%, respectively, with the method finally adopted by the present invention significantly outperforming the other two methods. For the multi-class object recognition task on the DOSR dataset, when the IoU threshold is 0.75, the mAP of the method finally adopted by the present invention is 43.81%, while the mAPs of the Baseline+GV and Baseline+Angle methods are only 28.26% and 17.97%. Figure 7 The following is a visual comparison of the detection results of three methods on the HRSC2016 and DOSR datasets. The first row shows the Baseline+Angle method, and the second row shows the proposed method. As can be seen, when directly predicting the angle of the target's rotation box, the predicted boxes of many targets have significant angular deviations. This angular deviation causes the predicted rotation box to deviate from the target's true position. However, the proposed method can more accurately reconstruct the target's rotation box, and the predicted rotation box is closer to the target's true boundary. This demonstrates the proposed method's prediction advantage in detecting the bounding box of directional targets.

[0155] Experimental results on the HRSC2016 dataset:

[0156] On the HRSC2016 dataset, the performance of mainstream remote sensing image ship detection methods is compared, including GWD, KLD, RIDet-O, SLA, DAL, and GVertex. The accuracy comparison of different methods is shown in Table 1.

[0157] Table 1 Accuracy comparison of typical methods on the HRSC2016 dataset (%)

[0158]

[0159]

[0160] As shown in Table 1, our method significantly surpasses other methods in object detection accuracy on the HRSC2016 dataset, achieving 90.44%. The mAP50 of other similar methods is below 90%, demonstrating the superiority of our method in ship target detection. The GVertex method and our method in Table 1 both utilize the same rotation box representation. Compared to GVertex, our method achieves a 2.24% improvement in mAP50, primarily due to our design and optimization of the network architecture. Regarding the rotation box representation, the GWD and KLD methods convert the rotation box into a two-dimensional Gaussian distribution and then optimize the loss function to improve the prediction accuracy. They outperform methods such as DAL and SLA, which directly predict the rotation box angle, under the mAP75 and mAP85 evaluation criteria, but their accuracy remains lower than that of our method. Compared to other methods, our method maintains a significant performance advantage with increasing the IoU threshold and under stricter evaluation criteria. This means that our method exhibits greater robustness and accuracy in boundary location prediction for large aspect ratio object detection. Figure 8 Figure 3 is the detection result of the method of the present invention on the HRSC2016 dataset. It can be seen from the detection results in the figure that the method of the present invention can effectively distinguish and identify ship targets of different types and in different environments. In particular, when multiple ships are docked side by side, the bounding box predicted by the method of the present invention is also relatively consistent with the actual position of the target.

[0161] Experimental results on the FGSD2021 dataset:

[0162] Compared with the HRSC2016 dataset, FGSD2021 is a multi-category target detection and recognition dataset that contains 20 different types of ships. Some ships have a high degree of similarity in appearance, making manual interpretation and discrimination more difficult. This poses a huge challenge to the task of fine-grained, high-precision target recognition. On the FGSD2021 dataset, a comparison of the fine-grained recognition accuracy of ship targets among the mainstream methods RoI-Trans, SCRDet, CSL, DCL, R3Det, RSDet, S2A-Net, BBAVectors, CHPDet, and the method of the present invention is shown in Table 2. As can be seen from Table 2, the mAP50 of the method of the present invention reached 91.28%, an increase of 1.99% over the CHPDet method of the same type. At the same time, it also outperforms other methods in the recognition of targets in multiple categories. Figure 9This is the fine-grained ship target recognition result of the method of the present invention on the FGSD2021 dataset. Different types of ships are distinguished by colors. It can be seen from the figure that the method of the present invention can effectively distinguish most categories of ship targets, but because the division of the dataset is still not detailed enough (19 types of ships are marked, and the remaining types of ships are marked as Oth), there are still some ship categories that are confused with each other.

[0163] Table 2 Comparison of the average accuracy of the proposed method and mainstream methods on the FGSD2021 dataset (%)

[0164]

[0165]

[0166] Experimental results of DOSR dataset:

[0167] The DOSR dataset also annotates 20 categories of ships, but compared with the FGSD2021 dataset, on the one hand, the objects annotated in the DOSR dataset are smaller in size (most of the objects are small boats) and the shapes of objects of different types are more similar; on the other hand, nearly half of the objects in the DOSR dataset have less than 100 objects. The lack of a training dataset will seriously affect the detection accuracy of the objects, making it more challenging to accurately predict the position of the object bounding box and the fine-grained category of the object. On the DOSR dataset, the accuracy comparison of the mainstream methods FR-FPN-O, R2CNN, RRPN, SCRDet, RetinaNet-O, R3Det, SCRDet++, RSDet, ReDet, EIRNet and the method of the present invention in fine-grained ship recognition is shown in Table 3. It can be seen from Table 3 that among the compared methods, the EIRNet method has the best accuracy performance, with an mAP of 61.39%. This method is a two-stage method that jointly determines the rotation box of the target by predicting the pixel-level information and the angle information of the target. The average accuracy of the proposed method can reach 64.56%, which is 3.17% higher than that of EIRNet, demonstrating the advantages of the proposed method in small target and fine-grained target ship recognition tasks. Figure 10 This is the fine-grained ship target recognition result of the method of the present invention on the DOSR dataset. It can be seen that there are interferences from man-made objects such as docks and buildings around the ship targets, and the targets are small in size, diverse in direction, and closely arranged. The method of the present invention can effectively detect different types of ship targets in complex backgrounds, especially can accurately distinguish ships docked side by side.

[0168] Table 3 Comparison of the average accuracy of the proposed method and mainstream methods on the DOSR dataset (%)

[0169]

[0170] Experimental results on the MAR20 dataset:

[0171] The method proposed in this paper is a general-purpose method for detecting and recognizing oriented targets in remote sensing images. It not only achieves superior results in ship target detection tasks on datasets such as HRSC2016, but is also applicable to other types of target detection and fine-grained recognition tasks. Therefore, a validation experiment was conducted on the MAR20 aircraft dataset, and the performance of the proposed method was compared with other methods, including FCOS-O, RetinaNet-O, S2A-Net, F-RCNN-O, DHead-O, O-RCNN, RoI-Trans, and GVertex, as shown in Table 4. The MAR20 dataset annotates 20 aircraft models. Different aircraft models have high similarities, differing only in certain components. Furthermore, due to varying imaging conditions such as lighting and cloud occlusion, aircraft of the same model in different images also exhibit significant visual occlusion and morphological differences. As shown in Table 4, for the fine-grained aircraft detection task in this dataset, the average accuracy of algorithms such as FCOS-O, RetinaNet-O, and A2S-Net all fell short of our proposed method. The RoI-Trans algorithm, a two-stage object detection algorithm, achieved a mAP50 of only 82.7%. Our proposed method achieved a mAP50 of 88.8%, a 6.1% improvement over RoI-Trans. Among the 20 aircraft categories, the A13 and A15 had the lowest accuracy and the worst recognition performance. Analysis of samples from the MAR20 dataset shows that the A13 and A15 are two fighter jet models with very similar appearances, resulting in poor recognition performance for these models. Our proposed algorithm, however, improved the A13's recognition accuracy from 69.9% to 80.3% and the A15's recognition accuracy from 47.9% to 80.9%, significantly improving the convolutional neural network's ability to distinguish between highly similar objects. Since the number of A18 targets is much lower than that of other types of targets, the recognition accuracy of other algorithms for A18 is also low. The present invention improves the recognition accuracy of A18 from 74.6% to 89.0%, showing that even when the training samples are insufficient, the algorithm of the present invention still has excellent target recognition capabilities. In addition, the RoI-Trans algorithm is a two-stage target detector, which has the common disadvantages of two-stage target detectors, namely, large amount of computation, slow speed, and reliance on manual intervention in anchor frame setting. Compared with similar methods, the CenterGV algorithm proposed in the present invention is a single-stage anchor-free target detector, which greatly overcomes the problems of large amount of computation and slow speed of two-stage target detectors, and has significant progress and speed advantages. Figure 11This is the fine-grained aircraft target recognition result of the method of the present invention on the dataset MAR20. It can be seen that the method of the present invention can effectively detect and recognize targets of different scenes, different lighting conditions, and different sizes and models.

[0172] Table 4 Comparison of the average accuracy of the proposed method and mainstream methods on the MAR20 dataset (%)

[0173]

[0174] It should be noted that in order to better demonstrate the effect, Figures 5 to 11 Color illustrations are used.

[0175] The remote sensing image fine-grained target detection method provided by the present invention can implement robust directional target detection and fine-grained target recognition for high-resolution remote sensing images, and can effectively improve the model's recognition ability for fine-grained target types. This method is suitable for the recognition task of fine-grained ships and aircraft targets. The target rotation bounding box representation method can accurately describe and represent the rotation bounding box of a directional target. The position of the rotation box is determined by predicting the offset of the target rotation bounding box vertices relative to the vertices of the horizontal rectangular box, which effectively alleviates the problems of inaccurate angle prediction when directly predicting the angle and discontinuous training process loss caused by periodic changes in angles. Based on the CenteNet algorithm, a single-stage anchor-free convolutional neural network architecture based on a feature alignment network pyramid is designed, which optimizes the feature alignment and fusion methods in the single-stage anchor-free target detection algorithm, and improves the convolutional neural network's ability to perceive the shape, direction changes, and spatial and semantic perception of remote sensing image targets. The effectiveness of the method of the present invention has been verified by training and experiments on a variety of open source remote sensing image target detection datasets. Compared with other algorithms, the method of the present invention effectively improves the accuracy of the prediction of the rotated bounding box of oriented targets in remote sensing images and significantly improves the recognition accuracy of fine-grained targets in remote sensing images.

[0176] Example 3:

[0177] Another embodiment of the present application relates to a remote sensing image directional target recognition device. The implementation details of the remote sensing image directional target recognition device of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details, and is not necessary for the implementation of this solution. The schematic diagram of the remote sensing image directional target recognition device of this embodiment can be as follows: Figure 12 As shown, it includes an image acquisition module 201 , an information prediction module 202 and a bounding box reconstruction module 203 .

[0178] The image acquisition module 201 is used to acquire target remote sensing images;

[0179] The information prediction module 202 is used to perform multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image in sequence to obtain a target feature map, and predict the rotation bounding box information based on the target feature map. Specifically, the rotation bounding box information includes the width and height of the rotation bounding box, the vertex offset, and the rotation bounding box type. The vertex offset includes the offset generated by the sliding of two adjacent vertices of the rotation bounding box along the edges corresponding to the two vertices on the horizontal rectangular box. The rotation bounding box type includes whether it is a horizontal rectangular box.

[0180] The bounding box reconstruction module 203 is configured to reconstruct a rotated bounding box based on the rotated bounding box information.

[0181] In a specific implementation, the rotated bounding box information may also include a center point category probability and a center point offset. The center point category probability refers to the probability that a directional target represented by the center point belongs to different target types or to different models of a given target type. The center point offset refers to the offset of the center point position.

[0182] Performing multi-scale feature extraction, feature selection, feature alignment, and feature fusion on target remote sensing images can significantly improve the ability to perceive changes in the shape and orientation of oriented targets during oriented target recognition. The resulting target feature map can predict the rotational bounding box information, including the width and height of the rotational bounding box, vertex offsets, and rotational bounding box type, improving the prediction accuracy of fine-grained targets and rotational bounding boxes. Since the vertex offset is the offset generated by sliding two adjacent vertices of the rotational bounding box along the corresponding two sides of the horizontal rectangular box, and the rotational bounding box type indicates whether it is a horizontal rectangular box, a sliding vertex-based representation of the rotational bounding box is achieved, making the reconstructed rotational bounding box representing oriented targets in remote sensing images accurate and reliable.

[0183] In some embodiments, the remote sensing image directional target recognition device further includes:

[0184] The model building and training module is used to build and train a directed target recognition model based on a single-stage anchor-free convolutional neural network architecture. The input of the directed target recognition model includes remote sensing images, and the output of the directed target recognition model includes a rotated bounding box.

[0185] Specifically, the directed target recognition model is used to perform multi-scale feature extraction, feature selection, feature alignment and feature fusion on the target remote sensing image in sequence to obtain a target feature map, and predict the rotated bounding box information based on the target feature map, and reconstruct the rotated bounding box based on the rotated bounding box information.

[0186] Directed target recognition is performed through a single-stage anchor-free convolutional neural network architecture directed target recognition model, which can effectively reduce the amount of computation, improve the detection speed, and well meet the prediction accuracy requirements of actual tasks in fine-grained target recognition tasks.

[0187] In some embodiments, the directed target recognition model includes: a multi-scale feature extraction module, which is used to perform multi-scale feature extraction on the target remote sensing image in sequence to obtain first feature maps of different scales; a feature alignment and fusion module, which is used to perform feature selection processing on the first feature maps of different scales to obtain second feature maps of different scales, perform feature alignment processing on the second feature maps of different scales to obtain third feature maps of different scales, and perform feature fusion on the second feature map and the third feature map to obtain a target feature map; a target information prediction module, which is used to predict the rotation bounding box information based on the target feature map; and a prediction result output module, which is used to reconstruct the rotation bounding box based on the rotation bounding box information.

[0188] By combining multi-scale feature extraction, feature alignment and fusion, the accuracy of the rotation bounding box prediction of the oriented target recognition model can be effectively improved, and the ability of the convolutional neural network to distinguish and identify similar targets can be enhanced.

[0189] In the specific implementation, the multi-scale feature extraction module uses a multi-scale convolutional neural network as the backbone network. In order to improve the convolutional neural network's ability to identify similar categories of targets in remote sensing images and improve the recognition accuracy of the convolutional neural network-based target detection algorithm for fine-grained directed targets such as ships and aircraft, remote sensing image directed fine-grained target detection and recognition is performed based on sliding vertices. The idea of predicting the target as a point is adopted, and the offset of the target's rotation bounding box vertices relative to the horizontal rectangular box vertices is predicted to determine the boundary of the target's rotation bounding box; at the same time, through a single-stage anchor-free convolutional neural network architecture, the prediction accuracy of fine-grained targets and rotation boxes is improved by enhancing the convolutional neural network's ability to perceive changes in target shape and direction.

[0190] By introducing the idea of sliding vertices to represent the rotated bounding box and feature selection, feature alignment and feature fusion into the single-stage anchor-free convolutional neural network architecture, it is possible to improve the spatial and semantic perception ability of the convolutional neural network for directional targets, improve the accuracy of the single-stage anchor-free convolutional neural network architecture for rotating bounding box prediction, and enhance the single-stage anchor-free convolutional neural network's ability to distinguish and identify similar targets.

[0191] In one example, the idea of sliding vertices to represent the target's rotating bounding box was introduced into the CenterNet architecture of the anchor-free target detection algorithm. At the same time, in order to improve the spatial and semantic perception ability of the convolutional neural network for directional targets, the following methods were designed: Figure 3The convolutional neural network architecture CenterGV, shown here, is based on a feature alignment network pyramid to improve the accuracy of anchor-free object detection algorithms for predicting rotated bounding boxes and enhance the convolutional neural network's ability to distinguish and identify similar objects. This network architecture primarily includes multi-scale feature extraction, feature alignment and fusion (FAF), and target information prediction.

[0192] Remote sensing images feature a wide variety of objects, significant size variations, and complex backgrounds. The receptive field of a single-scale feature map cannot effectively cover objects of varying sizes. Therefore, using high-performance convolutional neural networks to extract and preserve information at different scales is crucial for remote sensing imagery object detection. To extract multi-scale information from images, the multi-scale feature extraction module in this example uses the multi-scale convolutional neural network Res2Net50 as its backbone. This network has demonstrated excellent performance in tasks such as remote sensing image segmentation and object detection. Res2Net consists of several residual modules. Within each residual block, hierarchical residual connections are constructed, replacing the common single 3×3 convolution kernel. This significantly increases the network's receptive field and enables the representation of multi-scale features at a finer granularity. Based on the resolution of the feature maps output by the residual modules, the Res2Net architecture can be divided into five stages, with the feature maps output by these five stages denoted as C1, C2, C3, C4, and C5, respectively. Due to the downsampling mechanism of convolutional neural networks, the multi-scale feature maps C1, C2, C3, C4, and C5 extracted from remote sensing images using the backbone network have progressively decreasing resolution and increasing channel counts. Taking an input image of 800×800×3 pixels as an example (where 3 represents the RGB channels), the sizes of the C1, C2, C3, C4, and C5 feature maps are 400×400×64 pixels, 200×200×128 pixels, 100×100×256 pixels, 50×50×512 pixels, and 25×25×2048 pixels, respectively. Low-level feature maps have high resolution and contain more positional and detailed information, but they also have a smaller receptive field, less semantic information, and more noise. High-level feature maps have low resolution and poorer perception of spatial detail, but they have a larger receptive field and stronger semantic information. Fusion of features at different scales is an important means to improve target detection performance. In this example, four feature maps, C2, C3, C4, and C5, are used for subsequent feature fusion and target detection tasks.

[0193] In order to effectively fuse the spatial and semantic information of feature maps with different resolutions, the FPN structure needs to first upsample the low-resolution feature map to the same size as the high-resolution feature map, and then add the elements at the corresponding positions of the feature map. The feature fusion method of FPN is as follows: Figure 4As shown in (a) in . This method is simple and intuitive, but there is a phenomenon of spatial information misalignment in feature maps of different scales. The method of direct upsampling and then summing will destroy the continuity of the spatial information of the feature map to a certain extent, aggravate the information misalignment phenomenon, and damage the prediction of the target boundary and spatial position. In particular, the direction of the target in the remote sensing image is random, and the convolutional neural network's ability to perceive the target's spatial information is even more important for the accurate prediction of the boundary of a directional target. Based on the above considerations, in this example, the feature alignment and feature fusion ideas in the Feature-aligned Pyramid Network (FaPN) in the dense image prediction task are introduced into the remote sensing image fine-grained target detection task, in order to enhance the convolutional neural network's ability to perceive changes in target direction and shape, and alleviate the spatial information misalignment phenomenon between feature maps of different scales. Based on the FPN feature fusion method, FaPN introduces a feature selection module (Feature Selection Module, FSM) and a feature alignment module (Feature Alignment Module, FAM). The feature fusion method of FaPN is as follows: Figure 4 As shown in (b) in .

[0194] The feature alignment and fusion module includes:

[0195] a. Feature selection

[0196] Because feature maps of different scales have not only differences in resolution, but also differences in the number of channels. The FPN structure uses 1×1 convolution to reduce the channels of the low-resolution feature map to the same as the high-resolution feature map, which cannot selectively retain important information and suppress or discard useless information. Therefore, in the feature fusion process in this example, the feature selection module FSM is used to replace the 1×1 convolution, which can be used to enhance the important information of the feature map and accurately allocate the necessary spatial details. The core idea of ​​FSM is to use the channel attention mechanism to learn the weights of different channels of the feature map, realize the weighting of the feature map, and then add it to the original feature map through the residual connection, and finally use 1×1 convolution for further screening. The FSM implementation can be expressed as:

[0197]

[0198] u=σ(Conv 1×1 (z)) (3)

[0199] z=[z1,z2,z3,…,z D ] (4)

[0200]

[0201] Where C iis the input feature map (first feature map) (i∈{2,3,4}), C i =[c1,c2,c3,…,c D ], D is the number of channels of the input feature map; Output feature map for FSM (second feature map), D′ is the number of channels of the output feature map; u is the weight of different channels of the input feature map; Conv 1×1 (·) is a convolution layer with a kernel size of 1×1; σ(·) is a sigmoid activation function; H i and W i Represents the feature map C i height and width.

[0202] b. Feature alignment

[0203] Feature alignment mainly uses deformable convolution to learn the transformation offset of pixels in feature maps of different scales, that is, to adjust the spatial feature distribution of low-resolution feature maps according to the spatial details provided by high-resolution feature maps, to achieve alignment of spatial information and semantic information of feature maps of two scales, and to alleviate the damage of feature fusion method of element-by-element addition to the prediction of target boundary information and spatial information. In the feature alignment process, first C i+1 Feature map after upsampling Feature map after FSM filtering Splicing, using the convolution layer with a convolution kernel size of 3×3 to learn the offset D of the spatial information between the two group feature maps i+1 ; Then, according to the offset D i+1 , using deformable convolution to transform the feature map Aligning spatial features to feature maps The feature alignment implementation process can be expressed as:

[0204]

[0205] Where, is the feature map after feature alignment processing, is the result after upsampling the low-resolution feature map, DConv 3×3 (·) Deformable convolution with kernel size of 3×3, D i+1 It is the spatial information offset between feature maps of different scales; is the high-resolution feature map after feature selection, Conv 3×3 (·) represents a convolution layer with a convolution kernel size of 3×3, Concat(·) concatenates the feature maps according to the channel dimension, and Upsampling(·) is the upsampling function.

[0206] c. Feature fusion

[0207] Feature map after feature alignment The high-resolution feature map after feature selection is added and fused, and then the fused result is processed using a convolution layer with a convolution kernel size of 3×3 to eliminate the interference of noise and the feature aliasing effect during upsampling. The feature fusion process can be expressed as:

[0208]

[0209] Where, P i Represents the final fusion result of adjacent low-level feature maps and high-level feature maps. Feature maps C5, C4, and C3 are aligned and fused with the high-resolution feature map of the previous layer in sequence to obtain the final high-resolution feature map P2 for target information prediction, whose size is 1 / 4 of the input image.

[0210] In some embodiments, the target information prediction module includes:

[0211] Deformable convolution module, used to adaptively adjust and correct the receptive field of the target feature map;

[0212] Convolutional layer, used to extract features from the feature map obtained by the deformable convolution module;

[0213] The information prediction layer is used to predict the rotation bounding box information based on the feature map obtained by the convolutional layer.

[0214] By combining the deformable convolution module and the convolution layer, adaptive adjustment and correction of the receptive field of the target feature map are achieved, which can further enhance the perception of changes in target direction and shape. The deformable convolution has a flexible field of view and improves the perception of rotation and scale changes of oriented targets.

[0215] Continuing with the previous example, based on the feature map P2, we adopt the idea of predicting the target as a point and construct a detection head to predict the relevant information of the target's rotation bounding box. The detection head consists of 5 prediction branches, which predict the class probability of the target's center point respectively. Center point offset Width and height of the rotated bounding box Sliding Vertex Offset and rotated bounding box types Where W and H are the width and height of the input image, s is the downsampling factor of the input image (for example, s is 4), and K represents the category of the object to be detected. Each prediction branch consists of a deformable convolution module (for example, a 3×3 kernel size and 256 kernels) and a convolutional layer. This enables adaptive adjustment and correction of the receptive field of the feature map, further enhancing the prediction branch's ability to perceive changes in object orientation and shape.

[0216] In one example, the loss function L used to train the directed target recognition model is composed of the category probability loss L of the target center point. Q , the position offset loss of the center point L O , width and height loss L of the rotated bounding box B , the offset loss L of the sliding vertex G And the type loss L of the rotation bounding box A It consists of five parts.

[0217] The category probability loss L of the target center point Q The calculation method of is the focal loss function, that is:

[0218]

[0219] Where Y xyk ∈Q is the true value of the label, is the predicted value, (c x ,c y ) represents the position of the target center point on the original remote sensing image, δ is the standard deviation of the target rotation bounding box size, N is the number of targets, β and γ are parameters that control the degree of target contribution, and the same settings as the CenterNet algorithm are used, which are set to 2 and 4 respectively.

[0220] Rotation bounding box size loss L B for:

[0221]

[0222] Where, b=[b w ,b h ] represents the actual width and height of the horizontal rectangular box HBB, b∈B; represents the predicted value of the network,

[0223] Target center point offset loss L O for

[0224]

[0225] Where, Represents the true value of the center point offset, o∈O; Represents the predicted value of the target center point offset, s is the step size of feature map downsampling, which is 4.

[0226] Sliding vertex offset loss L G for:

[0227]

[0228] In the formula, g = [d1, d2] is the true value, is the network prediction value,

[0229] The hyperparameter α of the rotation bounding box type takes a value of 0 or 1, so it can be regarded as a two-classification problem. The loss L of the rotation bounding box type is A The cross entropy function is used for calculation, namely:

[0230]

[0231] Where α is the true value of the rotation bounding box type parameter, α∈A; is the predicted value of the network,

[0232] The total loss L can be expressed as:

[0233] L=λ1L Q +λ2L B +λ3L O +λ4L G +λ5L A (16)

[0234] Where λ1, λ2, λ3, λ4, and λ5 are set to 1.0, 0.5, 1.0, 1.0, 1.0, and 1.0 by default.

[0235] In some embodiments, training a directed target recognition model includes: obtaining remote sensing image target detection data sets HRSC2016, FGSD2021, DOSR and MAR20; dividing the four remote sensing image target detection data sets into training sets, validation sets and test sets respectively; and training the directed target recognition model based on the training set, validation set and test set to obtain a directed target recognition model that can recognize multiple categories and models.

[0236] By using the open source target detection datasets HRSC2016, FGSD2021, DOSR and MAR20, the directed target recognition model is trained, which can achieve accurate detection of multiple models of various directed targets, including cars, aircraft, ships, etc.

[0237] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0238] Example 4:

[0239] Another embodiment of the present application relates to an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the remote sensing image directional target recognition method in the above-mentioned embodiments.

[0240] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0241] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0242] Embodiment 5:

[0243] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for identifying directional targets in remote sensing images.

[0244] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0245] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for identifying directed targets in remote sensing images, characterized in that: include: Acquire target remote sensing images; performing multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image in sequence to obtain a target feature map, and predicting rotational bounding box information based on the target feature map, wherein the rotational bounding box information includes width and height of the rotational bounding box, vertex offsets, and a rotational bounding box type, wherein the vertex offsets include offsets generated by sliding two adjacent vertices of the rotational bounding box along edges corresponding to the two vertices on a horizontal rectangular box, and the rotational bounding box type includes whether it is a horizontal rectangular box; A rotated bounding box is reconstructed based on the rotated bounding box information.

2. The method for identifying directed targets in remote sensing images according to claim 1, wherein: Also includes: Constructing and training a directed target recognition model of a single-stage anchor-free convolutional neural network architecture, wherein the input of the directed target recognition model includes a remote sensing image and the output of the directed target recognition model includes a rotated bounding box; The directed target recognition model is used to perform multi-scale feature extraction, feature selection, feature alignment and feature fusion on the target remote sensing image in sequence to obtain a target feature map, predict the rotation bounding box information based on the target feature map, and reconstruct the rotation bounding box based on the rotation bounding box information.

3. The method for remote sensing image directed target recognition according to claim 2, wherein the directed target recognition model comprises: A multi-scale feature extraction module is used to extract multi-scale features from the target remote sensing image in sequence to obtain first feature maps of different scales; a feature alignment and fusion module, configured to perform feature selection processing on the first feature maps of different scales to obtain second feature maps of different scales, perform feature alignment processing on the second feature maps of different scales to obtain third feature maps of different scales, and perform feature fusion on the second feature map and the third feature map to obtain a target feature map; A target information prediction module, configured to predict the rotation bounding box information based on the target feature map; The prediction result output module is used to reconstruct a rotated bounding box based on the rotated bounding box information.

4. The method for identifying directed targets in remote sensing images according to claim 3, wherein the target information prediction module comprises: A deformable convolution module, configured to adaptively adjust and correct the receptive field of the target feature map; A convolutional layer, configured to extract features from the feature map obtained by the deformable convolution module; The information prediction layer is used to predict the rotation bounding box information based on the feature map obtained by the convolutional layer.

5. According to the remote sensing image directed target recognition method according to claim 3, the multi-scale feature extraction module adopts a multi-scale convolutional neural network as the backbone network, and the multi-scale feature extraction module is used to perform multi-scale feature extraction on the target remote sensing image in sequence based on the backbone network to obtain first feature maps of different scales, and the resolution of the first feature maps of different scales decreases step by step, and the channel data increases step by step.

6. The method for identifying directed targets in remote sensing images according to claim 2, wherein: Training the directed target recognition model includes: Obtain remote sensing image target detection datasets HRSC2016, FGSD2021, DOSR, and MAR20; The four remote sensing image target detection datasets are divided into training set, validation set and test set respectively; The directed target recognition model is trained based on a training set, a validation set, and a test set to obtain a directed target capable of recognizing multiple categories and models.

7. The method for identifying directed targets in remote sensing images according to claim 1, wherein: The type of the rotated bounding box is determined based on a hyperparameter, which is determined by the area ratio between the rotated bounding box and its minimum circumscribed horizontal rectangular box: When the area ratio is higher than a threshold, the hyperparameter is a first value, and the first value is used to indicate that the type of the rotated bounding box is a horizontal rectangular box; When the area ratio is not higher than a threshold, the hyperparameter is a second value, and the second value is used to indicate that the type of the rotated bounding box is not a horizontal rectangular box.

8. A remote sensing image directional target recognition device, characterized in that: include: Image acquisition module, used to acquire target remote sensing images; The target recognition module is configured to sequentially perform multi-scale feature extraction, feature selection, feature alignment, and feature fusion on the target remote sensing image to obtain a target feature map, predict rotated bounding box information based on the target feature map, and reconstruct a rotated bounding box based on the rotated bounding box information; the rotated bounding box information includes the width and height of the rotated bounding box, the vertex offset, and the type of the rotated bounding box; the vertex offset includes the offset generated by two adjacent vertices of the rotated bounding box sliding along the edges corresponding to the two vertices on the horizontal rectangular box; and the type of the rotated bounding box includes whether it is a horizontal rectangular box.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the remote sensing image directional target recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying directed targets in remote sensing images according to any one of claims 1 to 7 is implemented.

Images