Sar target recognition method, device and equipment fusing visual salient features and medium

Abstract

The application relates to a SAR target recognition method, device, equipment and medium fusing visual significant features. A significant binary image of a SAR image is obtained according to prior information, image feature extractors and significant feature extractors based on multilayer residual convolution are used to extract features of the SAR image and the significant binary image, and deep feature vectors and significant visual feature vectors are obtained. An optimized visual attention classifier is used to encode the feature vectors, multi-head attention fusion is performed, cross attention coding of the deep features by the significant visual features is realized, feature screening is realized, and a target recognition result is obtained. The method can be used for accurate target recognition of SAR images with complex backgrounds.

Description

Technical Field

[0001] This application relates to the field of SAR target automatic identification technology, and in particular to a SAR target identification method, apparatus, device and medium that integrates visual salient features. Background Technology

[0002] Synthetic Aperture Radar (SAR), as an active microwave imaging radar, possesses all-weather, all-day observation capabilities. This advantage makes it highly valuable in many remote sensing fields and has garnered widespread attention in recent years. Ship target identification is an important application area of ​​SAR remote sensing.

[0003] SAR ship target recognition methods are mainly divided into two categories: traditional methods and deep learning methods. Traditional methods, which identify ship targets based on human-designed features, have strong interpretability due to the statistical characteristics of the target image and the physical characteristics of the real target. However, they also have significant limitations: human-designed features do not fully characterize the target, and their design process is cumbersome and time-consuming; the errors between human-designed features and traditional classification algorithms are large as the target's scene changes, resulting in poor generalization ability and limiting the practical application scope of traditional methods.

[0004] While deep learning methods have achieved good results in SAR target recognition, automatic identification of ships presents challenges. Real-world applications frequently involve ships docked or densely packed together, resulting in complex backgrounds in SAR image slices. However, such slices are scarce in existing SAR ship recognition datasets. Most targets in existing datasets are on open sea surfaces, with only a few images containing near-shore objects or other ships in the background. Network models trained on these datasets perform well in identifying isolated ship targets on open sea surfaces, but when processing ship target slices with complex backgrounds, they face two main problems: firstly, the model lacks the ability to actively distinguish between the ship to be identified and the slice background, leading to features in the slice that do not belong to the ship interfering with the classification results; secondly, SAR images are superimposed images of different scatterers, further complicating image features and increasing the difficulty of classification. Summary of the Invention

[0005] Therefore, it is necessary to provide a SAR target recognition method, apparatus, device, and medium that integrates visual salient features to accurately identify targets on SAR images with complex backgrounds, addressing the aforementioned technical problems.

[0006] A SAR target recognition method that integrates visually salient features, the method comprising:

[0007] Acquire SAR images for target identification;

[0008] Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image;

[0009] After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0010] After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained;

[0011] The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

[0012] In one embodiment, segmenting the target in the SAR image based on prior information to obtain a saliency binary map corresponding to the SAR image includes:

[0013] Based on the prior information of the target bounding box in the SAR image, the SAR image is divided into two regions;

[0014] The saliency binary map is further generated by assigning the values ​​"1" to all pixels in the region within the diagonal frame and "0" to all pixels in the region outside the diagonal frame in the SAR image.

[0015] In one embodiment, the image feature extractor includes a plurality of sequentially connected residual convolutional blocks;

[0016] Each residual convolutional block includes two convolutional layers. When the residual convolutional block processes the input data, it performs batch normalization on the output data of each convolutional layer. At the same time, it uses residual connection to add the input data to the first convolutional layer and the output data of the last convolutional layer. Then, it uses the GeLU activation function to process the addition result to obtain the output features of the residual convolutional block.

[0017] In one embodiment, the saliency feature extractor includes an average pooling layer, a convolutional layer, and a GeLU activation function layer.

[0018] In one embodiment, in the optimized visual attention classifier:

[0019] The salient visual feature vector is transformed into a query vector input to the multi-head attention unit through the position encoding unit;

[0020] The deep feature vector is transformed into an address vector and a feature vector by the position encoding unit and input to the multi-head attention unit.

[0021] In one embodiment, the method further includes:

[0022] A dual-branch target recognition network is constructed based on the image feature extractor, salient feature extractor, and optimized visual attention classifier.

[0023] The SAR image and its corresponding saliency binary map are input into the dual-branch target recognition network to obtain the target recognition result.

[0024] In one embodiment, during the training of the dual-branch target recognition network:

[0025] First, the image feature extractor and the location encoding unit are pre-trained. After the pre-training is completed, the adjusted parameter values ​​in the image feature extractor and the location encoding unit are fixed. Then, other adjustable parameters in the dual-branch target recognition network are trained.

[0026] This application also provides a SAR target recognition device that integrates visual salient features, the device comprising:

[0027] The image acquisition module is used to acquire SAR images for target recognition.

[0028] The target segmentation image acquisition module is used to segment the targets in the SAR image based on prior information to obtain a saliency binary map corresponding to the SAR image;

[0029] The depth feature vector extraction module is used to segment the SAR image to obtain multiple image patches, and to extract features from the multiple image patches using an image feature extractor based on multi-layer residual convolution to obtain depth feature vectors.

[0030] The salient visual feature vector extraction module is used to process the salient binary map using a salient feature extractor to obtain a salient visual feature vector with the same specifications as the depth feature vector.

[0031] The target recognition module is used to process the depth feature vector and salient visual feature vector using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector and performs target recognition based on the fused features.

[0032] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0033] Acquire SAR images for target identification;

[0034] Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image;

[0035] After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0036] After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained;

[0037] The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

[0038] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0039] Acquire SAR images for target identification;

[0040] Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image;

[0041] After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0042] After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained;

[0043] The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

[0044] The aforementioned SAR target recognition method, apparatus, device, and medium that integrates visual salient features obtains a salient binary map of the SAR image based on prior information. It then uses an image feature extractor and a salient feature extractor constructed based on multi-layer residual convolution to extract features from both the SAR image and the salient binary map, obtaining depth feature vectors and salient visual feature vectors. An optimized visual attention classifier is then used to encode the feature vectors positionally, followed by multi-head attention fusion. This allows for cross-attention encoding of depth features by salient visual features, enabling feature filtering and ultimately yielding the target recognition result. This method can achieve accurate target recognition in SAR images with complex backgrounds. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a SAR target recognition method that integrates visually salient features in one embodiment.

[0046] Figure 2 This is a schematic diagram of an image feature extractor constructed based on multi-layer residual convolution in one embodiment;

[0047] Figure 3 This is a schematic diagram of the salient feature extractor in one embodiment;

[0048] Figure 4 This is a schematic diagram illustrating the results of three schemes for adding a saliency binary map to an input image in one embodiment;

[0049] Figure 5 This is a reference schematic diagram of the attention mechanism in one embodiment;

[0050] Figure 6This is a schematic diagram of the structure of a dual-branch target recognition network in one embodiment;

[0051] Figure 7 This is a schematic diagram of the training of a dual-branch target recognition network in one embodiment;

[0052] Figure 8 This is a structural block diagram of a SAR target recognition device that integrates visual salient features in one embodiment;

[0053] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In existing technologies, when using deep learning models for target recognition in SAR images, especially when the target is a ship, the SAR image slices of ship targets often have complex backgrounds due to the frequent occurrence of ships docking or densely packed berths in real-world scenarios. However, such slices are scarce in existing SAR ship recognition datasets. Most targets in existing datasets are located on open sea surfaces, with only a few images containing near-shore objects or other ships in the background. While network models trained using these datasets perform well in recognizing isolated ship targets on open sea surfaces, they face two main challenges when processing ship target slices with complex backgrounds.

[0056] First, the model lacks the ability to actively distinguish between the target ship and the background slice. In most training data, the ship hull is the only strong scattering source within the slice area, and its image brightness is significantly higher than that of the sea surface. In such slices, the sea surface and the ship can be distinguished based on the brightness differences in different areas. However, in situations such as when ships are docked or multiple ships are densely moored, the image brightness of other scatterers in the slice background is similar to that of the target ship, causing features in the slice that do not belong to the target ship to interfere with the classification results.

[0057] Secondly, the superposition of images from different scatterers complicates image features. Due to the inherent mechanism of SAR imaging, a portion of the backscattered echo from a scatterer often shifts and diffuses within the image, distributing to surrounding areas and superimposing with the images of other scatterers. On one hand, this causes part of the image of the ship to be identified to diffuse into the background area, increasing the difficulty of feature extraction; on the other hand, when the ship approaches other scatterers, the images of those scatterers are also superimposed on the ship's image. In image slices of open sea surfaces and isolated ships, because the backscattered echo from the sea surface is weak, the above superposition phenomenon mainly manifests as the diffusion of the ship's image into the background area, and the background area is relatively dark, making it easy to extract the features of the ship to be identified. However, when the background contains other scatterers, the superposition phenomenon between different scatterers complicates the image's feature space, increasing the difficulty of classification and recognition.

[0058] In response to the above problems, such as Figure 1 As shown, a SAR target recognition method that integrates visual salient features is provided, including the following steps:

[0059] Step S100: Acquire the SAR image to be identified.

[0060] Step S110: Based on prior information, the targets in the SAR image are segmented to obtain the saliency binary map of the corresponding SAR image.

[0061] Step S120: After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0062] Step S130: After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained.

[0063] Step S140: The depth feature vector and salient visual feature vector are processed by an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector and performs target recognition based on the fused features.

[0064] In this embodiment, a feature extractor based on residual convolution and a classifier based on visual attention are used to achieve SAR target recognition in an end-to-end manner. Specifically, to guide the deep learning model in distinguishing between target and background regions during classification, this method converts manually labeled oblique box information in the dataset into visual saliency maps. These maps are then processed into deep feature vectors by one branch of the network and fused with SAR image features extracted by another branch through a cross-attention mechanism. This reduces the interference of complex sliced ​​backgrounds on the model's end-to-end inference process, thereby improving the accuracy and reliability of the classification results.

[0065] In this embodiment, the SAR target recognition method proposed in this paper has a good recognition effect on SAR images with ships as targets and complex backgrounds, and can obtain accurate target recognition results. At the same time, this method can also be applied to the recognition of other types of targets, and can also achieve good recognition results. In this paper, ships are used as an example for illustration.

[0066] The basic process of deep learning-based SAR ship target recognition can be divided into three stages: preprocessing, feature extraction, and category mapping. The preprocessing stage involves operations such as slicing and image enhancement on the target image. During network training, various augmentation techniques are introduced, such as flipping, adding noise, and affine transformations, thereby expanding the dataset size while improving the network's ability to extract key features. Subsequently, feature extraction and category mapping are implemented by the feature extractor and classifier, respectively. These two are integrated into the network model and executed end-to-end. First, the feature extractor extracts depth features from the input data, and then the classifier maps the depth feature space to the category space. In this paper, the preprocessing stage will not be elaborated upon; the focus will be on the feature extraction and category mapping parts.

[0067] In step S100, the obtained SAR image is a SAR amplitude image.

[0068] In step S120, a residual convolution module is used to extract image depth features.

[0069] As one of the representative models in CNNs, the ResNet network introduced residual convolutional modules, introducing residual connections that are connected in parallel with a set of cascaded convolutional layers to form residual modules. This enhances the feature extraction capabilities of deeper neural network layers and addresses the performance degradation problem of deep networks. Furthermore, ResNet uses the ReLU activation function to strengthen the network's non-linear expressive power and employs batch normalization and dropout mechanisms to effectively improve training stability, helping to alleviate overfitting. Multiple convolutional layers can achieve a large receptive field while saving computation, and the stacking of multiple residual blocks gives the network greater depth, enabling it to express more abstract and complex features.

[0070] like Figure 2 As shown, this method uses a ResNet network model and multi-layer residual convolution to extract features from SAR ship targets, taking the SAR amplitude image as input. Taking a 384×384 SAR image slice as an example, the input image is first divided into 256 24×24 image patches. For each image patch, multi-layer residual convolution reduces its two-dimensional size while expanding its feature dimension, resulting in a 3×3 64-channel feature map, which is then flattened to a size of 1×576. The 256 flattened feature maps are then concatenated to obtain a feature vector with a length of 256 and a feature dimension of 576. It should be noted that the number of SAR amplitude image segments, the size of each segmented image patch, and the final feature vector size are all provided as examples; these values ​​can be defined according to actual needs.

[0071] In this embodiment, the image feature extractor includes multiple residual convolutional blocks connected in sequence. Each residual convolutional block includes two convolutional layers. When the residual convolutional block processes the input data, it performs batch normalization on the output data of each convolutional layer. Simultaneously, it uses residual connection to add the input data to the first convolutional layer to the output data of the last convolutional layer, and then uses the GeLU activation function to process the addition result to obtain the output features of the residual convolutional block.

[0072] Specifically, Figure 2 The diagram also illustrates the specific structure of the residual convolutional block. Each residual convolutional block contains two convolutional layers, with the output of each layer undergoing batch normalization. The residual connection adds the original input to the output of the final convolutional layer; when the two layers are of different sizes, a 1×1 convolution is used to transform the original input. The sum is then processed by the GeLU activation function, and the resulting feature map serves as the output of the residual convolutional block. The GeLU activation function is an improved version of the ReLU activation function, effectively avoiding the vanishing gradient problem of the ReLU activation function.

[0073] In this method, the purpose of introducing visual saliency information is to use it as auxiliary information to guide the classification network to distinguish between the ship to be identified and the slice background. When designing the specific representation of this auxiliary information, this method mainly considers the following factors: (1) the cost of manually annotating the information when creating training and testing datasets; (2) the feasibility of extracting the information from SAR image slices in actual application scenarios; and (3) the compatibility of the information representation method with the network inference process.

[0074] Currently, common annotation methods for SAR ship target detection and recognition datasets include horizontal bounding boxes, oblique bounding boxes, semantic segmentation, and keypoints. Among these, oblique bounding box annotation can describe the position, length, width, and orientation information of ship targets in an image. It is more suitable for the elongated shape of ship targets than horizontal bounding boxes and keypoints, and it saves on manual annotation costs compared to semantic segmentation. It has been widely used in newly released SAR ship detection and recognition datasets in recent years. In addition, research on oblique bounding box detection of ship targets in large-scene SAR images has flourished, and many results have been achieved in related research on extracting ship target contour information from SAR images. The existing research in both of these aspects can support oblique bounding box extraction of SAR ship targets in practical application scenarios. Therefore, considering both the annotation cost of training data and the feasibility of extracting measured data, this method uses oblique bounding boxes as the basic source of auxiliary information.

[0075] The diagonal frame is typically represented by key parameters such as coordinates, size, and orientation, with common formats including four-point and five-parameter representations. These representations are concise and accurate, easy to store, and can be directly used as supervisory information for network training. However, in this method, the diagonal frame parameters are not used as supervisory information but are part of the network input information, used to assist the network's classification and recognition process. If key parameters are still used, a contradiction arises between the locality of image features and the globality of the diagonal frame parameters. On the one hand, the network model processes SAR images layer by layer. The first layer activates neurons based on the values ​​of each pixel in the SAR image, and then passes this information to the next layer, expanding the receptive field layer by layer. Therefore, many features extracted by the intermediate layers of the network are localized. On the other hand, the global coordinates, overall target size, and orientation information used in the diagonal frame are all highly global, and their correspondence with each pixel and local region in the image is not intuitive, making it difficult to effectively participate in the feature processing of the intermediate layers.

[0076] Taking the above considerations into account, in this method, the oblique frame is converted into a binary image with the same size as the SAR image slice.

[0077] In step S110, segmenting the target in the SAR image based on prior information to obtain the corresponding saliency binary map of the SAR image includes: dividing the SAR image into two regions according to the prior information of the target frame in the SAR image, assigning all pixels in the region inside the frame and the region outside the frame to "1" and "0" respectively, and further generating the saliency binary map.

[0078] Specifically, a single-channel image of the same size as the target slice is generated and divided into two regions: inside and outside the diagonal frame. The region inside the diagonal frame corresponds to the main body of the ship target to be identified, while the region outside the diagonal frame corresponds to the slice background. Pixels in the two regions are assigned values ​​of 1 and 0, respectively, thus establishing a pixel-by-pixel correspondence between the auxiliary information based on the diagonal frame annotation and the SAR image slice. The resulting binary image utilizes the difference in pixel intensity to emphasize the visually intuitive perception of the "main body of the ship," hence it is a visually saliency binary image (hereinafter referred to as a saliency binary image).

[0079] Furthermore, in the segmented saliency binary image, the division between the ship's main body and the background region is not precise: on the one hand, the oblique frame only generally reflects the length, width, and orientation of the ship to be identified; in reality, the shape of the ship itself is not rectangular. On the other hand, due to the inherent mechanism of SAR imaging, the images of certain substructures of the ship often shift and diffuse, detaching from the main image region of the ship and mixing with the background region outside the oblique frame. For example, the echo energy of some strong scattering structures diffuses beyond the oblique frame along the azimuth and range directions. Also, in three-dimensional space perpendicular to the sea surface, some scattering structures at different heights will shift to different degrees in the imaging results due to different incident wave angles. Therefore, the background region contains part of the ship target's image and often reflects the scattering characteristics and spatial location information of certain substructures of the ship target. This information should be effectively extracted and utilized during the classification and identification process.

[0080] like Figure 3 As shown, the saliency feature extractor used to process saliency binary maps includes an average pooling layer, a convolutional layer, and a GeLU activation function layer, which fuses actual auxiliary information with image features.

[0081] Specifically, the saliency binary image is downsampled sequentially through an average pooling layer and a convolutional layer, and then expanded into a multi-channel image at the convolutional layer. The downsampled result is processed by the GeLU activation function and flattened into a one-dimensional vector as the output of this branch, with the same length and feature dimension as the feature extractor output. Taking the 384×384 example image in section 2.1 as an example, the corresponding saliency binary image undergoes 8×8 average pooling with a stride of 8 to obtain a 48×48 single-channel downsampled image, which is then subjected to 3×3 multi-channel convolution with a stride of 3 to obtain a feature map of size 16×16 with 576 channels. The receptive field of each pixel in the final feature map is a 24×24 region in the binary image, corresponding one-to-one with the 256 image patches obtained from the SAR image segmentation.

[0082] In step S130, different input information is processed by the image feature extractor and the salient feature extractor respectively. The outputs of the two are input into the optimized visual attention classifier, and after being marked by the shared position encoding unit, they are sent to the multi-head attention unit to ensure that the classifier can obtain the correspondence between the elements in the two vectors.

[0083] In this method, a salient feature extractor is used to process the salient binary map. This choice is made after analyzing different fusion methods, mainly considering factors such as information integrity and feature stability.

[0084] In this method, three fusion schemes were considered, including: (1) background masking scheme, which multiplies the saliency binary map with the SAR image slice pixel by pixel. This scheme masks the information of the background area and only retains the main body of the ship to be identified; (2) background weakening scheme, which changes the pixel value of the background area of ​​the saliency binary map to x∈(0,1) and then multiplies it with the SAR image slice pixel by pixel. This scheme emphasizes the main body of the ship to be identified and also retains the information of the background area to a certain extent; (3) dual-channel scheme: the SAR image slice and the saliency binary map are set as two channels of the input image. Figure 4 The diagram shows three possible solutions. Figure 4 (a) is the background masking scheme. Figure 4 (b) is the background weakening scheme, x = 0.5. Figure 4 (c) is a dual-channel scheme.

[0085] In the background masking scheme, information outside the main body of the ship is completely discarded. Due to the imprecision of the frame and the offset and diffusion characteristics of certain substructure images of the ship under SAR imaging conditions, this scheme loses some information about the ship target. In contrast, the background weakening scheme masks and weakens the background area of ​​the original SAR image slice based on a saliency binary map. After the image is input into the network, the overall activation level of neurons corresponding to the background area will be significantly lower than that of the main ship area, thus highlighting the target while preserving information from the background area.

[0086] However, the background weakening scheme still has two limitations: First, weakening the local features in the background region that are effective for classification reduces the activation values ​​of their corresponding neurons, hindering their effectiveness in the classification process. Second, from the perspective of feature space, this scheme leads to image feature instability. When images of the same or similar substructures are located within, outside, or crossed by the oblique frame boundary, the activation effects of corresponding neurons in the intermediate layers of the network will differ. Especially when the oblique frame boundary crosses the image, the corresponding image of the substructure is divided into two parts, and their proportions and positional relationships can lead to complex and diverse situations. The resulting diverse activation states disrupt the stability of image features, increasing the difficulty of extracting effective classification features.

[0087] The dual-channel approach, however, fully preserves the image features of the original image. After the dual-channel image is input into the network, the intermediate layers can use different parameter combinations to process the information provided by the two channels with a specific emphasis. For example, when a neuron's parameter combination assigns a higher weight to the SAR image channel, it can extract scattering and geometric features with strong stability. When a neuron's parameter combination assigns a higher weight to the binary image channel, it can effectively utilize the region segmentation information it represents to distinguish the hull from the background. However, the above parameter combinations depend on parameter tuning during network training. The SAR image and the saliency binary image lack independent processing procedures. The end-to-end training process cannot guarantee that the model effectively understands the differences in the information contained in the two channels, making it difficult for the model to achieve the desired effect.

[0088] In summary, the fusion method of directly adding the saliency binary map to the input image has limitations in terms of target information integrity and image feature stability. Therefore, in this method, the classification network uses an independent saliency branch (i.e., a saliency feature extractor) to process the saliency binary map, separating the image feature extraction process from the information processing of the saliency binary map, enabling the network to acquire complete and stable target features to be identified.

[0089] Furthermore, in step S130, the fusion of image depth features and saliency features is achieved through a self-attention mechanism.

[0090] Figure 5 Several attention encoding methods are demonstrated. Figure 5 middle, Figure 5 (a) represents the encoding process of the attention mechanism. Figure 5 (b) indicates multi-headed attention. Figure 5 (c) represents the Transformer module. The Transformer module structure is as follows: Figure 5As shown in (c), this module consists of a multi-head attention layer and a feed-forward network (FFN) layer, and incorporates residual connections and layer normalization, borrowing from the design of residual networks. Based on this, the ViT model is proposed, applying the Transformer to the field of image processing. ViT first segments the entire input 2D image, obtaining a set of P×P 2D image patches. Patch embedding and positional encoding are then used to transform the 2D image into the input vector of the Transformer module. Patch embedding flattens the 2D image before inputting it into a fully connected layer for feature mapping, while positional encoding marks the relative positional relationships between image patches, thus solving the problem of lost relative positional relationships caused by the sequence invariance of the Transformer module.

[0091] Specifically, when the image feature extractor transforms image patches, the patches undergo multiple layers of residual convolution. This shrinks the feature map length while expanding the number of channels, converting it from P×P×1 to 1×1×D, where D is the dimension of the input vector for the optimized visual attention classifier. This achieves the transformation from a 2D image to a 1D vector. Compared to a single fully connected layer, multiple layers of residual convolution can extract deeper features, and the receptive field of each element in the resulting 1D vector remains the entire image patch. Positional encoding uses a set of learnable parameters to label image patches at different locations, marking the relative positions between them. This solves the problem of lost relative positional relationships caused by the sequence invariance of subsequent multi-head attention units.

[0092] Specifically, in optimizing the visual attention classifier, the main approach is to fuse salient features of the target region and global features of the image through an improved multi-head attention module. The salient visual feature vector F sf Query vector after location encoding Deep feature vector F gf Used as a position-encoded key vector Sum value vector Where p(·) refers to the positional encoding function. The entire feature fusion strategy is mathematically represented as:

[0093] F fused =ζ{F sf ,F gf}+F gf (1)

[0094] In formula (1), ζ This represents an improved multi-head attention (MHA) mechanism, specifically calculated as follows:

[0095]

[0096] Head i (i = 1, ..., h) represents an instance of MHA computation, where M* is the parameter matrix.

[0097] Finally, the fused feature F fused Further processing yields the classification feature F. cls Then F cls The input is fed into a classifier for classification. During the feature transformation process, F... sf It contributed more important ship identification features, while F gf It contributes correlation features between the ship and the complex background. By fusing a self-attention mechanism, cross-attention coding fusion of target depth information and global feature information of SAR image was successfully achieved.

[0098] like Figure 6 As shown, the specific process of feature processing in this method is illustrated, along with the structure of the optimized visual attention classifier. It can be seen that the depth feature vector, after passing through a position encoding unit and layer normalization, is input as both an address vector and a feature vector into the multi-head attention unit. Simultaneously, the depth feature vector after the position encoding unit is fused with the output features of the multi-head attention unit to obtain intermediate features. These intermediate features then pass through a layer normalization layer and a feedforward network, and are further fused with the intermediate features via residual connections. Finally, a fully connected layer is used to classify the target category, thus achieving target recognition.

[0099] In this embodiment, when applying the proposed recognition method in practice, the feature extractor, saliency feature extractor, and optimized visual attention classifier can be regarded as a whole framework. That is, a two-branch target recognition network is constructed based on the image feature extractor, saliency feature extractor, and optimized visual attention classifier. The SAR image and the corresponding saliency binary map are input into the two-branch target recognition network to obtain the target recognition result.

[0100] In this embodiment, in the dual-branch target recognition network, the branch containing the image feature extractor is the image feature branch, which mainly implements the content in step S120, while the branch containing the salient feature extractor is the salient branch, which mainly implements the content in step S130. The two features extracted through these two branches are finally used for target recognition by the optimized visual attention classifier.

[0101] In order to effectively train the dual-branch target recognition network, in this embodiment, the training process is divided into two steps: a pre-training task and a classification task. The pre-training task adopts the Masked Image Modeling (MIM) task, which realizes the parameter initialization of some modules in the classification network, effectively reducing the training difficulty of the classification task.

[0102] In this embodiment, when training the dual-branch target recognition network: first, the image feature extractor and the position encoding unit are pre-trained, and after the pre-training is completed, the adjusted parameter values ​​in the image feature extractor and the position encoding unit are fixed, and then the other adjustable parameters in the dual-branch target recognition network are trained.

[0103] Specifically, the overall training process of the dual-branch target recognition network is as follows: Figure 7 As shown, the training process is divided into two steps, each performing a different training task. The first step is a pre-training task, where the encoder-decoder network shown in the figure is trained. The second step is a SAR ship target classification task, where the dual-branch SAR ship target recognition network proposed in this paper is trained.

[0104] Furthermore, during pre-training, the MIM (Model-In-Image) is used as the pre-training task to train the encoder-decoder network illustrated in the diagram end-to-end. This network's function is to extract feature information from the input image and reconstruct missing parts. The encoder part of the network is an image feature extractor, and the decoder part consists of cascaded visual attention modules and fully connected layers, incorporating positional encoding to model the spatial relationships of image features.

[0105] Specifically, the MIM (Mean Interpreter Model) method involves randomly masking local information in the input image, allowing the trained network to reconstruct the masked regions based on the remaining information. The masked information is then used as supervisory information during the training process. When the entire encoder-decoder network can reconstruct the missing information from the few remaining details in the image, it indicates that the network has achieved a relatively complete model of this type of image, possessing the ability to extract effective information and perform analytical reasoning. Specifically, the encoder module after MIM can be considered to have a strong ability to extract local image patch information, while the positional encoding module can effectively describe the spatial relationships of various local regions in the training image. Furthermore, this pre-training task allows the trained network to comprehensively learn various types of information appearing in the training image, rather than being limited to specific targets or a few partial features. Therefore, the feature extractor obtained in this way has good generalization potential.

[0106] In this method, the goal of employing the MIM method is to leverage the aforementioned properties of MIM to initialize the parameters of the feature extractor and positional encoder modules. Therefore, in designing the encoder-decoder network, the structures of the encoder and positional encoder modules are identical to those of the feature extractor and positional encoder in the proposed dual-branch network, so as to transfer the pre-trained parameters to the classification and recognition network.

[0107] In this embodiment, a pixel-wise mean squared error loss function is used when performing MIM.

[0108] In this embodiment, the training of the second step, SAR ship target classification task, is actually performed on the entire dual-branch target recognition network. Considering the rotation and translation invariance of ship target classification results, the sensitivity of the classification results to the relative positions of image patches is lower than that of pixel-by-pixel reconstruction of missing images in MIM. Therefore, the supervision effect of the classification task on position encoding parameters is weaker than that of the pre-training task. Continuing to update the position encoding parameters during the classification task training is not beneficial to improving its relative position modeling ability and may even cause parameter degradation. Simultaneously, the classifier's parameters are not pre-trained; their initialization method is random initialization, requiring adjustment through early training to initially adapt to the pre-trained feature extractor. Subsequently, the feature extractor and classifier continue training together to further fine-tune the parameters, thereby better adapting to the classification task.

[0109] Therefore, during the training of the classification task, in the two-branch object recognition network, the parameters of the saliency branch, multi-head attention unit, and fully connected layer are initialized randomly, while the parameters of the feature extractor and position encoding unit are transferred from the encoder-decoder network trained in the first step. After the transfer, the position encoding parameters are set to fixed parameters and are no longer adjusted during the training process; the parameters of the feature extractor are set to fixed parameters in the early stage of training, and then set to adjustable parameters in the later stage of training after the parameters of other parts of the network have initially stabilized, and are fine-tuned along with the parameters of other parts of the network.

[0110] In this embodiment, when training the above classification task, considering the characteristics of unbalanced sample sizes and different recognition difficulties in the existing SAR ship target dataset, a multi-class Focal-Loss loss function is used to calculate the classification loss, and the loss weights of samples of different sizes are balanced by class weighting to enhance the network's learning ability on the existing dataset.

[0111] Specifically, Focal-Loss is a classification loss function improved upon the cross-entropy loss function to address the imbalance between easy and difficult samples. Its expression is:

[0112] FL(p t )=-(1-pt ) γ log(p t (3)

[0113] In formula (3), pt represents the model's confidence in the predicted label, and the adjustment factor (1-p) t ) γ The introduction of γ allows the loss function value to assign higher loss weights to difficult samples based on the model's confidence level; the larger the coefficient γ, the more significant the weight difference between easy and difficult samples.

[0114] Based on the same adjustment principle, and considering the size of different categories of samples in the dataset, class weights are added to formula (3). For a sample whose true value is category i, its class weight is:

[0115]

[0116] In formula (4), N max N min These represent the maximum and minimum sample sizes for each category, N. i Let be the total number of samples in this class, and coefficients a and b scale the weights from (0,1) to (a,b). By introducing class weights, samples from smaller classes are assigned higher loss function values ​​while maintaining the same difficulty and confidence, thus strengthening the training process's focus on these types of targets. The loss function expression is:

[0117] loss(p t )=-w i ·FL(p t )=-w i (1-p t ) γ log(p t ) (5)

[0119] In the above-mentioned SAR target recognition method that integrates visual saliency features, a dual-branch classification and recognition network that integrates visual saliency information is proposed for SAR ship target images with complex backgrounds. The network uses oblique box annotations to generate a visual saliency binary map, which guides the model to distinguish between ship targets and background regions, reducing the interference of complex backgrounds on the recognition results. At the same time, classification and ablation experiments were carried out on this method based on satellite images. The results show that: (1) The image feature branch uses a feature extractor based on residual convolution, which is dedicated to processing SAR image slices. The processing is not affected by the region division and can extract stable SAR image features; (2) The saliency branch generates feature vectors based on the binary map, which are used to guide the network to perform flexible feature selection. The deep feature vector has a richer coding space, which can alleviate the problem of inaccurate region division caused by the imprecision of oblique boxes; (3) This paper introduces a visual attention mechanism based on cross attention and position coding, which can effectively integrate visual saliency information and SAR image features; (4) The MIM pre-training method used in this paper can perform good parameter initialization for the feature extractor and position coding, thereby reducing the training difficulty of the classification network and improving the classification performance of the model.

[0120] Compared with existing SAR ship identification methods that process simple background conditions, this method is more adaptable to complex real-world application scenarios. This method achieves effective fusion of SAR image features and auxiliary information by generating binary images, introducing a dual-branch structure, and employing a cross-attention mechanism.

[0121] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0122] In one embodiment, such as Figure 8 As shown, a SAR target recognition device integrating visual salient features is provided, comprising: a to-be-processed image acquisition module 200, a target segmentation image acquisition module 210, a depth feature vector extraction module 220, a salient visual feature vector extraction module 230, and a target recognition module 240, wherein:

[0123] The image acquisition module 200 is used to acquire SAR images for target recognition.

[0124] The target segmentation image acquisition module 210 is used to segment the target in the SAR image based on prior information to obtain a saliency binary map corresponding to the SAR image;

[0125] The depth feature vector extraction module 220 is used to perform image segmentation on the SAR image to obtain multiple image blocks, and to extract features from the multiple image blocks using an image feature extractor based on multi-layer residual convolution to obtain depth feature vectors.

[0126] The salient visual feature vector extraction module 230 is used to process the salient binary map using a salient feature extractor to obtain a salient visual feature vector with the same specifications as the depth feature vector.

[0127] The target recognition module 240 is used to process the depth feature vector and the salient visual feature vector using an optimized visual attention classifier to obtain a target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and the salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and the salient visual feature vector and performs target recognition based on the fused features.

[0128] Specific limitations regarding the SAR target recognition device that integrates visual salient features can be found in the limitations of the SAR target recognition method that integrates visual salient features described above, and will not be repeated here. Each module in the aforementioned SAR target recognition device that integrates visual salient features can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0129] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a SAR target recognition method that integrates visual salient features. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0130] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0132] Acquire SAR images for target identification;

[0133] Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image;

[0134] After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0135] After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained;

[0136] The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0138] Acquire SAR images for target identification;

[0139] Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image;

[0140] After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors.

[0141] After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained;

[0142] The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A SAR target recognition method that integrates visually salient features, characterized in that, The method includes: Acquire SAR images for target identification; Based on prior information, the targets in the SAR image are segmented to obtain a saliency binary map corresponding to the SAR image; After image segmentation of the SAR image, multiple image patches are obtained. An image feature extractor based on multi-layer residual convolution is used to extract features from the multiple image patches to obtain depth feature vectors. After processing the saliency binary map using a saliency feature extractor, a saliency visual feature vector with the same specifications as the depth feature vector is obtained; The depth feature vector and salient visual feature vector are processed using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector, and target recognition is performed based on the fused features.

2. The SAR target identification method according to claim 1, characterized in that, The step of segmenting the targets in the SAR image based on prior information to obtain a saliency binary map corresponding to the SAR image includes: Based on the prior information of the target bounding box in the SAR image, the SAR image is divided into two regions; The region within the diagonal frame and the region outside the diagonal frame in the SAR image are assigned the values ​​"1" and "0" respectively, and the saliency binary map is further generated.

3. The SAR target identification method according to claim 1, characterized in that, The image feature extractor includes multiple sequentially connected residual convolutional blocks; Each residual convolutional block includes two convolutional layers. When the residual convolutional block processes the input data, it performs batch normalization on the output data of each convolutional layer. At the same time, it uses residual connection to add the input data to the first convolutional layer and the output data of the last convolutional layer. Then, it uses the GeLU activation function to process the addition result to obtain the output features of the residual convolutional block.

4. The SAR target identification method according to claim 1, characterized in that, The salient feature extractor includes an average pooling layer, a convolutional layer, and a GeLU activation function layer.

5. The SAR target identification method according to claim 1, characterized in that, In the optimized visual attention classifier: The salient visual feature vector is transformed into a query vector input to the multi-head attention unit through the position encoding unit; The deep feature vector is transformed into an address vector and a feature vector by the position encoding unit and input to the multi-head attention unit.

6. The SAR target identification method according to any one of claims 1-5, characterized in that, The method further includes: A dual-branch target recognition network is constructed based on the image feature extractor, salient feature extractor, and optimized visual attention classifier. The SAR image and its corresponding saliency binary map are input into the dual-branch target recognition network to obtain the target recognition result.

7. The SAR target identification method according to claim 6, characterized in that, When training the dual-branch target recognition network: First, the image feature extractor and the location encoding unit are pre-trained. After the pre-training is completed, the adjusted parameter values ​​in the image feature extractor and the location encoding unit are fixed. Then, other adjustable parameters in the dual-branch target recognition network are trained.

8. A SAR target recognition device that integrates visual salient features, characterized in that, The device includes: The image acquisition module is used to acquire SAR images for target recognition. The target segmentation image acquisition module is used to segment the targets in the SAR image based on prior information to obtain a saliency binary map corresponding to the SAR image; The depth feature vector extraction module is used to segment the SAR image to obtain multiple image patches, and to extract features from the multiple image patches using an image feature extractor based on multi-layer residual convolution to obtain depth feature vectors. The salient visual feature vector extraction module is used to process the salient binary map using a salient feature extractor to obtain a salient visual feature vector with the same specifications as the depth feature vector. The target recognition module is used to process the depth feature vector and salient visual feature vector using an optimized visual attention classifier to obtain the target recognition result. The optimized visual attention classifier includes a position encoding unit and a multi-head attention unit. The position encoding unit converts the depth feature vector and salient visual feature vector into the input vector of the multi-head attention unit. The multi-head attention unit performs feature fusion on the depth feature vector and salient visual feature vector and performs target recognition based on the fused features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images