SAR image target recognition method based on text perception cross-modal transformer

Abstract

The application relates to the technical field of synthetic aperture radar image processing and multi-modal fusion, in particular to a SAR image target recognition method based on text perception cross-modal Transformer, which comprises the following steps: acquiring high-level language features and multi-scale initial visual features; performing bidirectional cross-scale fusion on the multi-scale initial visual features to obtain optimized multi-scale visual features; performing cross-modal coding enhancement on the high-level language features and the optimized multi-scale visual features to obtain visual language fusion features; generating an initial query vector of a decoder through a text perception query selection module; and performing iterative decoding inference on the visual language fusion features by using the initial query vector, so that a target recognition box specified by a text description is obtained and output. The method can solve the limitation of the prior art which only depends on a single visual mode, and further improve the semantic understanding ability, recognition accuracy, robustness and generalization ability of the model in a complex scene.

Description

Technical Field

[0001] The embodiments of this application relate to the fields of synthetic aperture radar image processing and multimodal fusion technology, and in particular to a SAR image target recognition method based on text-aware cross-modal Transformer. Background Technology

[0002] Synthetic Aperture Radar (SAR) has been widely used in disaster monitoring, urban planning, and marine surveillance due to its all-weather, all-day imaging advantages and high penetration. SAR image target recognition technology aims to automatically detect, locate, and classify targets in complex imaging scenes, and is a core component of SAR data interpretation. However, existing SAR image target recognition methods mainly rely on learning and matching visual features of SAR images (such as grayscale, contours, and scattering characteristics), which has inherent limitations. First, SAR images themselves are affected by factors such as speckle noise, imaging geometric distortion, and changes in observation angle, resulting in significant differences in the imaging characteristics of the same target under different conditions, leading to blurred features. Relying solely on visual features makes it difficult to establish a stable and robust target representation model, easily resulting in false detections and missed detections in complex backgrounds or among similar targets. Second, and more importantly, SAR images only contain electromagnetic scattering and geometric morphology information of the target, lacking high-level semantic information about the target's function, category, attributes, and scene context. This results in traditional methods only achieving a superficial resemblance, failing to meet the fine-grained semantic understanding requirements of practical applications. For example, they cannot distinguish the functional type of a building, the operational status of a ship, or the growth stage of a crop.

[0003] In practical applications, there is a wealth of textual information related to SAR observations, such as prior knowledge bases, geographic information descriptions, and historical interpretation reports. This text contains rich semantic knowledge and scene constraints. However, current technologies generally process images and textual information separately, adopting a "image processing separately, knowledge supplemented a posteriori" approach, failing to effectively integrate textual semantic information into the reasoning process of the recognition model. This approach is not only inefficient and reliant on human experience, but also makes the model lack the ability to use prior knowledge for reasoning and enhanced generalization. When faced with image quality degradation, small sample sizes, or unknown target scenes, the robustness and accuracy of recognition performance are severely limited.

[0004] Therefore, how to address the current technical problems that urgently need to be solved, such as existing SAR image target recognition methods relying solely on visual features, being unable to respond to text-specified recognition requirements, having poor robustness in recognizing targets in complex backgrounds, and being affected by speckle noise and geometric distortion in SAR images leading to insufficient recognition accuracy. Summary of the Invention

[0005] In view of this, embodiments of this application propose a SAR image target recognition method based on a text-aware cross-modal Transformer. Through a cross-modal network model architecture, it achieves deep fusion of text semantic features and SAR image visual features, enabling the model to accurately locate the target to be identified based on text descriptions. At the same time, it improves the model's adaptability to SAR image noise, distortion, and target scale changes, and finally outputs accurate target recognition boxes to meet the requirements of SAR image target recognition under text command guidance. This solves the limitation of existing technologies that rely on only a single visual modality, thereby improving the model's semantic understanding ability, recognition accuracy, robustness, and generalization ability in complex scenes.

[0006] To achieve the above objectives, embodiments of this application propose a SAR image target recognition method based on a text-aware cross-modal Transformer, the method comprising the following steps: High-level linguistic features and multi-scale initial visual features are obtained through a cross-modal network model. The high-level linguistic features are obtained by preprocessing and encoding the text description, while the multi-scale initial visual features are obtained by preprocessing and extracting features from SAR images. Bidirectional cross-scale fusion of initial multi-scale visual features is performed to obtain optimized multi-scale visual features; Cross-modal encoding enhancement is performed on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, which can be used to filter and enhance visual features using text semantics; Based on high-level language features and optimized multi-scale visual features, the initial query vector of the decoder is generated through the text-aware query selection module; The Transformer decoder based on deformable attention uses the initial query vector to iteratively decode and infer the visual-language fusion features, regresses to obtain and outputs the target recognition box specified by the text description, and trains and optimizes the cross-modal network model based on the combined loss function.

[0007] To achieve the above objectives, embodiments of this application also propose a SAR image target recognition system based on a text-aware cross-modal Transformer, the system comprising: The feature acquisition module is used to acquire high-level language features and multi-scale initial visual features through a cross-modal network model. The high-level language features are obtained by preprocessing and encoding the text description, and the multi-scale initial visual features are obtained by preprocessing and extracting features from the SAR image. The bidirectional cross-scale fusion module is used to perform bidirectional cross-scale fusion on initial multi-scale visual features to obtain optimized multi-scale visual features. The cross-modal coding enhancement module is used to perform cross-modal coding enhancement on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, so as to use text semantics to filter and enhance visual features; The vector generation module is used to generate the initial query vector for the decoder based on high-level language features and optimized multi-scale visual features through the text-aware query selection module. The iterative decoding and inference module is used for the Transformer decoder based on deformable attention. It uses the initial query vector to perform iterative decoding and inference on the visual-language fusion features, regresses to obtain and outputs the target recognition box specified by the text description, and trains and optimizes the cross-modal network model based on the combined loss function.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device, including: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a text-aware cross-modal Transformer-based SAR image target recognition method as described above.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a SAR image target recognition method based on a text-aware cross-modal Transformer as described above.

[0010] This application proposes a SAR image target recognition method based on a text-aware cross-modal Transformer. The method acquires high-level linguistic features and multi-scale initial visual features through a cross-modal network model. It then performs bidirectional cross-scale fusion on the initial multi-scale visual features to obtain optimized multi-scale visual features. Next, it enhances the high-level linguistic features and optimized multi-scale visual features through cross-modal coding to obtain visual-language fusion features, which are then used to filter and enhance visual features using text semantics. Based on the high-level linguistic features and optimized multi-scale visual features, a text-aware query selection module generates an initial query vector for the decoder. Finally, based on a deformable attention-based Transformer decoder, the initial query vector is used to iteratively decode and infer the visual-language fusion features, regressing to obtain the final visual-language fusion feature. The system outputs the target bounding box specified by the text description and trains and optimizes the cross-modal network model based on the combined loss function. Since high-level language features are obtained by preprocessing and encoding the text description, and multi-scale initial visual features are obtained by preprocessing and extracting features from the SAR image, the cross-modal network model achieves deep fusion of text semantic features and SAR image visual features. This allows the model to accurately locate the target to be identified based on the text description, while improving the model's adaptability to SAR image noise, distortion, and target scale changes. Finally, it outputs accurate target bounding boxes to meet the needs of SAR image target recognition under text-guided instructions. This addresses the limitations of existing technologies that rely on only a single visual modality, thereby improving the model's semantic understanding, recognition accuracy, robustness, and generalization ability in complex scenes.

[0011] Optionally, the cross-modal network model includes a language encoder and a visual backbone network, where the language encoder is a pre-trained BERT model and the visual backbone network is ResNet. The process of acquiring high-level language features and multi-scale initial visual features through the cross-modal network model includes: segmenting, normalizing, and data augmenting the SAR image; and segmenting and tokenizing the text description. The processed text description is then input into the pre-trained language encoder in the cross-modal network model for encoding, with an output dimension of [missing information]. High-level language features are extracted; the processed SAR image is input into the visual backbone network in the cross-modal network model to extract initial visual features at multiple scales.

[0012] Optionally, cross-scale fusion is performed on the initial visual features at multiple scales to obtain optimized multi-scale visual features, including: fusing the initial visual features at multiple scales through a bidirectional cross-scale fusion module; wherein, the bidirectional cross-scale fusion module performs top-down and bottom-up feature transfer and fusion, and fuses adjacent scale features through a RepConv block, thereby outputting optimized multi-scale visual features.

[0013] Optionally, cross-modal coding enhancement is performed on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, which utilize textual semantics to filter and enhance visual features. This includes: flattening the optimized multi-scale visual features as a query vector, and using high-level language features as keys and values ​​respectively, and calculating semantic features through a multi-head self-attention mechanism; calculating fusion coefficients based on semantic features and flattened multi-scale visual features by introducing learnable parameters; and modulating the semantic features and flattened multi-scale visual features using the fusion coefficients to obtain enhanced visual-language fusion features to suppress image noise and irrelevant background.

[0014] Optionally, based on high-level language features and optimized multi-scale visual features, an initial query vector for the decoder is generated through a text-aware query selection module, including: calculating the similarity between high-level language features and optimized multi-scale visual features; and selecting the top K image feature tokens most relevant to the text description from the similarity using the TopK function as the initial query vector.

[0015] Optionally, the Transformer decoder based on deformable attention iteratively decodes and infers the visual-language fusion features using the initial query vector, and regresses to obtain and outputs the target recognition box specified by the text description, including: inputting the initial query vector into the Transformer decoder based on deformable attention, iteratively decoding and inferring the visual-language fusion features to obtain the target core features; and using a detection head to perform bounding box regression on the target core features to obtain the coordinates of the target recognition box specified by the text description.

[0016] Optionally, training and optimizing the cross-modal network model based on the combined loss function includes: obtaining the target ground truth bounding boxes; during training of the cross-modal network model, optimizing the parameters of the cross-modal network model by minimizing the combined loss function based on the target ground truth bounding boxes and the target bounding boxes predicted by the model; wherein, the combined loss function is a weighted sum of the generalized intersection-union loss function and the L1 loss function; minimizing the combined loss function... The expression is as follows: ; ; ; in, This represents the generalized intersection-union loss function. Represents the L1 loss function; For the target true bounding box, The target bounding box predicted by the model; To surround simultaneously and The minimum bounding rectangle, and For loss weighting coefficients, This represents the number of iteration stages in the decoder. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0018] Figure 1 This is a flowchart of a SAR image target recognition method based on a text-aware cross-modal Transformer provided in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of a cross-modal network model provided in one embodiment of this application; Figure 3 This is a schematic diagram of a SAR public dataset detection result provided in one embodiment of this application; Figure 4 This is a flowchart illustrating an implementation of a text-aware cross-modal Transformer-based SAR image target recognition technology provided in an embodiment of this application. Figure 5 This is a schematic diagram of the structure of a SAR image target recognition system based on a text-aware cross-modal Transformer, provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0020] Synthetic Aperture Radar (SAR), with its advantages of all-weather, all-day operation and high penetration, has been widely used in disaster monitoring, resource surveys, and marine surveillance. SAR image recognition, as the core technology of SAR data interpretation, aims to automatically detect, locate, classify, and determine attributes from complex imaging scenes, providing a reliable basis for subsequent decision-making and applications. With the rapid development of high-resolution, multi-polarization SAR systems, target structures and scattering details have become richer, making traditional recognition methods based on manual features insufficient for high-precision interpretation. In recent years, intelligent recognition technologies, represented by deep learning, have significantly improved the accuracy of SAR image target recognition. However, most existing technologies rely solely on single SAR image information for feature learning and inference decisions, which still has significant limitations in real-world complex scenarios, making it difficult to achieve stable and reliable target recognition.

[0021] Relying solely on SAR images for target identification faces several inherent limitations. First, SAR images are affected by factors such as speckle noise, imaging geometric distortion, and changes in observation angle. The scattering characteristics of the same target vary significantly under different conditions, easily leading to blurred target features, unclear boundaries, and missing structural information. Simply relying on image grayscale, contours, and scattering features makes it difficult to establish a stable and invariant target representation model, easily resulting in misjudgments and missed detections in environments with similar targets and complex terrain. Second, SAR images only reflect the electromagnetic scattering characteristics and geometric shape of a target, lacking semantic information, attribute information, prior knowledge, and scene context information. They cannot distinguish high-level semantics with practical application value, such as function, purpose, and status, causing the identification results to remain at the "form-similar" level, failing to meet the requirements of refined identification and intelligent interpretation. For example, in urban planning, images alone are insufficient to distinguish the functions of different types of buildings; in agricultural monitoring, SAR images alone cannot directly determine key information such as crop type, growth stage, and irrigation status; and in marine monitoring, SAR images alone cannot directly determine key information such as ship type, operational status, and navigation attributes.

[0022] In practical applications, a wealth of textual information related to SAR observations exists and possesses significant application value. This includes prior knowledge bases, geographic information texts, site function descriptions, and historical interpretation reports. This type of textual information contains rich semantic knowledge, attribute definitions, category relationships, scene constraints, and prior rules, which can compensate for the deficiencies of image information at the semantic level. However, existing SAR image recognition methods generally fail to effectively integrate textual information into the recognition process, mostly still employing a "separate image processing, with knowledge supplemented a posteriori" approach. This results in a disconnect between textual and image information, making it difficult for knowledge to empower the recognition model. This approach is not only inefficient and overly reliant on human experience, but also prevents the model from using prior semantics for reasoning during the recognition process, severely limiting the intelligence level of SAR interpretation systems.

[0023] SAR image recognition technology that does not incorporate textual information has significant shortcomings in practical engineering applications. First, its semantic understanding is insufficient; the model can only learn visual features and cannot understand high-level semantics such as target category, attributes, functions, and relationships, making true intelligent interpretation difficult. Second, its generalization ability is limited; in scenarios with small samples, sparse samples, or unknown targets, the model struggles to make reasonable judgments due to a lack of external knowledge support. Third, its robustness is insufficient; when faced with image degradation problems such as noise, distortion, and occlusion, the lack of prior textual knowledge as constraints and corrections results in poor stability of recognition results. Fourth, knowledge is difficult to reuse; a large amount of domain-specific textual knowledge and expert experience cannot be embedded in the model, leading to system upgrades relying on extensive data annotation, resulting in high costs, long cycles, and poor scalability.

[0024] Therefore, introducing textual information into SAR image recognition has significant theoretical and practical value. The embodiments of this application, through multimodal fusion of text and image, can transform semantic knowledge, prior constraints, category descriptions, and attribute relationships in the text into model-learnable feature representations, complementing the scattering and structural features of SAR images. This enables joint reasoning of multimodal information at the data, feature, or decision levels. Introducing textual information significantly enhances the model's semantic understanding of targets, upgrading recognition from visual matching to knowledge-driven semantic understanding, effectively reducing interference from noise, distortion, and angle changes, and improving recognition robustness. Simultaneously, textual information provides knowledge support for small-sample and zero-sample recognition, reducing the model's dependence on a large number of labeled samples and improving cross-scene and cross-target generalization capabilities. Furthermore, with the help of textual semantic information, fine-grained attributes such as target function, state, and purpose can be discriminated, propelling SAR image interpretation from traditional target classification to higher-level semantic understanding.

[0025] In summary, existing SAR image-based recognition methods suffer from semantic gaps, insufficient robustness, weak fine-grained recognition capabilities, and difficulty in utilizing knowledge, making them inadequate for recognizing complex scenarios. Therefore, effectively incorporating textual information into the SAR image recognition process and constructing a cross-modal fusion recognition model is a crucial approach to overcoming existing technological bottlenecks and improving the intelligent interpretation of SAR images. Currently, related fusion mechanisms, feature alignment methods, and knowledge embedding methods remain imperfect. Therefore, researching SAR image recognition methods based on textual information enhancement has significant research value and broad application prospects, and is a key technical problem addressed in this application.

[0026] In view of this, embodiments of this application propose a SAR image target recognition method based on a text-aware cross-modal Transformer. Through a cross-modal network architecture, it achieves deep fusion of text semantic features and SAR image visual features, enabling the model to accurately locate the target to be identified based on the text description. At the same time, it improves the model's adaptability to SAR image noise, distortion, and target scale changes, and finally outputs an accurate target recognition box to meet the requirements of SAR image target recognition under text command guidance. This solves the limitation of existing technologies that rely on only a single visual modality, thereby improving the model's semantic understanding ability, recognition accuracy, robustness, and generalization ability in complex scenes.

[0027] One embodiment of this application proposes a SAR image target recognition method based on a text-aware cross-modal Transformer, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the SAR image target recognition method based on a text-aware cross-modal Transformer proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0028] The specific process of the SAR image target recognition method based on text-aware cross-modal Transformer proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Obtain high-level language features and multi-scale initial visual features through a cross-modal network model.

[0029] Among them, high-level language features are obtained by preprocessing and encoding the text description, and multi-scale initial visual features are obtained by preprocessing and extracting features from SAR images.

[0030] Understandably, this step, through preprocessing and basic feature extraction of the raw input data, lays the foundation for subsequent cross-modal fusion. The raw input data includes SAR images and their corresponding natural language text descriptions.

[0031] For example, a cross-modal network model includes a language encoder and a visual backbone network, where the language encoder is a pre-trained BERT model and the visual backbone network is a ResNet.

[0032] In one possible embodiment, step 101 includes: performing block segmentation, normalization, and data augmentation on the SAR image, and performing word segmentation and tokenization on the text description; inputting the processed text description into a pre-trained language encoder in a cross-modal network model for encoding, with an output dimension of... High-level language features are extracted; the processed SAR image is input into the visual backbone network in the cross-modal network model to extract initial visual features at multiple scales.

[0033] For example, such as Figure 2 As shown, the processing is divided into two parallel branches, which include a language branch and a visual branch.

[0034] In the language branch, the input natural language text description (e.g., "find the power tower in the southwest direction") is first preprocessed, including word segmentation and tokenization, transforming it into a token sequence that the model can process. This sequence is then fed into a pre-trained language encoder, such as the BERT model. This language encoder understands the deep semantics of the text and outputs a dimensional... Advanced language features .in, For channel dimension, Maximum length of the text token. (High-level language features) It incorporates the semantic information of text instructions, which is used to guide subsequent image recognition.

[0035] In the vision branch, the input SAR image undergoes preprocessing, including image segmentation, pixel value normalization, and possible data augmentation operations (such as random flipping and rotation) to improve the model's robustness. The processed image is then fed into a visual backbone network, such as ResNet. This network extracts features from the image at different levels and outputs a set of multi-scale initial visual features. .in, This represents the feature layer index. ResNet is used as the backbone network for the visual branch to extract visual features at different scales from SAR images. These features capture the low-level edge texture, mid-level component structure, and high-level semantic information of targets in SAR images, while preserving the scattering characteristics of targets unique to SAR imaging.

[0036] Step 102: Perform bidirectional cross-scale fusion on the initial multi-scale visual features to obtain optimized multi-scale visual features.

[0037] Understandably, due to the varying target scales in SAR images and the influence of speckle noise, the feature representation capability of a single scale is limited. This step aims to fuse visual information from different levels to enhance the representation capability of the feature pyramid, specifically through a bidirectional cross-scale fusion module.

[0038] In one possible embodiment, step 102 includes fusing initial visual features at multiple scales using a bidirectional cross-scale fusion module.

[0039] The bidirectional cross-scale fusion module performs top-down and bottom-up feature transfer and fusion, and fuses adjacent scale features through the RepConv block, thereby outputting optimized multi-scale visual features.

[0040] like Figure 2 As shown, the cross-modal network model also includes a bidirectional cross-scale fusion module for fusing multi-scale visual features extracted by the backbone network. This module contains two paths: Top-down approach: High-level features are upsampled using convolution and nearest-neighbor interpolation to match the size and number of channels of adjacent low-level features, and the fused semantic information is then passed down. High-level features have richer semantics but lower resolution.

[0041] Bottom-up approach: Low-level features are downsampled or channel-adjusted through convolution to match their size with adjacent high-level features, and precise spatial location information is passed up. Low-level features are more accurately located but have weaker semantic meaning.

[0042] When features from adjacent scales are fused, efficient feature integration can be achieved using a RepConv block (reparameterized convolutional block). Ultimately, this results in a set of optimized, more information-rich multi-scale visual features. .

[0043] Step 103: Perform cross-modal coding enhancement on the high-level language features and the optimized multi-scale visual features to obtain visual-language fusion features, so as to use text semantics to filter and enhance the visual features.

[0044] Understandably, this step is the core step in achieving text-guided image recognition. By constructing a cross-modal encoder based on a collaborative attention mechanism, the aim is to use text semantics as a filter and enhancer to screen and enhance visual features, suppress irrelevant background and noise, and highlight the target features specified in the text.

[0045] In one possible embodiment, step 103 includes: flattening the optimized multi-scale visual features as a query vector, and using the high-level language features as keys and values ​​respectively, and calculating semantic features through a multi-head self-attention mechanism; calculating fusion coefficients based on the semantic features and the flattened multi-scale visual features by introducing learnable parameters; and modulating the semantic features and the flattened multi-scale visual features using the fusion coefficients to obtain enhanced visual-language fusion features to suppress image noise and irrelevant background.

[0046] For example, to ensure feature effectiveness while reducing computational complexity, typically only the high-level visual features of the last layer (i.e., the highest level, with the strongest semantics) are considered. After being flattened, it is fed into the subsequent Transformer encoder.

[0047] For example, such as Figure 2 As shown, the visual features of the high-rise building after flattening As a query (Q), high-level language features Simultaneously, these visual features serve as both keys (K) and values ​​(V) in a multi-head self-attention module. This module calculates the correlation between each visual location feature and all text token features, resulting in a visual feature that is semantically highly correlated with the text description. .

[0048] Next, obtain the learnable parameters. This is used to adaptively control the degree of fusion. The fusion coefficient is calculated. The calculation formula is as follows: ; Understandably, the fusion coefficient This coefficient is used to measure the difference between the original visual features and the text-related features. Finally, the original features and cross-modal related features are modulated using this coefficient to obtain the enhanced visual-language fusion features. The calculation formula is as follows: ; in, This indicates the correlation between visual and linguistic representation features.

[0049] Understandably, the enhanced visual language fusion features It can effectively suppress SAR image noise and irrelevant background features, highlighting the visual features of the target specified in the text. Therefore, the model can ensure that it can focus on the semantic region of the text description.

[0050] Step 104: Based on high-level language features and optimized multi-scale visual features, the initial query vector of the decoder is generated through the text-aware query selection module.

[0051] In one possible embodiment, step 104 includes: calculating the similarity between high-level language features and optimized multi-scale visual features; and selecting the top K image feature tokens most relevant to the text description from the similarity using the TopK function as the initial query vector.

[0052] Understandably, in standard Transformer object detectors, the decoder's query vector is typically initialized randomly. To improve the model's ability to focus on specified targets in text and its convergence speed, this application designs a text-aware query selection module to intelligently initialize the query.

[0053] For example, the text-aware query selection module works by calculating image features. Text features The similarity between them. Specifically, this is achieved through tensor multiplication: ; in, For batch indexing, For image feature spatial location index, Indexed for text tokens. This is a channel index. This similarity matrix reflects the strength of the association between each image location and each text word.

[0054] It should be noted that the image features here... For the multi-scale visual features mentioned above Text features here For the high-level language features mentioned above .

[0055] Subsequently, the TopK function is used to select the K feature vectors from all image locations that are most relevant to the overall text features. These feature vectors are directly used as the initial query vectors for the decoder. In this way, at the very beginning of the decoder's operation, its attention is already guided to the image region most relevant to the text description, greatly improving the targeting and accuracy of the recognition process and increasing the model's convergence speed.

[0056] Step 105: Based on the Transformer decoder with deformable attention, the visual-language fusion features are iteratively decoded and inferred using the initial query vector to regress and output the target recognition box specified by the text description. The cross-modal network model is then trained and optimized based on the combined loss function.

[0057] Understandably, this step is used to achieve the final localization and recognition, and it can fuse the initial query vector with visual language features. They are then fed into a Transformer decoder based on multi-scale deformable attention.

[0058] In one possible embodiment, the Transformer decoder based on deformable attention iteratively decodes and infers the visual-language fusion features using an initial query vector to regress and output the target recognition box specified by the text description, including: inputting the initial query vector into the Transformer decoder based on deformable attention to iteratively decode and infer the visual-language fusion features to obtain the target core features; and using a detection head to perform bounding box regression on the target core features to obtain the coordinates of the target recognition box specified by the text description.

[0059] For example, this Transformer decoder performs multi-round iterative inference. In each round, the query vector adaptively focuses on visual-language fusion features through a deformable attention mechanism. The understanding of target features is gradually refined based on the most relevant spatial locations and feature scales. After several iterations, the Transformer decoder outputs optimized query features.

[0060] like Figure 2 As shown, these features are fed into a lightweight detection head (usually a feedforward network) to directly regress the bounding box coordinates of the target corresponding to each query. That is, the target recognition box.

[0061] The model training uses a combined loss function for end-to-end optimization. This combined loss function is a weighted sum of the GIoU loss function and the L1 loss function commonly used in bounding box regression. For details, please refer to the following example.

[0062] In one possible embodiment, training and optimizing the cross-modal network model based on a combined loss function includes: obtaining the target ground truth bounding box; and during the training of the cross-modal network model, training and optimizing the parameters of the cross-modal network model by minimizing the combined loss function based on the target ground truth bounding box and the target bounding box predicted by the model.

[0063] The combined loss function is the weighted sum of the generalized intersection-union loss function and the L1 loss function.

[0064] Minimize the combined loss function The expression is as follows: ; ; ; in, This represents the generalized intersection-union loss function. Represents the L1 loss function; For the target true bounding box, The target bounding box predicted by the model; To surround simultaneously and The minimum bounding rectangle, and For loss weighting coefficients, This represents the number of iteration stages in the decoder. The parameters of the entire cross-modal network model are optimized by minimizing the total loss using the backpropagation algorithm until the model converges.

[0065] In summary, such as Figure 2 As shown, this cross-modal network model extracts visual and textual features through parallel dual branches, enhances feature expression through multi-level fusion across scales and modalities, uses textual semantic intelligence to initialize queries, and finally achieves accurate target localization and recognition driven by textual instructions through a decoder.

[0066] Reference Figure 3 The diagram illustrates the detection results of the provided publicly available SAR dataset. When tested on the publicly available SAR dataset, this method demonstrated superior performance. In ship detection tasks, after receiving text instructions to detect ships, the model can accurately locate multiple ships against a complex sea background, and its output (red box) highly matches the manually labeled (green ellipse). In aircraft detection tasks, the model can also accurately locate aircraft targets in airports based on instructions. This demonstrates that the method provided in the embodiments of this application can effectively utilize text instructions to achieve accurate and robust target recognition in SAR images.

[0067] This application proposes a SAR image target recognition method based on a text-aware cross-modal Transformer. The method acquires high-level linguistic features and multi-scale initial visual features through a cross-modal network model. It then performs bidirectional cross-scale fusion on the initial multi-scale visual features to obtain optimized multi-scale visual features. Next, it enhances the high-level linguistic features and optimized multi-scale visual features through cross-modal coding to obtain visual-language fusion features, which are then used to filter and enhance visual features using text semantics. Based on the high-level linguistic features and optimized multi-scale visual features, a text-aware query selection module generates an initial query vector for the decoder. Finally, based on a deformable attention-based Transformer decoder, the initial query vector is used to iteratively decode and infer the visual-language fusion features, regressing and outputting the result. The target recognition box specified by the text description is generated, and the cross-modal network model is trained and optimized based on the combined loss function. Since the high-level language features are obtained by preprocessing and encoding the text description, and the multi-scale initial visual features are obtained by preprocessing and extracting features from the SAR image, the cross-modal network model achieves deep fusion of text semantic features and SAR image visual features. This allows the model to accurately lock the target to be identified based on the text description, while improving the model's adaptability to SAR image noise, distortion, and target scale changes. Finally, it outputs accurate target recognition boxes to meet the needs of SAR image target recognition under text command guidance. This solves the limitation of existing technologies that rely on only a single visual modality, thereby improving the model's semantic understanding ability, recognition accuracy, robustness, and generalization ability in complex scenes.

[0068] like Figure 4 As shown, Figure 4 This is a flowchart illustrating an implementation of a text-aware cross-modal Transformer-based SAR image target recognition technology, provided as an embodiment of this application.

[0069] First, the input SAR image and text command are preprocessed separately, and high-level linguistic features and multi-scale initial visual features are extracted through corresponding branch networks. Then, a bidirectional cross-scale fusion module is used to fuse the multi-level visual features vertically and horizontally, resulting in optimized multi-scale visual features. Next, a cross-modal encoder based on a collaborative attention mechanism is employed to enhance the association between text and visual features, suppressing noise and highlighting target features. Then, a text-aware query selection module initializes the decoder query, incorporating text semantics into the initial query to improve target focusing capabilities. Finally, the optimized initial query is input into the decoder for feature inference, the detection head performs bounding box regression, and a combined loss function is used for model training and optimization.

[0070] It is understood that the embodiments of this application achieve text-guided SAR target recognition by deeply fusing text commands with SAR images, breaking through the limitations of traditional methods that rely solely on visual features. This allows for customized target detection based on natural language commands, significantly improving recognition flexibility and practicality. Through cross-scale feature fusion and cross-modal coding, it effectively overcomes interference from target scale variations, geometric distortions, and speckle noise, greatly improving recognition accuracy and robustness. The text-aware query module optimizes model convergence efficiency, reduces false detection rates, and enhances target focusing capabilities. This method is optimized for SAR imaging characteristics, adapts to various satellite sensor data, and can stably detect multiple types of targets, demonstrating outstanding versatility and generalization ability. Simultaneously, it simplifies the operation process, lowers the professional usage threshold, and facilitates the efficient application of SAR images in emergency response, engineering, and other scenarios.

[0071] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0072] Another embodiment of this application proposes a SAR image target recognition system based on a text-aware cross-modal Transformer. The details of this SAR image target recognition system based on a text-aware cross-modal Transformer are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this example. Figure 5 This is a schematic diagram of the structure of a SAR image target recognition system based on a text-aware cross-modal Transformer proposed in this embodiment, including: The feature acquisition module 210 is used to acquire high-level language features and multi-scale initial visual features through a cross-modal network model; wherein, the high-level language features are obtained by preprocessing and encoding the text description, and the multi-scale initial visual features are obtained by preprocessing and feature extraction of SAR images. The bidirectional cross-scale fusion module 220 is used to perform bidirectional cross-scale fusion on the initial multi-scale visual features to obtain optimized multi-scale visual features. The cross-modal coding enhancement module 230 is used to perform cross-modal coding enhancement on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, so as to use text semantics to filter and enhance visual features; The vector generation module 240 is used to generate the initial query vector of the decoder through the text-aware query selection module based on high-level language features and optimized multi-scale visual features. The iterative decoding inference module 250 is used for the Transformer decoder based on deformable attention. It uses the initial query vector to perform iterative decoding inference on the visual-language fusion features, regresses to obtain and outputs the target recognition box specified by the text description, and trains and optimizes the cross-modal network model based on the combined loss function.

[0073] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0074] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0075] Another embodiment of this application provides an electronic device, such as Figure 6 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can realize a SAR image target recognition method based on text-aware cross-modal Transformer as described in the above method embodiment.

[0076] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0077] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0078] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a SAR image target recognition method based on a text-aware cross-modal Transformer as described in the above method embodiments.

[0079] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0080] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A SAR image target recognition method based on text-aware cross-modal Transformer, characterized in that, The method includes: High-level linguistic features and multi-scale initial visual features are obtained through a cross-modal network model. The high-level linguistic features are obtained by preprocessing and encoding the text description, while the multi-scale initial visual features are obtained by preprocessing and extracting features from SAR images. Bidirectional cross-scale fusion of initial multi-scale visual features is performed to obtain optimized multi-scale visual features; Cross-modal encoding enhancement is performed on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, which can be used to filter and enhance visual features using text semantics; Based on high-level language features and optimized multi-scale visual features, the initial query vector of the decoder is generated through the text-aware query selection module; The Transformer decoder based on deformable attention uses the initial query vector to iteratively decode and infer the visual-language fusion features, regresses to obtain and outputs the target recognition box specified by the text description, and trains and optimizes the cross-modal network model based on the combined loss function.

2. The method according to claim 1, characterized in that, The cross-modal network model includes a language encoder and a visual backbone network. The language encoder is a pre-trained BERT model, and the visual backbone network is ResNet. The process of obtaining high-level language features and multi-scale initial visual features through the cross-modal network model includes: The SAR images are segmented, normalized, and augmented, and the text descriptions are segmented and tokenized. The processed text description is input into the pre-trained language encoder in the cross-modal network model for encoding, and the output dimension is... Advanced language features; The processed SAR image is input into the visual backbone network in the cross-modal network model to extract initial visual features at multiple scales.

3. The method according to claim 2, characterized in that, The process of cross-scale fusion of initial multi-scale visual features to obtain optimized multi-scale visual features includes: The initial visual features of multiple scales are fused through a bidirectional cross-scale fusion module. The bidirectional cross-scale fusion module performs top-down and bottom-up feature transfer and fusion, and fuses adjacent scale features through the RepConv block, thereby outputting optimized multi-scale visual features.

4. The method according to claim 3, characterized in that, The process involves cross-modal encoding enhancement of high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, which utilize textual semantics to filter and enhance visual features, including: The optimized multi-scale visual features are flattened and used as query vectors, and the high-level language features are used as keys and values, respectively. Semantic features are then calculated through a multi-head self-attention mechanism. Based on semantic features and flattened multi-scale visual features, the fusion coefficient is calculated by introducing learnable parameters. By using fusion coefficients to modulate semantic features and flattened multi-scale visual features, enhanced visual-language fusion features are obtained to suppress image noise and irrelevant background.

5. The method according to claim 1, characterized in that, The initial query vector for the decoder, generated by the text-aware query selection module based on high-level language features and optimized multi-scale visual features, includes: Calculate the similarity between high-level language features and optimized multi-scale visual features; The TopK function is used to select the top K image feature tokens that are most relevant to the text description from the similarity scores, which are then used as the initial query vector.

6. The method according to claim 1, characterized in that, The Transformer decoder based on deformable attention iteratively decodes and infers the visual-language fusion features using an initial query vector, regresses to obtain and outputs the target recognition box specified by the text description, including: The initial query vector is input into the Transformer decoder based on deformable attention, and the visual-language fusion features are iteratively decoded and inferred to obtain the core features of the target. The detection head is used to perform bounding box regression on the core features of the target to obtain the coordinates of the target recognition box specified in the text description.

7. The method according to claim 6, characterized in that, The training and optimization of the cross-modal network model based on the combined loss function includes: Obtain the target's true bounding box; During the training of the cross-modal network model, the parameters of the cross-modal network model are optimized by minimizing the combined loss function based on the ground truth target bounding boxes and the target bounding boxes predicted by the model. The combined loss function is a weighted sum of the generalized intersection-union loss function and the L1 loss function. The expression is as follows: ； ； ； in, This represents the generalized intersection-union loss function. Represents the L1 loss function; For the target true bounding box, The target bounding box predicted by the model; To surround simultaneously and The minimum bounding rectangle, and For loss weighting coefficients, This represents the number of iteration stages in the decoder.

8. A SAR image target recognition system based on text-aware cross-modal Transformer, characterized in that, The system includes: The feature acquisition module is used to acquire high-level language features and multi-scale initial visual features through a cross-modal network model. The high-level language features are obtained by preprocessing and encoding the text description, and the multi-scale initial visual features are obtained by preprocessing and extracting features from the SAR image. The bidirectional cross-scale fusion module is used to perform bidirectional cross-scale fusion on initial multi-scale visual features to obtain optimized multi-scale visual features. The cross-modal coding enhancement module is used to perform cross-modal coding enhancement on high-level language features and optimized multi-scale visual features to obtain visual-language fusion features, so as to use text semantics to filter and enhance visual features; The vector generation module is used to generate the initial query vector for the decoder based on high-level language features and optimized multi-scale visual features through the text-aware query selection module. The iterative decoding and inference module is used for the Transformer decoder based on deformable attention. It uses the initial query vector to perform iterative decoding and inference on the visual-language fusion features, regresses to obtain and outputs the target recognition box specified by the text description, and trains and optimizes the cross-modal network model based on the combined loss function.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions that the processor can execute, and the processor is configured to, when executing the instructions, enable the electronic device to perform a text-aware cross-modal Transformer-based SAR image target recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can realize SAR image target recognition based on a text-aware cross-modal Transformer as described in any one of claims 1 to 7.

Images