Multi-modal feature fusion SAR image recognition method guided by ConTrans network based on cross attention

Abstract

The invention discloses a multi-modal feature fusion SAR (Synthetic Aperture Radar) image recognition method guided by a ConTrans network based on cross attention. The method comprises the following steps: acquiring an initial SAR image; preprocessing the initial SAR image to obtain a first SAR image; performing texture feature extraction on the first SAR image to obtain a gray-level co-occurrence matrix, and performing contrast calculation on the gray-level co-occurrence matrix to obtain a texture image; a target convolutional neural network is constructed, the target convolutional neural network comprises an input layer, a first feature extraction module, a second feature extraction module, five Trans blocks, five Conv blocks and a weighted average layer, the five Trans blocks comprise a first Trans block, a second Trans block, a third Trans block, a fourth Trans block and a fifth Trans block, and the structures of the first Trans block, the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block are the same. According to the invention, the technical problem that the classification accuracy is low when a single-mode SAR image is adopted for classification in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a cross-attention based ConTrans network guided multimodal feature fusion SAR image recognition method. Background Art

[0002] When an image contains rich feature information, building an efficient feature extraction model is extremely important for obtaining the content in the image. In order to achieve this goal, many models and methods have been proposed in the prior art, mainly including machine learning and deep learning methods. The attention mechanism with the ability to weigh important features is widely used in feature differentiation methods and has outstanding performance in the field of image target recognition. However, there is still a lot of room for improvement in the current classification performance, and there are still some problems that hinder researchers from improving model recognition. Many existing methods are based on learning single-modal raw SAR images, and single-modal images can only provide an expression of image information and cannot fully reflect the complete information of the image. The deep learning model can better perform image recognition by effectively extracting and fusing the features of multi-modal images. Therefore, the present invention proposes a multi-modal feature fusion SAR image classification technology guided by a cross-attention ConTrans network (Cross-Attention based ConTrans Network, CA-ConTransNet) - SAR images are recognized by iterative and deeper fusion of original image features and texture features. Summary of the invention

[0003] The embodiment of the present invention provides a multi-modal feature fusion SAR image recognition method guided by a ConTrans network based on cross-attention, so as to at least solve the technical problem of low classification accuracy when a single-modality SAR image is used for classification in the prior art.

[0004] According to one aspect of an embodiment of the present invention, a cross-attention based ConTrans network guided multimodal feature fusion SAR image recognition method is provided. The method may include: acquiring an initial SAR image; preprocessing the initial SAR image to obtain a first SAR image; extracting texture features from the first SAR image to obtain a grayscale co-occurrence matrix, performing contrast calculation on the grayscale co-occurrence matrix to obtain a texture image; constructing a target convolutional neural network, wherein the target convolutional neural network includes an input layer, a first feature extraction module, a second feature extraction module, five Trans blocks, five Conv blocks and a weighted average layer, the five Trans blocks include a first Trans block, a second Trans block, a third Trans block, a fourth Trans block and a fifth Trans block, the first Trans block, the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block have the same structure, the five Conv blocks include a first Conv block, a second Conv block, a third Conv block, a fourth Conv block and a fifth Conv block, the first Conv block, the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block have the same structure, each Trans block includes two layer normalizations, an MLP, a multi-head cross-attention feature fusion module and two fusion modules, each Conv block includes four 1×1 projection convolutions, two The first SAR image is processed by the first feature extraction module to obtain the initial original features; the texture image is processed by the second feature extraction module to obtain the initial texture features; the initial original features are processed by five Trans blocks to obtain the first classification result; the initial texture features are processed by five Conv blocks to obtain the second classification result; the first classification result and the second classification result are processed by the weighted average layer to obtain the target classification result of the initial SAR image.

[0005] Optionally, the preprocessing of the initial SAR image to obtain the first SAR image includes: filtering the initial SAR image by mean filtering to obtain a filtered SAR image; intercepting a 128×128 target area of ​​the filtered SAR image with the image center as the coordinate origin; scaling the 128×128 target area to 64×64; and grayscale normalizing the 64×64 target area to obtain the first SAR image.

[0006] Optionally, the processing the first SAR image by the first feature extraction module to obtain initial original features includes: inputting the first SAR image into a 7×7 convolution layer, a 3×3 maximum pooling layer and a projection layer in sequence to obtain the initial original features.

[0007] Optionally, the processing the texture image by the second feature extraction module to obtain the initial texture features includes: inputting the texture image into a 7×7 convolution layer and a 3×3 maximum pooling layer in sequence to obtain the initial texture features.

[0008] Optionally, the initial original features are processed through five Trans blocks to obtain a first classification result, including: in the first Trans block, the initial original features are input into layer normalization to obtain a global representation of the initial original features; a fine-grained local feature map of the initial texture features from the first Conv block is obtained; the global representation and the fine-grained local feature map are input into a multi-head cross-attention feature fusion module to obtain a first fused feature; the first fused feature and the initial original feature are input into the fusion module to obtain a second fused feature; the second fused feature is input into layer normalization and MLP in sequence to obtain a third fused feature; the third fused feature and the second fused feature are input into the fusion module to obtain the first original feature; the first original feature is processed in sequence by the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block to obtain a first classification result.

[0009] Optionally, the process of determining the fine-grained local feature map of the initial texture feature of the first Conv block is: inputting the initial texture feature into a 1×1 projection convolution, The spatial convolution of Convolution, average pooling layers, and layer normalization are performed to obtain fine-grained local feature maps of the initial texture features.

[0010] Optionally, the processing of the initial texture features by five Conv blocks to obtain the second classification result includes: in the first Conv block, the initial texture features are sequentially input into a 1×1 projection convolution, The first texture feature is obtained by performing spatial convolution and 1×1 projection convolution; the first texture feature is input into AW-Net to obtain the first adaptive weight of the initial texture feature; the first adaptive weight of the initial texture feature and the initial texture feature are input into the fusion module to obtain the first texture feature; the first texture feature is input into the 1×1 projection convolution to obtain the second texture feature; the fourth fusion feature from the first Trans block is obtained; the second texture feature and the fourth fusion feature are input into the fusion module to obtain the fifth fusion feature; the fifth fusion feature is input into The second adaptive weight of the fifth fusion feature is obtained by performing spatial convolution, 1×1 projection convolution and AW-Net; the second adaptive weight and the first adaptive weight are input into the fusion module to obtain the first texture feature; the first texture feature is processed in sequence by the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block to obtain the second classification result.

[0011] Optionally, the step of inputting the first texture feature into AW-Net to obtain a first adaptive weight of the initial texture feature includes: inputting the first texture feature into the maximum pooling layer and the average pooling layer, respectively, to obtain a maximum pooling feature and an average pooling feature; inputting the maximum pooling feature and the average pooling feature into MLP to obtain a maximum pooling texture feature and an average pooling texture feature; and inputting the maximum pooling texture feature and the average pooling texture feature into a fusion module and an activation layer in sequence to obtain a first adaptive weight.

[0012] Optionally, the process of determining the fourth fusion feature is: inputting the second fusion feature into an image interpolation module, 1×1 convolution and batch normalization in sequence to obtain the fourth fusion feature.

[0013] Beneficial effects of the present invention: The present invention proposes a multimodal feature fusion SAR image recognition method guided by a cross-attention ConTrans network. The constructed CA-ConTransNet can effectively perform feature modeling on SAR images for recognition. It focuses on the deep interaction of multimodal information and designs a multi-head cross-attention fusion mechanism. The original image features and texture image features are iteratively and effectively integrated at each stage of model training, which enhances the feature representation capability for classification and significantly improves the recognition accuracy of SAR images. It plays a key role in the interpretation of payload reconnaissance data in complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of a multimodal feature fusion SAR image recognition method guided by a ConTrans network based on cross-attention according to an embodiment of the present invention; Figure 2 is a schematic diagram of the CA-ConTransNet network structure according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a Trans block and a Conv block according to an embodiment of the present invention; Figure 4is a schematic diagram of the AW-Net structure according to an embodiment of the present invention; Figure 5 is a schematic diagram of a multi-head cross-attention feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] Example 1 According to an embodiment of the present invention, a multimodal feature fusion SAR image recognition method guided by a ConTrans network based on cross-attention is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0018] Figure 1 is a flow chart of a multimodal feature fusion SAR image recognition method guided by a ConTrans network based on cross attention according to an embodiment of the present invention, such as Figure 1 As shown, the method may include the following steps: Step S101, acquiring an initial SAR image.

[0019] In the technical solution provided in the above step S101 of the present invention, an initial SAR (Synthetic Aperture Radar) image is acquired.

[0020] Step S102: preprocessing the initial SAR image to obtain a first SAR image.

[0021] In the technical solution provided in the above step S102 of the present invention, the initial SAR image is preprocessed to obtain a first SAR image, wherein the first SAR image is a grayscale image.

[0022] Step S103: extract texture features from the first SAR image to obtain a gray level co-occurrence matrix, and perform contrast calculation on the gray level co-occurrence matrix to obtain a texture image.

[0023] In the technical solution provided in step S103 of the present invention, the first SAR image ( ) to extract texture features and calculate the gray-level co-occurrence matrix, where p is a point in the domain, (p) represents the midpoint of the image p The pixel value of i and j represents a set of pixel pairs in an image, yes p Along with another point Direction displacement, set the grayscale level to 64, select displacement d 1, direction The horizontal direction is 0°, and the image is traversed through a sliding window. The frequency of the grayscale combination of each adjacent pixel is counted and filled into the GLCM matrix. The matrix is ​​then normalized to represent the probability distribution. This method uses contrast as a texture feature based on GLCM input to describe the clarity of the image texture.

[0024] Step S104, constructing a target convolutional neural network, wherein the target convolutional neural network includes an input layer, a first feature extraction module, a second feature extraction module, five Trans blocks, five Conv blocks and a weighted average layer, the five Trans blocks include a first Trans block, a second Trans block, a third Trans block, a fourth Trans block and a fifth Trans block, the first Trans block, the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block have the same structure, the five Conv blocks include a first Conv block, a second Conv block, a third Conv block, a fourth Conv block and a fifth Conv block, the first Conv block, the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block have the same structure, each Trans block includes two layer normalizations, an MLP, a multi-head cross-attention feature fusion module and two fusion modules, each Conv block includes four 1×1 projection convolutions, two The spatial convolution, two AW-Nets and three fusion modules. The AW-Net includes a maximum pooling layer, an average pooling layer, MLP, a fusion module and an activation layer.

[0025] In the technical solution provided in the above step S104 of the present invention, a target convolutional neural network is constructed. Figure 2 is a schematic diagram of the CA-ConTransNet network structure according to an embodiment of the present invention, wherein the CA-ConTransNet network is a target convolutional neural network, such as Figure 2 As shown, the first feature extraction module includes a 7×7 convolution layer, a 3×3 maximum pooling layer and a projection layer, and the second feature extraction module includes a 7×7 convolution layer and a 3×3 maximum pooling layer. Figure 3 is a schematic diagram of the structure of a Trans block and a Conv block according to an embodiment of the present invention, such as Figure 3 As shown in Figure 2, the Trans block includes two layer normalizations, an MLP, a multi-head cross-attention feature fusion module, and two fusion modules. Each Conv block includes four 1×1 projection convolutions, two spatial convolution, two AW-Nets and three fusion modules, Figure 4 is a schematic diagram of the AW-Net structure according to an embodiment of the present invention. AW-Net is an adaptive weight network, such as Figure 4 As shown in Figure 1, the adaptive weight network consists of a maximum pooling layer, an average pooling layer, MLP, a fusion module, and an activation layer. Figure 5 is a schematic diagram of a multi-head cross-attention feature fusion module according to an embodiment of the present invention.

[0026] Step S105: Process the first SAR image by using a first feature extraction module to obtain initial original features.

[0027] In the technical solution provided in the above step S105 of the present invention, the first feature extraction module extracts features from the first SAR image to obtain initial original features.

[0028] Step S106: Process the texture image through a second feature extraction module to obtain initial texture features.

[0029] In the technical solution provided in the above step S106 of the present invention, the second feature extraction module processes the texture image to obtain initial texture features.

[0030] Step S107, processing the initial original features through five Trans blocks to obtain a first classification result.

[0031] In the technical solution provided in step S107 of the present invention, if Figure 2As shown, five Trans blocks process the initial raw features to obtain the first classification result.

[0032] Step S108, processing the initial texture features through five Conv blocks to obtain a second classification result.

[0033] In the technical solution provided in step S108 of the present invention, if Figure 2 As shown, five Conv blocks process the initial texture features to obtain the second classification result.

[0034] Step S109, processing the first classification result and the second classification result through a weighted average layer to obtain a target classification result of the initial SAR image.

[0035] In the technical solution provided in the above step S109 of the present invention, the first classification result and the second classification result are input into the weighted average layer to obtain the target classification result of the initial SAR image.

[0036] The above method of this embodiment is further introduced below.

[0037] As an optional implementation manner, step S102, preprocessing the initial SAR image to obtain a first SAR image, includes: filtering the initial SAR image by mean filtering to obtain a filtered SAR image; intercepting a 128×128 target area of ​​the filtered SAR image with the center of the image as the coordinate origin; scaling the 128×128 target area to 64×64; and performing grayscale normalization processing on the 64×64 target area to obtain the first SAR image.

[0038] In this embodiment, the initial SAR image of size 256×256 is mean filtered by a 3×3 kernel to obtain a filtered SAR image, thereby reducing the influence of coherent speckle noise on the image and improving image clarity. The filtered SAR image is taken with the image center as the coordinate origin, a 128×128 target area is intercepted to extract target details, and the area is scaled to 64×64 to reduce data dimensions and optimize calculation efficiency. The pixel values ​​are gray-scaled so that the gray-scale values ​​fall within the range of [0,1], and the initial SAR image is standardized.

[0039] As an optional implementation method, step S105, processing the first SAR image through the first feature extraction module to obtain initial original features, includes: inputting the first SAR image into a 7×7 convolution layer, a 3×3 maximum pooling layer and a projection layer in sequence to obtain the initial original features.

[0040] In this embodiment, if Figure 2 As shown, the first SAR image ( Figure 2The original image in the image is sequentially input into the 7×7 convolution layer, the 3×3 maximum pooling layer and the projection layer to obtain the initial original features ( Figure 2 ).

[0041] As an optional implementation method, step S106, processing the texture image through the second feature extraction module to obtain initial texture features, includes: inputting the texture image into a 7×7 convolution layer and a 3×3 maximum pooling layer in sequence to obtain initial texture features.

[0042] In this embodiment, if Figure 2 As shown in Figure 2, the texture image is sequentially input into the 7×7 convolution layer and the 3×3 maximum pooling layer to obtain the initial texture features ( Figure 2 texture features in ).

[0043] As an optional implementation method, step S107, the initial original features are processed through five Trans blocks to obtain a first classification result, including: in the first Trans block, the initial original features are input into layer normalization to obtain a global representation of the initial original features; a fine-grained local feature map of the initial texture features from the first Conv block is obtained; the global representation and the fine-grained local feature map are input into a multi-head cross-attention feature fusion module to obtain a first fused feature; the first fused feature and the initial original feature are input into the fusion module to obtain a second fused feature; the second fused feature is input into layer normalization and MLP in sequence to obtain a third fused feature; the third fused feature and the second fused feature are input into the fusion module to obtain the first original feature; the first original feature is processed in sequence by the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block to obtain a first classification result.

[0044] In this embodiment, if Figure 3 As shown, the initial original features ( Figure 3 The original features in ) are input to the layer normalization ( Figure 3 ), obtain the global representation of the initial original feature, obtain the fine-grained local feature map of the initial texture feature from the first Conv block, input the global representation and the fine-grained local feature map into the multi-head cross-attention feature fusion module, and obtain the first fused feature; Figure 5 As shown, the global representation extracted by the Trans block ( )( Figure 5 The original features in ) and the fine-grained local feature maps extracted by the Conv block ( )( Figure 5 The texture features in the image are simultaneously input into the multi-head cross-attention feature fusion module. The input of each modality corresponds to three weight matrices, and the query weight matrix , the key weight matrix , value weight matrix ,in and is a hyperparameter, query ,key Sum It can be defined as the following formula

[0045] The output of the multi-head cross-attention feature fusion module can be expressed as:

[0046] in, , , They are the queries corresponding to the global representation ,key Sum , , , The queries corresponding to the fine-grained local feature maps are ,key Sum represents the output of the multi-head cross-attention feature fusion module, Indicates connection, is the fully connected layer in the multi-head cross attention feature fusion module. and No change, but swap values ​​between the two modals , the attention weights are based on the similarity between different modalities, that is, the attention distribution of the features remains unchanged, but their values ​​are extracted from different modalities. The exchange of values ​​is conducive to the cross-modal information transfer between the original image and texture features, while retaining the attention calculation mechanism based on their respective features. It should be noted that Figure 5 The first CLS in is the vector after the global representation is reshaped. Figure 5 The second CLS in is the vector after reshaping of the fine-grained local feature map.

[0047] The first fused feature and the initial original feature are input into the fusion module to obtain the second fused feature, and the second fused feature is input into the layer normalization ( Figure 3 The third fusion feature is obtained by using the Layer Norm in the MLP and the third fusion feature; the third fusion feature and the second fusion feature are input into the fusion module to obtain the first original feature; the first original feature is processed in turn by the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block to obtain the first classification result.

[0048] As an optional embodiment, the process of determining the fine-grained local feature map of the initial texture feature of the first Conv block is: inputting the initial texture feature into 1×1 projection convolution, The spatial convolution of Convolution, average pooling layers, and layer normalization are performed to obtain fine-grained local feature maps of the initial texture features.

[0049] In this embodiment, if Figure 3 As shown, the initial texture features are sequentially input into the 1×1 projection convolution, The spatial convolution of Convolution, average pooling layers, and layer normalization are performed to obtain fine-grained local feature maps of the initial texture features.

[0050] As an optional implementation manner, step S108, the processing of the initial texture features by five Conv blocks to obtain a second classification result includes: in the first Conv block, the initial texture features are sequentially input into a 1×1 projection convolution, The first texture feature is obtained by performing spatial convolution and 1×1 projection convolution; the first texture feature is input into AW-Net to obtain the first adaptive weight of the initial texture feature; the first adaptive weight of the initial texture feature and the initial texture feature are input into the fusion module to obtain the first texture feature; the first texture feature is input into the 1×1 projection convolution to obtain the second texture feature; the fourth fusion feature from the first Trans block is obtained; the second texture feature and the fourth fusion feature are input into the fusion module to obtain the fifth fusion feature; the fifth fusion feature is input into The second adaptive weight of the fifth fusion feature is obtained by performing spatial convolution, 1×1 projection convolution and AW-Net; the second adaptive weight and the first adaptive weight are input into the fusion module to obtain the first texture feature; the first texture feature is processed in sequence by the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block to obtain the second classification result.

[0051] In this embodiment, if Figure 3 As shown, in the first Conv block, the initial texture features are sequentially input into the 1×1 projection convolution ( Figure 3 1×1 Conv-BN in The spatial convolution of Figure 3 3×3 Conv-BN in ), 1×1 projection convolution, and obtain the first texture feature; the first texture feature is input into AW-Net (such as Figure 4As shown), the first adaptive weight of the initial texture feature is obtained; the first adaptive weight of the initial texture feature and the initial texture feature are input into the fusion module to obtain the first texture feature; the first texture feature is input into the 1×1 projection convolution to obtain the second texture feature; the fourth fusion feature from the first Trans block is obtained; the second texture feature and the fourth fusion feature are input into the fusion module to obtain the fifth fusion feature; the fifth fusion feature is sequentially input into The second adaptive weight of the fifth fusion feature is obtained by performing spatial convolution, 1×1 projection convolution and AW-Net; the second adaptive weight and the first adaptive weight are input into the fusion module to obtain the first texture feature, and the first texture feature is processed in turn by the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block to obtain the second classification result.

[0052] As an optional implementation method, the first texture feature is input into AW-Net to obtain the first adaptive weight of the initial texture feature, including: inputting the first texture feature into the maximum pooling layer and the average pooling layer respectively to obtain the maximum pooling feature and the average pooling feature; inputting the maximum pooling feature and the average pooling feature into MLP to obtain the maximum pooling texture feature and the average pooling texture feature; inputting the maximum pooling texture feature and the average pooling texture feature into the fusion module and the activation layer in turn to obtain the first adaptive weight.

[0053] In this embodiment, if Figure 4 As shown, the first texture feature ( Figure 4 The input features X in are respectively input to the maximum pooling layer ( Figure 4 The maximum pooling feature and the average pooling feature are obtained by performing the Maxpool) and the average pooling layer (Avgpool) in the fusion module to obtain the maximum pooling feature and the average pooling feature. The maximum pooling feature and the average pooling feature are input into the MLP (shared multi-layer perceptron) to obtain the maximum pooling texture feature and the average pooling texture feature. The maximum pooling texture feature and the average pooling texture feature are input into the fusion module and the activation layer in turn to obtain the first adaptive weight ( Figure 4 ).

[0054] As an optional implementation manner, the process of determining the fourth fusion feature is: inputting the second fusion feature into the image interpolation module, 1×1 convolution and batch normalization in sequence to obtain the fourth fusion feature.

[0055] In this embodiment, if Figure 3 As shown in the figure, after the second fusion feature is reshaped, it is sequentially input into the image interpolation module (Interpolating), 1×1 convolution (1×1 Conv) and batch normalization (Batch Norm) to obtain the fourth fusion feature.

[0056] Experimental part: The MSTAR dataset was used to evaluate the performance of CA-ConTransNet. Taking the average recognition rate of 10 types of targets as the indicator, the recognition rate of CA-ConTransNet reached 99.61%. Compared with the classification experimental results based only on texture images (93.30%) and the classification experimental results based on original SAR images (95.06%), by fusing the feature information of the two modalities, the image recognition task can be better performed. The test performance of various classification methods was statistically analyzed. Compared with the current advanced model A-CNN (99.45%), the present invention achieved an improvement of 0.16%, which strongly proved the effectiveness of CA-ConTransNet.

[0057] In an embodiment of the present invention, an initial SAR image is acquired; the initial SAR image is preprocessed to obtain a first SAR image; texture features are extracted from the first SAR image to obtain a grayscale co-occurrence matrix, and contrast calculation is performed on the grayscale co-occurrence matrix to obtain a texture image; a target convolutional neural network is constructed, wherein the target convolutional neural network includes an input layer, a first feature extraction module, a second feature extraction module, five Trans blocks, five Conv blocks and a weighted average layer, the five Trans blocks include a first Trans block, a second Trans block, a third Trans block, a fourth Trans block and a fifth Trans block, and the first Trans block, the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block have the same structure, the five Conv blocks include a first Conv block, a second Conv block, a third Conv block, a fourth Conv block and a fifth Conv block, and the first Conv block, the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block have the same structure, each Trans block includes two layer normalizations, an MLP, a multi-head cross-attention feature fusion module and two fusion modules, and each Conv block includes four 1×1 projection convolutions, two The first SAR image is processed by the first feature extraction module to obtain the initial original features; the texture image is processed by the second feature extraction module to obtain the initial texture features; the initial original features are processed by five Trans blocks to obtain the first classification result; the initial texture features are processed by five Conv blocks to obtain the second classification result; the first classification result and the second classification result are processed by the weighted average layer to obtain the target classification result of the initial SAR image, which solves the technical problem of low classification accuracy when a single modality SAR image is used for classification in the prior art, and achieves the purpose of using the original image to characterize the image. GLCM is also used as the texture description of the image, and a novel multimodal feature fusion method is designed to learn and fuse the original CT image and GLCM image. The overall model adopts a dual-branch learning architecture to jointly learn the original CT image and texture features, and uses the designed Trans block to capture the contour representation and shape description of the original image, so that it can focus on the global structure and semantic information of the original CT image. The Conv block structure is designed to learn texture features and focus on local clues in the image, which helps to capture fine details and structures and facilitate the identification of texture differences of targets in different scenarios; focusing on the deep interaction of multimodal information, a multi-head cross-attention fusion mechanism is designed, which iteratively and effectively integrates the original image features and texture image features at various stages of model training, enhancing the technical effect of the feature representation capability used for classification.

[0058] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0059] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0060] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0062] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0063] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal feature fusion SAR image recognition method guided by ConTrans network based on cross attention, characterized in that: include: Acquire initial SAR image; Preprocessing the initial SAR image to obtain a first SAR image; Extracting texture features from the first SAR image to obtain a gray-level co-occurrence matrix, and performing contrast calculation on the gray-level co-occurrence matrix to obtain a texture image; Construct a target convolutional neural network, wherein the target convolutional neural network includes an input layer, a first feature extraction module, a second feature extraction module, five Trans blocks, five Conv blocks and a weighted average layer, the five Trans blocks include a first Trans block, a second Trans block, a third Trans block, a fourth Trans block and a fifth Trans block, the first Trans block, the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block have the same structure, the five Conv blocks include a first Conv block, a second Conv block, a third Conv block, a fourth Conv block and a fifth Conv block, the first Conv block, the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block have the same structure, each Trans block includes two layer normalizations, an MLP, a multi-head cross-attention feature fusion module and two fusion modules, each Conv block includes four 1×1 projection convolutions, two 3×3 spatial convolutions, two AW-Nets and three fusion modules, and the AW-Net includes a maximum pooling layer, an average pooling layer, an MLP, a fusion module and an activation layer; Processing the first SAR image by a first feature extraction module to obtain initial original features; The texture image is processed by a second feature extraction module to obtain initial texture features; The initial raw features are processed through five Trans blocks to obtain the first classification result; The initial texture features are processed through five Conv blocks to obtain the second classification result; The first classification result and the second classification result are processed by a weighted average layer to obtain the target classification result of the initial SAR image.

2. The method according to claim 1, characterized in that: The preprocessing of the initial SAR image to obtain the first SAR image includes: The initial SAR image is filtered by a mean filter to obtain a filtered SAR image; Take the center of the filtered SAR image as the coordinate origin and cut out a 128×128 target area; Scale the 128×128 target area to 64×64; The 64×64 target area is grayscale normalized to obtain the first SAR image.

3. The method according to claim 1, characterized in that The first SAR image is processed by the first feature extraction module to obtain initial original features, including: The first SAR image is sequentially input into a 7×7 convolutional layer, a 3×3 maximum pooling layer, and a projection layer to obtain the initial raw features.

4. The method according to claim 1, characterized in that The processing of the texture image by the second feature extraction module to obtain the initial texture features includes: The texture image is input into the 7×7 convolution layer and the 3×3 maximum pooling layer in sequence to obtain the initial texture features.

5. The method according to claim 1, characterized in that The initial original features are processed by five Trans blocks to obtain a first classification result, including: In the first Trans block, the initial raw features are input to the layer normalization to obtain the global representation of the initial raw features; Get a fine-grained local feature map of the initial texture features from the first Conv block; The global representation and fine-grained local feature map are input into the multi-head cross-attention feature fusion module to obtain the first fused feature; Inputting the first fused feature and the initial original feature into a fusion module to obtain a second fused feature; The second fused feature is sequentially input into layer normalization and MLP to obtain the third fused feature; Inputting the third fused feature and the second fused feature into a fusion module to obtain a first original feature; The first original feature is processed in sequence by the second Trans block, the third Trans block, the fourth Trans block and the fifth Trans block to obtain a first classification result.

6. The method according to claim 5, characterized in that The process of determining the fine-grained local feature map of the initial texture features of the first Conv block is: The initial texture features are input into the 1×1 projection convolution in sequence. The spatial convolution of Convolution, average pooling layers, and layer normalization are performed to obtain fine-grained local feature maps of the initial texture features.

7. The method according to claim 1, characterized in that The initial texture features are processed by five Conv blocks to obtain a second classification result, including: In the first Conv block, the initial texture features are sequentially input into 1×1 projection convolution, The spatial convolution and 1×1 projection convolution are used to obtain the first texture feature; Inputting the first texture feature into AW-Net to obtain a first adaptive weight of the initial texture feature; Inputting the first adaptive weight of the initial texture feature and the initial texture feature into a fusion module to obtain a first texture feature; Input the first texture feature into a 1×1 projection convolution to obtain a second texture feature; Get the fourth fused feature from the first Trans block; Inputting the second texture feature and the fourth fusion feature into a fusion module to obtain a fifth fusion feature; Input the fifth fusion feature into The spatial convolution, 1×1 projection convolution and AW-Net are used to obtain the second adaptive weight of the fifth fusion feature; Inputting the second adaptive weight and the first adaptive weight into a fusion module to obtain a first texture feature; The first texture feature is processed in sequence by the second Conv block, the third Conv block, the fourth Conv block and the fifth Conv block to obtain a second classification result.

8. The method according to claim 7, characterized in that The step of inputting the first texture feature into AW-Net to obtain a first adaptive weight of the initial texture feature includes: Input the first texture feature into the maximum pooling layer and the average pooling layer respectively to obtain the maximum pooling feature and the average pooling feature; Input the maximum pooling feature and the average pooling feature into MLP to obtain the maximum pooling texture feature and the average pooling texture feature; The maximum pooled texture features and the average pooled texture features are sequentially input into the fusion module and the activation layer to obtain the first adaptive weight.

9. The method according to claim 7, characterized in that: The process of determining the fourth fusion feature is as follows: The second fused feature is sequentially input into the image interpolation module, 1×1 convolution and batch normalization to obtain the fourth fused feature.

Images