Terahertz ISAR image target classification method based on space-frequency double-branch network

Abstract

The invention discloses a terahertz ISAR (Inverse Synthetic Aperture Radar) image target classification method based on a space-frequency double-branch network. The method comprises the following steps: performing data enhancement on a to-be-trained terahertz ISAR image, and dividing the to-be-trained terahertz ISAR image into a training set, a verification set and a test set; extracting initial features by using a stem module; the initial features are input into an MSFF network for multi-scale spatial feature and frequency feature extraction, so that the local feature perception capability and the global feature modeling capability of a model to a target are enhanced; splicing the spatial features and the frequency features, and inputting the spliced features to a classification layer for classification; carrying out gradient updating by calculating loss; and inputting a to-be-predicted terahertz ISAR image into the trained neural network, and outputting a classification result.

Description

Technical Field

[0001] The present invention relates to a radar image intelligent interpretation technology, and in particular to a terahertz ISAR image target classification method based on a space-frequency dual-branch network. Background Art

[0002] As a core technology for high-resolution radar imaging, Inverse Synthetic Aperture Radar (ISAR) has important application value in fields such as ocean monitoring, aerial reconnaissance, and space target identification due to its all-weather, long-range, and high-precision advantages. However, due to the unique imaging mechanism of ISAR images and the influence of measurement conditions, they are often accompanied by shortcomings such as small sample sizes and sensitivity to imaging parameters, making it difficult to effectively extract features, posing challenges to the automatic target classification (ATC) task based on ISAR images.

[0003] Current research on ISAR target classification primarily focuses on low-frequency data. Traditional target classification methods first design a feature extractor based on target characteristics, and then design a classifier for identification. However, these methods only achieve limited success under specific conditions. They often perform poorly when dealing with noise, blur, dynamic changes, and other issues in ISAR imagery, and lack scalability and robustness. With the development of artificial intelligence, various deep learning (DL) methods have also been applied to low-frequency ISAR target classification. Deep learning methods can automatically learn high-level features from data, eliminating the need for manually designed feature extractors and significantly improving target classification accuracy. However, existing research still primarily focuses on low-frequency ISAR imagery, with relatively little research on ISAR image classification in the terahertz band.

[0004] Compared to traditional low-frequency ISAR images, terahertz ISAR images exhibit significant advantages. They offer higher bandwidth, superior resolution, and exceptional sensitivity to target details and surface textures, with imaging quality approaching optical standards. They also perfectly preserve the inherent advantages of radar systems, enabling efficient operation even in complex environments such as long distances and high dynamic ranges.

[0005] Despite this, research on target classification in terahertz ISAR images is still in its early stages. Compared to lower-frequency data, terahertz ISAR images not only contain richer pixel information but also exhibit more complex scattering characteristics, which poses new challenges to the feature extraction and classification capabilities of existing target classification methods. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a terahertz ISAR image target classification method based on a space-frequency dual-branch network, aiming to improve the current situation of scarcity of terahertz ISAR image target classification methods, and at the same time solve the noise interference, dynamic change, and fine-grained classification problems of ISAR image target classification.

[0007] Technical solution: The terahertz ISAR image target classification method based on a space-frequency dual-branch network of the present invention comprises the following steps:

[0008] S1: Perform data augmentation on the terahertz ISAR images to be trained and then divide them into training set, validation set, and test set;

[0009] S2: Use the stem module to extract initial features;

[0010] S3: Input the initial features into the MSFF network to extract multi-scale spatial features and frequency features, thereby enhancing the model's ability to perceive local features of the target and model global features;

[0011] S4: splice the spatial features and frequency features and input them into the classification layer for classification;

[0012] S5: Update the gradient by calculating the loss;

[0013] S6: Input the terahertz ISAR image to be predicted into the trained neural network and output the classification result.

[0014] Furthermore, the operation steps of step S1 are:

[0015] Data augmentation was first performed on the terahertz ISAR images to be trained. The augmented dataset was then divided into a training set and a test set, with a training set to test set ratio of 0.7 and a validation set ratio of 0.2 to the training set. The augmentation methods included rotation, upside-down flipping, left-right flipping, adding Gaussian and salt-and-pepper noise, and performing brightness, contrast, saturation, and hue conversion operations.

[0016] Furthermore, the operation steps of step S2 are:

[0017] The input terahertz ISAR image is resized to 128×128 and then input into the stem module for extracting initial features. The stem module includes a 7×7 standard two-dimensional convolution with a stride of 1, followed by a 3×3 maximum pooling layer with a stride of 2.

[0018] Furthermore, the operation steps of step S3 are:

[0019] This space-frequency dual-branch network for terahertz ISAR image target classification is called the MSFF network. The MSFF network consists of a space branch and a frequency branch. The space branch includes a three-level multi-scale MSFE module, and the frequency branch includes a three-level multi-scale FDCA module. Both the MSFE module and the FDCA module are connected to a downsampling module, which includes a convolution block with a convolution kernel of 3, a stride of 2, and zero padding.

[0020] The MSFE module first uses 1×1 point-by-point convolution for dimensionality increase, and divides the data into three groups along the channel dimension. The data then enters two-dimensional 3×3, 5×5, and 7×7 depthwise separable convolutions, respectively. After each convolution layer, the GELU function is connected to splice the features and pass them through 1×1 depthwise separable convolution.

[0021] The FDCA module inputs the input feature map to the FDA module. The FDA module converts the image into the frequency domain through discrete Fourier transform, then uses 1×1 convolution, introduces the batch normalization BN layer, followed by the ReLU activation function, and then outputs to the MLP layer. The MLP layer contains two fully connected layers.

[0022] Furthermore, the operation steps of step S4 are:

[0023] The extracted multi-scale spatial features and multi-scale frequency features are concatenated and input into the classification layer for classification. The classification layer contains an adaptive average pooling layer and a fully connected layer.

[0024] Furthermore, the operation steps of step S5 are:

[0025] For the training dataset, the model is optimized using a loss function. The weights are saved after each iteration, and the optimal weights of the deep neural network are obtained through multiple training iterations. Gradient updates are performed using the cross-entropy loss function. The Adam optimizer is used for the training loss, and the learning rate is adjusted using a warmup plus cosine annealing strategy.

[0026] Furthermore, the operation steps of step S6 are:

[0027] The designed neural network is used to train the training data set, and then the terahertz ISAR image to be predicted is input into the trained neural network to output the classification result;

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention proposes a terahertz ISAR image target classification method based on a space-frequency dual-branch network. First, the terahertz ISAR image to be trained is subjected to data augmentation and then divided into a training set, a validation set, and a test set. The stem module is then used to extract initial features. The initial features are input into the MSFF network to extract multi-scale spatial and frequency features, thereby enhancing the model's ability to perceive local features of the target and model global features. The spatial and frequency features are then spliced ​​and input into the classification layer for classification.

[0030] Gradient updates are performed by calculating the loss; the terahertz ISAR image to be predicted is input into the trained neural network, and the classification result is output. This paper proposes a multi-scale spatial-frequency dual-branch network to achieve high-accuracy terahertz ISAR image target classification. Compared with existing technologies, this network more effectively integrates the multi-scale spatial and frequency features of the target, improving the classification performance of terahertz ISAR images with large attitude variations. The spatial branch of the proposed MSFF network can more effectively extract local spatial features at different scales in terahertz ISAR images. Furthermore, the introduction of the GELU function in the feature extraction process effectively avoids the loss of small target features during processing, further enriching the image's feature information. The frequency branch of the proposed MSFF network captures global information through frequency domain transformation, enhancing the model's ability to model the multi-scale global features of the target and significantly reducing the model's computational complexity. Therefore, this paper can improve the current scarcity of terahertz ISAR image target classification methods, addressing the noise interference, dynamic changes, and fine-grained classification issues in existing terahertz ISAR image target classification problems, improving the accuracy of terahertz ISAR image target classification, and reducing the model's computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the method of the present invention;

[0032] Figure 2 Schematic diagram of the MSFE module in the present invention;

[0033] Figure 3 Schematic diagram of the FDCA module in the present invention;

[0034] Figure 4 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] The terahertz ISAR image target classification method based on the space-frequency dual-branch network proposed in this invention is as follows: Figure 1 As shown in the figure, the terahertz ISAR image is first input into the stem module for initial feature extraction. It then passes through a multiscale spatial-frequency parallel dual-branch structure constructed in three hierarchical stages. The multiscale spatial branch, primarily composed of the MSFE module, uses multiscale convolution kernels to extract local spatial features from the terahertz ISAR image, enhancing the perception of the target's structure at different scales. The multiscale frequency branch, primarily composed of the FDCA module, converts the image to the frequency domain through frequency domain transformation to extract the target's global scattering pattern, enhancing global information modeling capabilities and thus improving classification performance.

[0037] Example 1

[0038] See also Figure 1 As shown, the terahertz ISAR image target classification method based on the space-frequency dual-branch network described in this embodiment includes the following steps:

[0039] S1: First perform data augmentation on the terahertz ISAR to be trained and then divide it into training set, validation set, and test set;

[0040] The operation steps of step S1 are:

[0041] The terahertz ISAR images to be trained are first enhanced, including rotation, upside-down flipping, left-right flipping, adding Gaussian noise and salt-and-pepper noise, brightness conversion, contrast conversion, saturation conversion, and hue conversion operations;

[0042] The enhanced terahertz ISAR images to be trained are divided into a training set and a test set. The ratio of the training set to the test set is 0.7, and the ratio of the validation set is 0.2 of the training set.

[0043] S2: Use the stem module to extract initial features;

[0044] The operation steps of step S2 are:

[0045] The input terahertz ISAR image is resized to 128×128 and then input into the stem module for extracting initial features. The stem module includes a 7×7 standard two-dimensional convolution with a stride of 1, followed by a 3×3 maximum pooling layer with a stride of 2.

[0046] S3: Input the initial features into the MSFF network to extract multi-scale spatial features and frequency features, thereby enhancing the model's ability to perceive local features of the target and model global features;

[0047] The operation steps of step S3 are:

[0048] This space-frequency dual-branch network for terahertz ISAR image target classification is called the MSFF network. The MSFF network includes a space branch and a frequency branch. The space branch includes a three-level multi-scale MSFE module, and the frequency branch includes a three-level multi-scale FDCA module.

[0049] The MSFE module first uses 1×1 point-by-point convolution for dimensionality increase, and divides the data after dimensionality increase into three groups along the channel dimension, which enter the two-dimensional 3×3, 5×5, and 7×7 depth-wise separable convolution respectively. After each convolution layer, the GELU function is connected to splice the features and pass the 1×1 depth-wise separable convolution. The expression is as follows:

[0050] X S =DS conv(σ·Conv multi (Split(PW conv(X t )))

[0051] Among them, X t Represents the feature map output by the stem module, X S It represents the feature map output by the multi-scale MSFE module after three levels of stages, PW conv represents point-by-point convolution, Split represents the division operation, Conv multi represents a multi-scale convolution operation consisting of two-dimensional 3×3, 5×5, and 7×7 depth-wise separable convolutions, and σ represents the GELU activation function;

[0052] The FDCA module inputs the input feature map to the FDA module. The FDA module converts the image to the frequency domain through discrete Fourier transform, then uses 1×1 convolution, introduces a batch normalization (BN) layer, followed by a ReLU activation function, and then outputs it to the MLP layer, which contains two fully connected layers.

[0053] For the feature X output by the stem module t ∈R h×w×C(h is the height of the feature map, w is the width of the feature map, and C is the number of channels of the feature map) to perform a discrete Fourier transform (DFT) to obtain the corresponding frequency representation. The specific formula of this process is as follows:

[0054]

[0055] Among them, u and v are the coordinates in Fourier space, X F (u, v) are the features X at different spatial locations t (h, w) corresponding to different frequency components;

[0056] In the frequency domain, the discrete Fourier transform transforms the input feature map X t The real and imaginary components are converted. A convolution operation is then performed using a 1×1 convolution kernel. After modulation, the real and imaginary components are separated, and the frequency domain is converted back to the spatial domain using the inverse DFT. This is then transferred to the Multilayer Perceptron (MLP) layer for the next step. The formula for the above process is as follows:

[0057] X R ,X I =DFT(X t )

[0058]

[0059] Among them, X R , X I The input feature map X is transformed into discrete Fourier transform t The real and imaginary components are converted into, Conv is 1×1 convolution, ConCat is the channel splicing operation, are the learned real and imaginary components, Inv DFT is the inverse discrete Fourier transform, BN is the batch normalization (BN) layer, and RELU is the ReLU activation function;

[0060] The structure of the MSFF network is shown in Table 1.

[0061] Table 1: Schematic diagram of the MSFF network structure

[0062]

[0063] S4: splice the spatial features and frequency features and input them into the classification layer for classification;

[0064] The operation steps of step S4 are:

[0065] The extracted multi-scale spatial features and multi-scale frequency features are concatenated and input to the classification layer for classification. The classification layer consists of an adaptive average pooling layer and a fully connected layer. The feature map output by the adaptive average pooling layer is 1×1 in size. The output feature map of the adaptive average pooling layer is input to the fully connected layer, which is a linear layer.

[0066] S5: Update the gradient by calculating the loss;

[0067] The operation steps of step S5 are:

[0068] For the training data set, the cross entropy loss function is used for gradient update. The Adam optimizer is used for the training loss, and the Warmup plus cosine annealing strategy is used to adjust the learning rate. The weight is saved once after each iteration, and the training iteration is repeated multiple times to obtain the optimal weight of the deep neural network.

[0069] S6: Input the terahertz ISAR image to be predicted into the trained neural network and output the classification result.

[0070] The operation steps of step S6 are:

[0071] The designed neural network is used to train the training data set, and then the terahertz ISAR image to be predicted is input into the trained neural network to output the classification result;

[0072] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A terahertz ISAR image target classification method based on a space-frequency dual-branch network, comprising the following steps: S1: Perform data augmentation on the terahertz ISAR images to be trained and then divide them into training set, validation set, and test set; S2: Use the stem module to extract initial features; S3: Input the initial features into the MSFF network to extract multi-scale spatial features and frequency features, thereby enhancing the model's ability to perceive local features of the target and model global features; S4: splice the spatial features and frequency features and input them into the classification layer for classification; S5: Update the gradient by calculating the loss; S6: Input the terahertz ISAR image to be predicted into the trained neural network and output the classification result.

2. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 1, characterized in that: The operation steps of step S1 are: The terahertz ISAR images to be trained are first data enhanced, and the enhanced dataset is divided into a training set and a test set. The ratio of the training set to the test set is 0.7, and the ratio of the validation set is 0.2 of the training set.

3. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 2, characterized in that: The enhancement method includes rotation, upside-down flipping, left-right flipping, adding Gaussian noise and salt-and-pepper noise, brightness transformation, contrast transformation, saturation transformation, and hue transformation operations.

4. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 3, characterized in that: The operation steps of step S2 are: adjusting the input terahertz ISAR image to 128×128, and then inputting it into the stem module for extracting initial features. The stem module includes a 7×7 standard two-dimensional convolution with a stride of 1, followed by a 3×3 maximum pooling layer with a stride of 2.

5. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 4, characterized in that: The operation steps of step S3 are as follows: the MSFF network includes a spatial branch and a frequency branch, the spatial branch includes a three-level multi-scale MSFE module, and the frequency branch includes a three-level multi-scale FDCA module. A downsampling module is connected after the MSFE module and the FDCA module, including a convolution block with a convolution kernel of 3, a step size of 2, and a padding of 0; The MSFE module first uses 1×1 point-by-point convolution to increase the dimension, and divides the data after the dimension increase into three groups along the channel dimension, which enter the two-dimensional 3×3, 5×5, and 7×7 depth-separable convolution respectively. After each convolution layer, the GELU function is connected to splice the features and pass the 1×1 depth-separable convolution. The expression is as follows: X S =DS conv(σ·Conv multi (Split(PW conv(X t ))) Among them, X t Represents the feature map output by the stem module, X S It represents the feature map output by the multi-scale MSFE module after three levels of stages, PW Conv represents point-by-point convolution, Split represents the division operation, Conv multi represents a multi-scale convolution operation consisting of two-dimensional 3×3, 5×5, and 7×7 depth-wise separable convolutions, and σ represents the GELU activation function; The FDCA module inputs the input feature map to the FDA module. The FDA module converts the image to the frequency domain through discrete Fourier transform, then uses 1×1 convolution, introduces a batch normalization (BN) layer, followed by a ReLU activation function, and then outputs it to the MLP layer, which contains two fully connected layers. For the feature X output by the stem module t ∈R h×w×C (h is the height of the feature map, w is the width of the feature map, and C is the number of channels of the feature map), and a discrete Fourier transform (DFT) is performed to obtain the corresponding frequency representation. The specific formula of this process is as follows: Among them, u and v are the coordinates in Fourier space, X F (u, v) are the features X at different spatial locations t (h, w) corresponding to different frequency components; In the frequency domain, the discrete Fourier transform transforms the input feature map X t The real and imaginary components are converted. A convolution operation is then performed using a 1×1 convolution kernel. After modulation, the real and imaginary components are separated, and the frequency domain is converted back to the spatial domain using the inverse DFT. This is then transferred to the multi-layer perceptron layer for further processing. The formula for the above process is as follows: X R ,X I =DFT(X t ) Among them, X R ,X I The input feature map X is transformed into discrete Fourier transform t The real and imaginary components are converted, Concat is the channel splicing operation, Conv is 1×1 convolution, are the learned real and imaginary components, Inv DFT is the inverse discrete Fourier transform, BN is the batch normalization layer, and RELU is the ReLU activation function.

6. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 5, characterized in that: The operation steps of step S4 are: splicing the extracted multi-scale spatial features and multi-scale frequency features, and inputting them into the classification layer for classification. The classification layer includes an adaptive average pooling layer and a fully connected layer.

7. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 6, characterized in that: The operation steps of step S5 are: for the training data set, using the loss function to optimize the model, saving the weight once after each iteration, and training iteratively multiple times to obtain the optimal weight of the deep neural network.

8. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 7, characterized in that: The cross entropy loss function is used for gradient update, the Adam optimizer is used for training loss, and the warmup plus cosine annealing strategy is used to adjust the learning rate.

9. The terahertz ISAR image target classification method based on a space-frequency dual-branch network according to claim 8, characterized in that: The operation steps of step S6 are: training the training data set using the designed neural network, then inputting the terahertz ISAR image to be predicted into the trained neural network, and outputting the classification result.

Images