Sequence ISAR image-oriented spatio-temporal joint target form change identification method

Abstract

The invention discloses a sequence ISAR image-oriented spatio-temporal joint target morphological change recognition method, which utilizes the advantage that an ISAR image has time sequence, and designs a pre-trained single-frame feature extraction network and a pre-trained time sequence feature recognition network. The method achieves the recognition of whether the time sequence sample of the ISAR image has morphological change or not and when the time sequence sample starts morphological change, reduces the mutual shielding between parts and the influence of electromagnetic scattering anisotropy on the morphological change image recognition rate, and enables the classification recognition result to be higher in precision. The introduced semi-wavelet residual feature extraction layer improves the perceptual ability of the model to the high and low frequency features of the ISAR image, the calculation efficiency and the perceptual ability to the target area; through the introduced gating linear attention unit, the inference capability of a linear attention mechanism on a long-time sequence and the information forgetting capability of the gating linear unit are combined, and the perception recognition capability of the model on the time sequence characteristics of the ISAR image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and recognition, and particularly relates to a spatio-temporal joint target morphological change recognition method for sequential ISAR images. Background Art

[0002] Inverse Synthetic Aperture Radar (ISAR) is widely used in military and civilian fields such as space target surveillance and situation awareness due to its all-weather, all-day, long-distance, and high-resolution imaging capabilities. Currently, ISAR imaging technology has been relatively mature, and can obtain well-focused high-resolution imaging results, providing important technical support for the morphological structure analysis of space targets. With the continuous increase in the number and types of space targets in recent years, the accurate and rapid monitoring of the morphological structure changes of space targets is of great significance for on-orbit health status monitoring of space targets. However, in actual monitoring scenarios, due to the mutual occlusion between components in ISAR images, it is easy to produce misjudgments when recognizing morphological changes in ISAR images with key components occluded, especially when the occluded components have morphological changes. At the same time, due to the anisotropy of ISAR images, there will be obvious pixel brightness differences on the images, which will have an adverse impact on morphological change recognition. In fact, the viewing angle of the ISAR image of a target monitored by radar for a long time tends to change continuously, which endows the ISAR image with temporal sequence. When a key component in the ISAR image is occluded or there are missing pixels, through the change of the viewing angle, the occluded component will be revealed, and the pixels may also be gradually completed. For a space target with morphological changes, after the viewing angle changes, the component will still be in a morphological change state such as defect. At the same time, by combining the state viewing angle change of the front and rear frames of the ISAR image in this way, the front and rear states of the key component can be compared to determine which frame in a temporal sequence sample has morphological changes, so as to realize the recognition of morphological changes in sequential ISAR images.

[0003] The problem of recognizing the morphological changes of ISAR images is actually to perceive whether the target structure presented by the ISAR images has changed. Essentially, it is consistent with the image change problem in computer vision. Therefore, the relevant methods of image change can be applied to the recognition of the morphological changes of ISAR images. In the past period of time, deep learning networks have made great progress, and their performance has been significantly improved, especially in the field of image recognition. The image change recognition method based on deep learning uses deep learning technology to automatically learn image features and recognizes image changes by modeling complex relationships. Such methods have made remarkable progress in recent years, especially in dealing with high-dimensional data, complex changes, and multi-source data fusion. For the recognition of the morphological changes of ISAR images, this method can accurately perceive the changes between the front and back frames of ISAR images by modeling the temporal relationship of multiple frames of ISAR images, thereby improving the correct rate of ISAR morphological change recognition.

[0004] Image change recognition is an important research direction in the fields of computer vision and remote sensing. It aims to identify the changed regions or targets by analyzing two or more images of the same scene at different times or under different conditions. This technology is widely used in many fields, such as climate change analysis in environmental monitoring, urban expansion in remote sensing, lesion detection in medical imaging, and industrial product quality inspection, etc. In the research on the morphological change recognition of spatial target ISAR images, the main focus is on whether the target has undergone morphological changes. Existing image change recognition methods can generally be divided into two categories: traditional image change recognition methods and deep learning-based image change recognition methods. Deep learning-based image change recognition methods use deep learning techniques to extract image features and identify image changes by modeling complex relationships. They can basically be divided into methods based on convolutional neural networks, methods based on Transformers, and methods based on unsupervised / semi-supervised learning. Methods based on convolutional neural networks mainly utilize the ability of CNNs to extract image features and combine different network structures to achieve the purpose of image change recognition. Methods based on Transformers are a popular research direction in the field of deep learning in recent years. They capture long-term dependencies in images through the attention mechanism to complete the recognition of image changes. Methods based on unsupervised / semi-supervised learning detect image changes by mining the internal laws of images themselves. These methods have made significant progress in recent years, especially in dealing with high-dimensional data, complex changes, and multi-source data fusion. However, these techniques have the following disadvantages respectively: Convolutional neural network-based image change recognition methods such as commonly used convolutional neural networks like CNN and VGG have simple network models, cannot associate spatio-temporal information, and cannot complete the recognition of sequential ISAR images. And like two-stream networks and Siamese networks, they are essentially a two-branch network that needs to compare the features of two images to obtain results, which will greatly increase the computational complexity of the network, and the network's ability to handle complex scenarios is limited. Transformer-based methods such as ViT focus more on changes within a single image and have limited ability to connect multiple frames of images. And some Transformer methods that fuse spatio-temporal information often require a large amount of labeled data. At the same time, due to the existence of the self-attention mechanism, the computational complexity of the model is relatively large, and a large amount of computing resources are required. Unsupervised / semi-supervised learning-based image change recognition methods, although they significantly reduce the dependence on labeled data by mining the internal laws of data, their accuracy is weaker than that of supervised methods, and the computational and implementation complexity is also higher than that of common supervised methods. In summary, existing methods are difficult to implement on the sequential ISAR morphological change image dataset due to reasons such as the inability to integrate spatio-temporal information or high computational complexity. Summary of the Invention

[0005] To solve the above problems existing in the prior art, the present invention provides a spatio-temporal joint target morphological change recognition method for sequential ISAR images. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] The present invention provides a spatio-temporal joint target morphological change recognition method for sequential ISAR images, and the method includes:

[0007] Generating an ISAR image dataset S according to the original spatial target model and its morphological change spatial target model through the polar coordinate format algorithm; processing the ISAR image dataset S by using a timing method to obtain an ISAR morphological change image timing dataset S';

[0008] Processing the ISAR morphological change image timing dataset S' by using a pre-trained ISAR morphological change image recognition model to obtain a morphological change result; the morphological change result includes whether there is a morphological change and the time when the morphological change starts; wherein, the pre-trained ISAR morphological change image recognition model is trained by using the ISAR image dataset S and the ISAR morphological change image timing dataset S', and the pre-trained ISAR morphological change image recognition model includes: a pre-trained single-frame feature extraction network SFExt, a pre-trained timing feature recognition network TFRec, and a network output layer connected in sequence.

[0009] In an embodiment of the present invention, processing the ISAR image dataset S by using a timing method to obtain an ISAR morphological change image timing dataset S' includes:

[0010] Sliding a window with a length of L on the ISAR image dataset S at a speed of 1 frame / time, obtaining a timing sample each time it slides, and forming the ISAR morphological change image timing dataset S' from all the timing samples.

[0011] In an embodiment of the present invention, the training process of the pre-trained ISAR morphological change image recognition model includes:

[0012] Training the single-frame feature extraction network by using the ISAR image dataset S to obtain a pre-trained single-frame feature extraction network SFExt;

[0013] Inputting the pre-processed ISAR morphological change image timing dataset S' into the pre-trained single-frame feature extraction network SFExt to obtain a timing feature vector S''; training the timing feature recognition network by using the timing feature vector S'' to obtain a pre-trained timing feature recognition network TFRec;

[0014] The network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec.

[0015] In an embodiment of the present invention, the single-frame feature extraction network includes:

[0016] A first convolutional layer, a batch normalization layer BN, a rectified linear unit ReLU, a max pooling layer MAXpool, a semi-wavelet residual feature extraction layer HWRes, and a double-branch structure connected in sequence; wherein, the double-branch structure includes 3 double-branch sampling substructures connected in sequence, and each double-branch sampling substructure includes: a semi-wavelet residual feature extraction layer as the upper branch, a second convolutional layer and a batch normalization layer as the lower branch.

[0017] In an embodiment of the present invention, the training process of the pre-trained single-frame feature extraction network SFExt includes:

[0018] Preprocess the ISAR image dataset S to obtain the preprocessed ISAR image;

[0019] Set a binary classification fully-connected layer at the end of the single-frame feature extraction network to output the classification result of whether there is a morphological change in a single image;

[0020] Set a single-frame feature training strategy, and train the single-frame feature extraction network a preset number of times according to the preprocessed ISAR image to obtain the trained single-frame feature extraction network; during the training process, use the Adam optimizer to update and learn the network parameters of the single-frame feature extraction network, and the learning rate update strategy is StepLR;

[0021] Remove the binary classification fully-connected layer in the trained single-frame feature extraction network to obtain the pre-trained single-frame feature extraction network SFExt.

[0022] In an embodiment of the present invention, a single training process of the single-frame feature extraction network includes:

[0023] Use the downsampling operation Down(·) composed of the first convolutional layer, the batch normalization layer BN, the rectified linear unit ReLU, the max pooling layer MAX pool, and the semi-wavelet residual feature extraction layer HWRes to perform preliminary feature extraction on the preprocessed ISAR image to obtain the main feature map;

[0024] Use the upper branch and the lower branch in the double-branch structure to downsample the main feature map to obtain the output feature map;

[0025] Unfold the output feature map into a one-dimensional feature vector, and splice the one-dimensional feature vector into a temporal feature according to the temporal relationship; use the temporal feature of the current training process as the input corresponding to the next training process.

[0026] In one embodiment of the present invention, the temporal feature recognition network includes:

[0027] A first normalization layer, a gated linear attention unit GLAU, a second normalization layer, and a multi-layer perception mechanism MLP connected in sequence; wherein,

[0028] The gated linear attention unit GLAU is a temporal data processing unit that combines a linear attention mechanism and a gating mechanism;

[0029] The multi-layer perception mechanism MLP is a combination of fully connected layers.

[0030] In one embodiment of the present invention, the training process of the pre-trained temporal feature recognition network TFRec includes:

[0031] According to the temporal relationship of the temporal samples in the ISAR morphological change image temporal dataset S′, splice the temporal feature vectors S″ to obtain a spliced temporal image feature vector;

[0032] Set a binary classification fully connected layer at the end of the temporal feature recognition network to output the classification result of whether the temporal sample has a morphological change;

[0033] Set a temporal feature training strategy, and after training the temporal feature recognition network a preset number of times according to the spliced temporal image feature vector, obtain the trained temporal feature recognition network; during the training process, use the Adam optimizer to update and learn the network parameters of the temporal feature recognition network, and the learning rate update strategy is StepLR;

[0034] Use the trained temporal feature recognition network as the pre-trained temporal feature recognition network TFRec.

[0035] In one embodiment of the present invention, a single training process of the temporal feature recognition network includes:

[0036] Process the spliced temporal image feature vector using the residual structure composed of the first normalization layer and the gated linear attention unit GLAU in series to obtain the temporal feature of the first stage of temporal recognition;

[0037] Process the temporal feature of the first stage of temporal recognition using the residual structure composed of the second normalization layer and the multi-layer perception mechanism MLP in series to obtain the temporal feature of the second stage of temporal recognition; use the temporal feature of the second stage of temporal recognition in the current training process as the input corresponding to the next training process.

[0038] In one embodiment of the present invention, the network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, including:

[0039] The temporal sample classification layer in the network output layer determines whether there is a morphological change in the temporal sample according to the output result of the pre-trained temporal feature recognition network TFRec:

[0040] If there is a morphological change, the change frame classification layer of itself is used to determine the frame at which the morphological change starts in the temporal sample with the morphological change according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, and according to the frame at which the morphological change starts, the occurrence of the morphological change and the time when the morphological change starts are output as the morphological change result;

[0041] If there is no morphological change, "no morphological change" is output as the morphological change result.

[0042] Advantages of the present invention:

[0043] In the solution provided by the present invention, the advantage that the ISAR image has temporality is utilized. By designing the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, an architecture combining feature extraction and temporal recognition is obtained, realizing the recognition of whether there is a morphological change in the temporal sample of the ISAR image and when the morphological change starts in the temporal sample, reducing the influence of mutual occlusion between components and the anisotropy of electromagnetic scattering on the recognition rate of the ISAR morphological change image, so that the classification and recognition result has higher accuracy. Further, the semi-wavelet residual feature extraction layer HWRes introduced in the pre-trained single-frame feature extraction network SFExt utilizes its own advantages in image feature extraction to improve the model's perception ability of high and low frequency features of the ISAR image, the model calculation efficiency, and the model's perception ability for the target area; the gated linear attention unit GLAU introduced in the pre-trained temporal feature recognition network TFRec combines the reasoning ability of the linear attention mechanism for long time series and the information forgetting ability of the gated linear unit, improving the model's perception and recognition ability of the temporal features of the ISAR image. Description of the Drawings

[0044] Figure 1 It is a schematic diagram of the steps of a spatio-temporal joint target morphological change recognition method for sequence ISAR images provided by an embodiment of the present invention;

[0045] Figure 2Schematic diagram of the structure of a pre-trained ISAR morphological change image recognition model provided by an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the structure of the pre-trained single-frame feature extraction network SFExt in a pre-trained ISAR morphological change image recognition model provided by an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of the structure of the half-wavelet residual feature extraction layer HWRes in a pre-trained single-frame feature extraction network SFExt provided by an embodiment of the present invention;

[0048] Figure 5 Schematic diagram of the structure of the pre-trained temporal feature recognition network TFRec in a pre-trained ISAR morphological change image recognition model provided by an embodiment of the present invention;

[0049] Figure 6 Schematic diagram of the structure of the gated linear attention unit GLAU in a pre-trained temporal feature recognition network TFRec provided by an embodiment of the present invention;

[0050] Figures 7a - 7d Schematic diagram of the target model in the simulation experiment of a spatio-temporal joint target morphological change recognition method for sequential ISAR images provided by an embodiment of the present invention;

[0051] Figure 8 Schematic diagram of the confusion matrix in the simulation of a spatio-temporal joint target morphological change recognition method for sequential ISAR images provided by an embodiment of the present invention;

[0052] Figures 9a - 9c Schematic diagram of the training set loss function and the test set confusion matrix in the simulation of a spatio-temporal joint target morphological change recognition method for sequential ISAR images provided by an embodiment of the present invention. Specific embodiments

[0053] The following further describes the present invention in detail with specific embodiments, but the embodiments of the present invention are not limited thereto.

[0054] In order to achieve the purpose of solving the problems of mutual occlusion between components of ISAR images and electromagnetic scattering anisotropy, an embodiment of the present invention provides a spatio-temporal joint target morphological change recognition method for sequential ISAR images, as Figure 1 shown, which may include:

[0055] S1. Generate an ISAR image dataset S according to the original space target model and its morphological change space target model by using the polar format algorithm; process the ISAR image dataset S by using a temporalization method to obtain an ISAR morphological change image temporal dataset S'.

[0056] Regarding S1, generating the ISAR image dataset S according to the original space target model and its morphological change space target model by using the polar format algorithm may include:

[0057] Generate an ISAR image without morphological change and an ISAR image with morphological change according to the original space target model and the morphological change space target model corresponding to the original space target model by using the polar format algorithm (PFA).

[0058] Obtain a continuously changing ISAR image dataset S according to the ISAR image without morphological change and the ISAR image with morphological change.

[0059] Specifically, it is realized by a fixed radar during imaging, enabling the target to rotate around an axis to obtain a continuously changing ISAR image dataset S.

[0060] Regarding S1, processing the ISAR image dataset S by using a temporalization method to obtain an ISAR morphological change image temporal dataset S' includes:

[0061] Slide a window with a length of L on the ISAR image dataset S at a speed of 1 frame per time. Each time it slides, a temporal sample is obtained. All the temporal samples constitute the ISAR morphological change image temporal dataset S'.

[0062] Specifically, the length L of the window is also the number of image frames included in a temporal sample. For a dataset with X frames of images, a total of X - L + 1 temporal samples can be obtained. In this way, the ISAR morphological change image temporal dataset S' can be obtained, which includes an ISAR image sample temporal dataset without morphological change and an ISAR image sample temporal dataset with morphological change. For the ISAR image sample temporal dataset with morphological change, it is necessary to replace its [1, m], m ≤ L / 2 frames of images with the ISAR image samples without morphological change corresponding to the rotation angles, so that a transition from without morphological change to with morphological change occurs in a temporal sample to simulate the actual space target morphological change monitoring scenario.

[0063] S2. Process the ISAR morphological change image time series dataset S' using the pre-trained ISAR morphological change image recognition model to obtain the morphological change result. The morphological change result includes whether there is a morphological change and the time when the morphological change starts. Among them, the pre-trained ISAR morphological change image recognition model is trained using the ISAR image dataset S and the ISAR morphological change image time series dataset S'. The pre-trained ISAR morphological change image recognition model includes: a pre-trained single-frame feature extraction network SFExt, a pre-trained time-series feature recognition network TFRec, and a network output layer connected in sequence.

[0064] In the embodiment of the present invention, constructing the pre-trained ISAR morphological change image recognition model is the key to accurate morphological change recognition.

[0065] The pre-trained ISAR morphological change image recognition model is a recognition model based on a feature extraction + time-series recognition architecture. As Figure 2 shown, it mainly includes three parts, from left to right are the pre-trained single-frame feature extraction network SFExt (Single-frame Feature Extraction), the pre-trained time-series feature recognition network TFRec (Time-sequence Feature Recognition), and the network output layer. Assume that the overall input of the pre-trained ISAR morphological change image recognition model is u loop time series samples, and each loop time series sample contains m continuously changing ISAR images. The value of m is equal to the length L of the window in S1. These images are used as the input of the pre-trained single-frame feature extraction network SFExt. After passing through this network, each image will obtain a one-dimensional single-frame feature vector with a length of q. The input of the pre-trained time-series feature recognition network TFRec is the time-series feature vector obtained by splicing the one-dimensional feature vectors of each loop time series sample in time series, that is, u groups of m×q time-series features, and the output is u groups of one-dimensional time-series feature vectors. The network output layer includes two branches. The time-series feature vector first passes through the first time-series sample classification layer (TsSoftmax) to determine whether there is a morphological change in the time-series sample. If there is a morphological change in the time-series sample, then execute the upper branch of the network output layer. The single-frame feature vector output by the pre-trained single-frame feature extraction network SFExt is integrated and passed through the change frame classification layer (ExSoftmax) to determine from which frame the morphological change starts in the time-series sample with morphological changes. If there is no morphological change in the time-series sample, then execute the lower branch of the network output layer and directly output the result.

[0066] Specifically, the network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained time-series feature recognition network TFRec, which may include:

[0067] The temporal sample classification layer in the network output layer determines whether there is a morphological change in the temporal sample according to the output result of the pre-trained temporal feature recognition network TFRec:

[0068] If there is a morphological change, the change frame classification layer of itself determines the frame at which the morphological change starts in the temporal sample with the morphological change according to the output result of the pre-trained single-frame feature extraction network SFExt and the output result of the pre-trained temporal feature recognition network TFRec, and outputs the time when the morphological change occurs and the morphological change starts as the morphological change result according to the frame at which the morphological change starts;

[0069] If there is no morphological change, it outputs no morphological change as the morphological change result.

[0070] The training process of the pre-trained ISAR morphological change image recognition model includes:

[0071] Using the ISAR image dataset S to train the single-frame feature extraction network to obtain the pre-trained single-frame feature extraction network SFExt;

[0072] Inputting the pre-processed ISAR morphological change image temporal dataset S′ into the pre-trained single-frame feature extraction network SFExt to obtain the temporal feature vector S″; using the temporal feature vector S″ to train the temporal feature recognition network to obtain the pre-trained temporal feature recognition network TFRec;

[0073] The network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec.

[0074] The single-frame feature extraction network, as Figure 3 shown, may include:

[0075] A first convolutional layer, a batch normalization layer BN, a rectified linear unit ReLU, a max pooling layer MAXpool, a semi-wavelet residual feature extraction layer HWRes, and a double-branch structure connected in sequence; wherein, the double-branch structure includes 3 double-branch sampling sub-structures connected in sequence, and each double-branch sampling sub-structure includes: a semi-wavelet residual feature extraction layer as the upper branch, a second convolutional layer and a batch normalization layer as the lower branch.

[0076] The overall structure of the single-frame feature extraction network is similar to the residual structure of the Residual Network (ResNet). For the input pre-processed ISAR image The downsampling operation Down(·) consisting of a first convolutional layer of 7×7, a batch normalization layer BN, a rectified linear unit ReLU, a max pooling layer MAX pool, and a half-wavelet residual feature extraction layer HWRes is used to perform the preliminary feature extraction on I M to obtain the main feature map represents the resolution of an image or image feature, where H and W represent the length and width dimensions of the image respectively. Among them, the output of the first convolutional layer of 7×7 is The output of the max pooling layer MAX pool is Then the preliminary feature extraction process can be expressed as follows:

[0077]

[0078] After obtaining the main feature map, the network will pass through a dual-branch structure composed of three groups of dual-branch substructures. The dual-branch substructure can include: a convolutional layer, a ReLU layer, and a HWRes feature extraction layer. In each group of the dual-branch structure, the upper branch and the lower branch will perform downsampling on the main feature map, and finally the upper and lower branches will be added together. Taking the first group of dual-branch substructures as an example, its calculation process can be expressed as follows:

[0079] M2 = BN(Conv(M1)) + HWRes(M1);

[0080] Among them, Finally, the feature map output by the pre-trained single-frame feature extraction network SFExt can be expressed as and it is unfolded into a one-dimensional feature vector.

[0081] Among them, the half-wavelet residual feature extraction layer HWRes, as Figure 4 shown, is a dual-branch network with the same input and output feature map sizes, and its specific structure is as described below:

[0082] The upper half-branch can be a half-wavelet attention block (HWAB), which is mainly used to obtain feature information in the wavelet domain, provide frequency domain analysis of the feature map, and accurately capture the rough contours and fine structures in the image. It is mainly composed of two parts: discrete wavelet transform (DWT) and spatial & channel attention.

[0083] The lower branch can be stacked by a Depthwise Over-parameterized Convolutional Layer (DO-Conv) and a Local Pyramid Attention (LPA), which is mainly used to improve the feature extraction efficiency and guide the network to focus on the target area.

[0084] The training process of the pre-trained single-frame feature extraction network SFExt includes:

[0085] Preprocess the ISAR image dataset S to obtain the preprocessed ISAR images;

[0086] Set a binary classification fully connected layer at the end of the single-frame feature extraction network to output the classification result of whether there is a morphological change in a single image;

[0087] Set the single-frame feature training strategy. After training the single-frame feature extraction network for a preset number of times according to the preprocessed ISAR images, obtain the trained single-frame feature extraction network; during the training process, use the Adam optimizer to update and learn the network parameters of the single-frame feature extraction network, and the learning rate update strategy is StepLR;

[0088] Remove the binary classification fully connected layer in the trained single-frame feature extraction network to obtain the pre-trained single-frame feature extraction network SFExt.

[0089] It can be understood that during training, the ISAR image dataset S can be divided into a training set S T and a test set S V at a preset ratio, use the training set S T to train the single-frame feature extraction network, and use the test set S V to test the single-frame feature extraction network. Preprocessing the ISAR image dataset S can include operations such as scaling, cropping, and normalizing the pictures in the ISAR image dataset S to meet the input requirements of the single-frame feature extraction network.

[0090] Preferably, the preset ratio can be set to 4:1, the preset number of times can be 20, and during training, the batch size can be set to 18.

[0091] A single training process of the single-frame feature extraction network can include:

[0092] The preprocessed ISAR image is subjected to preliminary feature extraction by the downsampling operation Down(·) composed of the first convolutional layer, batch normalization layer BN, rectified linear unit ReLU, max pooling layer MAX pool, and half-wavelet residual feature extraction layer HWRes to obtain the main feature map;

[0093] The main feature map is downsampled using the upper branch and the lower branch in the dual-branch structure to obtain the output feature map;

[0094] The output feature map is unfolded into a one-dimensional feature vector, and the one-dimensional feature vectors are concatenated into temporal features according to the temporal relationship; the temporal features of the current training process are used as the input corresponding to the next training process.

[0095] It can be understood that the half-wavelet residual feature extraction layer HWRes introduced in the pre-trained single-frame feature extraction network SFExt in the embodiments of the present invention utilizes its own advantages in image feature extraction to improve the model's perception ability of high and low frequency features of ISAR images, the model calculation efficiency, and the model's perception ability for target regions.

[0096] The temporal feature recognition network, as Figure 5 shown, may include:

[0097] The first normalization layer, gated linear attention unit GLAU, second normalization layer, and multi-layer perception mechanism MLP connected in sequence; where

[0098] The gated linear attention unit GLAU is a temporal data processing unit that combines the linear attention mechanism and the gating mechanism;

[0099] The multi-layer perception mechanism MLP is a combination of fully connected layers.

[0100] It can be understood that the temporal feature recognition network mainly captures and learns the long and short-term dependencies of temporal features. It can be seen that the overall architecture of the temporal feature recognition network is similar to that of a Transformer, and the input temporal data is processed through three basic units: the layer normalization layer (Layer Normalization, Norm), the gated linear attention unit (Gate Linear Attention Unit, GLAU), and the multi-layer perception mechanism (Multi-Layer Perceptron, MLP), and the changed feature representation is output. The gated linear attention unit GLAU is a temporal data processing unit that combines the linear attention mechanism and the gating mechanism, as Figure 6As shown, while reducing the computational complexity, information from different positions is aggregated to generate a context-aware feature representation. The multi-layer perceptron (MLP) is a combination of a series of fully connected layers, usually including non-linear activation functions. It is used to further transform the features generated by linear attention to capture more complex patterns.

[0101] Specifically, for the input ISAR image temporal features After passing through the residual structure composed of the first layer normalization layer and the gated linear attention unit (GLAU) in series, the temporal features of the first stage of temporal recognition can be obtained The calculation process can be as follows:

[0102] Seq1 = GLAU(Norm(Seq)) + Seq.

[0103] For the one-dimensional feature vector, it is concatenated in temporal order to form the temporal feature Seq = [M o1 ,..., M om , which represents the ISAR image temporal features obtained by a temporal sample passing through the pre-trained single-frame feature extraction network SFExt where m is the length of the temporal sample, and f = H / 32 × W / 32 is the size of the one-dimensional feature vector M o of.

[0104] The temporal features of the first stage of temporal recognition are processed using the residual structure composed of the second layer normalization layer and the multi-layer perceptron (MLP) in series to obtain the temporal features of the second stage of temporal recognition The calculation process can be expressed as follows:

[0105] Seq2 = MLP(Norm(Seq1)) + Seq1.

[0106] Among them, the structure of the gated linear attention unit (GLAU) is as Figure 6 shown. It is a module with the same size of the input and output temporal feature vectors, and is composed of two basic models: linear attention and gated linear unit (GLU). The specific structure is described as follows:

[0107] Based on the Attention-based Transform class models are widely used in the field of time series analysis, but its complexity in time and space is both O(n 2) level, because the representation of each position depends on the dot product similarity calculation with all other positions. Linear Attention is a variant of traditional Attention. It represents self-attention as a linear dot product of kernel feature maps and uses the associative property of matrix multiplication to reduce the complexity from O(n 2 ) to O(n). At the same time, in processing temporal information, the hidden layer state of Linear Attention can be calculated starting from the previous moment. Therefore, the computational complexity of the entire model is linear with respect to the sequence length, and it also has the ability of long-term recursive reasoning.

[0108] It can be understood that the linear attention GLU is essentially an activation function that combines a linear transformation and a gating mechanism to control the information flow. It is mainly composed of two branches: a linear transformation branch and a gating mechanism branch. During the training phase, in the linear transformation branch, the input x passes through a linear layer and outputs a feature vector T of the same size as the input a =W a x + b a . This branch mainly plays the role of transmitting the information flow. In the gating mechanism branch, the input is followed by a non-linear activation function (usually Sigmoid) after the linear layer to generate a gating signal G g =σ(W g x + b g ), where σ represents the passing weight of the information flow. This branch mainly plays the role of judging the importance of the information flow. The final output of GLU is the product of these two parts, as shown below:

[0109]

[0110] The above W a and W g are two different weight matrices respectively, and b a and b g are two bias terms respectively, represents element-wise multiplication.

[0111] The training process of the pre-trained temporal feature recognition network TFRec can include:

[0112] According to the temporal relationship of the temporal samples in the ISAR morphological change image temporal dataset S′, the temporal feature vectors S″ are concatenated to obtain the concatenated temporal image feature vector;

[0113] Set a binary classification fully connected layer at the end of the temporal feature recognition network to output the classification result of whether the temporal sample has morphological changes;

[0114] Set the time series feature training strategy. After training the time series feature recognition network a preset number of times based on the concatenated time series image feature vectors, the trained time series feature recognition network is obtained. During the training process, the Adam optimizer is used to update and learn the network parameters of the time series feature recognition network, and the learning rate update strategy is StepLR.

[0115] Use the trained time series feature recognition network as the pre-trained time series feature recognition network TFRec.

[0116] It can be understood that during training, the ISAR morphological change image time series dataset S′ can be divided into a training set S′ T and a test set S′ V in accordance with a preset ratio. Use the training set S′ T to train the single-frame feature extraction network, and use the test set S′ V to test the time series feature recognition network. Preprocess the ISAR morphological change image time series dataset S′ to obtain feature vectors with a one-dimensional length of 512 for each frame of image, which can include operations such as scaling, cropping, and normalization on the pictures in the ISAR morphological change image time series dataset S′ to meet the input requirements of the time series feature recognition network. According to the time series relationship of the time series samples in the ISAR morphological change image time series dataset S′, when concatenating the time series feature vectors S″ to obtain the concatenated time series image feature vectors, for a time series sample with a length of m, the corresponding feature vector of the time series image is 512×m, and when there are u groups of time series samples, the time series feature vector can be 512×m×u.

[0117] Preferably, the preset ratio can be set to 4:1, the preset number of times can be 20, and during training, the batch processing size (Batch size) can be set to 18.

[0118] A single training process of the time series feature recognition network can include:

[0119] Process the concatenated time series image feature vectors using the residual structure composed of the first-layer normalization layer and the gated linear attention unit GLAU in series to obtain the time series features in the first stage of time series recognition;

[0120] Process the time series features in the first stage of time series recognition using the residual structure composed of the second-layer normalization layer and the multi-layer perception mechanism MLP in series to obtain the time series features in the second stage of time series recognition; Use the time series features in the second stage of time series recognition in the current training process as the input corresponding to the next training process.

[0121] It can be understood that the gated linear attention unit GLAU introduced in the pre-trained temporal feature recognition network TFRec in the embodiments of the present invention combines the inference ability of the linear attention mechanism for long time series and the information forgetting ability of the gated linear unit, improving the model's perceptual recognition ability for the temporal features of ISAR images.

[0122] Based on the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, the network output layer obtains the morphological change result, which may include:

[0123] The temporal sample classification layer in the network output layer determines whether there is a morphological change in the temporal sample according to the output result of the pre-trained temporal feature recognition network TFRec:

[0124] If there is a morphological change, the change frame classification layer of itself is used to determine the frame at which the morphological change starts in the temporal sample with the morphological change according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, and according to the frame at which the morphological change starts, the time when the morphological change occurs and the start of the morphological change are output as the morphological change result;

[0125] If there is no morphological change, "no morphological change" is output as the morphological change result.

[0126] The spatio-temporal joint target morphological change recognition method for sequential ISAR images provided by the embodiments of the present invention adopts a feature extraction + temporal recognition architecture, including a half-wavelet transform residual structure (HWRes) and a gated linear attention unit (GLAU), realizing the feature extraction of ISAR images, and completing the recognition of ISAR temporal image samples by fusing single-frame ISAR image features into multi-frame features. Finally, the number of frames with morphological changes in the temporal samples is recognized by combining single-frame image features and multi-frame image features. It not only alleviates the adverse effects of mutual occlusion between components and electromagnetic scattering anisotropy on the recognition accuracy in single-frame ISAR image morphology recognition, but also, due to the introduction of the time dimension, can associate the states of the front and rear frames of ISAR images, enriching the temporal change features of the target and improving the recognition accuracy. In the embodiments of the present invention, the feature extraction part includes a half-wavelet transform residual structure (HWRes), which can effectively provide time-frequency localization analysis of the feature map. The temporal recognition part in the method of the present invention includes a gated linear attention unit (GLAU), which can capture long-range dependencies of temporal signals and enhance the network's learning of temporal dimension features.

[0127] Next, through simulation experiments, the beneficial effects of the embodiments of the present invention are verified:

[0128] Simulation conditions: The Hubble Space Telescope (HST) is selected as the space target, and its original space target model, the morphological change model with a single damaged sailboard, and the morphological change model with sailboard rotation are used as model inputs. As Figure 7a , Figure 7b and Figure 7d shown, 1440 time-series image results of the three models are obtained using the PFA algorithm. The training set and test set are divided and the temporalization of the dataset is completed according to Steps 3 and 4. At the same time, the Figure 7c time-series images generated by the model shown are used as part of the test set.

[0129] Design the SFExt single-frame recognition experiment. The experimental steps are shown in the training steps of the single-frame feature extraction network. Verify the feature extraction ability of the pre-trained single-frame feature extraction network SFExt for the single-frame dataset of ISAR morphological change images to ensure the effectiveness of the pre-trained single-frame feature extraction network SFExt.

[0130] Design the SFExt+TFRec time-series sample recognition experiment. The experimental steps are shown in the training steps of the pre-trained time-series feature recognition network TFRec. Verify the time-series recognition ability of the SFExt+TFRec network for time-series samples of ISAR morphological change images to ensure the effectiveness of the SFExt+TFRec network for time-series sample recognition.

[0131] Design the morphological change frame recognition experiment to verify the recognition effect of the SFExt+TFRec network on the presence or absence of morphological changes in time-series samples and the recognition effect of morphological change frames generated in time-series samples on the test set.

[0132] Simulation experiment content and result analysis

[0133] Figure 8 is a schematic diagram of the confusion matrix in the simulation of the spatio-temporal joint target morphological change recognition method for sequential ISAR images. Figures 9a - 9c is a schematic diagram of the loss function of the training set and the confusion matrix of the test set in the simulation of a spatio-temporal joint target morphological change recognition method for sequential ISAR images. Among them, Figure 9a is the loss function during the training of SFExt and the confusion matrix on the test set. It can be seen that the training process of SFExt is smooth and approaches convergence, and it can correctly recognize the presence or absence of morphological changes in most images on the test set. Figure 9b is the loss function during the training of SFExt+TFRec and the confusion matrix on the test set. It can be seen that the training process is smooth and the convergence speed is very fast, and it can correctly recognize the presence or absence of morphological changes in most time-series samples on the test set. Figure 9cIt is the confusion matrix for identifying the number of frames with morphological changes in the time-series samples using the trained ISAR morphological change image recognition model on the test set. It can be seen that for most of the time-series samples, the number of frames with morphological changes is correctly recognized, while most of the misrecognized parts are concentrated in the first column of the confusion matrix. That is to say, the network has a tendency to judge the first frame of the time-series sample with morphological changes as a morphological change frame.

[0134] Combined with Figure 8 For Figures 9a - 9c in the confusion matrix in, in practice, more attention is paid to the morphological change category. Therefore, the morphological change category is set as the positive class, and the normal morphological class is set as the negative class. Combined with Figure 8 Then the calculation formulas for the four evaluation indicators are as follows, and the calculation results are shown in Table 1:

[0135] Table 1

[0136]

[0137] The data in Table 1 are obtained through the following formulas:

[0138]

[0139] Among them, TP represents the true positive class, TN represents the true negative class, FP represents the false positive class, and FN represents the false negative class.

[0140] It can be seen from Table 1 that the four evaluation indicators of the test set of the feature extraction network and the time-series recognition network are at a relatively high level. Among them, the time-series recognition network has a significant improvement in precision compared with the feature extraction network. This shows that the method of the present invention for morphological change recognition using the time-series property of ISAR images is better than the morphological change recognition of only single-frame ISAR images, and largely alleviates the misjudgment caused by component occlusion and electromagnetic scattering anisotropy.

[0141] From the above results, it can be seen that the recognition method proposed in the embodiment of the present invention can realize the recognition of whether there are morphological changes in the time-series samples of ISAR images and the recognition of when the morphological changes start in the time-series samples, reduce the influence of component occlusion and anisotropy on the recognition rate of ISAR morphological change images, and thus support obtaining a higher morphological change recognition accuracy, with great innovation and feasibility.

[0142] The embodiments of the present invention utilize the advantage that ISAR images have temporality. By designing a pre-trained single-frame feature extraction network SFExt and a pre-trained temporal feature recognition network TFRec, an architecture combining feature extraction and temporal recognition is obtained, realizing the recognition of whether there are morphological changes in the temporal samples of ISAR images and when the morphological changes of the temporal samples start, reducing the influence of mutual occlusion between components and the anisotropy of electromagnetic scattering on the recognition rate of ISAR morphological change images, so that the classification and recognition results have higher accuracy. Further, the semi-wavelet residual feature extraction layer HWRes introduced in the pre-trained single-frame feature extraction network SFExt utilizes its own advantages in image feature extraction to improve the model's perception ability of high and low frequency features of ISAR images, the model calculation efficiency, and the model's perception ability for the target area; the gated linear attention unit GLAU introduced in the pre-trained temporal feature recognition network TFRec combines the reasoning ability of the linear attention mechanism for long time series and the information forgetting ability of the gated linear unit, improving the model's perception and recognition ability for the temporal features of ISAR images.

[0143] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0144] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A spatio-temporal joint target morphological change recognition method for sequential ISAR images, characterized in that Including: Generating an ISAR image dataset S according to an original spatial target model and its morphological change spatial target model by means of a polar coordinate format algorithm; Processing the ISAR image dataset S by means of a time series method to obtain an ISAR morphological change image time series dataset S'; Processing the ISAR morphological change image time series dataset S' by means of a pre-trained ISAR morphological change image recognition model to obtain a morphological change result; the morphological change result includes whether there is a morphological change and the time when the morphological change starts; wherein, the pre-trained ISAR morphological change image recognition model is trained by using the ISAR image dataset S and the ISAR morphological change image time series dataset S', and the pre-trained ISAR morphological change image recognition model includes: a pre-trained single-frame feature extraction network SFExt, a pre-trained time series feature recognition network TFRec and a network output layer connected in sequence.

2. The spatio-temporal joint target morphological change recognition method for sequence ISAR images according to claim 1, wherein, Processing the ISAR image dataset S by means of a time series method to obtain an ISAR morphological change image time series dataset S', including: Sliding a window with a length of L on the ISAR image dataset S at a speed of 1 frame per time, obtaining a time series sample each time of sliding, and forming the ISAR morphological change image time series dataset S' from all the time series samples.

3. A spatio-temporal joint target shape change recognition method for sequence ISAR images according to claim 1, characterized in that The training process of the pre-trained ISAR morphological change image recognition model includes: Training the single-frame feature extraction network by using the ISAR image dataset S to obtain a pre-trained single-frame feature extraction network SFExt; Inputting the pre-processed ISAR morphological change image time series dataset S' into the pre-trained single-frame feature extraction network SFExt to obtain a time series feature vector S"; training the time series feature recognition network by using the time series feature vector S" to obtain a pre-trained time series feature recognition network TFRec; The network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained time series feature recognition network TFRec.

4. A spatio-temporal joint target morphological change recognition method for sequence ISAR images according to claim 3, characterized in that The single-frame feature extraction network includes: A first convolutional layer, a batch normalization layer BN, a rectified linear unit ReLU, a max pooling layer MAX pool, a semi-wavelet residual feature extraction layer HWRes and a double-branch structure connected in sequence; wherein, the double-branch structure includes 3 double-branch sampling sub-structures connected in sequence, and each double-branch sampling sub-structure includes: a semi-wavelet residual feature extraction layer as the upper branch, a second convolutional layer and a batch normalization layer as the lower branch.

5. A spatio-temporal joint target morphological change recognition method for sequence ISAR images according to claim 4, characterized in that The training process of the pre-trained single-frame feature extraction network SFExt includes: Pre-processing the ISAR image dataset S to obtain a pre-processed ISAR image; Setting a binary classification fully-connected layer at the end of the single-frame feature extraction network to output a classification result of whether there is a morphological change in a single image. Set the single-frame feature training strategy. After training the single-frame feature extraction network a preset number of times based on the preprocessed ISAR image, obtain the trained single-frame feature extraction network. During the training process, use the Adam optimizer to update and learn the network parameters of the single-frame feature extraction network, and the learning rate update strategy is StepLR. Remove the binary classification fully connected layer in the trained single-frame feature extraction network to obtain the pre-trained single-frame feature extraction network SFExt.

6. A spatio-temporal joint target morphological change recognition method for sequential ISAR images according to claim 5, characterized in that A single training process of the single-frame feature extraction network includes: Perform preliminary feature extraction on the preprocessed ISAR image using the downsampling operation Down(·) composed of the first convolutional layer, batch normalization layer BN, rectified linear unit ReLU, max pooling layer MAX pool, and half-wavelet residual feature extraction layer HWRes to obtain the main feature map. Use the upper branch and the lower branch in the double-branch structure to downsample the main feature map to obtain the output feature map. Unfold the output feature map into a one-dimensional feature vector, and splice the one-dimensional feature vector into a temporal feature according to the temporal relationship. Use the temporal feature of the current training process as the input corresponding to the next training process.

7. A spatio-temporal joint target shape change recognition method for sequential ISAR images according to claim 3, characterized in that The temporal feature recognition network includes: The first normalization layer, gated linear attention unit GLAU, second normalization layer, and multi-layer perception mechanism MLP connected in sequence. Among them, The gated linear attention unit GLAU is a temporal data processing unit that combines the linear attention mechanism and the gated mechanism. The multi-layer perception mechanism MLP is a combination of fully connected layers.

8. A spatio-temporal joint target morphological change recognition method for sequence ISAR images according to claim 7, characterized in that, The training process of the pre-trained temporal feature recognition network TFRec includes: According to the temporal relationship of the temporal samples in the ISAR morphological change image temporal dataset S′, splice the temporal feature vectors S″ to obtain the spliced temporal image feature vector. Set a binary classification fully connected layer at the end of the temporal feature recognition network to output the classification result of whether the temporal sample has a morphological change. Set the temporal feature training strategy. After training the temporal feature recognition network a preset number of times based on the spliced temporal image feature vector, obtain the trained temporal feature recognition network. During the training process, use the Adam optimizer to update and learn the network parameters of the temporal feature recognition network, and the learning rate update strategy is StepLR. Use the trained temporal feature recognition network as the pre-trained temporal feature recognition network TFRec.

9. A spatio-temporal joint target morphological change recognition method for sequence ISAR images according to claim 8, characterized in that A single training process of the temporal feature recognition network includes: Process the spliced temporal image feature vector using the residual structure composed of the first normalization layer and the gated linear attention unit GLAU in series to obtain the temporal feature in the first stage of temporal recognition. Process the temporal feature in the first stage of temporal recognition using the residual structure composed of the second normalization layer and the multi-layer perception mechanism MLP in series to obtain the temporal feature in the second stage of temporal recognition. Use the temporal feature in the second stage of temporal recognition of the current training process as the input corresponding to the next training process.

10. A spatio-temporal joint target shape change recognition method for sequence ISAR images according to claim 8, characterized in that, The network output layer obtains the morphological change result according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, including: The temporal sample classification layer in the network output layer determines whether a morphological change occurs in the temporal sample according to the output result of the pre-trained temporal feature recognition network TFRec: If a morphological change occurs, the change frame classification layer of itself is used to determine the frame at which the morphological change begins in the temporal sample with the morphological change according to the output results of the pre-trained single-frame feature extraction network SFExt and the pre-trained temporal feature recognition network TFRec, and according to the frame at which the morphological change begins, the time when the morphological change occurs and the start of the morphological change are output as the morphological change result; If no morphological change occurs, "no morphological change" is output as the morphological change result.

Images