A target detection method based on graph reconstruction network

Abstract

The application belongs to the field of target detection, and discloses a target detection method based on a graph reconstruction network. The method first collects multispectral images for a period of time, performs dimension reduction and feature extraction on the multispectral images, and extracts physical feature maps, spatial feature maps and spectral feature maps of the multispectral images through graph embedding. The three feature maps of different node types obtained are connected through link edges and nodes, a graph pooling method based on self-attention is adopted, and a heterogeneous graph fusing multi-source features is obtained. The fused graph data is input in the time dimension, the time and space features of the data are obtained, multi-layer space-time graph convolution is used to extract information of different space-time scales, the convolution result of the space-time convolution is made consistent with the predicted target dimension through the convolution operation and the full connection operation of the CNN, and the full connection layer with weight sharing is used to classify and locate the target, so that the target detection task is completed. The space-time features obtained by the method make the detection more accurate.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more specifically to a target detection method based on graph reconstruction networks. Background Technology

[0002] Object detection is a crucial task in computer vision, aiming to accurately detect objects of interest in images or videos and label their locations. Multispectral images contain information across multiple spectral bands. They encompass not only spatial but also spectral information, overcoming the limitations of single-modal images. For dynamic and small targets, utilizing the multimodal information of multispectral images for target recognition allows for more accurate acquisition of target region and location information. Therefore, applying the characteristics of multispectral images to object detection tasks can improve accuracy and reliability. By leveraging the multiple spectral bands in multispectral images, algorithms can better distinguish targets from the background and extract richer feature information, thus achieving more accurate object detection results.

[0003] Early multispectral target recognition primarily relied on manual band selection. For example, specific feature bands were used to separate targets from complex field backgrounds, and polarization-multispectral image fusion was used to detect camouflaged targets based on their spectral characteristics. In recent years, traditional manual feature selection and fusion methods have been gradually replaced by convolutional neural networks (CNNs). Shaoting Zhang of the University of North Carolina verified the impact of feature fusion at different stages of a CNN on the performance of multispectral image target detection. Meanwhile, Hangil used CNNs and support vector regression to perform joint feature extraction from visible and far-infrared spectral images. Mingyi He et al. of Northwestern Polytechnical University used deep residual networks to extract features at different levels from multispectral remote sensing images, achieving end-to-end detection of salient targets.

[0004] However, CNNs, as a foundational network model, excel at processing spatial network data and establishing spatial local neighborhood relationships between pixels, easily neglecting the visual information of the data itself and the implicit relationships between irregular representations. The downsampling process in CNNs reduces the spatial resolution of feature images, inevitably leading to the loss of information about small objects. This makes it difficult for detection networks to learn representations from limited and distorted structural information. Furthermore, CNNs cannot extract temporal feature information between frames. Therefore, it is essential to propose a solution that addresses these problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a target detection method based on graph reconstruction networks. This method performs dimensionality reduction and feature extraction on multispectral images, performs graph embedding to obtain graph features, and utilizes spatiotemporal graph neural networks to fully obtain features in the time and spatial dimensions for target detection, thereby obtaining more accurate location and recognition accuracy.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is: a target detection method based on graph reconstruction networks, characterized by comprising the following steps:

[0007] (S1) Acquire multispectral images over a period of time, and perform dimensionality reduction and feature extraction on them;

[0008] (S2) Then, the physical feature map, spatial feature map and spectral feature map of the multispectral image are extracted by graph embedding.

[0009] (S3) Connect the feature maps of the three different node types obtained by linking edges and nodes, and use the graph pooling method based on self-attention to obtain a heterogeneous graph that fuses multi-source features.

[0010] (S4) Sort the fused graph data by time dimension and input it. Use multi-layer spatiotemporal graph convolution to obtain the temporal and spatial features of the data as well as spatiotemporal correlation.

[0011] (S5) The spatiotemporal convolution result is made consistent with the dimension of the predicted target through the convolution and fully connected operations of CNN. The target is classified and located through the weight-sharing fully connected layer.

[0012] Preferably, in step (S1), the multispectral image is captured by a multispectral camera capable of simultaneously acquiring three or more spectral bands.

[0013] Preferably, in step (S1), the feature dimensionality reduction and feature extraction are performed by using spatial spectral embedding to complete the weight allocation of the similarity of spectral features of different pixels, and by using manifold learning to perform similarity classification and feature dimensionality reduction on local neighborhood space and spectral information.

[0014] Preferably, in step (S2), the acquisition of the three feature maps is as follows: using the dimensionality-reduced spectral data, the physical feature map of the spectral data is extracted by combining infrared spectral features; using linear iterative clustering to determine the superpixel neighbor node information, constructing the edge connection relationship between nodes based on the spatial connectivity relationship of the superpixels, and extracting the spatial feature map; combining the spectral feature similarity of the target, sampling and recombining from different spectral band dimensions to obtain the target spectral feature distribution, and using a graph neural network to effectively represent the spectral data residing on the smooth manifold.

[0015] Preferably, in step (S3), the linked network model is a graph autoencoder, which includes, but is not limited to, graph convolutional autoencoders, variational graph convolutional autoencoders, and adversarial regularized graph autoencoders.

[0016] Preferably, in step (S3), the graph pooling method includes, but is not limited to, DiffPool, SAGPool, and ASAP.

[0017] Preferably, in step (S4), the spatiotemporal graph convolution uses different methods to extract features in the time dimension and the spatial dimension respectively. The networks used to extract features in the time dimension include, but are not limited to, RNN, GRU, LSTM, TCmodule, and Transformer. The networks used to extract features in the spatial dimension include, but are not limited to, GCN, GAT, and GCN combined with GAT.

[0018] Preferably, in step (S5), the detection method involves passing the extracted features through a convolution module after four levels of spatiotemporal graph convolution, and finally feeding them into the target detection module to classify and locate the target, thereby completing the target recognition task.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] By performing dimensionality reduction and feature extraction on multispectral images and then embedding them into graphs, multidimensional feature information including spatial, physical, and spectral features was obtained. The resulting graph structures were then merged to produce a heterogeneous graph containing multi-source information, which can better utilize this heterogeneous information to supplement the semantic associations and relative positional information of the target.

[0021] Meanwhile, the node and edge features in graph data represent the potential relationships between data. Its nodes are unordered and variable in size, and can update the state of a node based on neighboring nodes at any depth, thereby representing its attribute feature relationships. This feature makes it suitable for characterizing long-range spatial and spectral relationships in multispectral images.

[0022] Furthermore, to address the issue that CNNs cannot extract temporal features between frames, we utilize spatiotemporal graph convolution to extract temporal features. This allows us to more accurately determine the associated features of moving targets and global background information that change between frames. Attached Figure Description

[0023] Figure 1 This is a flowchart of a target detection method based on graph reconstruction networks according to the present invention.

[0024] Figure 2 This is a framework diagram of a target detection method based on graph reconstruction networks according to the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0026] See Figure 1 The present invention provides a target detection method based on graph reconstruction networks, comprising the following steps:

[0027] (S1) Acquire multispectral images over a period of time, and perform dimensionality reduction and feature extraction on them;

[0028] (S2) Then, the physical feature map, spatial feature map and spectral feature map of the multispectral image are extracted by graph embedding.

[0029] (S3) Connect the feature maps of the three different node types obtained by linking edges and nodes, and use the graph pooling method based on self-attention to obtain a heterogeneous graph that fuses multi-source features.

[0030] (S4) Sort the fused graph data by time dimension and input it. Use multi-layer spatiotemporal graph convolution to obtain the temporal and spatial features of the data as well as spatiotemporal correlation.

[0031] (S5) The spatiotemporal convolution result is made consistent with the dimension of the predicted target through the convolution and fully connected operations of CNN. The target is classified and located through the weight-sharing fully connected layer.

[0032] See Figure 1 In step (S1), the acquired multispectral data consists of multispectral images of four bands, and 1,000 multispectral images at different time periods were taken.

[0033] See Figure 1 In step (S1), the dimensionality reduction and feature extraction method first utilizes augmented vectors to fuse spectral and spatial information.

[0034] (1)

[0035] Where h(u,v) is a pixel on the image, (b1,b2,b3,b...) B () is a band array.

[0036] In this embodiment, images of 4 bands were acquired, so B=4.

[0037] The augmented vector [x1, x2, …, x] L ]∈ (B+2)L As training data, after normalization, for any x iIn supervised mode, the same category is classified, and the local neighborhood of the pixel is constructed by the k nearest neighbor algorithm. The similarity classification and feature dimensionality reduction of the local neighborhood spatial and spectral information are performed by manifold learning. Combined with spatial spectral polynomial local or neighborhood embedding, the weight allocation of the similarity of spectral features of different pixels in the local neighborhood is completed. Finally, the low-dimensional nonlinear explicit mapping between multispectral data is established by combining quantile matrix multiplication.

[0038] In a specific embodiment, the augmented vector contains 6 labeled elements.

[0039] See Figure 1 In step (S2), the spatial feature map is first segmented into superpixels by the SLIC algorithm. By calculating the spatial distance and spectral distance between pixels and balancing the weights, the superpixel cluster centers and range boundaries are iteratively updated to obtain a multispectral image data composed of superpixels. The edge connection relationship between nodes is constructed based on the spatial connectivity relationship of the superpixels.

[0040] See Figure 1 The spectral feature map described in step (S2) is constructed using a semi-supervised adjacency matrix method. Specifically, this method is constructed based on information provided by a finite amount of labeled data and a large amount of unlabeled data. Pseudo-labels are constructed using a hybrid model based on variational push Dirichlet processes, and the spatial spectral adjacency matrix is ​​constructed based on the intrinsic clustering algorithm in the data samples.

[0041] See Figure 1 In step (S3), the nodes and edges of the three obtained feature maps are analyzed, and a network structure based on graph autoencoder is used to connect the feature maps of three different types of nodes and edges.

[0042] Specifically, for each given graph, cosine similarity analysis is used to identify the node feature vectors between different graphs, retaining nodes with high similarity among the three graphs. For the three processed graphs, a graph convolutional network is used to compute the node representation z for each node. Then, the following formula is applied:

[0043] (2)

[0044] in It refers to the predicted probability between linked nodes (i, j), where The activation function is Sigmoid. Here, a probability greater than 0.8 is set for linking, and a probability less than 0.2 is set for not linking, resulting in a new graph after linking the three graphs.

[0045] See Figure 1In step (S3), the SAGpool method is used to extract and aggregate information from the nodes of the new graph.

[0046] Specifically, the extracted new graph is first processed through a graph neural network (GCN) convolution operation. The GCN learns feature representations for each node v∈V, aggregating the features of each node's neighboring nodes to obtain the feature representation of node v. For each node v, a self-attention mechanism is used to calculate an attention score z. Then, the top-k nodes are selected, and the number of nodes retained is determined by the pooling ratio k (here, we set k to 0.5). This yields an attention-based mask image. This mask image is then multiplied at corresponding nodes with the graph structure of the original input, which incorporates heterogeneous information, to obtain the final output graph, i.e., a heterogeneous graph fusing multi-source features.

[0047] See Figure 1 In step (S4), the network for extracting the time dimension is a TCmodule. This time module consists of two extended initiation layers.

[0048] Specifically, the output source of the entire temporal convolutional module is divided into two parts. The module's input is filtered through a dilated initiation layer consisting of a set of one-dimensional convolutional filters. The difference lies in the activation function following the dilated initiation layer. One part of the output source is processed by the tanh activation function after passing through the dilated initiation layer, which acts as a filter for the input. The other branch of the input is processed by the sigmoid activation function, which controls the amount of information that can pass to the next module. The dilated initiation layer uses a structure composed of filter sizes of 1×2, 1×3, 1×6, and 1×7, so that the aforementioned time period can be covered by combinations of these filter sizes.

[0049] In this example, we input 10 images with fused heterogeneous features at a time, that is, extract the temporal feature relationship from the original 10 frames of multispectral images.

[0050] See Figure 1 In step (S4), the feature network for extracting spatial dimensions is GCN plus GAT.

[0051] Specifically, after passing through the temporal module network, spatial features are extracted using the GCN layer. Then, the GAT graph attention layer is used to pass information between nodes and capture the dependencies between them. The features after passing through TCmodule, GCN, and GAT are processed again in the same way, and the features generated after each processing are used for further feature extraction.

[0052] In this example, we extract features at four levels and use the concat function to concatenate the obtained features to obtain multi-scale spatiotemporal features.

[0053] See Figure 1 In step (S5), after extracting the spatiotemporal features, the spatiotemporal convolution result is made consistent with the predicted target dimension through five layers of CNN convolution blocks. Finally, MLP is used to classify the acquired features to obtain the location information in the image and the category information, thus completing the target detection task.

[0054] In this example, training is performed on an Ubuntu 18.04.3 Linux operating system, a PyCharm compilation environment, a Python 3.9 programming language, a pytorch-cuda 11.7 deep learning application library, and a GeForce 3090 graphics card.

[0055] In this example, the IOU threshold is set to 0.5. A spatiotemporal graph convolutional network is built using PyTorch. Here, we set the loss function to the binary cross-entropy loss function:

[0056] (3)

[0057] Where y is the ground truth, which is 1 for positive examples and 0 for negative examples. This is the probability of the model's prediction accuracy. The learning rate is set to 0.01, the training epoch is set to 300, the batch size is 8, and the input consists of 1000 images.

[0058] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the limitations of the above embodiments. All technical solutions falling within the spirit and principle of the present invention are within the scope of protection of the present invention. It should be noted that any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A target detection method based on graph reconstruction networks, characterized in that, Includes the following steps: S1. Collect multispectral images over a period of time, process the image data using the spatial spectral embedding method to achieve weight allocation of spectral feature similarity between different pixels, and perform similarity classification, feature dimensionality reduction and extraction on the local neighborhood space and spectral information of the image through manifold learning. S2. Then, physical feature maps, spatial feature maps, and spectral feature maps of the multispectral image are extracted using graph embedding: First, based on the dimensionality-reduced spectral data and combined with infrared spectral information, feature maps reflecting the physical properties of equivalent temperature and equivalent area are extracted, and their graph structure is represented by a random walk graph embedding method; Second, the linear iterative clustering algorithm SLIC is used to perform superpixel segmentation on the multispectral image. By calculating the spatial and spectral distances between pixels and performing weight balancing, the superpixel cluster centers and their boundaries are iteratively updated, ultimately obtaining a multispectral image data composed of superpixels. Based on the spatial and spectral distances of the superpixels... Connectivity relationships are used to construct edge connections between nodes to extract spatial feature maps. Finally, based on the similarity of the target's spectral features, spectral information is sampled and recombined from different spectral band dimensions to form the target spectral feature distribution, which is then modeled using a semi-supervised adjacency matrix method. This semi-supervised adjacency matrix method is based on a limited number of labeled data and a large amount of unlabeled data. It uses a variational inference Dirichlet process hybrid model to generate pseudo-labels, combines the inherent clustering structure of the data to construct a spatial spectral adjacency matrix, and further utilizes GNN to model the spectral data residing on a smooth manifold, ultimately extracting the spectral feature map. S3. A graph autoencoder-based network structure is adopted, connecting the obtained feature maps of three dimensions and different node types through link edges and nodes. For each given feature map, cosine similarity is used to analyze the node feature vectors between different feature maps, and nodes with high similarity in the three maps are retained to achieve preliminary fusion of multi-source feature nodes. For the processed feature map, a graph convolutional network is used to calculate the node representation z corresponding to node i and node j. i and z j Then, the predicted probability of the link between node i and node j is calculated using the following formula: in Predict the probability of the link between node i and node j, where The Sigmoid activation function is used. Based on the link prediction probability, a node connection is established when the link prediction probability is greater than 0.8, and no connection is established when the link prediction probability is less than 0.2, thereby obtaining a new graph structure that fuses the three feature maps. Finally, the graph pooling method is used to extract and aggregate information from the nodes in the fused graph to achieve effective fusion of multi-source features. S4. Sort the fused graph data by time dimension and input it. Use multi-layer spatiotemporal graph convolution to obtain the temporal and spatial features of the data as well as spatiotemporal correlation. S5. After feature extraction through four layers of spatiotemporal graph convolution, the feature is input into the convolution module for further processing and finally sent to the target detection module to achieve target classification and localization, thereby completing the target recognition task. Through the convolution and fully connected operations of the convolutional neural network, the dimension of the spatiotemporal convolution output is made consistent with the dimension of the predicted target. With the help of the fully connected layer with weight sharing characteristics, the target is efficiently classified and located.

2. The target detection method based on graph reconstruction networks according to claim 1, characterized in that, In step S1, the multispectral image is captured by a multispectral camera capable of simultaneously acquiring three or more spectral bands.

3. The target detection method based on graph reconstruction networks according to claim 1, characterized in that, In step S3, the network structure based on graph autoencoders includes graph convolutional autoencoders, variational graph convolutional autoencoders, and adversarial regularized graph autoencoders.

4. The target detection method based on graph reconstruction network according to claim 1, characterized in that, In step S3, the graph pooling methods include DiffPool, SAGPool, and ASAP.

5. The target detection method based on graph reconstruction networks according to claim 1, characterized in that, In step S4, the spatiotemporal graph convolution uses different methods to extract features in the time and spatial dimensions. The networks used to extract features in the time dimension include RNN, GRU, LSTM, TCN, and Transformer, while the networks used to extract features in the spatial dimension include GCN, GAT, and GCN combined with GAT.

Images