Small target detection and tracking method in infrared images based on lightweight hypergraph network

Abstract

This application discloses a method for detecting and tracking weak targets in infrared images based on a lightweight hypergraph network, comprising: S1, collecting infrared images and constructing a complete infrared target database through a variational autoencoder of the semantic features of the infrared images; S2, constructing and training a hypergraph network target detection and recognition model using the complete infrared target database; and S3, using the hypergraph network target detection and recognition model to detect and track weak targets in the infrared images. This application achieves robust, robust, and real-time infrared weak target detection and tracking, solving the deployment problem of deep learning network models under the conditions of limited on-board / airborne computing power and storage resources.

Description

Technical Field

[0001] The present application relates to the field of small target detection, and in particular to a method for detecting and tracking small targets in infrared images based on a lightweight hypergraph network. Background Art

[0002] The infrared detection system uses the target's thermal radiation to achieve long-distance target detection and tracking. It is a passive detection system that uses infrared detectors to map the thermal radiation of objects in the scene into grayscale values ​​to form an infrared image. It has strong penetration, so it is less affected by weather and has strong environmental adaptability. Compared with visible light, it can better identify objects that are difficult to detect in complex and hidden environments. Therefore, it is widely used in military, medical, remote sensing, communications, climatology and other fields.

[0003] Since the infrared detection system is mainly used to detect long-distance targets, the target imaging area is small. The target often only occupies a few to dozens of pixels in the infrared imaging, and its geometric and texture features cannot be reflected. In addition, the imaging process may be disturbed by the atmosphere, causing the target radiation intensity to weaken and approach the background high-frequency information intensity, causing the target to be submerged in the background. In addition, the background of infrared detection is mostly clouds, sea surface or buildings, and the grayscale distribution of the background often has the characteristics of violent fluctuations and unstable spatial distribution. The high-frequency information intensity is close to the target, and the target usually has a very low signal-to-noise ratio. In summary, how to use infrared detection technology to achieve detection and tracking of infrared weak targets under limited samples / complex scenes is a research direction with great practical significance, and it is also an important technology that urgently needs to be solved.

[0004] The performance of existing infrared image weak target detection and tracking algorithms is often limited to situations such as sample imbalance and scene changeability. In recent years, infrared images have the characteristics of many scenes, large data volume but small target ratio, which makes it difficult for deep learning-based infrared image weak target detection and tracking technology to match the limited computing resources on board. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the infrared image dim target detection and tracking method based on a lightweight hypergraph network provided by the present invention solves the problems of noise interference, multiple false alarms and difficult deployment of current infrared image target detection and recognition methods under the conditions of limited on-board / airborne computing power and storage resources.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for detecting and tracking small targets in infrared images based on a lightweight hypergraph network, comprising:

[0007] S1, collect infrared images, and build a complete infrared target database through the semantic feature variational autoencoder of infrared images;

[0008] S2. Use the complete infrared target database to build and train the hypergraph network target detection and recognition model;

[0009] S3. Use the hypergraph network target detection and recognition model to detect and track small targets in infrared images.

[0010] Further: S1 includes:

[0011] S11, collect infrared images, and use Riemannian manifold learning to extract semantic features containing geometric attribute information in the infrared images;

[0012] S12. According to the semantic features of infrared images, a semantic feature variational autoencoder is constructed through variational inference theory;

[0013] S13. Train the semantic feature variational autoencoder, use the trained semantic feature variational autoencoder to expand the infrared image target information image, and build a complete infrared target database.

[0014] Further: the S11 includes:

[0015] S111, extracting manifold features of the infrared image on the symmetric positive definite matrix manifold as a covariance matrix;

[0016] S112, mapping the covariance matrix to the tangent space, and finally forming a semantic feature vector in the Euclidean space as a semantic feature containing geometric attribute information in the infrared image.

[0017] Further: in S12, the semantic feature variational autoencoder includes a generator, a decoder and a discriminator;

[0018] The generator is used to extract semantic features of infrared images;

[0019] The decoder is used to generate a reconstructed infrared image according to the semantic features of the real infrared image;

[0020] The discriminator is used to distinguish between real infrared images and reconstructed infrared images.

[0021] Further: In said S13, when training the semantic feature variational autoencoder, the network parameters in the generator, decoder and discriminator are optimized respectively by the back propagation algorithm, and the loss function L C for:

[0022]

[0023] in, KL (.) To find the relative quotient operation, and is the trainable model parameter of the semantic feature variational autoencoder, z is a hidden variable that follows a priori distribution , is the conditional probability of the generator, is used to balance the reconstruction loss and KL The weight hyperparameter between the divergences, is the approximate posterior of the decoder distribution, G (.) indicates a generator, D ( I )and D ( I fc ) represent the discriminator's response to the real image I and the resulting reconstructed image I fc The probability of true or false judgment, E I is the expected operation for judging the authenticity of the real image. is the expected operation for judging the authenticity of the generated reconstructed image. is the expected operation of semantic conditional loss, is the expectation operation of the reconstructed image loss.

[0024] Further: In S2, the step of detecting and identifying small targets in infrared images by the hypergraph network target detection and recognition model includes:

[0025] S21. Construct edge graph W E and the vertex graph W V , and through the edge graph W E and the vertex graph W V Extract low-level structural information of hypergraphs;

[0026] S22, extracting high-order structural information of the hypergraph from the hypergraph through vertex-edge convolution;

[0027] S23, the edge map W E and the vertex graph W V Aggregate edge-level and vertex-level structural information to obtain an enhanced edge graph and enhanced vertex graph ;

[0028] S24, the edge map and enhanced vertex graph Fusion, get a low-order hypergraph , the edge graph and enhanced vertex graph Aggregate the edge vertex structure information in to obtain a high-order hypergraph ;

[0029] S25. Through high-order hypergraph Extract the full local semantic features of small targets in infrared images through low-order hypergraphs Extract weak local background related information;

[0030] S26, combining the low-order structural information of the hypergraph with the high-order structural information of the hypergraph to enhance the orientation and size information of the weak target in the infrared image;

[0031] S27, taking the full local semantic features of the weak and small targets in the infrared image, the weak local background related information, and the orientation and size information of the weak and small targets in the infrared image as the detection and recognition results.

[0032] Further: In S23, the enhanced edge map and enhanced vertex graph The expression is:

[0033]

[0034] in, represents the number of sequence numbers of hyperedges, Indicates the number of times information is aggregated. n Indicates the total number of current infrared images. H j express I j The hypergraph adjacency matrix of Conv_e and Conv_v Represent the vertex convolution layer and edge convolution operation respectively, Indicated in The total weight of the hyperedges of the initial infrared image under the aggregation, Indicated in The total vertex weight of the initial infrared image under sub-aggregation, Indicated in Next aggregation j The total weight of the hyperedges of infrared images, Indicated in Next aggregation j Infrared image corresponding to the hyperedge eq and ep The weight between Indicated in Next aggregation j Infrared image corresponding to the hyperedge ek and eq The weight between Indicated in Next aggregation jThe total vertex weight of infrared images, Indicated in Next aggregation j The corresponding vertices of the infrared image vq and vp The weight between Indicated in Next aggregation j The corresponding vertices of the infrared image vk and vq The weight between .

[0035] Further: In S24, the low-order hypergraph and high-order hypergraphs The expression is:

[0036]

[0037] in, Fuse (.) indicates the structure information aggregation operation, Indicates the number of times information is aggregated. Indicated in t Second aggregation j A high-order hypergraph of an infrared image, Indicated in t Second aggregation j Infrared image hyperedge eq and ep , and the vertex vq and vp The high-order hypergraph under the structural information of Indicated in t 2-1st aggregation j Infrared image at the vertex vp The following corresponds to different hyperedges ek The edge hypergraph formed by Indicated in t 2-1st aggregation j Infrared image corresponding to the hyperedge ep The following correspond to different vertices vk The vertex hypergraph formed.

[0038] The beneficial effects of the present invention are:

[0039] 1. The data augmentation model of semantic feature variational autoencoder based on semantic conditions uses Riemannian manifold learning to extract semantic features containing geometric attribute information, which is used as a priori constraint to suppress background noise while improving the diversity and quality of generated samples, and establish a complete database of high-quality infrared targets with significant targets and diverse backgrounds;

[0040] 2. The infrared image weak target detection and recognition model based on hypergraph neural network utilizes the topological features of the hypergraph, mines nonlinear high-order correlations, suppresses background noise and eliminates pseudo targets, reduces the false alarm rate of the model, and realizes accurate and effective detection and tracking of weak targets in infrared videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the small target detection and tracking method in infrared images based on lightweight hypergraph network. DETAILED DESCRIPTION

[0042] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0043] Embodiment 1:

[0044] like Figure 1 As shown, in an embodiment, a method for detecting and tracking small targets in infrared images based on a lightweight hypergraph network is provided, comprising:

[0045] S1, collect infrared images, and build a complete infrared target database through the semantic feature variational autoencoder of infrared images;

[0046] S2. Use the complete infrared target database to build and train the hypergraph network target detection and recognition model;

[0047] S3. Use the hypergraph network target detection and recognition model to detect and track small targets in infrared images.

[0048] The S1 includes:

[0049] S11, collect infrared images, and use Riemannian manifold learning to extract semantic features containing geometric attribute information in the infrared images;

[0050] S12. According to the semantic features of infrared images, a semantic feature variational autoencoder is constructed through variational inference theory;

[0051] S13. Train the semantic feature variational autoencoder, use the trained semantic feature variational autoencoder to expand the infrared image target information image, and build a complete infrared target database.

[0052] The S11 includes:

[0053] S111, extracting manifold features of the infrared image on the symmetric positive definite matrix manifold as a covariance matrix;

[0054] S112, mapping the covariance matrix to the tangent space, and finally forming a semantic feature vector in the Euclidean space as a semantic feature containing geometric attribute information in the infrared image.

[0055] In S12, the semantic feature variational autoencoder includes a generator, a decoder and a discriminator;

[0056] The generator is used to extract semantic features of infrared images;

[0057] The decoder is used to generate a reconstructed infrared image according to the semantic features of the real infrared image;

[0058] The discriminator is used to distinguish between real infrared images and reconstructed infrared images.

[0059] In S2, the steps of detecting and identifying small targets in infrared images by the hypergraph network target detection and recognition model include:

[0060] S21. Construct edge graph W E and the vertex graph W V , and through the edge graph W E and the vertex graph W V Extract low-level structural information of hypergraphs;

[0061] S22, extracting high-order structural information of the hypergraph from the hypergraph through vertex-edge convolution;

[0062] S23, the edge map W E and the vertex graph W V Aggregate edge-level and vertex-level structural information to obtain an enhanced edge graph and enhanced vertex graph ;

[0063] S24, the edge map and enhanced vertex graph Fusion, get a low-order hypergraph , the edge graph and enhanced vertex graph Aggregate the edge vertex structure information in to obtain a high-order hypergraph ;

[0064] S25. Through high-order hypergraph Extract the full local semantic features of small targets in infrared images through low-order hypergraphs Extract weak local background related information;

[0065] S26, combining the low-order structural information of the hypergraph with the high-order structural information of the hypergraph to enhance the orientation and size information of the weak target in the infrared image;

[0066] S27, taking the full local semantic features of the weak and small targets in the infrared image, the weak local background related information, and the orientation and size information of the weak and small targets in the infrared image as the detection and recognition results.

[0067] Embodiment 2:

[0068] This embodiment is a supplement to Embodiment 1 and is a specific implementation method for supplementing the construction of a complete infrared target database.

[0069] Since most methods have problems such as lack of training samples, insufficient high-quality samples, and serious sample unevenness when realizing the recognition, detection, and tracking of small targets in infrared images, this embodiment first designs an infrared small target image generation model based on semantic conditional adversarial semantic feature variational autoencoder from the perspective of sample category imbalance and small samples, which includes the following steps:

[0070] S111, extracting the manifold features of the infrared image on the symmetric positive definite (SPD) matrix manifold as the covariance matrix;

[0071] The covariance matrix can more effectively describe the non-Euclidean geometric features of weak targets and complex backgrounds in infrared images;

[0072] S112, mapping the covariance matrix to the tangent space, and finally forming a semantic feature vector in the Euclidean space as a semantic feature containing geometric attribute information in the infrared image;

[0073] Among them, the pixels in the infrared image area d The dimensional feature map is represented as:

[0074]

[0075] in, f Indicates the mapping rule, is the infrared image at coordinates ( x , y ), For each pixel in x The first derivative in the direction, For each pixel in y The first derivative in the direction, For each pixel in x The second derivative in the direction, For each pixel, y The second derivative in the direction, is the local entropy, which is used to describe the intensity of grayscale changes in the local area. The characteristics of the image are filtered by Gabor filter groups of different scales and directions, which are used to characterize the frequency and direction characteristics of the target and are also robust to illumination and deformation.

[0076] By combining the underlying features of the target with mid-level semantic descriptions such as local entropy and Gabor features, the spatial information expression capability of the covariance descriptor is enriched. It can be calculated by the following formula:

[0077]

[0078]

[0079] in, i Representing different scales, n Represents the total number of pixels in the area. For i Scale k Pixels d dimensional feature map, is the intermediate parameter.

[0080] In manifold space ( ) and the geodesic distance in the tangent space ( ), the exponential operator defined on the differentiable manifold and logarithm operators , you can switch the manifold space and its x The tangent space at the position of , thus finally obtaining the semantic feature representation of the infrared image containing geometric attribute information is: .

[0081] After extending the model to the manifold space, the nonlinear correlation of small targets in the manifold space is explored through the probability density distribution of small targets in the low-dimensional space, so as to better distinguish the characteristics of infrared small targets and background features in complex scenes, thereby obtaining the nonlinear semantic feature representation of small targets in the manifold space of infrared images.

[0082] The extracted semantic features As a condition, a weak target data expansion model of infrared images based on semantic conditional semantic feature variational autoencoder is constructed with the help of variational inference theory. It constructs latent variables about target features by learning observed data, then samples from the probability density function representing the distribution of latent variables, and then reconstructs the real observed data through the decoder.

[0083] In order to obtain better decoder parameters, this embodiment converts the original image I , the reconstructed image generatedI fc Its semantic conditions The network parameters of the encoder, decoder and discriminator are optimized through the back propagation algorithm. The loss function of the model is L C for:

[0084]

[0085] in, KL (.) is a relative quotient operation used to measure the distance between two probability distributions. This term aims to make the posterior distribution close to the prior distribution. and is the trainable model parameter of the semantic feature variational autoencoder, z is a hidden variable that follows a priori distribution , is the conditional probability of the encoder, is the weight hyperparameter used to balance the reconstruction loss and KL divergence, is the approximate posterior of the decoder distribution, G (.) represents the generator network, D ( I )and D ( I fc ) represent the discriminator network for the real image I and the resulting reconstructed image I fc The probability of true or false judgment, E I is the expected operation for judging the authenticity of the real image. is the expected operation for judging the authenticity of the generated reconstructed image. is the expected operation of the semantic condition loss, which is used to ensure that the generated samples match the given semantic condition information. The expected operation of reconstructing the image loss is used to reduce the difference between the generated data and the original data.

[0086] During the training process, the discriminator tries to accurately distinguish between real infrared images and reconstructed infrared images, while the decoder tries to deceive the discriminator by reconstructing infrared images as realistically as possible with the help of semantic feature priors. Infrared images can be regarded as a superposition of background, noise and weak targets. Through the conditional constraints of semantic features, the autoencoder not only considers the global pixel-level features of most background representations, but also takes into account the instance-level features of weak target representations. While suppressing background noise, it alleviates the problem of missing attributes such as shape and texture of weak targets in generated samples, and improves the diversity and quality of generated samples. It is possible to extract weak target information from images with a high proportion of background and construct a universal data generation model, thereby establishing a complete database of high-quality infrared targets with prominent targets and diverse backgrounds.

[0087] Embodiment 3:

[0088] This embodiment is a supplement to Embodiment 1 and is a specific implementation method for constructing and training a hypergraph network target detection and recognition model using a complete infrared target database.

[0089] In order to solve the problem of multiple false alarms in long-term tracking caused by the lack of attributes such as shape and texture of small targets, this embodiment considers the local and global aspects of the data; multi-order model processing is performed on the local part of a single-frame infrared image to improve and enrich the feature information of small targets to achieve joint feature expression, and highlight the difference between small targets and the background, thereby effectively improving the accuracy of small target detection and tracking.

[0090] Since infrared images often include a large amount of background information, when using traditional graph neural networks to establish global correlations, noise will be introduced, and the intuitive pairwise connections between nodes (i.e., first-order relationships) are not sufficient to model high-order data in the real world, limiting their application in processing complex data in multiple scenarios. Hypergraph learning, through its flexible hyperedges, models complex, high-order, and multimodal data relationships, which is closer to the real object structure, has a wider perception domain and stronger high-order correlation modeling capabilities, and is more robust to noise.

[0091] To this end, this embodiment constructs a target detection and recognition model based on a hypergraph neural network, utilizes the topological features of the hypergraph, models the relationship between multiple nodes, and mines nonlinear high-order correlations, thereby learning more expressive global feature representations to achieve accurate and effective detection and tracking of weak targets in infrared videos.

[0092] In Hypergraph S G In each data point I j Considered as a hypergraph S G A vertex on vj , we assume I j is a central vertex, its k Nearest Neighbors and Hyperedges e Therefore, each hyperedge contains k +1 vertex, I j The corresponding adjacency matrix h ( v j , e m ) is defined as:

[0093]

[0094] in, Represents the pairwise Euclidean distance I j The neighborhood set of

[0095] After that, by designing the vertex convolution layer (i.e. edge graph W E ) and edge convolution layers (vertex graph W V ) Extract low-order structural information of the hypergraph;

[0096] Apply the edge graph and vertex graph to The secondary edge-level and vertex-level structural information aggregation enhances the low-order structural information and directly extracts high-order structural information from the hypergraph through vertex-edge convolution (hyperedges connecting multiple vertices);

[0097] Enhanced edge map and enhanced vertex graph The expression is:

[0098]

[0099] in, represents the number of sequence numbers of hyperedges, Indicates the number of times information is aggregated. n Indicates the total number of current infrared images. H j express I j The hypergraph adjacency matrix of Conv_e and Conv_v Represent the vertex convolution layer and edge convolution operation respectively, Indicated in The total weight of the hyperedges of the initial infrared image under the aggregation, Indicated in The total vertex weight of the initial infrared image under the secondary aggregation, Indicated in Next aggregation j The total weight of the hyperedges of infrared images, Indicated in Next aggregation j Infrared image corresponding to the hyperedge eq and ep The weight between Indicated in Next aggregation j Infrared image corresponding to the hyperedge ek and eq The weight between Indicated in Next aggregation j The total vertex weight of infrared images, Indicated in Next aggregation j The corresponding vertices of the infrared image vq and vp The weight between Indicated in Next aggregation j The corresponding vertices of the infrared image vk and vq The weight between .

[0100] The edge graph and enhanced vertex graph Fusion, get a low-order hypergraph , the edge graph and enhanced vertex graph Aggregate the edge vertex structure information in to obtain a high-order hypergraph , low-order hypergraph and high-order hypergraphs The expression is:

[0101]

[0102] in, Fuse (.) indicates the structure information aggregation operation, Indicates the number of times information is aggregated. Indicated in t Second aggregation j A high-order hypergraph of an infrared image, Indicated in t Second aggregation j Infrared image hyperedge eq and ep , and the vertex vq and vp The high-order hypergraph under the structural information of Indicated int 2-1st aggregation j Infrared image at the vertex vp The following corresponds to different hyperedges ek The edge hypergraph formed by Indicated in t 2-1st aggregation j Infrared image corresponding to the hyperedge ep The following correspond to different vertices vk The vertex hypergraph formed.

[0103] By and Fusion achieves effective combination of high-order and low-order structural information to enhance the global feature representation capability, and its expression is:

[0104]

[0105] The full local semantic features of the target are extracted from the high-order hypermap, and the weak local background related information is extracted from the low-order hypermap. The combination of high-order and low-order information can enhance the semantic features such as the orientation and size information of small targets in infrared images. By learning the local information in the same frame such as contour and amplitude, the utilization of limited geometric features is realized, the background noise is further suppressed, the pseudo-targets are eliminated, and the false alarm rate of the model is reduced.

[0106] Embodiment 4:

[0107] This embodiment is a supplement to Embodiment 1 or Embodiment 3, and is used to supplement the explanation of how to avoid the loss of neighborhood information in the graph nodes in the hypergraph.

[0108] In this embodiment, in order to alleviate the problem of difficulty in detecting infrared weak targets caused by factors such as rapid motion, target occlusion and similar background interference, considering the strong temporal continuity between infrared video frames, a graph capsule network is used to obtain the temporal relationship of weak targets between different frames, avoid the loss of neighborhood information in the graph nodes, and expand the nodes in the hypergraph operation from the scalar domain to the capsule form of the vector domain. The spatiotemporal continuity information is effectively captured through the self-attention routing mechanism, and the capsules containing similar features in different frames are adaptively selected for aggregation, so as to realize long-term dependency modeling of the features of weak infrared targets and avoid target loss, thereby enhancing the detection and tracking performance of infrared weak targets. Specifically, the first l Layer j The graph node capsule is , No. l The graph node capsules at layer +1 are ;

[0109] In this embodiment, the steps of implementing target detection and recognition through the graph capsule network specifically include:

[0110] A1. Each capsule Affine transformation matrix between two adjacent capsule layers Multiply them together to get the prediction vector for each capsule ;

[0111] The step A1 is used to project all capsules in each layer to the next layer;

[0112] A2. The prediction vector and coupling coefficient Combined with the calculation of l +1 layer of capsule output , whose expression is:

[0113]

[0114] in, is the prior matrix, which is used to make similar capsules more closely connected; is the prediction vector The transpose of To calculate the coupling coefficient through the self-attention algorithm, its expression is:

[0115]

[0116] in, For the l Layer and l +1 layer of self-attention matrix, Included j l The similarity measure of each capsule is used to calculate the corresponding coupling coefficient to learn the variability between different capsules; is the coefficient used to balance the coupling coefficient and the logarithmic prior to obtain a stable training process;

[0117] A3. Learn the differences between capsule graph nodes through self-attention routing strategy to assign different weights to them;

[0118] The purpose of step A3 is to enhance the feature aggregation capability of the hypergraph network, improve the ability to distinguish suspicious pixels on different frames in the infrared video, further suppress background noise and eliminate pseudo targets, and reduce the false alarm rate of the model;

[0119] A4. Generate target proposal regions through Region Proposal Networks (RPN), and use target proposal regions to achieve accurate and effective detection and tracking of weak targets in infrared videos.

[0120] Embodiment 5:

[0121] This embodiment is a supplement to Embodiments 1-4 and is used to provide a specific embodiment for realizing lightweight hypergraph network target detection and recognition model.

[0122] In this embodiment, in order to reduce the computational complexity and storage overhead of the deep learning network and improve its ability to meet practical application requirements, it can successfully complete satellite / airborne infrared target detection and recognition tasks. To this end, a lightweight optimization scheme for the above-mentioned deep network model is explored. Most networks have many convolutional layer parameters and low computational efficiency, so a sparse low-rank pruned separable convolution optimization strategy is proposed to reduce the amount of calculation and the amount of parameters.

[0123] First, depthwise separable convolution is introduced to replace conventional convolution operations. The core idea of ​​the depthwise separable convolution structure is to divide the calculation of a standard convolution layer into two steps: depthwise convolution layer and pointwise convolution layer, which are used to learn channel correlation and spatial correlation decoupling respectively.

[0124] Assume that the input feature map has N , size is A × B , output M feature maps, the standard convolutional layer uses N The size is k × k The convolution kernel size is k × k , the actual size of the operation is k × k × M ; Therefore, the computational complexity of the standard convolution can be expressed as A × B × N × k × k × M .

[0125] In the deep convolution layer, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel; that is: A × B × C × N The input feature map is divided into group = C Group, for each group of feature maps, k × k convolution; therefore, the computational complexity of this structure can be expressed as A × B × k × k × M (It is 1 / 1 of the computational effort of the standard convolutional layer N ).

[0126] Point convolution achieves "mixing" of information between feature channels. A × B × C × N The input is done M A 1×1 convolution operation changes the output dimension of the feature. The convolution method here is the same as the calculation method of the standard convolution layer, except that a 1×1 convolution kernel is used. The purpose is to make each new feature map contain the information of all feature maps of the input layer, and the calculation amount of the 1×1 convolution layer is A × B × N × M .

[0127] Based on the above analysis, compared with the standard convolution layer, the reduced computational effort of the depthwise separable convolution structure can be expressed as follows:

[0128]

[0129] in, is the convolution kernel size of the standard convolution layer, M is the number of input feature map channels, and N are the input and output feature map sizes respectively. From the above formula, we can see that the reduction in parameters is mainly related to In infrared images, N =1, assuming is the conventional minimum scale 3, that is =3, it reduces the amount of calculation by about 8~9 times, and when The larger it is, the more the computational effort is reduced.

[0130] Then, the above-mentioned depth-separable convolutional layer is subjected to sparse low-rank decomposition and pruning optimization;

[0131] Assume that the weight matrix of the convolution layer in a group in the current depth-separable convolution structure is expressed as ;in, R represents the field of real numbers, d Indicates the depth of the convolution kernel of the current convolution layer, c is the number of channels of the convolution kernel of the current convolution layer, and the weight matrix of the convolution layer Apply sparse low-rank constraints to improve the improved sparse low-rank convolution layer, decomposing the weight matrix into a low-rank matrix L and sparse matrices S , expressed as follows:

[0132]

[0133] Among them, the low-rank term L The rank ofr ,rank r The size of will affect the entire network structure. The rank of the matrix measures the correlation between the rows and columns of the matrix. Because the weight vectors are mainly distributed in some low-rank subspaces, using a few bases to reconstruct the weight matrix can remove redundancy and reduce parameters.

[0134] In order to reduce computational complexity and storage space, the low-rank matrix items L Further factorization is: L = UV ;

[0135] in, , , by taking the low-rank matrix L The standard convolutional layer is decomposed into a convolutional layer with a smaller number of channels but the same convolutional kernel size and a 1×1 convolutional layer, that is, multiple small matrices or tensors are used to replace the original matrix or tensor, with a rank of r The smaller the value, the lower the storage and computational complexity.

[0136] Therefore, the loss function of this part of the sparse low-rank constraint term is defined as:

[0137]

[0138] in, is a balancing parameter used to balance performance and sparsity. ii =1,2,3 ... c Indicates the convolution kernel of the current convolution layer. ii channels, c is the total number of convolution kernel channels, L ii Indicates ii The low-rank matrix term under the channel, S ii Indicates ii The sparse matrix entries under the channel, U ii and V ii Respectively represent ii The low-rank matrix entry under the channel L ii The two submatrix entries of the decomposition. Func (.) represents the network convolutional layer function, Func ( W ) represents the weight of the current convolutional layer, and min represents the minimum.

[0139] This step mainly imposes sparse low-rank constraints on the calculation process of the convolution layer to perform lightweight. WThe matrix rank is greatly reduced, and more information can be carried through a smaller low-rank matrix, trying to restore the representation ability of the complete weights. However, since only the redundant weights are reset to 0, the memory space occupied by the data has not changed, and the overall data volume has not decreased, but the computing efficiency has been improved.

[0140] In order to further reduce the convolution layer weight parameters, this embodiment performs sparse constraint pruning on the above output feature channels and outputs the weight matrix W Calculated according to its channel dimension L 1 norm, as shown below:

[0141]

[0142] in, express L 1 norm;

[0143] Set the clipping threshold P ,Will L The 1-norm results are sorted from small to large, and the weight matrix is ​​calculated according to the following rules: W Channel dimension clipping:

[0144]

[0145] Therefore, the channels with a norm of 0 in the matrix are discarded, and the non-zero channels are retained, thereby deleting redundant weight data, freeing up memory space, and reducing the model size to a certain extent. Subsequently, after completing layer-by-layer channel pruning, the model needs to be retrained so that the remaining weights in the model relearn the feature distribution, enhance the expressiveness of the weight parameters and the correlation of the weight parameters, and alleviate some precision losses.

[0146] In addition, a substantial reduction in the number of model parameters will inevitably lead to a loss in the accuracy of the entire network model. In order to recover the performance loss caused by the reduction in the number of network model parameters, a network input feature distillation algorithm is introduced to achieve performance recovery. In addition to the conventional classification loss and regression loss, the feature distillation algorithm also adds a feature loss constraint. The formula is as follows:

[0147]

[0148] in, is a balancing parameter used to balance the weights of the two losses. is the loss function without using knowledge distillation, which includes classification loss and bounding box regression loss; , represents the 2-norm, represents the characteristics of a certain layer of the teacher network, After lightweighting, the loss corresponds to the features in the same position. It uses Euclidean distance to measure the difference between two features and then approximates the distribution. By transferring knowledge, it achieves performance recovery to meet the requirements of actual application scenarios for detection and recognition accuracy.

[0149] This embodiment makes the deep sparse low-rank separable convolution structure not only streamline the network structure, but also ensures the sparse and low-rank nature of the weight matrix in each convolution layer. While minimizing the accuracy loss of the deep detection and tracking model, it effectively reduces the computational and storage overhead of the entire deep model, thereby achieving robust, real-time infrared dim target detection and tracking, and further solving the deployment problem of deep learning network models under the conditions of limited onboard / airborne computing power and storage resources.

[0150] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting and tracking small targets in infrared images based on a lightweight hypergraph network, characterized in that: include: S1, collect infrared images, and build a complete infrared target database through the semantic feature variational autoencoder of infrared images; S1 includes: S11, collect infrared images, and use Riemannian manifold learning to extract semantic features containing geometric attribute information in the infrared images; S12. According to the semantic features of infrared images, a semantic feature variational autoencoder is constructed through variational inference theory; S13, training a semantic feature variational autoencoder, using the trained semantic feature variational autoencoder to expand the infrared image target information image, and constructing a complete infrared target database; In S13, when training the semantic feature variational autoencoder, the network parameters in the generator, decoder, and discriminator are optimized by the back propagation algorithm, and the loss function is L C for: in, KL (.) To find the relative quotient operation, and is the trainable model parameter of the semantic feature variational autoencoder, z is a hidden variable that follows a priori distribution , is the conditional probability of the generator, is used to balance the reconstruction loss and KL The weight hyperparameter between the divergences, is the approximate posterior of the decoder distribution, G (.) indicates a generator, D ( I )and D ( I fc ) represent the discriminator's response to the real image I and the resulting reconstructed image I fc The probability of true or false judgment, E I is the expected operation for judging the authenticity of the real image. is the expected operation for judging the authenticity of the generated reconstructed image. is the expected operation of semantic conditional loss, is the expectation operation of the reconstructed image loss; S2. Use the complete infrared target database to build and train the hypergraph network target detection and recognition model; In S2, the steps of detecting and recognizing small targets in infrared images by the hypergraph network target detection and recognition model include: S21. Construct edge graph W E and the vertex graph W V , and through the edge graph W E and the vertex graph W V Extract low-level structural information of hypergraphs; S22, extracting high-order structural information of the hypergraph from the hypergraph through vertex-edge convolution; S23, the edge map W E and the vertex graph W V Aggregate edge-level and vertex-level structural information to obtain an enhanced edge graph and enhanced vertex graph ; S24, the edge map and enhanced vertex graph Fusion, get a low-order hypergraph , the edge graph and enhanced vertex graph Aggregate the edge vertex structure information in to obtain a high-order hypergraph ; S25. Through high-order hypergraph Extract the full local semantic features of small targets in infrared images through low-order hypergraphs Extract weak local background related information; S26, combining the low-order structural information of the hypergraph with the high-order structural information of the hypergraph to enhance the orientation and size information of the weak target in the infrared image; S27, taking the full local semantic features of the infrared image small target, the weak local background related information, and the orientation and size information of the infrared image small target as the detection and recognition result; S3. Use the hypergraph network target detection and recognition model to detect and track small targets in infrared images.

2. The infrared image small target detection and tracking method based on lightweight hypergraph network according to claim 1 is characterized in that: The S11 includes: S111, extracting manifold features of the infrared image on the symmetric positive definite matrix manifold as a covariance matrix; S112, mapping the covariance matrix to the tangent space, and finally forming a semantic feature vector in the Euclidean space as a semantic feature containing geometric attribute information in the infrared image.

3. The infrared image small target detection and tracking method based on lightweight hypergraph network according to claim 1 is characterized in that: In S12, the semantic feature variational autoencoder includes a generator, a decoder and a discriminator; The generator is used to extract semantic features of infrared images; The decoder is used to generate a reconstructed infrared image according to the semantic features of the real infrared image; The discriminator is used to distinguish between real infrared images and reconstructed infrared images.

4. The infrared image small target detection and tracking method based on lightweight hypergraph network according to claim 1 is characterized in that: In S23, the enhanced edge map and enhanced vertex graph The expression is: in, represents the number of sequence numbers of hyperedges, Indicates the number of times information is aggregated. n Indicates the total number of current infrared images. H j express I j The hypergraph adjacency matrix of Conv_e and Conv_v Represent the vertex convolution layer and edge convolution operation respectively, Indicated in The total weight of the hyperedges of the initial infrared image under the aggregation, Indicated in The total vertex weight of the initial infrared image under sub-aggregation, Indicated in Next aggregation j The total weight of the hyperedges of infrared images, Indicated in Next aggregation j Infrared images correspond to hyperedges eq and ep The weight between Indicated in Next aggregation j Infrared images correspond to hyperedges ek and eq The weight between Indicated in Next aggregation j The total vertex weight of infrared images, Indicated in Next aggregation j The corresponding vertices of the infrared image v and vp The weight between Indicated in Next aggregation j The corresponding vertices of the infrared image vk and v The weight between .

5. The infrared image small target detection and tracking method based on lightweight hypergraph network according to claim 4 is characterized in that: In S24, the low-order hypergraph and high-order hypergraphs The expression is: in, Fuse (.) indicates the structure information aggregation operation, Indicates the number of times information is aggregated. Indicated in t Second aggregation j A high-order hypergraph of an infrared image, Indicated in t Second aggregation j Infrared image hyperedge eq and ep , and the vertex v and vp The high-order hypergraph under the structural information of Indicated in t 2-1st aggregation j Infrared image at the vertex vp The following corresponds to different hyperedges ek The edge hypergraph formed by Indicated in t 2-1st aggregation j Infrared images correspond to hyperedges ep The following correspond to different vertices vk The vertex hypergraph formed.

Images