Infrared space adjacent target super-resolution sub-pixel positioning method

Abstract

The invention discloses an infrared space adjacent target super-resolution sub-pixel positioning method, which relates to the technical field of image processing, and is characterized by comprising the following steps: S1, obtaining an area containing a space adjacent target in an infrared image; s2, constructing an infrared space adjacent target VOC format data set; s3, the improved YOLOv5 neural network is trained; s4, inputting the infrared image into the trained improved YOLOv5 network, outputting the number of targets in the CSOs, and completing super-resolution processing; and S5, taking the number of the targets in the step 4) as input, and obtaining the sub-pixel centroid position of each target on the image plane by using an improved FCM clustering algorithm. According to the method, a small target detection layer and a convolution attention module are added in a training network, so that the training network is more sensitive to point target feature extraction, and the target number estimation effect of CSOs is improved; and an FCM clustering algorithm is combined with Lanczos3 interpolation and particle distribution, so that precise centroid positioning of the sub-pixels of the CSOs is realized, and prior information is provided for follow-up processing such as tracking and three-dimensional positioning.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for super-resolution sub-pixel positioning of adjacent targets in infrared space. Background Art

[0002] As human beings explore space more and more deeply, the application of space-based infrared target search and tracking technology in military and civilian fields is becoming more and more extensive. Space-based infrared detection system is an important means to achieve space monitoring, tracking and positioning. It has the advantages of good concealment, no restrictions on meteorological conditions, and all-weather operation. Due to the long infrared detection distance, whether it is a mid-stage target group, space debris or stars, it usually appears as a point target of several to dozens of pixels relative to the space-based infrared sensor. During the detection process, the target may change its posture and number due to reasons such as releasing bait or accompanying flight, forming a group of targets with similar distances, namely, spatially adjacent targets (CSOs). Due to the structural characteristics of the optical system, spatially adjacent targets present overlapping clustered image spots on the image plane, resulting in the inability to distinguish important information such as the number and position of the target. Therefore, it is of great significance to study the super-resolution algorithm of infrared spatially adjacent targets and distinguish the number of targets and the position parameters of each target from the image spots.

[0003] The algorithm of infrared spatial neighboring target super-resolution and sub-pixel positioning method is essentially to estimate the parameters of point targets using image plane response model and noise model. The early research on infrared CSOs super-resolution algorithm was mainly based on Bayesian theory. The infrared imaging model was used to establish the objective function based on maximum a posteriori probability estimation (MAP), maximum likelihood estimation (ML) or least squares estimation (LS), and the optimization algorithm was used to solve the target information. However, the objective function optimization problem of this method is complex and has obvious shortcomings in running speed. There is also a class of CSOs super-resolution methods based on sparse reconstruction theory. Through discretized sampling, an over-complete dictionary is constructed, a sparse measurement model is established, and the norm optimization problem is converted into a second-order cone programming problem for solution. The super-resolution algorithm based on sparse reconstruction theory can estimate the target position well and is less affected by noise, but the algorithm is complex, time-consuming and requires high prior knowledge. There is also a class of methods that decompose pixel clusters by distributing particles and then iterate the optimal solution using a clustering algorithm. This method can estimate a large number of targets, but it has the disadvantage of being sensitive to noise.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) Traditional infrared CSOs super-resolution and sub-pixel localization methods have weak target resolution capabilities. In particular, as the number of targets in a spatially adjacent target cluster increases, the accuracy of CSO number estimation decreases sharply. Traditional algorithms segment multi-target areas and calculate the number of targets based on the Gaussian features of infrared spatially adjacent targets. When two targets are highly overlapped or the number of aliased targets increases, traditional algorithms tend to fail.

[0006] (2) There is a prominent contradiction between the existing infrared CSOs super-resolution and sub-pixel positioning methods and the positioning accuracy of each target image plane and the algorithm running time. For example, the super-resolution positioning method based on Bayesian theory and sparse reconstruction theory has high target positioning accuracy but long calculation time; while the clustering algorithm runs fast but has low accuracy in extracting the centroid of the target image plane.

[0007] (3) Due to the long detection distance and weak target energy, the signal-to-noise ratio of infrared images is generally low. The current infrared CSOs super-resolution and sub-pixel positioning methods are greatly affected by image noise and have the disadvantage of poor noise robustness. Summary of the invention

[0008] The purpose of the present invention is to solve the above problems and provide a super-resolution sub-pixel positioning method for infrared spatial neighboring targets, which can use an improved YOLOv5 network to accurately estimate the number of spatial neighboring targets, and then use an improved fuzzy C-means clustering method to accurately estimate the corresponding target centroid positions in CSOs.

[0009] In order to achieve the above object, the technical solution of the present invention is as follows: a super-resolution sub-pixel positioning method for infrared space neighboring targets, the positioning method is based on an improved YOLOv5 network and a fuzzy C-means clustering algorithm, comprising the following steps:

[0010] S1: Acquire an infrared spatial neighboring target image, wherein the infrared spatial neighboring target image contains a region of spatial neighboring targets CSOs;

[0011] S2: Construct a VOC format dataset containing infrared spatial neighboring targets of different numbers and sizes;

[0012] S3: Train the improved YOLOv5 neural network;

[0013] S4: Input the infrared image into the trained improved YOLOv5 network, output the number of targets in CSOs, and complete the super-resolution processing;

[0014] S5: Taking the number of targets in S4 as input, the improved fuzzy C-means clustering algorithm is used to obtain the sub-pixel centroid position of each target on the image plane.

[0015] Preferably, in S1, the infrared space neighboring target image is obtained by simulation through an infrared target simulation system, and the number and position information of the targets in the infrared space neighboring target image are stored simultaneously.

[0016] Preferably, in S2, the construction method comprises the following steps:

[0017] S21: re-storing the infrared image target quantity and position information stored in S1 as an xml file in the VOC dataset format;

[0018] S22: Divide the infrared image dataset into two parts: training set and test set.

[0019] Preferably, in S3, the improved method of the improved YOLOv5 neural network is as follows:

[0020] A small target detection layer is added to the Neck and Backbone connection part of the YOLOv5 network, and the feature map is continued to be upsampled to obtain a larger feature map to increase the network's ability to extract small target features. At the same time, a convolutional attention module is added to the C3 module of the Backbone part of the YOLOv5 network to enable the network to focus on important target features and ignore unnecessary responses.

[0021] Preferably, in S5, the improved method of the improved fuzzy C-means clustering algorithm is: based on the pseudo downsampling idea, the fuzzy C-means clustering algorithm is combined with Lanczos3 interpolation and particle distribution.

[0022] Preferably, the acquisition of the sub-pixel centroid position comprises the following steps:

[0023] S51: Perform Lanczos3 interpolation processing on the infrared image:

[0024]

[0025] Among them, s uv represents the image pixel with coordinates (u, v), G(x) represents the Lanczos3 interpolation function;

[0026] S52: Perform convolution processing on the interpolated and expanded image to highlight the features of the target pixel; the convolution template is defined as:

[0027]

[0028] The target area is processed by zero padding of the surrounding area to obtain the target pixel matrix I;

[0029]

[0030] Slide the convolution mask on the pixel matrix to obtain the oversampled interpolation pixel matrix I p ;

[0031]

[0032] S53: According to the ratio of the gray value of each pixel in the pixel matrix to the total gray value of the target cluster, a certain number of particles are distributed on the interpolation pixel matrix obtained in S52; the number of particles distributed on the mth pixel of the interpolation pixel matrix is ​​expressed as N m :

[0033]

[0034] Among them, S m Represents the grayscale value of the mth pixel, and n represents the entire pixel matrix I p The total number of pixels, N IP Represents the total number of particles distributed on the pixel matrix;

[0035] S54: down-sample the image, map the interpolated pixel matrix back to the original image size, and obtain a sub-pixel particle set;

[0036] S55: Taking the estimated value of the number of spatially adjacent targets obtained in S4 as input, the FCM algorithm is used to cluster the sub-pixel particle set, and after continuous iterations, the sub-pixel cluster center representing the centroid position of the spatially adjacent targets is output.

[0037] Compared with the prior art, this solution has the following beneficial effects:

[0038] (1) The present invention uses a neural network instead of a traditional clustering algorithm to estimate the number of CSOs. The trained neural network learns the characteristics of various types of spatially adjacent targets and can classify more types of spatially adjacent target areas according to the extracted CSOs features, thus having an advantage in terms of the accuracy of quantity estimation.

[0039] (2) The present invention adds a small target detection layer and a CBAM attention mechanism to the YOLOv5 network to enhance the network’s sensitivity to small targets. These improvements enable the YOLOv5 network to better extract small target features from CSOs and improve the accuracy of quantity estimation without taking too much time.

[0040] (3) Considering that CSOs occupy fewer pixels and lack effective information in infrared images, the estimation accuracy of the target centroid position is greatly affected. Based on the pseudo-downsampling idea, the present invention uses Lanczos3 interpolation and particle distribution to highlight the grayscale characteristics of the CSOs target area and improve the FCM algorithm's extraction accuracy of the target centroid position. Under the condition that the number of particles is used properly, both the algorithm accuracy and the running time can obtain better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a method for super-resolution sub-pixel positioning of an infrared space neighboring target in an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the improved YOLOv5 network structure constructed in an embodiment of the present invention for realizing super-resolution of adjacent targets in infrared space;

[0043] Figure 3 is a schematic diagram of a confusion matrix of a network training result of target quantity classification of infrared CSOs images containing 2 to 7 targets in an embodiment of the present invention;

[0044] Figure 4 is an output result diagram of the test image in the embodiment of the present invention.

[0045] Figure 5 Schematic diagram of the image processing process of steps S51-S54 in an embodiment of the present invention; wherein, Figure (a) is the original image, Figure (b) is the extended image after Lanczos3 interpolation and convolution processing, Figure (c) is the particle distribution image, and Figure (d) is the image mapped back to the original size;

[0046] Figure 6 is a schematic diagram of the result of improving the FCM algorithm in step S5 in an embodiment of the present invention;

[0047] Figure 7 1 is a result diagram of the influence of target distance and number and image signal-to-noise ratio on the improved FCM algorithm in step S5 of the embodiment of the present invention; wherein, FIG (a) is a trend of the performance of the improved FCM algorithm changing with target distance and signal-to-noise ratio when the number of targets in CSOs is 2; FIG (b) is a trend of the performance of the improved FCM algorithm changing with the number of targets and signal-to-noise ratio when the number of targets in CSOs is 2 to 7. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below in conjunction with the embodiments.

[0050] Example:

[0051] like Figure 1-7 As shown, a method for super-resolution sub-pixel positioning of infrared spatial neighboring targets comprises the following steps:

[0052] S1: Acquire the area containing the CSOs in the infrared image.

[0053] The infrared space neighboring target image is simulated by an infrared target simulation system, and the number and position information of the targets in the infrared space neighboring target image are stored at the same time.

[0054] In the infrared imaging process, the diffraction effect of the optical system causes the energy of the point target in the imaging plane to spread to its neighboring pixels, forming a central bright spot, called the Airy disk. This diffraction effect is usually approximated by a two-dimensional Gaussian point spread function (PSF), and the expression on the image plane is:

[0055]

[0056] Among them, (x i ,y i ) represents the actual projection position of the target on the image plane, σ psf represents the spread of the target energy, which is determined by the sensor focal length and the detection wavelength. In this example, σ psf Set to 0.5.

[0057] By integrating a certain pixel of the infrared image, we can get the amplitude response of the pixel:

[0058]

[0059] Among them, (u, v) is the coordinate corresponding to the pixel, and d is the pixel size.

[0060] When there is a group of spatially adjacent targets in the image, that is, the number of targets is ≥ 2, the amplitude response of each pixel is the linear superposition of the response of a single target to that pixel.

[0061] Z=[θ1(u,v),θ2(u,v),...,θ K (u,v)]·s+n=Θ·s+n

[0062] Where s is the response intensity vector of each target and n is the noise.

[0063] In this embodiment, 28,000 infrared CSOs simulation images with a size of 64×64 are obtained through the infrared target simulation system. Among them, the number of CSOs targets in each image is between 2 and 7, the size of each target is randomly generated between 3×3, 5×5, 7×7, and 9×9, and the distance between the centroids of two adjacent targets in CSOs is between 0.8 and 3 pixels. During the simulation process, the number of CSOs targets in each image and the centroid position of each target are stored.

[0064] When the infrared imaging detection system operates outside the atmosphere, it will be affected by noise. The noise mainly comes from scattered noise, thermal noise and 1 / f noise. Their distribution approximately follows Gaussian distribution, and the noise of each pixel is independent. Therefore, in order to improve the authenticity, we add Gaussian noise of different intensities to the simulated image to describe it. The signal-to-noise ratio is defined as the ratio of signal power to noise power, and the calculation formula is:

[0065]

[0066] Among them, T represents the minimum value of all target peak gray values, B represents the background gray value, σ n Represents the standard deviation of the noise.

[0067] S2: Construct a VOC format dataset containing infrared spatial neighboring objects of different numbers and sizes.

[0068] Among them, S2 includes the following steps:

[0069] S21: re-store the infrared image target quantity and position information stored in S1 in the XML format required by the VOC dataset format;

[0070] S22: The infrared space neighboring target simulation images and their corresponding XML files are randomly divided into a training set and a test set in a ratio of 7:3.

[0071] S3: Train the improved YOLOv5 neural network.

[0072] Among them, a small target detection layer is added to the Neck and Backbone connection part of the YOLOv5 network, and the feature map is continued to be upsampled to obtain a larger feature map to increase the network's ability to extract small target features. At the same time, a convolutional attention module (CBAM) is added to the C3 module of the Backbone part of the YOLOv5 network to enable the network to focus on important target features and ignore unnecessary responses. The improved YOLOv5 can improve the target number estimation performance of CSOs, and the lightweight network does not cause too much time loss, effectively realizing super-resolution processing of CSOs.

[0073] In this embodiment, the training set in the infrared space neighboring target VOC dataset constructed by S2 is used as input to train the improved YOLOv5 network;

[0074] Figure 3 Confusion matrix of the network training results for target number classification of infrared CSOs images containing 2 to 7 targets.

[0075] S4: Input the infrared image into the trained improved YOLOv5 network, output the number of targets in CSOs, and complete the super-resolution processing;

[0076] Figure 4 It is the output result of the test image provided by the embodiment of the present invention.

[0077] In this embodiment, the test set in the infrared spatial neighboring target VOC dataset constructed by S2 is used as input, and the trained improved YOLOv5 network is used to output the number of targets in the infrared CSOs image to complete the super-resolution processing.

[0078] In the embodiment, in order to verify the effectiveness of the improved network compared to the original network, the noise-free simulated infrared CSOs image test set in S1 and S2 is used for verification, and the results are as follows. It can be seen that the improved YOLOv5 network is 11.1% higher than the mAP@0.5 of the initial network, and the target number estimation accuracy is improved by about 20.6%. Although the speed is slightly reduced, it can still meet the real-time requirements.

[0079] Table 1 Comparison of CSOs super-resolution performance of improved YOLOv5 network

[0080]

[0081] S5: Taking the number of targets in S4 as input, the improved fuzzy C-means (FCM) clustering algorithm is used to obtain the sub-pixel centroid position of each target on the image plane.

[0082] like Figure 5-7 As shown in S5, based on the pseudo downsampling idea, the FCM clustering algorithm is combined with Lanczos3 interpolation and particle distribution to achieve sub-pixel accurate centroid positioning of CSOs. The improved FCM clustering algorithm includes the following steps:

[0083] S51: Perform Lanczos3 interpolation processing on the infrared image:

[0084]

[0085] Among them, s uv Represents the image pixel with coordinates (u, v), and G(x) represents the Lanczos3 interpolation function.

[0086]

[0087] S52: Perform convolution processing on the interpolated and expanded image to highlight the features of the target pixel. The convolution template is defined as:

[0088]

[0089] The target area is filled with zeros around the target area to obtain the target pixel matrix I.

[0090]

[0091] Slide the convolution mask on the pixel matrix to obtain the oversampled interpolation pixel matrix I p .

[0092]

[0093] S53: According to the proportion of the gray value of each pixel in the pixel matrix to the total gray value of the target cluster, a certain number of particles are distributed on the interpolation pixel matrix obtained in S52. The number of particles distributed on the mth pixel of the interpolation pixel matrix is ​​expressed as N m :

[0094]

[0095] Among them, S m Represents the grayscale value of the mth pixel, and n represents the entire pixel matrix I p The total number of pixels, Indicates the total number of particles distributed on the pixel matrix.

[0096] S54: down-sample the image, map the interpolated pixel matrix back to the original image size, and obtain a sub-pixel particle set.

[0097]

[0098] Among them, (x g ,y g ) represents the coordinates of the particle in the original target area, (x p ,y p ) represents the coordinates of the particle in the interpolated pixel matrix.

[0099] S55: Taking the estimated value of the number of spatially adjacent targets obtained in S4 as input, clustering the sub-pixel particle set using the FCM algorithm, and after continuous iteration, outputting the sub-pixel cluster center representing the centroid position of the spatially adjacent targets. S55 specifically includes the following steps:

[0100] The objective function of the FCM algorithm is defined as:

[0101]

[0102] Among them, m is the membership factor; d ij Indicates the distance between the sample and the cluster center; u ij It is called membership degree, which indicates the probability that the jth sample in the data set belongs to the ith category. The expression of the above parameters is expressed as:

[0103] d ij =||χ j -v i || 2

[0104]

[0105] in, Represents the set of particles distributed on the pixel, each particle can be expressed as

[0106] v i Represents the cluster center, and the iteration formula is as follows:

[0107]

[0108] In the iterative process, the membership u ij The following constraints should be met:

[0109]

[0110] In this embodiment, in order to verify the effectiveness of the proposed method in a low signal-to-noise ratio infrared space adjacent target scene with a large number of targets and severe aliasing, the simulation data set in S1 is classified according to the number of targets in the image and the distance between the targets as the input of S5.

[0111] The algorithm performance evaluation indicator is the joint root mean square error (JRMSE) of target positioning estimation:

[0112]

[0113] Experimental results show that the target centroid positioning performance of the improved algorithm proposed by S5 decreases with the increase of the number of targets, the decrease of the target distance and the increase of noise. The improved algorithm is effective when SNR≥10dB, but the JRMSE increases with the decrease of signal-to-noise ratio. When there are only two targets in the CSOs area and SNR≥15dB, the JRMSE can be controlled within 0.3 pixels. When the number of targets is between 2 and 7 and SNR≥15dB, the JRMSE of target position estimation is less than 0.3 pixels.

[0114] The above specific embodiments are merely explanations of the present invention and are not limitations of the present invention. After reading this specification, those skilled in the art may make modifications to the embodiments without any creative contribution as needed. However, such modifications are protected by the patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A super-resolution sub-pixel positioning method for infrared space neighboring targets, the positioning method is based on an improved YOLOv5 network and a fuzzy C-means clustering algorithm. It is characterized in that The following steps are involved: S1: Acquire an infrared spatial neighboring target image, wherein the infrared spatial neighboring target image contains a region of spatial neighboring targets CSOs; S2: Construct a VOC format dataset containing infrared spatial neighboring targets of different numbers and sizes; S3: Train the improved YOLOv5 neural network; S4: Input the infrared image into the trained improved YOLOv5 network, output the number of targets in CSOs, and complete the super-resolution processing; S5: Taking the number of targets in S4 as input, the improved fuzzy C-means clustering algorithm is used to obtain the sub-pixel centroid position of each target on the image plane.

2. According to claim 1, a method for super-resolution sub-pixel positioning of adjacent targets in infrared space, Features: In S1, the infrared space neighboring target image is obtained by simulation through an infrared target simulation system, and the number and position information of the targets in the infrared space neighboring target image are stored.

3. According to claim 1, a method for super-resolution sub-pixel positioning of adjacent targets in infrared space, It is characterized in that In S2, the construction method comprises the following steps: S21: re-storing the infrared image target quantity and position information stored in S1 as an xml file in the VOC dataset format; S22: Divide the infrared image dataset into two parts: training set and test set.

4. The method for super-resolution sub-pixel positioning of adjacent targets in infrared space according to claim 1, It is characterized in that In S3, the improved method of the improved YOLOv5 neural network is as follows: A small target detection layer is added to the Neck and Backbone connection part of the YOLOv5 network, and the feature map is continued to be upsampled to obtain a larger feature map to increase the network's ability to extract small target features. At the same time, a convolutional attention module is added to the C3 module of the Backbone part of the YOLOv5 network to enable the network to focus on important target features and ignore unnecessary responses.

5. The method for super-resolution sub-pixel positioning of adjacent targets in infrared space according to claim 1, It is characterized in that In S5, the improved method of the improved fuzzy C-means clustering algorithm is: based on the pseudo downsampling idea, the fuzzy C-means clustering algorithm is combined with Lanczos3 interpolation and particle distribution.

6. The method for super-resolution sub-pixel positioning of adjacent targets in infrared space according to claim 5, It is characterized in that The acquisition of the sub-pixel centroid position comprises the following steps: S51: Perform Lanczos3 interpolation processing on the infrared image: Among them, s uv represents the image pixel with coordinates (u, v), G(x) represents the Lanczos3 interpolation function; S52: Perform convolution processing on the interpolated and expanded image to highlight the features of the target pixel; the convolution template is defined as: The target area is processed by zero padding of the surrounding area to obtain the target pixel matrix I; Slide the convolution mask on the pixel matrix to obtain the oversampled interpolation pixel matrix I p ; S53: According to the ratio of the gray value of each pixel in the pixel matrix to the total gray value of the target cluster, a certain number of particles are distributed on the interpolation pixel matrix obtained in S52; the number of particles distributed on the mth pixel of the interpolation pixel matrix is ​​expressed as N m : Among them, S m Represents the grayscale value of the mth pixel, and n represents the entire pixel matrix I p The total number of pixels, N IP Represents the total number of particles distributed on the pixel matrix; S54: down-sample the image, map the interpolated pixel matrix back to the original image size, and obtain a sub-pixel particle set; S55: Taking the estimated value of the number of spatially adjacent targets obtained in S4 as input, the FCM algorithm is used to cluster the sub-pixel particle set, and after continuous iterations, the sub-pixel cluster center representing the centroid position of the spatially adjacent targets is output.

Images