Anti-interference target recognition method based on fusion of multi-band spectral features

Abstract

The invention discloses an anti-interference target recognition method integrating multi-band spectral features. After performing morphological opening operation on a sequence of infrared multi-band images of a target under different interferences, an improved watershed algorithm is used to segment the target and interference in the image without distinguishing categories. The segmented sub-images are divided into a training set, a verification set and a test set, which are used to train and verify a recognition module including an information fusion module and an image binary classification convolutional neural network. In the actual measurement stage, a complete multi-band spectral infrared image is segmented into an indefinite number of sub-images, which are then merged as a batch and input into the trained recognition module, and the classification probabilities of all sub-images are calculated in parallel. The invention can effectively suppress the influence of complex interference on the target detection and recognition accuracy by performing real-time recognition on the target and interference in the sequence of multi-band infrared images collected by an infrared multi-band detector.

Description

Technical Field

[0001] The present invention relates to a technology in the field of digital image processing and pattern recognition, in particular to an anti-interference target recognition method integrating multi-band spectral features. Background Art

[0002] Infrared target recognition belongs to the field of pattern recognition, and is mainly aimed at the recognition of surface targets. With the development of infrared jamming technology, how to realize the recognition of point targets has become a bottleneck problem that affects practical applications. Since point targets have no shape information, the spectral characteristics become an important feature quantity. Different objects have different spectral radiation characteristics under different motion states. Capturing the fine spectral characteristics of the target can realize the recognition and confirmation of the target. Since the current infrared and laser jamming systems only simulate targets within a wide spectral range, it is difficult or even impossible for the jamming system to be consistent with the target in terms of the fine spectral structure characteristics. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes an anti-interference target recognition method that integrates multi-band spectral features, which can perform real-time recognition of targets and interferences in a multi-band infrared image sequence collected by an infrared multi-band detector, and can effectively suppress the influence of complex interference on target detection and recognition accuracy.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an anti-interference target recognition method integrating multi-band spectral features. After performing morphological opening operation on a sequence of infrared multi-band images of a target under different interferences, an improved watershed algorithm is used to segment the target and interference in the image without distinguishing categories. The segmented sub-images are divided into a training set, a verification set and a test set, which are used to train and verify a recognition module including an information fusion module and an image binary classification convolutional neural network. In the actual measurement stage, a complete multi-band spectral infrared image is segmented into an indefinite number of sub-images, which are merged as a batch and input into a trained recognition module, and the classification probabilities of all sub-images are calculated in parallel to realize anti-interference target recognition.

[0006] The interference includes types of continuous release from a point target in a certain direction or continuous release in all directions. After the interference is released, the infrared area detected will continue to expand. During the acquisition process, the infrared point target is always located in the center of the acquired image.

[0007] The morphological opening operation refers to: first performing an erosion operation on the grayscale image and then performing an expansion operation to eliminate some noise in the image and clarify the boundary between the target and the interference, so as to facilitate subsequent segmentation, specifically: Where: A is the infrared grayscale image to be processed, B is the selected specific structural element, ο represents the opening operation, represents the corrosion operation, Represents an expansion operation.

[0008] The segmentation is to select the infrared image in the first band and segment the target and interference therein without distinction, and segment the infrared images of the remaining bands in the same way as the first band, specifically: threshold segmentation is performed on the grayscale image after the opening operation, and the distance transformation operation is performed after the target or interference area is determined. In the image after threshold segmentation, 1 represents the target or interference area, and 0 represents the background, and a Euclidean distance transformation map is generated.

[0009] Dis(i,j)=min{Distance[(i,j),(x,y)],(x,y)∈B},where:

[0010] Dis is the Euclidean distance transformation map, and B is the set of target areas where all points with a value of 1 in the binary image. Find the valley in the defined extended distance map, that is, the local minimum value as a marker input to the watershed algorithm. The algorithm outputs the coordinates of the segmented target or interference point, and divides it into N×N rectangular sub-images with this as the center. Each sub-image only contains the target or interference. According to the segmentation result of this band, the image of the remaining bands is segmented in the same way, and each segmented sub-image of each band is merged as a channel into a multi-channel sub-image.

[0011] The improved watershed algorithm combines the results of distance transformation and grayscale difference transformation as the extended distance map, marks the segmentation sub-image located at the center of the original image as the target, and marks the remaining sub-images as interference, specifically:

[0012] Where: D is the extended distance map, Dis and Dif are the results of distance transformation and grayscale difference transformation respectively, Dif max and Dif min are the maximum and minimum values ​​of the grayscale difference transformation respectively.

[0013] The ratio of the training set, validation set and test set is preferably 2:2:1, wherein in the segmented sub-image of the same infrared image, if the number of interferences is greater than 5, the 5 interferences closest to the target are taken as samples in the data set.

[0014] The recognition module inputs the single-channel feature map after multi-channel image fusion into the binary classification convolutional neural network to complete the classification of the image. When the network is trained on the multi-band infrared target recognition data set, the input image is firstly subjected to mean normalization.

[0015] The multi-channel image fusion network in the recognition module learns the fusion mode of spectral features of different bands by itself through 1×1 convolution kernel, which is specifically: Where: f(x,y) represents the pixel value at a specific location, N is the number of bands of the acquired infrared image, α i is the parameter learned by the 1×1 convolution kernel in the information fusion module, that is, the weight of each channel in the fused feature map. The multi-channel image fusion network obtains the fused single-channel feature map without changing the size of the multi-channel image. Specifically: F:R N×H×W →R 1×H×W , where: F represents the fusion process, N is the number of bands of the acquired infrared image, and H and W are the height and width of the image.

[0016] The loss function of the training for the image binary classification convolutional neural network is:

[0017] Where: α is a hyperparameter used to balance the imbalance between dataset interference and target ratio, The probability that the network predicts that the sample is a positive example, that is, the target, y is the sample label, if the sample is a target, it is 1, otherwise it is 0.

[0018] The network training described herein uses a batch gradient descent algorithm to train the network while combining regularization methods such as weight decay and momentum.

[0019] The parallel calculation refers to: first, fixing the parameters of the trained multi-channel image fusion network, using the morphological segmentation algorithm to segment the target and interference in the image into sub-images, superimposing them into a batch input multi-channel image fusion network to calculate the classification probability tensor, matching the sub-image with each maximum probability category in the tensor according to the index, obtaining the category of each sub-image, and realizing the distinction between the target and the interference in a complex interference environment.

[0020] The present invention relates to a system for implementing the above method, comprising: a segmentation module, an information fusion module and an image binary classification convolutional neural network, wherein: the segmentation module segments the target and interference in the collected data, and the segmented sub-images are used as inputs of the information fusion module. The information fusion module fuses the spectral features of multiple bands of the same target or interference and inputs them into the image binary classification convolutional neural network. The system finally outputs the position of the target or interference and its corresponding category.

[0021] Technical Effects

[0022] The present invention solves the defect that the prior art is difficult to locate and distinguish targets and interference in infrared images as a whole; the targets and interference in the image are segmented by an improved watershed algorithm. Compared with the segmentation algorithm using the deep learning method, there is no need to label a large amount of data, and the segmentation accuracy is high while having a faster running speed; the information fusion module of the multi-band spectrum can effectively fuse multiple bands of information. Compared with the current single-band target detection and recognition algorithm, the multi-band spectral features are fused to improve the point target detection performance under complex interference conditions, so that the present invention does not need to label a large amount of data as a whole, and the segmentation accuracy is high while having a faster running speed, which significantly improves the point target detection performance under complex interference conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall flow chart of the algorithm of the present invention;

[0024] Figure 2 It is a schematic diagram of infrared image segmentation;

[0025] Figure 3 It is a schematic diagram of the multi-band spectral information fusion module. DETAILED DESCRIPTION

[0026] like Figure 1 As shown, this embodiment relates to an anti-interference target recognition method that integrates multi-band spectral features. First, based on morphological operations and an improved watershed algorithm, the target and interference are segmented indiscriminately from the infrared image, and then a multi-channel image fusion network is combined with an image binary classification convolutional neural network to achieve classification of the target and interference, specifically including:

[0027] The first step is to use an infrared multi-band detector to collect a sequence of images of the same target in five bands under different interferences. In this embodiment, the interference mode is set to be continuously released from the point target downward and continuously released to the surroundings (collecting 3930*5 images and 789*5 images respectively). After the interference is released, the infrared area detected will continue to expand. The resolution of all collected images is 128*128 pixels. The collected infrared image samples are as follows: Figure 2 In order to facilitate the subsequent annotation and training of multi-band infrared target recognition data sets, the target is ensured to be located in the center of the infrared image during the acquisition process to facilitate the subsequent annotation of the data;

[0028] The second step is to select the image of the first band in the multi-band infrared image and perform an opening operation on the image. The process can be expressed as: Wherein: A is the infrared grayscale image to be processed, B is the selected specific structural element, in this embodiment, the shape of B is 3×3, ο represents the opening operation, represents the corrosion operation, Represents dilation operation. Grayscale erosion operation and grayscale dilation operation The processes are:

[0029]

[0030] Where: A(x,y) is the infrared grayscale image to be processed, B(x,y) is the selected specific structure element, and (x,y) is an integer coordinate value pair.

[0031] After obtaining the infrared image after the opening operation, the threshold segmentation method is used to determine the target or interference area.

[0032] The threshold is: T = argmaxW0(t)*[U0(t)-U(t)] 2 +W1(t)*[U0(t)-U(t)] 2 , where: T is the gray value when the above formula takes the maximum value, t is the maximum threshold for segmentation, and all gray values ​​t in the traversal image are used as the segmentation threshold for foreground and background. The proportion of foreground points in the image is W0(t), the average grayscale of the foreground is U0(t), the proportion of background points in the image is W1(t), the average grayscale of the background is U1(t), and the average grayscale of the entire image is U(t) = [W0(t)*U0(t)+W1(t)*U1(t)] / [W0(t)+W1(t)].

[0033] After threshold segmentation, the distance transformation operation is performed. In the image after threshold segmentation, 1 represents the target or interference area, and 0 represents the background. The process of generating the Euclidean distance transformation map can be expressed as:

[0034] Dis(i,j)=min{Distance[(i,j),(x,y)],(x,y)∈B}

[0035] Where: D is the Euclidean distance transformation map, B is the set of target areas where all points with a value of 1 in the binary image are located.

[0036] In order to improve the accuracy of the watershed algorithm, the input of the improved watershed algorithm of this embodiment combines the results of distance transformation and grayscale difference transformation as an extended distance map, which can be expressed as: Where: D is the extended distance map, Dis and Dif are the results of distance transformation and grayscale difference transformation respectively, Dif max and Dif min are the maximum and minimum values ​​of the grayscale difference transformation respectively.

[0037] Find the valley in the defined extended distance map, that is, the local minimum value as a marker input to the watershed algorithm. The algorithm outputs the coordinates of the segmented target or interference point, and divides it into 7×7 rectangular sub-images with this as the center. Each sub-image only contains the target or interference. According to the segmentation result of the first band, the images of the remaining 4 bands are segmented in the same way, and each segmented sub-image of each band is merged as a channel into a 5-channel sub-image. The segmented sub-image located at the center of the original infrared image is marked as the target, and the remaining sub-images are marked as interference. The division of the training set, test set, and validation set is shown in Table 1 below.

[0038] Table 1 Division of multi-band infrared target recognition dataset (sub-images after segmentation)

[0039] Dataset Number of target subgraphs Number of interference subgraphs Training set 2000 6064 Validation set 1930 4972 Test Set 789 3183

[0040] Step 3: This embodiment constructs an information fusion module for 5-channel images, whose convolution kernel size is 1×1, and learns from the data set how to merge multi-channel images into single-channel images. The image processing process can be expressed as: Where: f(x,y) represents the pixel value at a specific location, α i It is the parameter learned by the 1×1 convolution kernel in the information fusion module, that is, the weight of each channel in the fused feature map.

[0041] The information fusion module obtains the fused single-channel feature map without changing the size of the multi-channel image. The process F can be expressed as: F:R 5×7×7 →R 1×7×7 .

[0042] The overall network structure designed in this embodiment uses the single-channel image binary classification MobileNet as the skeleton, and the single-channel feature map after multi-channel image fusion is input into the MobileNet network to complete the image classification. When the network is trained on the multi-band infrared target recognition dataset, the input image is firstly mean normalized.

[0043] For the training described above, the loss function for the image binary classification convolutional neural network is defined as:

[0044] Among them: α is a hyperparameter used to balance the imbalance between the interference and target ratio of the data set, which is set to 0.75 in this embodiment; γ is a hyperparameter used to adjust the rate of reduction of the weight of simple samples, which is set to 2 in this embodiment; The probability that the network predicts that the sample is a positive example, that is, the target, y is the sample label, if the sample is a target, it is 1, otherwise it is 0.

[0045] The network was trained using batch gradient descent (SGD) with the learning rate set to 0.005, batch size set to 64, weight decay set to 0.0005, and momentum set to 0.9. The best result was taken as the final network weight after 300 epochs of training.

[0046] The fourth step is the actual application stage. Infrared detectors are used to collect multi-band spectral infrared images in real time. The segmentation algorithm of this embodiment divides the target and interference in the image into sub-images, and then superimposes them into a batch input convolutional neural network to calculate the classification probability tensor. The sub-images are matched with the maximum probability categories in the tensor according to the index to obtain the category of each sub-image, so as to realize the distinction between targets and interference in complex interference environments, mark the targets with rectangles, and mark the interference with circles.

[0047] This example illustrates the effect of the method proposed in this embodiment through experiments. Through the specific implementation of this example introduced above, the segmentation effect of the segmentation algorithm proposed in this embodiment is as follows: Figure 2 The final experimental results on the self-collected multi-band infrared target recognition dataset are shown in Table 2.

[0048] Table 2 Experimental results of anti-interference target recognition

[0049]

[0050] As can be seen from Table 2, after the infrared images of the five bands collected are segmented and merged into sub-images of size 5×7×7, the method proposed in this embodiment can effectively fuse the information of different bands and accurately distinguish the categories of targets and interferences, with the recognition accuracy rates of targets and interferences being 80.99% and 97.68% respectively. The advantages of the lightweight classification network and hardware parallel computing adopted make the speed of the algorithm meet the real-time requirements of practical applications, and the FPS reaches about 60.

[0051] Based on the collection of multi-band infrared target sequences under different interferences, the proposed morphological image segmentation method in this embodiment accurately segments the target and interference in the image, and forms a multi-band infrared target recognition data set according to the segmentation results. Finally, the proposed information fusion module is combined with the existing classification convolutional neural network to accurately implement the target and interference classification while ensuring real-time performance, effectively integrating the spectral characteristics of multiple bands, and has the characteristics of flexible application and excellent comprehensive performance.

[0052] In summary, the present invention is based on multi-channel image fusion, and through an information fusion module based on 1*1 convolution design, it learns the fusion method of different band spectral features from the segmented sub-image data. It is different from the traditional scheme that relies on one band image for segmentation, so it can more efficiently distinguish between targets and interference.

[0053] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. An anti-interference target recognition method integrating multi-band spectral features, characterized in that: After morphological opening operation of infrared multi-band image sequences of targets under different interferences, the improved watershed algorithm is used to segment targets and interferences in the images without distinguishing categories. The segmented sub-images are divided into training sets, validation sets and test sets for training and validating the recognition module including the information fusion module and the image binary classification convolutional neural network. In the actual measurement stage, the complete multi-band spectral infrared image is segmented into an indefinite number of sub-images and then merged as a batch to input the trained recognition module, and the classification probabilities of all sub-images are calculated in parallel to achieve anti-interference target recognition; The morphological opening operation refers to: first performing an erosion operation on the grayscale image and then performing an expansion operation to eliminate part of the noise in the image and clarify the boundary between the target and the interference, so as to facilitate subsequent segmentation; The segmentation is to select the infrared image in the first band and segment the target and interference therein without distinction, and segment the infrared images in the remaining bands in the same manner as the first band; The improved watershed algorithm combines the results of distance transformation and grayscale difference transformation as an extended distance map, marks the segmented sub-image located at the center of the original image as the target, and marks the remaining sub-images as interference; The parallel calculation refers to: first, fixing the parameters of the trained multi-channel image fusion network, using the morphological segmentation algorithm to segment the target and interference in the image into sub-images, superimposing them into a batch input multi-channel image fusion network to calculate the classification probability tensor, matching the sub-image with each maximum probability category in the tensor according to the index, obtaining the category of each sub-image, and realizing the distinction between the target and the interference in a complex interference environment.

2. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The morphological opening operation is specifically: Among them: A is the infrared grayscale image to be processed, B is the selected specific structural element, Indicates the opening operation, represents the corrosion operation, Represents an expansion operation.

3. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The segmentation is specifically as follows: threshold segmentation is performed on the grayscale image after the opening operation, and a distance transformation operation is performed after the target or interference area is determined. In the image after threshold segmentation, 1 represents the target or interference area, and 0 represents the background, and a Euclidean distance transformation map Dis(i,j)=min{Distance[(i,j),(x,y)],(x,y)∈B} is generated, where: Dis is the Euclidean distance transformation map, B is the set of target areas for all points with a value of 1 in the binary image; the valley is found in the defined extended distance map, that is, the local minimum is used as a marker to input the watershed algorithm, and the algorithm outputs the coordinates of the segmented target or interference point, which is divided into N×N rectangular sub-images with this as the center. Each sub-image only contains the target or interference. According to the segmentation result of the band, the image of the remaining bands is segmented in the same way, and the segmented sub-images of each band are merged as a channel into a multi-channel sub-image.

4. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The improved watershed algorithm is specifically: Where: D is the extended distance map, Dis and Dif are the results of distance transformation and grayscale difference transformation respectively, Dif max and Dif min are the maximum and minimum values ​​of the grayscale difference transformation respectively.

5. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The ratio of the training set, the validation set and the test set is 2:2:1, wherein in the segmented sub-image of the same infrared image, if the number of interferences is greater than 5, the 5 interferences closest to the target are taken as samples in the data set.

6. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The recognition module inputs the single-channel feature map after multi-channel image fusion into the binary classification convolutional neural network to complete the classification of the image. When the network is trained on a multi-band infrared target recognition data set, the input image is firstly mean normalized.

7. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The multi-channel image fusion network in the recognition module learns the fusion mode of spectral features of different bands by itself through 1×1 convolution kernel, which is specifically: Where: f(x,y) represents the pixel value at a specific location, N is the number of bands of the acquired infrared image, α i is the parameter learned by the 1×1 convolution kernel in the information fusion module, that is, the weight of each channel in the fused feature map. The multi-channel image fusion network obtains the fused single-channel feature map without changing the size of the multi-channel image. Specifically: F:R N×H×W →R 1×H×W , where: F represents the fusion process, N is the number of bands of the acquired infrared image, and H and W are the height and width of the image.

8. The anti-interference target recognition method integrating multi-band spectral features according to claim 1 is characterized in that: The loss function of the training for the image binary classification convolutional neural network is: Where: α is a hyperparameter used to balance the imbalance between dataset interference and target ratio, The probability that the network predicts that the sample is a positive example, that is, the target, y is the sample label, if the sample is a target, it is 1, otherwise it is 0.

9. An anti-interference target recognition system integrating multi-band spectral features to implement any of the methods described in claims 1 to 8, characterized in that: include: Segmentation module, information fusion module and image binary classification convolutional neural network, among which: the segmentation module segments the target and interference in the collected data, and the segmented sub-image is used as the input of the information fusion module. The information fusion module fuses the spectral features of multiple bands of the same target or interference and inputs them into the image binary classification convolutional neural network. The system finally outputs the location of the target or interference and its corresponding category.

Images