An infrared small target method, device, equipment and storage medium
By combining spatial and temporal features in an infrared small target detection method, a spatiotemporal fusion feature map is generated, which solves the problem that existing methods rely solely on spatial features, resulting in limited performance improvement and achieving higher detection accuracy and generalization ability.
Patent Information
- Application Number
- CN202211437114.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-16
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2042-11-16
AI Technical Summary
Existing deep learning-based infrared small target detection methods mainly rely on salient features in the spatial domain, resulting in limited improvement in detection performance.
By combining the spatial and temporal features of the image to be processed, infrared small targets are detected through joint representation in the spatiotemporal dimensions, including spatial convolution and temporal convolution processing, and a spatiotemporal fusion feature map is generated for target localization.
It improves the performance of infrared small target detection, achieves a balance between detection rate and false alarm rate, and enhances detection accuracy and generalization ability.
Smart Images
Figure CN115690587B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of target detection, and particularly relates to an infrared small target detection method, device, equipment and storage medium. BACKGROUND
[0002] The purpose of infrared small target detection is to separate the target from the clutter background and accurately locate it, and it is widely used in civil and military fields, such as fire alarm systems, unmanned aerial vehicle vision systems and early warning systems. With the development of science and technology and the progress of society, great progress has been made in infrared small target detection.
[0003] In recent years, the infrared small target detection method based on deep learning has achieved good results due to its powerful feature learning ability. However, most of the existing infrared small target detection methods based on deep learning only use the spatial domain significant features of the target for detection, and such detection methods have certain problems in performance improvement. SUMMARY
[0004] Therefore, the present application aims to provide an infrared small target detection method, device, equipment and storage medium, which can combine the spatial domain features and time domain features of the to-be-processed image, detect based on the joint representation of the infrared small target in the space-time dimension, and thus improve the performance of target detection. The specific scheme is as follows:
[0005] In a first aspect, the present application provides an infrared small target detection method, comprising:
[0006] obtaining a plurality of original infrared images;
[0007] respectively performing spatial domain convolution processing on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images, and splicing the spatial domain feature maps of the original infrared images to obtain a spliced spatial domain feature map;
[0008] performing time domain convolution processing on the spliced spatial domain feature map to obtain a space-time fusion feature map corresponding to the plurality of original infrared images;
[0009] determining a current to-be-processed image from the plurality of original infrared images, and separating a space-time fusion feature map corresponding to the current to-be-processed image from the space-time fusion feature map;
[0010] performing positioning processing on an infrared small target in the current to-be-processed image based on the space-time fusion feature map corresponding to the current to-be-processed image to obtain a target positioning result.
[0011] Optionally, the spatial domain convolution processing on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images comprises:
[0012] performing a separate convolution operation on each of the original infrared images to obtain a preliminary spatial feature map of each of the original infrared images;
[0013] performing an Hourglass operation on the preliminary spatial feature map of each of the original infrared images to obtain a spatial feature map of each of the original infrared images; the Hourglass operation comprises a preset number of down-sampling processes.
[0014] Optionally, the splicing of the spatial feature maps of the original infrared images to obtain the spliced spatial feature map comprises:
[0015] splicing the spatial feature maps of the original infrared images in a preset dimension to obtain the spliced spatial feature map.
[0016] Optionally, the positioning of the infrared small target in the current to-be-processed image based on the spatio-temporal fusion feature map corresponding to the current to-be-processed image comprises:
[0017] fusing features in different channels in the spatio-temporal fusion feature map corresponding to the current to-be-processed image by using a preset convolution layer, and performing normalization after the fusion to obtain a segmentation result map corresponding to the current to-be-processed image;
[0018] positioning the infrared small target in the current to-be-processed image based on the segmentation result map.
[0019] Optionally, the positioning of the infrared small target in the current to-be-processed image based on the segmentation result map comprises:
[0020] determining a target binary map based on the segmentation result map;
[0021] determining a target centroid of an eight-neighbor connected domain corresponding to the target binary map by using a preset centroid determination rule, and determining a positioning result of the infrared small target in the current to-be-processed image based on the target centroid.
[0022] Optionally, the determining of the target binary map based on the segmentation result map comprises:
[0023] performing a target-oriented probability map separation operation on the segmentation result map to obtain a target channel map;
[0024] performing threshold segmentation processing on the target channel map to obtain the target binary map.
[0025] In a second aspect, the present application provides an infrared small target detection device, comprising:
[0026] An original image acquisition module is configured to acquire a plurality of frames of original infrared images;
[0027] A spatial domain convolution module is configured to perform spatial domain convolution processing on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images, and to splice the spatial domain feature maps of the original infrared images to obtain a spliced spatial domain feature map;
[0028] A time domain convolution module is configured to perform time domain convolution processing on the spliced spatial domain feature map to obtain a spatio-temporal fusion feature map corresponding to the plurality of frames of original infrared images;
[0029] A spatio-temporal fusion feature map separation module is configured to determine a current image to be processed from the plurality of frames of original infrared images, and separate a spatio-temporal fusion feature map corresponding to the current image to be processed from the spatio-temporal fusion feature map;
[0030] A target positioning module is configured to perform positioning processing on an infrared small target in the current image to be processed based on the spatio-temporal fusion feature map corresponding to the current image to be processed, to obtain a target positioning result.
[0031] Optionally, the target positioning module comprises:
[0032] A feature fusion sub-module is configured to perform fusion processing on features in different channels in the spatio-temporal fusion feature map corresponding to the current image to be processed by using a preset convolution layer;
[0033] A normalization processing sub-module is configured to perform normalization after the fusion processing to obtain a segmentation result map corresponding to the current image to be processed.
[0034] In a third aspect, the present application provides an electronic device, comprising:
[0035] A memory is configured to save a computer program;
[0036] A processor is configured to execute the computer program to implement the infrared small target detection method described above.
[0037] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, wherein the computer program is executed by a processor to implement the infrared small target detection method described above.
[0038] In the present application, a plurality of original infrared images are acquired; each of the original infrared images is subjected to spatial domain convolution processing to obtain a spatial domain feature map of each of the original infrared images, and the spatial domain feature maps of the original infrared images are spliced to obtain a spliced spatial domain feature map; the spliced spatial domain feature map is subjected to time domain convolution processing to obtain a spatio-temporal fusion feature map corresponding to the plurality of original infrared images; a current image to be processed is determined from the plurality of original infrared images, and a spatio-temporal fusion feature map corresponding to the current image to be processed is separated from the spatio-temporal fusion feature map; and the infrared small target in the current image to be processed is subjected to positioning processing based on the spatio-temporal fusion feature map corresponding to the current image to be processed to obtain a target positioning result. In this way, by processing and analyzing the spatio-temporal fusion feature map corresponding to the image to be processed, the infrared small target in the image to be processed is positioned, avoiding detection of only the spatial domain feature of the target. The present application combines the spatial domain feature and the time domain feature of the image to be processed and detects based on the joint representation of the infrared small target in the spatio-temporal dimension, thereby improving the performance of target detection. BRIEF DESCRIPTION OF DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.
[0040] Figure 1 A flow chart of an infrared small target detection method provided by the present application;
[0041] Figure 2 A flow chart of a specific infrared small target detection method provided by the present application;
[0042] Figure 3 An image comparison chart of an experimental result provided by the present application;
[0043] Figure 4 A structural schematic diagram of an infrared small target detection device provided by the present application;
[0044] Figure 5 A structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION
[0045] With reference to the drawings and in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.
[0046] In recent years, the infrared small target detection method based on deep learning has achieved good results due to its strong feature learning ability. However, most of the existing infrared small target detection methods based on deep learning only use the spatial domain feature of the target for detection, and such detection method has certain problems in performance improvement. Therefore, the present application combines the spatial domain feature and the time domain feature of the image to be processed, and detects based on the joint representation of the infrared small target in the space-time dimension, thereby improving the performance of target detection.
[0047] Referring to Figure 1 The embodiments of the present application disclose an infrared small target detection method, comprising:
[0048] Step S11, acquiring a plurality of frames of original infrared images.
[0049] In the present embodiment, a plurality of frames of original images containing infrared small targets are acquired. It can be understood that the plurality of frames of original infrared images can be acquired from historical infrared small target video sequences, and the number of original infrared image frames can be pre-set, for example, five frames of original infrared images are acquired.
[0050] Step S12, performing spatial domain convolution processing on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images, and splicing the spatial domain feature maps of the original infrared images to obtain a spliced spatial domain feature map.
[0051] In the present embodiment, the plurality of frames of original infrared images can be input into a pre-established spatial domain convolution module to perform spatial domain convolution processing on each of the original infrared images, thereby obtaining a spatial domain feature map of each of the original infrared images, and splicing the spatial domain feature maps of the original infrared images to obtain a spliced spatial domain feature map.
[0052] It should be noted that the spatial domain convolution processing is respectively performed on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images, and specifically, the spatial domain convolution processing can include: performing a separate convolution operation on each of the original infrared images to obtain a preliminary spatial domain feature map of each of the original infrared images; and performing an Hourglass operation on the preliminary spatial domain feature map of each of the original infrared images to obtain the spatial domain feature map of each of the original infrared images, wherein the Hourglass operation includes a preset number of down-sampling processes. It can be understood that the number of the separate convolution operation and the down-sampling process can be set by the user according to the user's own needs, for example, the separate convolution operation can be performed four times on each of the original infrared images to obtain the preliminary spatial domain feature map of each of the original infrared images, and then the Hourglass operation including four down-sampling processes is performed on the preliminary spatial domain feature map of each of the original infrared images.
[0053] It should be noted that the spatial domain feature maps of the original infrared images are spliced to obtain a spliced spatial domain feature map, and specifically, the spatial domain feature maps of the original infrared images can be spliced in a preset dimension to obtain the spliced spatial domain feature map. It can be understood that the dimension and the splicing method can be set by the user according to the user's own needs, for example, the spatial domain feature maps of the original infrared images can be spliced in series in a first dimension to obtain the spliced spatial domain feature map.
[0054] In step S13, the spliced spatial domain feature map is subjected to time domain convolution processing to obtain a spatio-temporal fusion feature map corresponding to the original infrared images.
[0055] In this embodiment, the spliced spatial domain feature map can be input into a pre-established time domain convolution module to perform time domain convolution processing on the spliced spatial domain feature map, so as to obtain a spatio-temporal fusion feature map corresponding to the original infrared images.
[0056] In step S14, a current image to be processed is determined from the original infrared images, and a spatio-temporal fusion feature map corresponding to the current image to be processed is separated from the spatio-temporal fusion feature map.
[0057] In one specific embodiment, the last one of the original infrared images can be directly taken as the current image to be processed, so as to subsequently locate the infrared small target in the last one of the image sequences input into the network. Of course, this embodiment does not exclude that a frame of image of interest can be selected from the original infrared images according to the actual application needs of the user as the current image to be processed.
[0058] Step S15, based on the spatio-temporal fusion feature map corresponding to the current to-be-processed image, the infrared small target in the current to-be-processed image is positioned and processed to obtain a target positioning result.
[0059] In this embodiment, a plurality of original infrared images are acquired; the spatial domain convolution processing is respectively performed on each of the original infrared images to obtain the spatial domain feature map of each of the original infrared images, and the spatial domain feature maps of the original infrared images are spliced to obtain a spliced spatial domain feature map; the spatio-temporal fusion feature map corresponding to the plurality of original infrared images is obtained by performing the time domain convolution processing on the spliced spatial domain feature map; the current to-be-processed image is determined from the plurality of original infrared images, and the spatio-temporal fusion feature map corresponding to the current to-be-processed image is separated from the spatio-temporal fusion feature map; the infrared small target in the current to-be-processed image is positioned and processed based on the spatio-temporal fusion feature map corresponding to the current to-be-processed image to obtain a target positioning result. In this way, by processing and analyzing the spatio-temporal fusion feature map corresponding to the to-be-processed image, the infrared small target in the to-be-processed image is positioned, which avoids detecting only the spatial domain feature of the target, and the performance of target detection is improved by combining the spatial domain feature and the time domain feature of the to-be-processed image and detecting based on the joint representation of the infrared small target in the spatio-temporal dimension.
[0060] Referring to Figure 2 As shown in the figure, the embodiment of the present application discloses a specific infrared small target detection method, which comprises:
[0061] Step S21, a plurality of original infrared images are acquired.
[0062] Step S22, the spatial domain convolution processing is respectively performed on each of the original infrared images to obtain the spatial domain feature map of each of the original infrared images, and the spatial domain feature maps of the original infrared images are spliced to obtain a spliced spatial domain feature map.
[0063] Step S23, the spatio-temporal fusion feature map corresponding to the plurality of original infrared images is obtained by performing the time domain convolution processing on the spliced spatial domain feature map.
[0064] Step S24, the current to-be-processed image is determined from the plurality of original infrared images, and the spatio-temporal fusion feature map corresponding to the current to-be-processed image is separated from the spatio-temporal fusion feature map.
[0065] Step S25, the features in different channels in the spatio-temporal fusion feature map corresponding to the current to-be-processed image are fused by using a preset convolution layer, and normalization is performed after the fusion processing to obtain a segmentation result map corresponding to the current to-be-processed image.
[0066] In this embodiment, the spatio-temporal fusion feature map corresponding to the current image to be processed can be input into a segmentation module comprising two two-dimensional convolution layers, and the features in different channels of the spatio-temporal fusion feature map corresponding to the current image to be processed are fused in the segmentation module to obtain a score channel representing a prediction target and a score channel representing a prediction background, respectively. Then, the score values of the two channels are normalized by using a Soffmax function to obtain a segmentation result map corresponding to the current image to be processed, wherein the segmentation result map comprises a probability map of the target and the background.
[0067] In step S26, a target binary image is determined based on the segmentation result map.
[0068] In this embodiment, it should be noted that the determination of the target binary image based on the segmentation result map can specifically include: performing a probability map separation operation on the segmentation result map for the target to obtain a target channel map; and performing threshold segmentation processing on the target channel map to obtain the target binary image. That is, the probability map classified as the target is separated from the segmentation result map to obtain the target channel map, and then the target channel map is subjected to threshold segmentation processing to obtain the target binary image, wherein the pixels with a value of 1 in the target binary image are classified as the foreground, and the pixels with a value of 0 are classified as the background.
[0069] In step S27, a target centroid of an eight-neighbor connected domain corresponding to the target binary image is determined by using a preset centroid determination rule, and a positioning result of the infrared small target in the current image to be processed is determined based on the target centroid.
[0070] In this embodiment, an eight-neighbor connected domain corresponding to the target binary image is obtained, and a target centroid of the eight-neighbor connected domain is determined by using a preset centroid determination rule, so as to determine a positioning result of the infrared small target in the current image to be processed based on the target centroid. In a specific implementation, the preset centroid determination rule can include a centroid determination rule based on an Image Moments algorithm.
[0071] The specific processes of steps S21 to S24 can refer to the corresponding contents disclosed in the foregoing embodiments, and will not be described here again.
[0072] In this embodiment, a plurality of original infrared images are acquired; each of the original infrared images is subjected to spatial domain convolution processing to obtain a spatial domain feature map of each of the original infrared images, and the spatial domain feature maps of the original infrared images are spliced to obtain a spliced spatial domain feature map; the spliced spatial domain feature map is subjected to time domain convolution processing to obtain a spatio-temporal fusion feature map corresponding to the plurality of original infrared images; a current image to be processed is determined from the plurality of original infrared images, and a spatio-temporal fusion feature map corresponding to the current image to be processed is separated from the spatio-temporal fusion feature map; features in different channels in the spatio-temporal fusion feature map corresponding to the current image to be processed are fused by using a preset convolution layer, and normalization is performed after the fusion to obtain a segmentation result map corresponding to the current image to be processed; a target binary map is determined based on the segmentation result map; a target centroid of an eight-neighbor connected domain corresponding to the target binary map is determined by using a preset centroid determination rule, and a positioning result of an infrared small target in the current image to be processed is determined based on the target centroid. Through the above scheme, the features in the channels in the spatio-temporal fusion feature map corresponding to the image to be processed are fused and normalized to obtain a segmentation result map, a target binary map is determined based on the segmentation result map, and a positioning result of an infrared small target is determined by using the target binary map. In this way, the target positioning result of the infrared small target is determined by calculating the target centroid of the eight-neighbor connected domain of the target binary map, the accuracy of target detection is improved, and thus the performance of target detection is improved.
[0073] The specific embodiments of the present application will be described below in combination with an experiment performed for the present application.
[0074] Experiment preparation: 32 sequences of different scenes are used in this experiment, and target labels are made by using point annotation. According to the signal-to-clutter ratio, 22 real sequences are used as a training set, and 10 sequences are used as a test set. In the test set, 6 sequences are real scenes, which have similar data distribution as the training set, and the other 4 sequences are scenes in which simulated targets are added to real backgrounds. Among them, the four simulated sequences with different data distributions can be selected to verify the generalization ability of the deep learning-based method in different data distribution sequences. Most of the infrared small targets in the 10 sequences in the test set are weak, less than 5x5 pixels, and the background is complex, so the performance of the algorithm in complex situations can be more simply and clearly evaluated. In addition, the gray scale of the real sequences in the experiment is 4096, and the gray scale of the simulated sequences is 256.
[0075] The experiment initializes the weights of the convolution layer using the Xavier method and initializes the bias values using the random initialization method with uniform distribution. In the experiment, the input has long time domain information of 5 discontinuous frames, each frame has a size of 512*512, the batch size is set to 2, the initial learning rate is set to 0.001, and all training is stopped after 10 epochs. The evaluation parameters used in the experiment include detection rate and false alarm rate, where the detection rate represents the ratio of the number of correctly detected targets to the total number of targets, and the false alarm rate represents the ratio of the number of false alarm pixels to the total number of pixels in the image.
[0076] The experimental process is as follows: first, input five frames of original infrared images into the target detection network, and the data dimension of the original infrared image is 1*H(high)*W(width). The five frames of original infrared images are input into the pre-established spatial convolution module to perform spatial convolution processing on each frame of the original infrared image respectively, thereby obtaining the corresponding five frames of spatial feature maps, and the five frames of spatial feature maps are spliced to obtain the spliced spatial feature map, wherein the spatial convolution module has four convolution layers, the sizes of the above convolution layers are 1*3, 3*1, 1*3 and 3*1 respectively, the activation function of each convolution layer is ReLU(Rectified Linear Unit, linear rectification unit) function, and the data dimension of each frame of original infrared image after each convolution is 16*H*W, 16*H*W, 32*H*W and 32*H*W respectively; and the pre-set sub-module is used to perform four times of down-sampling on each frame of original infrared image after convolution, and the first dimension of the image data is kept unchanged during the down-sampling process, and is always kept as 32. Finally, five frames of spatial feature maps corresponding to the five frames of original infrared images are obtained, and the data dimension of the spatial feature map is 32*H*W.
[0077] The five frames of spatial feature maps are concatenated in the first dimension to obtain the spliced spatial feature map of the five frames of images, and the dimension of the spliced spatial feature map is 5*32*H*W. The spliced spatial feature map is input into the pre-established temporal convolution module, the temporal convolution module has two temporal convolution layers (with a size of 5*1*1), and the activation function of each convolution layer is ReLU function, thereby obtaining a spatio-temporal fusion feature map corresponding to the five frames of original infrared images, and the data dimension of the spatio-temporal fusion feature map is 5*32*H*W.
[0078] A current image to be processed is determined from the five original infrared images, and a spatiotemporal fusion feature map corresponding to the current image to be processed is separated from the spatiotemporal fusion feature map, with a dimension of 32*H*W. The spatiotemporal fusion feature map corresponding to the current image to be processed is input into a pre-established segmentation module. The features in different channels in the spatiotemporal fusion feature map corresponding to the current image to be processed are fused by using a preset convolutional layer in the segmentation module, and then normalized by using a Softmax layer to obtain a segmentation result map (with a size of 2*H*W) corresponding to the current image to be processed. The preset convolutional layer is two two-dimensional convolutional layers (with a size of 1*1) for fusing features in different channels, and the activation function of each convolutional layer is a ReLU function. The dimension of data after each convolutional layer is 64*H*W, 2*H*W.
[0079] A probability map classified as a target is separated from the segmentation result map to obtain a target channel map, and then threshold segmentation processing is performed on the target channel map to obtain a target binary map with a size of 1*H*W. In the target binary map, pixels with a value of 1 are classified as targets, and pixels with a value of 0 are classified as backgrounds. An eight-neighbor connected domain is obtained for the target binary map, and a centroid of the eight-neighbor connected domain is calculated. The centroid is a target positioning result of an infrared small target.
[0080] Experimental results: In order to verify the effectiveness of the infrared small target detection method disclosed in the present application, the method in the present application can be compared with several existing advanced detection methods, wherein the method in the present application can be defined as STDFnet (i.e. Spatio-Temporal Decoupled Fusion Network). The existing advanced detection methods include single-frame detection algorithms, Top-hat (top-hat transformation algorithm), IPI (infrared patch-image model), PLCM (part local contrast measure), RIPT (reweighted infrared patch-tensor), and multi-frame detection algorithms such as NMoG algorithm (non-i.i.d. mixture of Gaussians), MSLSTIPT algorithm (Multiple Subspace Learning and Spatial-Temporal Infrared Patch-Tensor Model), and several deep learning-based methods including ACM algorithm (Asymmetric Contextual Modulation), ALCNet algorithm (Attentional Local Contrast Networks), and DNANet algorithm (Dense Nested Attention Network). Considering that the present experiment uses point annotation, the HPM loss (Hard Point Mining loss) is used to train other deep learning-based methods for comparison. Considering that the data distribution and target intensity are different between the real sequence and the simulated sequence, two groups of results are given respectively. The experimental results are shown in Tables I and II. Figure 3
[0081] Table I Real sequence results
[0082]
[0083]
[0084] Table II Simulated sequence results
[0085]
[0086] The results of detection on real sequences and simulation sequences are shown in Table 1 and Table 2. It can be seen that the infrared small target detection method disclosed by the application realizes the balance of detection rate and false alarm rate, and can maintain a high detection rate while reducing the false alarm rate.
[0087] For the present application, more spatial information and temporal information are extracted by using a spatio-temporal fusion network, wherein the spatial information is used to obtain more responses of suspicious targets, and the temporal information is used to maximize the suppression of false alarms. Therefore, good performance can be achieved in terms of detection rate and false alarm rate. As shown in Table 1, the detection rate of the present application on real sequences can be maintained at 99.59%, and the false alarm rate is only 4.9x10 -6 .
[0088] Compared with the other three deep learning-based methods, the performance of these methods is better than that of the traditional method in real sequences, but the performance of the other three deep learning-based methods is worse in simulation sequences with different data distribution from the training set. In the case of different spatial distribution of real sequences and simulation sequences, since the time distribution is similar, the intensity of both peaks appears when the target appears. The present application can capture the changes in the time dimension through the time convolution module, while the other three deep learning-based methods can only perceive spatial information. Therefore, the infrared small target detection method disclosed by the application improves the generalization ability of target detection.
[0089] Referring to Figure 4 , the application discloses an infrared small target detection device, comprising:
[0090] An original image acquisition module 11 is configured to acquire a plurality of frames of original infrared images;
[0091] A spatial domain convolution module 12 is configured to perform spatial domain convolution processing on each of the original infrared images to obtain a spatial domain feature map of each of the original infrared images, and to splice the spatial domain feature maps of the original infrared images to obtain a spliced spatial domain feature map;
[0092] A time domain convolution module 13 is configured to perform time domain convolution processing on the spliced spatial domain feature map to obtain a spatio-temporal fusion feature map corresponding to the plurality of frames of original infrared images;
[0093] A spatio-temporal fusion feature map separation module 14 is configured to determine a current image to be processed from the plurality of frames of original infrared images, and to separate a spatio-temporal fusion feature map corresponding to the current image to be processed from the spatio-temporal fusion feature map;
[0094] A target positioning module 15 is configured to perform positioning processing on an infrared small target in the current image to be processed based on the spatio-temporal fusion feature map corresponding to the current image to be processed, to obtain a target positioning result.
[0095] In this embodiment, a plurality of original infrared images are acquired; each of the original infrared images is subjected to spatial convolution processing to obtain a spatial feature map of each of the original infrared images, and the spatial feature maps of the original infrared images are spliced to obtain a spliced spatial feature map; the spliced spatial feature map is subjected to time domain convolution processing to obtain a spatio-temporal fusion feature map corresponding to the plurality of original infrared images; a current image to be processed is determined from the plurality of original infrared images, and a spatio-temporal fusion feature map corresponding to the current image to be processed is separated from the spatio-temporal fusion feature map; the infrared small target in the current image to be processed is subjected to positioning processing based on the spatio-temporal fusion feature map corresponding to the current image to be processed to obtain a target positioning result. In this way, by processing and analyzing the spatio-temporal fusion feature map corresponding to the image to be processed, the infrared small target in the image to be processed is located, avoiding detection of only the spatial features of the target. The present application combines the spatial features and the time domain features of the image to be processed and detects based on the joint representation of the infrared small target in the space-time dimension, thereby improving the performance of target detection.
[0096] In some specific embodiments, the spatial convolution module 12 can specifically include:
[0097] A separation convolution unit is configured to separately perform separation convolution operation on each of the original infrared images to obtain a preliminary spatial feature map of each of the original infrared images.
[0098] An Hourglass unit is configured to perform Hourglass operation on the preliminary spatial feature map of each of the original infrared images to obtain a spatial feature map of each of the original infrared images. The Hourglass operation includes a preset number of times of down-sampling processing.
[0099] In some specific embodiments, the spatial convolution module 12 can specifically include:
[0100] An image splicing unit is configured to splice the spatial feature maps of the original infrared images in a preset dimension to obtain a spliced spatial feature map.
[0101] In some specific embodiments, the target positioning module 15 can specifically include:
[0102] A fusion normalization unit is configured to perform fusion processing on the features in different channels in the spatio-temporal fusion feature map corresponding to the current image to be processed by using a preset convolution layer, and perform normalization after the fusion processing to obtain a segmentation result map corresponding to the current image to be processed.
[0103] In some specific embodiments, the infrared small target detection apparatus can specifically include:
[0104] A binary graph determination unit is configured to determine a target binary graph based on the segmentation result graph.
[0105] A centroid determination unit is configured to determine a target centroid of an eight-neighbor connected domain corresponding to the target binary graph by using a preset centroid determination rule, and determine a positioning result of an infrared small target in the current image to be processed based on the target centroid.
[0106] In some specific embodiments, the infrared small target detection apparatus can specifically include:
[0107] A channel graph determination unit is configured to perform a target probability graph separation operation on the segmentation result graph to obtain a target channel graph.
[0108] A threshold segmentation unit is configured to perform threshold segmentation processing on the target channel graph to obtain the target binary graph.
[0109] In some specific embodiments, the target positioning module 15 can specifically include:
[0110] A feature fusion sub-module is configured to fuse features in different channels in a spatio-temporal fusion feature graph corresponding to the current image to be processed by using a preset convolution layer.
[0111] A normalization processing sub-module is configured to perform normalization after the fusion processing to obtain a segmentation result graph corresponding to the current image to be processed.
[0112] Further, the embodiment of the present application also discloses an electronic device, Figure 5 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.
[0113] Figure 5 A structural schematic diagram of an electronic device 20 provided by the embodiment of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21, so as to realize the related steps in the infrared small target detection method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.
[0114] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.
[0115] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.
[0116] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the infrared small target detection method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.
[0117] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing disclosed infrared small target detection method. For the specific steps of the method, please refer to the corresponding contents disclosed in the foregoing embodiments, which will not be described here.
[0118] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the related contents.
[0119] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in a general manner in the foregoing description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0120] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The
[0121] Finally, it should be noted that, in the description of the application, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.
[0122] The above provides a detailed description of the technical solutions of the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.
Claims
1. A method for detecting small infrared targets, characterized in that, include: Acquire several frames of raw infrared images; Each frame of the original infrared image is subjected to spatial convolution processing to obtain a spatial feature map of each frame of the original infrared image, and the spatial feature maps of each frame of the original infrared image are stitched together to obtain a stitched spatial feature map. The step of performing spatial convolution processing on each frame of the original infrared image to obtain a spatial feature map of each frame includes: performing a split convolution operation on each frame of the original infrared image to obtain a preliminary spatial feature map of each frame of the original infrared image; performing an Hourglass operation on the preliminary spatial feature map of each frame of the original infrared image to obtain a spatial feature map of each frame of the original infrared image; the Hourglass operation includes a preset number of downsampling processes. The spliced spatial feature map is subjected to temporal convolution processing to obtain a spatiotemporal fusion feature map corresponding to the several frames of original infrared images; The current image to be processed is determined from the aforementioned frames of original infrared images, and the spatiotemporal fusion feature map corresponding to the current image to be processed is separated from the spatiotemporal fusion feature map; Based on the spatiotemporal fusion feature map corresponding to the current image to be processed, the infrared small target in the current image to be processed is located to obtain the target location result.
2. The infrared small target detection method according to claim 1, characterized in that, The step of stitching together the spatial feature maps of each frame of the original infrared image to obtain the stitched spatial feature map includes: The spatial feature maps of each frame of the original infrared image are stitched together along a preset dimension to obtain the stitched spatial feature map.
3. The infrared small target detection method according to any one of claims 1 to 2, characterized in that, The process of locating small infrared targets in the current image to be processed based on the spatiotemporal fusion feature map corresponding to the current image to be processed includes: The features in different channels of the spatiotemporal fusion feature map corresponding to the current image to be processed are fused using a preset convolutional layer, and normalized after fusion processing to obtain a segmentation result map corresponding to the current image to be processed. Based on the segmentation result image, the infrared small targets in the current image to be processed are located.
4. The infrared small target detection method according to claim 3, characterized in that, The process of locating small infrared targets in the current image to be processed based on the segmentation result image includes: Determine the target binary image based on the segmentation result image; The target centroid of the eight-neighbor connected region corresponding to the target binary image is determined using a preset centroid determination rule, and the positioning result of the infrared small target in the current image to be processed is determined based on the target centroid.
5. The infrared small target detection method according to claim 4, characterized in that, The step of determining the target binary image based on the segmentation result image includes: Perform a target-specific probabilistic map separation operation on the segmentation result map to obtain the target channel map; The target channel image is subjected to threshold segmentation to obtain the target binary image.
6. An infrared small target detection device, characterized in that, include: The raw image acquisition module is used to acquire several frames of raw infrared images; The spatial convolution module is used to perform spatial convolution processing on each frame of the original infrared image to obtain a spatial feature map of each frame of the original infrared image, and to stitch the spatial feature maps of each frame of the original infrared image to obtain a stitched spatial feature map; wherein, the step of performing spatial convolution processing on each frame of the original infrared image to obtain a spatial feature map of each frame of the original infrared image includes: performing a split convolution operation on each frame of the original infrared image to obtain a preliminary spatial feature map of each frame of the original infrared image; performing an Hourglass operation on the preliminary spatial feature map of each frame of the original infrared image to obtain a spatial feature map of each frame of the original infrared image; the Hourglass operation includes a preset number of downsampling processes. The temporal convolution module is used to perform temporal convolution processing on the stitched spatial feature map to obtain a spatiotemporal fusion feature map corresponding to the several frames of original infrared images; The spatiotemporal fusion feature map separation module is used to determine the current image to be processed from the several frames of original infrared images, and to separate the spatiotemporal fusion feature map corresponding to the current image to be processed from the spatiotemporal fusion feature map; The target localization module is used to perform localization processing on small infrared targets in the current image to be processed based on the spatiotemporal fusion feature map corresponding to the current image to be processed, so as to obtain the target localization result.
7. The infrared small target detection device according to claim 6, characterized in that, The target localization module includes: The feature fusion submodule is used to fuse features in different channels of the spatiotemporal fusion feature map corresponding to the current image to be processed using a preset convolutional layer. The normalization processing submodule is used to perform normalization after fusion processing to obtain a segmentation result map corresponding to the current image to be processed.
8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the infrared small target detection method as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the infrared small target detection method as described in any one of claims 1 to 5.