A Gunfire Detection Method Based on the Combination of Traditional Image Processing and Deep Learning

Abstract

This invention belongs to the field of target detection and recognition technology, and discloses a muzzle detection method based on a combination of traditional image processing and deep learning. The method includes the following steps: S1: Perform muzzle detection based on traditional image processing, using the energy distribution and temporal features of the muzzle image to initially select suspected targets; S2: Perform muzzle detection based on deep learning: use deep learning for modeling, and leverage the nonlinear mapping characteristics of neural networks to determine morphological differences between images and identify real muzzle targets. This invention can extract all muzzle targets, but it also results in a large number of false alarms. By using deep learning technology for further filtering, most false alarms can be eliminated while retaining the actual muzzle targets. Since deep learning only processes a small number of targets filtered out by traditional algorithms, the overall time cost is also relatively low.

Description

Technical Field

[0001] This invention belongs to the field of target detection and recognition technology, and relates to a flash detection method, specifically a flash detection method based on a combination of traditional image processing and deep learning. Background Technology

[0002] A sniper detection system based on muzzle flash detection uses a high-frame-rate infrared detector and a high-speed image processing platform to detect and identify the infrared radiation energy emitted by an enemy sniper's muzzle flash at the moment of firing, thus detecting the sniper. The muzzle flash detection method directly determines the system's performance.

[0003] Because the flash duration is very short (a few milliseconds), the system must use staring detection. Considering constraints such as detection field of view, detection distance, and processing speed, the flash occupies very few pixels in the infrared detector; therefore, the flash is a transient, weak infrared target.

[0004] The task of gunfire detection is to quickly and accurately identify gunfire targets from complex backgrounds with a large field of view.

[0005] Traditional infrared weak target detection methods can be divided into morphology-based detection methods and multi-directional gradient-based detection methods.

[0006] Morphological detection methods mainly perform Top-hat operations on images to eliminate false alarms and enhance weak targets. However, they are difficult to distinguish between strong clutter, fixed noise and real targets by using a single dilation and erosion structuring element, resulting in poor detection performance.

[0007] Multi-directional gradient-based detection algorithms select multiple feature points along the neighborhood of the target point, calculate their gradient directions, and extract the target region. However, relying solely on gradient features to extract candidate regions can easily introduce many false targets, reducing the algorithm's detection accuracy.

[0008] The rapid development of deep learning technology has provided new solutions to many problems that cannot be solved from the perspective of traditional image processing. It learns the inherent patterns and hierarchical representations of sample data, and the information gained during the learning process greatly aids in the interpretation of the data. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans. Deep learning technology is currently widely used in image denoising, image classification, and image enhancement.

[0009] Traditional methods are advantageous due to their ease of optimization and low time overhead. However, to ensure that gunfire can be extracted in all scenarios, the extraction results inevitably include a large number of false alarms. Deep learning techniques can achieve good detection results provided there is sufficient data, but the time overhead is insufficient for detecting transient weak targets directly in complex backgrounds with a large field of view. Summary of the Invention

[0010] (I) Purpose of the Invention

[0011] The purpose of this invention is to propose a two-stage flash detection method that combines traditional image processing techniques with deep learning techniques, thereby combining the advantages of both to obtain a better detection algorithm.

[0012] (II) Technical Solution

[0013] To address the aforementioned technical problems, this invention provides a flash detection method based on a combination of traditional image processing and deep learning, with the specific steps as follows:

[0014] S1: Gunfire detection based on traditional image processing

[0015] The energy distribution of the gunfire in infrared images generally exhibits a Gaussian-like distribution with high energy distribution in the mid-to-high frequency range and low energy distribution in the low frequency range. Moreover, in high-speed cameras, the gunfire lasts only a few frames from appearance to disappearance. By using the energy distribution and temporal characteristics of the gunfire images as constraints, potential targets can be preliminarily selected.

[0016] Furthermore, flash detection based on traditional image processing includes:

[0017] S11: Differential extraction of all suspected targets that change instantaneously;

[0018] S12: Remove large-area false targets by constraining the area of ​​suspected targets;

[0019] S13: By performing Gaussian fitting on the energy distribution of suspected targets, false targets that do not conform to the Gaussian distribution are removed;

[0020] S14: Remove false targets with excessively long durations using temporal constraints;

[0021] However, in real-world scenarios, factors such as water surface reflection and distant heat sources can generate false alarms that meet the above constraints, and these false alarms cannot be eliminated by traditional image processing alone.

[0022] S2: Deep Learning-Based Gunfire Detection

[0023] By observing the data detected by traditional image processing, it was found that there are morphological differences between real gunfire and false targets. For example, real targets are closer to circular in shape, while false targets are mostly irregular in shape. Therefore, this problem is defined as a common image binary classification problem. Deep learning technology is used to model this problem, and the nonlinear mapping characteristics of neural networks are used to distinguish the morphological differences between images.

[0024] From the perspective of image data, considering that these images containing suspected targets are composed of two-dimensional matrices and have significant regional morphological information, we use convolutional neural networks (CNNs), which are commonly used in the field of computer vision, to build a deep learning model. This network has the characteristics of translation invariance and rotation invariance. These two properties have obvious effects in image processing and are very suitable for image classification scenarios.

[0025] (III) Beneficial Effects

[0026] The above-mentioned technical solution provides a flash detection method based on the combination of traditional image processing and deep learning, which can extract all flash targets, but is accompanied by a large number of false alarms. By filtering again through deep learning technology, most false alarms can be eliminated while ensuring that flash targets are retained. Since deep learning only processes a small number of targets filtered out by traditional algorithms, the overall time cost is also relatively small. Attached Figure Description

[0027] Figure 1 This is the network structure of a deep learning model in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0029] like Figure 1 As shown, the flash detection method based on the combination of traditional image processing and deep learning in this embodiment includes the following steps:

[0030] S1: Gunfire detection based on traditional image processing

[0031] To extract targets from all scenarios and minimize false alarms, this stage fully utilizes various feature information from gunfire to create multiple constraints. The process is as follows:

[0032] S11: Differential extraction of all suspected regions

[0033] Gunfire is a transient target, and its energy changes over time. Therefore, by subtracting adjacent frames, all transient targets with this characteristic can be selected. Since the background is not completely static, the targets obtained after subtraction contain a large number of false alarms. In other words, subtraction can obtain all possible points in the frame that change instantaneously.

[0034] S12: Area Constraint

[0035] Extensive data analysis revealed that the number of pixels occupied by the gun flash in infrared images ranges from 2 to 50. This characteristic can be used to remove a large number of large-area false alarms. By applying area constraints, significant computational costs can be reduced, saving resources and accelerating detection time.

[0036] S13: Gaussian Fitting

[0037] Gun flash data exhibits a Gaussian distribution within the region, specifically a two-dimensional Gaussian distribution with high values ​​in the center and low values ​​around the perimeter. This characteristic can be used to further eliminate false alarms after area constraints are applied.

[0038] Given a suspected target region (two-dimensional matrix), expand the matrix into a square matrix in the original image with the maximum value as the center point (the center point of the Gaussian distribution has the largest value). If it exceeds the boundary of the original image, replace it with 0 to obtain the suspected target square matrix A.

[0039] Generate a Gaussian matrix of the same size as A using a normal Gaussian function, and then normalize it to obtain the Gaussian kernel K.

[0040] The Gaussian fitted value X is obtained by taking the modulus of the square matrix resulting from the dot product of A and K. The closer the distribution of A is to a Gaussian shape, the larger the value of X.

[0041] However, X cannot be used directly as a criterion for judgment because X is greatly affected by the area. The larger the area, the larger the value of X may be, which is obviously not what we want.

[0042] Therefore, we use a mean matrix B of A to eliminate the influence of area. The size of B is the same as that of A. The elements of B corresponding to the non-zero elements of A are set as the mean of A. We obtain the Gaussian fitting value Y of B by taking the modulus of the matrix after multiplying B by K. We divide X by Y to obtain the Gaussian fitting coefficient r of the suspected target. The larger r is, the better the matrix distribution fits the Gaussian distribution.

[0043] The formula for calculating the Gaussian fitting coefficients is:

[0044]

[0045]

[0046] Where A represents the target matrix, N represents the number of non-zero elements in matrix A, and K represents the normal Gaussian kernel.

[0047] Through experimental verification, the minimum threshold for r was ultimately set to 1.5. Areas below this threshold are considered false alarms, while areas above this threshold will be further evaluated. Gaussian fitting can remove heat sources such as faces and some reflections from the image.

[0048] S14: Timing Constraints

[0049] In high-speed infrared cameras, the entire process from the appearance to the disappearance of the gun flash does not exceed 3 frames. Therefore, for the suspected targets remaining after area constraints and Gaussian determination, temporal constraints are applied again. There are three key points in temporal determination: first, how to determine the appearance of the target; second, how to track the target; and finally, how to determine the disappearance of the target.

[0050] If a target appears, the pixel values ​​of the suspected area in the current frame are compared with those in the previous frame. If all values ​​increase, the target is determined to have appeared.

[0051] Since performing inter-frame differencing after a target appears will result in the loss of the target region, tracking the target is extremely difficult. Consider two scenarios: First, if the center point of the current target region is misaligned by no more than 3 pixels from the center point of the target region in the previous frame, it indicates that the target is not moving, but its energy is changing over time. Therefore, the suspected region is updated to the point of maximum energy to facilitate target disappearance determination. Second, if the center point of the current target region is misaligned by more than 3 but less than 20 pixels from the center point of the target region in the previous frame, it indicates that the target position has changed. In this case, the region obtained by differencing is not the complete target. The regions with center misalignment between 3 and 20 pixels are then combined to obtain the complete suspected region.

[0052] To determine if a target has disappeared, the suspected area is compared with the corresponding area in the current frame. If all values ​​decrease and the difference between the maximum value in the current frame (the frame in which the target disappeared) and the frame before the target appeared does not exceed 500, then the target is determined to have disappeared.

[0053] If the entire process from the appearance to the disappearance of the target does not exceed 3 frames, then the timing constraint is satisfied and it can be determined as gunfire.

[0054] S2: Gunfire Detection Based on Deep Learning

[0055] S21: Dataset Construction

[0056] Based on the suspected targets detected by traditional methods, a dataset is constructed for training a deep binary classification model.

[0057] Since all suspected targets are smaller than 8*8 pixels, all suspected targets are expanded into 8*8 square areas in the original image with the center. If they exceed the size of the original image, they are replaced with 0.

[0058] The dataset is split into training, validation, and test sets in an 8:1:1 ratio. The main reasons for using smaller regions instead of the original image for classification are as follows:

[0059] a) If the original image search area is too large, the deep learning model may not be able to accurately focus on the target area, leading to detection failure;

[0060] b) When using small regions, deep models only need to learn the features of suspected targets, which allows them to focus more on the high-level features of the targets and avoid being disturbed by the background.

[0061] c) Classifying small regions does not require a deep network structure, saving detection time.

[0062] Before feeding the image data into the model for training, the gunfire dataset first undergoes data preprocessing operations, such as translation, rotation, and normalization. The main purpose is to make the trained model more robust and have better generalization performance, so as to better cope with the uncertainties introduced by changes in the detection scene.

[0063] S22: Deep Learning Training

[0064] The convolutional neural network model uses a combination of 5 convolutional modules and 3 fully connected modules. Each convolutional module consists of a convolutional layer, a layer of normalization (BN), and a ReLU activation function. Each fully connected module consists of a fully connected layer, a ReLU activation function, and a Dropout regularization operation. The last layer of the network is a sigmoid activation function, which outputs a probability value for image classification. If the value exceeds 0.5, the image is considered a real object; otherwise, it is considered a false object. The overall network structure is as follows: Figure 1 As shown.

[0065] The processed dataset is then fed into a convolutional neural network model for training. Considering that this is a binary classification problem, the cross-entropy loss function, which is most commonly used in classification problems, is chosen to penalize the model.

[0066] The trained model was put into the test set for experimentation, and a classification accuracy of 99% was obtained.

[0067] Therefore, incorporating this model into a flash detection system, as a "precision detector," can accurately distinguish the morphological differences between real and false targets.

[0068] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A flash detection method based on a combination of traditional image processing and deep learning, characterized in that, Includes the following steps: S1: Gun flash detection is performed based on traditional image processing, and suspected targets are initially selected by constraining the energy distribution and temporal features of the gun flash image. Step S1 includes the following steps: S11: Differential extraction of all suspected targets that change instantaneously; S12: Remove large-area false targets by constraining the area of ​​suspected targets; S13: By performing Gaussian fitting on the energy distribution of suspected targets, false targets that do not conform to the Gaussian distribution are removed; S14: Remove false targets with excessively long durations using temporal constraints; In step S13, given a suspected target region, i.e. a two-dimensional matrix, with the maximum value as the center point, i.e. the center point of the Gaussian distribution with the maximum value, the matrix is ​​expanded into a square matrix in the original image. If it exceeds the boundary of the original image, it is replaced with 0, thus obtaining the suspected target square matrix A. Generate a Gaussian matrix of the same size as A using a normal Gaussian function, and then normalize it to obtain the Gaussian kernel K; The Gaussian fitted value X is obtained by taking the modulus of the square matrix after multiplying A and K by the dot product of A and K. Find the mean matrix B of A. The size of B is the same as that of A. The elements of B corresponding to the non-zero elements of A are set as the mean of A. The Gaussian fitting value Y of B is obtained by taking the modulus of the matrix after multiplying B by K. The Gaussian fitting coefficient r of the suspected target is obtained by dividing X by Y. The larger r is, the better the matrix distribution fits the Gaussian distribution. The formula for calculating the Gaussian fitting coefficients is: Where N represents the number of non-zero elements in matrix A, and K represents the normal Gaussian kernel; S2: Gun flash detection based on deep learning; Deep learning is used for modeling, and the nonlinear mapping characteristics of neural networks are used to distinguish morphological differences between images and identify real gunfire targets.

2. The flash detection method based on the combination of traditional image processing and deep learning as described in claim 1, characterized in that, In step S11, the gun flash is a transient target whose energy changes over time. By subtracting adjacent frames, all transient targets with this characteristic are selected.

3. The flash detection method based on the combination of traditional image processing and deep learning as described in claim 2, characterized in that, In step S12, the number of pixels occupied by the gun flash in the infrared image is between 2 and 50. This feature is used to remove large-area false alarms.

4. The flash detection method based on the combination of traditional image processing and deep learning as described in claim 3, characterized in that, In step S13, the minimum threshold of r is set to 1.

5.

5. The flash detection method based on the combination of traditional image processing and deep learning as described in claim 4, characterized in that, In step S14, if a target appears, the pixel values ​​of the suspected area in the current frame and the previous frame of the area are compared. If all values ​​increase, the target is determined to have appeared. To determine if a target has disappeared, the suspected area is compared with the corresponding area in the current frame. If all values ​​decrease and the difference between the maximum value in the current frame and the maximum value in the frame before the target appeared does not exceed 500, then the target is determined to have disappeared. If the entire process from the appearance to the disappearance of the target does not exceed 3 frames, then the timing constraint is satisfied and it can be determined as gunfire.

6. The flash detection method based on the combination of traditional image processing and deep learning as described in claim 5, characterized in that, In step S14, if the center point of the current target area is misaligned with the center point of the target area in the previous frame by no more than 3 pixels, it indicates that the target has not moved; if the center point of the current target area is misaligned with the center point of the target area in the previous frame by more than 3 but less than 20 pixels, it indicates that the target position has changed. At this time, the area obtained by difference is not a complete target. The area with the center misaligned between 3 and 20 pixels is combined to obtain a complete suspected area.

7. The flash detection method based on a combination of traditional image processing and deep learning as described in claim 6, characterized in that, Step S2 includes the following steps: S21: Dataset Construction Based on the suspected targets detected in step S1, a dataset is constructed for training a deep binary classification model; Expand the original image with an 8*8 square region centered on all suspected targets. If the region exceeds the size of the original image, replace it with 0. Use a training set:validation set:test set ratio of 8:1:1 to split the dataset. S22: Deep Learning Training The convolutional neural network model uses a combination of 5 convolutional modules and 3 fully connected modules. Each convolutional module consists of a convolutional layer, a layer of normalized batch normalization (BN), and an activation function ReLU. The fully connected module consists of a fully connected layer, an activation function ReLU, and a regularization operation Dropout. The last layer of the network is a sigmoid activation function, which outputs the probability value for image classification. If the value is greater than 0.5, it is judged as a real object; otherwise, it is a false object. The processed dataset is input into the convolutional neural network model for training, and the trained model is then placed into the test set for experimentation.

Images