Deep learning assisted gunshot detection and positioning method

Abstract

The invention relates to a deep learning assisted gunshot detection and positioning method, which belongs to the technical field of sound positioning and comprises a gunshot detection stage and a sound source direction of arrival (DOA) estimation stage. In the gunshot detection stage, captured sound signals are classified by using a convolutional neural network (CNN), sound signals belonging to gunshots enter the DOA estimation stage, and the sound source arrival direction of the gunshots is positioned by using the CNN; according to the method, the multi-channel spectrogram of the received signal is used as an input feature, the method has the characteristics of flexibility and low cost, more information of the received sound signal can be acquired from a longer-term angle, the CNN-based deep learning network is better utilized, and the input requirement of DOA estimation can be met only by using the two microphones with very small sizes.

Description

Technical Field

[0001] The present invention belongs to the technical field of sound localization and relates to a deep learning-assisted gunshot detection and localization method. Background Art

[0002] In the field of counter-terrorism and security, it is crucial to accurately identify the occurrence and direction of gun attacks. To achieve this goal, researchers have explored a variety of technical means.

[0003] An intuitive method is to use photoelectric sensors to capture visible signs such as gun muzzles. However, this method may not only increase system cost and volume, but also as a line-of-sight detection device, its performance is easily affected by surrounding obstacles. Another idea is to use acoustic means to complete the task by deploying acoustic detection sensors. Acoustic methods have advantages in size and cost, but noisy environments may have an adverse effect on them.

[0004] In recent years, many methods for gunfire detection and localization have been proposed. Traditional methods usually try to track gunfire signals in the time domain, such as shock waves and signs of muzzle blast, which have high sound pressure levels and usually appear as impulse noise. Some studies have used microphone sensor arrays arranged at different locations to determine the location of gunfire using the time difference between the arrival of the sound wave signal at each sensor. The effectiveness of this method has been verified to a certain extent in simulations and field tests, but its performance in noisy environments still needs further evaluation.

[0005] In addition, there are studies that use signal analysis and processing techniques for gunshot detection and localization. For example, a sound source detection system based on the non-negative matrix factorization (NMF) method is proposed, and the modified controlled response power phase transformation (SRP-Phat) method is used in the localization stage. However, the accuracy of this system in an impulse noise environment still needs to be improved.

[0006] In recent years, deep learning has achieved remarkable results in fields such as computer vision and natural language processing. Inspired by this, some studies have applied deep learning to acoustic event detection (SED) and direction of arrival (DOA) estimation, and proposed methods based on convolutional neural networks (CNN). SED and DOA are general concepts for gunshot detection and localization. Although there have been some attempts in neural network-based gunshot detection, such as extracting sound signal features by calculating coefficients for prediction, or using two-dimensional visualization of sound signals to train CNN for detection, these methods do not fully consider the types of noise in practical applications, which may affect their performance in the real world. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a deep learning-assisted gunshot detection and positioning method.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A deep learning-assisted gunshot detection and positioning method includes a gunshot detection stage and a sound source arrival direction DOA estimation stage; in the gunshot detection stage, the captured sound signal is classified, and for the sound signal belonging to the gunshot, the arrival direction of the sound source of the gunshot is then located in the DOA estimation stage;

[0010] The gunshot detection stage includes the following steps:

[0011] Use microphone arrays to capture sound signals, add sound source category labels to each sound signal, and construct training sets and test sets;

[0012] Construct a sound source category recognition model based on convolutional neural network CNN;

[0013] The sound signals in the training set are converted into a series of decibel-scale amplitude spectrogram subsequences of a specific length, namely, spectrograms, and input into the sound source category recognition model, and network training is performed in combination with the sound source category labels of each sound signal; and the trained sound source category recognition model is tested using the test set;

[0014] The captured sound signal is converted into a spectrogram, input into a trained sound source category recognition model, and a sound source category with the highest posterior probability is output. For the sound signal belonging to the gunshot, the direction of arrival of the sound source is estimated;

[0015] The sound source arrival direction DOA estimation stage comprises the following steps:

[0016] Use a microphone array consisting of two microphones to capture sound signals, add a sound source arrival direction label to each sound signal, and construct a training set and a test set;

[0017] Construct a DOA estimation model based on convolutional neural network (CNN);

[0018] The sound signals in the training set are converted into multi-channel spectrograms, input into the DOA estimation model, and network training is performed in combination with the sound source arrival direction labels of each sound signal; the trained DOA estimation model is tested using the test set;

[0019] The captured sound signal is converted into a multi-channel spectrogram, input into the trained DOA estimation model, and outputs a sound source arrival direction with the highest posterior probability.

[0020] Furthermore, the sound source category recognition model is based on CNN, which includes an input layer, 4 convolutional layer blocks, a fully connected layer, a softmax layer and a classification layer; the 4 convolutional layer blocks are each composed of a convolutional layer, a batch normalization layer, a Relu layer and a maximum pooling layer; the last two convolutional layer blocks of the 4 convolutional layers are also provided with a dropout layer.

[0021] Furthermore, the sound source categories include semi-automatic firearm sounds, automatic firearm sounds, shotgun sounds, and non-firearm sounds, and the sound signals belonging to the semi-automatic firearm sounds, automatic firearm sounds, and shotgun sounds are subjected to DOA estimation.

[0022] Furthermore, during the training process of the sound source category recognition model, the sound source category recognition model compares the output sound source category result with the highest posterior probability with the actual sound source category label, and feeds the comparison result back to the training process to optimize the network model.

[0023] Furthermore, the spectrogram is the form in which the time-frequency components of the sound signal are presented, which is essentially the square amplitude of the short-time Fourier transform (STFT) of the sound signal s(t):

[0024] Spectrogram(t,ω)=|STFT(t,ω)| 2

[0025] Where ω is the width of the window.

[0026] Furthermore, the DOA estimation model is based on CNN, which includes an input layer, 6 convolutional layer blocks, a fully connected layer, a softmax layer and a classification layer; the 6 convolutional layer blocks are each composed of a convolutional layer, a batch normalization layer and a Relu layer; the first, second, fourth and sixth blocks of the 6 convolutional layer blocks are provided with a max-pooling layer, and the last four blocks are provided with a dropout layer.

[0027] Furthermore, the DOA estimation model uses the clock azimuth method to describe the direction of arrival of the sound source and divides the direction of arrival of the sound source into 12 categories; the nth incoming spectrogram is denoted as S n , then the posterior probability of the DOA estimation model output is given by p(θ i |S n ) is given by, where θ i represents the i-th category of DOA, 1≤i≤12.

[0028] The beneficial effect of the present invention is that: in the gunshot detection stage, the present invention uses a trained CNN-based gunshot detector to extract and analyze the time-frequency characteristics of the input sound signal to mark the most likely sound source. If the source of the input recording is within the range of the gunshot, it is further passed to another CNN-based DOA estimation network to determine the most likely DOA of the gunshot. In the CNN-based DOA estimation network, the DOA estimation problem is converted into a 12-category classification problem, each category represents a possible DOA value range, and the DoA category with the highest posterior probability is estimated and output. The present invention proposes a flexible and low-cost input feature selection, that is, a multi-channel spectrogram of the received signal. The gunshot spectrogram from a specific direction has its own unique pattern, which can be learned by a deep learning network, and then DOA estimation is performed on it. The time length of the sound signal in this way can be any value, and more information about the received sound signal can be obtained from a longer-term perspective; the input in the form of a graphic can better utilize the CNN-based deep learning network; and only a very small size of dual microphones is required to meet the input requirements of DOA estimation.

[0029] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 A flow chart of the deep learning-assisted gunshot detection and localization method;

[0032] Figure 2 Spectrograms of different types of recordings; (a) is gunshot, (b) is human voice, (c) is background noise in an urban area, and (d) is wind at night;

[0033] Figure 3 The network architecture for the sound source category recognition model;

[0034] Figure 4 A clock position system for sound source localization;

[0035] Figure 5 Diagram for dual microphone setup;

[0036] Figure 6The multi-channel spectrograms of gunshots from different directions produced by the same gun; (a) is at the 3 o'clock direction, and (b) is at the 8 o'clock direction;

[0037] Figure 7 Network structure of the model for DOA estimation. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0040] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0041] The present invention provides a deep learning-assisted gunshot detection and positioning method, the flow chart is as follows: Figure 1 As shown, it includes the gunshot detection stage and the corresponding DOA estimation stage, and finally outputs the result.

[0042] In the gunshot detection phase, the trained CNN-based gunshot detector is used to extract and analyze the time-frequency features of the input sound signal to mark the most likely sound source. If the source of the input recording is within the range of the gunshot, it is further passed to another CNN-based DOA estimation network to determine the most likely DOA of the gunshot.

[0043] Specifically, in the gunshot detection stage, a sound source category recognition model based on a convolutional neural network (CNN) was constructed to detect any signs of gunshots in the surrounding area. The sound signal originating from a specific type of sound source has its own unique time-frequency characteristics and can be used as the input of the network. The basic idea is to let the network learn the relevant features for gunshot detection from the time-frequency components through training. Specifically, the time-frequency components of the received sound signal are used as input features and fed back to the network. These components are presented in the form of spectrograms, which essentially corresponds to calculating the square amplitude of the short-time Fourier transform (STFT) of the input sound signal s(t):

[0044] Spectrogram(t,ω)=|STFT(t,ω)| 2

[0045] Where ω is the width of the window.

[0046] Figure 2 (a)-(d) in the figure show some examples of spectrograms of sound signals, including gunshots and other types of recordings. It can be seen that the spectrograms of gunshots have their own unique patterns, which can be learned by the network that performs detection on them later.

[0047] Finally, a well-trained network should be able to identify the sound source type of the corresponding sound signal with the highest posterior probability.

[0048] The sound signal captured by the microphone array is first converted into a series of decibel-scale amplitude spectrogram subsequences of a specific length, and then used as the input of the network. The input layer of the sound source category recognition model is followed by 4 convolutional layer blocks. Each consists of a convolutional layer, a batch normalization layer, a Relu layer, and a maximum pooling layer. It is worth noting that the last two blocks have an additional dropout layer. The last three layers, namely the fully connected layer, the softmax layer, and the classification layer, can finally determine the input by marking the corresponding sound source. The present invention adopts four different possible sound source categories (semi-automatic firearms, automatic firearms, shotguns, and other sounds produced by non-firearms). Figure 3 The network architecture of the sound source category recognition model is given.

[0049] A large number of gunshot recordings are used to train a network that intends to learn the connection between the time-frequency characteristics of the sound signal captured by the microphone array and its corresponding sound source. This learning process utilizes a large number of labeled sound recordings (each label represents a specific sound source, such as a specific category of firearms) and uses them as training data. More specifically, the sound source classifier is trained on a data store containing pairs of sound recordings and their corresponding sound source category labels. Later in the verification phase, the classifier outputs a sound source category with the highest posterior probability given the sound recording input, and these results are compared with the actual category labels. The corresponding information from these comparisons is fed back into the training process to obtain a better network.

[0050] After training is complete, the pre-trained network continues to detect streaming audio captured from the microphone. Specifically, these incoming audios (with a specific time length) are stored in a buffer and are constantly labeled (classified) by the network. Eventually, the network outputs the most likely sound labels based on their probabilities. Once the sound detected by the network belongs to gunshots, the next step is to locate the direction of the source of these gunshots.

[0051] A deep CNN is used to learn the connection between the multi-channel spectrogram of the incoming microphone array signal and the DOA of the corresponding sound waves, leveraging a large amount of labeled (each label corresponds to a specific direction) training data.

[0052] Therefore, the DOA estimation problem is transformed into a Class I classification problem. Each class represents a possible range of DOA values, and the DOA class with the highest posterior probability is estimated.

[0053] In police and military operations, police and military personnel often use a clock position system to share the location information of potential targets. In this system, the observer imagines himself / herself on a horizontal clock face with 12 o'clock on the front, assuming that he / she is on the bow of the ship, and observes the time numbers located in the line of sight, such as Figure 4 As shown. For example, 12 o'clock indicates the front, 6 o'clock indicates the back, 3 o'clock indicates the right, and 9 o'clock indicates the left. Compared with the traditional way of describing directions with angles between [0°, 360°], this clock system is much simpler and causes much less confusion in practice. Therefore, in the present invention, the number of categories I is set to 12, corresponding to using the analogy of a 12-hour clock to describe the possible relative directions of the target.

[0054] In DOA estimation, it is first necessary to determine the feature representation that contains enough information. In the prior art, the phase component of the STFT coefficients of the signals received by multiple microphones is used as the input of the CNN. The basic principle is to enable the network to learn the relevant features of DOA estimation from the phase component through training. In practice, the phase components of the signals received by different microphones within a specific time range are used as input features and arranged into a matrix. This matrix is ​​defined as a phase map and then input into the deep learning network. A well-trained network should be able to estimate the DOA of the corresponding signal with the highest posterior probability. Although the aforementioned method has proved its effectiveness, it still exposes some shortcomings:

[0055] ① It can only estimate the DOA of the sound signal within a time frame of a static length. Considering the diversity of audio recordings, the length of the input recording needs to be more flexible.

[0056] ② Four or more microphones are required to form an array, which is inconvenient to carry and install. Therefore, a smaller array is needed to provide similar performance, by providing less equipment cost and more flexible deployment, so that the microphone array can be installed on a helmet.

[0057] ③CNN is widely considered to be the tool of choice for image (unstructured data) classification, but the matrices involved in existing technologies are more like structured data, which is to some extent beyond the professional scope of CNN.

[0058] Therefore, the present invention proposes a more flexible and less costly input feature selection, namely a multi-channel spectrogram of the received signal.

[0059] The existing microphone array is usually set with the observer position as the symmetric center. The present invention introduces a protruding structure different from the traditional design into the microphone array, such as Figure 5 As shown, the microphone array consisting of two microphones is placed slightly forward of the observer, so that the two microphones are still symmetrically placed, but the center of symmetry is located directly in front of the observer. This structure can solve the problem that the traditional microphone setting mode cannot distinguish sounds from symmetrical directions (for example, from 45° to the left front and 45° to the right rear).

[0060] Figure 6 (a) and (b) show the binaural spectrograms (Chanel 1 and Chanel 2) of gunfire signals from the same gun but different directions. It can be seen that the gunfire spectrograms from a specific direction have their own unique patterns, which can be learned by the deep learning network and then the DOA can be estimated.

[0061] The proposed deep learning framework consists of a training and a validation phase. In the training phase, the DOA classifier is trained on a data store containing pairs of multi-channel spectrograms and their corresponding DOA class labels. In the validation phase, the classifier outputs the DOA class with the highest posterior probability, given a multi-channel spectrogram input. With the multi-channel spectrogram as input, the task of the deep learning network is to generate the posterior probability of each DOA class. The nth incoming spectrogram is denoted as S n , then the posterior probability of the DOA estimation model output is given by p(θ i |S n ) is given by, where θ i represents the i-th category of DOA, 1≤i≤12. The network architecture is as follows Figure 7 As shown in Figure 1, the input layer is followed by 6 convolutional layer blocks. Each layer consists of a convolutional layer, a batch normalization layer, and a ReLU layer. It is worth noting that the first, second, fourth, and sixth blocks have an additional max-pooling layer, and the last four blocks have an additional dropout layer. The last three layers, namely the fully connected layer, the softmax layer, and the classification layer, are constructed by marking different corresponding DOAs (sound sources p(θ i |S n ) the largest layer), and finally determine the input. The present invention uses 12 possible DOAs of different sound source categories.

[0062] The network uses the training data set {{S n ,θn}|n=1,…,N}, where N is the total number of multichannel spectrograms in the set. In the validation phase, multichannel spectrograms of the test audio recordings are first generated, with the same settings used during training. These spectrograms are then fed as input to a pre-trained deep learning network, which ultimately outputs the predicted DOA class with the highest posterior probability for each test recording. These results are compared with the actual DOA class labels. The corresponding information from these comparisons is fed back into the training process to obtain a better network.

[0063] Gunshot detection experiment:

[0064] A microphone array consisting of two microphones was set up with a distance of 40 cm between the two microphones. To form the categories, the sound recordings were divided into four partitions corresponding to their sound source types, namely semi-automatic firearms, automatic firearms, shotguns, and others (sounds not generated by firearms). The sound sources were set to come from the far-field area.

[0065] An open source RIR (Room Impulse Response) generator is used to simulate different acoustic conditions. Training, testing, and validation data are generated by convolving shooting clips from different types of guns (more than 300 shooting records were used) with simulated RIRs. The location of the sound source in the simulated RIR is random and can be in any direction relative to the microphone array. This experimental example considers microphone arrays at different locations and convolves different types of noise. These randomly selected noises come from a set of noise samples that cover almost all possible scenarios in real life, including rainfall noise, engine noise, aircraft passing noise, periodic noise, and background noise in urban areas. The SNR level is randomly selected between -15 and 5dB.

[0066] Experimental comparison with RNMF-SRP-Phat: The ability of the proposed method to detect the source of gunfire under various SNR conditions was evaluated with that of RNMF-SRP-Phat (an existing gunfire detection solution using signal analysis and processing techniques), and the performance of the two in terms of overall detection accuracy was also compared. From the results, it can be seen that the solution proposed in the present invention is capable of locating gunfire signals and is equally comprehensive. Compared with NMF-SRP-Phat, detection accuracy was achieved under a range of SNR conditions from -15dB to 5dB. Given a complex acoustic scenario, RNMF-SRP-Phat has difficulty accurately locating gunfire signals, while the proposed solution is still able to locate gunfire signals. Under various SNR conditions from -15dB to 5dB, this method achieves higher overall detection accuracy compared to NMF-SRP-Phat.

[0067] Gunshot DOA estimation evaluation experiment: Considering two microphone arrays with a distance of 40cm between microphones, the entire DOA range ([0°, 360°]) is evenly divided into partitions, resulting in I = 12 DOA categories corresponding to the clock position system. An open source RIR generator is used to simulate different acoustic conditions. Training, test, and validation data are generated by convolving shooting clips from different types of guns (more than 300 shooting records are used) with the simulated RIR. Training, test, and validation data are generated in multiple evaluation scenarios. There are a total of about 12,000 sound clips for the 12 different DOA categories for each scenario.

[0068] For microphone arrays with symmetrical structures, first, the ability of the proposed method and CDOA (an existing CNN-based sound event detection scheme) to locate the source of gunfire under various SNR conditions is evaluated. From the results, it can be concluded that the proposed method is able to locate gunfire signals. In two different acoustic environments with different room sizes, reverberation times and source arrays, the performance of the proposed method is superior to CDOA in most cases. In addition, the robustness of the proposed method is studied by introducing small perturbations in the position of the gunfire source. Small perturbations are introduced to the position of the gunfire source by randomly moving in all directions (x, y, and z axes) with a maximum movement of 10 cm each. By comparison, it can be seen that compared with CDOA, at relatively low SNR levels, the robustness of the proposed method to such perturbations is stronger. By evaluating the performance under several other different acoustic conditions, it is found that the proposed method has strong robustness to perturbations under different SNR conditions.

[0069] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0070] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0071] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0072] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0073] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0074] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0075] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0076] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0077] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0078] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0079] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A deep learning-assisted gunshot detection and localization method, characterized in that: It includes a gunshot detection phase and a sound source arrival direction DOA estimation phase. In the gunshot detection phase, the captured sound signals are classified, and for the sound signals belonging to gunshots, the arrival direction of the sound source of the gunshots is located in the DOA estimation phase. The gunshot detection stage includes the following steps: Use microphone arrays to capture sound signals, add sound source category labels to each sound signal, and construct training sets and test sets; Construct a sound source category recognition model based on convolutional neural network CNN; The sound signals in the training set are converted into a series of decibel-scale amplitude spectrogram subsequences of a specific length, namely, spectrograms, and input into the sound source category recognition model, and network training is performed in combination with the sound source category labels of each sound signal; and the trained sound source category recognition model is tested using the test set; The captured sound signal is converted into a spectrogram, input into a trained sound source category recognition model, and a sound source category with the highest posterior probability is output. For the sound signal belonging to the gunshot, the direction of arrival of the sound source is estimated; The sound source arrival direction DOA estimation stage comprises the following steps: Use a microphone array consisting of two microphones to capture sound signals, add a sound source arrival direction label to each sound signal, and construct a training set and a test set; Construct a DOA estimation model based on convolutional neural network (CNN); The sound signals in the training set are converted into multi-channel spectrograms, input into the DOA estimation model, and network training is performed in combination with the sound source arrival direction labels of each sound signal; the trained DOA estimation model is tested using the test set; The captured sound signal is converted into a multi-channel spectrogram, input into the trained DOA estimation model, and outputs a sound source arrival direction with the highest posterior probability.

2. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: The sound source category recognition model is based on CNN, which includes an input layer, 4 convolutional layer blocks, a fully connected layer, a softmax layer and a classification layer; the 4 convolutional layer blocks are each composed of a convolutional layer, a batch normalization layer, a Relu layer and a maximum pooling layer; the last two convolutional layer blocks of the 4 convolutional layers are also provided with a dropout layer.

3. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: The sound source categories include semi-automatic gun sounds, automatic gun sounds, shotgun sounds, and non-gun sounds, and the sound signals belonging to the semi-automatic gun sounds, automatic gun sounds, and shotgun sounds are subjected to DOA estimation.

4. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: During the training process of the sound source category recognition model, the sound source category recognition model compares the output sound source category result with the highest posterior probability with the actual sound source category label, and feeds the comparison result back to the training process to optimize the network model.

5. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: The spectrogram is the form in which the time-frequency components of the sound signal are presented, which is essentially the square amplitude of the short-time Fourier transform (STFT) of the sound signal s(t): Spectrogram(t,ω)=|STFT(t,ω)|2 Where ω is the width of the window.

6. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: The DOA estimation model is based on CNN, which includes an input layer, 6 convolutional layer blocks, a fully connected layer, so ftmax layer and classification layer; the six convolutional layer blocks each consist of a convolutional layer, a batch normalization layer and a Rel u Layer composition; the first, second, fourth and sixth blocks of the 6 convolutional layer blocks are provided with max-pooling layers, and the last four blocks are provided with dropout layers.

7. The deep learning-assisted gunshot detection and localization method according to claim 1, characterized in that: The DOA estimation model uses the clock azimuth method to describe the direction of arrival of the sound source and divides the direction of arrival of the sound source into 12 categories; the nth incoming spectrogram is denoted as S n , then the posterior probability of the DOA estimation model output is given by p(θ i |S n ) is given by, where θ i represents the i-th category of DOA, 1≤i≤12.

Images