A sound target detection method integrating machine recognition and auditory brain-computer interface

Abstract

The present invention discloses a sound target detection method that integrates machine recognition and auditory brain-computer interface, comprising the following steps: obtaining a real-time sound signal and a sound sample training set, preprocessing and feature extraction of the real-time sound signal and the sound sample training set, and obtaining corresponding real-time acoustic features and training set acoustic features, respectively; constructing a confidence calculation model and a neural network model, and training the confidence calculation model and the neural network model based on the sound sample training set; obtaining the joint confidence of the real-time sound signal based on the trained confidence calculation model and the neural network model; judging the joint confidence, obtaining a brain-computer interface activation result, and then discriminating the real-time sound signal. The sound target detection method of the present invention can improve the anti-interference ability of the detection system and its robustness to new target samples, while reducing the burden on the brain-computer interface operator and improving the accuracy and efficiency of detection.

Description

Technical Field

[0001] The present invention belongs to the fields of human-machine collaboration science and brain-machine fusion technology, and in particular relates to a sound target detection method that integrates machine recognition and auditory brain-machine interface. Background Art

[0002] Target detection is a technology that extracts and analyzes environmental features to determine whether a target of interest exists. Common target detection systems include radar, optical, and acoustic detection. Acoustic detection, due to its passive operation, high concealment, resistance to obstruction and interference, and economical nature, has found widespread application across various industries. For example, the military uses acoustic technology to detect military vehicles and drones, nature reserves use it to track biological activity, and industry uses sound to detect damaged machinery.

[0003] However, target detection based on acoustic technology is often susceptible to interference from ambient noise, performs poorly in low signal-to-noise ratios, and suffers from low accuracy in identifying new targets. To address this issue, researchers have proposed numerous solutions. Traditional acoustic target detection relies primarily on machine learning techniques, which involve pre-acquiring data and features from detected sound segments, building a dataset for offline training to determine the parameters of the detection algorithm, and ultimately implementing it for practical online application. This approach offers the advantage of high automation and the ability to complete detection tasks in low-interference environments. However, because machine learning methods rely on the quality of the learning model and the training dataset, they often struggle to significantly improve detection robustness to noise. To address this, researchers have proposed a brain-computer interface (BCI)-based approach. This approach leverages changes in EEG signal characteristics when a person perceives a target sound during a target detection task to determine whether a neural representation corresponding to the sound target is present, thereby determining the presence of the detected target. This approach leverages the ability of humans to recognize targets in noisy environments and their robustness to new targets, addressing the challenges of traditional detection techniques. However, since this approach requires people to be in a state of auditory work for a long time, it greatly increases the operator's mental and physical exertion, easily causing fatigue and leading to a decrease in recognition accuracy, making it difficult to apply independently. Furthermore, neural networks can initially learn the time-frequency characteristics of target sounds, but because they rely on datasets, when datasets are insufficient, their generalization ability for different individuals within the same target class is poor. Statistical information, on the other hand, can collect common characteristics (such as harmonic characteristics and energy distribution) across different individuals within the same target class, making it suitable for further learning of target sound characteristics. Combining the two can complete target feature learning tasks at the machine recognition level.

[0004] In this context, in order to improve the anti-interference ability of the detection system and its robustness to new target samples while reducing the burden on brain-computer interface operators, it is urgent to propose a sound target detection method that integrates machine recognition and auditory brain-computer interface. Summary of the Invention

[0005] The purpose of this invention is to provide a sound target detection method that integrates machine recognition and auditory brain-computer interface technology. This method uses statistical information and the decision layer of a neural network classifier to obtain a joint confidence evaluation of the sound signal. This is used to measure the credibility of the machine recognition algorithm in distinguishing the current target. Based on the output joint confidence, a threshold judgment rule is used to determine whether brain-computer interface detection is enabled. The final detection result is output, integrating machine recognition and auditory brain-computer interface technology to determine whether the sound target exists. This method can efficiently complete the target detection task, improve the detection system's anti-interference ability and robustness to new target samples, and reduce the burden on the brain-computer interface operator, thereby solving the problems existing in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention provides a sound target detection method that integrates machine recognition and auditory brain-computer interface, comprising the following steps:

[0007] Acquiring a real-time sound signal and a sound sample training set, performing preprocessing and feature extraction on the real-time sound signal and the sound sample training set to obtain corresponding real-time acoustic features and training set acoustic features, respectively;

[0008] Constructing a confidence calculation model and a neural network model, and training the confidence calculation model and the neural network model based on the sound sample training set;

[0009] Obtaining a joint confidence of the real-time sound signal based on the trained confidence calculation model and the neural network model;

[0010] The joint confidence is judged to obtain the brain-computer interface activation result, and then the real-time sound signal is judged.

[0011] Optionally, the process of preprocessing and extracting features from the real-time sound signal and sound sample training set includes: preprocessing the real-time sound signal and sound sample training set by resampling and framing, baseline correction, and high-pass filtering, and then extracting corresponding Mel spectrum features based on short-time Fourier transform and Mel filter group.

[0012] Optionally, the process of obtaining the joint confidence of the real-time sound signal includes: obtaining a first confidence based on a trained confidence calculation model, and obtaining a second confidence based on a trained neural network model; normalizing the first confidence and the second confidence to obtain a joint confidence.

[0013] Optionally, the process of building a confidence calculation model includes:

[0014] The acoustic features of the training set are averaged in the time domain to obtain the energy mean vector of the sound sample training set; the average values of the energy mean vectors of the target sound samples and the non-target sound samples are obtained respectively, thereby obtaining the corresponding first energy distribution baseline and second energy distribution baseline; the distance between each type of sound sample and the corresponding baseline is counted, and the distance is fitted into an exponentially modified Gaussian probability density, thereby obtaining a confidence calculation model, wherein the sound sample training set includes target sound samples and non-target sound samples.

[0015] Optionally, the process of obtaining the first confidence level based on the trained confidence calculation model includes: obtaining the distances between the energy mean vector of the sound sample training set and the first energy distribution baseline and the second energy distribution baseline respectively; and calculating the cumulative distribution of the corresponding distances on the fitting distribution to obtain the first integral value and the second integral value; mapping the first integral value and the second integral value to a preset interval to obtain the first confidence level.

[0016] Optionally, the process of obtaining a second confidence level based on the trained neural network model includes: constructing a neural network model, training the neural network model based on the acoustic features of the target sound sample and the non-target sound sample, and then outputting a second confidence level based on the classifier decision layer of the trained neural network model.

[0017] Optionally, the process of distinguishing the real-time sound signal includes: presetting a brain-computer interface activation threshold, and when the joint confidence is higher than the threshold, outputting the joint confidence model's distinction result on the real-time sound signal; when the joint confidence is lower than the threshold, enabling brain-computer interface detection, and outputting the brain-computer interface's distinction result on the real-time sound signal.

[0018] The technical effects of the present invention are:

[0019] The present invention proposes a sound target detection method that integrates machine recognition and auditory brain-computer interface. It can integrate machine recognition of sound and sound target detection of auditory brain-computer interface, combine the advantages of the two methods to efficiently complete the target detection task, improve the anti-interference ability of the detection system and the robustness to new target samples, while reducing the burden on the brain-computer interface operator and improving the accuracy and efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0021] Figure 1This is a schematic diagram of the sound target detection structure that integrates machine recognition and auditory brain-computer interface in an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the Mel spectrum feature extraction process in an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of the structure of a neural network classifier in an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of a confidence model based on statistical information in an embodiment of the present invention;

[0025] Figure 5 FIG. 4 is a schematic diagram of the Mel spectrum of a target sound signal according to an embodiment of the present invention. FIG. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Example 1

[0029] like Figure 1 As shown, this embodiment provides a target detection method that integrates machine recognition and auditory brain-computer interface, including the following steps: obtaining real-time sound signals and sound sample training sets, preprocessing and feature extraction of the real-time sound signals and sound sample training sets, and obtaining corresponding real-time acoustic features and training set acoustic features respectively; constructing a confidence calculation model and a neural network model, and training the confidence calculation model and the neural network model based on the sound sample training set; obtaining the joint confidence of the real-time sound signal based on the trained confidence calculation model and neural network model; judging the joint confidence, obtaining the brain-computer interface activation result, and then judging the real-time sound signal.

[0030] This embodiment is used for sound target detection tasks under complex conditions with low signal-to-noise ratio, and specifically includes: a sound acquisition system, a feature extraction system, a joint confidence calculation system, and a human-machine fusion decision-making and judgment system.

[0031] The sound acquisition system is used to collect sound signals from the detection environment in real time, obtain real-time sound signals, and transmit the collected sound signals to the feature extraction system for signal preprocessing and feature extraction. In this example, a wireless microphone device (DJI MIC) is used to collect sound signals, and the sampling frequency is set to 48000Hz.

[0032] The feature extraction system is used to preprocess and extract features from the collected sound, including: resampling and framing of sound signals, baseline correction, high-pass filtering, and Mel spectrum feature extraction.

[0033] Resampling and framing include:

[0034] In order to reduce the amount of calculation, the sound signal is resampled to 16000 Hz and cut into 3 s signal blocks for subsequent analysis, that is, each signal block contains 16000×3=48000 sampling points.

[0035] Baseline correction methods include:

[0036] In order to eliminate the influence of zero drift in the signal, baseline correction is used to suppress the drift trend in the signal. The baseline correction filtering concept adopted in the present invention is sliding minus mean. For each segment of 48,000 sampling points, the 16,000 data points corresponding to the first 1 second are respectively subtracted from the average value of the signal of the first 1 second of the entire segment. For each data point of 1-3 seconds (16001-48000), the average value of the data of the first 1 second before each data point is respectively subtracted. The calculation is as follows:

[0037]

[0038] Among them, m i is the original sound signal of the i-th data point, X i is the sound signal after baseline correction.

[0039] High-pass filtering methods include:

[0040] The target (drone) sound in this embodiment is mainly concentrated in the medium and high frequencies. In order to preliminarily filter out the interference of environmental noise, a high-pass filter is used to filter out the low-frequency noise, and the filter cutoff frequency is 500 Hz.

[0041] like Figure 2 As shown, Mel spectrum feature extraction includes:

[0042] 1) Pre-emphasis:

[0043] In this embodiment, the target sound signal has a low-intensity high-frequency component and a high-intensity low-frequency component. To prevent a large difference in the intensity between the high- and low-frequency components of the signal, a high-pass filter is typically used to pre-emphasize the signal. Since high-pass filtering is involved in the preprocessing process, this step can be ignored.

[0044] 2) Frame division and windowing

[0045] In order to better obtain the relationship between signal frequency and intensity over time, the sound signal is divided into several frames, and a Fast Fourier Transform (FFT) is performed on each frame. After framing, in order to obtain a better sidelobe reduction amplitude, the present invention uses a Hanning window to add a window function (n fft =2048 hop_length=512)

[0046] 3) Mel filter bank

[0047] In this embodiment, the Mel filter bank is a triangular filter bank of equal height. The starting point of each filter is at the midpoint of the previous filter. The corresponding frequency is linear on the Mel scale, so it is called a Mel filter bank. The frequency corresponding to each filter can be converted into a Mel frequency by linearly dividing the maximum frequency into several frequency bands on the Mel scale, and then converted back to the actual frequency scale. The calculation of the conversion between frequency and Mel frequency is as follows:

[0048]

[0049] In this embodiment, the number of Mel filters n_Mel=512, and the power spectrum is converted into decibel (dB) units, with the reference being the power average value.

[0050] The joint confidence calculation system obtains a joint confidence evaluation of the sound sample based on the sample statistical information and the decision layer of the neural network classifier, and obtains the degree of confidence of the machine recognition on the judgment result of the current sound sample based on the joint confidence, including: training of the confidence model (a. based on statistical information b. based on the decision layer of the neural network classifier) and joint confidence calculation of the sound sample.

[0051] The training of the confidence model includes:

[0052] 1) Training set data classification

[0053] The sound signals in the training set were divided into two categories: target sounds and non-target sounds, which were used to train the joint confidence model. The non-target sound dataset consisted of recorded ambient sounds from busy roads, including human speech and activity, car traffic, and wind. The target sound dataset was a combination of recorded ambient sounds and two types of drone sound data.

[0054] 2) Preprocessing of training set data

[0055] The training set data is preprocessed and finally divided into 3s sound segments for model training, including: resampling and framing, baseline correction, and high-pass filtering.

[0056] 3) Confidence model based on statistical information

[0057] The Mel spectrum features of the sound signal are extracted based on the short-time Fourier transform method and the Mel scale filter bank, and the Mel spectrum features are averaged in the time domain to obtain the average energy distribution of each frequency band of the sound signal, that is, the energy mean vector. The Librosa library is used to complete the Mel spectrum feature extraction of the sound, and the Fourier transform parameter n is used to calculate the energy mean vector. fft =2048hop_length=512, number of Mel filters n_Mel=512, convert the power spectrum into decibel (dB) units, and use the power average as the reference. Then the dimension of the energy mean vector obtained after time domain averaging is 512-dimensional. The calculation method of the average energy mean vector is:

[0058]

[0059]

[0060] μ i,0 ,μ j,1 ∈C 1×Mel Num

[0061] Where μ i,0 Represents the energy mean vector of the i-th non-target sound sample, μ j,1 Represents the energy mean vector of the j-th target sound sample, and Mel Num represents the number of Mel filters.

[0062] Count the differences between each type of sound sample and its baseline, and define a certain sample μ ,0 (μ j,1 ) relative to the baseline M0(M1) is the Euclidean distance of the two vectors. The distances of various samples from the baseline are fitted into an exponentially modified Gaussian probability density, and the confidence calculation model based on sample statistical information is obtained:

[0063] distance(μ i,0 ,M0)=||μ i,0 -M0||2

[0064] distance(μ j,1 ,M1)=||μ j,1 -M1||2

[0065]

[0066]

[0067] Where, distance(μ i,0 (μ j,1 ), M0(M1)) is a certain type of sample μ i,0 (μ j,1 ) relative to M0(M1); and is the parameter estimate obtained by least square fitting. The fitted function represents the probability density of the distance between each type of sample and its baseline. It indicates the degree of change in the distance between the sample and its baseline. If the value is large, it means that the difference between the sample and its baseline varies in a large range, that is, the distribution of energy in each frequency band is relatively dispersed; if It is small, indicating that the difference between this type of sample and its baseline is small, that is, the distribution of energy in each frequency band is relatively consistent.

[0068] The joint confidence calculation of the sound sample includes:

[0069] 1) Confidence calculation based on statistical information

[0070] like Figure 4 As shown, the energy mean vector μ of the sound sample is calculated. sample And the distance from the baseline of the two types of samples ||μ sample -M0||2 and ||μ sample -M1||2.

[0071] By calculating the cumulative distribution of the distance on the fitted distribution and And by mapping the two integral values obtained to the (0,1) interval, the confidence of the sample based on statistical information is obtained:

[0072]

[0073]

[0074]

[0075] Where, The exponentially modified Gaussian probability density function is fitted using the training samples. ε is defined as an interval parameter. By adjusting ε to change the size of the integral interval, a better model can be obtained. pro,i is the cumulative distribution Normalized sample confidence.

[0076] 2) Confidence calculation based on the decision layer of the neural network classifier

[0077] The neural network model is trained according to the training samples, and the model architecture is as follows Figure 3 As shown, the activation function Sigmoid of the neural network output layer is used to output the confidence of the decision layer of the neural network classifier:

[0078]

[0079] Where, X i is the output of the previous hidden layer of the output layer, Cofidence_NN pro,i is the confidence that the sample belongs to class i.

[0080] 3) Joint confidence calculation

[0081] The joint confidence of the sound sample is obtained by normalizing the confidence based on statistical information and the confidence based on the decision layer of the neural network classifier. i :

[0082]

[0083] Where w1 and w2 are normalization parameters, satisfying w1+w2=1. The values of w1 and w2 can be adjusted to change the proportion of statistical information in the results and obtain a more optimal joint confidence model.

[0084] The human-machine fusion decision-making and judgment system determines whether to enable brain-computer interface detection through sound samples combined with confidence threshold judgment rules, integrating machine recognition of sound and sound target detection of auditory brain-computer interface.

[0085] Whether to enable brain-computer interface detection is determined based on the defined threshold T. When the confidence is higher than the threshold: Cofidence1>TorCofidence0>T, the machine model is considered to have a high "confidence" in considering the sample as a target or a non-target. Otherwise, the "confidence" of the machine model's discrimination result is low and the discrimination result is unreliable.

[0086] Secondly, when the confidence level is higher than the threshold, the system outputs the discrimination result of the machine model (target or non-target). When the confidence level is lower than the threshold, since the machine discrimination result is unreliable, it proves that the sound sample has complex components and is not suitable for machine judgment. Then the auditory brain-computer interface detection technology is enabled to output the discrimination result of the auditory brain-computer interface on the sound sample (target or non-target), such as Figure 5 The figure shows a schematic diagram of the Mel spectrum of the target sound signal.

[0087] The purpose of this embodiment is to integrate machine recognition and brain-computer interface technologies to propose a human-machine coordinated sound target detection system. In this system, based on statistical information and neural network output, the machine's confidence in the current target recognition is determined by setting a threshold to determine whether to enable auditory brain-computer interface detection, thereby handing over relatively simple sound targets to machine recognition and more complex targets to auditory brain-computer interface recognition. This system can effectively combine the advantages of both methods to complete target detection tasks, while improving the system's anti-interference ability and robustness to new target samples, while reducing the burden on the brain-computer interface operator and improving the accuracy and efficiency of detection.

[0088] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A sound target detection method integrating machine recognition and auditory brain-computer interface, characterized in that: The following steps are involved: Acquiring a real-time sound signal and a sound sample training set, performing preprocessing and feature extraction on the real-time sound signal and the sound sample training set to obtain corresponding real-time acoustic features and training set acoustic features, respectively; Constructing a confidence calculation model and a neural network model, and training the confidence calculation model and the neural network model based on the sound sample training set; Obtaining a joint confidence of the real-time sound signal based on the trained confidence calculation model and the neural network model; Determining the joint confidence level to obtain a brain-computer interface activation result, and then determining the real-time sound signal; The process of building a confidence calculation model includes: The acoustic features of the training set are averaged in the time domain to obtain the energy mean vector of the sound sample training set; the average values of the energy mean vectors of the target sound samples and the non-target sound samples are obtained respectively, thereby obtaining the corresponding first energy distribution baseline and second energy distribution baseline; the distance between each type of sound sample and the corresponding baseline is counted, and the distance is fitted into an exponentially modified Gaussian probability density, thereby obtaining a confidence calculation model, wherein the sound sample training set includes target sound samples and non-target sound samples.

2. The sound target detection method integrating machine recognition and auditory brain-computer interface according to claim 1 is characterized in that: The process of preprocessing and extracting features from the real-time sound signal and sound sample training set includes: resampling and framing, baseline correction, and high-pass filtering of the real-time sound signal and sound sample training set, and then extracting corresponding Mel spectrum features based on short-time Fourier transform and Mel filter bank.

3. The sound target detection method integrating machine recognition and auditory brain-computer interface according to claim 1 is characterized in that: The process of obtaining the joint confidence of the real-time sound signal includes: obtaining a first confidence based on a trained confidence calculation model, and obtaining a second confidence based on a trained neural network model; normalizing the first confidence and the second confidence to obtain a joint confidence.

4. The sound target detection method integrating machine recognition and auditory brain-computer interface according to claim 3 is characterized in that: The process of obtaining the first confidence level based on the trained confidence calculation model includes: obtaining the distances between the energy mean vector of the sound sample training set and the first energy distribution baseline and the second energy distribution baseline respectively; and calculating the cumulative distribution of the corresponding distances on the fitting distribution to obtain the first integral value and the second integral value; mapping the first integral value and the second integral value to a preset interval to obtain the first confidence level.

5. The sound target detection method integrating machine recognition and auditory brain-computer interface according to claim 3 is characterized in that: The process of obtaining a second confidence level based on the trained neural network model includes: constructing a neural network model, training the neural network model based on the acoustic features of the target sound sample and the non-target sound sample, and then outputting a second confidence level based on the classifier decision layer of the trained neural network model.

6. The sound target detection method integrating machine recognition and auditory brain-computer interface according to claim 1 is characterized in that: The process of distinguishing the real-time sound signal includes: presetting a brain-computer interface activation threshold, and when the joint confidence is higher than the threshold, outputting the joint confidence model's distinction result on the real-time sound signal; when the joint confidence is lower than the threshold, enabling brain-computer interface detection, and outputting the brain-computer interface's distinction result on the real-time sound signal.

Images