Sound event detection and localization method based on combined convolutional neural network

Abstract

A sound event detection and positioning method based on a combined convolutional neural network consists of the steps of data set preprocessing, feature extraction, construction of a combined convolutional neural network, training of the combined convolutional neural network, testing of the combined convolutional neural network, and detection and positioning of sound events. The present invention adopts a dual-branch convolutional neural network and a linear densely connected hybrid neural network. The conditional parameterized convolutional neural network dynamically calculates the convolution kernel according to the input signal and parameterizes the convolution kernel as a linear combination of n convolution kernels, thereby improving the learning ability and computational efficiency of the network. The linear densely connected hybrid neural network is adopted to focus on important information in the sound and ignore irrelevant sound information, thereby increasing the receptive field of the network and improving the accuracy of sound event detection and positioning. The present invention has the advantages of high recognition accuracy, stable network, and complete feature information extraction, and can be used for sound event detection and positioning.

Description

Technical Field

[0001] The present invention belongs to the field of speech signal processing and artificial intelligence technology, and specifically relates to the detection and positioning of sound signals. Background Art

[0002] Sound event detection and localization is to identify the temporal activity of each sound event, estimate their respective spatial position trajectories, and further associate them with their respective sound event labels. It is a combined task of sound event detection and arrival direction estimation. Sound event detection and localization has a wide range of applications in many technical fields such as robotics, smart cities, smart homes and industries, smart conferences, biodiversity monitoring, etc., and has very broad application prospects and important research value.

[0003] Early sound event detection and localization approaches addressed detection and localization separately, without linking the source location to the sound event. Detection typically employed methods such as Gaussian mixture models, hidden Markov models, or support vector machines. Localization, on the other hand, typically employed methods based on high-resolution spectral estimation, time delay estimation, and steerable beamforming.

[0004] With the rapid development of deep learning technology, a growing number of methods based on deep neural network models have significantly improved the performance of sound event detection and localization. Deep neural networks have achieved promising results for sound event detection, demonstrating the power of joint modeling for both detection and localization. Currently, most architectures combine convolutional neural networks with recurrent neural networks. However, the feature information extracted by convolutional and recurrent neural networks is relatively limited and often ignores important sound features, hindering further improvement in the accuracy of sound event detection and localization. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a sound event detection and positioning method based on a combined convolutional neural network with accurate speech emotion recognition, high recognition rate and good network stability.

[0006] The technical solution steps adopted to solve the above technical problems are as follows:

[0007] (1) Dataset preprocessing

[0008] Speech signals were extracted from the TAU spatial sound event dataset, which contains 11 sound event categories: throat clearing, coughing, doorbell pressing, door pushing, drawer sound, keyboard sound, door knocking, talking, laughing, book turning, and telephone ringing. The TAU spatial sound event dataset was divided into a training set and a test set in a ratio of 4:1.

[0009] (2) Feature extraction

[0010] Extract the Mel-spectrogram feature m from the speech signal as follows:

[0011]

[0012] Where f represents the actual frequency, and the generalized cross-correlation-phase transformation characteristic R is determined by the following formula: y1y2 (τ):

[0013]

[0014] Among them, G x1x2 (f) represents the power spectrum of the two signals x1 and x2, represents the phase shift weighting function, and τ represents the time delay.

[0015] (3) Constructing a combined convolutional neural network

[0016] The combined convolutional neural network is composed of a two-branch convolutional neural network, a linear densely connected hybrid neural network, and a fully connected layer connected in series.

[0017] The dual-branch convolutional neural network is composed of a first-branch convolutional neural network, a second-branch convolutional neural network, and a feature fusion network. The output end of the first-branch convolutional neural network and the second-branch convolutional neural network connected in parallel is connected to the input end of the feature fusion network.

[0018] The linear densely connected hybrid neural network is composed of a causal convolutional layer, a first void residual block, a second void residual block, a Transformer network, a third void residual block, and a depth-separable convolutional layer connected in series.

[0019] (4) Training combined convolutional neural networks

[0020] 1) Determine the objective function

[0021] The objective function includes the binary cross entropy loss function L, the mean square error loss function MSE, the evaluation function F1, the evaluation function ER, the evaluation function DOAerror, and the evaluation function Framerecall. The binary cross entropy loss function L is determined by the following formula:

[0022]

[0023] Among them, y is the true label value, y∈{0,1}, is the predicted probability value, The mean square error loss function MSE is determined as follows:

[0024]

[0025] Among them, n is the number of samples, n is a finite positive integer, y i is the actual value of the i-th sample, y i ′ is the predicted value of the i-th sample.

[0026] The evaluation function F1 of the comprehensive index of precision and recall is determined as follows:

[0027]

[0028]

[0029]

[0030] Among them, P is precision, R is recall, TP is true positive, FP is false positive, and FN is false negative.

[0031] The evaluation function ER of the number of errors is determined as follows:

[0032]

[0033] S(k)=min(FN(k),FP(k))

[0034] D(k)=max(0,FN(k)-FP(k))

[0035] I(k)=max(0,FP(k)-FN(k))

[0036] Among them, FN(k) represents the false negative event in the k-th segment of the system output, and FP(k) represents the false positive event in the k-th segment of the system output.

[0037] The evaluation function DOAerror is determined as follows:

[0038]

[0039] in, represents the reference angle at time t, represents the estimated angle at time t, Express estimate The total number of angles at time t, H refers to the Hungarian method for solving task allocation.

[0040] The evaluation function Framerecall is determined as follows:

[0041]

[0042] Among them, 1() means satisfying If the condition is true, output 1, otherwise output 0. Indicates reference The total number of angles at time t, where T represents the time frame and is not equal to 0.

[0043] 2) Training the combined convolutional neural network

[0044] The extracted Mel spectrum feature m and generalized cross-correlation-phase transform feature R y1y2 (τ) is sent to the dual-branch convolutional neural network for deep speech feature extraction. The deep speech features output by the dual-branch convolutional neural network are input into the linear densely connected hybrid neural network to model the time dependency of speech. During the training process, the learning rate r∈[10 -5 ,10 -3 ], the optimizer uses Adam optimizer, and iterates until the loss function L and MSE converge.

[0045] (5) Test the combined convolutional neural network

[0046] The test set is input into the trained combined convolutional neural network for testing.

[0047] (6) Detecting and locating sound events

[0048] The Sigmoid activation function in the fully connected layer is used for activation. The value range of the Sigmoid activation function is between 0 and 1. When the value of the Sigmoid activation function is greater than 0.5, the presence of a sound event is detected, and its arrival direction is estimated; when the distance to the sound event location is 1m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-40°, 40°]. When the distance to the sound event location is 2m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-20°, 20°]. The division is divided into 10° intervals.

[0049] Take all the reference azimuths in the tag as array M, and determine the estimated azimuth γ as follows:

[0050]

[0051] Take all the reference elevation angles in the tag as array N, and determine the estimated elevation angle β as follows:

[0052]

[0053] The estimated azimuth angle γ and the estimated elevation angle β are compared with the reference azimuth angle and the reference elevation angle through the evaluation function DOAerror to obtain the average angle error between the two.

[0054] In step (3) of the present invention, the first branch convolutional neural network is composed of a first conditional parameterized convolutional layer, a first batch normalization layer, and a first rectified linear unit layer connected in series.

[0055] The construction method of the first branch convolutional neural network is:

[0056] Output1(x)=ReLU(BN(CPConv1(x)))

[0057] Among them, CPConv1 is the first conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features.

[0058] The second branch convolutional neural network is composed of a second conditional parameterized convolutional layer, a second batch normalization layer, and a second rectified linear unit layer connected in series.

[0059] The construction method of the second branch convolutional neural network is:

[0060] Output2(x)=ReLU(BN(CPConv2(x)))

[0061] Among them, CPConv2 is the second conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features.

[0062] The feature fusion network is composed of a convolution layer, a global average pooling layer, a fully connected layer, and a softmax function layer connected in series. The convolution kernel size of the convolution layer is 1×1 and the step size is 1.

[0063] The first conditional parameterized convolution layer of the present invention is composed of an average pooling layer, a first fully connected layer, a deconvolution layer, and a sigmoid activation function layer connected in series in sequence.

[0064] The construction method of the first conditional parameterized convolutional layer is:

[0065] CPConv1(x)=(α1W1+...+α n W n )*x

[0066] Among them, α is the weight calculated by the routing function λ, W is the initialization weight, n is a finite positive integer, and * is the convolution operation.

[0067] The routing function λ is composed of an average pooling layer, the first fully connected layer, a deconvolution layer, and a Sigmoid activation function layer connected in series.

[0068] The construction method of routing function λ is:

[0069] λ=Sigmoid(Tconv(FC(AvgPool(x))))

[0070] Among them, AvgPool is average pooling, FC is full connection, Tconv is deconvolution, and Sigmoid is the activation function.

[0071] The second conditional parameterized convolutional layer of the present invention is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series in sequence.

[0072] The construction method of the second conditional parameterized convolutional layer is:

[0073] CPConv2(x)=(α1′W1+...+α n 'W n )*x

[0074] Among them, α′ is the weight calculated by the routing function ξ, W is the initialization weight, and n is a finite positive integer;

[0075] The routing function ξ is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series. The construction method of the routing function ξ is:

[0076] ξ=ReLU(BI(FC(LIPool(x))))

[0077] Among them, LIPool is local importance pooling, FC is full connection, BI is bilinear interpolation operation, and ReLU is rectified linear unit.

[0078] In step (3) of the present invention, the linear densely connected hybrid neural network is composed of a causal convolutional layer and a first void residual block, a second void residual block, a Transformer network, a third void residual block, and a depth-wise separable convolutional layer connected in series in sequence.

[0079] The construction method of the linear densely connected hybrid neural network is:

[0080] Z l =H l ([Z0,Z1,...,Z l-1 ])

[0081] Among them, [Z0,Z1,...,Z l-1 ] refers to the concatenation of feature maps generated from layer 0 to layer l, where l ranges from 1 to 5. l It is a composite operation that includes batch normalization, rectified linear unit, and convolution operations.

[0082] In step (3) of the present invention, the convolution kernel size of the causal convolution layer is 1×1 and the step size is 1; the first void residual block is composed of a void convolution layer, a sigmoid activation function layer, a tanh activation function layer, a first linear layer, a second linear layer, and a convolution layer, the output end of the void convolution layer is connected to the input end of the parallel sigmoid activation function layer and the tanh activation function layer, the output end of the sigmoid activation function layer is connected to the input end of the first linear layer, the output end of the tanh activation function layer is connected to the input end of the second linear layer, the output end of the first linear layer and the output end of the second linear layer are connected to the input end of the convolution layer; the structures of the second void residual block and the third void residual block are the same as that of the first void residual block.

[0083] The output method of each hole residual block is:

[0084] Y=(V g,i (sigmoid(W g,i *u i +b))+b′)⊙(V f,i (tanh(W f,i *u i +c))+c′)

[0085] Where * is the convolution operation, ⊙ is the element-wise dot product, W and V are learnable weights, b, b′, c, c′ are biases, i is the index of the layer, i takes values ​​of 1, 2, 3, f and g are filters and gates, tanh and sigmoid are activation functions, u i is the input of the dilated convolutional layer.

[0086] The convolution kernel size of the dilated convolution layer of the present invention is 3×3, the step size is 1, and the dilation rate is 2; the convolution kernel size of the convolution layer is 1×1, and the step size is 1.

[0087] In step (3) of the present invention, the convolution kernel size of the depthwise separable convolution layer is 3×3 and the step size is 1.

[0088] The present invention adopts a dual-branch conditional parameterized convolutional neural network and a densely connected hybrid neural network containing a linear layer. The dual-branch conditional parameterized convolutional neural network parameterizes the convolution kernel into a linear combination of n convolution kernels, thereby enhancing the learning ability of the network and enabling the network to extract richer sound feature information. Secondly, the densely connected hybrid neural network containing a linear layer can enable the network to focus on important information in the sound and ignore irrelevant sound information, thereby solving the problem that the feature information extracted by the convolutional neural network is relatively single and important sound feature information is ignored. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a flow chart of implementation 1 of the present invention.

[0090] Figure 2 It is a structural diagram of the combined convolutional neural network.

[0091] Figure 3 yes Figure 2 Schematic diagram of the structure of the dual-branch convolutional neural network.

[0092] Figure 4 yes Figure 3 Schematic diagram of the structure of the first conditionally parameterized convolutional layer in .

[0093] Figure 5 yes Figure 3 Schematic diagram of the structure of the second conditional parameterized convolutional layer in .

[0094] Figure 6 yes Figure 2 Schematic diagram of the structure of the linear densely connected hybrid neural network.

[0095] Figure 7 yes Figure 6 Schematic diagram of the structure of the first hole residual block in . DETAILED DESCRIPTION

[0096] The present invention will be further described below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0097] Example 1

[0098] exist Figure 1 In this embodiment, the sound event detection and positioning method based on the combined convolutional neural network consists of the following steps.

[0099] (1) Dataset preprocessing

[0100] Speech signals were extracted from the TAU spatial sound event dataset, which contains 11 sound event categories: throat clearing, coughing, doorbell pressing, door pushing, drawer sound, keyboard sound, door knocking, talking, laughing, book turning, and telephone ringing. The TAU spatial sound event dataset was divided into a training set and a test set in a ratio of 4:1.

[0101] (2) Feature extraction

[0102] Extract the Mel-spectrogram feature m from the speech signal as follows:

[0103]

[0104] Where f represents the actual frequency, and the generalized cross-correlation-phase transformation characteristic R is determined as follows: y1y2 (τ):

[0105]

[0106] Where G x1x2 (f) represents the power spectrum of the two signals x1 and x2, represents the phase shift weighting function, and τ represents the time delay.

[0107] (3) Constructing a combined convolutional neural network

[0108] exist Figure 2 In the embodiment, the combined convolutional neural network is composed of a two-branch convolutional neural network, a linear densely connected hybrid neural network, and a fully connected layer connected in series.

[0109] exist Figure 3 In the embodiment, the dual-branch convolutional neural network is composed of a first-branch convolutional neural network, a second-branch convolutional neural network, and a feature fusion network. The output end of the first-branch convolutional neural network and the second-branch convolutional neural network in parallel is connected to the input end of the feature fusion network.

[0110] The first branch convolutional neural network of this embodiment is composed of a first conditional parameterized convolutional layer, a first batch normalization layer, and a first rectified linear unit layer connected in series, and the second branch convolutional neural network is composed of a second conditional parameterized convolutional layer, a second batch normalization layer, and a second rectified linear unit layer connected in series.

[0111] The method for constructing the first branch convolutional neural network of this embodiment is:

[0112] Output1(x)=ReLU(BN(CPConv1(x)))

[0113] Among them, CPConv1 is the first conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features.

[0114] The method for constructing the second branch convolutional neural network of this embodiment is:

[0115] Output2(x)=ReLU(BN(CPConv2(x)))

[0116] Among them, CPConv2 is the second conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features.

[0117] The feature fusion network of this embodiment is composed of a convolutional layer, a global average pooling layer, a fully connected layer, and a softmax function layer connected in series. The convolution kernel size of the convolutional layer is 1×1 and the step size is 1.

[0118] exist Figure 4In the embodiment, the first conditional parameterized convolution layer is composed of an average pooling layer, a first fully connected layer, a deconvolution layer, and a sigmoid activation function layer connected in series.

[0119] The construction method of the first conditional parameterized convolutional layer is:

[0120] CPConv1(x)=(α1W1+...+α n W n )*x

[0121] Among them, α is the weight calculated by the routing function λ, W is the initialization weight, n is a finite positive integer, and * is the convolution operation.

[0122] The routing function λ is composed of an average pooling layer, the first fully connected layer, a deconvolution layer, and a Sigmoid activation function layer connected in series.

[0123] The construction method of routing function λ is:

[0124] λ=Sigmoid(Tconv(FC(AvgPool(x))))

[0125] Among them, AvgPool is average pooling, FC is full connection, Tconv is deconvolution, and Sigmoid is the activation function.

[0126] exist Figure 5 In the embodiment, the second conditional parameterized convolutional layer is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series.

[0127] The construction method of the second conditional parameterized convolutional layer is:

[0128] CPConv2(x)=(α1′W1+...+α n 'W n )*x

[0129] Among them, α′ is the weight calculated by the routing function ξ, W is the initialization weight, and n is a finite positive integer.

[0130] The routing function ξ is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series. The construction method of the routing function ξ is:

[0131] ξ=ReLU(BI(FC(LIPool(x))))

[0132] Among them, LIPool is local importance pooling, FC is full connection, BI is bilinear interpolation operation, and ReLU is rectified linear unit.

[0133] exist Figure 6 、 7 In the embodiment, the linear densely connected hybrid neural network is composed of a causal convolutional layer and a first void residual block, a second void residual block, a Transformer network, a third void residual block, and a depth-separable convolutional layer connected in series.

[0134] The construction method of the linear densely connected hybrid neural network is:

[0135] Z l =H l ([Z0,Z1,...,Z l-1 ])

[0136] Among them, [Z0,Z1,...,Z l-1 ] refers to the concatenation of feature maps generated from layer 0 to layer l, where l ranges from 1 to 5. l It is a composite operation that includes batch normalization, rectified linear unit, and convolution operations.

[0137] The output method of each hole residual block is:

[0138] Y=(V g,i (sigmoid(W g,i *u i +b))+b′)⊙(V f,i (tanh(W f,i *u i +c))+c′)

[0139] Where ⊙ is the element-wise dot product, W and V are learnable weights, b, b′, c, c′ are biases, i is the index of the layer, i takes values ​​of 1, 2, 3, f and g are filters and gates, tanh and sigmoid are activation functions, and u i is the input of the dilated convolutional layer.

[0140] The convolution kernel size of the causal convolution layer of this embodiment is 1×1 and the stride is 1. The first dilated residual block of this embodiment is composed of a dilated convolution layer, a sigmoid activation function layer, a tanh activation function layer, a first linear layer, a second linear layer, and a convolution layer. The output of the dilated convolution layer is connected to the input of the parallel sigmoid activation function layer and the tanh activation function layer, the output of the sigmoid activation function layer is connected to the input of the first linear layer, the output of the tanh activation function layer is connected to the input of the second linear layer, and the output of the first linear layer and the output of the second linear layer are connected to the input of the convolution layer. The structures of the second and third dilated residual blocks are the same as those of the first dilated residual block. The convolution kernel size of the dilated convolution layer is 3×3, the stride is 1, and the dilation rate is 2. The convolution kernel size of the convolution layer is 1×1 and the stride is 1. The convolution kernel size of the depthwise separable convolution layer of this embodiment is 3×3 and the stride is 1.

[0141] (4) Training combined convolutional neural networks

[0142] 1) Determine the objective function

[0143] The objective function includes the binary cross entropy loss function L, the mean square error loss function MSE, the evaluation function F1, the evaluation function ER, the evaluation function DOAerror, and the evaluation function Framerecall. The binary cross entropy loss function L is determined by the following formula:

[0144]

[0145] Among them, y is the true label value, y∈{0,1}, is the predicted probability value, The mean square error loss function MSE is determined as follows:

[0146]

[0147] Among them, n is the number of samples, n is a finite positive integer, y i is the actual value of the i-th sample, y i ′ is the predicted value of the i-th sample.

[0148] The evaluation function F1 of the comprehensive index of precision and recall is determined as follows:

[0149]

[0150]

[0151]

[0152] Among them, P is precision, R is recall, TP is true positive, FP is false positive, and FN is false negative.

[0153] The evaluation function ER of the number of errors is determined as follows:

[0154]

[0155] S(k)=min(FN(k),FP(k))

[0156] D(k)=max(0,FN(k)-FP(k))

[0157] I(k)=max(0,FP(k)-FN(k))

[0158] Among them, FN(k) represents the false negative event in the k-th segment of the system output, and FP(k) represents the false positive event in the k-th segment of the system output.

[0159] The evaluation function DOAerror is determined as follows:

[0160]

[0161] in, represents the reference angle at time t, represents the estimated angle at time t, Express estimate The total number of angles at time t, H refers to the Hungarian method for solving task allocation, and the evaluation function DOAerror is the error between the actual reference angle and the system estimated angle.

[0162] The evaluation function Framerecall is determined as follows:

[0163]

[0164] Among them, 1() means satisfying If the condition is true, output 1, otherwise output 0. Indicates reference The total number of angles at time t, where T represents the time frame and is not equal to 0.

[0165] 2) Training the combined convolutional neural network

[0166] The extracted Mel spectrum feature m and generalized cross-correlation-phase transform feature R y1y2 (τ) is sent to the dual-branch convolutional neural network for deep speech feature extraction. The deep speech features output by the dual-branch convolutional neural network are input into the linear densely connected hybrid neural network to model the time dependency of speech. During the training process, the learning rate r∈[10-5 ,10 -3 ], the learning rate r in this embodiment is 10 -4 , the optimizer uses Adam optimizer, and iterates until the loss function L and MSE converge.

[0167] (5) Test the combined convolutional neural network

[0168] The test set is input into the trained combined convolutional neural network for testing.

[0169] (6) Detecting and locating sound events

[0170] The Sigmoid activation function in the fully connected layer is used for activation. The value range of the Sigmoid activation function is between 0 and 1. When the value of the Sigmoid activation function is greater than 0.5, the presence of a sound event is detected, and its arrival direction is estimated; when the distance to the sound event location is 1m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-40°, 40°]. When the distance to the sound event location is 2m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-20°, 20°]. The division is divided into 10° intervals.

[0171] Take all the reference azimuths in the tag as array M, and determine the estimated azimuth γ as follows:

[0172]

[0173] Take all the reference elevation angles in the tag as array N, and determine the estimated elevation angle β as follows:

[0174]

[0175] The estimated azimuth angle γ and the estimated elevation angle β are compared with the reference azimuth angle and the reference elevation angle using the evaluation function DOAerror to calculate the average angular error between the two. The smaller the error, the more accurate the positioning.

[0176] Complete the sound event detection and localization method based on combined convolutional neural network.

[0177] Example 2

[0178] The sound event detection and localization method based on a combined convolutional neural network of this embodiment consists of the following steps.

[0179] (1) Dataset preprocessing

[0180] This step is the same as in Example 1.

[0181] (2) Feature extraction

[0182] This step is the same as in Example 1.

[0183] (3) Constructing a combined convolutional neural network

[0184] This step is the same as in Example 1.

[0185] (4) Training combined convolutional neural networks

[0186] 1) Determine the objective function

[0187] This step is the same as in Example 1.

[0188] 2) Training the combined convolutional neural network

[0189] The extracted Mel spectrum feature m and generalized cross-correlation-phase transform feature R y1y2 (τ) is sent to the dual-branch convolutional neural network for deep speech feature extraction. The deep speech features output by the dual-branch convolutional neural network are input into the linear densely connected hybrid neural network to model the time dependency of speech. During the training process, the learning rate r∈[10 -5 ,10 -3 ], the learning rate r in this embodiment is 10 -5 , the optimizer uses Adam optimizer, and iterates until the loss function L and MSE converge.

[0190] The other steps are the same as those in Example 1. The sound event detection and positioning method based on the combined convolutional neural network is completed.

[0191] Example 3

[0192] The sound event detection and localization method based on a combined convolutional neural network of this embodiment consists of the following steps.

[0193] (1) Dataset preprocessing

[0194] This step is the same as in Example 1.

[0195] (2) Feature extraction

[0196] This step is the same as in Example 1.

[0197] (3) Constructing a combined convolutional neural network

[0198] This step is the same as in Example 1.

[0199] (4) Training combined convolutional neural networks

[0200] 1) Determine the objective function

[0201] This step is the same as in Example 1.

[0202] 2) Training the combined convolutional neural network

[0203] The extracted Mel spectrum feature m and generalized cross-correlation-phase transform feature R y1y2 (τ) is sent to the dual-branch convolutional neural network for deep speech feature extraction. The deep speech features output by the dual-branch convolutional neural network are input into the linear densely connected hybrid neural network to model the time dependency of speech. During the training process, the learning rate r∈[10 -5 ,10 -3 ], the learning rate r in this embodiment is 10 -3 , the optimizer uses Adam optimizer, and iterates until the loss function L and MSE converge.

[0204] The other steps are the same as those in Example 1. The sound event detection and positioning method based on the combined convolutional neural network is completed.

[0205] In order to verify the beneficial effects of the present invention, the inventors conducted comparative experiments using the sound event detection and localization method of the combined convolutional neural network of Example 1 of the present invention and a convolutional recurrent neural network (hereinafter referred to as CRNN), a squeeze-excited convolutional recurrent neural network (hereinafter referred to as SE-CRNN), and a context-gated convolutional recurrent neural network (hereinafter referred to as CG-CRNN). The experimental results are shown in Table 1.

[0206] Table 1 Comparative experimental results of the present invention and three methods

[0207] Experimental methods Error rate ER F score F1 (%) DOA error (°) Frame recall (%) CRNN 0.28 85.4 24.6 85.7 SE-CRNN 0.25 85.8 23.9 86.7 CG-CRNN 0.19 88.7 24.4 90.3 Method of the present invention 0.23 87.1 22.4 90.2

[0208] As shown in Table 1, with DOA error as the main measurement indicator, the DOA error of the proposed method is reduced by 2.2 compared with the CRNN method, by 1.5 compared with the SE-CRNN method, and by 2.0 compared with the CG-CRNN method.

Claims

1. A sound event detection and localization method based on a combined convolutional neural network, characterized in that It consists of the following steps: (1) Dataset preprocessing Speech signals were extracted from the TAU spatial sound event dataset, which contains 11 sound event categories: throat clearing, coughing, doorbell pressing, door pushing, drawer sound, keyboard sound, door knocking, talking, laughing, book turning, and telephone ringing. The TAU spatial sound event dataset was divided into a training set and a test set in a 4:1 ratio. (2) Feature extraction Extract the Mel-spectrogram feature m from the speech signal as follows: Where f represents the actual frequency, and the generalized cross-correlation-phase transformation characteristic R is determined by the following formula: y1y2 (τ): Among them, G x1x2 (f) represents the power spectrum of the two signals x1 and x2, represents the phase transformation weighting function, τ represents the time delay; (3) Constructing a combined convolutional neural network The combined convolutional neural network is composed of a two-branch convolutional neural network, a linear densely connected hybrid neural network, and a fully connected layer connected in series; The dual-branch convolutional neural network is composed of a first-branch convolutional neural network, a second-branch convolutional neural network, and a feature fusion network. The output end of the first-branch convolutional neural network and the second-branch convolutional neural network connected in parallel is connected to the input end of the feature fusion network. The linear densely connected hybrid neural network is composed of a causal convolutional layer, a first atrous residual block, a second atrous residual block, a Transformer network, a third atrous residual block, and a depthwise separable convolutional layer connected in series. (4) Training combined convolutional neural networks 1) Determine the objective function The objective function includes the binary cross entropy loss function L, the mean square error loss function MSE, the evaluation function F1, the evaluation function ER, the evaluation function DOAerror and the evaluation function Framerecall. The binary cross entropy loss function L is determined by the following formula: Where y is the true label value, y∈{0,1}, is the predicted probability value, The mean square error loss function MSE is determined as follows: Where n is the number of samples, n is a finite positive integer, y i is the actual value of the i-th sample, y′ i is the predicted value of the i-th sample; The evaluation function F1 of the comprehensive index of precision and recall is determined as follows: Among them, P is precision, R is recall, TP is true positive, FP is false positive, and FN is false negative; The evaluation function ER of the number of errors is determined as follows: S(k)=min(FN(k),FP(k)) D(k)=max(0,FN(k)-FP(k)) I(k)=max(0,FP(k)-FN(k)) Where FN(k) represents the false negative event in the kth segment of the system output, and FP(k) represents the false positive event in the kth segment of the system output; The evaluation function DOAerror is determined as follows: in, represents the reference angle at time t, represents the estimated angle at time t, Express estimate The total number of angles at time t, H refers to the Hungarian method for solving task allocation; The evaluation function Framerecall is determined as follows: Among them, 1() means satisfying If the condition is true, output 1, otherwise output 0. Indicates reference The total number of angles at time t, where T represents the time frame and is not equal to 0; 2) Training the combined convolutional neural network The extracted Mel spectrum feature m and generalized cross-correlation-phase transform feature R y1y2 (τ) is sent to the dual-branch convolutional neural network for deep speech feature extraction. The deep speech features output by the dual-branch convolutional neural network are input into the linear densely connected hybrid neural network to model the time dependency of speech. During the training process, the learning rate r∈[10 -5 ,10 -3 ], the optimizer uses Adam optimizer, and iterates until the loss function L and MSE converge; (5) Test the combined convolutional neural network Input the test set into the trained combined convolutional neural network for testing; (6) Detecting and locating sound events The Sigmoid activation function in the fully connected layer is used for activation. The value range of the Sigmoid activation function is between 0 and 1. When the value of the Sigmoid activation function is greater than 0.5, the sound event is detected and its arrival direction is estimated. When the distance to the sound event location is 1m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-40°, 40°]. When the distance to the sound event location is 2m, the reference azimuth angle is [-180°, 180°] and the reference elevation angle is [-20°, 20°]. The division is based on an interval of 10. Take all the reference azimuths in the tag as array M, and determine the estimated azimuth γ as follows: Take all the reference elevation angles in the tag as array N, and determine the estimated elevation angle β as follows: The estimated azimuth angle γ and the estimated elevation angle β are compared with the reference azimuth angle and the reference elevation angle through the evaluation function DOAerror to obtain the average angle error between the two.

2. The sound event detection and localization method based on a combined convolutional neural network according to claim 1, characterized in that: In step (3), the first branch convolutional neural network is composed of a first conditional parameterized convolutional layer, a first batch normalization layer, and a first rectified linear unit layer connected in series; The construction method of the first branch convolutional neural network is: Output1(x)=ReLU(BN(CPConv1(x))) Among them, CPConv1 is the first conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features; The second branch convolutional neural network is composed of a second conditional parameterized convolutional layer, a second batch normalization layer, and a second rectified linear unit layer connected in series. The construction method of the second branch convolutional neural network is: Output2(x)=ReLU(BN(CPConv2(x))) Among them, CPConv2 is the second conditional parameterized convolution, BN is batch normalization, ReLU is the rectified linear unit, and x is the extracted Mel-spectrogram features and generalized cross-correlation-phase transform features; The feature fusion network is composed of a convolution layer, a global average pooling layer, a fully connected layer, and a softmax function layer connected in series. The convolution kernel size of the convolution layer is 1×1 and the step size is 1.

3. The sound event detection and localization method based on a combined convolutional neural network according to claim 2, characterized in that: The first conditional parameterized convolutional layer is composed of an average pooling layer, a first fully connected layer, a deconvolution layer, and a sigmoid activation function layer connected in series in sequence; The construction method of the first conditional parameterized convolutional layer is: CPConv1(x)=(α1W1+...+α n W n )*x Among them, α is the weight calculated by the routing function λ, W is the initialization weight, n is a finite positive integer, and * is the convolution operation; The routing function λ is composed of an average pooling layer, a first fully connected layer, a deconvolution layer, and a Sigmoid activation function layer connected in series. The construction method of routing function λ is: λ=Sigmoid(Tconv(FC(AvgPool(x)))) Among them, AvgPool is average pooling, FC is full connection, Tconv is deconvolution, and Sigmoid is the activation function; The second conditional parameterized convolutional layer is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series in sequence; The construction method of the second conditional parameterized convolutional layer is: CPConv2(x)=(α1′W1+...+α n 'IN n )*x Among them, α′ is the weight calculated by the routing function ξ, W is the initialization weight, and n is a finite positive integer; The routing function ξ is composed of a local importance pooling layer, a second fully connected layer, a bilinear interpolation layer, and a rectified linear unit layer connected in series. The construction method of the routing function ξ is: ξ=ReLU(BI(FC(LIPool(x)))) Among them, LIPool is local importance pooling, FC is full connection, BI is bilinear interpolation operation, and ReLU is rectified linear unit.

4. The sound event detection and localization method based on a combined convolutional neural network according to claim 1, characterized in that: In step (3), the linear densely connected hybrid neural network is composed of a causal convolutional layer, a first atrous residual block, a second atrous residual block, a Transformer network, a third atrous residual block, and a depthwise separable convolutional layer connected in series. The construction method of the linear densely connected hybrid neural network is: WITH l =H l ([Z0,Z1,...,Z l-1 ]) Among them, [Z0,Z1,...,Z l-1 ] refers to the concatenation of feature maps generated from layer 0 to layer l, where l ranges from 1 to 5; H l It is a composite operation that includes batch normalization, rectified linear unit, and convolution operations.

5. The sound event detection and localization method based on a combined convolutional neural network according to claim 1, characterized in that: In step (3), the convolution kernel size of the causal convolution layer is 1×1 and the step size is 1; the first atrous residual block is composed of a atrous convolution layer, a sigmoid activation function layer, a tanh activation function layer, a first linear layer, a second linear layer, and a convolution layer, the output end of the atrous convolution layer is connected to the input end of the parallel sigmoid activation function layer and the tanh activation function layer, the output end of the sigmoid activation function layer is connected to the input end of the first linear layer, the output end of the tanh activation function layer is connected to the input end of the second linear layer, the output end of the first linear layer and the output end of the second linear layer are connected to the input end of the convolution layer; the structures of the second atrous residual block and the third atrous residual block are the same as that of the first atrous residual block; The output method of each hole residual block is: Y=(V g,i (sigmoid(W g,i *u i +b))+b′)⊙(V f,i (tanh(W f,i *u i +c))+c′) Where * is the convolution operation, ⊙ is the element-wise dot product, W and V are learnable weights, b, b′, c, c′ are biases, i is the index of the layer, i takes values ​​of 1, 2, 3, f and g are filters and gates, tanh and sigmoid are activation functions, u i is the input of the dilated convolutional layer.

6. The sound event detection and localization method based on a combined convolutional neural network according to claim 5, characterized in that: The convolution kernel size of the dilated convolution layer is 3×3, the stride is 1, and the dilation rate is 2, and the convolution kernel size of the convolution layer is 1×1 and the stride is 1.

7. The sound event detection and localization method based on a combined convolutional neural network according to claim 1, characterized in that: In step (3), the convolution kernel size of the depthwise separable convolution layer is 3×3 and the stride is 1.

Images