Underwater target noise source classification model interpretability method based on gradient activation mapping

Abstract

The invention discloses an underwater target noise source classification model interpretability method based on gradient activation mapping. The first part comprises the following steps: preprocessing data, extracting an MFCC cepstrum of each frame of each type of audio, stacking the extracted cepstrum, and inputting the stacked cepstrum into a TDNN network to obtain voiceprint features of each type of audio; 2, performing normalization processing on the voiceprint data; using the normalized voiceprint data with labels as a feature space to train a classification model Cy; and finally, model parameters are saved after the training precision requirement is met. A third part: firstly, calculating an integration gradient size of each dimension of the voiceprint feature in the classification model by using an integration gradient method, and sequentially taking the integration gradient size as an index of the contribution degree of the integration gradient size; next, performing global average pooling on the gradient of the feature map output by the specific convolutional layer by using a class activation mapping method, and generating a heat map; and finally, combining the results obtained by the two methods to generate a fine-grained interpretable result.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and acoustic signal processing, and relates to an interpretability method of an underwater target noise source classification model based on gradient class activation mapping. Background Art

[0002] Underwater target noise source classification is an important technical task today, which aims to distinguish and identify different sound sources from the complex noise background in the marine environment. This technology has been widely used in many fields such as military defense, marine ecological research, underwater engineering and resource exploration. The key technologies involved are mainly the following: 1. Sound signal capture. The signals of underwater sound sources are usually collected by array hydrophones or single hydrophones. These devices can capture sounds from different directions and distances in complex marine environments. 2. Voiceprint feature extraction. Traditional voiceprint feature extraction technologies include Mel-Frequency Cepstral Coefficients (MFCC), linear predictive cepstral coefficients (LPCC) and spectral features, which help distinguish different types of sound sources during processing. 3. Pattern recognition and machine learning. Various statistical and machine learning algorithms are used to classify sound sources, including hidden Markov models (HMM), support vector machines (SVM), decision trees, etc. These technologies can learn patterns to distinguish different sound sources from the extracted voiceprint features. With the continuous development of deep learning technology in recent years, more and more researchers have begun to use deep neural networks for sound source classification, especially convolutional neural networks (CNN) and recurrent neural networks (RNN), which have shown excellent performance in processing sound data.

[0003] However, due to the complexity and nonlinearity of deep neural networks, it is often difficult to understand which parts of the spectrum the network makes classification decisions based on. In order to increase trust in the model and improve the efficiency of debugging models and datasets, researchers have begun to focus on the field of interpretability of underwater target noise classification models. The current mainstream interpretability methods are mainly divided into the following categories: 1. Gradient and activation-related interpretability methods. 2. Model post-processing interpretability methods. 3. Local approximation model. 4. Inverse method. For the first type of method, Selvaraju and Smilkov proposed the Grad-CAM method in "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", and Sundararajan et al. proposed the integrated gradient method in "Axiomatic Attribution for Deep Networks". The former uses the gradient information of the target category to weight the feature map of the convolutional layer, thereby generating a heat map for a specific output category, highlighting the important area; the latter provides an explanation by interpolating between the input features and a baseline and analyzing how the prediction score changes with the input. For the second type of methods, this type of method does not directly modify the original model, but provides explanations by analyzing the output and features of the model after the model is trained. There are mainly two methods: feature importance ranking and deviation analysis. For the third type of method, its core idea is to build a simple local model around the prediction of the model. This local model is easy to interpret and can be used to approximate the behavior of the original model. For the fourth type of method, there are mainly adversarial sample methods, which "cheat" the model by making slight modifications to the input samples, analyzing the sensitivity of the model to slight perturbations, and thus providing insights into the possible fragility and deviation of the model. These interpretability methods all aim to find the areas in the input data that have the greatest impact on the model classification. After obtaining these areas, researchers can focus on them and further improve the training effect of the model. However, the above methods are difficult to produce good explanations in the complex situation of underwater acoustic data, and they themselves cannot produce fine-grained visualization results. Summary of the invention

[0004] Based on the technical problems existing in the prior art, the present invention proposes an interpretability method for underwater target noise source classification model based on gradient class activation mapping, which focuses on the gradient of the output of a certain category in the convolutional neural network relative to the feature map of the last convolutional layer. These gradients can indicate the importance of each feature map to the final predicted output, and then mark which parts of the input the model considers to be the most important, thus realizing the interpretation of the underwater target noise classification model. Figure 1, the method consists of three parts: the first part extracts the voiceprint features of underwater sound source data as the unique identifier of a certain type of sound source, and uses it as the input of the subsequent classification model; the second part uses a multi-layer convolutional neural network model to classify the voiceprint features; the third part uses a combination of integrated gradients and gradient class activation mapping to interpret the trained network. The first part has the following process: first, the data is preprocessed, and the data set is required to be a multi-segment or underwater sound source audio data set with different classification labels; then for each type of audio, the MFCC cepstrum of each frame needs to be extracted; then the extracted cepstrum is stacked and input into the time delay neural network (TDNN) to obtain the voiceprint features of each type of audio; finally, the voiceprint features of different types are sorted to form a new voiceprint data set, which is used as a training data set. The second part has the following process: first, the voiceprint data is normalized; then the labeled normalized voiceprint data is used as the feature space to train the classification model C y ; Finally, save the model parameters after reaching the training accuracy requirement. The third part has the following process: First, use the integrated gradient method to calculate the integrated gradient size of each dimension of the voiceprint feature in the classification model, and use it as an indicator of its contribution in turn; Next, use the class activation mapping method to perform global average pooling on the gradient of the feature map output by a specific convolutional layer and generate a heat map; Finally, combine the results of the two methods to generate fine-grained interpretable results.

[0005] The technical solution of the present invention:

[0006] A method for interpretability of underwater target noise source classification model based on gradient class activation mapping, the steps are as follows:

[0007] (1) First, the audio data is preprocessed to determine the extraction parameters. Then, for each type of audio, the MFCC cepstrum of each frame needs to be extracted. Next, the extracted cepstrum is stacked and input into the TDNN network to obtain the voiceprint features of each type of audio. Finally, the voiceprint features of different types are sorted to form a new voiceprint dataset, which is used as the training dataset.

[0008] (1.1) First, the audio data format is standardized. Different types of underwater sound source audio are cut into wav format files of the same length, and then the parameters are determined: frame length s, window overlap length s ′ , retaining the total number of windows n and the dimension d of the MFCC vector. Among them, the frame length represents the length of each window slice, and we will extract MFCC once for each window; the window overlap length represents how much overlap is retained between the windows of two frames. In other words, the distance that each slice window slides back is ss ′; The total number of retained windows is the first n windows retained for MFCC extraction after multiple slicing. Assuming the total length of the original audio is x, the total number of windows N generated is:

[0009]

[0010] In order to unify the input matrix shape of TDNN, n≦N is uniformly taken as the final total number of windows.

[0011] (1.2) For these n windows, MFCC vectors are extracted. The extraction process is as follows: First, the frequency spectrum of the original window is subjected to fast Fourier transform (FFT).

[0012]

[0013] Where x[n] is the signal in a frame, X is the total number of sampling points, and Y(k) is the kth frequency point in the frequency domain.

[0014] The Fourier transformed result is passed through a set of Mel filter banks to convert the frequency into Mel scale.

[0015]

[0016] Where f is the frequency spectrum generated after fast Fourier transform.

[0017] The log energy of each Mel filter bank is then calculated.

[0018] S i =log(∑ k |Y(k)| 2 ×H i (k))

[0019] Among them, H i (k) represents the response of the ith Mel filter at frequency k. Y(k) is the result of the fast Fourier transform.

[0020] Finally, the logarithmic energy of each filter group is discrete cosine transformed to obtain the MFCC vector.

[0021]

[0022] According to the dimension d of the MFCC vector determined in the previous section, the generated MFCC vector c i After interception, we will eventually obtain n MFCC vectors of length d.

[0023] (1.3) Repeat steps (1.1) and (1.2), each time selecting a different frame length and window overlap length, to obtain enough MFCC vectors of a certain type of audio.

[0024] (1.4) In order to meet the characteristics of the TDNN neural network's timing, the MFCC vectors need to be stacked into a matrix form in sequence, and the shape of the matrix is ​​n×d. These MFCC matrices are input into the TDNN network as training sets, and the labels are set to the corresponding spectrum types. The loss function uses the cross entropy loss function, and the optimizer uses the stochastic gradient descent optimizer. The network is trained until the classification accuracy meets the requirements. At this time, the parameters of the embedding layer after the statistical pooling layer in the TDNN network are selected as the voiceprint features of this type of sound source, and the labels are processed. Finally, these labeled voiceprint features are sorted and re-made into a data set. At this point, the extraction of voiceprint data is completed, which serves as the training set for the next step of classification network training.

[0025] (2) First, normalize the voiceprint data; then use the labeled normalized voiceprint data as the feature space to train the classification model C y ; Finally, save the model parameters after the training accuracy requirements are met.

[0026] (2.1) In order to improve training efficiency and training accuracy, it is necessary to normalize the input space in advance. The mathematical formula is as follows:

[0027]

[0028] Among them, Input is the original input, μ is the average value of the original input, and σ is the standard deviation of the original input. It can make the input data more uniform and more suitable for network training. Finally, the final input data A of the model is obtained, which is a tensor with a length of r, where r is the length of the voiceprint feature obtained in the first part.

[0029] A=[a1,a2....,a r ]

[0030] (2.2) After obtaining the normalized data, the classification model C is trained using labeled data in the new feature space y , where the classification model C y The main structure is as follows:

[0031] The convolutional layer is used to extract local features of the data. It extracts features by applying a set of learnable filters to slide over the input data. Its mathematical formula is as follows:

[0032] S(p,q)=(I*K)(p,q)=∑ w ∑ v I(pw,qv)·K(w,v)

[0033] Among them, I(p,q) is the pixel value of the input feature map at the position (p,q), K(w,v) is the weight of the convolution kernel at the position (w,v), S(p,q) is the result of the output feature map at the position (i,j), and the values ​​of w and v are determined by the size of the convolution kernel;

[0034] The batch normalization layer is used to speed up the convergence of model training and enhance the robustness of the model to the initialization method. Assume that x is a feature of the input, there are Sum samples in a batch, and the batch mean μ B and batch variance The calculation formulas are as follows:

[0035]

[0036]

[0037] Normalize each feature by its mean and variance:

[0038]

[0039] Here, ε is a very small positive number used to prevent division by zero errors during normalization. This can cause numerical instability when the variance is very small, and ε ensures that there is always a small positive value in the denominator.

[0040] The SiLU activation function is non-monotonic and has the ability of self-regularization to a certain extent, which can help improve the generalization ability of the model. Its mathematical definition is as follows:

[0041]

[0042] The above is the classification model C y The main network structure.

[0043] For the input sample {A1,A2,A3,…,A N}, the loss function uses the cross entropy loss function, the optimizer selects the stochastic gradient descent optimizer, the batch size is B, the learning rate is α, and the training steps are as follows:

[0044] 1) Select sample subsets of size B from the input sample set in sequence;

[0045] 2) Input the subset into the network, and after the convolution layer, batch layer, SiLU activation function calculation, the classification score of the batch is finally obtained through the softmax activation function;

[0046] 3) Compare the scores of the batch with the true labels and calculate the loss value using the cross entropy loss function. The mathematical formula of the cross entropy loss function is as follows:

[0047]

[0048] Where C is the total number of categories, y c is an indicator indicating whether the sample belongs to category c. is the probability given by the model that the sample belongs to category c.

[0049] 4) Use stochastic gradient descent to update model parameters. The mathematical formula is as follows:

[0050]

[0051] Among them, α is the learning rate, is the current parameter θ t Next, for the current input batch B t The gradient of the calculated loss function.

[0052] 5) Repeat the above process until the classification accuracy of the model reaches the required level, and then use the model C y Save for subsequent interpretive analysis.

[0053] (3) First, the integrated gradient method is used to calculate the integrated gradient size of each dimension of the voiceprint feature in the classification model, which is used as an indicator of its contribution. Next, the class activation mapping method is used to perform global average pooling on the gradient of the feature map output by a specific convolutional layer, and a heat map is generated. Finally, the results obtained by the two methods are combined to generate fine-grained interpretable results.

[0054] (3.1) Use the integrated gradient method to generate the contribution of each feature value in the original input to the classification result. For a neural network model nn, its essence is to convert the input A = [a1, a2...., a r ] is mapped to a predicted value nn(A), and now we need to measure a i The importance of the predicted value can be calculated using the following formula:

[0055]

[0056] Among them, A ′ is the baseline input, which is a completely black image or a tensor of all 0s, and β is a scalar from 0 to 1. is the model input relative to a i The partial derivative of (a i -a ′ i) is the difference between the input and the baseline; the integrated gradient method calculates the gradient change process of the model from a "no information" input to the real input data, and integrates the gradient change of each pixel in this process to obtain its contribution to the classification result; this change process is a linear change process, which is gradually approximated using the linear interpolation method. The mathematical formula is as follows:

[0057] A ′ (β) = A ′ +β×(AA ′ )

[0058] Where β is a scalar from 0 to 1, which is used to control the degree of interpolation from the baseline to the actual input. For the final integrated gradient score IG i (A), the higher the score, the higher the i The more important, the less important. Finally, retain the feature values ​​corresponding to the top K highest integrated gradient scores to obtain the integrated gradient explanation of the classification model.

[0059] (3.2) Use the gradient class activation mapping method to generate a heat map to identify the feature areas in the original input that have an important impact on the classification. This patent performs class activation analysis on the last convolutional layer. Assume that the convolutional layer generates K feature maps, and the model determines that the classification category of the current input is c. First, according to the forward propagation calculation of the model, each feature map A of the convolutional layer can be obtained. K Next, for the target category c, the score of this category is calculated by back propagation relative to each feature map A K Gradient Then, for each feature map, the global average of these gradients is calculated through global average pooling to obtain the weight score of the feature value for the target category. The calculation formula is as follows:

[0060]

[0061] Where Z is the feature map A K The total number of elements in , r, t are the spatial indices in the feature map. Finally, use the weight Perform a weighted linear combination of the feature maps to obtain the final heat map L c , and its calculation formula is:

[0062]

[0063] The ReLU function only focuses on features that have a positive impact on category c, thereby filtering out parts that are not helpful or have a counterproductive effect on the classification results.

[0064] (3.3) Finally, the first K pixels obtained in (3.1) are retained, and the rest are all set to 0. By performing matrix multiplication between the heat map obtained in (3.2) and the result of the integrated gradient, only the part with a higher integrated gradient score can be retained in the heat map to generate a fine-grained explanation result.

[0065] Beneficial effects of the present invention: By realizing the visualization of model decisions, the interpretability method proposed in this patent can significantly enhance the user's trust in the model prediction results. For fields with high technical barriers and high accuracy requirements such as underwater sound source classification, it is particularly important to understand how the model makes decisions. At the same time, this method can not only help researchers and developers understand the basis of model decisions, but also guide them to perform more effective feature engineering and model tuning. By identifying the features that have the greatest impact on the classification results, researchers can focus on these key factors, thereby improving the accuracy and efficiency of the model. In the data preprocessing stage, this patent innovatively extracts effective voiceprint features through TDNN to enrich the input data set of the model. Such processing significantly improves the quality of the data set and the accuracy of subsequent model training. In the work of interpretable analysis, the present invention combines the advantages of multiple methods to achieve a more accurate and fine-grained interpretation result. Finally, through visual analysis and interpretation tools, this method can reveal previously unobvious or unrecognized sound source features and promote scientific research in the field of underwater acoustic signal processing.

[0066] Application prospects of the present invention: 1) This technology can be used to monitor the underwater environment, identify and classify the activities of underwater organisms, such as the calls of whales or the noise of ships, which is of great significance to ecological protection and marine biological research. 2) In the military and security fields, it is extremely important to accurately identify underwater sound sources such as submarines and mines. The model interpretability provided by this method can help improve the recognition ability of sonar systems and enhance national defense security. 3) In the industrial field, in the development and maintenance of offshore oil and gas fields, accurately identifying the sounds emitted by underwater machinery and equipment can help prevent accidents and failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is an organizational structure diagram of the underwater sound source classification model interpretability method described in the present invention.

[0068] Figure 2 This is the voiceprint extraction process described in the present invention.

[0069] Figure 3 This is the classification model training process described in the present invention.

[0070] Figure 4 This is the model explanation process described in the present invention. DETAILED DESCRIPTION

[0071] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0072] An interpretability method for underwater target noise source classification model based on gradient class activation mapping includes voiceprint feature extraction of underwater target noise source, training of sound source classification model and visual interpretation of the classification model.

[0073] Reference Figure 2 ,The specific operation process of underwater target noise source soundprint feature extraction is as follows:

[0074] Step 1. According to the parameters: frame length s, window overlap length s ′ , keep the total number of windows n, and slice the original audio:

[0075]

[0076] Take n≦N as the final total number of windows.

[0077] Step 2. Perform fast Fourier transform on each window segment:

[0078]

[0079] Step 3. Process using Mel filter:

[0080]

[0081] Step 4. Calculate the logarithmic energy of each Mel filter:

[0082] S i =log(∑ k |Y(k)| 2 ×H i (k))

[0083] Step 5. Perform discrete cosine transform on the logarithmic energy of each group:

[0084]

[0085] Get the d-dimensional MFCC vector c i :

[0086] c i =(c1,c2,..,c d )

[0087] Step 6. Stack the MFCC vectors of n windows together to form the input matrix C:

[0088]

[0089] It is used as training input into the TDNN network to generate the voiceprint feature F of a certain type of sound source. i , repeat steps 1 to 6, changing the length of the audio slice each time, until a sufficient number of voiceprint features are generated.

[0090] Reference Figure 3 , the classification model training process is as follows:

[0091] Step 7. F i Perform standard normalization:

[0092]

[0093] The classification model input data A is obtained. In order to facilitate the calculation of the convolution layer, it is converted from a vector to a square matrix:

[0094]

[0095] Step 8. Obtain the convolutional feature map A through the convolution layer ′ , after the batch processing layer, for A ′ Each feature point x i Calculate the batch mean:

[0096]

[0097] Calculate batch variance:

[0098]

[0099] Get each feature point after the batch processing layer

[0100]

[0101] Step 9. After SiLU activation function:

[0102]

[0103] Step 10. Calculate the classification score after the softmax activation function, and then use the cross entropy loss function to calculate the loss of this training:

[0104]

[0105] Step 11. Update the model using stochastic gradient descent:

[0106]

[0107] Step 12. Repeat steps 7 to 11 until the model classification accuracy reaches the requirement, and save the model file C at this time. y .

[0108] Reference Figure 4 , the model interpretation process is as follows:

[0109] Step 13. Calculate the integrated gradient IG for each pixel i (x):

[0110]

[0111] Take the pixel points corresponding to the first K largest integrated gradients:

[0112] IG k (x)=(x1,x2,..,x k )

[0113] Step 14. For classification model C y The K feature maps generated by the last convolutional layer are back-propagated according to the target category c to calculate the score of the category relative to each feature map A K Gradient For each feature map, the global average of these gradients is calculated through global average pooling to obtain the weight score of the feature value for the target category

[0114]

[0115] Step 15. Use weights Perform a weighted linear combination of the feature maps to obtain the final heat map L c :

[0116]

[0117] Step 16. IG obtained according to step 13 k (x), process the original input data A and keep only k The pixel values ​​of the pixels in (x) are all set to 0, and the rest are obtained as A IG Finally, A IG With L c Multiply them together to get the final interpretation result.

Claims

1. An interpretability method for underwater target noise source classification model based on gradient class activation mapping, characterized in that: Here are the steps: (1) First, preprocess the audio and determine the extraction parameters; then, for each type of audio, extract the MFCC cepstrum of each frame; then stack the extracted cepstrum and input it into the TDNN network to obtain the voiceprint features of each type of audio; finally, organize the voiceprint features of different types to form a new voiceprint dataset, which is used as the training dataset; (1.1) First, standardize the audio format and cut different types of underwater sound source audio into wav format files of the same length. Then determine the parameters: frame length s, window overlap length s ′ , retain the total number of windows n and the dimension d of the MFCC vector; where the frame length s represents the length of each window slice, and MFCC will be extracted once for each window; the window overlap length s ′ Indicates how much overlap is retained between the windows of two frames, that is, the distance each slice window slides backward is ss ′ ; The total number of retained windows is the first n windows retained for MFCC extraction after multiple slicing; Assuming the total length of the original audio is x, the total number of windows N generated is: In order to unify the input matrix shape of TDNN, n≦N is uniformly taken as the final total number of windows; (1.2) For n windows, the MFCC vector is extracted. The extraction process is as follows: first, the spectrum of the original window is subjected to fast Fourier transform; Where x[n] is the signal in a frame, X is the total number of sampling points, and Y(k) is the kth frequency point in the frequency domain; The Fourier transformed result is passed through a set of Mel filter banks to convert the frequency into Mel scale; Where f is the spectrum generated after fast Fourier transform; Then the logarithmic energy of each Mel filter bank is calculated; S i =log(Σ k |Y(k)| 2 ×H i (k)) Among them, H i (k) represents the response of the i-th Mel filter at frequency k; Finally, the logarithmic energy of each Mel filter bank is discrete cosine transformed to obtain the MFCC vector; According to the dimension d of the MFCC vector determined in the previous section, the generated MFCC vector c i After interception, n MFCC vectors of length d will be obtained; (1.3) Repeat steps (1.1) and (1.2), each time selecting a different frame length and window overlap length, to obtain enough MFCC vectors of a certain type of audio; (1.4) In order to meet the time-series characteristics of the TDNN neural network, the MFCC vectors need to be stacked into a matrix form in sequence, and the shape of the MFCC matrix is ​​n×d; these MFCC matrices are input into the TDNN neural network as training sets, and the labels are set to the corresponding spectrum types. The loss function uses the cross entropy loss function, and the optimizer uses the stochastic gradient descent optimizer. The network is trained until the classification accuracy meets the requirements; at this time, the parameters of the embedding layer after the statistical pooling layer in the TDNN neural network are selected as the voiceprint features of this type of sound source, and the labels are processed; finally, these labeled voiceprint features are sorted and re-made into a data set. At this point, the voiceprint data extraction is completed, which serves as the training set for the next step of classification network training; (2) First, normalize the voiceprint data; then use the labeled normalized voiceprint data as the feature space to train the classification model C y ;Finally, save the model parameters after reaching the training accuracy requirements; (2.1) In order to improve training efficiency and training accuracy, it is necessary to normalize the input space in advance. The mathematical formula is as follows: Among them, Input is the original input, μ is the average value of the original input, and σ is the standard deviation of the original input; finally, the final input data A of the model is obtained, which is a tensor with a length of r, where r is the length of the voiceprint feature obtained in the first part; <h2 style=";text-align:left;direction:ltr">A=[a1,a2....,a<h2 style=";text-align:left;direction:ltr"> r <h2 style=";text-align:left;direction:ltr"> ] (2.2) After obtaining the normalized data, the classification model C is trained using labeled data in the new feature space. y , where the classification model C y The main structure is as follows: The convolution layer is used to extract local features of the data. It slides over the input data using a set of learnable filters to extract features. The mathematical formula is as follows: S(p,q)=(I*K)(p,q)=∑ w ∑ v I(p-w,q-v)·K(w,v) Among them, I(p,q) is the pixel value of the input feature map at the position (p,q), K(w,v) is the weight of the convolution kernel at the position (w,v), S(p,q) is the result of the output feature map at the position (i,j), and the values ​​of w and v are determined by the size of the convolution kernel; Batch Normalization layer is used to speed up the classification model C y The convergence speed of training and enhancement of classification model C y Robustness to initialization method; assuming x i is a feature of the input. There are Sum samples in a batch and the batch mean is μ B and batch variance The calculation formulas are as follows: Normalize each feature by its mean and variance: Here, ε is a very small positive number used to prevent division by zero errors during normalization; because when the variance is very small, it may cause numerical instability, and ε can ensure that there is always a small positive value in the denominator; SiLU activation function, which is non-monotonic and has the ability of self-regularization to a certain extent, helps improve the classification model C y The generalization ability of is defined mathematically as follows: The above is the classification model C y The main network structure in For the input sample set {A1,A2,A3,…,A N }, the loss function uses the cross entropy loss function, the optimizer selects the stochastic gradient descent optimizer, the batch size is B, the learning rate is α, and the training steps are as follows: 1) Select sample subsets of size B from the input sample set in sequence; 2) Input the sample subset into the classification model C y , after the calculation of the convolution layer, batch normalization layer, and SiLU activation function, the classification score of the batch is finally obtained through the softmax activation function; 3) Compare the classification scores of the batch with the true labels, and use the cross entropy loss function to calculate the loss value. The mathematical formula of the cross entropy loss function is as follows: Where C is the total number of categories, y c is an indicator indicating whether the sample belongs to category c. is the probability that the sample given by the model belongs to category c; 4) Update the classification model C using stochastic gradient descent y Parameters, the mathematical formula is as follows: Among them, α is the learning rate, is the current parameter θ t Next, for the current input batch B t The gradient of the calculated loss function; 5) Repeat the above process until the classification model C y The classification accuracy reaches the requirement, and the classification model C y Save for subsequent interpretive analysis; (3) First, the integrated gradient method is used to calculate the integrated gradient size of each dimension of the voiceprint feature in the classification model, and each dimension is used as an indicator of its contribution. Next, the class activation mapping method is used to perform global average pooling on the gradient of the feature map output by a specific convolutional layer and generate a heat map. Finally, the results obtained by the two methods are combined to generate a fine-grained interpretable result. (3.1) Use the integrated gradient method to generate the contribution of each feature value in the original input to the classification result; for a neural network model nn, its essence is to convert the input A = [a1, a2...., a r ] is mapped to a predicted value nn(A), and now we need to measure a i For the importance of the predicted value, the following calculation formula is used: Among them, A ′ is the baseline input, which is a completely black image or a tensor of all 0s, and β is a scalar from 0 to 1. is the model input relative to a i The partial derivative of (a i -a i ′ ) is the difference between the input and the baseline; the integrated gradient method calculates the gradient change process of the model from a "no information" input to the real input data, and integrates the gradient change of each pixel in this process to obtain its contribution to the classification result; this change process is a linear change process, which is gradually approximated using the linear interpolation method. The mathematical formula is as follows: A ′ (β)=A ′ +β×(A-A ′ ) Among them, β is a scalar from 0 to 1, which is used to control the degree of interpolation from the baseline to the actual input; for the final integrated gradient score IG i (A), the higher the score, the higher the i The more important, the less important; finally, retain the feature values ​​corresponding to the top K highest integrated gradient scores to obtain the integrated gradient explanation of the classification model; (3.2) Use the gradient class activation mapping method to generate a heat map to identify the feature areas in the original input that have an important impact on the classification; this method performs class activation analysis on the last convolutional layer. Assume that the convolutional layer generates K feature maps, and the classification model determines that the classification category of the current input is c; first, according to the forward propagation calculation of the classification model, each feature map A of the convolutional layer is obtained. K Activation value; Next, for the target category c, the score of this category is calculated by back propagation relative to each feature map A K Gradient Then, for each feature map, the global average of these gradients is calculated through global average pooling to obtain the weight score of the feature value for the target category. The calculation formula is as follows: Among them, Z is the feature map A K The total number of elements in, r, t are the spatial indexes in the feature map; finally, use the weight Perform a weighted linear combination of the feature maps to obtain the final heat map L c , and its calculation formula is: The SiLU activation function only focuses on features that have a positive influence on category c, thereby filtering out parts that are not helpful or have a counterproductive effect on the classification results; (3.3) Finally, the first K pixels obtained in (3.1) are retained, and the rest are all set to 0; the heat map obtained in (3.2) is matrix multiplied with the result of the integrated gradient, and only the part with a higher integrated gradient score is retained in the heat map to generate a fine-grained interpretation result.

Images