Dog Barking Emotion Recognition Method Based on MFF-ScSEnet

Abstract

The present invention discloses a method for dog barking emotion recognition based on MFF-ScSEnet, which fuses the extraction of different features of audio signals and enhances the spectral ripple information during network training. Step 1: Convert the audio signal into a mel-spectrogram; Step 2: Convert the audio signal into a sinc-spectrogram; Step 3: Form a feature spectrogram; Step 4: Perform dog barking emotion recognition based on the feature spectrogram; Use the generated feature spectrogram to perform dog barking emotion recognition through the backbone network resnet18; Embed the scSEnet attention module after downsampling in Resnet18. The method of the present invention has the characteristics of high recognition accuracy, and the recognition accuracy reaches 91.45%.

Description

Technical Field

[0001] The present invention relates to a method for recognizing the emotion of a dog's barking sound based on MFF-ScSEnet. Background Art

[0002] Nowadays, more and more people keep pets at home to provide companionship in their spare time. However, for those who keep pet dogs for the first time, it is relatively complicated to recognize the emotions of pet dogs. Moreover, the owner is sometimes away from home for a long time, and pet dogs left alone at home for a long time will develop separation anxiety. Nowadays, the Internet of Things is becoming more and more powerful, and it is feasible to use Internet of Things technology to recognize the current emotional state of pet dogs and prompt the breeders. Experienced breeders mainly recognize the emotional state of pet dogs through the barking sound of dogs, the facial state of dogs, and the posture of dogs. However, considering from the aspects of technology and cost, it is difficult to obtain the facial state features and posture features of pet dogs. The real-time acquisition of sound features can be achieved only with a recording device, and information that cannot be captured by image data can also be obtained from the sound features. The recording of the barking sound of dogs is not restricted by light and venue. By letting the pet dog wear a collar with a recording device, the barking sound can be collected, analyzed, and processed in real time, so as to realize the real-time detection and recognition of the emotional state of pet dogs, and transmit the information to the user through the Internet of Things, which helps the user distinguish the emotions of pets.

[0003] In summary, it is necessary to design a method for recognizing the emotion of a dog through its barking sound. Summary of the Invention

[0004] There are mainly two technical problems to be solved by the present invention. The first is to extract the characteristic information that can recognize the emotion of a pet dog from the barking sound of the dog. The second is that the information extraction by the filter in the process of generating the spectrogram will cause information loss, which limits the learning ability of the emotion recognition of the barking sound of the dog. For the first problem, the present invention proposes a method of using a mel-spectrogram to extract the frequency characteristics from the barking sound of the dog, which can extract the effective information in the audio data and filter out the useless information. For the second problem, the present invention proposes a feature fusion network (MFF-ScSEnet). First, the Mel-spectrogram with the advantage of low-frequency characteristics and the Sinc spectrogram with the advantage of timbre characteristics are extracted from the barking sound of the dog through the Mel filter and the Sincnet filter respectively, and pre-decision fusion is performed on them. Then, in order to enhance the sound ripple information of the spectrogram and reduce the influence of the spectrogram noise information on recognition, the ScSEnet attention module is introduced into the backbone network, and this method has the characteristic of high recognition accuracy.

[0005] The technical solution of the invention is as follows:

[0006] A method for recognizing the emotion of dog barking based on MFF-ScSEnet, which fuses the extraction of different features of audio signals and enhances the spectral ripple information in the training network. The overall process includes the following steps:

[0007] Step 1: Convert the audio signal into a mel-spectrogram;

[0008] The audio signal corresponding to the dog barking is converted into a 160-dimensional mel-spectrogram through pre-emphasis and Mel filters;

[0009] Step 2: Convert the audio signal into a sinc-spectrogram;

[0010] The audio signal corresponding to the dog barking passes through the Sincnet filter, then uses 160 convolutional kernels with a length of 251, and then through adaptive average pooling to obtain a 1024-dimensional output to obtain the sinc-spectrogram;

[0011] Step 3: Form a feature spectrogram;

[0012] The mel-spectrogram and the Sinc-spectrogram are connected horizontally to form a new feature spectrogram; the feature spectrogram is the mei-sinc spectrogram;

[0013] Step 4: Recognize the emotion of dog barking based on the feature spectrogram; [[ID=2३]]

[0014] The generated feature spectrogram is used to recognize the emotion of dog barking through the backbone network resnet18; the scSEnet attention module is embedded after downsampling in Resnet18.

[0015] The scSEnet attention module is used to enhance the sound ripple information of the spectrogram, reduce redundant information such as network learning noise, better reflect the details of the spectrogram, and improve the accuracy of dog barking emotion recognition.

[0016] The specific steps of Step 1 are as follows:

[0017] The pre-emphasized (pre-emphasis means compensating for the high-frequency components of the input signal) audio waveform is framed and windowed, and the short-time Fourier transform STFT is performed to obtain the spectral matrix; the process is as follows:

[0018]

[0019] w(t - r) is a Hamming window centered at r; the length of the Hamming window w(t - r) is set to 512, the window shift is 256, f represents frequency, y is the pre-emphasized barking audio signal, r represents the frame obtained from the STFT of the current window, and x(f, r) represents the finally obtained spectral matrix. Specifically, a two-dimensional spectrogram matrix x(f, r) is obtained by performing a Fourier transform in the window.

[0020] The two-dimensional spectrogram matrix x(f, r) is transformed through a mel filter, and the relationship is as follows:

[0021]

[0022] where f mel is the calculated Mel-scale frequency, and f is the normal Hertz frequency. The mel filter is set to 160. Finally, the output matrix is transformed into a mel-spectrogram through the specshow function in librosa.

[0023] In step three, the mel-spectrogram and the Sinc-spectrogram are connected horizontally, and the hstack function in numpy is used to merge the two separate spectrograms into a new feature spectrogram.

[0024] In step four, the ScSEnet attention module is added to the first convolutional layer of the backbone network Resnet18, that is, after downsampling.

[0025] ScSEnet consists of a channel attention (cSE) mechanism and a spatial attention (sSE) mechanism.

[0026] cSE processes the feature map through global average pooling and 1D convolution to obtain a vector group, then uses the sigmoid function for normalization, and then multiplies it with the original feature map to obtain a calibrated feature map.

[0027] sSE directly performs a 1*1 convolution on the feature map, then uses sigmoid for activation to obtain a spatial attention map, and then directly multiplies it with the original feature map to complete the spatial information calibration.

[0028] Finally, the feature maps generated by cSE and sSE are summed to form a new attention feature map to enhance the sound ripple information of the spectrogram and reduce redundant information such as network learning noise. Improve the recognition performance in the final recognition result.

[0029] Beneficial effects:

[0030] The method for recognizing the emotion of dog barking based on MFF-ScSEnet of the present invention is a feature fusion method, aiming to utilize the mel-spectrum diagram to extract the features in dog barking to recognize the emotional state of pet dogs, while improving the comprehensiveness of information extraction in the process of generating the spectrum diagram for dog barking feature extraction, and enhancing the sound ripple information of the spectrum diagram and reducing the influence of the spectrum diagram noise information on recognition.

[0031] In the first aspect, in order to make the data closer to the real situation and make the model designed by the present invention have a wide adaptability, we use a recording device to set up a corresponding environment in a stray dog rescue station to record dog barking data. The data contains noises such as wind sounds and car honks in real life. Then, let multiple people with long-term experience in raising pet dogs classify the collected data. Finally, we establish an audio data set including five expressions: happy, angry, lonely, anxious, and sad.

[0032] In the second aspect, in order to reduce the loss of signal features in audio feature extraction, the present invention converts the features extracted by the Sincnet filter into a Sinc-spectrum diagram, and fuses the advantage of the Mel spectrum diagram in extracting low-frequency features with the advantages of the Sinc spectrum diagram in timbre and harmonics to obtain the Mel-Sinc spectrum diagram of dog barking. The proposed feature fusion spectrum diagram can improve the accuracy of dog barking emotion recognition.

[0033] In the third aspect, in order to enhance the sound ripple information of the spectrum diagram and reduce redundant information such as network learning noise, the present invention embeds the scSEnet attention module after downsampling in Resnet18 to better reflect the details of the spectrum diagram and improve the accuracy of dog barking emotion recognition.

[0034] The example of the present invention is to better extract features from dog barking and fully absorb the timbre and pitch information of the audio signal, and proposes a feature fusion method (Mel-Sinc spectrum diagram). It combines the strong absorption ability of the Mel filter for low-frequency information and the advantages of the Sincnet filter in timbre and harmonics. And because there is environmental noise and other interferences in the natural acquisition process of dog barking, in order to better learn the ripple information of the spectrum diagram of dog barking and reduce redundant information such as learning environmental noise. The scSEnet is added to the residual network Resnet18 to enhance the effective information of dog barking. Brief Description of the Drawings

[0035] Figure 1 It is the overall framework diagram for the emotion recognition of dog barking of the present invention;

[0036] Figure 2 It is the network framework diagram for the emotion recognition of dog barking of the present invention;

[0037] Figure 3It is a flowchart of the resnet residual structure;

[0038] Figure 4 It is the clipping process of the dataset and the box plot ( Figure 4 (A) is the clipping process of the dataset, Figure 4 (B) is the box plot of each category and quantity);

[0039] Figure 5 It is the Mel-spectrum diagram and Sinc-spectrum diagram of the audio segment. Detailed implementation manners

[0040] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0041] Figure 1 It is the overall flowchart of the present invention. The audio samples of dog barking are passed through Mel and Sincnet filters to generate two spectrum diagrams. The generated data is split into a training set and a test set with a ratio of 8:2. Then, the two spectrum diagrams are linearly spliced to generate a new spectrum diagram. Next, it is passed through the network Resnet18, and after downsampling in the network, the ScSEnet attention mechanism is embedded. Finally, the prediction recognition result is obtained through the softmax function.

[0042] Figure 2 It is the network of ScSE-Resnet18. ResNet (Residual Neural Network) was proposed by four Chinese including Kaiming He from Microsoft Research Institute. By using the ResNet Unit, a 152-layer Neural network was successfully trained and won the championship in the ILSVRC2015 competition, achieving great success in the field of deep convolutional network image classification. Both VGGnet and GoogLeNet obtained good performance by increasing the network depth. However, problems such as overfitting and network degradation would occur when the network depth was relatively deep. The author addressed the degradation problem (that is, as the network depth increases, the accuracy reaches saturation, and continuing to increase the depth leads to a rapid decline in accuracy). A residual learning module was designed, and its structure is as Figure 3 shown. Stacked non-linear layers are used to fit the residual mapping F(x) = H(x) - x, rather than directly fitting H(x). At this time, the basic mapping H(x) can be expressed as F(x) + x. x is the input of the previous layer, passing through a residual function F, and the final output is H = F(X) + x.

[0043] The attention module can generate attention-aware features, which can be used to emphasize or select important information of the target processing object and suppress some irrelevant detailed information. Such as Senet, CBAMnet, etc. The scSEnet attention module is an improvement based on SEnet by Roy AG et al. Three variants of the SE module, namely cSE, sSE, and scSE, are proposed, and the squeeze-and-excitation (SE) module is proposed. The SE block eliminates spatial dependence through global average pooling to learn specific channels. This module is used for channel recalibration of feature maps in image classification. (scSE) recalibrates the feature map along the channel and space respectively. cSE is the channel attention module, and sSE is the spatial attention module. The essence of the scSE module is the superposition of two attention mechanisms. After each backbone output U of the encoder enters the scSE module, it will enter two branches. The following branch is called the sSE branch. The branch performs a 1*1 convolution operation to obtain a weight matrix with the same length and width as U. The matrix is multiplied by U to obtain the spatially recalibrated feature UsSE. The upper branch is called the cSE branch. This branch first obtains a weight matrix with the same number of channels as U through a max-pooling operation. This matrix passes through two fully connected (FC) layers. The number of neurons in the first FC layer is half of the number of channels (to reduce computational complexity). The number of neurons in the second FC layer is equal to the number of channels. After the activation function, non-linearity is added. The matrix restored to the number of channels by the second FC is multiplied by U to obtain the channel-wise recalibrated feature UcSE. Finally, the features recalibrated along the channel and space are merged and output as UscSE. Experiments have proved that such a module can enhance meaningful features and suppress useless features. Since Resnet18 can only enter the residual module after downsampling. Information redundancy occurs during the downsampling process, and the model has poor discrimination ability for redundant information, thus reducing the classification performance of the network. The present invention improves Resnet18 by embedding the ScSEnet module into Resnet18 after downsampling. Obtain AcSE-resnet18 to enhance the ripple information of the spectrogram. Improve the classification performance of the model.

[0044] As Figures 1-4 , the method for recognizing dog barking emotions based on MFF-ScSEnet of the present invention includes the following steps:

[0045] Step 1: Convert the audio signal into a mel spectrogram. Perform frame addition and windowing on the pre-emphasized audio waveform, perform short-time Fourier transform STFT to obtain a matrix, and convert the matrix into a spectrogram. The process is as follows:

[0046] Set the length of the Hamming window w(t - r) to 512, the window shift to 256, f represents frequency, y is the pre-emphasized barking audio signal, r represents the frame obtained by the STFT of the current window, and x(f, r) represents the finally obtained spectral matrix. In (1), y(t) is the pre-emphasized time-domain signal, and w(t - r) is the Hamming window centered at r. The two-dimensional spectrogram matrix x(f, r) is obtained by performing a Fourier transform within the window. Since the human ear can perceive the frequency range from 20 to 2000 Hz, and the frequency resolution ability is uneven, the subjective perception of the frequency domain demarcation is non-linear, just like a filter bank that filters out other frequencies. The mel frequency is a non-linear frequency inspired by the auditory characteristics of the human ear, and this also reflects that the Mel spectrogram has a strong ability to learn low-frequency signals. The two-dimensional spectrogram matrix x(f, r) is converted through a mel filter, and its relationship is:

[0047]

[0048] where f mel is the calculated Mel-scale frequency, and f is the normal Hertz frequency. The function of the mel filter: improve the absorption of low-frequency signals in the audio signal.

[0049] The mel filter bank mimics the human ear to filter speech. M triangular filters are set within the frequency range of a barking audio. In the example, it is set to 512. The layout of the filters is from dense to sparse. As the Hertz frequency increases, the width of the filter increases from small to large. There is a 50% overlap between each filter to avoid information loss. On the Mel scale, these filters are shown to be of equal width. Finally, the output matrix is converted into a spectrogram through the specshow function in librosa. After the audio undergoes a Fourier transform, a matrix is generated;

[0050] Step 2: Convert the audio signal into a Sinc-spectrogram, and the Sinc-spectrogram is interpretable. Learn the features in the audio waveform through the cut-off frequency of the Sinc filter. The Sincnet filter is implemented using a set of mathematical operations as described below:

[0051] The function h[n] is a finite impulse response (FIR) filter used as a convolution filter to perform time-domain convolution:

[0052]

[0053] X[n] is the audio signal, and y[n] is the filtered output. g[n] is a predefined learnable function that depends on the parameters θ of the band-pass filter learned through one-dimensional convolution. This function depends on the parameter θ:

[0054] y[n] = x[n] * g[n, θ] (2)

[0055] rect(·) is the rectangular function in the frequency domain, that is, a band-pass filter. The amplitude of the band-pass filter is the difference between two low-pass filters. f1 and f2 are the cut-off frequencies of the low-pass filters, which are the starting frequency and the cut-off frequency of the band-pass filter respectively:

[0056]

[0057] Here, the sinc function is defined as sin c(x) = sin(x) / x, and the Fourier transform is rect(·):

[0058] g[n, f1, f2] = 2f2 sin c(2πf2n) - 2f1 sin c(2πf1n) (4)

[0059] For the learned band-pass filter, the cut-off frequencies f1 and f2 must ensure f2 ≥ f2 and f1 ≥ ≥ 0. Therefore, the parameter inputs actually used in (3) and (4) are:

[0060] f1 abc = |f1| (5)

[0061] f2 abc = f1 + |f2 - f1| (6)

[0062] To improve the leakage of the signal spectrum, the function is multiplied by a window function:

[0063] g w [n, f1, f2] = g[n, f1, f2] * w[n] (7)

[0064] The window function is the Hamming window:

[0065]

[0066] The role of the Hamming window function is to reduce the leakage of spectral energy;

[0067] Step 3: Convert the original audio into a 160-dimensional mel spectrogram. For the sinc spectrogram, use 160 convolutional kernels with a length of 251, and then obtain a 1024-dimensional output through adaptive average pooling to obtain the sinc spectrogram. Then, horizontally connect the spectrograms output respectively to form a new feature spectrogram. The steps to obtain the sinc spectrogram are existing mature steps.

[0068] Step 4: Perform barking sound emotion recognition based on the feature spectrogram

[0069] After the first convolutional layer (downsampling) of the backbone network Resnet 18, the ScSEnet attention module is added. The ScSEnet consists of a channel attention (cSE) and a spatial attention (sSE) mechanism. For cSE, the feature map is processed through global average pooling and 1D convolution to obtain a vector group, which is then normalized using the sigmoid function and multiplied by the original feature map to obtain a calibrated feature map. For sSE, the feature map is directly convolved with a 1*1 kernel, and then activated using sigmoid to obtain a spatial attention map, which is then directly multiplied by the original feature map to complete the spatial information calibration. Finally, the feature maps generated by cSE and sSE are summed to form a new attention feature map, so as to enhance the sound ripple information of the spectrogram and reduce redundant information such as network learning noise. The recognition performance is improved in the final recognition result.

[0070] To prove the effectiveness of this invention patent, the following experiments were conducted.

[0071] The barking dog audio established in this invention was obtained by setting up an experimental environment in a stray dog rescue station using a recording device. There are many breeds of dogs in this stray dog rescue station, such as Chinese rural dogs, Pomeranians, and Poodles, etc., and the collected audio data is relatively rich. We collected various barks by binding a recording pen around the dog's neck through wearing, and then used Adobe Audition software to edit and process the audio. The cutting process is as Figure 4 shown in (A). Each audio with unequal length is cut into audio segments about 4 seconds long, and each audio does not overlap with each other. To ensure the reliability of the data, all audio segments are judged and averaged by multiple personnel with more than three years of experience in raising pet dogs. All the data is divided into five categories: happy, angry, anxious, lonely, and sad. There are a total of 4440 audio files, and the box plot is as Figure 4 shown in (B). Since the data set is recorded from a real environment, there will be situations where background noise and dog barks overlap alternately, and the loudness and duration of dog barks will also change. Before training the network, this patent converts the cut audio segments into mel spectrograms with a size of 160*251 and sinc spectrograms with a size of 160*1024. The visualization diagrams of Phasianus - colchicus and Alectoris - graeca are as Figure 5 shown.

[0072] This invention patent was experimented under the hardware environment of CPU i9-10900K and GPU RTX 3090-24G. Based on the Pytorch deep learning framework, with version 1.7.0+cu110, learning rate of 0.001, Batch-size of 64, loss function of Cross-Entropy, trained for 200 Epochs, and the learning rate was decayed to one-tenth every 30 Epochs. The division ratio of the training set to the test set was 8:2, with 3552 dog bark segments in the training set and 888 dog bark segments in the test set. This invention patent proposed a feature fusion network MFF-ScSEnet. To reduce the loss of dog bark information, Resnet18 was used as the backbone network in this experiment to verify the effectiveness of feature fusion.

[0073] The experimental results are shown in Table 1. Resnet18 was used to identify the emotional state reflected by dog barks. Using the mel spectrogram as the input, the accuracy rate was 90.19%. Without changing the backbone network, a Sincnet filter was added for feature extraction to form a new spectrogram. The accuracy rate reached 91.05%, an increase of 0.86% compared to the baseline. Finally, ScSEnet was embedded after the downsampling of the backbone network to better reflect the details of the spectrogram, and the accuracy rate reached 91.28%, an increase of 1.09% compared to the baseline.

[0074] Table 1

[0075] Network model Input feature Accuracy rate Resnet18 Mel spectrogram 90.19％ Resnet18 Mel-Sinc spectrogram 91.05％ ScSE-Resnet18 Mel-Sinc spectrogram 91.28％

[0076] The recognition rate of the network for the dataset has a certain relationship with the data for dividing the training set and the test set. Therefore, the self-built dataset was divided three times, namely Split1, Split2, and Split3. The experimental results are shown in Table 2:

[0077] Table 2

[0078] Partition times Resnet18 MFF--ScSEnet Split1 90.01％ 91.19％ Split2 90.19％ 91.28％ Split3 90.33％ 91.45％

[0079] In the three divided datasets, the proposed model was embedded in Resnet18, and the classification accuracy rate increased by up to 1.18% at most, followed by 1.12%, and the lowest also increased by 1.09% based on the backbone network. Generally speaking, it stably improved the performance of the backbone network. The experiments in Table 1 were all carried out under the Split2 division.

[0080] The present invention discloses a method for identifying the emotion of dog barking based on deep learning of MFF-ScSEnet. First, the method extracts the Mel-spectrogram with the advantage of low-frequency features and the Sinc spectrogram with the advantage of timbre features from the dog barking audio through the Mel filter and the Sincnet filter respectively, and performs pre-decision fusion on them. Then, in order to enhance the sound ripple information of the spectrogram and reduce the influence of the spectrogram noise information on the recognition, the ScSEnet attention module is introduced into the backbone network to improve the network recognition performance. The MFF-ScSEnet dog barking recognition method proposed by the present invention achieves a recognition accuracy of 91.45% on the self-built dog barking dataset with a relatively low computational cost.

Claims

1. A method for recognizing the emotion of dog barking based on MFF-ScSEnet, characterized in that, Fuse the extraction of different features of the audio signal and enhance the spectral ripple information in the training network. The overall process includes the following steps: Step 1: Convert the audio signal into a mel-spectrogram; Convert the audio signal corresponding to the barking sound into a 160-dimensional mel-spectrogram through pre-emphasis and mel filters; Step 2: Convert the audio signal into a sinc-spectrogram; Pass the audio signal corresponding to the barking sound through the Sincnet filter, then use 160 convolutional kernels with a length of 251, and then obtain a 1024-dimensional output through adaptive average pooling to obtain the sinc spectrogram; Step 3: Form a feature spectrogram; Horizontally connect the mel-spectrogram and the Sinc-spectrogram to form a new feature spectrogram; the feature spectrogram is the mei-sinc spectrogram; Step 4: Perform barking sound emotion recognition based on the feature spectrogram; Perform barking sound emotion recognition on the generated feature spectrogram through the backbone network resnet18; embed the scSEnet attention module after downsampling in Resnet18.

2. The method for barking emotion recognition based on MFF-ScSEnet according to claim 1, characterized in that, The specific steps of Step 1 are as follows: Frame and window the pre-emphasized audio waveform, and perform short-time Fourier transform STFT to obtain the spectral matrix; the process is as follows: w(t - r) is the Hamming window centered at r; set the length of the Hamming window w(t - r) to 512, the window shift to 256, f represents the frequency, y is the pre-emphasized barking sound audio signal, r represents the frame obtained by STFT of the current window, and x(f, r) represents the finally obtained spectral matrix; Convert the two-dimensional spectrogram matrix x(f, r) through the mel filter, and its relationship is: where f mel is the calculated Mel-scale frequency, and f is the normal Hertz frequency. Set the Mel filter to 160, and finally convert the output matrix into a Mel spectrogram through the specshow function in librosa.

3. The method for barking emotion recognition based on MFF-ScSEnet according to claim 2, wherein, In Step 3, horizontally connect the mel-spectrogram and the Sinc-spectrogram, and use the hstack function in numpy to merge the two separate spectrograms into a new feature spectrogram.

4. The method for recognizing dog barking emotions based on MFF-ScSEnet according to claim 2, wherein In Step 4, add the ScSEnet attention module after the first convolutional layer (i.e., downsampling) of the backbone network Resnet18; ScSEnet consists of a channel attention (cSE) mechanism and a spatial attention (sSE) mechanism; cSE processes the feature map through global average pooling and 1D convolution to obtain a vector group, then uses the sigmoid function for normalization, and then multiplies it with the original feature map to obtain a calibrated feature map; sSE directly performs 1*1 convolution on the feature map, then uses sigmoid for activation to obtain the spatial attention map, and then directly multiplies it with the original feature map to complete the spatial information calibration; Finally, sum the feature maps generated by cSE and sSE to form a new attention feature map.

Images