Pig sound event detection method and device based on channel attention and residual gated convolution

By extracting the depth characteristics of pig sound data based on channel attention and residual gating convolution, the problem of high misjudgment rate and time-consuming and labor-consuming for pig epidemic detection in the prior art is solved. It realizes fast and accurate detection of pig sound events, and supports automated monitoring of pig epidemics.

CN114937461BActive Publication Date: 2024-12-03SOUTH CHINA AGRICULTURAL UNIVERSITY +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210660972.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-13
Publication Date
2024-12-03
Estimated Expiration
2042-06-13

AI Technical Summary

Technical Problem

The existing methods for detecting pig epidemics rely on human observation, have a high misjudgment rate, are time-consuming and labor-intensive, and are prone to pig stress and disease spread, and lack automated and informative monitoring methods.

Method used

The pig sound event detection method based on channel attention and residual gating convolution is adopted. By collecting and preprocessing sound data, the residual gating convolution module and channel attention module are used to extract depth features, combined with bidirectional cyclic recursion-form The feed neural network performs frame-level classification and outputs the time-domain distribution results of pig sound events.

Benefits of technology

It realizes fast and accurate detection of pig sound events, reduces labor costs, provides the ability to monitor pig abnormal status in real time, and enhances the efficiency of early detection and treatment of epidemics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114937461B_ABST
    Figure CN114937461B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for detecting pig sound events based on channel attention and residual gated convolution. The method includes: firstly collecting pig sound data and preprocessing the sound data; then using fast Fourier transform to obtain a spectrogram, and using spectral subtraction to remove environmental noise; then using a single-parameter double-threshold endpoint detection method to detect and crop the segments containing pig sounds in the spectrogram, and converting these segments into mel spectrograms; next, constructing a deep feature extractor based on channel attention and residual gated convolution to extract the sound deep features based on the mel spectrogram; finally, designing a frame-level sound classifier based on a bidirectional recurrent feedforward neural network to process and analyze the deep features of the sound, thereby presenting the time domain distribution results of the pig sound events of the entire audio segment. The present invention can accurately and quickly extract the time domain distribution results of pig sound events in a long-term, multi-noise farm environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of audio detection and sound recognition, and in particular relates to a method and device for detecting pig sound events based on channel attention and residual gated convolution. Background Art

[0002] Animal diseases are frequent and the risks of internal and external disease spread have become a major challenge to the development of animal husbandry. Existing methods for detecting pig diseases mainly rely on human detection, close observation, and experience. These methods have a high misjudgment rate, are time-consuming and labor-intensive, and are prone to cause stress reactions in pigs and the spread of diseases. Sound signals are a representation of the state of pigs and a way for pigs to communicate with each other. They include feedback information about the state and needs of the internal body of pigs, which contains rich information. Experienced animal husbandry workers can understand the current condition of pigs through the calls of pigs. For example, when pig diseases occur, pigs will produce sound changes due to changes in body temperature, respiratory infections, and other reasons. For example, blue ear disease, swine influenza, pseudorabies, porcine bronchitis and other diseases can cause pigs to cough and wheeze. Monitoring the sound events of coughing, wheezing, roaring and other common symptomatic symptoms of many diseases through audio detection technology is conducive to the early detection and early treatment of diseases, and is an effective means to achieve automation and informatization of disease monitoring. Summary of the invention

[0003] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method and device for detecting pig sound events based on channel attention and residual gated convolution. The present invention demonstrates how to perform sound event detection in audio data based on channel attention and residual gated convolution modules, and can accurately and quickly extract the time domain distribution results of pig sound events in a long-term, noisy farm environment.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention provides a pig sound event detection method based on channel attention and residual gated convolution, comprising the following steps:

[0006] Collect pig sound data in the farm, and mark the pig sound event categories and start and end times in some pig sound data to form a labeled standard data set;

[0007] Preprocessing the sound signal in the standard data set to obtain a preprocessed spectrogram;

[0008] Extracting segments containing pig sounds from the preprocessed spectrograms and converting these segments into mel-spectrograms;

[0009] A sound deep feature extractor is constructed, and the Mel spectrum map is input into the deep feature extractor for feature extraction to obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and utilize the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields;

[0010] A frame-level sound classifier is designed based on a bidirectional recursive-feedforward neural network. The frame-level sound classifier is used to perform frame-level classification on the deep feature map to obtain a sound event label for each frame, and finally the time-domain distribution result of the pig sound event is output. The time-domain distribution result of the pig sound event contains the event category and start and end time of the pig sound.

[0011] As a preferred technical solution, the preprocessing of the sound signal in the standard data set is specifically as follows:

[0012] Resample the sound signal to unify the sound signal to the same sampling rate;

[0013] Pre-emphasize the resampled sound signal and compensate for the high-frequency components of the audio signal

[0014] Performing short-time discrete Fourier transform on the pre-emphasized sound signal to obtain a spectrum diagram of the sound signal;

[0015] Denoise the spectrogram using spectral subtraction.

[0016] As a preferred technical solution, the extraction of segments containing pig sounds from the preprocessed spectrogram and conversion of these segments into mel spectrograms is specifically as follows:

[0017] The single-parameter double-threshold endpoint detection method based on short-time energy calculates the short-time energy of each frame of the sound signal, determines whether the frame contains pig sound according to the short-time energy, cuts the spectrum corresponding to the frame containing pig sound, and obtains the segment containing pig sound in the spectrum; the pig sound signal y(n) of the vth frame of the pig sound signal y v (n), with short-term energy E v The calculation formula is:

[0018]

[0019] Wherein, L represents the frame length, and n represents the sampling point number of the pig sound sample; in the single-parameter dual-threshold endpoint detection method: the single parameter is the short-time energy after amplitude normalization; the dual threshold is to set two different thresholds as thresholds, wherein the low threshold determines whether the frame is a sound frame, and the high threshold determines whether it contains pig sound; when the short-time energy exceeds the high threshold, it is regarded as the beginning of the frame containing pig sound, and when it is lower than the low threshold, it is regarded as the end of the frame containing pig sound;

[0020] The spectrum image segment containing pig sounds is passed through the Mel filter bank to obtain the Mel spectrum image. The conversion formula is:

[0021]

[0022] In the formula, f represents the original frequency of the audio data, and m represents the converted Mel frequency.

[0023] As a preferred technical solution, the residual gated convolution module includes: a plurality of residual gated convolution units, each unit including a gated convolution layer and a residual skip layer; for the lth layer of residual gated convolution unit, the following two steps are performed:

[0024] The first step is to transform the feature map X l The convolution modules U and V in the gated convolution layer are input respectively; secondly, they are activated by the sigmoid function and the linear function respectively to obtain the sound feature map and the attention mask; thirdly, the sound feature map and the attention mask are element-by-element multiplied to obtain the output Y of the lth gated convolution layer. l , the calculation formula is as follows:

[0025] Y l =(U*X l +b)⊙σ(V*X l +c)

[0026] Where b and c are offsets, σ ​​is the sigmoid function calculation, * is the convolution operation, and ⊙ is the Hadamard product operation;

[0027] The second step is to use the output R of the lth layer residual gated convolution unit l The output Y of the lth gated convolutional layer l With the input feature map X l Add together to form a residual skip layer connection. The expression of this step is:

[0028]

[0029] As a preferred technical solution, the channel attention module includes: a global maximum pooling, an average pooling layer, and a two-layer neural network; the execution steps of the channel attention module are as follows:

[0030] First, the output R of the l-th layer residual gated convolution module l The global average pooling layer and the global maximum pooling layer are input respectively, and the global average operation and the global spatial maximum value operation are performed on each channel of the feature map, so as to extract the global features of each channel of the feature map and form a global information embedding; then the channel global features are input into a two-layer neural network, and the parameters of the two-layer neural network are shared. The activation function of the first layer network is the ReLu function, and the activation function of the second layer uses the sigmoid function, so as to limit the output features to the range of [0,1], and obtain the sound depth feature map;

[0031] The expression of the global average pooling layer is:

[0032]

[0033] Among them, W is the length of the feature map, H is the width of the feature map, r c (i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, AP l is the global average information embedding of the lth layer;

[0034] The expression of the global maximum pooling layer is:

[0035]

[0036] Among them, R WH represents a feature map with a length of W and a width of H, r c (i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, MP l Globally maximize information embedding for the lth layer;

[0037] In summary, the expression of the entire process of the channel attention module is:

[0038] M = σ(MLP(AvgPool(R WH ))+MLP(MaxPool(R WH )))

[0039] Wherein, M is the output of the channel attention module, AvgPool is the global average pooling operation, MLP is the fully connected neural network, and MaxPool is the global maximum pooling operation.

[0040] As a preferred technical solution, the frame-level sound classifier is designed based on a bidirectional recurrent feedforward neural network, specifically:

[0041] The sound depth feature is input into the bidirectional recursive module to obtain the label probability vector of the sound at each moment; then the output value is input into the three-layer fully connected layer to obtain the sound event label corresponding to each frame; finally, the average value of the time domain label within a certain period of time is calculated to obtain the pig sound event label corresponding to the period of time, thereby outputting the time domain distribution of the pig sound event; for time t, the bidirectional recursive module performs forward calculation and reverse calculation at the same time, considering the past deep feature information and the future deep feature information, then at a certain time t, the probability vector h output by the lth layer is t (l) The expression is:

[0042]

[0043]

[0044]

[0045] in, Respectively represent the forward output value and reverse output value of the lth hidden layer at time t, W (l) represents the weight matrix of the recurrent neuron in the lth layer, and represents the forward weight matrix and reverse weight matrix of the hidden state of the recurrent neuron in the lth layer, and represents the forward offset and reverse offset of the recurrent neuron in the lth layer, Controlled from Hide state to The hidden state calculation, Controlled from Hide state to The calculation of the hidden state is Do the connection operation, the result is h t (l) ; For a certain time t, the sound event label is expressed as:

[0046] O′(t)=O(t)⊙Z loc (t)

[0047] Where ⊙ represents the Hadamard product, Z loc (t) represents the time domain position vector at time t, O(t) represents the result of the probability vector after passing through the feedforward neural network classifier, and O ′ (t) is the classification result at time t; the audio label in a certain time period T is determined by the average value of all frames in this time period, so the label result 0″ in a certain time period corresponds to the expression:

[0048]

[0049] As a preferred technical solution, the sound depth feature extractor is connected to the frame-level sound classifier to form an end-to-end network model. The end-to-end network model training process uses binary cross entropy as the loss function, which is expressed as:

[0050]

[0051] Among them, E is the binary cross entropy loss function, O k and P k They represent the estimated value and label value at sample index k respectively, and the total amount of data is K.

[0052] On the other hand, the present invention provides a pig sound event detection system based on channel attention and residual gated convolution, including a data acquisition module, a preprocessing module, a conversion module, a feature extraction module and a detection module;

[0053] The data collection module is used to collect pig sound data in the farm, and mark the pig sound event categories and start and end times in some pig sound data to form a labeled standard data set;

[0054] The preprocessing module is used to preprocess the sound signal in the standard data set to obtain a preprocessed spectrogram;

[0055] The conversion module is used to extract the segments containing pig sounds from the preprocessed spectrogram and convert these segments into Mel spectrograms.

[0056] The feature extraction module is used to construct a sound deep feature extractor, input the Mel spectrum map into the deep feature extractor for feature extraction, and obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and use the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields;

[0057] The detection module is used to design a frame-level sound classifier based on a bidirectional recursive-feedforward neural network, use the frame-level sound classifier to perform frame-level classification on the deep feature map, obtain the sound event label of each frame, and finally output the time domain distribution result of the pig sound event, which contains the event category and start and end time of the pig sound.

[0058] Another aspect of the present invention provides an electronic device, the electronic device comprising:

[0059] at least one processor; and,

[0060] a memory communicatively connected to the at least one processor; wherein,

[0061] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the pig sound event detection method based on channel attention and residual gated convolution.

[0062] On the other hand, the present invention provides a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, the method for detecting pig sound events based on channel attention and residual gated convolution is implemented.

[0063] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0064] 1. The present invention proposes a pig sound event detection method based on channel attention and residual gated convolution. Compared with the method of judging the state of pigs by manually listening to the sounds of pigs, it saves a lot of analysis time and reduces labor costs. The abnormal state of pigs can be monitored in real time through sound.

[0065] 2. The present invention preprocesses the sound signals in the pig sound data set by means of resampling, pre-emphasis, short-time discrete Fourier transform, etc., obtains a spectrum, and uses spectral subtraction to remove noise. The unique acoustic characteristics of the pig sound signal are enhanced, and the acoustic signal distributed in the time domain is converted into a spectrum, reducing the noise in the measured environment - the farm. This is conducive to the acoustic modeling of subsequent pig sound events.

[0066] 3. The present invention proposes a sound deep feature extractor based on a multi-channel attention mechanism and residual gated convolution to extract deep features of pig sound data. Compared with the current convolution-based sound deep feature extractor, the deep features extracted by the feature extractor of the present invention contain richer and more effective acoustic characteristics of pigs, control the amount of data in the network, pay attention to the relationship between feature map channels, make full use of the interdependence between channels of the deep feature map, and adaptively calibrate the feature responses under different receptive fields.

[0067] 4. According to the deep features of pig sound data, the present invention designs a frame-level sound classifier based on a bidirectional recursive-feedforward neural network to obtain the sound event label of each frame of pig sound, with fast classification speed and high accuracy.

[0068] 5. The present invention can generate a time domain distribution diagram of pig sound events based on the pig sound event detection results, which contains rich temporal evolution information of pig sound events, can reflect the temporal evolution of the pig's state, and provide a reference for detecting the pig's physiological state. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 is a flow chart of a pig sound event detection method based on channel attention and residual gated convolution according to an embodiment of the present invention;

[0071] Figure 2 Schematic diagram of a model of a channel attention mechanism according to an embodiment of the present invention;

[0072] Figure 3 It is a schematic diagram of extracting sound depth features by a sound depth feature extractor based on channel attention and residual gated convolution according to an embodiment of the present invention;

[0073] Figure 4 It is a schematic diagram of a bidirectional recurrent-feedforward neural network detecting the time domain distribution of a sound event based on a sound depth feature according to an embodiment of the present invention;

[0074] Figure 5 It is a block diagram of a pig sound event detection system based on channel attention and residual gated convolution according to an embodiment of the present invention;

[0075] Figure 6 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0077] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0078] The present application is a method for detecting pig sound events based on channel attention and residual gated convolution. The difficulty lies in constructing a sound deep feature extractor based on channel attention and residual gated convolution, and using the residual gated convolution module to extract the deep features of the Mel spectrum map in spatial distribution, guiding the network to learn the high-level semantic information of the audio. The multi-channel feature map extracted by the residual gated convolution module will enter the channel attention module, making full use of the interdependence between channels in the deep feature map, and adaptively calibrating the feature responses under different receptive fields; based on a bidirectional recurrent recursive-feedforward neural network, a frame-level sound classifier is designed. The deep feature map is classified at the frame level to obtain the sound event label of each frame. The network takes the sound deep feature as input, outputs the label probability vector of the pig sound event at each moment, and finally outputs the time domain distribution map of the pig sound event.

[0079] like Figure 1 As shown, the pig sound event detection method based on channel attention and residual gated convolution in this embodiment includes the following steps:

[0080] (1) Collect sound data of pigs in the farm, and annotate the categories and start and end times of four events, namely, coughing, panting, roaring, and snoring, in some of the sound data to form a labeled standard data set and an unlabeled data set to be tested;

[0081] (2) Preprocess the sound signals in the data set, including resampling, pre-emphasis, and short-time discrete Fourier transform of the sound signals to obtain a spectrum diagram, and use spectral subtraction to denoise; specifically:

[0082] (2-1) The sound signal is resampled to facilitate unified audio processing; this embodiment uses 16kHz as the standard sampling rate, interpolates the audio below 16kHz, and performs decimation on the audio above 16kHz, thereby unifying the sound signal to the same sampling rate.

[0083] (2-2) Pre-emphasis processing is performed on the resampled sound signal to compensate for the high-frequency components of the audio signal. Since the sound signal of the pig is pronounced through the lips, the signal frequency is affected by the glottal excitation and the radiation of the mouth and nose, and the high-frequency band is severely attenuated. Therefore, pre-emphasis processing is used to compensate for the high-frequency components of the audio signal.

[0084] Furthermore, a one-stage FIR high-pass digital filter is used to restore the high-frequency part, and its expression is:

[0085] y(n)=x(n)-αx(n-1)

[0086] In the formula, x(n) is the sound sampling value at time n, and the result after pre-emphasis processing is y(n), where a∈(0.9,1.0) is the pre-emphasis coefficient.

[0087] (2-3) Performing a short-time discrete Fourier transform on the pre-emphasized sound signal, including framing and windowing the sound signal, and performing a fast Fourier transform on each frame of the signal; this process utilizes the short-time stationary nature of the audio signal to divide the audio signal into multiple short segments, each of which is called a frame. There is a certain overlap between adjacent frames to ensure the continuity of adjacent frames; the length of each frame is called the frame length, and the length of the non-overlapping part between adjacent frames is called the frame shift; a Hamming window is used to perform a windowing operation on each frame to reduce spectral leakage; finally, a fast Fourier transform is performed on each frame to obtain a spectrum diagram of the sound signal.

[0088] (2-4) Spectral subtraction is used to denoise the spectrum. This method can effectively remove additive noise and is suitable for the most common steady or slowly changing environmental noise in farms. The frequency domain distribution of the noise is quite different from the frequency domain distribution of the sound made by pigs.

[0089] (3) Extract the segments containing pig sounds from the preprocessed spectrogram and convert them into Mel-spectrograms

[0090] In this embodiment, step (3) uses a single-parameter dual-threshold endpoint detection method based on short-time energy to detect and crop the preprocessed spectrum graph, specifically:

[0091] (3-1) The single-parameter double-threshold endpoint detection method based on short-time energy calculates the short-time energy of each frame of sound signal to determine whether the frame contains pig sound. For the pig sound signal y(n) of the vth frame, the pig sound signal y v (n), with short-term energy E v The calculation formula is:

[0092]

[0093] Where L represents the frame length, and n represents the sampling point number of the pig sound sample.

[0094] Furthermore, the single parameter in the single-parameter dual-threshold endpoint detection method based on short-time energy is the short-time energy after amplitude normalization, and the dual threshold is to set two different thresholds as thresholds. The low threshold determines whether the frame is a sound frame, and the high threshold determines whether it contains pig sounds. The algorithm starts as a frame containing pig sounds when the short-time energy exceeds the high threshold, and ends as a frame containing pig sounds when it is lower than the low threshold. The spectrogram corresponding to the frame containing pig sounds is cropped to obtain the fragment containing pig sounds in the spectrogram.

[0095] (3-2) Mel-spectrogram is a nonlinear spectrum obtained by passing the spectrum through the Mel filter group. Its conversion process is based on the logarithmic spectrum represented by the Mel scale and the linear cosine transform. It can sensitively capture the changes in the low-frequency band of the spectrum and is insensitive to the changes in the high-frequency band, which is in line with the characteristics of the human auditory system.

[0096] The specific steps are: pass the spectrum image segment containing pig sound through the Mel filter bank to obtain the Mel spectrum image, and the conversion formula is:

[0097]

[0098] In the formula, f represents the original frequency of the audio data, and m represents the converted Mel frequency.

[0099] (4) Construct a sound deep feature extractor, input the Mel spectrum map into the deep feature extractor for feature extraction, and obtain a deep feature map; Figure 2 As shown, the sound deep feature extractor includes a residual gated convolution module and a channel attention module. The residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and utilize the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields;

[0100] Further, such as Figure 3 As shown, in the sound depth feature extractor:

[0101] (4-1) The structure of a single residual gated convolution module includes: multiple residual gated convolution units, each unit contains a gated convolution layer and a residual skip layer. For the lth layer of the gated convolution unit, two steps are performed in the lth layer of the gated convolution unit:

[0102] The first step, feature map X l The convolution modules U and V in the gated convolution layer are input respectively; then, they are activated by the sigmoid function and the linear function respectively to obtain the sound feature map and the attention mask; the above two are then element-by-element multiplication to obtain the output Y of the gated convolution unit of the lth layer l .

[0103] Furthermore, the calculation formula of the gated convolution unit is:

[0104] Y l =(U*X l +b)⊙σ(V*X l +c)

[0105] Where b and c are offsets, σ ​​is the sigmoid function calculation, * is the convolution operation, and ⊙ is the Hadamard product operation;

[0106] In the second step, the lth layer residual gated convolution unit outputs R l The output of the gated convolutional unit at layer l is Y l With input feature X l Add together to form a residual skip layer connection, which is expressed as:

[0107]

[0108] (4-2) The structure of a single channel attention module includes: a global maximum pooling layer, an average pooling layer, and a two-layer neural network.

[0109] In this embodiment, the output R of the l-th layer residual gating module is firstly l , respectively input into the global average pooling layer and the global maximum pooling layer, perform global average operation and take the global spatial maximum value operation on each channel of the feature map, so as to extract the global features of each channel of the feature map and form global information embedding; then input the channel global features into the two-layer neural network. The two-layer neural network parameters are shared, the activation function of the first layer network is the ReLu function, and the activation function of the second layer uses the sigmoid function, so as to limit the output features to the range of [0,1] and obtain the sound depth feature map. The expression of the global average pooling layer is:

[0110]

[0111] Among them, W is the length of the feature map, H is the width of the feature map, r c (i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, AP l is the global average information embedding of the lth layer;

[0112] The expression of the global maximum pooling layer is:

[0113]

[0114] Among them, R WH represents a feature map with a length of W and a width of H, r c(i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, MP l Globally maximize the information embedding for the lth layer.

[0115] In summary, the expression of the entire module process is:

[0116] M = σ (MLP (AvgPool (R WH ))+MLP(MaxPool(R WH )))

[0117] Wherein, M is the output of the channel attention module, AvgPool is the global average pooling operation, MLP is the fully connected neural network, and MaxPool is the global maximum pooling operation.

[0118] (5) A frame-level sound classifier is designed based on a bidirectional recursive-feedforward neural network, and the frame-level sound classifier is used to perform frame-level classification on the deep feature map to obtain a sound event label for each frame, and finally output a time domain distribution result of the pig sound event, wherein the time domain distribution result of the pig sound event includes the event category and start and end time of the pig sound.

[0119] like Figure 4 As shown, step (5) is specifically as follows:

[0120] First, input the sound deep features into the bidirectional recursive module to obtain the label probability vector of the sound at each moment; then input the output value into the three-layer fully connected layer to obtain the sound event label corresponding to each frame; finally, calculate the average value of the time domain label within a certain period of time to obtain the pig sound event label corresponding to the period of time, and then output the time domain distribution of the pig sound event. At time t, the bidirectional recursive module performs forward and reverse calculations at the same time, taking into account the past deep feature information and the future deep feature information. At a certain time t, the probability vector h output by the lth layer is t (l) The expression is:

[0121]

[0122]

[0123]

[0124] in, Respectively represent the forward output value and reverse output value of the lth hidden layer at time t, W (l) represents the weight matrix of the recurrent neuron in the lth layer, and represents the forward weight matrix and reverse weight matrix of the hidden state of the recurrent neuron in the lth layer, and represents the forward offset and reverse offset of the recurrent neuron in the lth layer, Controlled from Hide state to The hidden state calculation, Controlled from Hide state to The calculation of the hidden state is Do the connection operation, the result is h t (l) ; For a certain time t, the sound event label is expressed as:

[0125] O′(t)=O(t)⊙Z loc (t)

[0126] Where ⊙ represents the Hadamard product, Z loc (t) represents the time domain position vector at time t, O(t) represents the result after the probability vector passes through the feedforward neural network classifier, and O′(t) is the classification result at time t. The audio label within a certain time period T is determined by the average value of all frames within the time period, so the label result 0″ within a certain time period corresponds to the expression:

[0127]

[0128] (6) The sound deep feature extractor is connected to the frame-level sound event classifier to form an end-to-end network model, and training is performed simultaneously. The training process uses binary cross entropy as the loss function, and its corresponding expression is:

[0129]

[0130] Where E is the binary cross entropy loss function, O k and P k They represent the estimated value and label value at sample index k respectively, and the total amount of data is K.

[0131] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0132] Based on the same idea as the pig sound event detection method based on channel attention and residual gated convolution in the above-mentioned embodiment, the present invention also provides a pig sound event detection system based on channel attention and residual gated convolution, which can be used to execute the above-mentioned pig sound event detection method based on channel attention and residual gated convolution. For ease of explanation, the structural schematic diagram of the embodiment of the pig sound event detection system based on channel attention and residual gated convolution only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated structure, or combine certain components, or arrange the components differently.

[0133] See also Figure 5 In another embodiment of the present application, a pig sound event detection system 100 based on channel attention and residual gated convolution is provided, the system comprising a data acquisition module 101, a preprocessing module 102, a conversion module 103, a feature extraction module 104 and a detection module 105;

[0134] The data collection module 101 is used to collect pig sound data in the farm, and mark the pig sound event categories and start and end times in part of the pig sound data to form a labeled standard data set and an unlabeled data set to be tested.

[0135] The preprocessing module 102 is used to preprocess the sound signal in the standard data set to obtain a preprocessed spectrogram;

[0136] The conversion module 103 is used to extract the segments containing pig sounds from the preprocessed spectrogram and convert these segments into Mel-spectrograms.

[0137] The feature extraction module 104 is used to construct a sound deep feature extractor, input the Mel spectrum map into the deep feature extractor for feature extraction, and obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and use the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields;

[0138] The detection module 105 is used to design a frame-level sound classifier based on a bidirectional recursive-feedforward neural network, use the frame-level sound classifier to perform frame-level classification on the deep feature map, obtain the sound event label of each frame, and finally output the time domain distribution result of the pig sound event, which contains the event category and start and end time of the pig sound.

[0139] It should be noted that the pig sound event detection system based on channel attention and residual gated convolution of the present invention corresponds one to one with the pig sound event detection method based on channel attention and residual gated convolution of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the pig sound event detection method based on channel attention and residual gated convolution are applicable to the embodiment of pig sound event detection based on channel attention and residual gated convolution. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here.

[0140] In addition, in the implementation of the pig sound event detection system based on channel attention and residual gated convolution in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the pig sound event detection system based on channel attention and residual gated convolution is divided into different program modules to complete all or part of the functions described above.

[0141] See also Figure 6 In one embodiment, an electronic device for implementing a pig sound event detection method based on channel attention and residual gated convolution is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a pig sound event detection program 203.

[0142] The first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 may also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 may not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the pig sound event detection program 203, but also be used to temporarily store data that has been output or is to be output.

[0143] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0144] Figure 6 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0145] The pig sound event detection program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve: collecting pig sound data in the farm, and marking the pig sound event categories and start and end times in part of the pig sound data to form a labeled standard data set and an unlabeled data set to be detected;

[0146] Preprocessing the sound signal in the standard data set to obtain a preprocessed spectrogram;

[0147] Extracting segments containing pig sounds from the preprocessed spectrograms and converting these segments into mel-spectrograms;

[0148] A sound deep feature extractor is constructed, and the Mel spectrum map is input into the deep feature extractor for feature extraction to obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and utilize the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields;

[0149] A frame-level sound classifier is designed based on a bidirectional recursive-feedforward neural network. The frame-level sound classifier is used to perform frame-level classification on the deep feature map to obtain a sound event label for each frame, and finally the time-domain distribution result of the pig sound event is output. The time-domain distribution result of the pig sound event contains the event category and start and end time of the pig sound.

[0150] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A pig sound event detection method based on channel attention and residual gated convolution, characterized in that: The steps include: Collect pig sound data in the farm, and mark the pig sound event categories and start and end times in some pig sound data to form a labeled standard data set; Preprocessing the sound signal in the standard data set to obtain a preprocessed spectrogram; Extracting segments containing pig sounds from the preprocessed spectrograms and converting these segments into mel-spectrograms; A sound deep feature extractor is constructed, and the Mel spectrum map is input into the deep feature extractor for feature extraction to obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and utilize the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields; A frame-level sound classifier is designed based on a bidirectional recursive-feedforward neural network, and the frame-level sound classifier is used to perform frame-level classification on the deep feature map to obtain a sound event label for each frame, and finally output a time-domain distribution result of a pig sound event, wherein the time-domain distribution result of the pig sound event includes an event category and a start and end time of the pig sound; The residual gated convolution module includes: a plurality of residual gated convolution units, each unit including a gated convolution layer and a residual skip layer; for the lth layer of residual gated convolution unit, the following two steps are performed: The first step is to transform the feature map X l The convolution modules U and V in the gated convolution layer are input respectively; secondly, they are activated by the sigmoid function and the linear function respectively to obtain the sound feature map and the attention mask; thirdly, the Hadamard product of the sound feature map and the attention mask is calculated to obtain the output Y of the lth gated convolution layer. l , the calculation formula is as follows: Y l =(U*X l +b)⊙σ(V*X l +c) Where b and c are offsets, σ ​​is the sigmoid function calculation, * is the convolution operation, and ⊙ is the Hadamard product operation; The second step is to use the output R of the lth layer residual gated convolution unit l The output Y of the lth gated convolutional layer l With the input feature map X l Add together to form a residual skip layer connection. The expression of this step is: The channel attention module includes: a global maximum pooling, an average pooling layer, and a two-layer neural network; the execution steps of the channel attention module are as follows: First, the output R of the l-th layer residual gated convolution module l The global average pooling layer and the global maximum pooling layer are input respectively, and the global average operation and the global spatial maximum value operation are performed on each channel of the feature map, so as to extract the global features of each channel of the feature map and form a global information embedding; then the channel global features are input into a two-layer neural network, and the parameters of the two-layer neural network are shared. The activation function of the first layer network is the ReLu function, and the activation function of the second layer uses the sigmoid function, so as to limit the output features to the range of [0,1], and obtain the sound depth feature map; The expression of the global average pooling layer is: Among them, W is the length of the feature map, H is the width of the feature map, r c (i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, AP l is the global average information embedding of the lth layer; The expression of the global maximum pooling layer is: Among them, R WH represents a feature map with a length of W and a width of H, r c (i, j) is the pixel value of the feature map at coordinate (i, j) on channel c, MP l Globally maximize information embedding for layer l; In summary, the expression of the entire process of the channel attention module is: M=σ(MLP(AvgPool(R WH ))+MLP(MaxPool(R WH ))) Wherein, M is the output of the channel attention module, AvgPool is the global average pooling operation, MLP is the fully connected neural network, and MaxPool is the global maximum pooling operation.

2. According to claim 1, the pig sound event detection method based on channel attention and residual gated convolution is characterized in that: The preprocessing of the sound signal in the standard data set is specifically: Resample the sound signal to unify the sound signal to the same sampling rate; Pre-emphasize the resampled sound signal and compensate for the high-frequency components of the audio signal Performing short-time discrete Fourier transform on the pre-emphasized sound signal to obtain a spectrum diagram of the sound signal; Denoise the spectrogram using spectral subtraction.

3. According to claim 1, the pig sound event detection method based on channel attention and residual gated convolution is characterized in that: The extracting of the segments containing pig sounds from the preprocessed spectrogram and converting these segments into Mel spectrograms is specifically as follows: The single-parameter double-threshold endpoint detection method based on short-time energy calculates the short-time energy of each frame of the sound signal, determines whether the frame contains pig sound according to the short-time energy, cuts the spectrum corresponding to the frame containing pig sound, and obtains the segment containing pig sound in the spectrum; the pig sound signal y(n) of the vth frame of the pig sound signal y v (n), with short-term energy E v The calculation formula is: In the formula, L represents the frame length, and n represents the sampling point number of the pig sound sample; In the single-parameter dual-threshold endpoint detection method: the single parameter is the short-time energy after amplitude normalization; the dual threshold is to set two different thresholds as thresholds, wherein the low threshold determines whether the frame is a sound frame, and the high threshold determines whether it contains pig sound; When the short-time energy exceeds the high threshold, the frame containing the pig sound begins, and when it is lower than the low threshold, the frame containing the pig sound ends; The spectrum image segment containing pig sounds is passed through the Mel filter bank to obtain the Mel spectrum image. The conversion formula is: In the formula, f represents the original frequency of the audio data, and m represents the converted Mel frequency.

4. According to claim 1, the pig sound event detection method based on channel attention and residual gated convolution is characterized in that: The frame-level sound classifier is designed based on a bidirectional recurrent feedforward neural network, specifically: The sound deep features are input into the bidirectional recurrent module to obtain the label probability vector of the sound at each moment; then the output value is input into the three-layer fully connected layer to obtain the sound event label corresponding to each frame; Finally, the average value of the time domain label within a certain period is calculated to obtain the pig sound event label corresponding to the period, thereby outputting the time domain distribution of the pig sound event; for time t, the bidirectional recursive module performs forward calculation and reverse calculation at the same time, considering the past deep feature information and the future deep feature information. At a certain time t, the probability vector h output by the lth layer is t (l) The expression is: in, Respectively represent the forward output value and reverse output value of the lth hidden layer at time t, W (l) represents the weight matrix of the recurrent neuron in the lth layer, and represents the forward weight matrix and reverse weight matrix of the hidden state of the recurrent neuron in the lth layer, and represents the forward offset and reverse offset of the recurrent neuron in the lth layer, Controlled from Hide state to The hidden state calculation, Controlled from Hide state to The calculation of the hidden state is Do the connection operation, the result is h t (l) ; For a certain time t, the sound event label is expressed as: O ′ (t)=O(t)⊙Z loc (t) Where ⊙ represents the Hadamard product, Z loc (t) represents the time domain position vector at time t, O(t) represents the result after the probability vector passes through the feedforward neural network classifier, and O′(t) is the classification result at time t; the audio label within a certain time period T is determined by the average value of all frames within the time period, so the label result 0″ within a certain time period corresponds to the expression:

5. According to claim 1, the pig sound event detection method based on channel attention and residual gated convolution is characterized in that: The sound depth feature extractor is connected to the frame-level sound classifier to form an end-to-end network model. The end-to-end network model training process uses binary cross entropy as the loss function, which is expressed as: Among them, E is the binary cross entropy loss function, O k and P k They represent the estimated value and label value at sample index k respectively, and the total amount of data is K.

6. A pig sound event detection system based on channel attention and residual gated convolution, characterized in that: A pig sound event detection method based on channel attention and residual gated convolution applied to any one of claims 1-5, comprising a data acquisition module, a preprocessing module, a conversion module, a feature extraction module and a detection module; The data collection module is used to collect pig sound data in the farm, and mark the pig sound event categories and start and end times in some pig sound data to form a labeled standard data set; The preprocessing module is used to preprocess the sound signal in the standard data set to obtain a preprocessed spectrogram; The conversion module is used to extract the segments containing pig sounds in the preprocessed spectrogram and convert these segments into mel spectrograms; The feature extraction module is used to construct a sound deep feature extractor, input the Mel spectrum map into the deep feature extractor for feature extraction, and obtain a deep feature map; the sound deep feature extractor includes a residual gated convolution module and a channel attention module, the residual gated convolution module is used to extract the deep features of the Mel spectrum map in spatial distribution, and guide the network to learn the high-level semantic information of the sound; the channel attention module is used to process the multi-channel feature map extracted by the residual gated convolution module, and use the mutual dependence between the channels of the deep feature map to adaptively calibrate the feature responses under different receptive fields; The detection module is used to design a frame-level sound classifier based on a bidirectional recursive-feedforward neural network, use the frame-level sound classifier to perform frame-level classification on the deep feature map, obtain the sound event label of each frame, and finally output the time domain distribution result of the pig sound event, which contains the event category and start and end time of the pig sound.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the pig sound event detection method based on channel attention and residual gated convolution as described in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the pig sound event detection method based on channel attention and residual gated convolution described in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Sound scene classification method based on attention mechanism and double-path deep residual network

    CN111754988A

  • Acoustic event detection method based on pixel-based attention of capsule networks model

    CN113012714A