Baby cry classification method based on Transformer fusion model
By adopting a fusion model based on Transformer in infant crying classification, combining the spectrum map enhancement module and attention mechanism module, the problem of insufficient feature extraction and robustness in the existing technology is solved, and a more efficient fusion classification of infant crying is achieved performance.
Patent Information
- Application Number
- CN202210236093.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-11
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2042-03-11
AI Technical Summary
The existing infant crying classification method cannot adaptively obtain distinguishable features, ignores the information interaction between features inside and between the spectrogram feature channels, and in the case of limited labeling data, it is difficult to improve the robustness of the spectrogram features.
Using the infant crying classification method based on the Transformer fusion model, the robust feature representation is extracted through the spectrum enhancement module, and the spatial and channel attention mechanism module is used to extract the distinguishable feature representation inside the channel and between channels. Finally, Transformer is used to extract the The modules incorporate these feature representations to improve classification performance.
It effectively improves the robustness and accuracy of infant crying classification, and can extract more distinctive feature representations under limited data, which enhances the generalization ability of the model.
Smart Images
Figure CN114595772B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer speech technology, and in particular to a method for classifying infant crying sounds based on a Transformer fusion model. Background Art
[0002] Automatic baby cry classification is a crucial research field in bioengineering. It uses medical and engineering techniques to analyze the baby cry signals to distinguish the physiological and pathological conditions of baby cry. Unlike the verbal information of adult speech, it is difficult to identify what babies are trying to convey through their crying. Therefore, it is of great significance to study and design an effective baby cry classification model to efficiently obtain and identify the physiological and pathological conditions of baby cry.
[0003] Traditional baby cry classification models are mostly based on single-branch classification models. They are mainly divided into models based on traditional machine learning classifiers, including MLP, SVM, decision tree, etc., and models based on deep learning classifiers, including Resnet-50, migration of Resnet-50 combined with SVM model, graph convolution model, and R-CNN series methods. However, traditional machine learning classifiers have many limitations, such as small data scale, poor generalization ability, and difficulty in applying to complex and changeable actual scenarios. Combined with deep learning classifier methods, most of them only focus on deeply extracting complex baby cry feature representations, and fail to consider how to expand the limited baby cry samples and fully explore the distinguishable feature representations within the baby cry channel and between channels.
[0004] In summary, the existing classification of baby crying has the following problems:
[0005] (1) Existing baby cry classification methods are unable to adaptively obtain distinguishable features and ignore the information interaction of features within and between channels of the spectrum graph.
[0006] (2) Similarly, for the problem of limited labeled infant crying data, due to the sensitivity of infant crying data and the time-consuming nature of transcribing the original crying data, existing methods cannot effectively increase the robustness of spectral graph features. Summary of the invention
[0007] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a method for classifying baby cries based on the Transformer fusion model. From the perspective of information fusion, the present invention uses a spectrum enhancement module to extract robust feature representations to expand baby cry data; on the other hand, it uses a spatial and channel attention mechanism module to extract distinguishable feature representations within and between channels, and finally uses a Transformer module to fuse the feature representations of the first two to make full use of the fusion idea to further improve the classification performance.
[0008] Technical solution: A method for classifying infant crying sounds based on a Transformer fusion model of the present invention comprises the following steps:
[0009] Step (1), inputting the baby crying audio data to be classified, preprocessing the input audio data through the audio processing module, and generating a spectrogram;
[0010] Step (2), constructing a training model and preliminarily extracting spectrogram features;
[0011] The Resnet-50 pre-trained model is used as the base network. The parameters of the first 8 convolutional layers in the Resnet-50 model are fixed. Only the parameters of the last two layers are involved in the training process. Then two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are attached to the base network to obtain the training model.
[0012] Step (3), for the spectrogram features obtained in step (2), respectively extract robust feature representation, intra-channel distinguishable feature representation, and inter-channel distinguishable feature representation through the spectrogram enhancement module and the attention mechanism module;
[0013] Step (4), the two feature representations obtained in step (3) are fused through the Transformer fusion module; the fused feature map is then used for infant crying classification, and the final classification result is obtained through multiple iterative training.
[0014] Furthermore, the preprocessing in step (1) refers to converting audio file samples (including wav, pcm, mp3 and other format samples) into a spectrogram of size 256×256 through audio processing software (such as Sound Exchange, etc.), wherein the horizontal axis of the single-channel spectrogram represents time and the vertical axis represents frequency.
[0015] Furthermore, the method for constructing the training basic network model in step (2) is:
[0016] The Resnet-50 pre-trained model is used as the backbone network, and then the parameters of the first 8 convolutional layers in the Resnet-50 model are fixed, including the 8 groups of convolutional layers from Conv1 to Conv8 in Resnet-50. After the training process, the parameters of the last two layers are updated; then two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are appended to the backbone network to form the required basic network model for preliminary extraction of spectrogram features.
[0017] Furthermore, the specific method for extracting the robustness features of the spectrogram by the spectrogram enhancement module in step (3) is: performing mask operations on the time domain channel and the frequency domain channel of the spectrogram respectively, that is, setting two masked frequency domain channels, and the random value range is between 0 and 20; setting two masked time domain channels, and the random value range is between 10 and 30, because the crying sound of an infant does not have strong semantic information in the speech time sequence, the time distortion operation is removed.
[0018] Here, when training data is insufficient, the spectrogram enhancement module can be used to perform spectrum enhancement on audio data and dynamically expand the data set; spectrogram enhancement can not only help the neural network better learn the feature representation of the spectrogram, but also increase the robustness of the training network against time domain deformation and partial frequency domain segment loss, ultimately effectively improving the performance of the final audio classifier.
[0019] Furthermore, the attention mechanism module in step (3) includes a channel attention mechanism and a spatial attention mechanism, and the specific working process is as follows:
[0020] For the channel attention mechanism, global maximum pooling is first used for each feature map to aggregate channel information; then the feature descriptor generated by global maximum pooling is sent to the two-layer perceptron; formula (1) is as follows:
[0021] P c =M mlp (M max (P)), (1)
[0022] The final feature map is generated by multiplying the channel attention map and the original feature map; as shown in formula (2):
[0023]
[0024] Among them, M max (·) represents the global maximum pooling of feature maps between channels, M mlp (·) is a two-layer perceptron, P represents the input feature map, P c is the generated channel attention map, P f1 Represents the feature map generated by the channel attention mechanism.
[0025] For the spatial attention mechanism, the global maximum pooling of each point in the feature map is introduced, and then the feature descriptor generated by the global maximum pooling is sent to the two-layer perceptron; finally, the feature map is generated by multiplying the channel attention map and the original feature map. Formula (3) is as follows:
[0026]
[0027] Where M in formula (3) max(·) represents the global maximum pooling of each point in the feature map, P f2 Represents the feature map generated by the spatial attention mechanism.
[0028] The present invention can obtain more robust feature embedding through the dual-branch spectrogram enhancement module and the attention module compared to the single-branch spectrogram enhancement module.
[0029] Furthermore, the Transformer fusion module of step (4) does not set position encoding, and uses a layer of Transformer Block to fuse robust feature representation, distinguishable feature representation within the channel, and distinguishable feature representation between channels; the sentence token in the Transformer fusion module is set to 49, and the size of each token is 1×1×128 (w×h×c); finally, the final classification result is obtained after 2 layers of fully connected layers and softmax.
[0030] On the one hand, the present invention considers the learning of channel internal features, which is conducive to the information interaction between channels; on the other hand, based on the channel attention and spatial attention modules, it learns the distinguishable feature representation between channels. Therefore, Transformer is used here to fuse these two branch modules and organically integrate the learned features.
[0031] Furthermore, in the iterative training process of step (4), the stochastic gradient method is used for 200 iterations, the batch size of each iteration is set to 32, the model learning rate is set to 0.0001, and the result does not change after every ten iterations, and the final model is saved for model testing;
[0032] The model was trained and tested using a 5-fold cross validation method, and the final 5-fold average test accuracy was used as the classification result.
[0033] Beneficial effects: Under the problem of limited baby cry samples and how to effectively mine the distinguishable feature representation of baby cry, the present invention proposes a spectrum map enhancement using a random mask module combined with a spatial and channel attention mechanism to obtain a better feature representation for baby cry classification. At the same time, the Transformer mechanism is integrated to achieve the complementary effect between features and further improve the robustness of baby cry classification.
[0034] Compared with the prior art, the present invention has the following advantages:
[0035] (1) The training model constructed by the present invention is a new type of bilinear fusion network that mines audio spectrum features at multiple levels.
[0036] (2) The spectrogram enhancement module of the present invention extracts the robust feature representation of the spectrogram within the channel, which can increase the number of audios and dynamically expand the data set.
[0037] (3) The attention mechanism module of the present invention extracts distinguishable feature representations and fully exploits the distinguishable features between channels.
[0038] (4) The feature fusion module of the present invention can realize the complementary effect between features, highlight useful information, suppress redundant information, and further enhance the representation ability of feature graphs. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is the overall classification flow chart of the present invention;
[0040] Figure 2 A schematic diagram of a network model of an embodiment;
[0041] Figure 3 A sample audio spectrum diagram of an embodiment;
[0042] Figure 4 It is a characteristic matrix schematic diagram of an embodiment;
[0043] Figure 5 is a schematic diagram of the enhanced spectrum in the embodiment;
[0044] Figure 6 Schematic diagram of the classification result of baby crying types in the embodiment. DETAILED DESCRIPTION
[0045] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.
[0046] like Figure 1 As shown, a method for classifying infant crying sounds based on a Transformer fusion model of the present invention comprises the following steps:
[0047] Step (1), inputting the audio data of the baby crying sound to be classified, and using Sound Exchange audio processing software to convert the audio sample into a spectrogram of size 256×256, wherein the horizontal axis of the single-channel spectrogram represents time and the vertical axis represents frequency;
[0048] Step (2), constructing a training model and preliminarily extracting spectrogram features;
[0049] like Figure 2As shown in the figure, the Resnet-50 pre-trained model is used as the base network, and then the parameters of the first 8 convolutional layers in the Resnet-50 model are fixed. Only the parameters of the last two layers will participate in the training process. Then, two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are attached to the base network to obtain the training model.
[0050] Step (3), for the spectrogram features obtained in step (2), respectively extract robust feature representation, intra-channel distinguishable feature representation, and inter-channel distinguishable feature representation through the spectrogram enhancement module and the attention mechanism module;
[0051] Step (4), the two feature representations obtained in step (3) are fused through the Transformer fusion module; the fused feature map is then used for infant crying classification, and the final classification result is obtained through multiple iterative training.
[0052] Embodiment 1:
[0053] This embodiment includes the following steps:
[0054] Step (1): First, the training data is processed into a 256×256 spectrogram using the SOX tool as the input for the subsequent model. The audio parameter settings include: the sample rate is set to 44100 Hz, the sample size is 16 bits, the sample encoding is set to 16-bit (signed integer) signed integer PCM, and the number of channels is 3. The specific audio spectrogram sample is as follows: Figure 3 As shown;
[0055] Step (2): Use the Resnet-50 pre-trained model as the basic network and perform fine-tuning to build a training model. Specifically:
[0056] The Resnet-50 pre-trained model is used as the backbone network, and then the parameters of the first 8 convolutional layers in the Resnet-50 model are fixed, including the 8 groups of convolutional layers from Conv1 to Conv8 in Resnet-50. During the training process, the parameters of the last two layers are updated; then two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are added to the backbone network to form the required basic network model for preliminary extraction of spectrogram features and generate a 1024×32 feature matrix, such as Figure 4 shown.
[0057] Step (3), the extracted spectrum features are further optimized through two branches;
[0058] A spectrum enhancement module is used. In this embodiment, the setting of the distorted time domain signal is removed, and only the time domain channel and the frequency domain channel of the spectrum are masked; specifically, two masked frequency domain channels are set, and the random value range is between 0 and 20; two masked time domain channels are also set, and the random value range is between 10 and 30; after enhancing the spectrum of one of the spectrum graphs, such as Figure 5 shown.
[0059] The other one uses the spatial and channel attention modules. For the channel attention module, each spectrum map feature passes through a layer of global maximum pooling (MaxPooling) and a two-layer MLP network to obtain a channel attention feature map. After that, the channel attention feature map is multiplied with the original input feature, and the result is used as the input of the spatial attention module. In the spatial attention module, the global maximum pooling of each point in the feature map is introduced; then the feature descriptor generated by the global maximum pooling is sent to the two-layer perceptron; finally, the channel attention map is multiplied with the original feature map to generate a feature map.
[0060] Step (4), using the Transformer fusion module to fuse the two representations obtained above. In this embodiment, the position encoding setting in the Transformer module is removed, and a layer of Transformer Block is used to fuse the spectrum features optimized by the above two branches; wherein, the sentence block (token) is set to 49, and the size of each token is 1×1×128 (w×h×c); finally, after two layers of fully connected layers and softmax, the final classification result is obtained. After classifying one of the spectrum graphs, the result is as follows Figure 6 shown.
[0061] The classification results of this implementation include hungry, sleepy, and wakeup categories. The final classification detection of the baby's crying audio can determine the meaning of the baby's crying. For example, hungry means expressing hunger through crying, wakeup means waking up, and sleepy means sleepy, thereby knowing the baby's physiological needs.
[0062] In the iterative training process of this embodiment, the stochastic gradient method is used for 200 iterations, the batch size of each iteration is set to 32, the model learning rate is set to 0.0001, and the result does not change after every ten iterations. The final model is saved for model testing; the model is trained and tested using a 5-fold cross-validation method, and the final 5-fold average test accuracy is used as the classification result.
[0063] Embodiment 2:
[0064] In order to verify the rationality and effectiveness of the technical solution of the present invention, this embodiment selects a subset of the Baby2020 dataset for experiments, and uses acc (accuracy) as an objective evaluation indicator of the classification results. This embodiment is implemented based on the deep learning framework Pytorch, and uses an image processing unit (GPU) to accelerate calculations, 12GB of memory, and an Nvidia GeForce GTX2080Ti graphics card.
[0065] For the Baby2020 data subset, it contains 3713 samples of three types from healthy babies aged 0 to 3 months, of which 1058 samples are in the Hungry category, 1257 samples are in the Sleepy category, and 949 samples are in the Wakeup category. Among them, 2790 audio audios are used as training data and 743 audio audios are used as test data. Table 1 lists the experimental results of the present invention. The acc reaches 83.14%, and the classification performance is better than other similar methods, achieving effective classification of the target.
[0066] Table 1 Classification results of the Baby2020 data subset
[0067]
[0068] It can be seen from the above embodiments that the present invention can greatly improve the classification accuracy and efficiency of baby crying audio, making it easier to understand the needs expressed by the baby crying in a timely manner.
Claims
1. A method for classifying infant crying sounds based on a Transformer fusion model, characterized in that: The following steps are involved: Step (1), inputting the baby crying audio data to be classified, preprocessing the input audio data through the audio processing module, and generating a spectrogram; Step (2), constructing a basic network model and preliminarily extracting spectrum graph features; The Resnet-50 pre-trained model is used as the base network. The parameters of the first 8 convolutional layers in the Resnet-50 model are fixed. Only the parameters of the last two layers are involved in the training process. Then, two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are attached to the base network to obtain the base network model. Step (3), for the spectrogram features obtained in step (2), respectively extract robust feature representation, intra-channel distinguishable feature representation, and inter-channel distinguishable feature representation through the spectrogram enhancement module and the attention mechanism module; The specific method of extracting robust features of the spectrogram by the spectrogram enhancement module is as follows: masking operations are performed on the time domain channel and the frequency domain channel of the spectrogram respectively, that is, two masked frequency domain channels are set, and the random value range is between 0 and 20; two masked time domain channels are set, and the random value range is between 10 and 30; The attention mechanism module includes channel attention mechanism and spatial attention mechanism. The specific working process is as follows: For the channel attention mechanism, global maximum pooling is first used for each feature map to aggregate channel information; then the feature descriptor generated by global maximum pooling is sent to the two-layer perceptron; formula (1) is as follows: P c =M mlp (M max (P)), (1) The final feature map is generated by multiplying the channel attention map and the original feature map; as shown in formula (2): Among them, M max (·) represents the global maximum pooling of feature maps between channels, M mlp (·) is a two-layer perceptron, P represents the input feature map, P c is the generated channel attention map, P f1 Represents the feature map generated by the channel attention mechanism; For the spatial attention mechanism, the global maximum pooling of each point in the feature map is introduced, and then the feature descriptor generated by the global maximum pooling is sent to the two-layer perceptron; finally, the feature map is generated by multiplying the channel attention map and the original feature map. Formula (3) is as follows: Where M in formula (3) max (·) represents the global maximum pooling of each point in the feature map, P f2 Represents the feature map generated by the spatial attention mechanism; Step (4), the robust feature representation obtained in step (3), the distinguishable feature representation within the channel, and the distinguishable feature representation between channels are fused through the Transformer fusion module; the fused feature map is then used for infant cry classification, and the final classification result is obtained through multiple iterative training.
2. The method for classifying infant crying sounds based on the Transformer fusion model according to claim 1, characterized in that: The preprocessing in step (1) refers to converting the audio file samples into a spectrogram of size 256×256 through an audio processing module, wherein the horizontal axis of the spectrogram of a single channel represents time and the vertical axis represents frequency.
3. The method for classifying infant crying sounds based on the Transformer fusion model according to claim 1, characterized in that: The method of constructing the basic network model in step (2) is: The Resnet-50 pre-trained model is used as the basic network, and then the parameters of the first 8 convolutional layers in the Resnet-50 model are fixed. Only the parameters of the last two layers are involved in the training process. Then, two fully connected layers with 1024 neurons and a dropout layer with a rate of 0.7 are attached to the basic network to obtain the basic network model.
4. The method for classifying infant crying sounds based on the Transformer fusion model according to claim 1, characterized in that: The Transformer fusion module of step (4) does not set position encoding, and uses a layer of Transformer Block to fuse robust feature representation, distinguishable feature representation within the channel, and distinguishable feature representation between channels; The sentence tokens in the Transformer fusion module are set to 49, and the size of each token is 1×1×128 (w×h×c); finally, the final classification result is obtained through 2 layers of fully connected layers and softmax.
5. The method for classifying infant crying sounds based on the Transformer fusion model according to claim 1, characterized in that: In the iterative training process of step (4), the stochastic gradient method is used for 200 iterations, the batch size of each iteration is set to 32, the model learning rate is set to 0.0001, and the result does not change after every ten iterations. The final model is saved for model testing; The model was trained and tested using a 5-fold cross validation method, and the final 5-fold average test accuracy was used as the classification result.