Deep learning-based voice classification method and device, storage medium, and computer

Abstract

The present invention provides a sound classification method and device, a storage medium, and a computer. The sound classification method includes: providing a training set, a validation set, and a convolutional neural network model; obtaining time domain features and frequency domain features from a sample by windowing, superimposing the extracted time domain features and frequency domain features to obtain a time-frequency combination feature of the sample; inputting the time-frequency combination feature obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination feature obtained based on the validation samples in the validation set into the convolutional neural network model for verification, and obtaining a trained and verified convolutional neural network model through multiple training and verification cycles. In this way, while taking into account the changes in frequency energy, the overall sound information also has a high degree of recognition, and the accuracy of classification can also be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a sound classification method and device based on deep learning, a storage medium, and a computer. Background Art

[0002] Over the past decade, with the advancement of hardware computing power, neural networks have gradually emerged as a prominent force among machine learning algorithms. The advent of convolutional neural networks (CNNs), in particular, has enabled neural networks to achieve far greater accuracy in image classification than other algorithms. Compared to the booming field of image classification, research on sound classification has been less extensive. Sound classification has numerous real-life applications, such as determining whether a car engine is malfunctioning, distinguishing the function of a moving bearing, and properly installing keyboard keycaps. Traditional sound analysis techniques extract features such as energy intensity, zero-crossing rate, and short-term energy, or extract spectrograms through time-frequency analysis and then apply clustering or decision tree algorithms to these features. While these methods can achieve some success, their accuracy still requires significant improvement.

[0003] Therefore, it is necessary to propose a new solution to overcome the related problems. Summary of the Invention

[0004] The purpose of the present invention is to provide a sound classification method and device, storage medium and computer, which can combine the characteristics of time domain and frequency domain, so that while taking into account the frequency energy changes, it also has a high recognition of the overall sound information, and can also improve the accuracy of classification.

[0005] To achieve the purpose of the invention, according to one aspect of the present invention, the present invention provides a sound classification method, which includes: providing a training set, a validation set and a convolutional neural network model, wherein the training set includes multiple training samples, the validation set includes one or more validation samples, and each sample is a segment of labeled sound data; adding a first window to a sample, calculating and extracting time domain features for the sound data in the first window, adding a second window to the sound data in the first window, converting the sound data in the second window from the time domain to the frequency domain, and extracting frequency domain features, superimposing the extracted time domain features and frequency domain features to obtain the time-frequency combination features of the sample; inputting the time-frequency combination features obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination features obtained based on the validation samples in the validation set into the convolutional neural network model for verification, and obtaining a trained and verified convolutional neural network model through multiple training and verification.

[0006] According to another aspect of the present invention, the present invention provides a sound classification device, which includes: a feature extraction module, which adds a first window to each sample, calculates and extracts time domain features from the sound data in the first window, adds a second window to the sound data in the first window, converts the sound data in the second window from the time domain to the frequency domain, and then extracts frequency domain features, and superimposes the extracted time domain features and frequency domain features to obtain the time-frequency combination features of the sample, wherein the data input into the feature extraction module includes a training set, a validation set, and a test set, the training set includes multiple training samples, the validation set includes one or more validation samples, and the test set includes one or more test samples, each sample is a segment of labeled sound data; a convolutional neural network model, which is configured to receive the time-frequency combination features obtained based on the training samples in the training set for training, and receive the time-frequency combination features obtained based on the validation samples in the validation set for verification, and obtain the trained and verified convolutional neural network model through multiple training and verification.

[0007] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, which are executed during the program execution to perform a sound classification method. The sound classification method includes: providing a training set, a verification set, and a convolutional neural network model, wherein the training set includes multiple training samples, the verification set includes one or more verification samples, and each sample is a segment of labeled sound data; adding a first window to a sample, calculating and extracting time domain features from the sound data in the first window, adding a second window to the sound data in the first window, converting the sound data in the second window from the time domain to the frequency domain, and extracting frequency domain features, superimposing the extracted time domain features and frequency domain features to obtain the time-frequency combination features of the sample; inputting the time-frequency combination features obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination features obtained based on the verification samples in the verification set into the convolutional neural network model for verification, and obtaining a trained and verified convolutional neural network model through multiple training and verification.

[0008] According to another aspect of the present invention, the present invention provides a computer, which includes a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute a sound classification method. The sound classification method includes: providing a training set, a verification set, and a convolutional neural network model, wherein the training set includes multiple training samples, and the verification set includes one or more verification samples, each sample is a segment of labeled sound data; adding a first window to a sample, calculating and extracting time domain features for the sound data in the first window, adding a second window to the sound data in the first window, converting the sound data in the second window from the time domain to the frequency domain, and extracting frequency domain features, superimposing the extracted time domain features and frequency domain features to obtain the time-frequency combination features of the sample; inputting the time-frequency combination features obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination features obtained based on the verification samples in the verification set into the convolutional neural network model for verification, and obtaining a trained and verified convolutional neural network model through multiple training and verification.

[0009] Compared with the existing technology, the present invention uses a two-layer window, selects a large window for calculation of time domain features, and uses a smaller window for short-time Fourier transform of frequency domain features. Combining the characteristics of time domain and frequency domain, it takes into account the frequency energy changes while also having a high recognition rate for the overall sound information; in addition, a structurally optimized convolutional neural network model is used to extract deep time-frequency features for classification through convolution, which can improve noise resistance, tolerance to system errors and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of a time-frequency combination feature obtained by superimposing frequency domain features and time domain features according to an embodiment of the present invention;

[0011] Figure 2 1 is a flow chart of a sound classification method according to an embodiment of the present invention;

[0012] Figure 3 FIG. 1 is a schematic structural diagram of a sound classification device in one embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0014] Figure 2 FIG. 1 is a flow chart of the sound classification method 100 in one embodiment of the present invention. Figure 2As shown, the sound classification method 100 is generally divided into a training phase and a use phase.

[0015] like Figure 2 As shown, the sound classification method 100 includes the following steps or operations.

[0016] Step 110: Provide a training set, a validation set, and a convolutional neural network (CNN) model.

[0017] The training set includes multiple training samples, and the validation set includes one or more validation samples, each of which is a piece of labeled sound data. The labels may include a yes (OK) label and a no (NG) label, wherein the labels may be manually calibrated, with samples labeled as yes being considered acceptable and samples labeled as no being considered unacceptable. Of course, the labels may also include multiple categories, such as first-category labels, second-category labels, third-category labels, etc., so that the trained convolutional neural network model can classify the input samples into the first, second, or third categories.

[0018] In addition, a test set may be provided, comprising one or more test samples. The test samples are used to test the convolutional neural network model to determine whether the convolutional neural network model can be formally used. Of course, in some embodiments, a test set may not be provided depending on application needs.

[0019] The sample acquisition process includes: providing an initial sample; preprocessing the initial sample to form the sample. Preferably, the sample set (sample composition set) can be expanded by adding noise, data perturbation, speed perturbation, tuning, etc.

[0020] Specifically, sound is collected through a recording device. If background sound needs to be recorded separately as background data in a noisy scene, multiple initial samples can be obtained. The preprocessing includes: merging data channels, such as merging the left and right channels into one data channel; and / or converting all initial samples to the same sampling rate through resampling; and / or cropping long initial samples and padding short initial samples; and / or performing noise reduction or sound enhancement on the initial samples; and labeling the initial samples. The labeling of the initial samples can be done manually.

[0021] In one embodiment, the sample set can be randomly divided into a training set, a validation set, and a test set according to a certain ratio. It should be noted that each sample assigned to the training set is called a training sample, assigned to the validation set is called a validation sample, and assigned to the test set is called a test sample. In fact, there is no essential difference in technology between training samples, validation samples, and test samples.

[0022] Step 121: add a first window to a sample, and calculate and extract time domain features from the sound data in the first window.

[0023] Specifically, the first window is a rectangular window, and the window size is a first predetermined time length, such as about 1 second. The time domain features include one or more of mean, standard deviation, amplitude, root mean square, maximum point, skewness factor, kurtosis factor, margin factor and crest factor.

[0024] Step 122 , adding a second window to the sound data in the first window, converting the sound data in the second window from the time domain to the frequency domain, and then performing frequency domain feature extraction.

[0025] The sound data in the second window is converted from the time domain to the frequency domain by short-time Fourier transform. The second window selects the Hanning window, the window size is the second predetermined time length, such as 20ms, the first predetermined time length is N times the second predetermined time length, and N is an integer greater than or equal to 2, such as N=50. Specifically, the sound data in the frequency domain in the second window is converted to the Mel (MEL) scale, and the logarithm is calculated to obtain the frequency domain features. Converting the data to the Mel scale allows the convolutional neural network (CNN) model to have the same frequency recognition as humans.

[0026] Step 123, the extracted time domain features and frequency domain features are superimposed to obtain the time-frequency combined features of the sample. Figure 1 As shown, the horizontal direction is time, the lower part is the time domain feature, and the upper part is the frequency domain feature. Since the first window is N times the second window, Figure 1 In the upper middle part, N frequency domain features are arranged in time sequence, corresponding to the time domain features in time, so that a time-frequency combination feature can be formed, also known as a feature matrix.

[0027] In this way, the convolutional neural network model can use the information features of both time and frequency spaces at the same time, and because the features come from different time scales, it can retain time dependencies over longer distances, enabling the convolutional neural network model to better mine deep features.

[0028] For ease of understanding, steps 121, 122, and 123 may be collectively referred to as a time-frequency combination feature extraction step, which is used to extract time-frequency combination features from the sample. Through the time-frequency combination feature extraction step, time-frequency combination features can be extracted from samples such as training samples and validation samples. This step can be applied to training sets, validation sets, and test sets.

[0029] In step 130, the time-frequency combination features obtained based on the training samples in the training set are input into the convolutional neural network model for training, and the time-frequency combination features obtained based on the verification samples in the verification set are input into the convolutional neural network model for verification. Through continuous training and verification, a convolutional neural network model after training and verification is obtained.

[0030] Preferably, the convolutional neural network model includes a convolution layer, a pooling layer, a dense block, a fully connected layer and a sigmoid function layer connected in sequence, an activation function layer and a normalization (batch-norm) layer are sequentially arranged between the convolution layer and the pooling layer, an activation function layer and a normalization layer are sequentially arranged between the dense block and the fully connected layer, and the sigmoid function layer is used to shrink the result between 0 and 1. Specifically, there are two dense blocks, respectively referred to as the first dense block and the second dense block, and there are two fully connected layers, respectively referred to as the first fully connected layer and the second fully connected layer. The first dense block is connected to the pooling layer, and the activation function layer and the normalization layer are sequentially arranged between the first dense block and the second dense block. The activation function layer and the normalization layer are sequentially arranged between the second dense block and the first fully connected layer. After the second fully connected layer is connected to the second fully connected layer, the normalization layer is used to speed up training and highlight the distribution differences between the data. The inclusion of a regularization term in the loss function ensures good generalization even during long training periods, enabling the neural network to better learn features and achieve higher accuracy. In the convolutional neural network model, dense layers are used to extract deep features. These layers have superior deep feature extraction capabilities, helping to obtain better features. A fully connected layer is used as a classifier at the end, and a sigmoid function layer is used to shrink the results to a range of 0-1.

[0031] During the training process, the parameters are optimized through the BP (Back Propagation) algorithm. When the loss function becomes stable, the training is terminated and the convolutional neural network model is saved.

[0032] Step 140 : extracting time-frequency combination features from the test sample in the same manner as step 120 .

[0033] Step 150: Use the trained and verified convolutional neural network model to detect and classify the time-frequency combination features of the test sample.

[0034] like Figure 2 As shown, steps 110-130 can be collectively referred to as the training and verification phase of the convolutional neural network model, and steps 140 and 150 can be referred to as the testing phase.

[0035] Application examples:

[0036] A well-known manufacturer is testing bearing failures in packaging equipment. Bearings in equipment support rotation and reduce friction, making them a crucial component of mechanical equipment. A bearing failure can cause significant damage to the equipment. Real-time bearing detection can mitigate losses caused by bearing failures. Currently, the manufacturer uses a stethoscope to listen to different bearing locations daily. Because the bearings are located in a very small space, listening with a stethoscope can easily interfere with equipment operation. Therefore, artificial intelligence (AI) has been chosen to replace manual labor, enabling real-time detection without damaging either the equipment or personnel.

[0037] Detected defect category: good bearing or bearing failure, that is, "yes mark" represents good bearing, "no mark" represents bearing failure.

[0038] Detection Difficulties: Bearings are located inside machinery, where background noise is particularly high. Bearings are very small objects, and their low friction results in low noise. Even after applying noise reduction algorithms, the signal-to-noise ratio is still not very high. Capturing the desired signal amidst the noise and classifying it is crucial to the final accuracy. Traditional methods extract MFCC or Log-Mel features in the frequency domain and classify them using a neural network. However, this method classifies sound segments with high noise energy, resulting in low accuracy.

[0039] Considering that bearing noise is generated by mechanical rotation and exhibits temporal periodicity, windowed time-domain features better reflect bearing rotation, while frequency-domain features better reflect changes in energy at different frequencies over time. Using a noise reduction algorithm and time-frequency analysis to obtain combined time-frequency features, the superimposed time-frequency features simultaneously reflect both frequency energy changes and short-term time-domain energy. Training is performed based on a convolutional neural network, using the cross-entropy loss function. The training process performs well on the validation set, achieving very high accuracy. Training is terminated when the loss function stabilizes, producing the final convolutional neural network model.

[0040] Finally, the system was tested on the test set. While ensuring the over-detection rate was 0, the missed detection rate was kept below 0.5%, and the overall accuracy exceeded 99%.

[0041] It can be seen that the present invention uses two layers of windows, selects a large window for calculation of time domain features, and uses a smaller window for short-time Fourier transform of frequency domain features. Combining the characteristics of time domain and frequency domain, it takes into account the frequency energy changes while also having a high recognition rate for the overall sound information; in addition, a structurally optimized convolutional neural network model is used to extract deep time-frequency features for classification through convolution, which can improve noise resistance, tolerance to system errors and classification accuracy.

[0042] According to another aspect of the present invention, the present invention can also be implemented as a sound classification device. Figure 3 FIG. 3 is a schematic diagram of the structure of a sound classification device 300 in one embodiment of the present invention. The sound classification device 300 includes a feature extraction module 310 and a convolutional neural network model 320 .

[0043] The feature extraction module 310 adds a first window to each sample, calculates and extracts time domain features from the sound data in the first window, adds a second window to the sound data in the first window, converts the sound data in the second window from the time domain to the frequency domain, and then extracts frequency domain features. The extracted time domain features and frequency domain features are superimposed to obtain the time-frequency combination features of the sample, wherein the data input into the feature extraction module includes a training set, a validation set, and a test set, wherein the training set includes multiple training samples, the validation set includes one or more validation samples, and the test set includes one or more test samples, and each sample is a segment of labeled sound data.

[0044] The convolutional neural network model 320 is configured to receive time-frequency combination features obtained based on training samples in the training set for training, and receive time-frequency combination features obtained based on verification samples in the verification set for verification. Through continuous training and verification, a trained and verified convolutional neural network model is obtained. The trained and verified convolutional neural network model 320 is used to detect the time-frequency combination features obtained based on the test samples in the test set.

[0045] Since the sound classification device 300 is essentially technically consistent with the sound classification method 100 , the overlapping parts will not be repeated here.

[0046] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, which are executed when the program is executed to perform the above-mentioned sound classification method. The sound classification method includes: providing a training set, a validation set, and a convolutional neural network model, wherein the training set includes multiple training samples, and the validation set includes one or more validation samples, each sample being a segment of labeled sound data; adding a first window to a sample, calculating and extracting time domain features from the sound data in the first window, adding a second window to the sound data in the first window, converting the sound data in the second window from the time domain to the frequency domain, and extracting frequency domain features, superimposing the extracted time domain features and frequency domain features to obtain the time-frequency combination features of the sample; inputting the time-frequency combination features obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination features obtained based on the validation samples in the validation set into the convolutional neural network model for verification, and through continuous training and verification, obtaining a trained and verified convolutional neural network model. The remaining steps of the sound classification method 100 are not repeated here.

[0047] According to another aspect of the present invention, a computer is provided, comprising a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to perform the above-mentioned sound classification method. The sound classification method comprises: providing a training set, a validation set, and a convolutional neural network model, wherein the training set comprises multiple training samples, and the validation set comprises one or more validation samples, each sample being a segment of labeled sound data; adding a first window to a sample, calculating and extracting time domain features from the sound data within the first window, adding a second window to the sound data within the first window, converting the sound data within the second window from the time domain to the frequency domain, and then extracting frequency domain features; superimposing the extracted time domain features and frequency domain features to obtain a time-frequency combination feature of the sample; inputting the time-frequency combination feature obtained based on the training samples in the training set into the convolutional neural network model for training, and inputting the time-frequency combination feature obtained based on the validation samples in the validation set into the convolutional neural network model for validation, and through continuous training and validation, obtaining a trained and validated convolutional neural network model. The remaining steps of the sound classification method 100 are not repeated here.

[0048] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0049] In this document, directional terms such as front, back, top, and bottom are defined based on the positions of components in the accompanying drawings and relative to each other, and are intended only for clarity and convenience in describing the technical solution. It should be understood that the use of these directional terms should not limit the scope of protection claimed in this application.

[0050] In the absence of conflict, the above embodiments and features in the embodiments may be combined with each other.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sound classification method, characterized in that: It includes: Providing a training set, a validation set, and a convolutional neural network model, wherein the training set includes multiple training samples, the validation set includes one or more validation samples, and each sample is a segment of labeled sound data; A first window is added to a sample, and time domain features are extracted by calculating the sound data in the first window. A second window is added to the sound data in the first window, and the sound data in the second window is converted from the time domain to the frequency domain and then the frequency domain features are extracted. The extracted time domain features and frequency domain features are superimposed to obtain the time-frequency combination features of the sample, wherein the first window is a rectangular window, and the window size is a first predetermined time length. The second window is a Hanning window, and the size of the second window is a second predetermined time length. The first predetermined time length is N times the second predetermined time length, and N is an integer greater than or equal to 2. The time domain features extracted based on the first window are combined with the frequency domain features extracted based on the N second windows to form a time-frequency combination feature, wherein the upper part of the time-frequency combination feature is the frequency domain feature, and the lower part is the time domain feature. The N frequency domain features of the upper part are arranged in sequence according to time and correspond to the time domain features of the lower part according to time; The time-frequency combination features obtained based on the training samples in the training set are input into the convolutional neural network model for training, and the time-frequency combination features obtained based on the verification samples in the verification set are input into the convolutional neural network model for verification. After multiple training and verification, a convolutional neural network model after training and verification is obtained. The convolutional neural network model includes a convolution layer, a pooling layer, a dense block, a fully connected layer and a sigmoid function layer connected in sequence. An activation function layer and a normalization layer are sequentially arranged between the convolution layer and the pooling layer, and an activation function layer and a normalization layer are sequentially arranged between the dense block and the fully connected layer. The sigmoid function layer is used to shrink the result between 0 and 1. There are two dense blocks, respectively called the first dense block and the second dense block; there are two fully connected layers, respectively called the first fully connected layer and the second fully connected layer; the first dense block is connected to the pooling layer; an activation function layer and a normalization layer are sequentially arranged between the first dense block and the second dense block; an activation function layer and a normalization layer are sequentially arranged between the second dense block and the first fully connected layer; after the second fully connected layer is connected to the first fully connected layer, the normalization layer is used to speed up training and highlight the distribution differences between the data.

2. The sound classification method according to claim 1, wherein: The method further includes: providing a test set, wherein the test set includes one or more test samples; The time-frequency combination features obtained based on the test samples in the test set are input into the trained and verified convolutional neural network model for testing and detection.

3. The sound classification method according to claim 1, wherein: The sample acquisition process includes: Provide an initial sample; and Preprocessing the initial samples to form the samples, the preprocessing including: converting all the initial samples to the same sampling rate by resampling; and / or cropping long initial samples and padding short initial samples; and / or performing noise reduction or sound enhancement on the initial samples; Marking the initial sample, The sample set is divided into a training set and a validation set.

4. The sound classification method according to claim 1, wherein: The time domain characteristics include one or more of mean, standard deviation, amplitude, root mean square, maximum point, skewness factor, kurtosis factor, margin factor and crest factor, The sound data in the second window is converted from the time domain to the frequency domain through short-time Fourier transform, the sound data in the frequency domain is converted to the Mel (MEL) scale, and the logarithm is calculated to obtain the frequency domain features.

5. The sound classification method according to claim 1, wherein: During the training process, the parameters are optimized using the BP algorithm, and the training is terminated when the loss function becomes stable.

6. A sound classification device, characterized in that: It includes: A feature extraction module, which adds a first window to each sample, calculates and extracts time domain features from the sound data in the first window, adds a second window to the sound data in the first window, converts the sound data in the second window from the time domain to the frequency domain, and then extracts frequency domain features, and superimposes the extracted time domain features and frequency domain features to obtain a time-frequency combination feature of the sample, wherein the data input into the feature extraction module includes a training set and a verification set, the training set includes multiple training samples, the verification set includes one or more verification samples, each sample is a segment of labeled sound data, the first window is a rectangular window, the window size is a first predetermined time length, the second window is a Hanning window, the size of the second window is a second predetermined time length, the first predetermined time length is N times the second predetermined time length, N is an integer greater than or equal to 2, the time domain features extracted based on the first window and the frequency domain features extracted based on the N second windows are combined to form a time-frequency combination feature, the upper part of the time-frequency combination feature is a frequency domain feature, and the lower part is a time domain feature, the N frequency domain features of the upper part are arranged in sequence according to time, and correspond to the time domain features of the lower part according to time; A convolutional neural network model is configured to receive time-frequency combination features obtained based on training samples in the training set for training, receive time-frequency combination features obtained based on verification samples in the verification set for verification, and obtain a trained and verified convolutional neural network model through multiple training and verification. The convolutional neural network model includes a convolution layer, a pooling layer, a dense block, a fully connected layer and a sigmoid function layer connected in sequence. An activation function layer and a normalization layer are sequentially arranged between the convolution layer and the pooling layer, and an activation function layer and a normalization layer are sequentially arranged between the dense block and the fully connected layer. The sigmoid function layer is used to shrink the result between 0 and 1. There are two dense blocks, respectively called the first dense block and the second dense block; there are two fully connected layers, respectively called the first fully connected layer and the second fully connected layer; the first dense block is connected to the pooling layer; an activation function layer and a normalization layer are sequentially arranged between the first dense block and the second dense block; an activation function layer and a normalization layer are sequentially arranged between the second dense block and the first fully connected layer; after the second fully connected layer is connected to the first fully connected layer, the normalization layer is used to speed up training and highlight the distribution differences between the data.

7. The sound classification device according to claim 6, wherein: The data input into the feature extraction module includes a test set, which includes one or more test samples. The time-frequency combination features obtained based on the test samples in the test set are detected using a convolutional neural network model that has been trained and verified.

8. A storage medium, characterized in that: It stores program instructions, which are executed to perform the sound classification method according to any one of claims 1 to 5.

9. A computer, characterized in that: The method comprises a processor and a memory, wherein program instructions are stored in the memory, and the processor runs the program instructions to execute the sound classification method according to any one of claims 1 to 5.

Images