Convolutional neural network model tbsp-net for predicting abnormal state based on audio data

Abstract

The application provides a convolutional neural network model TBSP-Net for predicting an abnormal state based on audio data, which comprises an input module, a feature extraction module and a classification module, the input module acquires audio data at a preset sampling frequency, and pre-processes the acquired audio data, converts the acquired audio data reflecting only time domain information into composite data having both time and frequency, prevents overfitting, then uses the feature extraction module to extract features from the converted composite data, and uses the classification module to classify the features, thereby improving the diagnostic prediction accuracy of the abnormal state of an individual.

Description

Technical Field

[0001] This invention belongs to the field of abnormal state diagnosis and prediction, and in particular relates to a convolutional neural network model TBSP-Net for predicting abnormal states based on audio data. Background Technology

[0002] An abnormal state refers to an object's working condition under abnormal circumstances. Its manifestations can be diverse. While various abnormal states are exposed externally and easily detected, internal anomalies are difficult to detect due to visual confinement. For example, it's challenging to detect potential internal anomalies in operating machines, requiring disassembly or invasive methods to investigate the cause, and the results are not always accurate. However, some abnormal states, if the machine continues to operate, are accompanied by audio signals with different characteristics. Therefore, audio data can be modeled to obtain the corresponding internal abnormal states without disassembling or invasively probing. Currently, diagnostic models based on audio data exist for predicting and diagnosing abnormal states. However, for different faults with similar audio data, the diagnostic predictions are not accurate enough. Therefore, how to model and predict abnormal states based on audio data to better predict these states remains a topic worthy of further investigation. Summary of the Invention

[0003] In view of the problems existing in the prior art, the main objective of the present invention is to provide a convolutional neural network model TBSP-Net based on audio data to predict abnormal states, so as to alleviate overfitting, improve the accuracy of pulmonary tuberculosis diagnosis prediction on an individual basis, and fill the gap in predicting pulmonary tuberculosis positivity based on audio data.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] This invention provides a convolutional neural network model TBSP-Net for predicting abnormal states based on audio data, including an input module, a feature extraction module, and a classification module. The input module acquires audio data at a preset sampling frequency and preprocesses the acquired audio data. The preprocessing of the audio data includes the following steps:

[0006] The acquired audio data is processed using a control unit to generate a vector. ;

[0007] Make the vector The time-shift vector is obtained by randomly shifting the time vector left or right in the time domain. The calculation formula is:

[0008] ;

[0009] in, Representative vector No. The position of each sampling point after rolling; Representative vector Index of sampling points; Representative vector The length of the scrolling time in the time domain; Represents the audio sampling frequency; This represents the modulo operation, used to implement vector operations. The loop scrolls; For vectors Length;

[0010] The obtained time shift vector Perform a short-time Fourier transform to obtain its time-frequency neighborhood features. The calculation formula is:

[0011] ;

[0012] in, Represents frequency; Representing the The starting sampling point between time periods; Representative vector Length; Represents the size of the sliding window; Represents frequency index; Represents a window function; Represents the input frame; This represents the distance between two sliding window frames; Represents the imaginary unit;

[0013] The acquired time-frequency neighborhood features Random masking is performed in the frequency domain to obtain the frequency domain vector. The calculation formula is:

[0014] ;

[0015] in, Representing the Frequency component indexes randomly selected over a time period This represents the frequency or number of time-domain blocks.

[0016] The obtained frequency domain vector The input is fed into the feature extraction module to obtain the feature vector;

[0017] The acquired feature vectors are input into the classification module to classify the acquired feature vectors.

[0018] As a further description of the above technical solution, in the step "the input module acquires audio data at a preset sampling rate", the preset sampling rate is 44100Hz.

[0019] As a further description of the above technical solution, in the step "using the control unit to trim the acquired audio data, thereby generating a vector..." In this context, the control unit is a time constant. One control channel lasts for 0.5 seconds.

[0020] As a further description of the above technical solution, the feature extraction module includes a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, and a sixth convolutional block connected in sequence.

[0021] Each convolutional block includes a convolutional layer, a batch normalization layer, and a max pooling layer.

[0022] As a further description of the above technical solution, the convolutional layer of the first convolutional block is composed of a first convolutional kernel with a size of 5×5 and no activation function, wherein the stride of the first convolutional kernel is 2.

[0023] The convolutional layers of the second, third, fourth, fifth, and sixth convolutional blocks are all composed of a second convolutional kernel with a size of 3×3 and no activation function, wherein the stride of the second convolutional kernel is 2.

[0024] As a further description of the above technical solution, the batch normalization layers of the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, the fifth convolutional block, and the sixth convolutional block all carry the ReLU function.

[0025] The max pooling layers of the first, second, third, fourth, fifth, and sixth convolutional blocks are all composed of pooling kernels of size 2×2 and have no activation function.

[0026] As a further description of the above technical solution, the classification module includes a first fully connected layer, a second fully connected layer, and a classification layer; wherein,

[0027] The input of the first fully connected layer is connected to the output of the max pooling layer of the sixth convolutional block through an information transmission channel;

[0028] The input of the second fully connected layer is connected to the output of the first fully connected layer through an information transmission channel;

[0029] The input of the classification layer is connected to the output of the second fully connected layer through an information transmission channel.

[0030] As a further description of the above technical solution, a Dropout layer is added after both the first fully connected layer and the second fully connected layer, and the dropout probability value of the Dropout layer added after the first fully connected layer is 0.5.

[0031] The dropout probability value of the Dropout layer added after the second fully connected layer is 0.3.

[0032] As a further description of the above technical solution, both the first fully connected layer and the second fully connected layer carry the sigmod function.

[0033] As a further description of the above technical solution, the convolutional neural network model TBSP-Net based on audio data for predicting abnormal states is used for the diagnosis and prediction of abnormal states of individuals.

[0034] In summary, the outstanding effects of this invention are:

[0035] The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data provided by this invention can convert the acquired audio data that only reflects time-domain information into composite data containing both time and frequency information in its input module, which can prevent overfitting. Then, its feature extraction module extracts features from the converted composite data, and the classification module classifies it, thereby improving the accuracy of diagnosis and prediction of abnormal states on an individual basis. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a simplified structural diagram of TBSP-Net, a convolutional neural network model for predicting abnormal states based on audio data, in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In the description of this invention, it should be noted that the terms "upper," "middle," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0040] Please see Figure 1 This invention provides a convolutional neural network model TBSP-Net for predicting abnormal states based on audio data. It comprises an input module, a feature extraction module, and a classification module. The input module acquires audio data at a preset sampling frequency and preprocesses it, converting the acquired audio data, which only reflects time-domain information, into composite data containing both time and frequency information. The feature extraction module extracts features from the preprocessed composite data. The feature vectors extracted by the feature extraction module are then input into the classification module for classification, thus achieving the classification of abnormal states and improving the accuracy of diagnosing and predicting abnormal states on an individual basis.

[0041] Specifically, in this embodiment, the sampling frequency of the input module for acquiring audio data is 44100Hz. Of course, in other optional embodiments, the sampling frequency can also be set to other values ​​as needed.

[0042] Specifically, in this embodiment, the input module preprocesses the sampled audio data, including the following steps:

[0043] First, it uses a control unit to trim the acquired audio data, thereby generating a vector. In this process, the control unit is a time constant. With one control channel every 0.5 seconds, the sampled audio data will be trimmed to... ;

[0044] Next, the vector Data augmentation is performed, specifically in the time domain, by randomly shifting the data left / right to obtain the time-shift vector. The calculation formula is as follows:

[0045] ;

[0046] in, Represents a pair of vectors No. The position of each sampling point after rolling; Representative vector Index of sampling points; Representative vector The scrolling time in the time domain, in seconds, with positive values ​​representing scrolling to the left and negative values ​​representing scrolling to the right; Represents the audio sampling frequency; This represents the modulo operation, used to implement vector operations. The loop scrolls; For vectors Length;

[0047] Then, the obtained time shift vector A short-time Fourier transform was performed to obtain its time-frequency neighborhood features. The calculation formula is as follows:

[0048]

[0049] in, Represents frequency; Representing the The starting sampling point between time periods; Representative vector Length; Represents the size of the sliding window; Represents frequency index; Represents a window function; Represents the input frame; This represents the distance between two sliding window frames; Representing the imaginary unit, this process yields the time shift vector. The power spectrum;

[0050] Then, the acquired time-frequency neighborhood features Data augmentation is performed by randomly masking the data in the frequency domain, thereby obtaining the frequency domain vector. The calculation formula is as follows:

[0051]

[0052] in, Representing the Frequency component indexes randomly selected over a time period This represents the frequency or number of time-domain blocks.

[0053] This transforms audio data that only reflects time-domain information into composite data containing both time and frequency information in the input module, which can alleviate overfitting and ensure the performance of subsequent diagnostic predictions.

[0054] Then, the obtained frequency domain vector The feature vector is obtained by inputting it into the feature extraction module. Finally, the obtained feature vector is input into the classification module to classify the obtained feature vector, thus realizing the classification of abnormal states and realizing the prediction and diagnosis of abnormal states on an individual basis, with excellent prediction and diagnosis performance.

[0055] Specifically, in this embodiment, the feature extraction module includes a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, and a sixth convolutional block connected in sequence; each convolutional block includes a convolutional layer, a batch normalization layer, and a max pooling layer, and the audio data sampled by the input module is preprocessed into a frequency domain vector. Then, the data is input to the feature extraction module. First, it passes through the convolutional layer of the first convolutional block for feature extraction. Then, the feature map output by the convolutional layer is passed to the batch normalization layer included in the first convolutional block for feature standardization. Then, it is passed to the max pooling layer of the first convolutional block for feature compression and information filtering. The compressed and filtered output data is passed to the next convolutional block, where the convolutional layer, batch normalization layer, and max pooling layer included in it further perform feature extraction, feature standardization, feature compression, and information filtering. Finally, the output data compressed and filtered by the max pooling layer of the sixth convolutional block is flattened and then input to the classification module for abnormal state classification.

[0056] Specifically, in this embodiment, the convolutional layer of the first convolutional block consists of a first convolutional kernel with a size of 5×5 and no activation function, wherein the stride of the first convolutional kernel is 2; the convolutional layers of the second, third, fourth, fifth and sixth convolutional blocks are all composed of a second convolutional kernel with a size of 3×3 and no activation function, wherein the stride of the second convolutional kernel is 2.

[0057] Specifically, in this embodiment, the batch normalization layers of the first, second, third, fourth, fifth, and sixth convolutional blocks all carry ReLU functions; the max pooling layers of the first, second, third, fourth, fifth, and sixth convolutional blocks are all composed of pooling kernels of size 2×2, without activation functions.

[0058] Specifically, in this embodiment, the classification module includes a first fully connected layer, a second fully connected layer, and a classification layer. The input of the first fully connected layer is connected to the output of the max-pooling layer of the sixth convolutional block via an information transmission channel; the input of the second fully connected layer is connected to the output of the first fully connected layer via an information transmission channel; and the input of the classification layer is connected to the output of the second fully connected layer via an information transmission channel. This allows the output data, compressed and filtered by the max-pooling layer of the sixth convolutional block, to be flattened and first passed to the first fully connected layer for feature extraction. Then, the output data extracted by the first fully connected layer is passed to the second fully connected layer for feature extraction, and finally, the output data extracted by the first fully connected layer is passed to the classification layer for classification, thereby identifying different abnormal states.

[0059] Specifically, in this embodiment, a Dropout layer is added after both the first fully connected layer and the second fully connected layer to prevent the model from overfitting and improve the model's generalization ability. The dropout probability value of the Dropout layer added after the first fully connected layer is 0.5, and the dropout probability value of the Dropout layer added after the second fully connected layer is 0.3.

[0060] Specifically, in this embodiment, both the first fully connected layer and the second fully connected layer carry the sigmoid function as the activation function to ensure that the model network fits well and can accurately classify abnormal states in the future.

[0061] Specifically, in this embodiment, the convolutional neural network model TBSP-Net, which predicts abnormal states based on audio data, is used for the diagnostic prediction of abnormal states in individuals. It should be understood that in this embodiment, the "abnormal state" refers to a non-disease state that requires attention, which can be a non-disease state requiring long-term continuous monitoring, a non-disease state requiring intermittent monitoring, or a non-disease state requiring short-term continuous monitoring, etc. For example, after a transformer in a substation is switched on and energized, the alternating magnetic flux in the core generates a force vibration in the silicon steel sheets of the core, producing a "humming" sound. The loudness of this sound is proportional to the voltage and current applied to the transformer. During normal operation, this sound is emitted uniformly. However, when abnormal current or voltage occurs, or when the transformer experiences core faults, loose parts, or inter-turn short circuits, the characteristics of this sound will change differently. Therefore, this model can be used to achieve long-term continuous monitoring of transformers, thereby accurately diagnosing and predicting faults in individual transformers. That is, it can ultimately diagnose and predict whether the anomaly of a single transformer is caused by abnormal current or voltage, or by core faults, loose parts, or inter-turn short circuits. Therefore, in this embodiment, the classification layer consists of 5 nodes, corresponding to the sound signals of 4 typical abnormal states and the normal operating state. Of course, in other embodiments, the number of nodes in the classification layer can be set according to the number of categories of states to be monitored.

[0062] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any changes, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A convolutional neural network model TBSP-Net for predicting abnormal states based on audio data, characterized in that, It includes an input module, a feature extraction module, and a classification module. The input module acquires audio data at a preset sampling frequency and preprocesses the acquired audio data. The preprocessing of the audio data includes the following steps: The acquired audio data is processed using a control unit to generate a vector v; The vector v is randomly shifted left / right in the time domain to obtain the time-shifted vector v. roll The calculation formula is: v roll [i]=v[(i+r*Fs)mod N],for i=0,1,…,N-1; Among them, v roll [i] represents the position of the i-th sampling point of vector v after scrolling; i represents the index of the sampling point of vector v; r represents the scrolling time of vector v in the time domain; Fs represents the audio sampling frequency; mod represents the modulo operation, used to implement the cyclic scrolling of vector v; N is the length of vector v; The obtained time shift vector v roll Perform a short-time Fourier transform to obtain its time-frequency neighborhood feature V[w,m], calculated as follows: Where w represents frequency; m represents the starting sampling point between the m-th time intervals; len represents the length of vector v; win_len represents the size of the sliding window; k represents the frequency index; win[k] represents the window function; input represents the input frame; hop_len represents the distance between two sliding window frames; and j represents the imaginary unit. The acquired time-frequency neighborhood features V[w,m] are randomly masked in the frequency domain to obtain the frequency domain vector S(m,k), calculated as follows: Where, m k F_mask represents the index of a frequency component randomly selected in the m-th time period, and F_mask represents the number of frequencies or time domains to be masked. The obtained frequency domain vector S(m,k) is input into the feature extraction module to obtain the feature vector; The acquired feature vectors are input into the classification module to classify the acquired feature vectors.

2. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 1, characterized in that, In the step "the input module acquires audio data at a preset sampling rate", the preset sampling rate is 44100Hz.

3. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 1, characterized in that, In the step "using the control unit to trim the acquired audio data to generate vector v", the control unit is a control channel with a time constant τ of 0.5s.

4. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 1, characterized in that, The feature extraction module includes a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, and a sixth convolutional block connected in sequence; Each convolutional block includes a convolutional layer, a batch normalization layer, and a max pooling layer.

5. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 4, characterized in that, The first convolutional block consists of a first convolutional kernel of size 5×5 with no activation function, wherein the stride of the first convolutional kernel is 2. The convolutional layers of the second, third, fourth, fifth, and sixth convolutional blocks are all composed of a second convolutional kernel with a size of 3×3 and no activation function, wherein the stride of the second convolutional kernel is 2.

6. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 5, characterized in that, The batch normalization layers of the first, second, third, fourth, fifth, and sixth convolutional blocks all carry the ReLU function; The max pooling layers of the first, second, third, fourth, fifth, and sixth convolutional blocks are all composed of pooling kernels of size 2×2 and have no activation function.

7. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 4, characterized in that, The classification module includes a first fully connected layer, a second fully connected layer, and a classification layer; wherein... The input of the first fully connected layer is connected to the output of the max pooling layer of the sixth convolutional block through an information transmission channel; The input of the second fully connected layer is connected to the output of the first fully connected layer through an information transmission channel; The input of the classification layer is connected to the output of the second fully connected layer through an information transmission channel.

8. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 7, characterized in that, A Dropout layer is added after both the first fully connected layer and the second fully connected layer. The dropout probability value of the Dropout layer added after the first fully connected layer is 0.

5. The dropout probability value of the Dropout layer added after the second fully connected layer is 0.

3.

9. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 7, characterized in that, Both the first fully connected layer and the second fully connected layer carry the sigmod function.

10. The TBSP-Net convolutional neural network model for predicting abnormal states based on audio data according to claim 1, characterized in that, The TBSP-Net convolutional neural network model based on audio data for predicting abnormal states is used for the diagnosis and prediction of abnormal states in individuals.

Images