Abnormal sound detection method based on multi-task learning

Abstract

The invention discloses an abnormal sound detection method based on multiple tasks, and belongs to the technical field of sound detection, and the method comprises the steps: obtaining a to-be-detected sound signal; preprocessing the sound signal to be detected; wherein the preprocessing process at least comprises the steps of performing data enhancement on a sound signal to be detected, and generating a target Log-Mel spectrogram; inputting a sound signal to be detected and the target Log-Mel spectrogram into the trained multi-task learning network, and outputting a multi-modal feature and a classification abnormal value; inputting the multi-modal features into a trained Gaussian mixture model to generate an abnormal score; and performing anomaly judgment on the sound signal to be detected based on the anomaly score and the classification anomaly value. Through collaborative design of a multi-task learning framework and multi-modal feature fusion, the problems that a traditional anomaly detection method is weak in generalization ability and high in false alarm rate in a complex industrial environment are solved, and high-precision and high-robustness abnormal sound detection is achieved.

Description

Technical Field

[0001] This invention relates to the field of sound detection technology, specifically an abnormal sound detection method based on multi-task learning. Background Technology

[0002] In the era of Industry 4.0, intelligent manufacturing and intelligent operation and maintenance have become paramount in leading the transformation and upgrading of the industrial sector. Their core lies in improving production efficiency and operational reliability through intelligent technologies. Machine Condition Monitoring (MCM), as a key supporting technology in intelligent manufacturing and intelligent operation and maintenance systems, plays a crucial role in ensuring the continuous and stable operation of industrial equipment and reducing the total lifecycle maintenance cost. Among these methods, abnormal sound detection based on sound signals has been widely used in the fault diagnosis of various industrial equipment. However, traditional abnormal sound detection methods mainly rely on single-modal features for detection, resulting in limitations and low accuracy.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide an abnormal sound detection method based on multi-task learning.

[0005] To achieve the above objectives, this invention provides an abnormal sound detection method based on multi-task learning, comprising: acquiring a sound signal to be detected; preprocessing the sound signal to be detected; wherein the preprocessing process includes at least data augmentation of the sound signal to be detected and generating a target Log-Mel spectrogram; inputting the sound signal to be detected and the target Log-Mel spectrogram into a trained multi-task learning network, and outputting multimodal features and classification outliers; wherein the multi-task learning network includes a dual-branch module and multiple classifiers; the dual-branch module includes a temporal encoder and a frequency encoder; the temporal encoder generates a temporal encoder by processing the sound signal to be detected. The frequency domain encoder generates a frequency domain feature representation by processing the target Log-Mel spectrum. The multimodal features are generated by fusing the time domain feature representation with the frequency domain feature representation. The classifications of the multiple classifiers include machine speed range classification, temperature range classification, microphone placement classification, data augmentation operation classification, and machine type classification. The machine type classification is used to distinguish centrifuges from other machines. The result of the machine type classification is used to determine the classification outliers. The multimodal features are input into a trained Gaussian mixture model to generate anomaly scores. Based on the anomaly scores and the classification outliers, the sound signal to be detected is judged to be abnormal.

[0006] In one embodiment of the present invention, the multi-task learning network is trained through the following steps: acquiring sound data to obtain raw audio signals; preprocessing the raw audio signals; the data preprocessing includes dataset partitioning, loading, generating spectral features, data augmentation, and normalization; wherein, the dataset partitioning includes a training set and a test set; training the multi-task learning network based on the audio data in the training set, and testing it using the audio data in the test set.

[0007] In one embodiment of the present invention, the acquisition of sound data includes: acquiring normal sound samples and abnormal sound samples from centrifuges and other types of machines through a microphone; the original audio signal includes the normal sound samples and the abnormal sound samples; wherein, the amount of data of the normal sound samples is increased by controlling three variables: rotation speed, microphone placement position, and temperature, and the abnormal sound samples are obtained by the sound of damaged machines.

[0008] In one embodiment of the present invention, generating spectral features includes: performing a short-time Fourier transform on the original audio signal to obtain a spectrogram; and performing Mel filtering and logarithmic operation on the spectrogram to obtain a Log-Mel spectrogram.

[0009] In one embodiment of the present invention, for the original audio signal, the data enhancement methods include: pitch shifting, time stretching, adding noise, and time shifting; for the Log-Mel spectrogram, the data enhancement methods include: time masking and frequency masking.

[0010] In one embodiment of the present invention, the structure of the temporal encoder includes a convolutional network layer and a Transformer encoder; the convolutional network layer extracts local features of the original audio signal through 8 layers of 1D convolution operations and downsamples the time dimension; the Transformer encoder is used for global context modeling to capture the dependency relationship of features in different time periods; the structure of the frequency domain encoder includes 8 layers of two-dimensional convolutional layers; wherein, the 2nd to 8th layers are frequency-adaptive dynamic convolutional layers.

[0011] In one embodiment of the present invention, the plurality of classifiers includes a first classifier, a second classifier, a third classifier, a fourth classifier, and a fifth classifier; the first classifier is used for machine speed range classification; the second classifier is used for temperature range classification; the third classifier is used for microphone placement location classification; the fourth classifier is used for machine type classification; and the fifth classifier is used for data augmentation operation classification.

[0012] In one embodiment of the present invention, the fourth classifier is used to output the probability of belonging to a centrifuge and to determine outliers based on the probability.

[0013] In one embodiment of the present invention, the step of inputting the multimodal features into a trained Gaussian mixture model to generate anomaly scores includes: inputting the multimodal features into the Gaussian mixture model, calculating the negative log-likelihood of the multimodal features relative to the Gaussian mixture model, and generating the anomaly scores.

[0014] In one embodiment of the present invention, the step of determining the abnormality of the sound signal to be detected based on the abnormality score and the classification outlier includes: performing a standardized weighted summation based on the abnormality score and the classification outlier to obtain a total abnormality score; if the total abnormality score is higher than a set threshold, then the sound signal to be detected is determined to be an abnormal sound; if the total abnormality score is not higher than the set threshold, then the sound signal to be detected is determined to be a normal sound.

[0015] Compared with existing technologies, the abnormal sound detection method based on multi-task learning according to the present invention has the following beneficial effects:

[0016] First, this application proposes a dual-branch architecture to overcome the limitations of traditional abnormal sound detection methods that rely solely on features from a single modality. It processes sound signals from two complementary perspectives: the time domain and the frequency domain. Relying solely on the Log-Mel spectrum results in the loss of time-domain details, while relying solely on the raw audio data lacks global spectral information. This application designs a dual-branch network consisting of a time-domain encoder (processing the raw audio signal) and a frequency-domain encoder (processing the Log-Mel spectrum). By splicing and fusing time-domain and frequency-domain features along the channel dimension, it preserves the instantaneous dynamic characteristics of the time domain (such as short-duration pulses from mechanical impacts) and the spectral distribution of the frequency domain (such as harmonic energy changes). For some short-duration impact anomalies, the time-domain branch can accurately capture the timing of the pulse, while the frequency-domain branch can simultaneously extract the high-frequency spectral abrupt changes accompanying the impact. The fusion of these two branches significantly reduces the false negative rate.

[0017] Furthermore, in industrial settings, centrifuges often experience significant domain shifts in their operating environment (e.g., temperature differences between the source and target workshops, different speed ranges between old and new equipment, microphone placement adjustments, etc.). Traditional methods are prone to misjudgment during cross-domain testing. This framework, however, achieves significant domain generalization through multi-task constraints. This application constructs a multi-task learning framework that enables the neural network to learn more general feature representations of normal sound samples, effectively solving the domain shift problem. A main task and four auxiliary tasks are cleverly designed. The main task is binary classification of machine types, treating machines other than centrifuges as pseudo-anomalies. Through "pseudo-anomaly comparison," the model focuses on the unique normal sound features of centrifuges. While the normal sounds of other machines are not abnormal, they differ fundamentally from the acoustic characteristics of centrifuges. In distinguishing between the two, the model automatically filters out "acoustic features not common to centrifuges," learning the stable vibration modes and spectral patterns unique to centrifuges, thereby further enabling effective detection of abnormal sounds from industrial centrifuge equipment.

[0018] As can be seen from the above, this application solves the problems of weak generalization ability and high false alarm rate of traditional anomaly detection methods in complex industrial environments by using a collaborative design of multi-task learning framework and multi-modal feature fusion, and achieves high-precision and high-robust anomaly sound detection. Attached Figure Description

[0019] Figure 1 This is a flowchart of the steps of an abnormal sound detection method based on multi-task learning according to an embodiment of the present invention;

[0020] Figure 2 This is a block diagram of a multi-task learning network according to an embodiment of the present invention;

[0021] Figure 3 This is a block diagram of a time-domain encoder according to an embodiment of the present invention;

[0022] Figure 4 This is a block diagram of a frequency domain encoder according to an embodiment of the present invention;

[0023] Figure 5 This is a flowchart of a frequency-domain adaptive dynamic convolution process according to an embodiment of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0025] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0026] Please see Figure 1 This application provides an abnormal sound detection method based on multi-task learning, which specifically includes steps 101-105.

[0027] Step 101: Acquire the sound signal to be detected.

[0028] That is, during the application process, environmental sound signals can be collected in real time as the sound signals to be detected.

[0029] Step 102: Preprocess the sound signal to be detected.

[0030] The preprocessing process includes at least data augmentation of the sound signal to be detected and generation of the target Log-Mel spectrum.

[0031] The method for generating the target Log-Mel spectrum includes: performing a short-time Fourier transform on the original audio signal to obtain a spectrogram, and then performing Mel filtering and logarithmic operations on the spectrogram to obtain the target Log-Mel spectrum.

[0032] The data enhancement methods for the sound signal to be detected include: pitch shift, time stretching, adding noise, and time shift.

[0033] Data augmentation can also be performed on the target Log-Mel spectrum, including time masking and frequency masking.

[0034] Step 103: Input the sound signal to be detected and the target Log-Mel spectrogram into the trained multi-task learning network, and output multimodal features and classification outliers.

[0035] The multi-task learning network includes a dual-branch module and multiple classifiers. The dual-branch module comprises a time-domain encoder and a frequency-domain encoder. The time-domain encoder processes the sound signal to be detected to generate a time-domain feature representation. The frequency-domain encoder processes the target Log-Mel spectrum to generate a frequency-domain feature representation. By fusing the time-domain and frequency-domain feature representations, multimodal features are generated. The multiple classifiers perform classifications including machine speed range classification, temperature range classification, microphone placement classification, data augmentation operation classification, and machine type classification. Machine type classification is used to distinguish centrifuges from other machines. The results of machine type classification are used to identify outliers.

[0036] Step 104: Input the multimodal features into the trained Gaussian mixture model to generate anomaly scores.

[0037] Step 105: Based on the anomaly score and the classification outlier, determine the anomaly of the sound signal to be detected.

[0038] In summary, the abnormal sound detection method based on multi-task learning provided in this application has the following beneficial effects, including:

[0039] First, this application proposes a dual-branch architecture to overcome the limitations of traditional abnormal sound detection methods that rely solely on features from a single modality. It processes sound signals from two complementary perspectives: the time domain and the frequency domain. Relying solely on the Log-Mel spectrum results in the loss of time-domain details, while relying solely on the raw audio data lacks global spectral information. This application designs a dual-branch network consisting of a time-domain encoder (processing the raw audio signal) and a frequency-domain encoder (processing the Log-Mel spectrum). By splicing and fusing time-domain and frequency-domain features along the channel dimension, it preserves the instantaneous dynamic characteristics of the time domain (such as short-duration pulses from mechanical impacts) and the spectral distribution of the frequency domain (such as harmonic energy changes). For some short-duration impact anomalies, the time-domain branch can accurately capture the timing of the pulse, while the frequency-domain branch can simultaneously extract the high-frequency spectral abrupt changes accompanying the impact. The fusion of these two branches significantly reduces the false negative rate.

[0040] Furthermore, in industrial settings, centrifuges often experience significant domain shifts in their operating environment (e.g., temperature differences between the source and target workshops, different speed ranges between old and new equipment, microphone placement adjustments, etc.). Traditional methods are prone to misjudgment during cross-domain testing. This framework, however, achieves significant domain generalization through multi-task constraints. This application constructs a multi-task learning framework that enables the neural network to learn more general feature representations of normal sound samples, effectively solving the domain shift problem. A main task and four auxiliary tasks are cleverly designed. The main task is binary classification of machine types, treating machines other than centrifuges as pseudo-anomalies. Through "pseudo-anomaly comparison," the model focuses on the unique normal sound features of centrifuges. While the normal sounds of other machines are not abnormal, they differ fundamentally from the acoustic characteristics of centrifuges. In distinguishing between the two, the model automatically filters out "acoustic features not common to centrifuges," learning the stable vibration modes and spectral patterns unique to centrifuges, thereby further enabling effective detection of abnormal sounds from industrial centrifuge equipment.

[0041] As can be seen, this application solves the problems of weak generalization ability and high false alarm rate of traditional anomaly detection methods in complex industrial environments by using a collaborative design of multi-task learning framework and multi-modal feature fusion, and achieves high-precision and high-robust anomaly sound detection.

[0042] The training process of the multi-task learning network is explained below.

[0043] Optionally, the multi-task learning network is trained through the following steps: acquiring sound data to obtain raw audio signals; preprocessing the raw audio signals; data preprocessing includes dataset partitioning, loading, generating spectral features, data augmentation, and normalization; wherein, dataset partitioning includes training set and test set; training the multi-task learning network based on audio data in the training set, and testing it with audio data in the test set.

[0044] Optionally, sound data is collected, including: collecting normal sound samples and abnormal sound samples from centrifuges and other types of machines through a microphone; the raw audio signal includes normal sound samples and abnormal sound samples; wherein, the amount of normal sound sample data is increased by controlling three variables: rotation speed, microphone placement position, and temperature, and abnormal sound samples are obtained by using the sound of damaged machines.

[0045] Specifically, normal and abnormal sound samples from centrifuges and other types of machines in the source workshop can be collected using microphones. Data diversity is increased by controlling three variables: rotation speed, microphone placement, and temperature; this metadata is also automatically recorded. Simultaneously, a small number of normal and abnormal sound samples from centrifuges in the target workshop are collected to test and evaluate the model's domain generalization ability. Abnormal sound samples are collected by intentionally damaging the target machines. All audio files are saved in .wav format, with the temperature, microphone placement, and rotation speed parameters (which are saved as metadata) recorded in the filename.

[0046] The dataset partitioning process included: all samples collected from the source workshop were randomly divided into training and test sets at a 9:1 ratio, ensuring no overlap between the two sets; normal sound samples collected from the target workshop were divided into training and test sets at a 1:1 ratio, while all abnormal sound samples from the target workshop were used as the test set. The final training set included normal sound samples from centrifuges in both the source and target workshops, as well as normal sound samples from other types of machines in the source workshop. The test set included a small number of normal and abnormal sound samples from centrifuges in both the source and target workshops.

[0047] It should be noted that, simply put, the data partitioning here means that the training set contains only normal sound samples, with no abnormal sound samples, and includes centrifuges and other machines from the source workshop, as well as a small number of centrifuges from the target workshop. The test set contains both normal and abnormal sounds from the source and target workshops, but only includes centrifuges. The method in this application primarily targets centrifuges; other types of machines do not appear in the test set but are included in the training set as a disturbance target to encourage the model to learn more general feature representations of normal sounds.

[0048] The reading and loading process can specifically include: using the librosa library to read the first channel of an audio file ending in .wav and setting the sampling rate to 16kHz to obtain the raw audio signal that can be input into the neural network.

[0049] Optionally, generating spectral features includes: performing a short-time Fourier transform on the original audio signal to obtain a spectrogram; and performing Mel filtering and logarithmic operations on the spectrogram to obtain a Log-Mel spectrogram.

[0050] Specifically, a short-time Fourier transform is performed on the original audio signal to obtain a spectrogram. Then, Mel filtering and logarithmic operations are performed on the spectrogram to obtain a Log-Mel spectrogram. The frame shift is set to 512, the window size is set to 2048, and the number of Mel filter bands is set to 128.

[0051] Optionally, for the original audio signal, data augmentation methods include: pitch shifting, time stretching, adding noise, and time shifting; for the Log-Mel spectrogram, data augmentation methods include: time masking and frequency masking.

[0052] It should be noted that each data augmentation method has a 50% probability of being applied.

[0053] Pitch Shift: Randomly adjusts the pitch within the range of [-10, 10] semitones;

[0054] Time Stretch: Changes the audio speed at random rates of [0.5, 2], stretches and then resamples to maintain the original length.

[0055] Add noise (White Noise Injection): Inject Gaussian white noise with a signal-to-noise ratio (SNR) of [-6, 6].

[0056] Time Shifting: Randomly shifts the audio signal forward or backward by an amount ranging from 0 to half the signal length.

[0057] Time masking: Randomly masks a continuous time segment on the spectrum, with the mask length being less than 1 / 10 of the total signal length.

[0058] Frequency masking: Randomly masks a continuous frequency range on a spectrum graph. The mask length is less than 1 / 10 of the total length of the frequency dimension.

[0059] Specifically, normalization means performing dataset-level normalization on the original audio signal and Log-Mel spectrogram, which improves training stability and accelerates convergence.

[0060] The structure of multi-task learning networks is explained below. Please refer to [link / reference]. Figure 2 During the training phase, the original audio signal and the Log-Mel spectrogram, both audio features, are input into the multi-task learning network. Specifically, the original audio signal (audio) is input into the temporal encoder to obtain the temporal feature representation q1, and the Log-Mel spectrogram is input into the frequency encoder to obtain the frequency feature representation q2 of the sound sample. Then, q1 and q2 are fused to obtain the multimodal feature q. Finally, the multimodal feature q is fed into five classifiers to obtain the classification results.

[0061] Optionally, please refer to Figure 3The structure of the temporal encoder includes a convolutional network layer and a Transformer encoder. The convolutional network layer extracts local features of the original audio signal through 8 layers of 1D convolution operations and downsamples the time dimension. The Transformer encoder is used for global context modeling to capture the dependencies between features in different time periods.

[0062] Specifically, the raw audio signal is input into a temporal encoder to obtain a temporal feature representation q1 of the sound. The temporal encoder consists of a CNN and a Transformer encoder. The CNN extracts local features of the audio signal through eight layers of 1D convolutional operations and downsamples the temporal dimension. The resulting feature representation is then input into the Transformer encoder, which is used for global context modeling to capture the dependencies between features at different time intervals.

[0063] exist Figure 3 In Chinese and English, the meanings are as follows: Conv 1d represents one-dimensional convolution; Batch Normalization 1d represents one-dimensional batch normalization; Max-Pool 1d represents one-dimensional maximum pooling; Transformer Encoder represents transformer encoder; ConvBlock×5: indicates that the convolutional block (ConvBlock) consisting of convolution (Conv), batch normalization (Batch Normalization) and maximum pooling (Max-Pool) is repeated 5 times.

[0064] Please see Figure 4 The structure of the frequency domain encoder includes eight two-dimensional convolutional layers; among them, layers 2 to 8 are frequency-adaptive dynamic convolutional layers.

[0065] exist Figure 4 In Chinese and English, the meanings are as follows: Mel bands represents the Log-Mel spectrum; Time represents time; Conv represents convolution; Batch Normalization represents batch normalization; Avg-Pool represents average pooling; FDY represents adaptive dynamic convolution; ConvBlock×7 represents a convolutional block (ConvBlock) consisting of adaptive dynamic convolution (FDY), batch normalization (BatchNormalization), and average pooling (Avg-Pool) repeated 7 times.

[0066] The Log-Mel spectrum is input into a frequency domain encoder to obtain the frequency domain representation q2 of the sound sample. The frequency domain encoder is as follows: Figure 4As shown, the frequency encoder uses 8 layers of 2D convolutions to extract local time-frequency features. In this application, the last 7 layers of convolutions do not use traditional 2D convolutions, but are replaced by frequency-adaptive dynamic convolutions (FDY). The FDY frequency-adaptive kernel releases the translational equivariance of the 2D convolutions along the frequency axis to achieve physical consistency with the time-frequency patterns in the sound event. Specifically, the FDY first performs average pooling on the input features along the time dimension, then applies two 1D convolutions along the channel axis to compress the channel dimension into the number of basic convolution kernels, and finally applies Softmax to obtain frequency-adaptive attention weights. The frequency-adaptive weights are used to perform a weighted summation of the basic kernels to generate the frequency-adaptive convolution kernel. This application is the first to apply frequency-adaptive dynamic convolution to a self-supervised learning abnormal sound detection task. The workflow of the frequency-adaptive dynamic convolution is as follows: Figure 5 As shown.

[0067] exist Figure 5 In Chinese and English, the meanings are as follows: attention refers to the attention mechanism; avgpool overtime refers to the average pooling over time; conv1D refers to one-dimensional convolution; BN refers to batch normalization; ReLU refers to the linear rectified function; basis kernels refer to the basis kernels; and conv2D refers to two-dimensional convolution.

[0068] The formula for frequency-adaptive dynamic convolution is as follows: where t represents the time dimension, f represents the frequency dimension, x(t,f) represents the input of the frequency-adaptive dynamic convolution, and W... i b represents the weight of the i-th basis kernel. i Let K represent the bias term of the i-th basic kernel, and K represent the total number of basic kernels, π. i (f,x) represents the frequency-adaptive attention weight of the i-th base convolutional kernel, and y(t,f,x) represents the final output of the frequency-dynamic convolutional layer.

[0069] y i (t,f)=W i *x(t,f)+b i ;

[0070]

[0071] (3) The time-domain feature representation q1 and the frequency-domain feature representation q2 are fused to obtain the multimodal feature q. The fusion method is to splice them along the channel dimension. The fused multimodal feature q can capture the instantaneous dynamic characteristics of the audio signal in the time dimension and the spectral information in the frequency dimension. This fusion mechanism not only preserves the fine-grained temporal details and global energy distribution characteristics of the original signal, but also constructs a more discriminative comprehensive feature representation by mining the potential correlation between the time and frequency domains.

[0072] Then, the fused multimodal features q are input into multiple classifiers.

[0073] In the embodiments of this application, the multiple classifiers include a first classifier, a second classifier, a third classifier, a fourth classifier, and a fifth classifier; the first classifier is used for machine speed range classification; the second classifier is used for temperature range classification; the third classifier is used for microphone placement location classification; the fourth classifier is used for machine type classification; and the fifth classifier is used for data augmentation operation classification.

[0074] Specifically, the multimodal feature q is fed into the first classifier to obtain the classification result. The first classifier has a three-layer perceptron structure, mapping the multimodal feature q to rotational speed range categories. This also corresponds to the first auxiliary task—classifying the machine's rotational speed. The first and second layers use ReLU activation functions, and the last layer uses softmax activation functions, outputting an L-dimensional probability distribution vector (L is the number of rotational speed ranges, and each dimension represents the probability that a sample belongs to the corresponding rotational speed range). The loss function is l1, using cross-entropy loss. The structures of the subsequent second to fifth classifiers are completely similar to the first classifier, except that the activation function and weight matrix of the last layer are slightly different. The formula for the loss function l1 is as follows: where N is the number of samples, L is the number of categories, and y i,c It is the true label of the i-th sample. This represents the predicted probability of the classifier.

[0075]

[0076] Specifically, the multimodal feature q is fed into the second classifier to obtain the classification result. The second classifier maps the multimodal feature q to temperature range categories, which corresponds to the second auxiliary task—classifying temperature ranges. The second classifier adopts a three-layer perceptron structure that is completely consistent with the first classifier: the activation function of the first and second layers is ReLU, and the last layer uses the softmax activation function, outputting a K-dimensional probability distribution vector (K is the number of temperature ranges, and each dimension represents the probability that the sample belongs to the corresponding temperature range). The loss function is l2, using cross-entropy loss. By minimizing l2, the model learns the association between audio features and temperature ranges (for example, in a high-temperature environment, the sound of a machine may have a specific spectral shift due to the thermal expansion of parts, while in a low-temperature environment, different vibration characteristics may appear due to changes in lubrication). The significance of this auxiliary task is to allow the model to capture temperature-related sound characteristics, thereby more accurately distinguishing between "normal sound changes caused by temperature fluctuations" and "real abnormal mechanical sounds" in the main task, improving robustness to environmental interference. The formula for the loss function l2 is as follows: where N is the number of samples, K is the number of categories, and y i,c It is the true label of the i-th sample. This represents the predicted probability of the classifier.

[0077]

[0078] Specifically, the multimodal feature q is fed into the third classifier to obtain the classification result. The third classifier maps the multimodal feature q to the category of microphone placement position, which corresponds to the third auxiliary task—classifying the microphone placement position. The structure of the third classifier is consistent with the first and second classifiers. The first two layers use the ReLU activation function to mine feature patterns related to the acquisition position; the last layer uses the softmax activation function to output an M-dimensional probability distribution (M is the number of position categories), representing the probability of a sample corresponding to each microphone position. The loss function is l3, using cross-entropy loss. By optimizing l3, the model learns the influence of different acquisition positions on sound features. The design significance of this task is to help the model distinguish between "sound feature changes caused by microphone position differences" and "feature changes caused by mechanical anomalies," avoiding mistaking different acquisition conditions for anomalies, thereby enhancing its adaptability to diverse acquisition scenarios in actual deployment. The formula for the loss function l3 is as follows: where N is the number of samples, M is the number of categories, and y i,c It is the true label of the i-th sample. This represents the predicted probability of the classifier.

[0079]

[0080] Optionally, a fourth classifier is used to output the probability of belonging to a centrifuge and to determine outliers based on the probability.

[0081] Specifically, the multimodal feature q is fed into the fourth classifier to obtain the classification result, which is a binary classification problem. The fourth classifier maps the feature vector q to a binary label for the machine type, where a label of 1 represents "centrifuge" and a label of 0 represents "other machine types". This corresponds to the main task—centrifuge abnormal sound detection. Other machine types are considered pseudo-anomalies, prompting the neural network to learn a more general feature representation of the normal sound of a centrifuge. The structure of the fourth classifier is similar to that of the first to third classifiers, but the last layer uses the sigmoid activation function (compressing the output to the [0, 1] interval) to obtain the probability e that the sample belongs to a centrifuge. Then the value of the classification outlier p2 can be derived, which serves as a basis for anomaly judgment. The loss function is l4, using binary cross-entropy loss. By optimizing this loss, the model learns the differences in sound features between centrifuges and other machine types, thereby more accurately focusing on the feature representation of the target machine (centrifuge), providing targeted feature support for subsequent anomaly detection tasks. The formula for calculating the classification outlier p2 of the i-th sample is as follows:

[0082] p2 = 1 - e i ;

[0083] The formula for calculating the loss function l4 is as follows: where N represents the number of samples, e i The probability that the model predicts the i-th sample as "centrifuge" is y. i This represents the true label of the i-th sample.

[0084]

[0085] Specifically, the multimodal feature q is fed into the fifth classifier to obtain the classification result. The fifth classifier maps the multimodal feature q to a combination label of data augmentation methods, which corresponds to the fourth auxiliary task—multi-label classification of which data augmentation operations were used on the input audio samples. This task aims to enable the model to understand the impact of different augmentation strategies on sound features, thereby improving robustness to various environmental noises and disturbances. The first and second layers of the fifth classifier use the ReLU activation function, and the last layer uses the Sigmoid activation function, outputting a T-dimensional vector, where each dimension represents whether the corresponding augmentation method was used (probability value ∈ [0, 1]). The loss function is l5, using multi-label binary cross-entropy loss. By performing multi-label classification of various data augmentation strategies, the fifth classifier enables the model to autonomously identify the impact of different disturbances on sound features, significantly enhancing robustness to changes in environmental noise and recording conditions. The formula for the loss function l5 is as follows: where N is the number of samples, T is the number of classes, and y i,j It is the j-th label of the i-th sample. This represents the predicted probability of the i-th sample for the j-th label.

[0086]

[0087] Total loss function This is a weighted sum of the loss functions of each classifier.

[0088] In this model, α1, α2, α3, α4, and α5 are hyperparameters used to balance the losses of different components. The neural network achieves collaborative learning between the main task and multiple auxiliary tasks by jointly optimizing the total loss function. This multi-task learning method transforms prior constraints such as rotation speed, temperature, position, machine type, and enhancement method into differentiable regularization terms, enabling the neural network to learn more general feature representations of normal sound, greatly enhancing generalization performance and effectively solving the domain offset problem.

[0089]

[0090] Next, in the testing and evaluation phase, the trained multi-task learning network is used as a feature extractor with frozen parameters. The audio data from the test set is input into the pre-trained multi-task learning network to obtain the feature embedding q and the output p2 of the fourth classifier. The feature embedding q will then be used as input for Gaussian clustering.

[0091] It should be noted that the above is a detailed description of the training process. For the description of the application process, please refer to the relevant description. This application will not repeat it.

[0092] Optionally, inputting multimodal features into a trained Gaussian mixture model to generate anomaly scores includes: inputting multimodal features into a Gaussian mixture model, calculating the negative log-likelihood of the multimodal features relative to the Gaussian mixture model, and generating anomaly scores.

[0093] A Gaussian mixture model (GMM) is pre-trained using normal centrifuge sound samples from the training set. The GMM, through a weighted combination of multiple Gaussian distributions, fits the probability distribution of features from normal samples. Finally, the feature embedding q obtained in the feature extraction stage is input into the GMM, and the negative log-likelihood of the feature embedding q relative to the GMM is calculated to obtain the anomaly score p1.

[0094] Optionally, based on the anomaly score and the classified outlier, an anomaly determination is made on the sound signal to be detected, including: performing a standardized weighted summation based on the anomaly score and the classified outlier to obtain a total anomaly score; if the total anomaly score is higher than a set threshold, the sound signal to be detected is determined to be an abnormal sound; if the total anomaly score is not higher than the set threshold, the sound signal to be detected is determined to be a normal sound.

[0095] The total anomaly score is the standardized weighted sum of p1 and p2. Here, p1 is the anomaly score calculated by the Gaussian mixture model, and p2 is the anomaly value derived from the fourth classifier. If the total anomaly score exceeds a set threshold A... θ If the sound is positive, it is an abnormal sound; otherwise, it is a normal sound.

[0096] The formula for calculating the total abnormality score is as follows: in and These are the mean and standard deviation of p1 calculated on the training set. and λ is the mean and standard deviation of p2 calculated on the training set, and λ is a weighting coefficient used to adjust the relative contributions of the two terms.

[0097] The formula for determining an anomaly is as follows: A θ The abnormal threshold is set manually, and P is the abnormal score obtained in the previous step.

[0098]

[0099] In summary, the abnormal sound detection method based on multi-task learning proposed in this application can achieve the following results:

[0100] First, fully utilize the inherent information of the machine sound samples themselves for self-supervised learning.

[0101] This application proposes a novel data acquisition method that controls three variables—temperature, rotation speed, and microphone placement—to increase data diversity. This diverse data increases the difficulty of training the neural network and reduces overfitting. Furthermore, this application utilizes four intrinsic information parameters—temperature, rotation speed, microphone placement, and machine type—for training. These four parameters are automatically recorded and saved during acquisition, requiring no manual annotation or a large number of abnormal sound samples for supervised learning. Compared to supervised learning, this method not only significantly improves efficiency but also learns more general feature representations of normal sound samples, enabling generalization to new machines.

[0102] First, a dual-branch time-frequency feature fusion architecture is used to detect abnormal sounds using multimodal features.

[0103] This application's dual-branch architecture overcomes the limitations of traditional abnormal sound detection methods that rely solely on single-modal features, processing sound signals from two complementary perspectives: the time domain and the frequency domain. Relying solely on the Log-Mel spectrum results in the loss of time-domain details, while relying solely on the raw audio lacks global spectral information. This application designs a dual-branch network with a time-domain encoder (processing the raw audio signal) and a frequency-domain encoder (processing the Log-Mel spectrum). By splicing and fusing time-domain and frequency-domain features along the channel dimension, it preserves the instantaneous dynamic characteristics of the time domain (such as short-duration pulses from mechanical impacts) and the frequency-domain spectral distribution (such as harmonic energy changes). For some short-duration impact anomalies, the time-domain branch can accurately capture the pulse's temporal location, while the frequency-domain branch can simultaneously extract the high-frequency spectral abrupt changes accompanying the impact. The fusion of these two branches significantly reduces the false negative rate.

[0104] Third, to overcome the shortcomings of traditional abnormal sound detection methods that use two-dimensional convolution, this application uses frequency-adaptive dynamic convolution.

[0105] For the frequency domain encoder, this application employs frequency-adaptive dynamic convolution instead of traditional two-dimensional convolution. This is achieved through frequency-dependent attention weights π. i (f,x) generates an adaptive convolution kernel, releasing the translational equivariance on the frequency axis, which conforms to the physical characteristics of sound time-frequency patterns. This enables the frequency domain encoder to more accurately capture subtle feature changes at different frequencies, significantly improving the ability of frequency domain features to distinguish abnormal patterns and avoiding feature mis-extraction caused by insufficient frequency axis adaptation in traditional convolution.

[0106] Fourth, a multi-task learning framework enhances the generality of feature representations.

[0107] In industrial settings, centrifuges often experience significant domain shifts in their operating environment (e.g., temperature differences between the source and target workshops, different speed ranges between new and old equipment, microphone placement adjustments, etc.). Traditional methods are prone to misjudgment during cross-domain testing. This framework, however, achieves significant domain generalization through multi-task constraints. This application constructs a multi-task learning framework that enables neural networks to learn more general feature representations of normal sound samples, effectively solving the domain shift problem. A main task and four auxiliary tasks are cleverly designed. The main task is a binary classification of machine types, treating machines other than centrifuges as pseudo-anomalies. Through "pseudo-anomaly comparison," the model focuses on the unique normal sound features of centrifuges. While the normal sounds of other machines are not abnormal, they differ fundamentally from the acoustic characteristics of centrifuges. In distinguishing between the two, the model automatically filters out "acoustic features not common to centrifuges," learning the stable vibration modes and spectral patterns unique to centrifuges. The four auxiliary tasks target four types of variables most prone to domain shifts in industrial scenarios (machine parameters, environmental conditions, acquisition methods, and noise disturbances), forming a "variable-feature" mapping learning.

[0108] (1) Rotational speed range classification task: By learning the regular changes in sound features at different rotational speeds, the model can distinguish between "normal feature differences caused by rotational speed fluctuations" and "abnormal feature mutations." This significantly reduces the false detection rate caused by domain offset (changes in machine parameters).

[0109] (2) Temperature Range Classification Task: This task addresses the physical effects of temperature on mechanical components (e.g., changes in frictional sound due to a decrease in bearing lubrication coefficient at high temperatures), guiding the model to capture the correlation pattern between temperature and features. When the test environment temperature differs significantly from the training set (e.g., the training set is predominantly 25℃, while the test set contains samples at 40℃), the model can automatically identify feature shifts caused by temperature as within the normal range, avoiding classifying them as abnormal and improving its adaptability to environmental temperature fluctuations.

[0110] (3) Microphone Placement Classification Task: Learn the acoustic attenuation patterns caused by different acquisition positions (e.g., rich high-frequency details in close-range acquisition and higher low-frequency proportion in distant acquisition), enabling the model to isolate the interference of "positional differences" on sound characteristics. For example, the sound characteristics of the same centrifuge collected 1 meter in front of the workshop and 5 meters to the side are significantly different. This task helps the model understand that this difference is a normal phenomenon caused by acquisition conditions, rather than a machine malfunction, thus adapting to diverse microphone installation scenarios in actual deployment.

[0111] (4) Data Augmentation Classification Task: By identifying the perturbations to features caused by augmentation operations such as noise addition and time stretching, the model establishes a "perturbation-feature" mapping relationship. When test samples are interfered with by real environmental noise (such as factory background noise) or recording equipment errors, the model can quickly identify these perturbations as "non-abnormal feature changes," thereby focusing on the abnormal signals of the machine itself. Compared with traditional techniques, this method can effectively reduce false detections caused by complex environmental noise interference and significantly improve the model's adaptability to diverse noise scenarios in actual deployment.

[0112] Loss functions were constructed for each of the five task categories, and the total loss function was expressed as a weighted sum of the five task categories. The constraints of the five task categories were integrated into a unified optimization objective. The main task ensured that the model did not deviate from the learning direction of "normal centrifuge features," while the auxiliary task, through gradient backpropagation, suppressed the model's excessive focus on "features related to interfering variables"—for example, when temperature changes cause feature fluctuations, the loss of the temperature classification task forces the model to classify such fluctuations as "explainable normal changes" rather than abnormal signals. Ultimately, the feature representation learned by the model will strip away the influence of variables such as rotation speed, temperature, and position, focusing on the essential acoustic laws of normal centrifuge operation. This allows the model to maintain stable anomaly detection performance in cross-workshop and cross-condition testing scenarios, effectively solving the domain generalization bottleneck of traditional methods.

[0113] Fifth, fusion of abnormal scores.

[0114] This application standardizes and weights the negative log-likelihood of the Gaussian mixture model and the outliers derived from the fourth classifier to obtain a total anomaly score. This overcomes the limitations of single scores (such as reconstruction error or classification probability alone)—the Gaussian mixture model reflects the degree to which features deviate from the normal distribution (statistical perspective), while the classification model reflects the difference from the normal features of the centrifuge (discriminative perspective). After fusion, the synergistic effect of these two perspectives improves the detection rate of rare anomalies.

[0115] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0116] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0117] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0118] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0119] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0120] It should be noted that the information interaction and execution process between the above systems / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0123] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. An abnormal sound detection method based on multi-task learning, characterized in that, include: Acquire the sound signal to be detected; The sound signal to be detected is preprocessed; wherein the preprocessing process includes at least data augmentation of the sound signal to be detected and generation of a target Log-Mel spectrogram; The sound signal to be detected and the target Log-Mel spectrogram are input into a trained multi-task learning network, which outputs multimodal features and classification outliers. The multi-task learning network includes a dual-branch module and multiple classifiers. The dual-branch module includes a temporal encoder and a frequency-domain encoder. The temporal encoder processes the sound signal to be detected to generate a temporal feature representation. The frequency-domain encoder processes the target Log-Mel spectrogram to generate a frequency-domain feature representation. The multimodal features are generated by fusing the temporal and frequency-domain feature representations. The classifications by the multiple classifiers include machine speed range classification, temperature range classification, microphone placement classification, data augmentation operation classification, and machine type classification. The machine type classification is used to distinguish centrifuges from other machines. The result of the machine type classification is used to determine the classification outliers. The multimodal features are input into the trained Gaussian mixture model to generate anomaly scores; Based on the anomaly score and the classification anomaly value, the sound signal to be detected is determined to be abnormal.

2. The abnormal sound detection method based on multi-task learning as described in claim 1, characterized in that, The multi-task learning network is trained through the following steps: Acquire sound data to obtain the raw audio signal; The original audio signal is preprocessed; the data preprocessing includes dataset partitioning, loading, generating spectral features, data augmentation, and normalization; wherein, dataset partitioning includes training set and test set; The multi-task learning network is trained using audio data from the training set and tested using audio data from the test set.

3. The abnormal sound detection method based on multi-task learning as described in claim 2, characterized in that, The acquisition of sound data includes: Normal and abnormal sound samples from centrifuges and other types of machines are collected using a microphone; the raw audio signal includes both the normal and abnormal sound samples. Specifically, the amount of data for the normal sound samples is increased by controlling three variables: rotation speed, microphone placement, and temperature, while the abnormal sound samples are obtained by using the sound of a damaged machine.

4. The abnormal sound detection method based on multi-task learning as described in claim 2, characterized in that, The generated spectral features include: The original audio signal is subjected to a short-time Fourier transform to obtain a spectrum. Mel filtering and logarithmic operation are performed on the spectrum to obtain the Log-Mel spectrum.

5. The abnormal sound detection method based on multi-task learning as described in claim 4, characterized in that, For the original audio signal, the data enhancement methods include: pitch shifting, time stretching, adding noise, and time shifting; For the Log-Mel spectrum, the data augmentation methods include time masking and frequency masking.

6. The abnormal sound detection method based on multi-task learning as described in claim 2, characterized in that, The structure of the temporal encoder includes a convolutional network layer and a Transformer encoder; The convolutional network layer extracts local features of the original audio signal through 8 layers of 1D convolution operations and downsamples the time dimension. The transformer encoder is used for global context modeling to capture the dependencies of features in different time periods. The structure of the frequency domain encoder includes eight two-dimensional convolutional layers; wherein, layers 2 to 8 are frequency-adaptive dynamic convolutional layers.

7. The abnormal sound detection method based on multi-task learning as described in claim 2, characterized in that, The multiple classifiers include a first classifier, a second classifier, a third classifier, a fourth classifier, and a fifth classifier; The first classifier is used for classifying machine speed ranges; The second classifier is used for temperature range classification; The third classifier is used to classify the microphone placement location; The fourth classifier is used for machine type classification; The fifth classifier is used for data augmentation operation classification.

8. The abnormal sound detection method based on multi-task learning as described in claim 7, characterized in that, The fourth classifier is used to output the probability of belonging to a centrifuge, and to determine outliers based on the probability.

9. The abnormal sound detection method based on multi-task learning as described in claim 1, characterized in that, The step of inputting the multimodal features into the trained Gaussian mixture model to generate anomaly scores includes: The multimodal features are input into the Gaussian mixture model, and the negative log-likelihood of the multimodal features relative to the Gaussian mixture model is calculated to generate the anomaly score.

10. The abnormal sound detection method based on multi-task learning as described in claim 1, characterized in that, The step of determining anomalies in the sound signal to be detected based on the anomaly score and the classification anomaly value includes: The total anomaly score is obtained by performing a standardized weighted summation based on the anomaly score and the classified outliers. If the total anomaly score is higher than a set threshold, the sound signal to be detected is determined to be an abnormal sound. If the total abnormal score is not higher than the set threshold, then the sound signal to be detected is determined to be normal sound.

Images