A general data enhancement method and device for industrial acoustic signals

Abstract

This application proposes a general data augmentation method and apparatus for industrial acoustic signals, belonging to the field of acoustic signal fault diagnosis technology. The method includes: acquiring actual acoustic signal data to be augmented; preprocessing the actual acoustic signal data; extracting features from the preprocessed actual acoustic signal data, including Mel-frequency cepstral coefficients and logarithmic fundamental frequency; inputting the Mel-frequency cepstral coefficients and logarithmic fundamental frequency into a pre-trained star-shaped generative adversarial network for data augmentation, and obtaining the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier. This application improves the data augmentation effect while solving the problems of insufficient and imbalanced data, reducing the number of training generative models, and improving model training efficiency, thus having broad application prospects and practical value.

Description

Technical Field

[0001] This invention belongs to the field of acoustic signal fault diagnosis technology, specifically relating to a general data enhancement method and apparatus for industrial acoustic signals. Background Technology

[0002] In the context of intelligent manufacturing, industrial equipment is exhibiting a trend towards high automation and large-scale production. Fault detection in industrial equipment has become a crucial component of condition-based maintenance management, serving as a vital means to ensure safe and reliable equipment operation. Currently, with the continuous development of advanced acoustic-vibration sensors and big data-related technologies, machine learning and deep learning have been successfully applied in practical industrial fault detection and diagnosis, overcoming the low accuracy and efficiency problems of traditional manual inspection. Compared to vibration signal detection technology, industrial acoustic signal detection has a wider range of applications, is not limited by space, and offers advantages such as non-contact measurement, high installation flexibility, and low cost. Therefore, analyzing the sound signals generated during machine operation and extracting their acoustic signature features to achieve real-time and accurate fault identification holds great promise for ensuring industrial safety.

[0003] However, in the actual operation of industrial equipment, situations involving prolonged fault states are relatively rare, resulting in a limited and incomplete amount of historical fault cases and information. This scarcity and imbalance of fault data makes it difficult to train efficient and accurate fault diagnosis models, leading to poor generalization ability of data-driven fault detection and diagnosis methods, which struggle to meet the fault diagnosis needs of actual industrial production. To address this issue, some techniques exist, such as using computer simulation, data transformation, and generative adversarial networks to augment training samples. However, these data augmentation techniques still suffer from problems, such as low quality, insufficient diversity, and low efficiency of the generated fault samples. Therefore, to solve the problems of lack of fault training data and class imbalance in practical industrial fault diagnosis tasks, and to further improve the detection performance of existing fault diagnosis methods, a general data augmentation method for fault detection and diagnosis of industrial acoustic signals is needed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application proposes a general data augmentation method and apparatus for industrial acoustic signals, aiming to solve the problems of insufficient and imbalanced training data in practical industrial fault diagnosis tasks, and improve the detection accuracy and generalization of existing fault diagnosis models. This method can generate high-quality, diverse industrial acoustic signal fault data using a relatively simple network structure, thereby enhancing training data while improving model training efficiency.

[0005] In a first aspect, this application proposes a general data enhancement method for industrial acoustic signals, comprising:

[0006] Collect actual acoustic signal data to be enhanced;

[0007] The actual acoustic signal data is preprocessed;

[0008] Feature extraction is performed on the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency.

[0009] The Mel frequency cepstral coefficients and logarithmic fundamental frequency are input into a pre-trained star-shaped generative adversarial network for data augmentation to obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier.

[0010] The generator is used to generate data for multiple target domains. The generator structure includes: a first convolutional layer, a first batch normalization layer, a first gated linear unit, four first downsampling layers, a first transposed convolutional layer, a second batch normalization layer, a second gated linear unit, three first upsampling layers, and a second transposed convolutional layer.

[0011] The discriminator is used to determine whether the data of multiple target domains output by the generator are consistent with the target label. The structure of the discriminator includes: a second convolutional layer, a third batch normalization layer, a third gated linear unit, three second downsampling layers, a third convolutional layer, an activation function layer, a first pooling layer, a fourth convolutional layer, and an output true or false layer.

[0012] The auxiliary classifier is used to determine the label category of the data in multiple target domains output by the generator. The structure of the auxiliary classifier includes: a fifth convolutional layer, a fourth batch normalization layer, a fourth gated linear unit, a fully connected layer, three third downsampling layers, a sixth convolutional layer, a softmax layer, and a second pooling layer.

[0013] The training process of the pre-trained star-shaped generative adversarial network includes:

[0014] Step S1: Construct samples for a star-shaped generative adversarial network and their corresponding labels;

[0015] Step S2: Define the structure of the generator, the discriminator, and the auxiliary classifier; define the loss function of the generator, the discriminator, and the auxiliary classifier; initialize the learning rate and learning rate decay of the generator, the learning rate and learning rate decay of the discriminator, the weight hyperparameters of the generator's loss function, the weight hyperparameters of the discriminator's loss function, and the weight hyperparameters of the auxiliary classifier's loss function.

[0016] Step S3: Randomly shuffle the samples and corresponding sample labels of the star-shaped generative adversarial network to obtain the shuffled samples and corresponding sample labels.

[0017] Step S4: Use the disordered samples and their corresponding labels to train the generator, discriminator, and auxiliary classifier;

[0018] Step S5: Optimize the generator using the first Adam optimizer, optimize the discriminator using the second Adam optimizer, and optimize the auxiliary classifier using the third Adam optimizer;

[0019] Step S6: Determine whether the current number of training iterations is less than the first set value; if the current number of training iterations is less than the first set value, proceed to step S7; if the current number of training iterations is greater than or equal to the first set value, use the first Adam optimizer to update the learning rate and learning rate decay rate of the generator, use the second Adam optimizer to update the learning rate and learning rate decay rate of the discriminator, and use the third Adam optimizer to update the learning rate and learning rate decay rate of the auxiliary classifier.

[0020] Step S7: Calculate the loss function of the generator, the loss function of the discriminator, and the loss function of the auxiliary classifier using the current learning rate and the learning rate decay rate;

[0021] Step S8: Determine whether the current number of training iterations has reached the second set value. If the second set value is greater than the first set value, proceed to step S9; otherwise, increment the current number of training iterations by 1 and proceed to step S4 to continue training.

[0022] Step S9: Construct and save the pre-trained star-shaped generative adversarial network using the current generator loss function, discriminator loss function, and auxiliary classifier loss function.

[0023] The generator's loss function is obtained by weighted summation of the generator's adversarial loss function, the generator's domain classification loss function, the generator's cycle consistency loss function, and the generator's identity mapping loss function.

[0024] The loss function of the discriminator is the adversarial loss function of the discriminator;

[0025] The loss function of the auxiliary classifier is the domain classification loss function of the discriminator.

[0026] The adversarial loss function, domain classification loss function, cycle consistency loss function, identity mapping loss function, adversarial loss function, and domain classification loss function of the generator are calculated as follows:

[0027]

[0028] in, Let the generator be the adversarial loss function. Let the discriminator be the adversarial loss function. The domain classification loss function for the generator. Let the domain classification loss function be the discriminator. Let the generator's cycle consistency loss function be . Let G(x,c) be the identity mapping loss function of the generator, x be the source domain consisting of samples of all classes, y be the target domain consisting of the labels corresponding to the samples, c be the label of the target domain, and c′ be the label of the source domain; G(x,c) is the data generated using the data of the source domain x to generate the data of the target domain y labeled c; D(G(x,c),c) is the discriminator that distinguishes whether the data generated by the generator is similar to the data of the target domain labeled c; ·~p(·) is the distribution of the data; p C (c|G(x,c)) represents the probability of class c in the target domain y generated by the generator, p C (c|y) represent the probabilities of the true class c in the target domain y generated by the generator; E represents the mathematical expectation.

[0029] Secondly, this application proposes a general data enhancement device for industrial acoustic signals, comprising:

[0030] The data acquisition module is used to acquire the actual acoustic signal data to be enhanced.

[0031] The data preprocessing module is used to preprocess the actual acoustic signal data;

[0032] The feature extraction module is used to extract features from the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency.

[0033] The data augmentation module is used to input the Mel frequency cepstral coefficients and logarithmic fundamental frequency into a pre-trained star-shaped generative adversarial network to perform data augmentation and obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier.

[0034] Thirdly, this application proposes an electronic device comprising: one or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the general data enhancement method for industrial acoustic signals.

[0035] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the general data enhancement method for industrial acoustic signals.

[0036] Beneficial effects:

[0037] This application proposes a general data augmentation method and apparatus for industrial sound signals. This application uses the StarGAN network for data augmentation, which significantly improves the data augmentation effect. At the same time, it solves the problems of insufficient and imbalanced data. This application reduces the number of training models by simplifying the network structure of the generator, discriminator and auxiliary classifier, thereby improving the model training efficiency. This application has broad application prospects and practical value. Attached Figure Description

[0038] Figure 1 A flowchart of a general data enhancement method for industrial acoustic signals according to an embodiment of this application;

[0039] Figure 2 A schematic flowchart of a general data enhancement method for industrial acoustic signals according to an embodiment of this application;

[0040] Figure 3 The star-shaped generative adversarial network model structure diagram of this application embodiment is shown in (a) generator structure diagram, (b) discriminator structure diagram, and (c) auxiliary classifier structure diagram.

[0041] Figure 4 Flowchart of the training process of the pre-trained star-shaped generative adversarial network in this application embodiment;

[0042] Figure 5 Comparison of time-domain waveforms and spectrograms of different types of sound signals in embodiments of this application;

[0043] Figure 6 This application provides a schematic diagram of a general data enhancement device for industrial acoustic signals. Detailed Implementation

[0044] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] Among the techniques for addressing the problem of limited training data in this field, several data augmentation methods exist. Traditional computer simulation methods can quickly generate a large number of training samples by scaling, slicing, and flipping one-dimensional acoustic signals. However, the data generated by these traditional data transformation methods lacks diversity, potentially leading to information loss or distortion, and in some cases, introducing additional noise or interference signals, thereby reducing the detection accuracy of fault diagnosis models. Another method to address imbalanced datasets is the Synthetic Minority Over-sampling Technique (SMOTE). This method analyzes minority class samples for random sampling, artificially synthesizes new samples, and adds them to the dataset for data augmentation. However, this method is difficult to set the sampling rate and does not consider the overall data distribution, easily leading to model overfitting. This results in the diagnostic model learning overly specific information and lacks generalization ability, thus it cannot be considered a general data augmentation solution. Another background technique, most closely related to this application, utilizes Generative Adversarial Networks (GANs) for data augmentation. This method consists of a generator and a discriminator. The generator is responsible for generating new synthetic samples, and the discriminator is responsible for evaluating the differences between the generated samples and the target samples. Through continuous training and game-like interaction, the generator gradually produces data that increasingly resembles real-world industrial fault samples. However, basic generative adversarial networks (GANs) are prone to gradient vanishing and pattern collapse problems. Although further research, such as Conditional Generative Adversarial Networks (CGANs) and CycleGANs, has improved upon basic GANs to enhance the quality of generated data, issues remain, including insufficient diversity of generated data and low data augmentation efficiency due to the need to train multiple generative models when handling multi-domain generation tasks.

[0046] In summary, the aforementioned data augmentation techniques still have some problems, such as low quality and insufficient diversity of the generated industrial data, and low training efficiency. Therefore, this application proposes a general data augmentation method and apparatus for industrial sound signals, aiming to solve the problems of insufficient data and class imbalance in fault detection and diagnosis. The method in this application is based on StarGAN (Star Generative Adversarial Network), consisting of a generator, a discriminator, and an auxiliary classifier, used to achieve data augmentation. This application employs a novel deep learning model based on StarGAN (Generative Adversarial Network). By learning from a limited amount of existing normal and faulty sample training data, it realizes a general data augmentation method for fault detection and diagnosis of industrial sound signals. The generator of this method learns the time-spectral features of the extracted industrial sound signals to generate new and diverse fault samples as an expansion of the training set, solving the problem of limited training data in practical industrial fault diagnosis tasks and improving the detection accuracy and generalization of existing fault diagnosis models. During the generator training process, auxiliary labels are introduced as additional inputs to guide the generator in generating samples related to specific fault types, enhancing the relevance of the generated data and making the generated fault data more accurate and controllable. Compared with other generative adversarial networks, this method can generate sound signals from normal sound signals to different types of fault sound signals using only one generator, one discriminator, and one auxiliary classifier. This simplifies the network structure, reduces model complexity, and improves training efficiency. This application provides a scalable industrial acoustic signal data augmentation method that supports acoustic signal generation across multiple normal / fault state categories, enhancing the training dataset for fault diagnosis models. It possesses strong versatility and practicality, applicable to acoustic signal fault detection and diagnosis tasks in various industrial fields. By using the StarGAN network for data augmentation, this application generates high-quality, diverse fault data similar to actual industrial acoustic signals, improving fault detection and diagnosis performance. It overcomes the scarcity of fault samples in real-world industrial scenarios by generating a large amount of high-quality fault data, while auxiliary labels guide the generator to generate samples related to specific fault types, alleviating data imbalance. The synergistic effect of a generator, a discriminator, and an auxiliary classifier simplifies the network structure, enabling one-to-many domain model training, reducing the number of training models, and improving training efficiency. This application has broad application prospects and practical value. Experiments were conducted on a dataset of actual transformer equipment operating status acoustic signals, demonstrating that this method can serve as a general data augmentation solution applicable to various industrial equipment acoustic signal fault detection and diagnosis fields.

[0047] Example 1:

[0048] This embodiment proposes a general data enhancement method for industrial acoustic signals, such as... Figure 1 , Figure 2 As shown, it includes:

[0049] Step S100: Acquire the actual acoustic signal data to be enhanced;

[0050] In this embodiment, it is understood that: historical actual acoustic signal data needs to be collected first to train the star-shaped generative adversarial network (GAN). After the pre-trained GAN completes training, as described in detail in step S1, when data augmentation is needed, real-time actual acoustic signal data to be augmented is collected, or historical actual acoustic signal data to be augmented can be collected as a training set for data augmentation. The real-time actual acoustic signal data or training set includes actual acoustic signal data under normal and various fault categories of operating states. The actual acoustic signal data refers to acoustic signal data collected in actual industrial settings.

[0051] Step S101: Preprocess the actual acoustic signal data;

[0052] In this embodiment, in order to fully capture the key features of industrial sound signals, the actual collected sound signal data is first preprocessed by framing, windowing and short-time Fourier transform using the librosa library in Python to obtain the preprocessed actual sound signal data.

[0053] Step S102: Extract features from the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency.

[0054] In this embodiment, Mel frequency cepstral coefficients (MFCCs) and logarithmic fundamental frequency (Log F0) features are extracted from each preprocessed actual acoustic signal data. The Mel frequency cepstral coefficients and logarithmic fundamental frequency are then normalized to zero mean and variance, and used as input to the pre-trained star-shaped generative adversarial network.

[0055] Typically, feature extraction from sound signals uses either the original one-dimensional time-domain features or frequency-domain features derived through Fourier transform (such as zero-crossing rate, energy spectrum, and formants). However, unlike speech signals, which are primarily generated by vocal cord vibrations, industrial sound signals originate from various mechanical, electrical, and physical systems, exhibiting diverse generation methods. Industrial sound signals display greater complexity and randomness; therefore, using only the original time-domain features (which only describe signal changes over time and have weak analytical capabilities for frequency components) and frequency-domain features (with uniform linear frequency scales, unable to target key frequency information in industrial sound signals; and requiring greater computational complexity) cannot adequately capture the crucial features of industrial sound signals. However, Mel frequency cepstral coefficients are obtained through Mel transform (a non-parametric domain representation based on the human auditory perception system, capable of mapping linear frequency scales to perceptually relevant scales, mimicking how humans perceive and distinguish different frequencies). However, using only Mel frequency cepstral coefficients limits frequency resolution. Therefore, this invention also introduces logarithmic fundamental frequency as a supplement, which enables better harmonic analysis and captures frequency changes and periods of industrial sound signals.

[0056] Step S103: Input the Mel frequency cepstral coefficients and logarithmic fundamental frequency into the pre-trained star-shaped generative adversarial network to perform data augmentation and obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier.

[0057] In this embodiment, data augmentation is achieved by constructing a StarGAN model, which is the core content of this application. This model consists of a generator (G), a discriminator (D), and an auxiliary classifier (C), realizing the conversion and generation of acoustic signal features under various operating states in industrial scenarios. In this data augmentation stage, by training the generator, discriminator, and classifier, the model can learn the mapping relationship between acoustic signals in different operating state domains of the equipment, transforming the input normal state acoustic signal into acoustic signals of different fault types. This data augmentation method can effectively increase the diversity of training data and improve the generalization ability and fault detection accuracy of the fault diagnosis model.

[0058] The generator, such as Figure 3 As shown in (a), the generator structure used to generate data for multiple target domains includes: a first convolutional layer, a first batch normalization layer, a first gated linear unit, four first downsampling layers, a first transposed convolutional layer, a second batch normalization layer, a second gated linear unit, three first upsampling layers, and a second transposed convolutional layer.

[0059] In this embodiment, conventional generator methods typically use one-dimensional CNNs to extract features. The generator G network model designed in this application is a 2-1-2 CNN. This is because upsampling and downsampling using 2D convolutions preserve the original input signal structure while achieving feature transformation; one-dimensional convolutions primarily extract important feature information and capture overall dynamic changes in the features. Specifically, the generator G network model is constructed by sequentially passing the input data through convolution, batch normalization, gated linear units, four downsampling operations, transposed convolution, batch normalization, gated linear units, three upsampling operations, and transposed convolution to finally obtain the output data. The generator's function is to generate various target domain Y data using source domain X samples and target sample labels c as input.

[0060] The discriminator, such as Figure 3 As shown in (b), the discriminator is used to determine whether the data of multiple target domains output by the generator are consistent with the target label. The discriminator structure includes: a second convolutional layer, a third batch normalization layer, a third gated linear unit, three second downsampling layers, a third convolutional layer, an activation function layer, a first pooling layer, a fourth convolutional layer, and an output true or false layer.

[0061] In this embodiment, the discriminator uses a two-dimensional CNN to extract input features (the purpose is to more fully capture the features of the two-dimensional Mel spectrum of the input); in addition, the last layer of the discriminator usually uses a fully connected layer to distinguish the similarity between the two input signals. However, in the adversarial training process of real industrial sound signals and generated industrial sound signals, in order to reduce the difficulty of multi-parameter learning of the discriminator, this application uses a convolutional layer instead to realize the output of the discrimination results between the generated samples and real samples.

[0062] In this embodiment, the auxiliary classifier, such as Figure 3 As shown in (c), the structure of the auxiliary classifier used to determine the label category of the data in the multiple target domains output by the generator includes: a fifth convolutional layer, a fourth batch normalization layer, a fourth gated linear unit, a fully connected layer, three third downsampling layers, a sixth convolutional layer, a softmax layer, and a second pooling layer.

[0063] In this embodiment, the process of constructing the auxiliary classifier C network model involves sequentially processing the input data through convolution, batch normalization, gated linear units, fully connected layers, three downsampling operations, convolution again, softmax classification, and pooling, ultimately outputting the classification result. The classifier acts as an auxiliary discriminator, used to determine which label category in the target domain Y the data generated by the generator G belongs to, thereby optimizing the entire data augmentation training process.

[0064] The pre-trained star-shaped generative adversarial network, such as Figure 4 As shown, the training process includes:

[0065] Step S1: Construct samples for a star-shaped generative adversarial network and their corresponding labels;

[0066] In this embodiment, the sample construction is similar to steps S100 to S102. First, historical actual sound signal data is collected, including historical actual sound signal data of normal and various fault categories of operation. Second, the historical actual sound signal data is preprocessed, including: framing, windowing, and short-time Fourier transform, to obtain preprocessed historical actual sound signal data. Finally, feature extraction is performed on the preprocessed historical actual sound signal data. The extracted features include: Mel-frequency cepstral coefficients and logarithmic fundamental frequency. The Mel-frequency cepstral coefficients and logarithmic fundamental frequency extracted from the historical signal data are used as samples, and the corresponding real fault types are used as the corresponding sample labels to train the star-shaped generative adversarial network, resulting in a pre-trained star-shaped generative adversarial network.

[0067] Step S2: Define the structure of the generator, the discriminator, and the auxiliary classifier; define the loss function of the generator, the discriminator, and the auxiliary classifier; initialize the learning rate and learning rate decay of the generator, the learning rate and learning rate decay of the discriminator, the weight hyperparameters of the generator's loss function, the weight hyperparameters of the discriminator's loss function, and the weight hyperparameters of the auxiliary classifier's loss function.

[0068] In this embodiment, the following four preparations are required:

[0069] (1) Define the training hardware device. The training device in this embodiment contains at least one GPU (graphics processing unit) by default to improve the computing speed.

[0070] (2) Define the structure of the generator, the discriminator, and the auxiliary classifier. The network model structures of the generator G, discriminator D, and auxiliary classifier C are as follows: Figure 3 (a) Figure 3 (b) and Figure 3 As shown in (c), for a detailed description, please refer to the description of the generator, discriminator and auxiliary classifier in step S103. This embodiment will not repeat the description.

[0071] (3) Define the loss function of the generator, the loss function of the discriminator, and the loss function of the auxiliary classifier;

[0072] The generator's loss function is obtained by weighted summation of the generator's adversarial loss function, the generator's domain classification loss function, the generator's cycle consistency loss function, and the generator's identity mapping loss function.

[0073] The loss function of the discriminator is the adversarial loss function of the discriminator;

[0074] The loss function of the auxiliary classifier is the domain classification loss function of the discriminator.

[0075] In this embodiment, the loss functions in the generator G, discriminator D, and classifier C and They are defined as follows:

[0076]

[0077] in, Let the generator be the adversarial loss function. Let the discriminator be the adversarial loss function. The domain classification loss function for the generator. Let the domain classification loss function be the discriminator. Let the generator's cycle consistency loss function be . λ is the identity mapping loss function for the generator. cls Let λ be the weight hyperparameter of the domain classification loss of the generator G. cyc Let λ be the weight hyperparameter of the cycle consistency loss of generator G. id Let be the weight hyperparameters of the identity mapping loss of the generator G.

[0078] The adversarial loss function, domain classification loss function, cycle consistency loss function, identity mapping loss function, adversarial loss function, and domain classification loss function of the generator are calculated as follows:

[0079]

[0080] in, Let the generator be the adversarial loss function. Let the discriminator be the adversarial loss function. The domain classification loss function for the generator. Let the domain classification loss function be the discriminator. Let the generator's cycle consistency loss function be . Let G(x,c) be the identity mapping loss function of the generator, x be the source domain consisting of samples of all classes, y be the target domain consisting of the labels corresponding to the samples, c be the label of the target domain, and c′ be the label of the source domain; G(x,c) is the data generated using the data of the source domain x to generate the data of the target domain y labeled c; D(G(x,c),c) is the discriminator that distinguishes whether the data generated by the generator is similar to the data of the target domain labeled c; ·~p(·) is the distribution of the data; p C(c|G(x,c)) represents the probability of class c in the target domain y generated by the generator, p C (c|y) represent the probabilities of the true class c in the target domain y generated by the generator; It represents the mathematical expectation.

[0081] Generative adversarial loss and domain classification loss functions are typically used to train generative models; however, due to the complexity of modeling industrial acoustic signals, relying solely on these two loss functions can lead to instability during training. Furthermore, the scarcity of anomalous events in reality results in insufficient training samples for the generator and discriminator, making it difficult for the generator to capture the features of industrial acoustic signals and resulting in low-quality generated samples. Therefore, to effectively capture the temporal sequence and structural spatial features of acoustic signals generated relative to the operation of industrial equipment, a cycle consistency loss is introduced. Simultaneously, to ensure feature consistency between the generated acoustic signals and real acoustic signals, an identity mapping loss is introduced. The purpose of these two loss functions is to make the generated signals more realistic and more similar to actual acoustic signals, while simultaneously making the generator training process more stable.

[0082] In this embodiment, the domain classification loss function serves to assist the discriminator in generating an acoustic signal that corresponds to the set label category c, while also making the generated acoustic signal closer to the real acoustic signal.

[0083] The network model training parameters are set, and these parameters can be modified according to different application scenarios. In this invention, data augmentation verification will be performed using audio signals of various fault states from substation transformer equipment that have been actually collected. The specific training parameters are set as follows:

[0084] (4) Initialize the learning rate and learning rate decay rate of the generator, the learning rate and learning rate decay rate of the discriminator, the weight hyperparameters of the generator's loss function, the weight hyperparameters of the discriminator's loss function, and the weight hyperparameters of the auxiliary classifier's loss function. The specific initial values ​​are as follows:

[0085] ① Learning rate and learning rate decay settings:

[0086] The generator's learning rate: self.g_lr = 1 × 10 -4 ;

[0087] The discriminator's learning rate: self.d_lr = 1 × 10 -4 ;

[0088] The classifier's learning rate: self.c_lr = 1 × 10 -4 ;

[0089] The generator's learning rate decay rate is: self.decay_g = 1 × 10^6 -9 ;

[0090] The learning rate decay of the discriminator: self.decay_d = 1 × 10 -9 ;

[0091] The learning rate decay of the classifier: self.decay_c = 1 × 10 -9 ;

[0092] The number of iterations at which the learning rate begins to decay: self.start_decay = 1 × 10 3 ;

[0093] ② Weight hyperparameter settings for the loss function in the generator:

[0094] The weight hyperparameter of the cycle consistency loss function is: self.lambda_cycle = 3;

[0095] Domain classification loss function weight hyperparameter: self.lambda_classifier = 2;

[0096] The weight hyperparameter of the identity mapping loss function is: self.lambda_identity = 2;

[0097] ③ Model saving settings:

[0098] Save the model every 2000 iterations: self.step_save = 2000.

[0099] Step S3: Randomly shuffle the samples and corresponding sample labels of the star-shaped generative adversarial network to obtain the shuffled samples and corresponding sample labels.

[0100] In this embodiment, before each training round begins, it is necessary to generate and shuffle random data pairs for training data of multiple categories based on the data label categories, and then perform a training round cycle.

[0101] Step S4: Use the disordered samples and their corresponding labels to train the generator, discriminator, and auxiliary classifier;

[0102] Step S5: Optimize the generator using the first Adam optimizer, optimize the discriminator using the second Adam optimizer, and optimize the auxiliary classifier using the third Adam optimizer;

[0103] In this embodiment, three Adam optimizers are created: a first Adam optimizer, a second Adam optimizer, and a third Adam optimizer, which are used to update the parameters of the generator, discriminator, and classifier models, respectively. The optimizers control their parameters according to the learning rates g_lr, d_lr, and c_lr set in step S2, so as to gradually optimize them and better fit the training data.

[0104] Step S6: Determine whether the current number of training iterations is less than the first set value; if the current number of training iterations is less than the first set value, proceed to step S7; if the current number of training iterations is greater than or equal to the first set value, use the first Adam optimizer to update the learning rate and learning rate decay rate of the generator, use the second Adam optimizer to update the learning rate and learning rate decay rate of the discriminator, and use the third Adam optimizer to update the learning rate and learning rate decay rate of the auxiliary classifier.

[0105] Step S7: Calculate the loss function of the generator, the loss function of the discriminator, and the loss function of the auxiliary classifier using the current learning rate and the learning rate decay rate;

[0106] Step S8: Determine whether the current number of training iterations has reached the second set value. If the second set value is greater than the first set value, proceed to step S9; otherwise, increment the current number of training iterations by 1 and proceed to step S4 to continue training.

[0107] Step S9: Construct and save the pre-trained star-shaped generative adversarial network using the current generator loss function, discriminator loss function, and auxiliary classifier loss function.

[0108] In practice, as the number of training epochs increases, it is determined whether the current training epoch exceeds a set value of 1000 (the first set value of 1000 can be defined according to the application scenario). If the current training epoch exceeds 1000, the first Adam optimizer updates the generator's learning rate and its decay rate, the second Adam optimizer updates the discriminator's learning rate and its decay rate, and the third Adam optimizer updates the auxiliary classifier's learning rate and its decay rate. The learning rates of the generator, discriminator, and classifier are updated with a decay rate of 1×10⁻⁹. If the current training epoch does not exceed 1000, the generator learning rate is: self.g_lr = 1×10⁻⁴; the discriminator learning rate is: self.d_lr = 1×10⁻⁴; and the classifier learning rate is: self.c_lr = 1×10⁻⁴. Then, calculate the loss functions of the generator, discriminator, and auxiliary classifier using the current learning rate and the learning rate decay rate, referring to formulas (1) to (9). Determine whether the current training rounds have reached the set value of 2000 (the second set value of 2000 can be defined according to the application scenario); if the training rounds have reached the set value of 2000, construct and save the pre-trained star-shaped generative adversarial network using the current loss functions of the generator, discriminator, and auxiliary classifier; if the training rounds have not reached the set value of 2000, increment the current training count by 1 and proceed to step S4 to continue training.

[0109] Figure 5 This section presents a comparison of the time-domain waveforms and spectrograms of different categories of acoustic signals generated using the star-shaped generative adversarial network (GAN) data augmentation method proposed in this invention. To verify the invention's ability to generate different categories of industrial acoustic signals, this embodiment uses actually collected substation transformer operating status acoustic signal data to train the generation model, generating four different fault state acoustic signals using normal state audio. Figure 5 The mid-time domain waveforms show that the generated acoustic signals of the four different fault states are quite similar. Furthermore, comparing their spectrograms reveals that the energy of the generated acoustic signals and the original target fault acoustic signals remains largely consistent across the entire frequency band. Therefore, the data enhanced by the method of this invention can be used as supplementary data for training the fault diagnosis model.

[0110] Table 1 shows the comparison results of the three fault diagnosis models before and after using the data augmentation method of this invention. This embodiment uses actual collected transformer operating status audio data and generated operating status audio data for verification. Before data augmentation, 80% of the original data was used as the training set; after data augmentation, 80% of the original data plus the augmented data generated by the StarGAN model of this invention were used together as the training set. To ensure fairness, 20% of the original data was used uniformly as the test set, and the SVM, BP, and LSTM diagnostic models all used default parameter settings. The recognition accuracy results of the three fault diagnosis models before and after using data augmentation are shown in Table 1. As can be seen from Table 1, after using the data augmentation method of this invention, the recognition accuracy of all three diagnostic models is improved compared to training with only the original data.

[0111] Table 1 Comparison of identification results of three fault diagnosis models before and after using the data augmentation method of this invention.

[0112]

[0113] Therefore, through experimental verification using a dataset of acoustic signals related to the operating status of transformer equipment, it can be concluded that the star-shaped generative adversarial network model proposed in this invention can serve as a general data augmentation method to generate acoustic signal samples of different domain categories as an extension of the training data for industrial fault detection and diagnosis models, thereby improving the accuracy of fault identification.

[0114] This application achieves the following technical effects:

[0115] 1. Improved data augmentation effect: By using the StarGAN network for data augmentation, this invention can generate high-quality and diverse fault data similar to actual industrial acoustic signals, which is beneficial to improving the performance of fault detection and diagnosis;

[0116] 2. Addressing the issues of insufficient and imbalanced data: By generating a large amount of high-quality fault data, the problem of scarce fault samples in actual industrial scenarios is overcome. At the same time, the generator is guided by auxiliary labels to generate samples related to specific fault types, thus alleviating the problem of data imbalance.

[0117] 3. Improved training efficiency: By working together with a generator, a discriminator, and an auxiliary classifier, the network structure is simplified, enabling one-to-many domain model training, reducing the number of training models, and improving model training efficiency.

[0118] 4. It has broad application prospects and practical value: The present invention has been tested on a dataset of acoustic signals of transformer equipment operation status that has been collected in practice, which proves that the method can be used as a general data enhancement solution and applied to the field of acoustic signal fault detection and diagnosis of various industrial equipment in practice.

[0119] Example 2:

[0120] This embodiment proposes a general data enhancement device for industrial acoustic signals, such as... Figure 6 As shown, it includes: a data acquisition module, a data preprocessing module, a feature extraction module, and a data augmentation module; the data acquisition module, data preprocessing module, feature extraction module, and data augmentation module are connected in sequence.

[0121] The data acquisition module is used to acquire the actual acoustic signal data to be enhanced.

[0122] The data preprocessing module is used to preprocess the actual acoustic signal data;

[0123] The feature extraction module is used to extract features from the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency.

[0124] The data augmentation module is used to input the Mel frequency cepstral coefficients and logarithmic fundamental frequency into a pre-trained star-shaped generative adversarial network to perform data augmentation and obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier.

[0125] Example 3:

[0126] This embodiment proposes an electronic device, including: one or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the general data enhancement method for industrial sound signals.

[0127] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the general data enhancement method for industrial sound signals as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0128] The processor is used to execute all or part of the steps in the general data enhancement method for industrial acoustic signals as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0129] The processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the general data enhancement method for industrial acoustic signals described in the above embodiments.

[0130] Example 4:

[0131] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0132] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the general data enhancement method for industrial sound signals described in the various embodiments of this application.

[0133] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes, on which computer programs are stored. When the computer program is executed by a processor, it can implement the various steps of the general data enhancement method for industrial sound signals described above.

[0134] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0135] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A general data enhancement method for industrial acoustic signals, characterized in that, include: Collect actual acoustic signal data to be enhanced; The actual acoustic signal data is preprocessed; Feature extraction is performed on the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency. The Mel frequency cepstral coefficients and logarithmic fundamental frequency are input into a pre-trained star-shaped generative adversarial network for data augmentation to obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier. The generator is used to generate data for multiple target domains. The generator structure includes: a first convolutional layer, a first batch normalization layer, a first gated linear unit, four first downsampling layers, a first transposed convolutional layer, a second batch normalization layer, a second gated linear unit, three first upsampling layers, and a second transposed convolutional layer. The discriminator is used to determine whether the data of multiple target domains output by the generator are consistent with the target label. The structure of the discriminator includes: a second convolutional layer, a third batch normalization layer, a third gated linear unit, three second downsampling layers, a third convolutional layer, an activation function layer, a first pooling layer, a fourth convolutional layer, and an output true or false layer. The auxiliary classifier is used to determine the label category of the data in multiple target domains output by the generator. The structure of the auxiliary classifier includes: a fifth convolutional layer, a fourth batch normalization layer, a fourth gated linear unit, a fully connected layer, three third downsampling layers, a sixth convolutional layer, a softmax layer, and a second pooling layer. The training process of the pre-trained star-shaped generative adversarial network includes: Step S1: Construct samples for a star-shaped generative adversarial network and their corresponding labels; Step S2: Define the structure of the generator, the discriminator, and the auxiliary classifier; define the loss function of the generator, the discriminator, and the auxiliary classifier; initialize the learning rate and learning rate decay of the generator, the learning rate and learning rate decay of the discriminator, the weight hyperparameters of the generator's loss function, the weight hyperparameters of the discriminator's loss function, and the weight hyperparameters of the auxiliary classifier's loss function. Step S3: Randomly shuffle the samples and corresponding sample labels of the star-shaped generative adversarial network to obtain the shuffled samples and corresponding sample labels. Step S4: Use the disordered samples and their corresponding labels to train the generator, discriminator, and auxiliary classifier; Step S5: Optimize the generator using the first Adam optimizer, optimize the discriminator using the second Adam optimizer, and optimize the auxiliary classifier using the third Adam optimizer; Step S6: Determine whether the current number of training iterations is less than the first set value; if the current number of training iterations is less than the first set value, proceed to step S7; if the current number of training iterations is greater than or equal to the first set value, use the first Adam optimizer to update the learning rate and learning rate decay rate of the generator, use the second Adam optimizer to update the learning rate and learning rate decay rate of the discriminator, and use the third Adam optimizer to update the learning rate and learning rate decay rate of the auxiliary classifier. Step S7: Calculate the loss function of the generator, the loss function of the discriminator, and the loss function of the auxiliary classifier using the current learning rate and the learning rate decay rate; Step S8: Determine whether the current number of training iterations has reached the second set value. If the second set value is greater than the first set value, proceed to step S9; otherwise, increment the current number of training iterations by 1 and proceed to step S4 to continue training. Step S9: Construct and save the pre-trained star-shaped generative adversarial network using the current generator loss function, discriminator loss function, and auxiliary classifier loss function.

2. The general data augmentation method for industrial acoustic signals according to claim 1, characterized in that, The generator's loss function is obtained by weighted summation of the generator's adversarial loss function, the generator's domain classification loss function, the generator's cycle consistency loss function, and the generator's identity mapping loss function. The loss function of the discriminator is the adversarial loss function of the discriminator; The loss function of the auxiliary classifier is the domain classification loss function of the discriminator.

3. The general data enhancement method for industrial acoustic signals according to claim 2, characterized in that, The adversarial loss function, domain classification loss function, cycle consistency loss function, identity mapping loss function, adversarial loss function, and domain classification loss function of the generator are calculated as follows: ； ； ； ； ； ； in, Let the generator be the adversarial loss function. Let the discriminator be the adversarial loss function. The domain classification loss function for the generator. Let the domain classification loss function be the discriminator. Let the generator's cycle consistency loss function be . Let be the identity mapping loss function for the generator, x be the source domain consisting of samples from all classes, y be the target domain consisting of the labels corresponding to the samples, and c be the label of the target domain. Tags for the source domain; To generate data for the target domain y label c using data from the source domain x, To help the discriminator determine whether the data generated by the generator is similar to the data of the target domain label c; For the distribution of data; The probability of class c in the target domain y generated by the generator. This represents the probability of the true class c in the target domain y generated by the generator; Represents the mathematical expectation. This indicates the calculation of the expected value under the conditions that the source domain sample x follows the source domain sample distribution p(x) and the target domain label c follows the target domain label distribution p(c). This indicates that the target domain label c follows the target domain label distribution p(c), and the target domain y follows the data distribution given the label c. Calculate the mathematical expectation under the given conditions. Indicates in the source domain tag The target domain label distribution p(c) is followed, and the source domain data x follows the given source domain label. Data distribution under certain conditions Calculate the expected value of the target domain label c under the condition that it follows the target domain label distribution p(c). This indicates that the source domain label c′ follows the target domain label distribution p(c), and the source domain data x follows the given source domain label. Data distribution under certain conditions Calculate the mathematical expectation under the given conditions. express The order norm.

4. A general-purpose data enhancement device for industrial acoustic signals, characterized in that, include: The data acquisition module is used to acquire the actual acoustic signal data to be enhanced. The data preprocessing module is used to preprocess the actual acoustic signal data; The feature extraction module is used to extract features from the preprocessed actual acoustic signal data. The extracted features include: Mel frequency cepstral coefficients and logarithmic fundamental frequency. The data augmentation module is used to input the Mel frequency cepstral coefficients and the logarithmic fundamental frequency into a pre-trained star-shaped generative adversarial network to perform data augmentation and obtain the data augmentation result. The star-shaped generative adversarial network includes a generator, a discriminator, and an auxiliary classifier. The generator is used to generate data for multiple target domains. The generator structure includes: a first convolutional layer, a first batch normalization layer, a first gated linear unit, four first downsampling layers, a first transposed convolutional layer, a second batch normalization layer, a second gated linear unit, three first upsampling layers, and a second transposed convolutional layer. The discriminator is used to determine whether the data of multiple target domains output by the generator are consistent with the target label. The structure of the discriminator includes: a second convolutional layer, a third batch normalization layer, a third gated linear unit, three second downsampling layers, a third convolutional layer, an activation function layer, a first pooling layer, a fourth convolutional layer, and an output true or false layer. The auxiliary classifier is used to determine the label category of the data in multiple target domains output by the generator. The structure of the auxiliary classifier includes: a fifth convolutional layer, a fourth batch normalization layer, a fourth gated linear unit, a fully connected layer, three third downsampling layers, a sixth convolutional layer, a softmax layer, and a second pooling layer. The training process of the pre-trained star-shaped generative adversarial network includes: Step S1: Construct samples for a star-shaped generative adversarial network and their corresponding labels; Step S2: Define the structure of the generator, the discriminator, and the auxiliary classifier; define the loss function of the generator, the discriminator, and the auxiliary classifier; initialize the learning rate and learning rate decay of the generator, the learning rate and learning rate decay of the discriminator, the weight hyperparameters of the generator's loss function, the weight hyperparameters of the discriminator's loss function, and the weight hyperparameters of the auxiliary classifier's loss function. Step S3: Randomly shuffle the samples and corresponding sample labels of the star-shaped generative adversarial network to obtain the shuffled samples and corresponding sample labels. Step S4: Use the disordered samples and their corresponding labels to train the generator, discriminator, and auxiliary classifier; Step S5: Optimize the generator using the first Adam optimizer, optimize the discriminator using the second Adam optimizer, and optimize the auxiliary classifier using the third Adam optimizer; Step S6: Determine whether the current number of training iterations is less than the first set value; if the current number of training iterations is less than the first set value, proceed to step S7; if the current number of training iterations is greater than or equal to the first set value, use the first Adam optimizer to update the learning rate and learning rate decay rate of the generator, use the second Adam optimizer to update the learning rate and learning rate decay rate of the discriminator, and use the third Adam optimizer to update the learning rate and learning rate decay rate of the auxiliary classifier. Step S7: Calculate the loss function of the generator, the loss function of the discriminator, and the loss function of the auxiliary classifier using the current learning rate and the learning rate decay rate; Step S8: Determine whether the current number of training iterations has reached the second set value. If the second set value is greater than the first set value, proceed to step S9; otherwise, increment the current number of training iterations by 1 and proceed to step S4 to continue training. Step S9: Construct and save the pre-trained star-shaped generative adversarial network using the current generator loss function, discriminator loss function, and auxiliary classifier loss function.

5. An electronic device, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the general data enhancement method for industrial acoustic signals as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the general data enhancement method for industrial acoustic signals as described in any one of claims 1 to 3.

Images