Mechanical fault identification method based on vibration and audio signal fusion
By integrating real-time vibration and audio signals of mechanical equipment, feature extraction and fusing are performed to perform fault diagnosis, which solves the problem of misdiagnosis and high missed diagnosis rate of diagnostic methods based on a single physical quantity in the prior art, achieving higher diagnostic accuracy and reliability.
Patent Information
- Application Number
- CN202510219885.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-12-20
- Filing Date
- 2025-02-26
- Publication Date
- 2025-07-01
AI Technical Summary
In the prior art, the mechanical fault diagnosis method based on a single physical quantity has a high rate of misdiagnosis and misdiagnosis, and the diagnostic results are low. It is difficult to accurately identify the fault source in complex mechanical equipment and complex environments.
The mechanical fault identification method based on the fusion of vibration and audio signals is adopted. By obtaining the real-time audio signal and real-time vibration signal of the mechanical equipment, the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model are respectively input, and the real-time feature map is obtained and the feature fusion is performed, and the classifier is input to the fault diagnosis.
By fusing the characteristic diagram of vibration and audio signals, mechanical equipment failures can be diagnosed more accurately and reliably, the rate of misdiagnosis and missed diagnosis can be reduced, and the accuracy and reliability of diagnostic results can be improved.
Smart Images
Figure CN120234753A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of mechanical fault diagnosis, and particularly to a mechanical fault identification method based on the fusion of vibration and audio signals. Background Art
[0002] Traditional fault diagnosis methods can, to a certain extent, capture and identify some problems in mechanical systems, but most of them rely on the analysis of a single physical quantity (e.g., vibration signal). This method performs well in scenarios where fault characteristics are obvious and / or environmental interference is small. However, in the face of increasingly complex mechanical equipment structures, diverse fault modes, and complex and changeable industrial environments, its limitations are gradually emerging. Specifically, single vibration data may not accurately reflect the overall health status of the mechanical system, especially when there are interference factors of this single physical quantity in the environment (such as other vibration signals) or the initial symptoms of the fault are not obvious. Traditional analysis methods often struggle to accurately identify the fault source, resulting in a relatively high misdiagnosis and missed diagnosis rate and low reliability of the diagnosis results. Summary of the Invention
[0003] The present invention provides a mechanical fault identification method based on the fusion of vibration and audio signals to solve the problems of relatively high misdiagnosis and missed diagnosis rates and low reliability of the diagnosis results in the existing diagnosis method based on a single physical quantity.
[0004] The present invention provides a mechanical fault identification method based on the fusion of vibration and audio signals, including the following steps.
[0005] Obtain the real-time audio signal and real-time vibration signal of the mechanical equipment.
[0006] Input the real-time audio signal and real-time vibration signal into the audio feature extraction network and vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0007] Input the real-time audio feature map and real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0008] Input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0009] Wherein, the fault diagnosis model is trained based on sample signals and the fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals.
[0010] A mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention. Both the audio feature extraction network and the vibration feature extraction network are U-Net feature extraction networks. The U-Net feature extraction network includes: an encoder, a decoder, and a convolutional module. The convolutional module connects the encoder and the decoder. The encoder includes: N downsampling modules, and the decoder includes: N upsampling modules, where N is greater than or equal to 2. The convolutional module connects the last downsampling module and the last upsampling module. The convolutional module is used to convolve the downsampled feature map obtained by the last downsampling module and then input it into the last upsampling module.
[0011] A mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention. The U-Net feature extraction network further includes: N hybrid local channel attention modules. The i-th hybrid local channel attention module is used to receive the i-th downsampled feature map of the i-th downsampling module, fuse the channel features and spatial features of the i-th downsampled feature map, and fuse the global features and local features, and output the fused i-th downsampled feature map to the i-th upsampling module, where i = 1, 2,..., N.
[0012] A mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention. The feature fusion module is a feature fusion module based on the attention mechanism, which is used to calculate the attention map for the concatenated feature map of the real-time audio feature map and the real-time vibration feature map, and multiply the attention map and the concatenated feature map to obtain the real-time fusion feature map.
[0013] A mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention. The feature fusion module includes: a feature concatenation sub-module, a channel attention sub-module, and a spatial attention sub-module.
[0014] The feature concatenation sub-module is used to concatenate the real-time audio feature map and the real-time vibration feature map to obtain the concatenated feature map.
[0015] The channel attention sub-module is used to calculate the channel attention map according to the concatenated feature map, and multiply the channel attention map and the concatenated feature map to obtain the intermediate concatenated feature map.
[0016] The spatial attention sub-module is used to calculate the spatial attention map according to the intermediate concatenated feature map, and multiply the spatial attention map and the intermediate concatenated feature map to obtain the real-time fusion feature map.
[0017] A mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention. The fault diagnosis model is iteratively trained in the following manner.
[0018] Input the sample audio signal and the sample vibration signal into the audio feature extraction network and the vibration feature extraction network of the initial model respectively to obtain a sample audio feature map and a sample vibration feature map.
[0019] Input the sample audio feature map and the sample vibration feature map into the feature fusion module of the initial model to obtain a sample fusion feature map.
[0020] Input the sample fusion feature map into the classifier of the initial model to obtain the fault prediction result output by the classifier.
[0021] Substitute the fault prediction result and the fault label corresponding to the sample audio signal and the sample vibration signal into the loss function of the classifier. When the loss function converges, obtain the fault diagnosis model. Otherwise, perform backpropagation, update the parameters of the initial model, and continue iterative training.
[0022] The present invention also provides a mechanical fault identification device based on the fusion of vibration and audio signals, including the following units.
[0023] A real-time signal acquisition unit for acquiring the real-time audio signal and the real-time vibration signal of the mechanical equipment.
[0024] A feature map extraction unit for inputting the real-time audio signal and the real-time vibration signal into the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0025] A feature map fusion unit for inputting the real-time audio feature map and the real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0026] A classification and diagnosis unit for inputting the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0027] Wherein, the fault diagnosis model is trained based on the sample signal and the fault label corresponding to the sample signal, and the sample signal includes: a sample audio signal and a sample vibration signal.
[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the program, it implements the mechanical fault identification method based on the fusion of vibration and audio signals as described in any one of the above.
[0029] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the mechanical fault identification method based on the fusion of vibration and audio signals as described in any one of the above.
[0030] The present invention also provides a computer program product, including a computer program, which when executed by a processor, implements the mechanical fault identification method based on the fusion of vibration and audio signals as described in any one of the above.
[0031] The mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention includes obtaining real-time audio signals and real-time vibration signals of a mechanical device; Inputting the real-time audio signal and the real-time vibration signal into an audio feature extraction network and a vibration feature extraction network of a fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map; inputting the real-time audio feature map and the real-time vibration feature map into a feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map; inputting the real-time fusion feature map into a classifier of the fault diagnosis model to obtain a fault diagnosis result output by the classifier; wherein, the fault diagnosis model is trained based on sample signals and fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals. Since the fusion feature map contains more angular feature information of the operating state of the mechanical device compared to a single feature map, it can diagnose mechanical device faults more accurately and reliably, reduce the misdiagnosis rate and missed diagnosis rate, and improve the accuracy and reliability of the diagnosis result. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0033] Figure 1 is a schematic flowchart of the mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention.
[0034] Figure 2 is a schematic structural diagram of a fault diagnosis model in the mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention.
[0035] Figure 3 is one of the schematic structural diagrams of the feature extraction network of the fault diagnosis model in the mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention.
[0036] Figure 4 is the other schematic structural diagram of the feature extraction network of the fault diagnosis model in the mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention.
[0037] Figure 5 Yes Figure 3 And Figure 4 Schematic diagram of the structure of the hybrid local channel attention module in the feature extraction network of
[0038] Figure 6 Schematic diagram of feature fusion of the fault diagnosis model in the mechanical fault identification method based on the fusion of vibration and audio signals provided by the present invention.
[0039] Figure 7 Schematic diagram of the structure of the mechanical fault identification device based on the fusion of vibration and audio signals provided by the present invention.
[0040] Figure 8 Schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners
[0041] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Apparently, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.
[0042] The mechanical fault identification method based on the fusion of vibration and audio signals according to the embodiment of the present invention, as Figure 1 And Figure 2 shown, includes the following steps S110 and S140.
[0043] Step S110: Obtain the real-time audio signal and real-time vibration signal of the mechanical equipment. Specifically, the real-time audio signal and real-time vibration signal of the mechanical equipment can be respectively collected by an audio sensor and a vibration sensor. In this step, the real-time audio signal and real-time vibration signal can be respectively obtained from the audio sensor and the vibration sensor.
[0044] In this step, it may further include denoising the obtained real-time audio signal and real-time vibration signal.
[0045] Specifically, empirical mode decomposition can be used to perform denoising operations on the real-time audio signal and real-time vibration signal. Empirical mode decomposition is an adaptive decomposition method suitable for non-linear and non-stationary signals, which can remove noise and retain useful information in the signal.
[0046] When applying empirical mode decomposition, it is first necessary to find the moments corresponding to all the maximum and minimum values in the original signal , denoted as And , tRepresents a moment. Then, the maximum and minimum values are respectively fitted with spline curves to obtain the upper envelope curve and the lower envelope curve . The specific process is as follows: ; .
[0047] Among them, is the spline function. Subsequently, the mean value of the upper and lower envelopes is calculated : .
[0048] Then, this mean value is subtracted from the original signal to obtain the initial mode component : .
[0049] Repeat the above steps to perform extreme point detection, envelope curve fitting, mean value calculation, and subtraction operation on until meets the Intrinsic Mode Functions (IMF) conditions (for example, the mean value approaches 0, or the standard deviation satisfies a certain threshold). Suppose the signal after iterations is . Then, when meets the IMF conditions, is the first : .
[0050] Then, repeat the above process for to extract the second that meets the IMF conditions. Repeat this process until the residual no longer contains decomposable oscillation components. Finally, the signal is decomposed into several that meet the IMF conditions and a residual : .
[0051] The advantage of empirical mode decomposition is that it does not require pre-set basis functions and can adaptively decompose the mode functions that reflect the essential characteristics of the signal.
[0052] Step S120: Input the real-time audio signal and the real-time vibration signal into the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model respectively to obtain the real-time audio feature map and the real-time vibration feature map. Specifically, the audio feature extraction network is used to extract features from the real-time audio signal to obtain the real-time audio feature map, and the vibration feature extraction network is used to extract features from the real-time vibration signal to obtain the real-time vibration feature map.
[0053] Step S130: Input the real-time audio feature map and the real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map. Specifically, this feature fusion module can utilize the attention mechanism to adaptively adjust the respective fusion weights of the real-time audio feature map and the real-time vibration feature map according to the importance and relevance of different modality information, so as to achieve the fusion of audio features and vibration features, thereby improving the accuracy and reliability of mechanical equipment fault identification.
[0054] Step S140: Input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier. Among them, the fault diagnosis model is trained based on sample signals and the fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals.
[0055] Specifically, after obtaining a large number of sample audio signals and sample vibration signals, the sample audio signals and sample vibration signals can be preprocessed (including denoising and normalization, etc.). After preprocessing, a group of sample audio signals and sample vibration signals are labeled. Briefly, when this group of sample audio signals and sample vibration signals are generated under the known fault state of the mechanical equipment, the corresponding fault label is marked as 1, otherwise marked as 0. At this time, the classifier can adopt a Support Vector Machine (SVM). Further, when this group of sample audio signals and sample vibration signals are generated under the known fault state of the mechanical equipment, different markings can be made according to different fault types. For example: 0 represents no fault, 1 represents fault type one, 2 represents fault type two, 3 represents fault type three. At this time, the classifier can adopt a Multi-class Support Vector Machine.
[0056] In the mechanical fault identification method based on the fusion of vibration and audio signals in this embodiment, the audio signal feature map and the vibration signal feature map are fused to obtain a multi-modal fusion feature map, and the mechanical equipment is fault diagnosed based on the fusion feature map. Since the fusion feature map contains more angular feature information about the operating state of the mechanical equipment than a single feature map, it can diagnose mechanical equipment faults more accurately and reliably, reduce the misdiagnosis rate and missed diagnosis rate, and improve the accuracy and reliability of the diagnosis result.
[0057] It should be noted that during the operation of mechanical equipment, both sound and vibration are continuous. The audio sensor and vibration sensor also continuously collect audio signals and vibration signals. In this embodiment, in order to construct a sample data set, the continuous signals are segmented at a predetermined time interval to obtain a number of signal segments. For example, a signal segment is divided every 0.1 seconds, and the duration of the signal segment is 0.1 seconds, thereby obtaining multiple groups of sample audio signals and sample vibration signals, and fault labels are calibrated for each group of sample audio signals and sample vibration signals. Correspondingly, the real-time audio signal and real-time vibration signal are also obtained by segmenting the corresponding real-time continuous signals at the above-mentioned predetermined time interval.
[0058] Furthermore, to avoid sudden situations or abnormal sounds and vibrations caused by improper accidental operations of users on mechanical equipment being misjudged as faults, in step S140, after obtaining the fault diagnosis result output by the classifier, it further includes counting the fault diagnosis results within a preset duration (such as: 10 seconds). In the case where more than 5%-10% of the fault diagnosis results are faults, it is determined that the mechanical equipment has a fault, making the fault diagnosis result more accurate and avoiding misjudgment.
[0059] In some embodiments, both the audio feature extraction network and the vibration feature extraction network are U-Net feature extraction networks. As Figure 3 shown, the U-Net feature extraction network includes: an encoder, a decoder, and a convolution module. The convolution module connects the encoder and the decoder; the encoder includes: N downsampling modules, and the decoder includes: N upsampling modules, where N is greater than or equal to 2. For example, N = 5. The convolution module connects the last downsampling module and the last upsampling module. The convolution module is used to input the downsampled feature map obtained by the last downsampling module after convolution into the last upsampling module. This convolution module can be a 1×1 convolution module. Moreover, for the first downsampling module, a 1×1 input convolution module is further provided at its input end. The 1×1 input convolution module is used to output the input original signal (real-time audio signal or real-time vibration signal) after convolution to the first downsampling module; for the first upsampling module, a 1×1 output convolution module is further provided at its output end. The 1×1 output convolution module is used to output the upsampled feature map output by the first upsampling module after convolution, thereby obtaining a real-time audio feature map or a real-time vibration feature map.
[0060] In this embodiment, a traditional U-Net feature extraction network can be adopted. Its characteristic is that it has a symmetric encoder-decoder structure, and the details of the original input information are maintained through skip connections. When applying the U-Net network structure to one-dimensional signals (audio signals and vibration signals), its feature extraction ability can effectively capture the long-distance dependencies in the audio, and at the same time, the high-frequency details of the signal are retained through skip connections. The encoder part of the U-Net gradually extracts the global features of the signal through multi-layer convolution operations, while the decoder restores the time structure of the signal through upsampling, so that the details of the signal are retained. Its structure design enables the model to not only capture the long-term changes in the signal, but also handle short-term changes, which helps to improve the accuracy and robustness of signal processing tasks.
[0061] Since the feature maps adopted by the encoder and decoder of the traditional U-Net feature extraction network have relatively large semantic differences in feature fusion. The semantic differences may lead to "mismatches" of information, causing the model to lose some details during the synthesis process or being unable to effectively retain the deep semantic information. The result may be that the feature map of the signal finally extracted by the U-Net is lacking in details, or unable to accurately express complex context relationships. To narrow the semantic differences, enhance the expression of different features of the encoder, eliminate the interference of primary noise, and improve the extraction of the signal spectrum features by the U-Net feature extraction network, in some embodiments, the U-Net feature extraction network further includes: N mixed local channel attention modules (Mixed local channel attention, MLCA). As Figure 4 shown, the i-th mixed local channel attention module is used to receive the i-th downsampled feature map of the i-th downsampling module, fuse the channel features and spatial features of the i-th downsampled feature map, fuse the global features and local features, and output the fused i-th downsampled feature map to the i-th upsampling module, where i = 1, 2,..., N.
[0062] Specifically, the structure of the mixed local channel attention module is as Figure 5 shown, and its corresponding attention mechanism algorithm is as follows in steps one to six.
[0063] Step 1: Input the feature map (the downsampled feature map after passing through the downsampling module) into the local average pooling module. The local pooling focuses on the feature information in the local area. The local average pooling result of the local average pooling module is split into two paths. One path is input into the first Reshape module, and the other path is input into the global average pooling module. Local average pooling can capture local detailed information by aggregating the features in the local area, helping the network to capture local features without losing too much spatial information. The global and local parallel processing design transmits local information and global information to different modules respectively, which helps the network to comprehensively process features from two perspectives and improves the expressive ability of the model.
[0064] Step 2: The local average pooling result input into the first Reshape module passes through the first Reshape module, the first 1D convolutional module (i.e., a 1×1 convolutional module), and the third Reshape module in sequence to obtain the first intermediate feature map. This process is equivalent to a kind of feature selection, strengthening the attention to useful features. Among them, the first 1D convolutional module is used to compress the feature channels while keeping the spatial dimension unchanged.
[0065] Step 3: The local average pooling result input into the global average pooling module undergoes global average pooling processing in the global average pooling module. Global pooling captures the statistical information of the entire feature map. After the feature map after global pooling is rearranged by the second Reshape module, it then passes through the second 1D convolutional module and the first unpooling module (UNAP) to obtain the second intermediate feature map. Among them, the second 1D convolutional module is used to compress the feature channels while keeping the spatial dimension unchanged, ensuring that the global information is not redundant during the transmission process and further strengthening the retention of global information.
[0066] Step 4: Add the first intermediate feature map and the second intermediate feature map. Combine the global context information with the local pooling features through an addition operation to obtain the third intermediate feature map processed by local and global attention. This can improve the model's ability to recognize complex patterns and make full use of the complementarity between local features and global features.
[0067] Step 5: Perform an unpooling operation on the third intermediate feature map again through the second unpooling module to restore the third intermediate feature map to the original spatial dimension and obtain the fourth intermediate feature map. This step enables the network to further process information at the spatial level. This step helps to avoid the loss of spatial information.
[0068] Step 6: Multiply the fourth intermediate feature map by the input feature map to obtain the output feature map.
[0069] It can be understood that: for the audio feature extraction network, Figure 3 and 4The original signal in Figure 3 and 4 is a real-time audio signal (a sample audio signal during model training), and the signal feature is a real-time audio feature map (a sample audio feature map during model training); for the vibration feature extraction network,
[0070] The original signal in
[0071] such as Figure 6 shown, the feature splicing sub-module is used to splice the real-time audio feature map and the real-time vibration feature map to obtain the spliced feature map, and the feature splicing sub-module can be implemented by the concat function.
[0072] The channel attention sub-module is used to calculate the channel attention map according to the spliced feature map, and multiply the channel attention map by the spliced feature map to obtain an intermediate spliced feature map.
[0073] The spatial attention sub-module is used to calculate the spatial attention map according to the intermediate spliced feature map, and multiply the spatial attention map by the intermediate spliced feature map to obtain the real-time fusion feature map.
[0074] In this embodiment, the feature fusion module realizes the fusion according to the mutual dependence between two modalities of audio and vibration among different channels to obtain a real-time fusion feature map. Mechanical faults often manifest themselves through abnormal vibrations and sound waves. Vibration signals can capture the minute movements of objects, while audio signals can better reflect the sound wave characteristics caused by faults. Fusing the feature maps of these two signals can more comprehensively describe the state of the system and make up for the information loss of a single signal in certain frequency domains or time domains. Vibration signals provide accurate displacement, velocity, and acceleration data, while audio signals may be more sensitive to abnormal sounds in different frequency ranges. After combining the two, the real-time fusion feature map can more comprehensively and accurately reveal the fault state of the system, enhance the robustness and accuracy of recognition, and thus have a higher diagnostic effect compared with a single feature.
[0075] In some embodiments, the fault diagnosis model is iteratively trained in the following manner.
[0076] Input the sample audio signal and the sample vibration signal into the audio feature extraction network and the vibration feature extraction network of the initial model respectively to obtain a sample audio feature map and a sample vibration feature map.
[0077] Input the sample audio feature map and the sample vibration feature map into the feature fusion module of the initial model to obtain a sample fusion feature map.
[0078] Input the sample fusion feature map into the classifier of the initial model to obtain the fault prediction result output by the classifier.
[0079] Substitute the fault prediction result and the fault label corresponding to the sample audio signal and the sample vibration signal into the loss function (e.g., Multiclass SVM loss) of the classifier. When the loss function converges, obtain the fault diagnosis model. Otherwise, perform backpropagation, update the parameters of the initial model, and continue iterative training.
[0080] The mechanical fault identification device based on the fusion of vibration and audio signals provided by the present invention will be described below. The mechanical fault identification device based on the fusion of vibration and audio signals described below can be correspondingly referred to the mechanical fault identification method based on the fusion of vibration and audio signals described above.
[0081] The mechanical fault identification device based on the fusion of vibration and audio signals according to the embodiment of the present invention, as Figure 7 shown, includes the following functional units.
[0082] A real-time signal acquisition unit 710, configured to acquire a real-time audio signal and a real-time vibration signal of a mechanical device.
[0083] A feature map extraction unit 720, configured to input the real-time audio signal and the real-time vibration signal into the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0084] A feature map fusion unit 730, configured to input the real-time audio feature map and the real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0085] A classification diagnosis unit 740, configured to input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0086] Wherein, the fault diagnosis model is trained based on the sample signal and the fault label corresponding to the sample signal, and the sample signal includes: a sample audio signal and a sample vibration signal.
[0087] In some embodiments, both the audio feature extraction network and the vibration feature extraction network are U-Net feature extraction networks. The U-Net feature extraction network includes: an encoder, a decoder, and a convolutional module. The convolutional module connects the encoder and the decoder. The encoder includes: N downsampling modules, and the decoder includes: N upsampling modules, where N is greater than or equal to 2. The convolutional module connects the last downsampling module and the last upsampling module, and is used to input the downsampled feature map obtained by the last downsampling module into the last upsampling module after convolution.
[0088] In some embodiments, the U-Net feature extraction network further includes: N hybrid local channel attention modules. The i-th hybrid local channel attention module is used to receive the i-th downsampled feature map of the i-th downsampling module, fuse the channel features and spatial features of the i-th downsampled feature map, and fuse the global features and local features, and output the fused i-th downsampled feature map to the i-th upsampling module, where i = 1, 2,..., N.
[0089] In some embodiments, the feature fusion module is a feature fusion module based on an attention mechanism, and is used to calculate an attention map for the concatenated feature map of the real-time audio feature map and the real-time vibration feature map, and multiply the attention map and the concatenated feature map to obtain the real-time fusion feature map.
[0090] In some embodiments, the feature fusion module includes: a feature concatenation sub-module, a channel attention sub-module, and a spatial attention sub-module.
[0091] The feature concatenation sub-module is used to concatenate the real-time audio feature map and the real-time vibration feature map to obtain the concatenated feature map.
[0092] The channel attention sub-module is used to calculate a channel attention map according to the concatenated feature map, and multiply the channel attention map and the concatenated feature map to obtain an intermediate concatenated feature map.
[0093] The spatial attention sub-module is used to calculate a spatial attention map according to the intermediate concatenated feature map, and multiply the spatial attention map and the intermediate concatenated feature map to obtain the real-time fusion feature map.
[0094] In some embodiments, the fault diagnosis model is iteratively trained in the following manner.
[0095] Input the sample audio signal and the sample vibration signal into the audio feature extraction network and the vibration feature extraction network of the initial model respectively to obtain a sample audio feature map and a sample vibration feature map.
[0096] Input the sample audio feature map and the sample vibration feature map into the feature fusion module of the initial model to obtain a sample fusion feature map.
[0097] Input the sample fusion feature map into the classifier of the initial model to obtain the fault prediction result output by the classifier.
[0098] Substitute the fault prediction result and the fault label corresponding to the sample audio signal and the sample vibration signal into the loss function of the classifier. When the loss function converges, obtain the fault diagnosis model. Otherwise, perform backpropagation, update the parameters of the initial model, and continue iterative training.
[0099] Figure 8 An example of a schematic physical structure diagram of an electronic device is shown as Figure 8 shown. The electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute a mechanical fault identification method based on the fusion of vibration and audio signals. The method includes the following steps.
[0100] Obtain the real-time audio signal and the real-time vibration signal of the mechanical device.
[0101] Input the real-time audio signal and the real-time vibration signal into the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0102] Input the real-time audio feature map and the real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0103] Input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0104] Among them, the fault diagnosis model is trained based on the sample signal and the fault label corresponding to the sample signal. The sample signal includes: a sample audio signal and a sample vibration signal.
[0105] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0106] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mechanical fault identification method based on the fusion of vibration and audio signals provided by the above-mentioned various methods. The method includes the following steps.
[0107] Obtain the real-time audio signal and real-time vibration signal of the mechanical equipment.
[0108] Input the real-time audio signal and real-time vibration signal into the audio feature extraction network and vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0109] Input the real-time audio feature map and real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0110] Input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0111] Wherein, the fault diagnosis model is trained based on sample signals and the fault labels corresponding to the sample signals. The sample signals include: sample audio signals and sample vibration signals.
[0112] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the mechanical fault identification method based on the fusion of vibration and audio signals provided by the above-mentioned various methods. The method includes the following steps.
[0113] Obtain the real-time audio signal and real-time vibration signal of the mechanical equipment.
[0114] Input the real-time audio signal and the real-time vibration signal into the audio feature extraction network and the vibration feature extraction network of the fault diagnosis model respectively to obtain a real-time audio feature map and a real-time vibration feature map.
[0115] Input the real-time audio feature map and the real-time vibration feature map into the feature fusion module of the fault diagnosis model to obtain a real-time fusion feature map.
[0116] Input the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier.
[0117] Wherein, the fault diagnosis model is trained based on sample signals and the fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals.
[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.
[0119] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A mechanical fault identification method based on vibration and audio signal fusion, characterized in that: include: Obtain real-time audio signals and real-time vibration signals of mechanical equipment; Inputting the real-time audio signal and the real-time vibration signal into an audio feature extraction network and a vibration feature extraction network of a fault diagnosis model respectively to obtain a real-time audio feature graph and a real-time vibration feature graph; Inputting the real-time audio feature graph and the real-time vibration feature graph into a feature fusion module of the fault diagnosis model to obtain a real-time fusion feature graph; Inputting the real-time fusion feature graph into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier; The fault diagnosis model is obtained by training based on sample signals and fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals.
2. The mechanical fault identification method based on vibration and audio signal fusion according to claim 1 is characterized in that: The audio feature extraction network and the vibration feature extraction network are both U-Net feature extraction networks, and the U-Net feature extraction network includes: an encoder, a decoder and a convolution module, and the convolution module connects the encoder and the decoder; the encoder includes: N layers of downsampling modules, and the decoder includes: N layers of upsampling modules, N is greater than or equal to 2, and the convolution module connects the last layer of downsampling modules and the last layer of upsampling modules, and the convolution module is used to convolve the downsampling feature map obtained by the last layer of downsampling module and input it into the last layer of upsampling module.
3. The mechanical fault identification method based on vibration and audio signal fusion according to claim 2 is characterized in that: The U-Net feature extraction network also includes: N hybrid local channel attention modules, the i-th hybrid local channel attention module is used to receive the i-th down-sampling feature map of the i-th layer down-sampling module, fuse the channel features and spatial features of the i-th down-sampling feature map, fuse the global features and local features, and output the fused i-th down-sampling feature map to the i-th layer up-sampling module, where i=1,2,…,N.
4. The mechanical fault identification method based on vibration and audio signal fusion according to claim 1 is characterized in that: The feature fusion module is a feature fusion module based on the attention mechanism, which is used to calculate the attention map for the spliced feature map of the real-time audio feature map and the real-time vibration feature map, and multiply the attention map and the spliced feature map to obtain the real-time fused feature map.
5. The mechanical fault identification method based on vibration and audio signal fusion according to claim 4 is characterized in that: The feature fusion module includes: a feature splicing submodule, a channel attention submodule and a spatial attention submodule; The feature splicing submodule is used to splice the real-time audio feature map and the real-time vibration feature map to obtain the spliced feature map; The channel attention submodule is used to calculate a channel attention map according to the splicing feature map, and multiply the channel attention map with the splicing feature map to obtain an intermediate splicing feature map; The spatial attention submodule is used to calculate a spatial attention map based on the intermediate splicing feature map, and multiply the spatial attention map with the intermediate splicing feature map to obtain the real-time fusion feature map.
6. The mechanical fault identification method based on vibration and audio signal fusion according to any one of claims 1 to 5, characterized in that: The fault diagnosis model is iteratively trained as follows: Inputting the sample audio signal and the sample vibration signal into the audio feature extraction network and the vibration feature extraction network of the initial model respectively to obtain a sample audio feature map and a sample vibration feature map; Inputting the sample audio feature map and the sample vibration feature map into the feature fusion module of the initial model to obtain a sample fusion feature map; Inputting the sample fusion feature map into the classifier of the initial model to obtain a fault prediction result output by the classifier; The fault prediction result and the fault labels corresponding to the sample audio signal and the sample vibration signal are substituted into the loss function of the classifier. When the loss function converges, the fault diagnosis model is obtained. Otherwise, back propagation is performed to update the parameters of the initial model and then continue iterative training.
7. A mechanical fault identification device based on the fusion of vibration and audio signals, characterized in that: include: A real-time signal acquisition unit, used to acquire real-time audio signals and real-time vibration signals of mechanical equipment; A feature map extraction unit, used for inputting the real-time audio signal and the real-time vibration signal into an audio feature extraction network and a vibration feature extraction network of a fault diagnosis model respectively, so as to obtain a real-time audio feature map and a real-time vibration feature map; A feature graph fusion unit, used for inputting the real-time audio feature graph and the real-time vibration feature graph into a feature fusion module of the fault diagnosis model to obtain a real-time fusion feature graph; A classification diagnosis unit, used for inputting the real-time fusion feature map into the classifier of the fault diagnosis model to obtain the fault diagnosis result output by the classifier; The fault diagnosis model is obtained by training based on sample signals and fault labels corresponding to the sample signals, and the sample signals include: sample audio signals and sample vibration signals.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the mechanical fault identification method based on vibration and audio signal fusion as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mechanical fault identification method based on the fusion of vibration and audio signals as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the mechanical fault identification method based on the fusion of vibration and audio signals as described in any one of claims 1 to 6 is implemented.