A multimodal sound feature fusion fluid transfer device fault classification acceleration method and system
By using a multimodal sound feature fusion method and a neural network model to automatically identify faults in fluid transmission equipment, the problem of inaccurate detection in existing technologies is solved, and rapid, stable and accurate fault identification is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN UNIV OF TECH
- Filing Date
- 2025-12-22
- Publication Date
- 2026-06-05
Smart Images
Figure CN122153701A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fault type identification acceleration technology, specifically to a method and system for accelerating fault classification of fluid transport equipment using multimodal sound feature fusion. Background Technology
[0002] In recent years, my country's infrastructure construction has continued to advance, and the industrial sector has flourished. The prosperity of traditional energy industries such as oil and natural gas has opened up a vast market space for the energy transportation industry. However, as the core carrier of energy transportation, fluid transmission systems can trigger a series of serious consequences in the event of an accident: not only will they cause multi-dimensional environmental pollution of soil, water, and air, but they may also trigger serious incidents such as fires and explosions, resulting in significant property damage and personal injury, posing a serious threat to industry development and social security.
[0003] The causes of failures in fluid transmission systems are complex and diverse, spanning the entire equipment lifecycle. During manufacturing, core components such as pumps, pipes, and valves are prone to inherent defects such as internal holes, dents, and scratches. In practical applications, pumps frequently experience impeller damage, inlet blockage, bearing wear, and leaks; valves face issues like seal failure, jamming, vibration, corrosion, or damage; and pipes often exhibit corrosion, cracks, breaks, and deformation. These failures are mostly internal or latent damage, difficult to observe directly, posing a significant challenge to accurate identification.
[0004] Current mainstream fault detection technologies include electromagnetic leakage detection, magnetic particle testing, ultrasonic testing, long-distance eddy current detection, and X-ray detection, but all have significant limitations and cannot meet the comprehensiveness and accuracy requirements of practical testing. In contrast, equipment health status detection methods based on sound signals have gained widespread attention in related fields due to their advantages such as not affecting normal system operation, high stability, and high sensitivity. However, this technology is still in the experimental exploration stage and is not yet mature in terms of detection accuracy, portability, and engineering applications, and there is still a certain gap before its practical application. Summary of the Invention
[0005] The purpose of this invention is to propose a method and system for accelerating the classification of fluid transmission equipment faults by fusing multimodal sound features, so as to achieve rapid and accurate identification of faults in fluid transmission equipment.
[0006] According to a first aspect of the present disclosure, a method for accelerating fault classification of fluid transport equipment based on multimodal acoustic feature fusion is provided, comprising the following steps: Collect sound data of different fault types and normal operating conditions of fluid transmission equipment to construct a fault sound dataset; train a neural network model based on the dataset, the model integrating a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module and a fault diagnosis module; Acquire the real-time operating sound signal of the target fluid transmission component, and preprocess the signal to obtain a clean sound signal; The clean sound signal is input into the multi-scale temporal convolutional fusion module to extract temporal fusion features; at the same time, the clean sound signal is input into the frequency domain feature extraction module, and after processing, the temporal features after frequency domain processing are obtained. The time-domain fused features and the frequency-domain processed time-domain features are fused using a dual-domain feature fusion module to generate dual-domain fused features. The dual-domain fusion features are input into the Transformer feature fusion module, and deep-level fusion features are extracted after feature depth mining. The deep-level fusion features are input into the fault diagnosis module, which performs feature classification processing and outputs the fault identification results of the fluid transmission equipment.
[0007] In another implementation of the present invention, the fault sound dataset includes sound data of pump mechanical damage, valve seal damage, pipeline faults, and normal operating conditions.
[0008] In another implementation of the present invention, the multi-scale temporal convolutional fusion module adopts the backbone network structure of EEGNet, sets convolutional layers with different kernel sizes, and the parameters include the number of filters NF, the kernel size KE, and the random deactivation ratio Dropout, wherein the ratio of KE to NF is a multiple relationship, and the features extracted at different scales are fused by splicing operation.
[0009] In another implementation of the present invention, the processing procedure of the frequency domain feature extraction module is as follows: first, the feature is downsampled by one-dimensional convolution and mapped to a channel, and then normalized by adaptive pooling to a fixed-length discrete time domain feature as the FFT input; the spatial domain latent code is converted to the frequency domain by FFT to obtain the amplitude component and the phase component; the amplitude component is fused by point convolution, and the phase component is learned by two layers of convolution combined with INR interpolation.
[0010] In another implementation of the present invention, the amplitude components are represented by point convolution as follows: In the formula: For attention weights, For attention value vectors, Convolution weights for dual-source feature encoding points, Generate point convolution weights for the attention weights; Generate pointwise convolution weights for the attention value vector, with GELU as the activation function; (Signal branch) For spatial branching, For the offset correction term, each element This corresponds to "the b-th sample, the c-th channel, and the f-th frequency point".
[0011] In another implementation of the present invention, the dual-domain feature fusion module fuses the time-domain fused features and the frequency-domain processed time-domain features using a complex Gabor wavelet activation function, wherein the complex Gabor wavelet activation function is: Where w0 is the center frequency in the frequency domain, and v0 is the standard deviation constant of the Gaussian function. It is a vector in the time domain.
[0012] In another implementation of the present invention, the Transformer feature fusion module adopts an encoder structure with 128 input channels, 128 dimensions, and 8 heads for the multi-head attention mechanism. The encoder unit adopts a 2-layer stacked network structure, with each layer containing a multi-head attention mechanism layer, a residual connection layer, a layer normalization layer, a feedforward neural network layer, a secondary residual connection layer, and a secondary layer normalization layer.
[0013] According to a second aspect of the present disclosure, a multimodal acoustic feature fusion-based fault classification acceleration system for fluid transport devices is provided, comprising: The dataset construction and training module collects sound data of different fault types and normal operating states of fluid transmission equipment to construct a fault sound dataset; based on this dataset, a neural network model is trained, which integrates a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module, and a fault diagnosis module. The signal preprocessing module acquires the real-time operating sound signal of the target fluid transmission component, preprocesses the signal to obtain a clean sound signal; The time-domain and frequency-domain feature extraction module inputs the clean sound signal into the multi-scale temporal convolution fusion module to extract the time-domain fusion features; at the same time, the clean sound signal is input into the frequency-domain feature extraction module, and after processing, the time-domain features after frequency-domain processing are obtained. The fusion module fuses the time-domain fused features with the frequency-domain processed time-domain features through the dual-domain feature fusion module to generate dual-domain fused features; The deep feature mining module inputs the dual-domain fusion features into the Transformer feature fusion module, and extracts the deep fusion features after feature depth mining. The fault diagnosis output module inputs the deeply fused features into the fault diagnosis module, which then performs feature classification processing and outputs the fault identification results of the fluid transmission equipment.
[0014] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory. The processor is an FPGA that executes the program to implement the described method for accelerating fault classification of fluid transport devices by multimodal sound feature fusion.
[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the aforementioned method for accelerating fault classification of fluid transport equipment by multimodal sound feature fusion.
[0016] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows: 1. No manual on-site inspection or disassembly testing is required. Fault diagnosis is completed automatically, greatly reducing human resource costs. At the same time, it avoids subjective errors caused by differences in human experience and visual fatigue, ensuring objective and consistent results. The detection covers all kinds of internal hidden and complex faults in fluid transmission equipment without omission. It is also fast-responding, easy to operate, less affected by environmental interference, highly stable, and can accurately capture subtle fault differences with high sensitivity.
[0017] 2. Based on the FPGA processor, compared with the serial computing of the CPU and the general computing architecture of the GPU, there is no need for redundant computing, and the power consumption is significantly reduced. The parallel computing characteristics are adapted to the core algorithm, reducing instruction scheduling latency and the response speed is faster. The low power consumption design can be paired with a portable power supply, making it suitable for scenarios without stable power supply such as in the field and remote sites, with outstanding portability.
[0018] 3. By adopting a dataset-driven neural network training mode, and supplementing the model parameters such as the number of new fault types and adjusting the number of filters and the size of convolution kernels, it can be adapted to different application environments, identify more fault and defect types, and continuously improve the recognition speed and accuracy. The multimodal sound feature fusion framework has strong versatility and, after dataset and parameter adaptation, can be extended to equipment fault diagnosis scenarios in other industries such as machinery manufacturing, power equipment, and aerospace. Attached Figure Description
[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.
[0020] Figure 1 A flowchart of an accelerated fault classification method for fluid transport equipment based on multimodal sound feature fusion; Figure 2 This is a schematic diagram of the encoder for the Transformer feature fusion module. Figure 3 This is a schematic diagram of the structure of a multi-scale temporal convolutional fusion module; Figure 4 This is a diagram showing the main parameters of the multi-scale temporal convolutional fusion module; Figure 5 Here is a flowchart of the frequency domain feature extraction module. Figure 6 Flowchart for the dual-domain feature fusion module; Figure 7 This is a flowchart of the fault diagnosis module. Detailed Implementation
[0021] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.
[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0024] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.
[0025] Example 1: like Figure 1 As shown, this embodiment provides a method for accelerating fault classification in fluid transport equipment by multimodal sound feature fusion, including the following steps: S1. A sound sensor collects sound data of different fault types and normal operating conditions of fluid transmission equipment to construct a fault sound dataset; a neural network model is trained based on this dataset, the model integrating a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module, and a fault diagnosis module; Specifically, the fault sound dataset includes sound data of pump mechanical damage, valve seal damage, pipeline faults, and normal operation. Pump mechanical damage includes impeller damage, bearing wear, and leakage; valve seal damage includes seal failure, jamming, vibration, and corrosion; and pipeline faults include corrosion, cracks, fractures, and deformation. After normalization, the dataset is divided into training and testing sets for training and validation of neural network models.
[0026] S2. Acquire the real-time operating sound signal of the target fluid transmission component, preprocess the signal to obtain a clean sound signal; Specifically, sound preprocessing includes filtering and pre-emphasis operations, which aim to remove various interference noises generated during the fault sound acquisition process, including external environmental noise, equipment operating noise, and background noise generated by the internal liquid flow during normal equipment operation, thereby obtaining a pure target sound signal and providing reliable data support for subsequent feature extraction and fault diagnosis.
[0027] S3. Input the clean sound signal into the multi-scale temporal convolution fusion module to extract temporal fusion features; at the same time, input the clean sound signal into the frequency domain feature extraction module, and obtain the frequency domain processed temporal features after processing. Specifically, the multi-scale temporal convolutional fusion module can simultaneously extract features at different temporal and spatial scales by configuring convolutional layers with different kernel sizes, and then fuse these features to effectively enrich the diversity of signal features.
[0028] This module adopts the EEGNet backbone network structure, enabling it to extract multi-scale temporal and spatial EEG features. Its core parameters include the number of filters (NF), kernel size (KE), and dropout ratio. The standard EEGNet parameters are set to NF=8, KE=(1,64), and Dropout=0.5. In this embodiment, the NF, KE, and Dropout parameters of this unit are as follows: Figure 3 As shown, KE and NF maintain a 4-fold ratio. The structure of the multi-scale temporal convolutional fusion module is as follows. Figure 4 As shown, the temporal and spatial features extracted at different scales are concatenated and then used in the subsequent feature fusion and classification process.
[0029] For the preprocessed audio signal, downsampling and channel mapping are first performed through one-dimensional convolution, and then adaptive pooling is used to normalize it into a discrete time-domain feature of fixed length 1024. This feature is used as the input of Fast Fourier Transform (FFT). After converting the audio time-domain signal to the frequency domain through FFT, the amplitude and phase components in the frequency domain are processed in a targeted manner. Finally, the processed frequency features are mapped back to the time domain by using Inverse Fast Fourier Transform (IFFT).
[0030] Directly using static convolution kernels in the frequency domain only enhances a specific frequency range, which is unsuitable for feature fusion tasks. However, by learning feature content to generate weights, implicit neural representations can be viewed as a dynamic interpolation method in a continuous space, which can adaptively enhance information in the frequency domain without excessively altering the overall frequency distribution.
[0031] like Figure 5 As shown, since amplitude and phase are represented in different ways, they are processed separately.
[0032] First, FFT is used to encode the latent data from the spatial domain. and After converting to the frequency domain and obtaining the transformed data, their amplitude components are further obtained. ), and phase components , . 1. Amplitude section Point convolution has the characteristics of not spanning multiple positions in the frequency domain and having no overlap, which can accurately focus on the interaction of channel-dimensional information, thereby capturing the correlation features between channels more efficiently.
[0033] To standardize the dimensionality of dual-source amplitude and offset correction terms and provide complete and dimension-compatible input data for point convolutional coding, an input integration operation is required first: concatenating signal branches A(f) sequentially according to channel indices. spe ), Spatial branch A(f) spa The element Aᵢ obtained after integrating the offset correction term δx and the offset correction term δx is... n ^((b,c,f)) corresponds to the amplitude or offset value of "the b-th sample, the c-th channel, and the f-th frequency point", respectively.
[0034] Therefore, for the amplitude component, the point convolution form of the fusion function is expressed as: In the formula: The attention weights are essentially three-dimensional tensors defined in the real number domain, with dimensions (B, 32, 513). Here, the first dimension B represents the number of fault samples processed in a batch, the second dimension 32 corresponds to the number of feature groups of the attention mechanism, and the third dimension 513 is the number of frequency domain sampling points after the time-domain signal undergoes a Fast Fourier Transform (FFT). The attention value vector is a four-dimensional tensor defined in the real number domain, with dimensions (B, 32, 64, 513). Here, the first dimension B represents the number of fault samples processed in the batch; the second dimension 32 represents the number of feature groups in the attention mechanism; the third dimension 64 corresponds to the number of frequency domain amplitude feature channels output by a single branch (signal branch / spatial branch); and the fourth dimension 513 represents the number of frequency domain sampling points after the time-domain signal undergoes a Fast Fourier Transform (FFT).
[0035] The convolution weights for the two-source feature encoding points serve as the core input for integrating the two-source amplitude features and the offset correction term. The encoding process is performed to convert it into features with high-dimensional discriminative power; Point convolution weights are generated for the attention weights, which are used to map the attention weights from the high-dimensional encoded features mentioned above. These are learnable parameters. Point convolution weights are generated for the attention value vector, which are used to map attention value vectors from high-dimensional encoded features. These weights are also learnable parameters.
[0036] GELU, as an activation function, performs a non-linear transformation on the encoded features of the integrated input. This enhances the discriminative power of the features and effectively avoids the vanishing gradient problem, ensuring the stability of the model training process. Reshape, the tensor dimension reconstruction function, adjusts the dimensional structure of the attention value vector without changing the feature values or the total number of elements, ensuring a precise match with the attention weights. This provides dimensional compatibility support for subsequent weighted fusion operations.
[0037] Phase part Phase components contain crucial information such as texture details, but point convolutions struggle to fully capture spatial representations. Therefore, this invention employs 3×3 convolutions to learn phase information. Furthermore, small changes in the frequency domain can easily trigger significant fluctuations in the spatial domain; hence, phase learning still utilizes INR (Implicit Neural Representation) interpolation. The specific processing of the phase components consists of two 3×3 convolution layers, whose output parameters correspond to the fused phase components. Finally, the processed frequency features are mapped back to time domain features using Inverse Fast Fourier Transform (IFFT). It is worth noting that a single frequency point in frequency space can correspond to multiple pixels at different locations in the spatial domain. Therefore, the receptive field of INR in the frequency domain is expanded in the spatial domain, thereby enhancing the ability to capture spatial information.
[0038] S4. The time-domain fused features and the frequency-domain processed time-domain features are fused using the dual-domain feature fusion module to generate dual-domain fused features; like Figure 6 As shown, the core objective of dual-domain feature fusion is to achieve seamless coupling between "time-domain multi-scale representation" and "time-domain representation restored after frequency-domain processing." To achieve this objective, this invention introduces a real-valued Gabor wavelet activation function with good time-frequency compactness. This function, through the synergistic effect of Gaussian envelope (achieving time-domain energy concentration) and cosine carrier (achieving frequency-domain energy concentration), adapts to the time-frequency correlation requirements of dual-domain features. Its function expression is as follows: Where w0 is the center frequency in the frequency domain, v0 is the standard deviation constant of the Gaussian function, and x is a vector in the time (or space) domain.
[0039] Using the aforementioned real-valued Gabor wavelet function, the time-domain multi-scale representation and the time-domain features restored after frequency-domain processing are fused together. S5. Input the dual-domain fusion features into the Transformer feature fusion module, and extract deep-level fusion features after feature depth mining; The Transformer feature fusion module in this model adopts an encoder structure, such as... Figure 2As shown, its core parameter configuration and data flow logic are as follows: The number of input channels of the Transformer module is set to twice the number of feature channels, where the number of feature channels is 64, so the number of input channels is 128. This setting can match the output feature dimension after the dual-branch frequency domain features and spatial features are fused by Gabor modulation and convolution; the core dimension of the model is set to 128, which is consistent with the number of input channels, to ensure the dimensionality of features in the entire process of mapping, encoding and restoration.
[0040] The multi-head attention mechanism uses 8 heads, which splits the input features into 8 parallel sub-features, each with a dimension of 16. Eight independent weight matrices are used to project the query vector, key vector, and value vector onto the input features to comprehensively capture the correlation information across different dimensions of the features. The dimensions of the projected query vector, key vector, and value vector are all (batch size, sequence length, 128). Similarly, the dimensions of the query vector, key vector, and value vector corresponding to each attention head after splitting are also (batch size, sequence length, 16). After attention weight calculation and weighted aggregation, the 8 parallel sub-features are concatenated to restore the feature output to (batch size, sequence length, 128).
[0041] The encoder unit adopts a two-layer stacked network structure. Each layer contains, in sequence, a multi-head attention mechanism layer, a residual connection layer, a layer normalization layer, a feedforward neural network layer, a secondary residual connection layer, and a secondary layer normalization layer. The parameters of the two encoder units are independent, and feature enhancement and correlation modeling are achieved through iterative encoding. The layer normalization layer is used to calibrate the feature distribution, ensuring the stability of feature representation. The residual connection layer uses a method of adding the sub-layer input and sub-layer output in tandem to effectively alleviate the gradient vanishing problem.
[0042] The hidden layer dimension of the feedforward neural network is set to 256. Internally, it consists of two fully connected linear layers and a GELU activation function. The first linear layer expands the encoded features from 128 dimensions to a 256-dimensional feature space. The GELU activation function introduces a non-linear transformation to enhance the feature representation capability. Subsequently, the second linear layer restores the feature dimension to 128 dimensions, forming a complete processing flow of "dimensional expansion - non-linear enhancement - dimensional restoration". The dropout regularization rate is set to 0.1. Dropout regularization operations are embedded at the outputs of both the multi-head attention mechanism layer and the feedforward neural network layer to improve the model's generalization ability and avoid overfitting.
[0043] The maximum sequence length of the positional encoding module is set to 1024. It constructs an encoding vector based on sine and cosine functions that matches the core dimensions of the model. This encoding vector is then added to the mapped features, injecting temporal positional information into the features and ensuring effective modeling of temporal features. The encoder layer sets the input features to meet the "batch size priority" format requirement: the original input feature format is (batch size, 128, sequence length), which is then processed by dimensionality transformation before being sent to the Transformer module. After encoding, the feature dimensions are restored to 128 dimensions through the output linear projection layer, and then restored to the original feature format through inverse dimensionality transformation, ensuring smooth feature flow between the Transformer module and upstream / downstream modules.
[0044] S6. Input the deep-level fusion features into the fault diagnosis module, which will then perform feature classification processing and output the fault identification results of the fluid transmission equipment.
[0045] like Figure 7 As shown, the fault diagnosis module takes the multi-scale spatiotemporal-frequency fusion features generated in the previous stage as input and outputs the classification probabilities of various fault states and normal operating states of the fluid transmission equipment.
[0046] The processing flow is as follows: the fused features pass sequentially through convolutional layers, flattening layers, fully connected layers, and activation layers to complete the fine-tuning, dimensionality transformation, and classification mapping of features. Key parameters are set as follows: the convolutional kernel size of the convolutional layer is set to 108×1, which is used to perform local feature enhancement and dimensionality regularization on the fused features; the classification probability is calculated using the Softmax function to normalize the probability distribution of multiple categories, and finally outputs accurate fault type identification results.
[0047] The hardware platform acceleration end outputs and displays the sound recognition accuracy corresponding to various faults and normal equipment operation status through the Universal Asynchronous Receiver Transmitter (UART) serial communication protocol.
[0048] Example 2: This embodiment provides a multimodal sound feature fusion-based fault classification acceleration system for fluid transport equipment, including: The dataset construction and training module collects sound data of different fault types and normal operating states of fluid transmission equipment to construct a fault sound dataset; based on this dataset, a neural network model is trained, which integrates a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module, and a fault diagnosis module. The signal preprocessing module acquires the real-time operating sound signal of the target fluid transmission component, preprocesses the signal to obtain a clean sound signal; The time-domain and frequency-domain feature extraction module inputs the clean sound signal into the multi-scale temporal convolution fusion module to extract the time-domain fusion features; at the same time, the clean sound signal is input into the frequency-domain feature extraction module, and after processing, the time-domain features after frequency-domain processing are obtained. The fusion module fuses the time-domain fused features with the frequency-domain processed time-domain features through the dual-domain feature fusion module to generate dual-domain fused features; The deep feature mining module inputs the dual-domain fusion features into the Transformer feature fusion module, and extracts the deep fusion features after feature depth mining. The fault diagnosis output module inputs the deeply fused features into the fault diagnosis module, which then performs feature classification processing and outputs the fault identification results of the fluid transmission equipment.
[0049] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.
[0050] Example 3: An electronic device is provided for running the aforementioned "accelerated method for fault classification of fluid transport equipment based on multimodal sound feature fusion". The electronic device includes: a processor, a memory, and optional communication interfaces / display devices / input devices, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S6 of the method described in Embodiment 1, specifically including but not limited to: S1. Collect sound data of different fault types and normal operation status of fluid transmission equipment to construct a fault sound dataset; train a neural network model based on the dataset, the model integrating a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module and a fault diagnosis module; S2. Acquire the real-time operating sound signal of the target fluid transmission component, preprocess the signal to obtain a clean sound signal; S3. Input the clean sound signal into the multi-scale temporal convolution fusion module to extract temporal fusion features; at the same time, input the clean sound signal into the frequency domain feature extraction module, and obtain the frequency domain processed temporal features after processing. S4. The time-domain fused features and the frequency-domain processed time-domain features are fused using the dual-domain feature fusion module to generate dual-domain fused features; S5. Input the dual-domain fusion features into the Transformer feature fusion module, and extract deep-level fusion features after feature depth mining; S6. Input the deep-level fusion features into the fault diagnosis module, which will then perform feature classification processing and output the fault identification results of the fluid transmission equipment.
[0051] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor is an FPGA; the memory can be RAM, ROM, flash memory, or a disk array. The device can interact with local / remote data storage (acquiring observation data and outputting inversion results) through a communication interface. The above hardware configuration does not constitute a limitation of the present invention.
[0052] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to execute the method steps S1 to S6 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.
[0053] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.
[0054] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0055] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.
Claims
1. A method for accelerating fault classification in fluid transport equipment using multimodal acoustic feature fusion, characterized in that, Includes the following steps: Collect sound data of different fault types and normal operating conditions of fluid transfer equipment, and construct a fault sound dataset; A neural network model is trained based on this dataset. The model integrates a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module, and a fault diagnosis module. Acquire the real-time operating sound signal of the target fluid transmission component, and preprocess the signal to obtain a clean sound signal; The clean sound signal is input into the multi-scale temporal convolutional fusion module to extract temporal fusion features; at the same time, the clean sound signal is input into the frequency domain feature extraction module, and after processing, the temporal features after frequency domain processing are obtained. The time-domain fused features and the frequency-domain processed time-domain features are fused using a dual-domain feature fusion module to generate dual-domain fused features. The dual-domain fusion features are input into the Transformer feature fusion module, and deep-level fusion features are extracted after feature depth mining. The deep-level fusion features are input into the fault diagnosis module, which performs feature classification processing and outputs the fault identification results of the fluid transmission equipment.
2. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 1, characterized in that, The fault sound dataset includes sound data of pump mechanical damage, valve seal damage, pipeline faults, and normal operating conditions.
3. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 1, characterized in that, The multi-scale temporal convolutional fusion module adopts the EEGNet backbone network structure and sets convolutional layers with different kernel sizes. The parameters include the number of filters NF, the kernel size KE, and the random deactivation ratio Dropout. The ratio of KE to NF is a multiple relationship. Features extracted at different scales are fused through a splicing operation.
4. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 1, characterized in that, The processing procedure of the frequency domain feature extraction module is as follows: first, it is downsampled by one-dimensional convolution and mapped to channels, and then it is normalized by adaptive pooling into discrete time domain features of fixed length as FFT input; the spatial domain latent code is transformed to the frequency domain by FFT to obtain amplitude components and phase components; the amplitude components are fused by point convolution, and the phase components are learned by two layers of convolution combined with INR interpolation.
5. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 4, characterized in that, The amplitude components are fused together using point convolution as follows: In the formula: For attention weights, For attention value vectors, Convolution weights for dual-source feature encoding points, Generate point convolution weights for the attention weights; Generate pointwise convolution weights for the attention value vector, with GELU as the activation function; (Signal branch) For spatial branching, For the offset correction term, each element This corresponds to "the b-th sample, the c-th channel, and the f-th frequency point".
6. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 1, characterized in that, The dual-domain feature fusion module fuses the time-domain fused features with the frequency-domain processed time-domain features using a complex Gabor wavelet activation function. The complex Gabor wavelet activation function is: Where w0 is the center frequency in the frequency domain, and v0 is the standard deviation constant of the Gaussian function. It is a vector in the time domain.
7. The method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion according to claim 1, characterized in that, The Transformer feature fusion module adopts an encoder structure with 128 input channels, 128 dimensions, and 8 heads for the multi-head attention mechanism. The encoder unit adopts a 2-layer stacked network structure, with each layer containing a multi-head attention mechanism layer, a residual connection layer, a layer normalization layer, a feedforward neural network layer, a secondary residual connection layer, and a secondary layer normalization layer.
8. A multimodal acoustic feature fusion-based fault classification acceleration system for fluid transport equipment, characterized in that, include: The dataset construction and training module collects sound data of different fault types and normal operating conditions of fluid transmission equipment to construct a fault sound dataset. A neural network model is trained based on this dataset. The model integrates a multi-scale temporal convolution fusion module, a frequency domain feature extraction module, a dual-domain feature fusion module, a Transformer feature fusion module, and a fault diagnosis module. The signal preprocessing module acquires the real-time operating sound signal of the target fluid transmission component, preprocesses the signal to obtain a clean sound signal; The time-domain and frequency-domain feature extraction module inputs the clean sound signal into the multi-scale temporal convolution fusion module to extract the time-domain fusion features; at the same time, the clean sound signal is input into the frequency-domain feature extraction module, and after processing, the time-domain features after frequency-domain processing are obtained. The fusion module fuses the time-domain fused features with the frequency-domain processed time-domain features through the dual-domain feature fusion module to generate dual-domain fused features; The deep feature mining module inputs the dual-domain fusion features into the Transformer feature fusion module, and extracts the deep fusion features after feature depth mining. The fault diagnosis output module inputs the deeply fused features into the fault diagnosis module, which then performs feature classification processing and outputs the fault identification results of the fluid transmission equipment.
9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor is an FPGA executing the program, it implements the method for accelerating fault classification of fluid transport equipment based on multimodal sound feature fusion as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the method for accelerating fault classification of fluid transport equipment by multimodal sound feature fusion as described in any one of claims 1-7.