Industrial Equipment Condition Monitoring Method and Device Based on Multimodal Data

Abstract

This disclosure presents an embodiment of a method and apparatus for industrial equipment condition monitoring based on multimodal data. One specific implementation of the method includes: real-time signal acquisition of a target industrial equipment in operation to obtain a real-time signal information sequence; fusing and denoising the real-time acoustic and vibration signals included in each real-time signal information in the sequence to generate denoised real-time signal information; extracting dynamic signal features from each denoised real-time signal information in the sequence to generate real-time signal features; generating a fused signal feature sequence based on the real-time signal feature sequence; identifying the equipment condition based on the fused signal feature sequence to generate equipment condition information; and controlling the target industrial equipment based on the equipment condition information. This implementation achieves real-time and effective condition monitoring of industrial equipment in operation.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of industrial equipment condition monitoring, sensor data processing, and machine learning, and specifically to industrial equipment condition monitoring methods and apparatus based on multimodal data. Background Technology

[0002] Condition monitoring of critical industrial equipment (such as gas turbines) is one of the necessary means to ensure the safe operation of the equipment. Currently, the mainstream methods are mainly divided into the following two types: one is to monitor the condition of industrial equipment through regular maintenance and inspection; the other is to trigger condition monitoring of industrial equipment through fixed rules set by expert systems.

[0003] However, when using the above method, the following technical problems often arise: First, the regular maintenance method has the drawback of being outdated, especially in that it cannot promptly identify sudden abnormal states during equipment operation. Second, monitoring methods based on fixed rules have extremely high rule maintenance costs, and they are difficult to effectively cover the complex and diverse types of faults for industrial equipment with complex structures, resulting in a lack of effective triggering for some abnormal states. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure propose a method and apparatus for industrial equipment condition monitoring based on multimodal data to solve the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide a method for industrial equipment condition monitoring based on multimodal data. The method includes: real-time signal acquisition of a target industrial equipment in operation to obtain a real-time signal information sequence, wherein the target industrial equipment is a gas turbine used to drive a generator to generate electricity, and the target industrial equipment is provided with M monitoring areas, where M≥2; the real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area; and fusing and denoising the real-time acoustic signals and real-time vibration signals included in each real-time signal information sequence to generate denoised real-time signal information, thus obtaining denoised real-time signal information. The sequence comprises a denoised acoustic signal and a denoised vibration signal. Dynamic signal features are extracted from each denoised real-time signal in the sequence to generate real-time signal features, resulting in a real-time signal feature sequence composed of acoustic and vibration signal features. A fused signal feature sequence is generated based on this sequence. Equipment status is identified based on the fused signal feature sequence to generate equipment status information, including: area identifier, status type, and status confidence level. Equipment control is then performed on the target industrial equipment based on the equipment status information.

[0007] Secondly, some embodiments of this disclosure provide an industrial equipment condition monitoring device based on multimodal data. The device includes: a real-time signal acquisition unit configured to acquire real-time signals from a target industrial equipment in operation to obtain a real-time signal information sequence, wherein the target industrial equipment is a gas turbine used to drive a generator to generate electricity, and the target industrial equipment is provided with M monitoring areas, M≥2, and the real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area; and a fusion and noise reduction unit configured to fuse and reduce the real-time acoustic signals and real-time vibration signals included in each real-time signal information in the real-time signal information sequence to generate noise-reduced real-time signal information, thereby obtaining a noise-reduced real-time signal information sequence, wherein the noise-reduced real-time signal... The information consists of denoised acoustic signals and denoised vibration signals; a dynamic signal feature extraction unit is configured to extract dynamic signal features from each denoised real-time signal in the denoised real-time signal information sequence to generate real-time signal features, resulting in a real-time signal feature sequence, wherein the real-time signal features consist of acoustic signal features and vibration signal features; a generation unit is configured to generate a fused signal feature sequence based on the real-time signal feature sequence; an equipment status identification unit is configured to identify the equipment status based on the fused signal feature sequence to generate equipment status information, wherein the equipment status information includes: area identifier, status type, and status confidence; and an equipment control unit is configured to control the target industrial equipment based on the equipment status information.

[0008] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0009] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0010] The above embodiments of this disclosure have the following beneficial effects: The industrial equipment condition monitoring method based on multimodal data, as described in some embodiments of this disclosure, achieves real-time and effective condition monitoring of industrial equipment in operation. Specifically, real-time signal acquisition is performed on the target industrial equipment in operation to obtain a real-time signal information sequence. The target industrial equipment is a gas turbine used to drive a generator to generate electricity. The target industrial equipment is configured with M monitoring areas, where M≥2. The real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area. In practice, for precision-structured industrial equipment, considering the cost of sensor setup and signal acquisition, this disclosure performs signal acquisition from both acoustic and vibration signal dimensions. Next, the real-time acoustic signals and real-time vibration signals included in each real-time signal information sequence are fused and denoised to generate denoised real-time signal information, resulting in a denoised real-time signal information sequence. The denoised real-time signal information is composed of denoised acoustic signals and denoised vibration signals. In practice, conventional methods often process different signal dimensions independently. However, this disclosure considers that vibration signals are elastic waves transmitted within mechanical structures, while acoustic signals are pressure waves excited by vibration. This means there is a signal correlation between acoustic and vibration signals, but also differences in their signal expression. Therefore, a fusion-based denoising method is used to improve signal quality and filter noise interference. Next, dynamic signal feature extraction is performed on each denoised real-time signal in the denoised real-time signal information sequence to generate real-time signal features, resulting in a real-time signal feature sequence. These real-time signal features consist of acoustic and vibration signal features. In practice, acoustic and vibration features differ due to different abnormal states. Feature extraction based on the overall signal can easily lead to key features being masked by other non-key features. Therefore, dynamic signal feature extraction is used to enhance the representation of key features. Furthermore, a fused signal feature sequence is generated based on the real-time signal feature sequence. This signal fusion method transforms different features (acoustic and vibration signal features) to the same feature dimension. Furthermore, equipment status identification is performed based on the aforementioned fused signal feature sequence to generate equipment status information, which includes: area identifier, status type, and status confidence level. This enables the location and status identification of monitoring areas exhibiting anomalies. Finally, based on the aforementioned equipment status information, equipment control is implemented on the target industrial equipment. This method achieves real-time and effective status monitoring of industrial equipment in operation. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0012] Figure 1 This is a flowchart of some embodiments of the industrial equipment condition monitoring method based on multimodal data according to this disclosure; Figure 2 This is a schematic diagram of the real-time signal information sequence acquisition process; Figure 3 This is a schematic diagram of the network structure of the fusion noise reduction network; Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the industrial equipment condition monitoring device based on multimodal data according to this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of an industrial equipment condition monitoring method based on multimodal data according to the present disclosure. This industrial equipment condition monitoring method based on multimodal data includes the following steps: Step 101: Real-time signal acquisition is performed on the target industrial equipment in operation to obtain a real-time signal information sequence.

[0020] In some embodiments, the execution subject (e.g., a computing device) of the industrial equipment condition monitoring method based on multimodal data can perform real-time signal acquisition on the target industrial equipment in the equipment operation state to obtain a real-time signal information sequence.

[0021] The target industrial equipment is a gas turbine used to drive a generator to produce electricity. The target industrial equipment has M monitoring areas, where M ≥ 2. Each monitoring area is a region within the target industrial equipment used to collect real-time acoustic and vibration signals. Specifically, the monitoring areas may be located at: the bearing housing, the rotor area, or the thrust bearing area. High-temperature resistant acoustic sensors and high-temperature resistant vibration sensors can be installed within the monitoring areas. The high-temperature resistant acoustic sensors are high-temperature resistant sensors used for acoustic signal acquisition. The high-temperature resistant vibration sensors are high-temperature resistant sensors used for vibration signal acquisition. Real-time signal information includes real-time acoustic and vibration signals corresponding to the same monitoring area. Real-time acoustic signals can be acquired by high-temperature resistant acoustic sensors. Real-time vibration signals can be acquired by high-temperature resistant vibration sensors.

[0022] As an example, see Figure 2 The diagram illustrates the acquisition process of the real-time signal information sequence. First, the computing device sends a device operation status confirmation request to the target industrial equipment. Upon receiving the request, the target equipment replies with an operation status identifier. The device operation status request confirms whether the target equipment is in operation. The operation status identifier indicates whether the target equipment is in operation. For example, a "1" identifier indicates the equipment is in operation, while a "0" identifier indicates it is not in operation. Next, upon receiving the operation status identifier, the computing device simultaneously sends signal acquisition commands to the high-temperature resistant acoustic sensors and high-temperature resistant vibration sensors installed in each of the M monitoring areas. Figure 2For example, the M monitoring areas may include: monitoring area A1, monitoring area A2, monitoring area A3, monitoring area A4, and monitoring area A5. The monitoring areas are set in order, so M real-time signal information will be obtained, which constitute a real-time signal information sequence.

[0023] It should be noted that the aforementioned computing devices can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed on the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0024] Step 102: Fusion and noise reduction are performed on the real-time acoustic signal and real-time vibration signal included in each real-time signal information sequence to generate noise-reduced real-time signal information, thus obtaining the noise-reduced real-time signal information sequence.

[0025] In some embodiments, the execution entity may fuse and denoise the real-time acoustic signal and real-time vibration signal included in each real-time signal information in the real-time signal information sequence to generate denoised real-time signal information, thereby obtaining a denoised real-time signal information sequence.

[0026] The real-time signal information after noise reduction refers to the real-time signal information after signal denoising. This real-time signal information consists of the denoised acoustic signal and the denoised vibration signal. Specifically, the vibration signal is the elastic wave transmitted within the mechanical structure, while the acoustic signal is the pressure wave excited by vibration. Therefore, there is a signal correlation between the acoustic and vibration signals, especially for the same vibration source; the acoustic and vibration signals can be considered as signal expressions in two different dimensions.

[0027] In practice, considering that the signal noise corresponding to vibration signals is mainly transmitted from nearby mechanical structures, and the signal noise corresponding to acoustic signals is mainly transmitted through the air, especially since, unless it is signal noise generated by extremely strong sound waves, the signal noise generated by sound waves generally will not excite high-temperature resistant vibration sensors, the noise between real-time acoustic signals and real-time noise-reduced signals collected from the same monitoring area can be estimated and eliminated through fusion noise reduction. Especially in the harsh working environment of the target industrial equipment, this method can effectively suppress noise interference compared to independent processing. Specifically, firstly, for the real-time signal information, including real-time acoustic signals and real-time vibration signals, the cross-power spectrum and auto-power spectrum at the frequency point are calculated to estimate the correlation function value, thereby determining whether the real-time acoustic signal and real-time vibration signal originate from the same source at that frequency point. Before calculating the spectrum, the real-time vibration signal and real-time acoustic signal need to be normalized. Next, for the high-coherence frequency band, an LMS (Least Mean Square) adaptive filter is used, taking the real-time vibration signal as input, to filter out signals emitted by the signal source due to abnormal conditions from the real-time acoustic signal, thus obtaining signal noise specific to the real-time acoustic signal. Similarly, an LMS (Least Mean Square) adaptive filter is used, taking the real-time acoustic signal as input, to filter out signals emitted by the signal source due to abnormal conditions from the real-time vibration signal, thus obtaining signal noise specific to the real-time vibration signal. Finally, based on the two obtained signal noises, noise stripping is performed on the real-time acoustic signal and the real-time vibration signal respectively, thus obtaining the denoised real-time signal information including the denoised acoustic signal and the denoised vibration signal.

[0028] In some optional implementations of certain embodiments, the execution entity performs fusion and noise reduction on the real-time acoustic signal and real-time vibration signal included in each real-time signal information in the real-time signal information sequence to generate noise-reduced real-time signal information, including: Step S1: Perform signal normalization processing on the real-time acoustic signal and real-time vibration signal included in the above real-time signal information to obtain normalized acoustic signal and normalized vibration signal.

[0029] The normalized acoustic signal is the real-time acoustic signal after normalization. The normalized vibration signal is the real-time vibration signal after normalization.

[0030] In practice, for example, the max-min normalization method can be used to normalize the real-time acoustic signal and real-time vibration signal included in the aforementioned real-time signal information, respectively, to obtain normalized acoustic and vibration signals. This transforms the real-time vibration and acoustic signals to the same numerical range and distribution pattern, thereby eliminating differences in dimensions and amplitude magnitudes, facilitating subsequent feature extraction and processing at the same scale. Alternatively, a normalization layer can also be used to normalize the real-time acoustic signal and real-time vibration signal included in the aforementioned real-time signal information, respectively, to obtain normalized acoustic and vibration signals.

[0031] Step S2: Through the acoustic feature extraction branch, the initial signal features are extracted from the normalized acoustic signal to obtain the initial acoustic signal features.

[0032] The initial acoustic signal features are the signal features corresponding to the normalized acoustic signal. The acoustic feature extraction branch is a feature extraction network used to extract signal features from the normalized acoustic signal.

[0033] As an example, see Figure 3 The diagram shows the network structure of the fusion denoising network, which includes branches for acoustic feature extraction, vibration feature extraction, feature cross-projection, residuals, acoustic signal reconstruction, and vibration signal reconstruction. The input to the acoustic feature extraction branch is the normalized acoustic signal. Specifically, the normalized acoustic signal is input to the acoustic feature extraction branch in a frame-by-frame manner. The frame window can be 512, and the frame overlap ratio is 50%. The acoustic feature extraction branch consists of a one-dimensional depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function. The one-dimensional depthwise separable convolutional layer corresponds to Depthwise and Pointwise convolution operations. The Depthwise convolution operation has 1 input channel, 16 output channels, a kernel size of 5, and a stride of 1. The Pointwise convolution operation has 16 input channels, 16 output channels, a kernel size of 1×1, and an output dimension of 512×16. The batch normalization layer contains 16 channels to divide the output of the pointwise convolution operation into 512×1×16 segments, thereby achieving normalization processing for each independent channel. The output dimension of the acoustic feature extraction branch is K×512×16, where K is the number of frames.

[0034] Step S3: Through the vibration feature extraction branch, the initial signal features of the normalized vibration signal are extracted to obtain the initial vibration signal features.

[0035] The acoustic feature extraction branch and the vibration feature extraction branch are configured in parallel and have the same network structure. The initial vibration signal features are the signal features corresponding to the normalized vibration signal. The vibration feature extraction branch is a feature extraction network used to extract signal features from the normalized vibration signal.

[0036] As an example, see further. Figure 3 The vibration feature extraction branch takes a normalized vibration signal as its input. Specifically, the normalized vibration signal is input to the vibration feature extraction branch in frames. The frame window can be 512, with a frame overlap of 50%. The vibration feature extraction branch consists of a one-dimensional depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function. The one-dimensional depthwise separable convolutional layer corresponds to both Depthwise and Pointwise convolution operations. The Depthwise convolution operation has 1 input channel, 16 output channels, a kernel size of 5, and a stride of 1. The Pointwise convolution operation has 16 input channels, 16 output channels, a kernel size of 1×1, and an output dimension of 512×16. The batch normalization layer contains 16 channels to divide the output of the Pointwise convolution operation into 512×1×16, thus achieving normalization processing for each independent channel. The output dimension of the vibration feature extraction branch is K×512×16, where K is the number of frames.

[0037] In practice, the acoustic feature extraction branch and the vibration feature extraction branch use the same network structure, which ensures that the initial vibration signal features and the initial acoustic signal features are mapped to the same feature space. At the same time, the redesigned, lightweight network structure can ensure processing speed while effectively extracting features, thus meeting the requirements of real-time processing.

[0038] Step S4: By using the feature cross-projection branch, the above-mentioned initial acoustic signal features and the above-mentioned initial vibration signal features are projected and filtered to obtain the projected acoustic signal features and the projected vibration signal features.

[0039] The feature cross-projection branch is the network structure used for feature cross-projection filtering. For example... Figure 3 As shown, the feature cross-projection branch consists of a first projection branch and a second projection branch. The first projection branch corresponds to the initial acoustic signal features. The second projection branch corresponds to the initial vibration signal features. The network structures of the first and second projection branches are identical and symmetrical. The feature dimensions of the projected acoustic signal features and the projected vibration signal features are identical, both being K×512×16.

[0040] In practice, taking the first projection branch as an example, the first projection branch consists of a 1D convolutional layer and a Sigmoid activation function. The kernel size of the 1D convolutional layer is 5, and the stride is 1. Specifically, firstly, the 1D convolutional layer and Sigmoid activation function in the first projection branch compress the initial vibration signal into a K×512×6 projection vector. The vector values ​​in the projection vector range from [0,1]. Next, the projection vector calculated by the first projection branch and the initial acoustic signal are multiplied element-wise to obtain the projected acoustic signal features. Taking the second projection branch as an example, the second projection branch also consists of a 1D convolutional layer and a Sigmoid activation function. The kernel size of the 1D convolutional layer is 5, and the stride is 1. Specifically, the 1D convolutional layer and Sigmoid activation function in the second projection branch compress the initial acoustic signal into a K×512×6 projection vector. The vector values ​​in the projection vector range from [0,1]. Next, the projection vector calculated by the second projection branch and the initial vibration signal are multiplied element-wise to obtain the projected vibration signal characteristics. Since the vibration signal and acoustic signal originate from the same source, they exhibit coherence in their fault characteristics. However, their unique noise characteristics often do not appear in the signal characteristics of the other mode. Therefore, this disclosure uses projection vector control to preserve the common source characteristics and suppress unique noise characteristics. Furthermore, the projection process is efficient and concise, effectively meeting processing speed requirements.

[0041] Step S5: Based on the above initial acoustic signal characteristics, the above initial vibration signal characteristics, the above projected acoustic signal characteristics, and the above projected vibration signal characteristics, perform signal reconstruction to obtain the denoised acoustic signal and denoised vibration signal included in the real-time signal information after noise reduction.

[0042] In practice, since the initial acoustic signal features and the projected acoustic signal features have the same feature dimensions, as do the initial vibration signal features and the projected vibration signal features, the initial acoustic signal features and the projected acoustic signal features can be superimposed element-wise, processed by the ReLU activation function, and then transposed and convolved to obtain the denoised acoustic signal. Similarly, the initial vibration signal features and the projected vibration signal features can be superimposed element-wise, processed by the ReLU activation function, and then transposed and convolved to obtain the denoised vibration signal.

[0043] Optionally, the signal reconstruction based on the initial acoustic signal features, the initial vibration signal features, the projected acoustic signal features, and the projected vibration signal features to obtain the denoised real-time signal information includes the denoised acoustic signal and the denoised vibration signal, including: Step S51: By using residual branching, the initial acoustic signal features and the projected acoustic signal features are residually fused to obtain fused acoustic signal features.

[0044] As an example, see further. Figure 3 The residual branch is connected to the feature cross-projection branch. The residual branch consists of a feature stacking layer and a ReLU activation function. The feature stacking layer can element-wise stack the initial acoustic signal features and the projected acoustic signal features.

[0045] Step S52: By using residual branching, the initial vibration signal features and the projected vibration signal features are residually fused to obtain fused vibration signal features.

[0046] The feature dimension of the fused vibration signal features is K×512×16.

[0047] In practice, since the initial vibration signal features, the projected vibration signal features, the initial acoustic signal features, and the projected acoustic signal features all have the same feature dimensions, the network structure can be simplified by sharing a single residual branch. Specifically, the feature overlay layer can overlay the initial vibration signal features and the projected vibration signal features element by element.

[0048] Step S53: Through the acoustic signal reconstruction branch, perform time-domain reconstruction on the above-mentioned fused acoustic signal features to obtain the denoised acoustic signal included in the real-time signal information after denoising.

[0049] The acoustic signal reconstruction branch is a network used to recover the acoustic signal from the fused acoustic signal features. The feature dimension of the fused acoustic signal features is K×512×16.

[0050] Step S54: Through the vibration signal reconstruction branch, perform time-domain reconstruction on the above-mentioned fused vibration signal features to obtain the denoised vibration signal included in the real-time signal information after denoising.

[0051] The acoustic signal reconstruction branch and the vibration signal reconstruction branch are set up in parallel and have the same network structure. For example... Figure 3 As shown, both the acoustic signal reconstruction branch and the aforementioned vibration signal reconstruction branch include a transposed convolutional layer.

[0052] In practice, the aforementioned fusion denoising network effectively reduces the number of parameters by optimizing the network structure, thereby achieving sub-millisecond inference speeds. By introducing a mutual denoising mechanism between the two modes, accurate and effective homogeneous denoising is achieved.

[0053] Step 103: Perform dynamic signal feature extraction on each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features and obtain a real-time signal feature sequence.

[0054] In some embodiments, the execution entity may perform dynamic signal feature extraction on each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features and obtain a real-time signal feature sequence.

[0055] The real-time signal features consist of acoustic signal features and vibration signal features. The acoustic signal features are obtained by frequency division feature extraction of the denoised acoustic signal. The vibration signal features are obtained by frequency division feature extraction of the denoised vibration signal.

[0056] In practice, firstly, wavelet transform can be used to decompose the denoised acoustic signal and denoised vibration signal into different frequency bands in the real-time signal information. Then, by extracting features from each frequency band (e.g., extracting statistical features and time-frequency features from different frequency bands), the acoustic and vibration signal features in the real-time signal are obtained.

[0057] In some optional implementations of certain embodiments, the execution entity performs dynamic signal feature extraction on each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features, including: Step S1: Perform signal frequency band decomposition on the above-mentioned noise-reduced real-time signal information, including the noise-reduced acoustic signal and the noise-reduced vibration signal, to obtain the first frequency band signal group and the second frequency band signal group.

[0058] The first frequency band signal group corresponds to the aforementioned noise-reduced acoustic signals. The first frequency band signals in the first frequency band signal group are acoustic signals corresponding to different frequency bands after signal decomposition. The second frequency band signal group corresponds to the aforementioned noise-reduced vibration signals. The second frequency band signals in the second frequency band signal group are vibration signals corresponding to different frequency bands after signal decomposition. The first frequency band signals in the first frequency band signal group correspond to the low-frequency band, mid-frequency band, and high-frequency band, respectively. The second frequency band signals in the second frequency band signal group correspond to the low-frequency band, mid-frequency band, and high-frequency band, respectively.

[0059] In practice, the denoised acoustic signal and the denoised vibration signal included in the above-mentioned real-time signal information can be decomposed into a first frequency band signal group and a second frequency band signal group by using Discrete Wavelet Transform (DWPT).

[0060] As an example, assuming that the noise-reduced acoustic signal and the noise-reduced vibration signal in the real-time signal information include the same sampling frequency, such as Fs (Hz), then the signal frequency range of the low-frequency band corresponding to the first and second frequency band signal groups is 0 ~ Fs / 6 Hz, the corresponding mid-frequency band signal frequency range is Fs / 6 ~ Fs / 3 Hz, and the corresponding high-frequency band signal frequency range is Fs / 3 ~ Fs / 2 Hz.

[0061] Step S2: Extract frequency band features from each first frequency band signal in the first frequency band signal group to generate first frequency band signal features and obtain the first frequency band signal feature group.

[0062] Among them, the signal characteristics of the first frequency band are the signal characteristics corresponding to the first frequency band signal.

[0063] In practice, a pre-trained three-way convolutional network can be used to extract features from three first-frequency band signals in parallel, thus obtaining the first-frequency band signal feature group. The three-way convolutional network consists of three parallel convolutional networks, each composed of a 1D convolutional layer, a batch normalization layer, a ReLU activation function, and an average pooling layer.

[0064] Step S3: Extract frequency band features from each second frequency band signal in the second frequency band signal group to generate second frequency band signal features and obtain the second frequency band signal feature group.

[0065] Among them, the signal characteristics of the second frequency band are the signal characteristics corresponding to the second frequency band signal.

[0066] In practice, a pre-trained three-way convolutional network can be used to extract features from three second-frequency band signals in parallel, thus obtaining the second-frequency band signal feature group. The three-way convolutional network consists of three parallel convolutional networks, each composed of a 1D convolutional layer, a batch normalization layer, a ReLU activation function, and an average pooling layer.

[0067] Step S4: Perform attention-weighted fusion on the first frequency band signal features in the first frequency band signal feature group to obtain the acoustic signal features included in the real-time signal features.

[0068] The attention-weighted fusion method uses a 1×3 weight matrix to weight and fuse the first frequency band signal features in the aforementioned first frequency band signal feature group. The weight matrix contains three weight values, the sum of which is 1. For example, the weight matrix can be dynamically calculated using an attention mechanism, or it can be calculated using a weighted network. The concatenated features obtained by stitching together the first frequency band signal features in the first frequency band signal feature group can be used as input to the weight matrix. The weight matrix consists of one input layer, two hidden layers, and one output layer. The first hidden layer has 24 nodes, the second hidden layer has 16 nodes, and the output layer has 3 nodes. The output layer outputs a 1×3 weight matrix. By adjusting the weight values, the effective feature representation of the frequency band containing the anomaly is ensured.

[0069] Step S5: Perform attention-weighted fusion on the second frequency band signal features in the second frequency band signal feature group to obtain the vibration signal features included in the real-time signal features.

[0070] The attention-weighted fusion method uses a 1×3 weight matrix to weight and fuse the second-band signal features from the aforementioned second-band signal feature group. The weight matrix contains three weight values, the sum of which is 1. For example, the weight matrix can be dynamically calculated using an attention mechanism, or it can be calculated using a weighted network. The concatenated features obtained by stitching together the various second-band signal features from the second-band signal feature group can be used as input to the weight matrix. The weight matrix consists of one input layer, two hidden layers, and one output layer. The first hidden layer has 24 nodes, the second hidden layer has 16 nodes, and the output layer has 3 nodes. The output layer outputs a 1×3 weight matrix. By adjusting the weight values, the effective feature representation of the frequency band containing the anomaly is ensured.

[0071] Step 104: Generate a fused signal feature sequence based on the real-time signal feature sequence.

[0072] In some embodiments, the aforementioned execution entity may generate a fused signal feature sequence based on the real-time signal feature sequence.

[0073] In this sequence, the fused signal features in the fused signal feature sequence correspond one-to-one with the real-time signal features in the real-time signal feature sequence. The fused signal features are obtained by fusing the acoustic signal features and vibration signal features included in the corresponding real-time signal features.

[0074] In practice, since vibration signals and sound signals are different dimensions of signal expression for the same signal source, and the feature dimensions of acoustic signal features and vibration signal features are consistent, and the dual-path network design with the same structure ensures the consistency of the feature space during feature processing, the acoustic signal features and vibration signal features included in the real-time signal features can be directly spliced ​​along the channel to obtain fused signal features.

[0075] Step 105: Perform device status identification based on the fused signal feature sequence to generate device status information.

[0076] In some embodiments, the aforementioned execution entity can perform device status identification based on the fused signal feature sequence to generate device status information.

[0077] The equipment status information represents the monitoring area where an anomaly exists, and its corresponding anomaly type. The equipment status information includes: area identifier, status type, and status confidence level. The area identifier represents the area number of the monitoring area where the anomaly exists. The status type represents the anomaly type. The status confidence level represents the confidence level of the status type. The status confidence level ranges from [0,1].

[0078] In practice, firstly, for each fused signal feature in the fused signal feature sequence, a multi-classifier can be used to generate a state type and state confidence score, taking the fused signal feature as input. When the state confidence score is greater than a preset confidence threshold and the state type is a risk state type from a preset risk state type list, device status information is generated based on the region number, state type, and state confidence score corresponding to the fused signal feature. The multi-classifier can be composed of a feature shaping network and a Softmax function. The feature shaping network consists of multiple sequential fully connected layers. In some optional implementations of certain embodiments, the execution entity performs device state identification based on the fused signal feature sequence to generate device state information, including: Step S1: For each fused signal feature in the above fused signal feature sequence, perform the following identification steps: Step S11: Perform feature decoding on the above fused signal features to obtain the decoded signal features.

[0079] The decoded signal features are obtained by shaping the fused signal features through a downsampling network. The downsampling network consists of three 1D convolutional layers and four fully connected layers. The feature dimension of the decoded signal features is 1×64.

[0080] Step S12: Classify the states based on the above-mentioned decoded signal characteristics to obtain candidate device state information.

[0081] The aforementioned candidate device status information includes: region identifier, status type, and status confidence level.

[0082] In practice, firstly, a multi-classifier network containing three fully connected layers and one softmax function is used as input to decode signal features to generate the state type and state confidence of candidate device state information. Next, since there is a one-to-one correspondence between the fused signal features and the monitoring area, the region identifier of the candidate device state information is obtained by determining the region number corresponding to the fused signal features.

[0083] Step S2: Filter the obtained candidate device status information sequence to obtain device status information.

[0084] Multiple filtering conditions can be set to filter the obtained candidate device status information sequence to obtain the device status information. For example, multiple filtering conditions could be: the status confidence level is greater than a preset confidence threshold; the status type is a preset risk status type.

[0085] Step 106: Control the target industrial equipment based on the equipment status information.

[0086] In some embodiments, the aforementioned executing entity may perform equipment control on the target industrial equipment based on equipment status information.

[0087] In practice, multiple control trigger rules can be pre-set, with each rule bound to a corresponding device control operation. By combining the device status information, including the status type and status confidence level, the trigger rules are automatically activated, thereby achieving device control of the target industrial equipment.

[0088] In some optional implementations of certain embodiments, the execution entity performs equipment control on the target industrial equipment based on the equipment status information, including: Step S1: Determine the hazard level based on the area identifier and status type included in the above equipment status information.

[0089] The hazard levels can be categorized into three levels: Level 1, Level 2, and Level 3. Level 1 hazard corresponds to high risk. Level 2 hazard corresponds to medium risk. Level 3 hazard corresponds to low risk.

[0090] In practice, the location of a hazard can be determined based on area identification, and the hazard level can be mapped based on the state type. Specifically, different hazard levels have pre-set corresponding state type lists. For example, the state type list corresponding to a Level 1 hazard level may include: rotor fracture, overspeed, blade shedding, etc. The state type list corresponding to a Level 2 hazard level may include: severe bearing wear, severe seal leakage, etc. The state type list corresponding to a Level 3 hazard level may include: minor bearing wear, minor seal leakage, etc.

[0091] Step S2: Determine the equipment dependency level of the current power generation task on the aforementioned target industrial equipment.

[0092] Among them, the equipment dependency level represents the necessity of the target industrial equipment for the current power generation task.

[0093] In practice, for each power generation task, critical and non-critical equipment can be pre-defined and stored in an equipment dependency list. Critical equipment refers to irreplaceable industrial equipment that the power generation task depends on. Non-critical equipment refers to replaceable industrial equipment that the power generation task depends on. Therefore, the necessity of a target industrial equipment for the current power generation task can be determined by querying the equipment dependency list. Equipment dependency levels can include: primary equipment dependency level and secondary equipment dependency level. Primary equipment dependency level indicates that the industrial equipment is critical. Secondary equipment dependency level indicates that the industrial equipment is non-critical.

[0094] Step S3: Determine the control level based on the above hazard level and the above equipment dependency level.

[0095] The control level represents the necessity of equipment control. Control levels can include: Level 1, Level 2, and Level 3. Level 1 indicates only alarm information is pushed; Level 2 indicates derating control of industrial equipment; and Level 3 indicates emergency shutdown control of industrial equipment.

[0096] In practice, a lightweight decision tree model can be trained to obtain the control level by taking the hazard level and equipment dependency level as input. A three-level quantization approach decouples the hazard level, equipment dependency level, and control level, facilitating dynamic adjustments based on actual needs and improving the interpretability of the level determination process.

[0097] Step S4: In response to the above control level being the first control level and the status confidence level included in the above equipment status information being within the first confidence level range, an alarm message is pushed to the control terminal corresponding to the above target industrial equipment.

[0098] The control terminal is the operating terminal corresponding to the target industrial equipment. Alarm information is used to provide alerts to the operators corresponding to the control terminal.

[0099] Step S5: In response to the control level being the second control level and the status confidence level included in the equipment status information being within the second confidence level range, derating control is triggered for the target industrial equipment.

[0100] The upper limit of the first confidence interval is smaller than the lower limit of the second confidence interval.

[0101] In practice, the target industrial equipment can be de-capacitated according to a preset ratio (e.g., 80%).

[0102] Step S6: In response to the control level being the third control level and the status confidence level included in the equipment status information being in the third confidence level region, a shutdown control for the target industrial equipment is triggered.

[0103] The upper limit of the second confidence interval is smaller than the lower limit of the third confidence interval. In particular, the first, second, and third confidence intervals constitute a continuous interval.

[0104] In practice, a shutdown command can be sent to the target industrial equipment to achieve shutdown control of the target industrial equipment.

[0105] In some optional implementations of some embodiments, the above method further includes: Step S1: Synchronize the equipment status information to the equipment twin corresponding to the target industrial equipment.

[0106] Among them, the equipment twin is a digital virtual mapping corresponding to the aforementioned target industrial equipment.

[0107] In practice, equipment status information can be synchronized to the monitoring area in the equipment twin that corresponds to the area identifier included in the equipment status information.

[0108] Step S2: The device control operation and the updated device status are displayed synchronously on the above-mentioned device twin.

[0109] In practice, when different equipment control operations are triggered according to the control level, the specific equipment control operations, as well as the equipment status after the execution and update of the equipment control operations, are updated in the equipment twin through data synchronization.

[0110] The above embodiments of this disclosure have the following beneficial effects: The industrial equipment condition monitoring method based on multimodal data, as described in some embodiments of this disclosure, achieves real-time and effective condition monitoring of industrial equipment in operation. Specifically, real-time signal acquisition is performed on the target industrial equipment in operation to obtain a real-time signal information sequence. The target industrial equipment is a gas turbine used to drive a generator to generate electricity. The target industrial equipment is configured with M monitoring areas, where M≥2. The real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area. In practice, for precision-structured industrial equipment, considering the cost of sensor setup and signal acquisition, this disclosure performs signal acquisition from both acoustic and vibration signal dimensions. Next, the real-time acoustic signals and real-time vibration signals included in each real-time signal information sequence are fused and denoised to generate denoised real-time signal information, resulting in a denoised real-time signal information sequence. The denoised real-time signal information is composed of denoised acoustic signals and denoised vibration signals. In practice, conventional methods often process different signal dimensions independently. However, this disclosure considers that vibration signals are elastic waves transmitted within mechanical structures, while acoustic signals are pressure waves excited by vibration. This means there is a signal correlation between acoustic and vibration signals, but also differences in their signal expression. Therefore, a fusion-based denoising method is used to improve signal quality and filter noise interference. Next, dynamic signal feature extraction is performed on each denoised real-time signal in the denoised real-time signal information sequence to generate real-time signal features, resulting in a real-time signal feature sequence. These real-time signal features consist of acoustic and vibration signal features. In practice, acoustic and vibration features differ due to different abnormal states. Feature extraction based on the overall signal can easily lead to key features being masked by other non-key features. Therefore, dynamic signal feature extraction is used to enhance the representation of key features. Furthermore, a fused signal feature sequence is generated based on the real-time signal feature sequence. This signal fusion method transforms different features (acoustic and vibration signal features) to the same feature dimension. Furthermore, equipment status identification is performed based on the aforementioned fused signal feature sequence to generate equipment status information, which includes: area identifier, status type, and status confidence level. This enables the location and status identification of monitoring areas exhibiting anomalies. Finally, based on the aforementioned equipment status information, equipment control is implemented on the target industrial equipment. This method achieves real-time and effective status monitoring of industrial equipment in operation.

[0111] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an industrial equipment condition monitoring device based on multimodal data. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this industrial equipment condition monitoring device based on multimodal data can be specifically applied to various electronic devices.

[0112] like Figure 4 As shown, an industrial equipment condition monitoring device 400 based on multimodal data in some embodiments includes: a real-time signal acquisition unit 401, a fusion and noise reduction unit 402, a dynamic signal feature extraction unit 403, a generation unit 404, an equipment condition identification unit 405, and an equipment control unit 406. The real-time signal acquisition unit 401 is configured to acquire real-time signals from a target industrial equipment in operation to obtain a real-time signal information sequence. The target industrial equipment is a gas turbine used to drive a generator to generate electricity. The target industrial equipment has M monitoring areas, where M ≥ 2. The real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area. The fusion and noise reduction unit 402 is configured to fuse and reduce the real-time acoustic signals and real-time vibration signals included in each real-time signal information in the real-time signal information sequence to generate a denoised real-time signal. The system obtains a denoised real-time signal information sequence, wherein the denoised real-time signal information consists of denoised acoustic signals and denoised vibration signals; a dynamic signal feature extraction unit 403 is configured to extract dynamic signal features from each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features, thereby obtaining a real-time signal feature sequence, wherein the real-time signal features consist of acoustic signal features and vibration signal features; a generation unit 404 is configured to generate a fused signal feature sequence based on the real-time signal feature sequence; a device status identification unit 405 is configured to perform device status identification based on the fused signal feature sequence to generate device status information, wherein the device status information includes: area identifier, status type, and status confidence; and a device control unit 406 is configured to perform device control on the target industrial equipment based on the device status information.

[0113] It is understandable that the units recorded in the industrial equipment condition monitoring device 400 based on multimodal data are related to the reference... Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the industrial equipment condition monitoring device 400 based on multimodal data and the units contained therein, and will not be repeated here.

[0114] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 500 suitable for implementing some embodiments of the present disclosure. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0115] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0116] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0117] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0118] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0119] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0120] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: perform real-time signal acquisition on a target industrial device in operation, obtaining a real-time signal information sequence, wherein the target industrial device is a gas turbine used to drive a generator to generate electricity, the target industrial device is provided with M monitoring areas, M≥2, and the real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area; and fuse and denoise the real-time acoustic signals and real-time vibration signals included in each real-time signal information in the aforementioned real-time signal information sequence to generate denoised real-time signal information, obtaining denoised real-time signal information. The system comprises a sequence of noise-reduced real-time signals, wherein the noise-reduced real-time signal information consists of noise-reduced acoustic signals and noise-reduced vibration signals. Dynamic signal features are extracted from each noise-reduced real-time signal in the sequence to generate real-time signal features, resulting in a real-time signal feature sequence, which consists of acoustic signal features and vibration signal features. A fused signal feature sequence is generated based on this sequence. Equipment status is identified based on this fused signal feature sequence to generate equipment status information, which includes: area identifier, status type, and status confidence level. Equipment control is then performed on the target industrial equipment based on this equipment status information.

[0121] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and Python, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0124] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for monitoring the condition of industrial equipment based on multimodal data, characterized in that, include: Real-time signal acquisition is performed on the target industrial equipment in the equipment operation state to obtain a real-time signal information sequence. The target industrial equipment is a gas turbine used to drive a generator to generate electricity. The target industrial equipment is equipped with M monitoring areas, M≥2. The real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area. The real-time acoustic signal and real-time vibration signal included in each real-time signal information in the real-time signal information sequence are fused and denoised to generate denoised real-time signal information, resulting in a denoised real-time signal information sequence, wherein the denoised real-time signal information is composed of denoised acoustic signal and denoised vibration signal. Dynamic signal features are extracted from each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features, resulting in a real-time signal feature sequence, wherein the real-time signal features consist of acoustic signal features and vibration signal features. Based on the real-time signal feature sequence, a fused signal feature sequence is generated; Device status is identified based on the fused signal feature sequence to generate device status information, wherein the device status information includes: region identifier, status type, and status confidence level; Based on the equipment status information, the target industrial equipment is controlled.

2. The industrial equipment condition monitoring method based on multimodal data according to claim 1, characterized in that, The step of fusing and denoising the real-time acoustic signal and real-time vibration signal included in each real-time signal information in the real-time signal information sequence to generate denoised real-time signal information includes: The real-time acoustic signal and real-time vibration signal included in the real-time signal information are respectively subjected to signal normalization processing to obtain normalized acoustic signal and normalized vibration signal; The initial acoustic signal features are obtained by extracting initial signal features from the normalized acoustic signal through the acoustic feature extraction branch. The vibration feature extraction branch extracts initial signal features from the normalized vibration signal to obtain initial vibration signal features. The acoustic feature extraction branch and the vibration feature extraction branch are set in parallel and have the same network structure. By using a feature cross-projection branch, the initial acoustic signal features and the initial vibration signal features are projected and filtered to obtain the projected acoustic signal features and the projected vibration signal features. Signal reconstruction is performed based on the initial acoustic signal characteristics, the initial vibration signal characteristics, the projected acoustic signal characteristics, and the projected vibration signal characteristics to obtain the denoised acoustic signal and the denoised vibration signal, which are included in the real-time signal information after noise reduction.

3. The industrial equipment condition monitoring method based on multimodal data according to claim 2, characterized in that, The step of reconstructing the signal based on the initial acoustic signal features, the initial vibration signal features, the projected acoustic signal features, and the projected vibration signal features to obtain the denoised real-time signal information includes the denoised acoustic signal and the denoised vibration signal, including: By using residual branching, the initial acoustic signal features and the projected acoustic signal features are residually fused to obtain fused acoustic signal features; By using residual branching, the initial vibration signal features and the projected vibration signal features are residually fused to obtain fused vibration signal features; By using the acoustic signal reconstruction branch, the fused acoustic signal features are reconstructed in the time domain to obtain the denoised acoustic signal included in the real-time signal information after denoising. The fused vibration signal features are reconstructed in the time domain through the vibration signal reconstruction branch to obtain the denoised vibration signal included in the real-time signal information after denoising. The acoustic signal reconstruction branch and the vibration signal reconstruction branch are set in parallel and have the same network structure.

4. The industrial equipment condition monitoring method based on multimodal data according to claim 3, characterized in that, The step of extracting dynamic signal features from each denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features includes: The noise-reduced acoustic signal and noise-reduced vibration signal, which are included in the real-time signal information after noise reduction, are decomposed into a first frequency band signal group and a second frequency band signal group. Frequency band features are extracted from each first frequency band signal in the first frequency band signal group to generate first frequency band signal features, thus obtaining a first frequency band signal feature group; Frequency band features are extracted from each second frequency band signal in the second frequency band signal group to generate second frequency band signal features, thus obtaining a second frequency band signal feature group; Attention-weighted fusion is performed on the first frequency band signal features in the first frequency band signal feature group to obtain the acoustic signal features included in the real-time signal features; Attention-weighted fusion is performed on the second frequency band signal features in the second frequency band signal feature group to obtain the vibration signal features included in the real-time signal features.

5. The industrial equipment condition monitoring method based on multimodal data according to claim 4, characterized in that, The step of identifying the device status based on the fused signal feature sequence to generate device status information includes: For each fused signal feature in the fused signal feature sequence, perform the following identification steps: The fused signal features are then decoded to obtain the decoded signal features; Based on the characteristics of the decoded signal, state classification is performed to obtain candidate device state information, wherein the candidate device state information includes: region identifier, state type, and state confidence level; The obtained candidate device status information sequence is filtered to obtain the device status information.

6. The industrial equipment condition monitoring method based on multimodal data according to claim 5, characterized in that, The step of controlling the target industrial equipment based on the equipment status information includes: The hazard level is determined based on the area identifier and status type included in the equipment status information; Determine the equipment dependency level of the current power generation task on the target industrial equipment; The control level is determined based on the hazard level and the equipment dependency level. In response to the control level being the first control level and the status confidence level included in the device status information being within the first confidence interval, an alarm message is pushed to the control terminal corresponding to the target industrial equipment; In response to the control level being the second control level and the state confidence level included in the device status information being within the second confidence interval, derating control is triggered for the target industrial equipment; In response to the control level being the third control level and the state confidence level included in the equipment status information being in the third confidence region, a shutdown control for the target industrial equipment is triggered.

7. The industrial equipment condition monitoring method based on multimodal data according to claim 6, characterized in that, The method further includes: The equipment status information is synchronized to the equipment twin corresponding to the target industrial equipment; The device twin synchronously displays device control operations and updated device status.

8. An industrial equipment condition monitoring device based on multimodal data, characterized in that, include: The real-time signal acquisition unit is configured to acquire real-time signals from a target industrial device in operation to obtain a real-time signal information sequence. The target industrial device is a gas turbine used to drive a generator to generate electricity. The target industrial device is provided with M monitoring areas, where M≥2. The real-time signal information includes real-time acoustic signals and real-time vibration signals corresponding to the same monitoring area. The fusion denoising unit is configured to fuse and denoise each real-time signal information in the real-time signal information sequence, including the real-time acoustic signal and the real-time vibration signal, to generate denoised real-time signal information and obtain a denoised real-time signal information sequence, wherein the denoised real-time signal information is composed of denoised acoustic signal and denoised vibration signal. The dynamic signal feature extraction unit is configured to extract dynamic signal features from each of the denoised real-time signal information in the denoised real-time signal information sequence to generate real-time signal features and obtain a real-time signal feature sequence, wherein the real-time signal features are composed of acoustic signal features and vibration signal features. The generation unit is configured to generate a fused signal feature sequence based on the real-time signal feature sequence; The device status identification unit is configured to identify the device status based on the fused signal feature sequence to generate device status information, wherein the device status information includes: area identifier, status type and status confidence level; The equipment control unit is configured to control the target industrial equipment based on the equipment status information.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images