Fault diagnosis model for fusing attention mechanism with chemical process bimodal information

Through the fusion and attention mechanism of MPR-CNN and Yolov7 models, a collaborative modal fusion unit is built, which solves the problem of ignoring visual data in the traditional chemical process, achieves more efficient fault diagnosis, and meets the needs of modern chemical production.

CN118981746APending Publication Date: 2024-11-19TIANJIN UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411119393.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Fault detection and diagnosis of traditional chemical processes mainly relies on text data analysis, ignoring the full extraction of visual data, resulting in poor diagnostic effects and difficult to meet the needs of modern chemical production.

Method used

The multi-mode representation-convolution neural network (MPR-CNN) model is fused with the Yolov7 model, and a collaborative modal fusion unit is constructed in combination with the attention mechanism, which extracts dual-modal information in the chemical process, and realizes the deep interaction between text and visual data. Protect the integrity of language information.

Benefits of technology

By making full use of dual-modal data, richer features are extracted, and the accuracy and efficiency of chemical process fault diagnosis is improved to meet the needs of industrial applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981746A_ABST
    Figure CN118981746A_ABST
Patent Text Reader

Abstract

The invention relates to a fault diagnosis model using an attention mechanism to fuse chemical process bimodal information. The system is composed of a text modal feature extraction unit, a visual modal extraction unit and a collaborative modal fusion unit. The text modal feature extraction uses an MPR-CNN model, and multiple different types of data after multiple pre-extraction form a three-dimensional data feature matrix; the visual modality is extracted by using a data feature matrix converted by a YoV7 model; the collaborative modal fusion unit is composed of a plurality of newly constructed multi-modal attention decoders and a multi-modal interaction unit, and a fault diagnosis model for fusing chemical process bimodal information by using an attention mechanism is formed. The invention aims to solve the problem that the traditional fault detection and diagnosis of the chemical process only analyzes the text data generated in the chemical process and ignores the full extraction of visual data. Potential richer features of data are extracted by fully utilizing bimodal data, so that fault diagnosis is more efficient.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a new fault diagnosis model that uses an attention mechanism to fuse bimodal information of a chemical process. This is to solve the problem that traditional fault detection and diagnosis of chemical processes only analyzes text data generated in the chemical process and ignores the full extraction of visual data. The MPR-CNN model is innovatively fused with the Yolov7 model to extract bimodal information, and a new collaborative modal fusion unit using an attention mechanism is constructed to protect the integrity of the language information in the bimodal information. By using this new collaborative fusion module, it is possible to fully utilize bimodal data extraction to obtain potentially richer features of the data, making fault diagnosis more efficient. Background Art

[0002] With the development of scale, complexity and intelligence in the chemical industry, once a chemical process fails, it may cause huge economic losses, environmental pollution, and even threaten the safety of people's lives. The demand for safety, reliability and product quality in chemical process systems is increasing, which makes fault detection and diagnosis an area of ​​great concern. In the past, fault detection and diagnosis of chemical processes mainly relied on the experience of experts. However, this method is inefficient and difficult to meet the needs of modern chemical production. Therefore, the research on fault diagnosis of complex chemical processes has important practical application value.

[0003] With the development of artificial intelligence technology, current research relies on various deep learning algorithms. From the initial artificial neural network to the deep learning network with complex structure coupling, the fault diagnosis performance is constantly improving.

[0004] However, the data generated by chemical processes are diverse. In the fault diagnosis research of typical chemical processes (such as TE processes), only text modality data is used, ignoring the full extraction of other modality data (such as visual data), resulting in poor diagnostic effect. Such performance is difficult to meet the needs of actual industrial processes. In order to make the data-driven fault diagnosis model have practical application value, it is of great significance to further improve the fault diagnosis rate and diagnosis timeliness.

[0005] The present invention utilizes the multi-modal representation-convolutional neural network model (MPR-CNN), and on its basis, intends to develop a fault diagnosis model that integrates the attention mechanism and bimodal information of the chemical process. By utilizing the classification and recognition advantages of artificial intelligence, the text modal information of variables such as temperature and pressure in the chemical process and the visual modal information captured in real time are fully utilized to achieve real-time analysis, thereby quickly and accurately determining the cause of the system disturbance and assisting the decision-making of the control system. Summary of the invention

[0006] In order to solve the problem that traditional process fault detection and diagnosis only analyzes the text data generated in the chemical process and ignores the full extraction of visual data, making it difficult to further improve the fault diagnosis performance, this paper proposes a fault diagnosis model that integrates the bimodal information of the chemical process with an attention mechanism. The bimodal data (text modal data and visual modal data) generated in the chemical process is fully utilized to achieve high-accuracy diagnosis of chemical process faults.

[0007] The technical solution of the present invention is as follows:

[0008] A fault diagnosis model that uses an attention mechanism to fuse bimodal information of a chemical process; it is composed of a text modal feature extraction unit, a visual modal extraction unit and a collaborative modal fusion unit; the text modal feature extraction uses an MPR-CNN model to combine multiple pre-extracted data of various different types into a three-dimensional data feature matrix; the visual modal extraction uses a data feature matrix converted from a YoloV7 model; the collaborative modal fusion unit uses multiple newly constructed multimodal attention decoders (T AttLayer) and multimodal interaction (MI) units to form a fault diagnosis model that uses an attention mechanism to fuse bimodal information of a chemical process.

[0009] The invention discloses a fault diagnosis model that integrates bimodal information of chemical process by integrating attention mechanism. Firstly, bimodal data of chemical process is collected to generate bimodal data of normal operation and abnormal bimodal data caused by process variable disturbance. The data are labeled according to the process state of the data source, and the obtained samples are divided into training set and test set in the ratio of 8:2. The MPR-CNN algorithm is used for text modal feature extraction. The YoloV7 model is used to extract features for visual modal extraction. Multiple multimodal attention decoders (T Att Layer) and multimodal interaction (MI) units are used in the collaborative modal fusion unit to fuse the features of bimodal information. The obtained training set is used for training and the test set is used to evaluate its performance. If the performance meets the actual industrial process diagnosis requirements, the model is applied as an online diagnosis model. In the online diagnosis process, the bimodal data is input into the model, and the fault diagnosis result is finally output to judge whether the whole chemical process is in a fault state. If so, the specific fault cause is given.

[0010] The attention mechanism integrates the fault diagnosis model of the chemical process bimodal information; the text modal information extraction process uses the multi-modal representation-convolutional neural network (MPR-CNN) method; the chemical process text modal data is sliced ​​X = (X1, X2, ..., X N ), discrete Fourier transform and aggregate approximation are performed on the sliced ​​data respectively; Fourier transform is performed to obtain in The aggregation approximation is in Y I With Y II After splicing, a convolutional neural network is performed to extract text modal features.

[0011] The attention mechanism is integrated into the fault diagnosis model of the chemical process bimodal information; the high-dimensional and multivariate feature information contained in the text modal information is fully utilized, and the multimodal representation (MPR-CNN) method is used in the extraction process; the chemical process text modal data X = (X1, X2, ..., X N ), respectively, perform discrete Fourier transform and aggregate approximation. Through the discrete Fourier transform formula After getting in Approximation formula by aggregation get in Finally, X and Y I and After merging, the data are input into the convolutional neural network to extract the text modality feature matrix.

[0012] The attention mechanism integrates the fault diagnosis model of the chemical process bimodal information; the visual modal data P corresponding to the text modal data is converted into (P1, P2, ..., P N ) is input into the YoloV7 model to extract the visual modality feature matrix.

[0013] The attention mechanism described herein fuses the fault diagnosis model of the bimodal information of the chemical process; the fusion module inputs the text modal features and the visual modal features into a collaborative modal fusion structure composed of a multimodal attention decoder (T AttLayer) and multimodal interaction (MI) to achieve deep interaction between the two modal features, and then inputs them into the decoder for decoding, and finally outputs the fused modal feature information.

[0014] The attention mechanism integrates the fault diagnosis model of the chemical process bimodal information; the bimodal attention decoder uses a linear layer to project the text modality feature information and the visual modality feature information into a key and a value, respectively, and then uses two keys from the two modalities to generate a mutual attention matrix; is the scaling factor, F is the matrix after the two modal data pass through their respective feature extraction networks and then through the linear layer; the mutual attention matrix describes the relationship strength between all elements of the text modality feature matrix and the visual modality feature matrix in the chemical process, and then the mutual attention matrix is ​​normalized on the visual axis, and applied to the features of the visual modality, the output is generated using the attentive visual and textual information; and the output of the T Att module is generated together with the text information initially passed through another branch.

[0015] The attention mechanism integrates the fault diagnosis model of the chemical process bimodal information; the bimodal interaction module is modified on the basis of GRU, and the input text modality feature is F n ; The input visual features are The text feature F n With visual features After matrix multiplication, after the linear layer, L n With the original text feature F n Concatenate to get the text feature F containing visual information n+1 ; Visual features With L n After concatenation, the nonlinear transformation is performed by ReLU activation function and then L n Perform matrix multiplication to obtain visual features containing text modality features The bimodal interaction module helps to protect the integrity of the language information in the bimodal information and prevent the information from being lost and distorted during the transmission of the network; The input to the fully connected layer decoder outputs the corresponding fault type.

[0016] The specific instructions are as follows:

[0017] A fault diagnosis model that fuses bimodal information of chemical process with attention mechanism; it is composed of text modal feature extraction unit, visual modal extraction unit and collaborative modal fusion unit; text modal feature extraction uses multi-mode representation-convolutional neural network model (MPR-CNN), and different types of data of chemical process text modal data after multiple pre-extraction are input into the convolutional neural network to extract text modal feature matrix; visual modal feature extraction uses YoloV7 model, and chemical process visual modal data is converted into data feature matrix through YoloV7 model; collaborative modal fusion unit is composed of newly constructed bimodal attention (T AttLayer) unit and bimodal interaction (MI) unit, which realizes deep interaction of two modal features, so as to produce an artificial intelligence fault diagnosis algorithm with high diagnostic accuracy, strong generalization ability and industrial application value.

[0018] The fault diagnosis model using the attention mechanism to fuse the bimodal information of the chemical process; firstly, the bimodal data of the chemical process is collected, and the bimodal data of normal operation is first generated, and then the process state in the chemical process is disturbed (such as the temperature increase of the crystallizer), and the abnormal operation bimodal data and the abnormal bimodal data generated by the extreme case (the feed pipe is blocked) are generated; the original text modal data is X = (X1, X2, ..., X N ), X i =(x i,1 ,x i,2 ,...,x i,M ) T ,i=1,2,...,N. The corresponding original visual modality is P=(P1,P2,...,P N ), N is the number of historical data collected by the process once, M is the number of process variables, X i is the process data corresponding to the i-th variable. Label according to the process status of the bimodal data source, such as normal operation if the data source is normal, temperature increase of the crystallizer if the data source is increased, and corresponding status for some extreme cases such as the feed pipe being blocked. Merge all labeled data to get samples. Divide these samples into training set and test set in a ratio of 8:2;

[0019] The chemical process bimodal fault diagnosis model fused with the attention mechanism; in order to make full use of the high-dimensional and multivariate feature information contained in the text modality information, the extraction process uses the multimodal representation (MPR-CNN) method. The chemical process text modality data X = (X1, X2, ..., X N ) respectively perform discrete Fourier transform and aggregate approximation. Through the discrete Fourier transform formula After getting in Approximation formula by aggregation get in Finally, X and Y Ⅰ and Y Ⅱ After merging, input into the convolutional neural network to extract the text modality feature matrix F n .

[0020] The fault diagnosis model of chemical process bimodal information fused with attention mechanism; visual modal data P = (P1, P2, ..., P N ) is input into the YoloV7 model to extract the visual modality feature matrix.

[0021] The fault diagnosis model that uses the attention mechanism to fuse the bimodal information of the chemical process; inputs the text modal features and the visual modal features into a collaborative modal fusion structure composed of a bimodal attention decoder (T AttLayer) and bimodal interaction (MI) to achieve deep interaction between the two modal features, and then inputs them into a decoder (a fully connected layer for multi-classification) for decoding, and finally outputs the fused modal feature information.

[0022] The fault diagnosis model that uses the attention mechanism to fuse the bimodal information of the chemical process; the bimodal attention decoder uses a linear layer to project the textual modal feature information and the visual modal feature information into a key and a value, respectively, and then uses two keys from the two modalities to generate a mutual attention matrix. is the scaling factor, and F is the matrix after the two modal data pass through their respective feature extraction networks and then through the linear layer. The mutual attention matrix describes the relationship strength between all elements of the text modality feature matrix and the visual modality feature matrix in the chemical process. The mutual attention matrix is ​​then normalized on the visual axis and applied to the features of the visual modality to generate the output using the attention visual and text information. The output of the T Att module is generated together with the text information that initially passes through another branch.

[0023] The fault diagnosis model of chemical process bimodal information fused with attention mechanism; the bimodal interaction module is modified on the basis of GRU, and the input text modality feature is F n The input visual features are The text feature F n With visual features After matrix multiplication, after the linear layer, L n With the original text feature F n Concatenate to get the text feature F containing visual information n+1 . Visual Features With L n After concatenation, the nonlinear transformation is performed by ReLU activation function and then L n Perform matrix multiplication to obtain visual features containing text modality features The bimodal interaction module helps to protect the integrity of the language information in the bimodal information and prevent the information from being lost and distorted during the transmission of the network. The input to the fully connected layer decoder outputs the corresponding fault type.

[0024] A dual-modal fault diagnosis model with a structure of MPR-CNN algorithm to extract text modal features, YoloV7 model to extract visual modal features and collaborative modal fusion unit is used for fault diagnosis of chemical processes, which has the characteristics of high accuracy and strong generalization ability.

[0025] The present invention aims to solve the problem that traditional chemical process fault detection and diagnosis only analyzes the text data generated in the chemical process while ignoring the full extraction of visual data. The MPR-CNN model is innovatively fused with the Yolov7 model to extract bimodal information, and a new collaborative modal fusion unit using an attention mechanism is constructed to protect the integrity of the language information in the bimodal information. By using this new collaborative fusion module, it is possible to fully utilize the bimodal data extraction to obtain potentially richer features of the data, making fault diagnosis more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the fault diagnosis model of the present invention that integrates the attention mechanism with bimodal information of chemical process.

[0027] Figure 2 This is a structural diagram of the fault diagnosis model that integrates the attention mechanism of the present invention with bimodal information of chemical processes.

[0028] Figure 3 This is the processing flow chart of the text modal information extraction module.

[0029] Figure 4 This is a structural diagram of the bimodal attention unit of the present invention.

[0030] Figure 5 This is a structural diagram of the dual-modal interaction unit of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] The present invention connects the text modality information extraction process with the visual modality information extraction process in parallel. In the process of fusion of the two modalities, a new dual-modality attention decoder and dual-modality interaction unit structure are proposed, which can further realize the full mining of features and improve the fault diagnosis performance. Figure 1As shown in the figure, firstly, data of textual modality and visual modality are collected from the chemical process, and a bimodal dataset is generated after data preprocessing and annotation. The fault diagnosis model that integrates the constructed attention mechanism with the bimodal information of the chemical process is trained on the dataset. The trained model is evaluated, and if the effect is good, it will be put into the online chemical process for fault identification, otherwise the parameters will continue to be fine-tuned. A semi-batch crystallization process is described as an example of fault diagnosis. The semi-batch crystallization process system is acetylsalicylic acid (ASA), which is realized by the flow control strategy. ASA crystal slurry in low-temperature ethanol and high-temperature pure water are mixed in a mixing container to generate a high-temperature saturated ASA solution, and water-ethanol is used as the feed of the crystallizer. The crystallizer is pre-loaded with a certain volume of ASA solution saturated at the crystallization temperature (low temperature), and the flow rate of the feed flow is controlled by the flow-supersaturation control (F-SSC) strategy to keep the supersaturation constant throughout the crystallization process. The crystallization process is stopped when the supersaturation naturally drops to almost 0. Under normal conditions, the running time of each batch of crystallization is set to 30 minutes. Six types of faults and eight process variables are proposed.

[0035]

[0036] The first step is to collect six types of fault data and normal state data, including text modal data and visual modal data in each second. Each fault type runs for 10 minutes (faults are added after 20 minutes of normal operation), the sampling time is set to 0.5 seconds, and a total of 8400 sets of text modal data are included in the normal state. Acetylsalicylic acid morphology pictures are taken every 4 seconds. The total text modal data at intervals of 4 seconds are divided into a group. A total of 1050 acetylsalicylic acid morphology pictures and 1050 sets of text modal data of the acetylsalicylic acid generation process. The text modal data is X = (X1, X2, ..., X N ), X i =(x i,1 ,x i,2 ,...,x i,M ) T ,i=1,2,...,N. The corresponding original visual modality is P=(P1,P2,...,P N ), N is the number of historical data collected by the process once, M is the number of process variables, X iis the process data corresponding to the i-th variable. Label according to the process state of the data source. The visual modal information and text modal information of acetylsalicylic acid are used as the data set for training and evaluating the fault diagnosis model that integrates the attention mechanism and the bimodal information of the chemical process. The data set is divided into a training set and a test set at a ratio of 8:2. The information containing only the text modality is used as the data set for training and evaluating other chemical process fault diagnosis models that only use text information. The data set is also divided into a training set and a test set at a ratio of 8:2. The second step is to construct the branch for text modal information extraction, the branch for visual modal information extraction and the information fusion module. The third step is to use the training set to train the constructed model and train the control model at the same time. The fourth step is to evaluate the performance of the trained model with the test set and compare it with the control model. If the performance meets the actual industrial process diagnosis needs, the model is applied as an online diagnosis model.

[0037] Figure 2 The structure of the fault diagnosis model that integrates the attention mechanism with the bimodal information of the chemical process is demonstrated: the network input includes the process variable text data and image visual data generated by the semi-batch crystallization process, and the text modal features and visual modal features are obtained after passing through the text and visual modal representation networks respectively. Then, the text modal features and visual modal features are input into the collaborative modal fusion structure composed of the bimodal attention decoder (T AttLayer) and bimodal interaction (MI) to achieve deep interaction of the two modal features. Finally, the features of the two modal interactions are output to the decoder, and the type of fault is output.

[0038]

[0039] Figure 3 The processing flow of the text modality information extraction module is shown: multi-modal representation (MPR) and convolutional neural network (CNN). The text modality data of acetylsalicylic acid is X = (X1, X2, ..., X N ), N is 1050. For X=(X1,X2,...,X N ) respectively perform discrete Fourier transform and aggregate approximation. Through the discrete Fourier transform formula After getting in Approximation formula by aggregation get in Finally, X and Y Ⅰ and Y Ⅱ After merging, input into the convolutional neural network to extract the text modality feature matrix F nThe convolutional neural network is set as Conv(32)-Pool(2×2)-Conv(64)-Pool(1×2)-Conv(32)-FC(32)-FC(16). Conv represents the convolution layer, Pool represents the pooling layer, and FC represents the fully connected layer.

[0040]

[0041] Figure 4 The bimodal attention unit structure is shown: The bimodal attention decoder takes inputs from two modalities and converts them into two roles: index and value. A linear layer is used to project text features and visual features into keys and values ​​respectively, and then the two keys from the two modalities are used to generate the mutual attention matrix. The mutual attention matrix is ​​then normalized on the visual axis and applied to the features of the visual modality. Finally, the output is generated using the attention visual and text information. The output of the TAtt module is generated together with the acetylsalicylic acid text modality features that initially pass through another branch.

[0042] Figure 5 The bimodal interaction unit structure is shown: the bimodal interaction module is modified based on GRU, and the input text feature is F n The input visual features come from the bimodal attention unit The text feature F n With visual features After matrix multiplication, it passes through the linear layer, that is, L n With the original text feature F n Concatenate to get the text feature F containing visual information n+1 . Visual Features With L n After concatenation, the nonlinear transformation is performed by ReLU activation function and then L n Perform matrix multiplication to obtain visual features containing text information The bimodal interaction module helps to protect the integrity of the language information in the bimodal information and prevent the information from being lost and distorted during the transmission of the network. The fault type to be predicted is output through a fully connected layer decoder. The fully connected layer decoder is FC(64)-FC(32)-FC(6).

[0043] The control CNN model that only inputs the text modal information of acetylsalicylic acid is set to Conv(32)-Pool(2×2)-Conv(64)-Pool(1×2)-Conv(32)-FC(64)-FC(32)-FC(6). At the same time, the loss function of the two neural networks is set to MSE. The evaluation indicators such as R^2 of the two models are compared. If the fault diagnosis model that integrates the attention mechanism and the bimodal information of the chemical process is 3% more accurate than the model that only uses the text modal information, its fault diagnosis performance is considered satisfactory and it will be used for online fault diagnosis. Otherwise, its internal structure is fine-tuned and retrained until the requirements are met (modify the structure of the convolution part, other hyperparameters, etc.).

[0044] Online stage:

[0045] (1) Collect data from chemical processes online and perform data preprocessing.

[0046] (2) Input the obtained sample matrix into the designed CNN model and give the predicted diagnosis result.

[0047] If abnormal, provide specific information on the cause of the abnormality.

[0048] The present invention proposes a new fault diagnosis model that uses an attention mechanism to fuse bimodal information of a chemical process. This is to solve the problem that traditional fault detection and diagnosis of chemical processes only analyzes text data generated in the chemical process while ignoring the full extraction of visual data. The multi-modal representation-convolutional neural network model (MPR-CNN) is innovatively fused with the Yolov7 model to extract bimodal information, and a new collaborative modal fusion unit using an attention mechanism is constructed to protect the integrity of the language information in the bimodal information. By using this new collaborative fusion module, it is possible to fully utilize bimodal data extraction to obtain potentially richer features of the data, making fault diagnosis more efficient. This allows for quick and accurate judgment of the cause of system disturbances, assisting in the decision-making of the control system.

[0049] The technical solutions disclosed and proposed by the present invention can be realized by those skilled in the art by referring to the contents of this article and appropriately changing the conditions, routes and other links. Although the methods and preparation techniques of the present invention have been described through preferred embodiments, relevant technicians can obviously modify or re-combine the methods and technical routes described herein without departing from the content, spirit and scope of the present invention to realize the final preparation technology. It should be particularly noted that all similar substitutions and modifications are obvious to those skilled in the art, and they are all considered to be included in the spirit, scope and content of the present invention.

Claims

1. A fault diagnosis model that uses an attention mechanism to fuse bimodal information of a chemical process; characterized in that: It is composed of a text modal feature extraction unit, a visual modal extraction unit and a collaborative modal fusion unit; the text modal feature extraction uses the MPR-CNN model to combine multiple pre-extracted data of various types into a three-dimensional data feature matrix; the visual modal extraction uses the data feature matrix converted by the YoloV7 model; the collaborative modal fusion unit is composed of multiple newly constructed multimodal attention decoders (T AttLayer) and multimodal interaction (MI) units, forming a fault diagnosis model that uses the attention mechanism to fuse the bimodal information of the chemical process.

2. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 1; characterized in that: First, the bimodal data of the chemical process is collected to generate bimodal data of normal operation and abnormal bimodal data caused by process variable disturbances; the data is labeled according to the process state of the data source, and the obtained samples are divided into training set and test set in a ratio of 8:2; the MPR-CNN algorithm is used for text modal feature extraction; the YoloV7 model is used to extract features for visual modal extraction; In the collaborative modality fusion unit, multiple multimodal attention decoders (T Att Layer) and multimodal interaction (MI) units are used to perform feature fusion of bimodal information; the obtained training set is used for training and the performance is evaluated with the test set; if the performance meets the actual industrial process diagnosis requirements, the model is applied as an online diagnosis model; During the online diagnosis process, the bimodal data is input into the model, and finally the fault diagnosis result is output to determine whether the entire chemical process is in a fault state. If so, the specific cause of the fault is given.

3. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 1; characterized in that: The text modal information extraction process uses the multi-modal representation-convolutional neural network (MPR-CNN) method; the chemical process text modal data is sliced ​​X = (X1, X2, ..., X N ), discrete Fourier transform and aggregate approximation are performed on the sliced ​​data respectively; Fourier transform is performed to obtain in The aggregation approximation is in Y I With Y II After splicing, a convolutional neural network is performed to extract text modal features.

4. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 3; characterized in that: To make full use of the high-dimensional and multivariate feature information contained in the text modal information, the multi-modal representation (MPR-CNN) method is used in the extraction process; the chemical process text modal data X = (X1, X2, ..., X N ), respectively, perform discrete Fourier transform and aggregate approximation. Through the discrete Fourier transform formula After getting in Approximate formula by aggregation get in Finally, X and Y I and After merging, the data are input into the convolutional neural network to extract the text modality feature matrix.

5. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 1; characterized in that: The visual modality data P corresponding to the text modality data is P=(P1, P2, ..., P N ) is input into the YoloV7 model to extract the visual modality feature matrix.

6. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 1; characterized in that: The fusion module inputs the text modal features and visual modal features into the collaborative modal fusion structure composed of the multimodal attention decoder (TAttLayer) and multimodal interaction (MI) to achieve deep interaction between the two modal features, and then inputs them into the decoder for decoding, and finally outputs the fused modal feature information.

7. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 6; characterized in that: The bimodal attention decoder uses a linear layer to project the textual modality feature information and the visual modality feature information into keys and values ​​respectively, and then uses the two keys from the two modalities to generate the mutual attention matrix; is the scaling factor, F is the matrix after the two modal data pass through their respective feature extraction networks and then through the linear layer; the mutual attention matrix describes the relationship strength between all elements of the text modality feature matrix and the visual modality feature matrix in the chemical process, and then the mutual attention matrix is ​​normalized on the visual axis, and applied to the features of the visual modality, the output is generated using the attentive visual and textual information; and the output of the T Att module is generated together with the text information initially passed through another branch.

8. The fault diagnosis model of chemical process bimodal information fused with attention mechanism according to claim 6; characterized in that: The bimodal interaction module is modified based on GRU, and the input text modality feature is F n ; The input visual features are The text feature F n With visual features After matrix multiplication, after the linear layer, L n With the original text feature F n Concatenate to get the text feature F containing visual information n+1 ; Visual features With L n After concatenation, the nonlinear transformation is performed by ReLU activation function and then L n Perform matrix multiplication to obtain visual features containing text modality features The bimodal interaction module helps to protect the integrity of the language information in the bimodal information and prevent the information from being lost and distorted during the transmission of the network; The input to the fully connected layer decoder outputs the corresponding fault type.

Citation Information

Cited By

  • Wind power blade intelligent diagnosis method and system based on large model visual semantic fusion

    CN120339671A

  • Motor fault diagnosis method based on knowledge embedding and time-frequency feature extraction

    CN121327709A

  • Phosphorus chemical industry production equipment online state monitoring and intelligent diagnosis system based on multi-parameter fusion

    CN121561339A

  • Intelligent detection method for chemical device based on multi-source data fusion

    CN122490387A