Petrochemical engineering anomaly detection method and system based on multi-modal attention fusion

Abstract

The invention relates to a petrochemical engineering anomaly detection method and system based on multi-modal attention fusion, and the method comprises the steps: obtaining a real-time video frame of a petrochemical plant, and carrying out the preprocessing of the real-time video frame, and obtaining a binary image; and calling a convolutional neural network to extract original features from the binary image, and performing mapping and weighting processing on the original features in combination with the attention weight to obtain fusion features. And performing linear projection on the fusion features to calculate attention output based on the projection matrix of each head, and performing splicing and linear mapping on the attention output of all the heads. Carrying out the splicing and linear mapping of the attention output of all heads, and carrying out the residual connection and normalization of the fusion features, and obtaining a normalization result. And inputting the normalization result into a classification head to output an abnormal alarm of a corresponding class. The multi-head self-attention classification network is combined, the anomaly detection precision is improved, the false alarm rate is reduced, and the alarm real-time performance is high.

Description

Technical Field

[0001] This invention relates to the field of petrochemical production and anomaly detection technology, and in particular to a petrochemical anomaly detection method and system based on multimodal attention fusion. Background Technology

[0002] Currently, petrochemical production sites typically employ either single-video surveillance or UWB-based positioning monitoring. Existing video surveillance systems largely rely on traditional image processing algorithms, which have limitations in recognizing abnormal behaviors in complex backgrounds (such as low-hanging seatbelts or abnormal operator movements). UWB-based positioning systems are primarily used for location and tracking, but struggle to identify subtle behavioral features. Applying these technologies alone makes it difficult to achieve efficient fusion of multimodal data and real-time, accurate anomaly detection. Directly fusing video and UWB features through weighted summation or feature stitching, such as linear fusion (e.g., weighted summation), fails to capture the non-linear interaction between action and position and lacks multi-scale information retention (ignoring local details of action or global trends in position).

[0003] Therefore, traditional safety monitoring systems for petrochemical plants cannot fully leverage the complementary features of vision and UWB under complex operating conditions, resulting in slow system response, low accuracy of anomaly analysis, delayed alarm information, and a higher likelihood of false alarms. Summary of the Invention

[0004] Therefore, it is necessary to address the aforementioned technical problems by providing a petrochemical anomaly detection method and system based on multimodal attention fusion that can fully exploit the complementary features of vision and UWB under complex working conditions, and combine them with a multi-head self-attention classification network to achieve high accuracy, low false alarm rate, and high real-time performance.

[0005] This invention provides a method for detecting anomalies in petrochemical industries based on multimodal attention fusion, the method comprising:

[0006] Real-time video frames of a petrochemical plant are acquired, and the real-time video frames are preprocessed to obtain a binary image.

[0007] A convolutional neural network is invoked to extract the original features from the binary image, and attention weights are used to map and weight the original features to obtain fused features.

[0008] The fused features are linearly projected to calculate the attention output based on the projection matrix of each head, and the attention outputs of all heads are concatenated and linearly mapped.

[0009] Based on the concatenation and linear mapping of the attention outputs of all heads, the fused features are residually connected and normalized to obtain the normalized result;

[0010] The normalization result is input into the classification head to output an anomaly alert for the corresponding class.

[0011] In one embodiment, acquiring real-time video frames of a petrochemical plant and preprocessing the real-time video frames to obtain a binary image includes:

[0012] The real-time video frames are acquired by the surveillance camera and converted into grayscale images. The grayscale images are then smoothed using a Gaussian kernel to obtain filtered image data.

[0013] Background information in the filtered image data is separated by background subtraction to determine the foreground target in the filtered image data. The foreground target is then binarized to obtain the binarized foreground image and background model corresponding to the real-time video frame.

[0014] In one embodiment, the invocation of the convolutional neural network extracts original features from the binary image, and maps and weights the original features using attention weights to obtain fused features, including:

[0015] The convolutional neural network is invoked to extract multi-scale features from the binarized foreground image and determine the pooled feature map to construct a multi-scale feature vector.

[0016] The target object in the binarized foreground image is located by sending a pulse signal based on the multi-scale feature vector using ultra-wideband technology, and the spatial position of the target object is calculated by using a multi-base station triangulation method.

[0017] In one embodiment, the step of invoking a convolutional neural network to extract original features from the binary image, and mapping and weighting the original features with attention weights to obtain fused features, further includes:

[0018] The instantaneous error of the ultra-wideband technology in locating the target object by sending pulse signals is obtained, and the Kalman filter is used to optimize the instantaneous error to obtain the UWB positioning data of the target object.

[0019] Based on the UWB positioning data and spatial location, the multi-scale features are mapped to the query subspace, key subspace, and value subspace corresponding to multi-head attention, and the attention weight of each subspace is calculated.

[0020] In one embodiment, the linear projection of the fused features to calculate the attention output based on the projection matrix of each head, and the concatenation and linear mapping of the attention outputs of all heads, includes:

[0021] The UWB positioning data at all visual locations are weighted and aggregated to obtain UWB feature vectors;

[0022] A multi-head parallel approach is used to concatenate the UWB feature vectors along the channels to perform a linear mapping on the concatenated UWB feature vectors, thereby mapping the concatenated UWB feature vectors to the original dimension.

[0023] In one embodiment, the step of linearly projecting the fused features to calculate the attention output based on the projection matrix of each head, and concatenating and linearly mapping the attention outputs of all heads, further includes:

[0024] Using the UWB feature vector as the query subspace and the multi-scale feature vector as the key space or value subspace, the real-time video frames are supplemented into the UWB positioning data through the attention weight matrix to obtain the first fusion sub-feature;

[0025] Using the multi-scale feature vector as the query subspace and the UWB feature vector as the key subspace or value subspace, the UWB positioning data is supplemented into the real-time video frame through attention weights to obtain the second fusion sub-feature;

[0026] The fusion ratio of the first fusion sub-feature and the second fusion sub-feature is calculated by a gating network, and the first fusion sub-feature and the second fusion sub-feature are weighted and fused based on the fusion ratio to obtain the normalized fusion feature.

[0027] In one embodiment, inputting the normalization result into the classification head to output an anomaly alert for the corresponding class includes:

[0028] The normalized fusion features are input into the classification head, and global average pooling is performed on the fusion features. The pooling results are then input into multiple fully connected layers to obtain the classification results of each layer.

[0029] Based on the classification results of each layer and the preset decision threshold, the alarm probability of each type of anomaly is output so that when the alarm probability exceeds the safety threshold, an alarm for the corresponding type of anomaly is sent.

[0030] This invention also provides a petrochemical anomaly detection system based on multimodal attention fusion, used to implement the petrochemical anomaly detection method based on multimodal attention fusion described in any of the above claims, the system comprising:

[0031] An image preprocessing module is used to acquire real-time video frames from a petrochemical plant and preprocess the real-time video frames to obtain a binary image.

[0032] The feature extraction and fusion module is used to call a convolutional neural network to extract original features from the binary image, and combine attention weights to map and weight the original features to obtain fused features;

[0033] A multi-head attention output module is used to linearly project the fused features to calculate the attention output based on the projection matrix of each head, and to concatenate and linearly map the attention outputs of all heads.

[0034] The feature normalization module is used to concatenate and linearly map the attention outputs of all heads, perform residual connection and normalization on the fused features, and obtain the normalization result.

[0035] The anomaly alarm module is used to input the normalization result into the classification head to output anomaly alarms for the corresponding class.

[0036] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the petrochemical anomaly detection method based on multimodal attention fusion as described above.

[0037] The present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the petrochemical anomaly detection method based on multimodal attention fusion as described above.

[0038] The aforementioned petrochemical anomaly detection method and system based on multimodal attention fusion extracts original features from binary images using real-time video frames combined with a convolutional neural network. These features are then mapped and weighted using attention weights to obtain fused features. Next, the fused features are linearly projected, and attention outputs are calculated based on the projection matrix of each head. The attention outputs of all heads are then concatenated and linearly mapped. Finally, based on the concatenation and linear mapping of the attention outputs from all heads, the fused features are residually connected and normalized to obtain a normalized result. This normalized result is then input into a classification head to output an anomaly alarm for the corresponding class. This method not only achieves bidirectional information compensation and deep coupling, overcoming the shortcomings of existing methods relying solely on a single camera or UWB which are susceptible to occlusion and multipath interference, but also generates fusion weights in real-time based on on-site conditions and historical alarm feedback, automatically amplifying the data contribution of key risk areas. This solves the problem that traditional fixed weights or manually set weights cannot cope with complex scene changes. In addition, multi-head self-attention can simultaneously capture the spatiotemporal correlation between multiple cameras and multiple UWB nodes, and output multi-level alarms ranging from early warning to emergency, making up for the shortcomings of existing models that can only give simple "present / absent" alarms and lack hierarchical guidance. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating the petrochemical anomaly detection method based on multimodal attention fusion provided by this invention;

[0041] Figure 2 This is a schematic diagram of the data preprocessing process for a petrochemical anomaly detection method based on multimodal attention fusion in a specific embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the adaptive feature fusion process of the petrochemical anomaly detection method based on multimodal attention fusion in a specific embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the multi-head self-attention discrimination process of the petrochemical anomaly detection method based on multimodal attention fusion in a specific embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention;

[0045] Figure 6 An internal structural diagram of the computer device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The following is combined with Figures 1-6 This invention describes a method and system for detecting anomalies in the petrochemical industry based on multimodal attention fusion.

[0048] like Figure 1 As shown, in one embodiment, a petrochemical anomaly detection method based on multimodal attention fusion includes the following steps:

[0049] Step S110: Acquire real-time video frames of the petrochemical plant and preprocess the real-time video frames to obtain a binary image.

[0050] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes the following steps in step S110:

[0051] Step S111: Acquire real-time video frames through a surveillance camera and convert the real-time video frames into grayscale images. Perform Gaussian kernel smoothing on the grayscale images to obtain filtered image data.

[0052] Step S112: Background information in the filtered image data is separated by background subtraction to determine the foreground target in the filtered image data. The foreground target is then binarized to obtain the binarized foreground image and background model corresponding to the real-time video frame.

[0053] Combination Figures 2 to 4 As shown in the specific embodiment, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes steps 1 to 4:

[0054] Step 1: Video image preprocessing and feature extraction.

[0055] Specifically, in practical applications, raw video images often contain noise and background interference, making preprocessing crucial for improving the accuracy of subsequent feature extraction. The raw video images are real-time video frames captured by surveillance cameras, often depicting on-site construction operations and equipment operation in industrial settings. After preprocessing and feature extraction, these images are used to identify various dynamic behaviors of personnel or equipment.

[0056] In the image preprocessing process, the first step involves grayscale conversion and noise filtering. The color image is converted to a grayscale image (RGB image to grayscale) to reduce computational complexity while preserving brightness information. Grayscale images effectively reduce data dimensionality, decrease computational load, and facilitate subsequent edge and texture feature extraction. Next, the image is smoothed using a Gaussian kernel to suppress noise and retain edge information, resulting in filtered image data. This step reduces data dimensionality and noise, adapting to the complex lighting and environmental interference of petrochemical environments.

[0057] Secondly, background modeling and binarization are performed on the image to separate the foreground from the background, highlight the target area, and facilitate abnormal behavior detection. For dynamic videos, background subtraction algorithms are commonly used to extract foreground targets. Binarization simplifies image analysis, reduces computational complexity, and adaptive thresholding improves adaptability to dynamic scenes. The expression is:

[0058]

[0059]

[0060] In the formula, Binarize the foreground image; The filtered image data; Optimize the foreground difference coefficient to enhance the distinction between the foreground and background; The threshold is adaptive and dynamically adjusted to suit different scenarios; Background model; For the binarized foreground target, Indicates other situations.

[0061] Subsequent thresholding can yield a binary image for further target detection, removing background information and retaining only key motion or abnormal behavior areas for easier subsequent processing. Examples include: key motion and abnormal behavior detection such as people not following designated routes, seatbelts being worn low or high, vehicles traveling at high speeds or in the wrong direction, unauthorized entry into danger zones, and fall detection.

[0062] It should be noted that the criteria for judging abnormal behavior are as follows: based on the behavior classification results after feature fusion, a threshold or model output probability is set to determine whether the behavior is normal or abnormal. Specific judgment indicators include: speed of position change, movement trajectory, behavior pattern (such as climbing, prolonged stagnation, and out-of-bounds operation), etc.

[0063] Step S120: Invoke the convolutional neural network to extract the original features from the binary image, and combine the attention weights to map and weight the original features to obtain fused features.

[0064] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes the following steps in step S120:

[0065] Step S121: Call the convolutional neural network to extract multi-scale features from the binarized foreground image and determine the pooled feature map to construct a multi-scale feature vector.

[0066] Step S122: Using ultra-wideband technology, pulse signals are sent to locate the target object in the binarized foreground image based on multi-scale feature vectors. At the same time, the spatial position of the target object is calculated using multi-base station triangulation.

[0067] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention further includes the following steps in step S120:

[0068] Step S123: Obtain the instantaneous error of the ultra-wideband technology pulse signal for positioning the target object, and call the Kalman filter to optimize the instantaneous error to obtain the UWB positioning data of the target object.

[0069] Step S124: Based on UWB positioning data and spatial location, map multi-scale features to the query subspace, key subspace and value subspace corresponding to multi-head attention, and calculate the attention weight of each subspace.

[0070] Combination Figures 2 to 4 As shown in the specific embodiment, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes step 2, feature extraction. Multi-level, multi-scale features of the background-subtracted binary image are extracted using a convolutional neural network (CNN). The main steps include convolution, activation, and pooling. That is, in a multi-layer CNN, after multiple convolutions, activations, and pooling, a multi-scale feature map from low-level edges to high-level semantics can be obtained. That is, multi-scale feature vectors After global average pooling, the output is a 2048-dimensional vector: .

[0071] For details, see Figure 2 As shown, firstly, UWB positioning data processing is performed. UWB (Ultra-Wideband) technology performs high-precision positioning by sending pulse signals. It mainly uses the multi-base station triangulation method to calculate the target position and locate fixed and moving objects, such as high-hanging and low-use seat belts, and distance detection scenarios such as people falling.

[0072] Secondly, data filtering is performed on the UWB positioning data to address the instantaneous errors in the UWB data. These instantaneous errors (UWB ranging noise: multipath propagation, electromagnetic interference; non-true linear factors: measurement equipment calibration errors; environmental influences: reflection, occlusion; dynamic target changes: target attitude, velocity changes) reflect the impact of environmental interference on the equipment. The goal is to reduce these instantaneous errors to improve detection reliability. A Kalman filter can be used for state estimation. The optimized state estimate after Kalman filtering, i.e., the optimized target coordinates, is a smoothed, high-precision position.

[0073] The expression for the UWB positioning data feature mapping is as follows:

[0074]

[0075] In the formula, It is a 2048-dimensional vector, a non-linear high-dimensional representation containing location information; As an activation function, it enhances the nonlinear expression; For bias vectors, ; This is the weight matrix. ; The original UWB features are 3D coordinates (x, y, z). ).

[0076] After the above processing, high-precision UWB positioning data is output. For example, video frames capture the action of a worker not wearing a safety belt, generating 2048-dimensional visual features. UWB coordinates (x, y, z) represent the location of the high-altitude work area, mapped to 2048 dimensions. This provides a consistent semantic space for integration.

[0077] Step 3: Nonlinear multi-scale adaptive feature fusion.

[0078] For details, see Figure 3 As shown, firstly, the original features are linearly mapped to Query / Key / Value, that is, the original features are mapped to the query (Q, Query), key (K, Key), and value (V, Value) space required for attention, and representations of different subspaces are learned for each:

[0079]

[0080]

[0081] In the formula, It is a visual query; It is the key to vision; It is the visual value; It's a UWB query; It is a UWB key; It is the UWB value.

[0082] Next, visual "query" of UWB: This involves having each image spatial location "question" all UWB measurement points to capture its correlation with different temporal / distance information. This process requires first calculating attention weights. :

[0083]

[0084] In the formula, Used to characterize the numerical stability of the inner product. This is used to normalize on each row so that the sum of the "attention" of each spatial location to all UWB points is 1.

[0085] Subsequently, the weighted aggregated UWB values :

[0086]

[0087] In the formula, each visual position obtains a weighted feature vector from UWB.

[0088] Then perform multi-head splicing and output mapping, if using Head parallel, the above After splicing along the channel, perform another linear mapping:

[0089]

[0090] In the formula, Restore to the original dimension.

[0091] Residual connectivity and normalization are used to preserve the original visual information and stabilize the gradient.

[0092]

[0093] In this embodiment, visual features are used. As a query, based on video frame features As a key / value pair, video frame information is selectively added to the UWB data using an attention weight matrix.

[0094] UWB "Query" Visual: Symmetrical, allowing UWB features to "read" visual semantics:

[0095]

[0096]

[0097] Multi-head splicing and output mapping, if using Head parallel, the above After splicing along the channel, perform another linear mapping: Residual + Normalization: Based on visual features As a query, with UWB features As a key / value pair, UWB information is selectively supplemented into the visual field through an attention weight matrix.

[0098] Finally, feature fusion is performed through weighted summation:

[0099]

[0100] In the formula, It is learnable. Weights are calculated on the features before fusion using a "gated network". In this way, each forward process adaptively provides a fusion ratio based on the current visual + UWB features. For example, when the steam concentration in the pipe gallery is high, the UWB weight is automatically increased. Compared with fixed ratio fusion, this example can be adjusted "at will" according to real-time operating conditions, avoiding the awkward situation of "high acquisition, low utilization" of data from high-mounted equipment.

[0101] Step S130: Perform linear projection on the fused features to calculate the attention output based on the projection matrix of each head, and concatenate and linearly map the attention outputs of all heads.

[0102] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes the following steps in step S130:

[0103] Step S131: Weighted aggregation of UWB positioning data at all visual locations is performed to obtain UWB feature vectors.

[0104] Step S132: Multi-head parallel concatenation of UWB feature vectors along the channel is used to perform linear mapping on the concatenated UWB feature vectors, so as to map the concatenated UWB feature vectors to the original dimension.

[0105] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention further includes the following steps in step S130:

[0106] Step S133: Using UWB feature vectors as the query subspace and multi-scale feature vectors as the key subspace or value subspace, real-time video frames are supplemented into the UWB positioning data through the attention weight matrix to obtain the first fusion sub-feature.

[0107] Step S134: Using multi-scale feature vectors as the query subspace and UWB feature vectors as the key subspace or value subspace, UWB positioning data is supplemented into the real-time video frame through attention weights to obtain the second fusion sub-feature.

[0108] Step S135: Calculate the fusion ratio of the first fusion sub-feature and the second fusion sub-feature using a gating network, and perform weighted fusion of the first fusion sub-feature and the second fusion sub-feature based on the fusion ratio to obtain the normalized fusion feature.

[0109] Step S140: Based on the concatenation and linear mapping of the attention outputs of all heads, residual connections and normalization are performed on the fused features to obtain the normalized result.

[0110] Step S150: Input the normalization result into the classification head to output the corresponding class's abnormal alarm.

[0111] In some embodiments, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes the following steps in step S150:

[0112] Step S151: Input the normalized fused features into the classification head, perform global average pooling on the fused features, and input the pooling results into multiple fully connected layers to obtain the classification results of each layer.

[0113] Step S152: Based on the classification results of each layer and the preset decision threshold, output the alarm probability of each type of anomaly, so as to send an alarm for the corresponding type of anomaly when the alarm probability exceeds the safety threshold.

[0114] Combination Figures 2 to 4 As shown in the specific embodiment, the petrochemical anomaly detection method based on multimodal attention fusion provided by the present invention includes step 4, multi-head self-attention discrimination.

[0115] For details, see Figure 4 As shown, firstly, a linear mapping to Q / K / V is performed, starting with... Do Head linear projection:

[0116]

[0117] The projection matrix for each head is as follows:

[0118]

[0119] Next, for each head, calculate the attention output:

[0120]

[0121]

[0122] Here, "self-attention" refers to using the fused features themselves to "question each other".

[0123] Then, the outputs of all the heads are concatenated and linearly mapped back to the original dimension:

[0124]

[0125] In the formula, .

[0126] Finally, perform residual joins and normalization:

[0127]

[0128] Through the above process, the features at each location are fused with global context information for discrimination.

[0129] Next, The normalized result is fed into the classification head, which outputs the alarm probability / category.

[0130] In this embodiment, to obtain a fixed-length vector, global average pooling (or max pooling) can be performed:

[0131]

[0132] After that, Several fully connected layers are added:

[0133]

[0134] The last layer is classified. :

[0135]

[0136] In the formula, It is the number of alarm categories (e.g., in binary classification). (Normal / Alarm)

[0137] Probability output: For binary alarms, the probability of the positive class ("alarm") can be taken. .like If this occurs, an alarm will be triggered. The threshold value is... It can be set to 0.5, or tuned on the validation set using ROC curves.

[0138] The following describes the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention. The petrochemical anomaly detection system based on multimodal attention fusion described below can be referred to in correspondence with the petrochemical anomaly detection method based on multimodal attention fusion described above.

[0139] like Figure 5 As shown, in one embodiment, a petrochemical anomaly detection system based on multimodal attention fusion includes an image preprocessing module 510, a feature extraction and fusion module 520, a multi-head attention output module 530, a feature normalization module 540, and an anomaly alarm module 550.

[0140] The image preprocessing module 510 is used to acquire real-time video frames of the petrochemical plant and preprocess the real-time video frames to obtain a binary image.

[0141] The feature extraction and fusion module 520 is used to call the convolutional neural network to extract the original features from the binary image, and combine the attention weights to map and weight the original features to obtain fused features.

[0142] The multi-head attention output module 530 is used to linearly project the fused features to calculate the attention output based on the projection matrix of each head, and to concatenate and linearly map the attention outputs of all heads.

[0143] The feature normalization module 540 is used to perform residual connection and normalization on the fused features based on the attention outputs of all heads by concatenating and linearly mapping them, and to obtain the normalized result.

[0144] The anomaly alarm module 550 is used to input the normalization result into the classification head to output anomaly alarms for the corresponding class.

[0145] In this embodiment, the image preprocessing module 510 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is specifically used for:

[0146] Real-time video frames are acquired by a surveillance camera and converted into grayscale images. The grayscale images are then smoothed using a Gaussian kernel to obtain filtered image data.

[0147] Background information in the filtered image data is separated by background subtraction to determine the foreground target in the filtered image data. The foreground target is then binarized to obtain the binarized foreground image and background model corresponding to the real-time video frame.

[0148] In this embodiment, the feature extraction and fusion module 520 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is specifically used for:

[0149] The convolutional neural network is invoked to extract multi-scale features from the binarized foreground image and determine the pooled feature map to construct a multi-scale feature vector.

[0150] The target object in the binary foreground image is located by transmitting pulse signals based on multi-scale feature vectors using ultra-wideband technology, and the spatial position of the target object is calculated by multi-base station triangulation method.

[0151] In this embodiment, the feature extraction and fusion module 520 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is further used for:

[0152] The instantaneous error of the target object being located by sending pulse signals using ultra-wideband technology is obtained, and a Kalman filter is used to optimize the instantaneous error to obtain the UWB positioning data of the target object.

[0153] Based on UWB positioning data and spatial location, multi-scale features are mapped to the query subspace, key subspace, and value subspace corresponding to multi-head attention, and the attention weight of each subspace is calculated.

[0154] In this embodiment, the multi-head attention output module 530 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is specifically used for:

[0155] We perform weighted aggregation of UWB positioning data from all visual locations to obtain UWB feature vectors.

[0156] A multi-head parallel approach is adopted to concatenate UWB feature vectors along the channels, and then perform linear mapping on the concatenated UWB feature vectors to map them to the original dimensions.

[0157] In this embodiment, the multi-head attention output module 530 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is further used for:

[0158] Using UWB feature vectors as the query subspace and multi-scale feature vectors as the key subspace or value subspace, real-time video frames are supplemented into the UWB positioning data through the attention weight matrix to obtain the first fusion sub-feature.

[0159] Using multi-scale feature vectors as the query subspace and UWB feature vectors as the key subspace or value subspace, UWB positioning data is supplemented into real-time video frames through attention weights to obtain the second fusion sub-feature.

[0160] The fusion ratio of the first fusion sub-feature and the second fusion sub-feature is calculated by a gating network, and the first fusion sub-feature and the second fusion sub-feature are weighted and fused based on the fusion ratio to obtain the normalized fusion feature.

[0161] In this embodiment, the anomaly alarm module 550 of the petrochemical anomaly detection system based on multimodal attention fusion provided by the present invention is specifically used for:

[0162] The normalized fused features are input into the classification head, and global average pooling is performed on the fused features. The pooling results are then input into multiple fully connected layers to obtain the classification results for each layer.

[0163] Based on the classification results of each layer and the preset decision threshold, the alarm probability of each type of anomaly is output so that when the alarm probability exceeds the safety threshold, an alarm for the corresponding type of anomaly is sent.

[0164] Figure 6 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 6As shown. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a petrochemical anomaly detection method based on multimodal attention fusion, which includes:

[0165] Acquire real-time video frames from a petrochemical plant and preprocess the real-time video frames to obtain binary images;

[0166] The convolutional neural network is invoked to extract the original features from the binary image, and attention weights are used to map and weight the original features to obtain fused features.

[0167] The fused features are linearly projected to compute attention outputs based on the projection matrix of each head, and the attention outputs of all heads are concatenated and linearly mapped.

[0168] Based on the concatenation and linear mapping of the attention outputs of all heads, the fused features are residually connected and normalized to obtain the normalized result;

[0169] Input the normalization result into the classification header to output anomaly alerts for the corresponding class.

[0170] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a petrochemical anomaly detection method based on multimodal attention fusion, the method comprising:

[0172] Acquire real-time video frames from a petrochemical plant and preprocess the real-time video frames to obtain binary images;

[0173] The convolutional neural network is invoked to extract the original features from the binary image, and attention weights are used to map and weight the original features to obtain fused features.

[0174] The fused features are linearly projected to compute attention outputs based on the projection matrix of each head, and the attention outputs of all heads are concatenated and linearly mapped.

[0175] Based on the concatenation and linear mapping of the attention outputs of all heads, the fused features are residually connected and normalized to obtain the normalized result;

[0176] Input the normalization result into the classification header to output anomaly alerts for the corresponding class.

[0177] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements a petrochemical anomaly detection method based on multimodal attention fusion, the method comprising:

[0178] Acquire real-time video frames from a petrochemical plant and preprocess the real-time video frames to obtain binary images;

[0179] The convolutional neural network is invoked to extract the original features from the binary image, and attention weights are used to map and weight the original features to obtain fused features.

[0180] The fused features are linearly projected to compute attention outputs based on the projection matrix of each head, and the attention outputs of all heads are concatenated and linearly mapped.

[0181] Based on the concatenation and linear mapping of the attention outputs of all heads, the fused features are residually connected and normalized to obtain the normalized result;

[0182] Input the normalization result into the classification header to output anomaly alerts for the corresponding class.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0184] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A petrochemical anomaly detection method based on multimodal attention fusion, characterized in that, The method includes: Real-time video frames of a petrochemical plant are acquired, and the real-time video frames are preprocessed to obtain a binary image. A convolutional neural network is invoked to extract the original features from the binary image, and attention weights are used to map and weight the original features to obtain fused features. The fused features are linearly projected to calculate the attention output based on the projection matrix of each head, and the attention outputs of all heads are concatenated and linearly mapped. Based on the concatenation and linear mapping of the attention outputs of all heads, the fused features are residually connected and normalized to obtain the normalized result; The normalization result is input into the classification head to output an anomaly alert for the corresponding class.

2. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 1, characterized in that, The process of acquiring real-time video frames from a petrochemical plant and preprocessing the real-time video frames to obtain a binary image includes: The real-time video frames are acquired by the surveillance camera and converted into grayscale images. The grayscale images are then smoothed using a Gaussian kernel to obtain filtered image data. Background information in the filtered image data is separated by background subtraction to determine the foreground target in the filtered image data. The foreground target is then binarized to obtain the binarized foreground image and background model corresponding to the real-time video frame.

3. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 2, characterized in that, The process involves calling a convolutional neural network to extract original features from the binary image, and then mapping and weighting these original features using attention weights to obtain fused features, including: The convolutional neural network is invoked to extract multi-scale features from the binarized foreground image and determine the pooled feature map to construct a multi-scale feature vector. The target object in the binarized foreground image is located by sending a pulse signal based on the multi-scale feature vector using ultra-wideband technology, and the spatial position of the target object is calculated by using a multi-base station triangulation method.

4. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 3, characterized in that, The step of calling a convolutional neural network to extract original features from the binary image, and combining attention weights to map and weight the original features to obtain fused features, also includes: The instantaneous error of the ultra-wideband technology in locating the target object by sending pulse signals is obtained, and the Kalman filter is used to optimize the instantaneous error to obtain the UWB positioning data of the target object. Based on the UWB positioning data and spatial location, the multi-scale features are mapped to the query subspace, key subspace, and value subspace corresponding to multi-head attention, and the attention weight of each subspace is calculated.

5. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 4, characterized in that, The step of linearly projecting the fused features to calculate the attention output based on the projection matrix of each head, and concatenating and linearly mapping the attention outputs of all heads, includes: The UWB positioning data at all visual locations are weighted and aggregated to obtain UWB feature vectors; A multi-head parallel approach is used to concatenate the UWB feature vectors along the channels to perform a linear mapping on the concatenated UWB feature vectors, thereby mapping the concatenated UWB feature vectors to the original dimension.

6. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 5, characterized in that, The step of linearly projecting the fused features to calculate the attention output based on the projection matrix of each head, and concatenating and linearly mapping the attention outputs of all heads, further includes: Using the UWB feature vector as the query subspace and the multi-scale feature vector as the key space or value subspace, the real-time video frames are supplemented into the UWB positioning data through the attention weight matrix to obtain the first fusion sub-feature; Using the multi-scale feature vector as the query subspace and the UWB feature vector as the key subspace or value subspace, the UWB positioning data is supplemented into the real-time video frame through attention weights to obtain the second fusion sub-feature; The fusion ratio of the first fusion sub-feature and the second fusion sub-feature is calculated by a gating network, and the first fusion sub-feature and the second fusion sub-feature are weighted and fused based on the fusion ratio to obtain the normalized fusion feature.

7. The petrochemical anomaly detection method based on multimodal attention fusion according to claim 1, characterized in that, The step of inputting the normalization result into the classification head to output an anomaly alert for the corresponding class includes: The normalized fusion features are input into the classification head, and global average pooling is performed on the fusion features. The pooling results are then input into multiple fully connected layers to obtain the classification results of each layer. Based on the classification results of each layer and the preset decision threshold, the alarm probability of each type of anomaly is output so that when the alarm probability exceeds the safety threshold, an alarm for the corresponding type of anomaly is sent.

8. A petrochemical anomaly detection system based on multimodal attention fusion, characterized in that, For implementing the petrochemical anomaly detection method based on multimodal attention fusion as described in any one of claims 1 to 7, the system comprises: An image preprocessing module is used to acquire real-time video frames from a petrochemical plant and preprocess the real-time video frames to obtain a binary image. The feature extraction and fusion module is used to call a convolutional neural network to extract original features from the binary image, and combine attention weights to map and weight the original features to obtain fused features; A multi-head attention output module is used to linearly project the fused features to calculate the attention output based on the projection matrix of each head, and to concatenate and linearly map the attention outputs of all heads. The feature normalization module is used to concatenate and linearly map the attention outputs of all heads, perform residual connection and normalization on the fused features, and obtain the normalization result. The anomaly alarm module is used to input the normalization result into the classification head to output anomaly alarms for the corresponding class.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the petrochemical anomaly detection method based on multimodal attention fusion as described in any one of claims 1 to 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the petrochemical anomaly detection method based on multimodal attention fusion as described in any one of claims 1 to 7.

Images