Industrial security anomaly detection method and device based on multi-modal data fusion, and medium

Abstract

The invention provides a multi-modal data fusion industrial security anomaly detection method, which comprises the steps of collecting multi-modal data, designing a double-flow space-time neural network, extracting video stream space-time characteristics in the multi-modal data, constructing a time sequence causal convolutional network to extract sensing stream time sequence characteristics in the multi-modal data, and embedding equipment topology position codes. Dynamically calibrating the video stream features and the sensing stream features through a cross attention mechanism to obtain calibrated video stream features and sensing stream features; constructing a feature abstract hierarchy, extracting single-hierarchy abstract features of the frequency flow features and the sensing flow features according to the feature abstract hierarchy, and performing cross-scale feature interactive fusion on the features of different hierarchies by adopting a bidirectional feature pyramid network to obtain fused features; and inputting the fusion features into a knowledge graph for reasoning, outputting an abnormal confidence coefficient, and reinforcing a learning model to generate an abnormal judgment threshold according to the abnormal confidence coefficient, the environmental parameters and the learning model.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent manufacturing technology, and in particular relates to an industrial safety anomaly detection method, equipment and medium based on multimodal data fusion. Background Art

[0002] In the existing technology, the current industrial safety monitoring field has many technical difficulties that need to be solved in terms of multimodal data processing, environmental adaptation, abnormal pattern recognition and alarm strategies. In terms of heterogeneous fusion of multimodal data, the core obstacles of video streams (RGB / infrared) and sensor signals (vibration, temperature, etc.) in industrial scenes are the mismatch of spatiotemporal dimensions and the misalignment of semantic levels. For example, the difference between the video frame rate and the sensor sampling rate is 2-3 orders of magnitude, and there is a lack of unified representation of pixel features and equipment status parameters. Traditional time window alignment methods will cause distortion of temporal features. Existing cross-modal fusion technologies, such as early fusion based on feature splicing or late fusion at the decision level, cannot capture fine-grained spatiotemporal correlations, and the feature fusion error rate is high under complex working conditions. In terms of dynamic environmental adaptability, the time-varying characteristics of industrial field environmental parameters (temperature, humidity, load rate) and equipment degradation effects jointly lead to detection threshold drift. Existing fixed threshold solutions, such as the rigid threshold intervals recommended by the ISO 10816 standard, experience a surge in false alarm rates when operating conditions fluctuate by more than ±15%. Traditional adaptive methods, however, fail to consider the physical constraints of equipment and can result in dangerous missed detections in sudden load changes. When it comes to identifying complex anomaly patterns, industrial anomalies are characterized by the coupling of multiple factors, with a high probability of mechanical failures and process parameter anomalies occurring simultaneously. Existing single-modal detection models, lacking domain knowledge embedding, offer limited accuracy in identifying latent correlated faults, such as lubricant temperature rise caused by bearing wear. Current knowledge-driven approaches, such as expert systems based on rule engines, struggle to process unstructured observational data, resulting in over 90% of alarm decisions relying on manual experience. Traditional alarm systems employ a single "alarm upon threshold crossing" model, lacking risk stratification and emergency response mechanisms. In large-scale continuous production systems, instantaneous false alarms can trigger cascading shutdowns, resulting in millions of yuan in economic losses per hour. Existing solutions lack the ability to coordinate decision-making across multiple subsystems, leading to a high rate of erroneous emergency responses. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems of mismatch in spatiotemporal dimensions and misalignment of semantic levels in multimodal data in the prior art, and the problem of detection threshold drift in dynamic environments. A multimodal data fusion industrial safety anomaly detection method is proposed, and a cross-modal feature fusion network and a dynamic threshold adjustment mechanism are developed to improve the accuracy of anomaly detection in complex industrial scenarios, reduce the false alarm rate, shorten the alarm response time, reduce the economic losses caused by false alarms, and improve the reliability and safety of the system.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0005] The multimodal data fusion industrial safety anomaly detection method comprises the following steps:

[0006] Collect multimodal data, design a two-stream spatiotemporal neural network, extract the spatiotemporal features of the video stream from the multimodal data, build a temporal causal convolutional network to extract the temporal features of the sensor stream from the multimodal data, and embed the device topology position code. Dynamically calibrate the video stream features and sensor stream features through the cross-attention mechanism to obtain the calibrated video stream features and sensor stream features.

[0007] Constructing a feature abstraction hierarchy, extracting single-level abstract features of the frequency flow features and the sensor flow features according to the feature abstraction hierarchy, and using a bidirectional feature pyramid network to interactively fuse features of different levels across scales to obtain fused features;

[0008] The fused features are input into the knowledge graph for reasoning, and the anomaly confidence is output. Based on the anomaly confidence and environmental parameters, the reinforcement learning model generates an anomaly judgment threshold.

[0009] Furthermore, collecting multimodal data includes:

[0010] Industrial cameras and sensors are used to synchronously collect multimodal data. The data collection process of industrial cameras and sensors is controlled by a unified clock signal, and the collected raw data is preprocessed.

[0011] Furthermore, the dual-stream spatiotemporal neural network is designed to extract the spatiotemporal features of the video stream in the multimodal data, and a temporal causal convolutional network is constructed to extract the temporal features of the sensor stream in the multimodal data, and the device topology position code is embedded. The video stream features and sensor stream features are dynamically calibrated through the cross-attention mechanism. The calibrated video stream features and sensor stream features include:

[0012] A 3D convolutional neural network is used to extract the spatiotemporal features of the video stream. By sliding the convolution kernel in the time dimension, it simultaneously captures the feature changes in space and time, and divides the time dimension of the video stream into segments synchronized with the sensor sampling period.

[0013] Encode the physical location information of the sensor into a vector and concatenate it with the sensor flow features;

[0014] The video stream features and sensor stream features are projected into the query, key and value spaces respectively, the attention weight matrix between the video stream and the sensor stream is calculated, and the video stream and sensor stream features are weightedly fused using the attention weight matrix to obtain the calibrated video stream features and sensor stream features.

[0015] Furthermore, the constructing of the feature abstraction level and extracting the single-level abstract features of the frequency flow features and the sensor flow features according to the feature abstraction level include:

[0016] Construct a four-level feature abstraction hierarchy, including pixel level, object level, scene level, and semantic level. Each level corresponds to different feature granularity, gradually abstracting and extracting information from low-level pixel features to high-level semantic features.

[0017] The pixel level includes features extracted from the raw data;

[0018] The object level involves identifying objects in images or local patterns in sensor signals through local feature extraction;

[0019] The scene level involves combining objects or local patterns to generate a description of the entire scene;

[0020] The semantic level involves further abstracting scene-level features to generate an understanding of the overall semantics.

[0021] Furthermore, the bidirectional feature pyramid network is used to interactively fuse features at different levels across scales, and the fused features obtained include:

[0022] A bidirectional feature pyramid network is used to establish bidirectional connections between single levels;

[0023] For a single layer, its input features come from the previous layer and the next layer, and the output features are obtained through weighted fusion;

[0024] In the single-level feature fusion process, learnable weights are introduced to dynamically adjust the contribution of features at different levels.

[0025] Furthermore, it also includes introducing deformable convolution to adaptively adjust the receptive field. For a single convolution kernel position, its offset is calculated, and the shape and position of the convolution kernel are adjusted through learnable parameters; the standard convolution kernel is adjusted using the offset to obtain a deformable convolution kernel, and a convolution operation is performed.

[0026] Furthermore,

[0027] Model the sub-device and alarm logic relationship as a Petri net, where a single sub-device corresponds to a single place, and a single alarm logic relationship corresponds to a single transition;

[0028] Define the state transition rules of the Petri net. When the tag of a single place meets the triggering condition of the transition, the transition is triggered and the tag is transferred from a single place to another place.

[0029] When a transition is triggered, the corresponding alarm action is executed according to the alarm logic relationship corresponding to the transition;

[0030] The sub-devices and emergency response measures are modeled as a decision tree, where a single node corresponds to a single condition judgment and a single branch corresponds to a single emergency response measure.

[0031] Furthermore, it also includes,

[0032] Deploy edge computing nodes at industrial sites to process multimodal data in real time. A single edge computing node collects and processes video streams and sensor data in its area.

[0033] Build a knowledge base in the cloud to store and manage industrial equipment operating data, historical abnormality records, equipment topology information, and expert knowledge;

[0034] The cloud-based knowledge base adopts a distributed storage architecture, interacts with edge computing nodes for data, and receives real-time data and preliminary detection results uploaded by edge computing nodes.

[0035] In order to achieve the above-mentioned objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program running on the processor, and when the processor runs the program, it executes the steps of the industrial safety anomaly detection method of multimodal data fusion as described above.

[0036] In order to achieve the above objectives, the present invention also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, execute the steps of the industrial safety anomaly detection method based on multimodal data fusion as described above.

[0037] The present invention proposes a multimodal data fusion industrial safety anomaly detection method, device, and medium, which have the following beneficial effects:

[0038] 1. Improve the accuracy of multimodal data fusion: Through the spatiotemporal attention alignment module and multi-scale feature pyramid fusion technology, the spatiotemporal dimension mismatch (frame rate difference of up to 3 orders of magnitude) and semantic level misalignment of video stream (200fps) and sensor signal (1kHz) are effectively solved. Through the dual-stream spatiotemporal network and cross-attention mechanism, dynamic calibration of spatiotemporal features is achieved, and the fusion error rate is reduced to. At the same time, the bidirectional feature pyramid network (BiFPN) combined with deformable convolution supports cross-scale feature interaction from pixel level to semantic level, improves the accuracy of anomaly detection, and improves recognition accuracy in scenarios with concurrent mechanical failures;

[0039] 2. Upgraded recognition capabilities for complex abnormal patterns. Traditional single-modal models have insufficient accuracy in identifying latent correlated faults (such as lubricating oil temperature rise caused by bearing wear). This invention uses a knowledge graph inference engine combined with a graph neural network (GNN) to implement abnormal evolution path reasoning, thereby improving the recognition rate of latent faults. For example, the knowledge graph automatically associates the causal chain of "bearing wear → vibration spectrum change → oil temperature rise" and combines multimodal fusion features to accurately identify early complex faults, avoiding reliance on manual experience.

[0040] 3. Intelligent hierarchical alarm and emergency response optimization. The traditional "threshold crossing alarm" mode is prone to cause cascading shutdowns in large-scale continuous production systems. The present invention designs a five-level risk warning system (observation → warning → alarm → emergency → shutdown) and an alarm logic engine based on Petri nets to achieve cross-subsystem collaborative decision-making. Through dynamic risk classification and emergency command distribution, the alarm response time is shortened and the emergency error operation rate is reduced. For example, when the "warning level" is triggered, a drone re-inspection is initiated. After confirming the abnormality, it is upgraded to the "emergency level" for local shutdown to avoid a full production stop.

[0041] 5. System deployment costs and real-time performance are optimized by adopting an edge-cloud hybrid architecture (NVIDIA Jetson AGX + cloud knowledge base). Edge nodes support real-time processing of 200fps video streams and 1kHz sensor data, and the cloud implements thousand-node device cluster management, reducing system deployment costs.

[0042] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 Flowchart of the multimodal data fusion industrial safety anomaly detection method of the present invention;

[0045] Figure 2 Schematic diagram of the process of dynamic calibration of multimodal data of the present invention;

[0046] Figure 3 Schematic diagram of the process of multi-scale feature pyramid fusion of the present invention;

[0047] Figure 4 A schematic diagram of the process flow of the multi-level linkage alarm strategy of the present invention;

[0048] Figure 5 Schematic diagram of the process of the edge-cloud collaboration mechanism of the present invention;

[0049] Figure 6 A schematic structural diagram of an electronic device 300 for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0051] Example 1

[0052] Figure 1 The flowchart of the industrial safety anomaly detection method based on multimodal data fusion of the present invention is as follows. Figure 1 , the industrial safety anomaly detection method based on multimodal data fusion of the present invention is described in detail.

[0053] In step 101, industrial cameras and vibration / temperature sensors are deployed to provide multimodal data for subsequent anomaly detection.

[0054] Optionally, data acquisition uses high-precision industrial cameras and multiple sensors for simultaneous multimodal data acquisition. The industrial cameras capture video streams at a frame rate of 200 frames per second (fps) and support both RGB and infrared imaging, ensuring clear image information under varying lighting conditions. Simultaneously, vibration and temperature sensors collect vibration and temperature signals from the device in real time at a sampling rate of 1kHz, ensuring complete capture of high-frequency dynamic characteristics.

[0055] In this embodiment, to achieve precise synchronization of multimodal data, a hardware trigger mechanism is employed to control the data acquisition process of the industrial camera and sensor using a unified clock signal. Specifically, a high-precision clock module is deployed that generates a synchronization pulse signal at a frequency of 1kHz. Both the industrial camera and the vibration / temperature sensor are equipped with a hardware trigger interface. When the synchronization pulse signal is received, the industrial camera and sensor begin data acquisition simultaneously. This hardware trigger mechanism effectively resolves the problem of time deviation during multimodal data acquisition, ensuring precise alignment of the video stream and sensor signals in the time dimension.

[0056] Optionally, data preprocessing is also included to perform preliminary processing on the collected raw data. For video stream data, data preprocessing begins with image denoising and color correction to ensure image quality meets the requirements of subsequent analysis. For vibration and temperature signals, data preprocessing includes signal filtering and baseline correction to eliminate the effects of environmental noise and sensor drift. The preprocessed data is transmitted to subsequent analysis modules via a high-speed data transmission interface (such as GigE or USB 3.0), ensuring real-time data integrity.

[0057] As an example, assume the system is deployed on a large piece of industrial equipment containing multiple critical components, such as bearings, gears, and motors. Industrial cameras are installed at key locations on the equipment, capturing images of the equipment's operating status at a frame rate of 200 fps. Vibration sensors and temperature sensors are mounted on the bearings and motor, respectively, collecting vibration and temperature signals at a sampling frequency of 1 kHz. A high-precision clock module generates a 1 kHz synchronization pulse signal, which controls data acquisition from the industrial camera and sensors via a hardware trigger interface. The captured video stream undergoes image denoising and color correction, while the vibration and temperature signals are filtered and baseline corrected before being transmitted to subsequent analysis modules via a high-speed data transmission interface. This precisely synchronized data acquisition method enables the system to acquire high-quality multimodal data, providing a reliable foundation for subsequent anomaly detection.

[0058] In step 102, a dual-stream spatiotemporal Transformer network is designed, 3D-CNN is used to extract the spatiotemporal features of the video stream, a temporal causal convolutional network is constructed to process the sensor stream, and dynamic calibration of the spatiotemporal features is achieved through a cross-attention mechanism.

[0059] Optionally, the video stream feature extraction includes: extracting spatiotemporal features of the video stream using a 3D-CNN (three-dimensional convolutional neural network). The 3D-CNN can simultaneously capture feature changes in space and time by sliding the convolution kernel in the time dimension.

[0060] Specifically, the input video stream data is V∈R T×H×W×C , where T is the number of time frames, H and W are the height and width of the image respectively, and C is the number of channels. The convolution kernel of 3D-CNN is K∈R t×h×w×C×C′ , where t, h, and w are the convolution kernel sizes in time, height, and width, respectively, and C′ is the number of output channels. The convolution operation can be expressed as:

[0061] V′=Conv3D(V,K)

[0062] where V′∈R T′×H′×W′×C′ is the extracted spatiotemporal features.

[0063] To align with the temporal dimension of the sensor data, the temporal dimension of the video stream is divided into segments synchronized with the sensor sampling period. Specifically, assuming the sensor sampling frequency is 1kHz and the video frame rate is 200fps, every 5 frames of video corresponds to a sensor sampling point.

[0064] Optionally, sensor stream feature extraction includes: processing sensor stream data using a temporal causal convolutional network (TCCN). TCCN ensures that the output of the current time step depends only on the current and previous time steps through causal convolution operations, thus avoiding leakage of future information. The input sensor data is S∈R T×D , where T is the number of time steps and D is the sensor dimension. The convolution kernel of TCCN is K′∈R k×D×D′ , where k is the convolution kernel size and D′ is the output dimension. The causal convolution operation can be expressed as:

[0065] S′=CausalConv(S, K′)

[0066] where S′∈R T×D′ is the extracted time series feature.

[0067] In order to enhance the physical meaning of sensor features, the device topology position encoding is embedded. Specifically, the physical location information of the sensor is encoded as a vector P∈R Dp , and concatenate it with the sensor feature S′ to obtain the enhanced feature S″∈R T ×(D′+Dp) .

[0068] Optionally, the cross-attention mechanism includes: dynamically calibrating the video stream and sensor stream features through the cross-attention mechanism. Specifically, first, the video stream features V′ and the sensor stream features S″ are projected into the query, key, and value spaces respectively:

[0069] Q V =W Q V′,K V =W K V′,V V =W V V′

[0070] Q S =W Q S″,K S =W K S″,V S =W V S″

[0071] Where W Q 、W K and W V is the learnable projection matrix.

[0072] Calculate the attention weight matrix A∈R between the video stream and the sensor stream T×T :

[0073]

[0074] where d k is the dimension of the key vector.

[0075] Use the attention weight matrix to perform weighted fusion of video stream and sensor stream features:

[0076] V″=AV S ,S″′=A T V V

[0077] Finally, the calibrated video stream feature V″∈R is obtained T×H′×W′×C′ and sensor flow features S″′∈R T×(D′+Dp) .

[0078] Optionally, feature alignment includes setting a time window and performing spatial grid division. The role of the time window is to divide the continuous time series data into segments of fixed length for subsequent processing. The time window is to match the sampling period of industrial cameras and sensors, while ensuring that the data segment contains sufficient information. The selection of the time window needs to balance computational efficiency and information integrity. Too short may lead to insufficient information, while too long may introduce redundancy. Spatial grid division is commonly used to process image data, that is, dividing the image into fixed-size areas (for example, 32×32 pixels). This division method helps to extract local features and reduce computational complexity. Combined with the high frame rate data of industrial cameras, this division can generate a large number of local features in a short time, which is convenient for subsequent feature alignment and analysis.

[0079] In this embodiment, multimodal data (image, vibration, temperature) comes from different sensors and may be asynchronous in time or space. By using time windows and spatial gridding, we can achieve temporal synchronization and spatial alignment of multimodal data, providing consistent input for subsequent model training.

[0080] As an example, assume that the system is deployed on a large industrial equipment that contains multiple key components such as bearings, gears, and motors. The industrial camera captures the operating status image of the equipment at a frame rate of 200fps, and the vibration sensor and temperature sensor collect vibration and temperature signals at a sampling frequency of 1kHz. Figure 2 As shown in the figure, a 3D-CNN is first used to extract spatiotemporal features from the video stream, dividing the time dimension into segments synchronized with the sensor sampling period. A temporal causal convolutional network is then used to process the vibration and temperature signals and embed the physical location information of the device. Finally, a cross-attention mechanism is used to dynamically align the features of the video and sensor streams, resulting in calibrated video and sensor stream features. This precise feature alignment enables the system to capture fine-grained spatiotemporal correlations and improve the accuracy of anomaly detection.

[0081] In step 103, a four-layer feature abstraction hierarchy is constructed, a bidirectional feature pyramid network (BiFPN) is used to implement cross-scale feature interaction, and a deformable convolution is introduced to adaptively adjust the receptive field.

[0082] In this embodiment, the receptive field is the size of the area mapped on the input image by the pixels on the feature map output by each layer of the convolutional neural network.

[0083] Optionally, four feature abstraction levels are constructed: pixel level, object level, scene level, and semantic level. Each level corresponds to a different feature granularity, gradually abstracting and extracting information from low-level pixel features to high-level semantic features.

[0084] In this embodiment, pixel-level features are extracted directly from raw data, such as pixel values in video streams and raw sample values of sensor signals. Object-level features are extracted to identify objects in images or local patterns in sensor signals. Scene-level features combine multiple objects or local patterns to form a description of the entire scene. Semantic-level features are further abstracted to form an understanding of the overall semantics, such as device status or anomaly type.

[0085] Optionally, a bidirectional feature pyramid network (BiFPN) is used to achieve cross-scale feature interaction. BiFPN uses bidirectional connections and weighted fusion to efficiently interact and fuse features at different levels.

[0086] Optionally, BiFPN establishes bidirectional connections between each level, allowing features to flow freely between upper and lower levels. Specifically, for each level L, its input feature F L From the previous level F L-1 and the next level F L+1 , the output feature F is obtained by weighted fusion L ′:

[0087] F L ′=Conv(w L-1 F L-1 +w L+1 F L+1 )

[0088] where w L-1 and w L+1 is the learnable weight and Conv is the convolution operation.

[0089] Optionally, weighted fusion includes introducing learnable weights in the feature fusion process of each level to dynamically adjust the contribution of features at different levels. Specifically, for each level l, its output feature F L "for:

[0090] F L =Conv(w L F L ′+w L-1 F L-1 ′+w L+1 F L+1 ′)

[0091] where w L 、w L-1 and w L+1 are learnable weights.

[0092] Optionally, a deformable convolution is introduced to adaptively adjust the receptive field to accommodate features of different scales and shapes. Deformable convolution adds an offset to the standard convolution kernel, enabling the convolution kernel to adaptively adjust its shape and position to better capture irregular features.

[0093] Optionally, for each convolution kernel position p, its offset Δp is calculated, and the shape and position of the convolution kernel are adjusted by learnable parameters. Specifically, the offset Δp can be expressed as:

[0094] Δp=Conv(F L )

[0095] Where Conv is the convolution operation, F L is the input feature.

[0096] Deformable convolution operation: Use the offset Δp to adjust the standard convolution kernel to obtain the deformable convolution kernel K d , and perform convolution operation:

[0097] F L ″′=DeformConv(F L , K d )

[0098] Among them, DeformConv is a deformable convolution operation.

[0099] As an example, assume that the system is deployed on a large industrial equipment that contains multiple key components such as bearings, gears, and motors. Figure 3As shown in the figure, pixel-level features are first extracted from the video stream and sensor signals, such as the pixel values of the video stream and the raw sample values of the sensor signals. Then, through local feature extraction, objects in the image or local patterns in the sensor signals are identified to form object-level features. Next, multiple objects or local patterns are combined to form a description of the entire scene, resulting in scene-level features. Finally, the scene-level features are further abstracted to form an understanding of the overall semantics, such as device status or anomaly type, to obtain semantic-level features. Through the bidirectional connection and weighted fusion of BiFPN, features at different levels are efficiently interacted and fused, and the receptive field is adaptively adjusted through deformable convolution to ultimately obtain the fused features. Through this multi-scale feature fusion approach, the system can capture feature information of different granularities and improve the accuracy of anomaly detection.

[0100] In step 104, the fused features are input into the knowledge graph for reasoning, and the anomaly confidence is output. Based on the anomaly confidence and environmental parameters, the reinforcement learning model generates an anomaly judgment threshold.

[0101] Optionally, the industrial ontology knowledge base defines the triple relationship between equipment, process and risk, providing structured prior knowledge for anomaly detection and reasoning. Graph neural networks (GNNs) can directly act on knowledge graphs, support reasoning on equipment state migration paths, and help discover potential risk evolution patterns.

[0102] Optionally, the fused semantic features are input into the knowledge graph engine to match the equipment-process-risk triples (such as "bearing wear → lubricating oil temperature rise → equipment shutdown risk") and output the anomaly confidence level.

[0103] In this embodiment, the environmental parameter generation process includes inputting operating condition characteristics and equipment physical constraints. The operating condition characteristics include a 6-dimensional vector (temperature, humidity, load rate, vibration amplitude, equipment operating time, and maintenance records). The equipment physical constraints include preset limit parameters (such as maximum allowable temperature and vibration threshold).

[0104] In this embodiment, the operating condition characteristics of the equipment are characterized by six-dimensional feature vectors such as temperature, humidity, and load rate. This multi-dimensional feature space design can fully reflect the status of the equipment in different environments and provide a basis for subsequent dynamic threshold adjustment.

[0105] Alternatively, a parameter generator based on reinforcement learning (RL) can be developed. Reinforcement learning (RL) is a method well-suited for complex decision-making problems. In this scenario, RL can be used to optimize the decision boundary of the device state space in real time. The core idea is to use a reward function to guide the model to continuously adjust parameters to find the optimal threshold setting.

[0106] In this embodiment, the optimal decision boundary for the device state space is generated in real time. This generation requires a combination of real-time data and historical experience. The reinforcement learning model can gradually approach the optimal solution by continuously interacting with the environment (e.g., receiving real-time sensor data), thereby dynamically adjusting the threshold.

[0107] Optionally, the reinforcement learning model includes:

[0108] State space (State): current working condition characteristics + real-time equipment status (such as the confidence level of abnormalities after fusion).

[0109] Action space (Action): Dynamically adjusted judgment threshold range (such as temperature threshold ± ΔT).

[0110] Reward function (Reward):

[0111]

[0112] The reinforcement learning model updates its state based on real-time data and selects the action that maximizes the long-term reward (i.e., the optimal threshold). If the generated threshold exceeds the device's safety range, it is forcibly truncated to the preset limit.

[0113] In this embodiment, in practical applications, the choice of reinforcement learning model (such as deep Q-network (DQN) or policy gradient method) and the design of reward function are crucial. It is necessary to ensure the quality and diversity of training data to avoid overfitting or convergence to a local optimum.

[0114] During the model training phase, historical data (including normal / abnormal labels) is used to initialize model parameters for offline pre-training. The Q-value table or policy network is updated based on real-time alarm feedback (TP / FP / FN) for online fine-tuning.

[0115] In step 105, a risk warning system is designed, and an alarm logic reasoning engine based on Petri nets is developed to implement a cross-subsystem collaborative emergency response decision tree.

[0116] Optionally, a risk warning system can be designed, with levels of observation, warning, alarm, emergency, and shutdown. Each level corresponds to a different risk level, with the response level increasing gradually from low risk to high risk.

[0117] In this embodiment, the observation level: The system detects a minor anomaly that does not reach the warning threshold, records the anomaly information, and continuously monitors it. The warning level: The system detects an anomaly approaching the warning threshold, issues a warning signal, and notifies relevant personnel to conduct an inspection. The alarm level: The system detects an anomaly exceeding the alarm threshold, issues an alarm signal, and initiates preliminary emergency measures. The emergency level: The system detects a serious anomaly, activates the emergency response mechanism, and takes urgent measures to prevent the accident from escalating. The shutdown level: The system detects an extreme anomaly and immediately shuts down the system to prevent equipment damage or casualties.

[0118] Alternatively, develop an alarm logic reasoning engine based on Petri nets to implement a cross-subsystem collaborative emergency response decision tree. Petri nets are a mathematical model used to describe concurrent systems that can effectively handle complex logical relationships and state transitions.

[0119] Petri Net Model Construction: Model the system's subsystems and alarm logic relationships as Petri nets. Each subsystem corresponds to a place, and each alarm logic relationship corresponds to a transition. The connection between places and transitions is represented by arcs.

[0120] State transition rules: Define the state transition rules of the Petri net. When the token of a place meets the triggering condition of the transition, the transition is triggered and the token is transferred from one place to another. Specifically, for transition t, its triggering condition can be expressed as:

[0121]

[0122] Where ·t represents the input place of transition t, M(p) represents the number of labels of place p, and W(p,t) represents the weight from place p to transition t.

[0123] Alarm logic reasoning: Alarm logic reasoning is implemented through state transitions in Petri nets. When a transition is triggered, the system executes the corresponding alarm action based on the corresponding alarm logic relationship. For example, when transition t1 is triggered, the system issues a warning signal; when transition t2 is triggered, the system issues an alarm signal.

[0124] Optionally, a cross-subsystem collaborative emergency response decision tree can be implemented to ensure rapid and accurate response to anomalies in complex industrial scenarios. The decision tree uses multi-level conditional judgments to gradually narrow the response scope and ultimately determine the optimal emergency response measures.

[0125] Decision tree construction involves modeling the system's subsystems and emergency response measures as a decision tree. Each node corresponds to a conditional judgment, and each branch corresponds to an emergency response measure. For example, the root node determines the anomaly type, the child nodes determine the anomaly severity, and the leaf nodes determine the specific emergency response measures.

[0126] Collaborative response mechanism: This achieves a coordinated response across subsystems through multi-level conditional judgments within a decision tree. When a subsystem detects an anomaly, the system gradually determines the optimal emergency response based on the conditions in the decision tree and notifies the relevant subsystems to execute it. For example, if subsystem A detects an anomaly, the system determines the optimal emergency response as shutdown based on the conditions in the decision tree and notifies subsystems B and C to execute the shutdown.

[0127] As an example, assume that the system is deployed on a large industrial equipment that contains multiple key components such as bearings, gears, and motors. Figure 4 As shown in the figure, the system first detects a slight increase in bearing temperature, but does not reach the warning threshold. It records the abnormality and continuously monitors the situation, entering the observation level. Then, as the bearing temperature continues to rise and approaches the warning threshold, the system issues a warning signal, notifying relevant personnel for inspection, and entering the warning level. Then, as the bearing temperature exceeds the alarm threshold, the system issues an alarm signal, initiates preliminary emergency measures, and enters the alarm level. Subsequently, as the bearing temperature rises sharply, the system activates the emergency response mechanism, taking urgent measures to prevent the accident from escalating, and enters the emergency level. Finally, as the bearing temperature reaches an extreme abnormal value, the system immediately shuts down to prevent equipment damage or casualties, entering the shutdown level. Using the Petri net alarm logic inference engine, the system executes the corresponding alarm actions based on the alarm logic relationships corresponding to the transitions. Through a cross-subsystem coordinated emergency response decision tree, the system gradually determines the optimal emergency response based on the conditions in the decision tree and notifies the relevant subsystems to execute it. This multi-level coordinated alarm strategy enables the system to respond to anomalies quickly and accurately, improving the reliability and safety of industrial safety monitoring.

[0128] In step 106, a hybrid architecture of edge computing nodes (NVIDIA Jetson AGX) and cloud knowledge base is adopted to complete the deployment of the entire system.

[0129] Alternatively, as Figure 5As shown in the figure, NVIDIA Jetson AGX edge computing nodes are deployed at key locations on the industrial site to process multimodal data in real time. Each edge computing node is responsible for collecting and processing video streams and sensor data in its area. Specifically, video streams are collected by industrial cameras at a frame rate of 200fps, and sensor data (such as vibration, temperature, etc.) are collected at a sampling rate of 1kHz. The edge computing nodes extract the spatiotemporal features of video streams and sensor data through 3D-CNN and temporal causal convolutional networks respectively, and realize dynamic calibration of spatiotemporal features through the cross-attention mechanism. The edge computing nodes are also responsible for performing preliminary anomaly detection and alarm logic reasoning to ensure rapid local response to anomalies.

[0130] Optionally, a knowledge base is built in the cloud to store and manage industrial equipment operating data, historical anomaly records, equipment topology information, and expert knowledge. The cloud-based knowledge base interacts with edge computing nodes through an API interface, receiving real-time data and preliminary detection results uploaded by edge computing nodes. The cloud-based knowledge base uses a distributed storage architecture to ensure high data availability and scalability. Specifically, the cloud-based knowledge base includes the following modules:

[0131] Data storage module: stores the operating data and historical abnormality records of industrial equipment, and uses a combination of time series databases (such as InfluxDB) and relational databases (such as PostgreSQL) to ensure efficient storage and query of data.

[0132] Knowledge management module: stores and manages device topology information, expert knowledge, and abnormal pattern libraries, using a combination of graph databases (such as Neo4j) and document databases (such as MongoDB) to ensure efficient management and retrieval of knowledge.

[0133] Data analysis module: performs complex data analysis and abnormal pattern recognition, using machine learning algorithms (such as random forest, support vector machine, etc.) and deep learning algorithms (such as LSTM, Transformer, etc.) to ensure accurate recognition of complex abnormal patterns.

[0134] Optionally, a collaborative mechanism between edge computing nodes and cloud-based knowledge bases can be implemented to ensure efficient system operation and accurate anomaly detection. Specifically, edge computing nodes are responsible for real-time processing of multimodal data and performing preliminary anomaly detection and alarm logic reasoning locally. When an anomaly is detected by an edge computing node, the anomaly data and preliminary detection results are uploaded to the cloud-based knowledge base. The cloud-based knowledge base performs further data analysis and anomaly pattern recognition, and updates the anomaly pattern library and expert knowledge in the knowledge base. The cloud-based knowledge base is also responsible for coordinating the collaborative work of multiple edge computing nodes to ensure a quick and accurate response to anomalies in complex industrial scenarios.

[0135] This paper proposes a multimodal data fusion method for industrial safety anomaly detection, combining cross-modal feature fusion capabilities with a knowledge-driven reasoning mechanism. By combining advanced deep learning models (such as spatiotemporal Transformer, BiFPN, and deformable convolution) with knowledge graph technology, this architecture enables efficient and reliable intelligent decision-making in complex industrial environments.

[0136] Example 2

[0137] Reference below Figure 6 , which shows a block diagram of an electronic device 300 suitable for implementing an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0138] like Figure 6 As shown, the electronic device 300 may include a processing device 310, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 320 or a program loaded from a storage device 380 into a random access memory (RAM) 330. The processing device 310 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processing device 310 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processing device 310 performs the various methods and processes described above.

[0139] Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 330. The processing device 310, the ROM 320, and the RAM 330 are connected to each other via a bus 340. An input / output (I / O) interface 350 is also connected to the bus 340.

[0140] Typically, the following devices may be connected to the I / O interface 350: an input device 360 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 370 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 380 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 390. The communication device 390 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0141] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 390, or installed from the storage device 380, or installed from the ROM 320. When the computer program is executed by the processing device 310, the above-mentioned functions defined in the method of the embodiment of the present application are performed. Alternatively, in other embodiments, the processing device 310 may be configured to execute the method in any other appropriate manner (for example, by means of firmware): collecting multimodal data, designing a dual-stream spatiotemporal neural network, extracting spatiotemporal features of the video stream in the multimodal data, constructing a temporal causal convolutional network to extract temporal features of the sensor stream in the multimodal data, and embedding the device topological position code, dynamically calibrating the video stream features and the sensor stream features through a cross-attention mechanism to obtain calibrated video stream features and sensor stream features; constructing a feature abstraction hierarchy, extracting single-level abstract features of the frequency stream features and sensor stream features according to the feature abstraction hierarchy, and using a bidirectional feature pyramid network to perform cross-scale feature interaction on features of different levels to obtain fused features; inputting the fused features into the knowledge graph for reasoning, outputting anomaly confidence, and generating anomaly judgment thresholds based on the anomaly confidence and environmental parameters using a reinforcement learning model.

[0142] Example 3

[0143] The computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0144] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0146] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: collects multimodal data, designs a dual-stream spatiotemporal neural network, extracts the spatiotemporal features of the video stream in the multimodal data, constructs a temporal causal convolutional network to extract the temporal features of the sensor stream in the multimodal data, and embeds the device topological position code, dynamically calibrates the video stream features and the sensor stream features through the cross-attention mechanism, and obtains the calibrated video stream features and sensor stream features; constructs a feature abstraction hierarchy, extracts the single-level abstract features of the frequency stream features and the sensor stream features according to the feature abstraction hierarchy, and uses a bidirectional feature pyramid network to perform cross-scale feature interaction and fusion of features of different levels to obtain fused features; inputs the fused features into the knowledge graph for reasoning, outputs the anomaly confidence, and generates the anomaly judgment threshold based on the anomaly confidence and environmental parameters using the reinforcement learning model.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0150] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof.

[0151] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0156] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0157] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0158] The embodiments described above are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0159] In the description of the embodiments of the present application, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the application is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the electric vehicle or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0160] In the description of the embodiments of the present application, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0161] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. A multimodal data fusion industrial safety anomaly detection method, characterized in that: include: Collect multimodal data, design a two-stream spatiotemporal neural network, extract the spatiotemporal features of the video stream from the multimodal data, build a temporal causal convolutional network to extract the temporal features of the sensor stream from the multimodal data, and embed the device topology position code. Dynamically calibrate the video stream features and sensor stream features through the cross-attention mechanism to obtain the calibrated video stream features and sensor stream features. Constructing a feature abstraction hierarchy, extracting single-level abstract features of the frequency flow features and the sensor flow features according to the feature abstraction hierarchy, and using a bidirectional feature pyramid network to interactively fuse features of different levels across scales to obtain fused features; The fused features are input into the knowledge graph for reasoning, and the anomaly confidence is output. Based on the anomaly confidence and environmental parameters, the reinforcement learning model generates an anomaly judgment threshold.

2. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1 is characterized in that: The collecting of multimodal data includes: Industrial cameras and sensors are used to synchronously collect multimodal data. The data collection process of industrial cameras and sensors is controlled by a unified clock signal, and the collected raw data is preprocessed.

3. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1 is characterized in that: The dual-stream spatiotemporal neural network is designed to extract the spatiotemporal features of the video stream from the multimodal data. A temporal causal convolutional network is constructed to extract the temporal features of the sensor stream from the multimodal data and embed the device topology position code. The video stream features and sensor stream features are dynamically calibrated through the cross-attention mechanism. The calibrated video stream features and sensor stream features include: A 3D convolutional neural network is used to extract the spatiotemporal features of the video stream. By sliding the convolution kernel in the time dimension, it simultaneously captures the feature changes in space and time, and divides the time dimension of the video stream into segments synchronized with the sensor sampling period. Encode the physical location information of the sensor into a vector and concatenate it with the sensor flow features; The video stream features and sensor stream features are projected into the query, key and value spaces respectively, the attention weight matrix between the video stream and the sensor stream is calculated, and the video stream and sensor stream features are weightedly fused using the attention weight matrix to obtain the calibrated video stream features and sensor stream features.

4. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1 is characterized in that: The step of constructing a feature abstraction level and extracting single-level abstract features of the frequency flow features and the sensor flow features according to the feature abstraction level includes: Construct a four-level feature abstraction hierarchy, including pixel level, object level, scene level, and semantic level. Each level corresponds to different feature granularity, gradually abstracting and extracting information from low-level pixel features to high-level semantic features. The pixel level includes features extracted from the raw data; The object level involves identifying objects in images or local patterns in sensor signals through local feature extraction; The scene level involves combining objects or local patterns to generate a description of the entire scene; The semantic level involves further abstracting scene-level features to generate an understanding of the overall semantics.

5. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1 is characterized in that: The bidirectional feature pyramid network is used to interactively fuse features at different levels across scales, and the fused features obtained include: A bidirectional feature pyramid network is used to establish bidirectional connections between single levels; For a single layer, its input features come from the previous layer and the next layer, and the output features are obtained through weighted fusion; In the single-level feature fusion process, learnable weights are introduced to dynamically adjust the contribution of features at different levels.

6. The industrial safety anomaly detection method based on multimodal data fusion according to claim 5 is characterized in that: It also includes the introduction of deformable convolution to adaptively adjust the receptive field. For the position of a single convolution kernel, its offset is calculated, and the shape and position of the convolution kernel are adjusted through learnable parameters; the standard convolution kernel is adjusted using the offset to obtain a deformable convolution kernel, and a convolution operation is performed.

7. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1 is characterized in that: Also includes, The sub-device and alarm logic relationship is modeled as a Petri net, where a single sub-device corresponds to a single place, and a single alarm logic relationship corresponds to a single transition; Define the state transition rules of the Petri net. When the tag of a single place meets the triggering condition of the transition, the transition is triggered and the tag is transferred from a single place to another place. When a transition is triggered, the corresponding alarm action is executed according to the alarm logic relationship corresponding to the transition; The sub-devices and emergency response measures are modeled as a decision tree, where a single node corresponds to a single condition judgment and a single branch corresponds to a single emergency response measure.

8. The industrial safety anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: Also includes, Deploy edge computing nodes at industrial sites to process multimodal data in real time. A single edge computing node collects and processes video streams and sensor data in its area. Build a knowledge base in the cloud to store and manage industrial equipment operating data, historical abnormality records, equipment topology information, and expert knowledge; The cloud-based knowledge base adopts a distributed storage architecture, interacts with edge computing nodes for data, and receives real-time data and preliminary detection results uploaded by edge computing nodes.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a program running on the processor, and the processor executes the industrial safety anomaly detection method of multimodal data fusion according to any one of claims 1 to 8 when running the program.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the industrial safety anomaly detection method based on multimodal data fusion according to any one of claims 1 to 8 is executed.

Images