Industrial equipment multi-modal data real-time anomaly detection method, device, equipment, medium and product

Abstract

The invention discloses an industrial equipment multi-modal data real-time anomaly detection method, device, equipment, medium and product, and relates to the field of industrial equipment multi-modal data anomaly detection.The method comprises the steps that multi-modal sensor data and spatio-temporal characteristics are fused, and enhanced spatio-temporal fusion characteristics are determined; the multi-mode sensor data comprises an equipment vibration signal, a current signal and a temperature signal; performing feature engineering processing on the enhanced space-time fusion features, and determining processed space-time fusion features; based on an isolated forest model, performing anomaly detection on the processed space-time fusion features, and determining an anomaly score; constructing a dynamic early warning mechanism, determining whether the multi-modal sensor data is abnormal or not according to the abnormal score and the dynamic early warning mechanism, and determining a judgment result; the multi-modal sensor data and the judgment result are visually displayed through a visual framework, the spatial-temporal feature modeling capability can be improved, and dynamic adaptability is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial equipment multi-modal data anomaly detection, and particularly relates to an industrial equipment multi-modal data real-time anomaly detection method, device, equipment, medium and product. BACKGROUND

[0002] Equipment health monitoring technology has become a core link to ensure production safety and efficiency. The current mainstream scheme mainly relies on traditional statistical methods, machine learning and deep learning technology, but all have significant limitations. Traditional methods are based on fixed thresholds (such as ISO 10816 vibration standards) and manual experience, with a false positive rate of up to 15%-20% under complex working conditions, and cannot capture multi-sensor collaborative features; machine learning methods can improve accuracy through feature engineering, but have insufficient time series correlation modeling capabilities, with detection delays generally exceeding 100ms; deep learning methods (such as CNN, LSTM) can achieve 95% F1-score on the PHM2012 dataset, but the model parameter volume is large and relies on GPU computing power, making it difficult to deploy on edge devices. More importantly, existing technologies generally separate time and space feature analysis, resulting in insufficient detection capability for complex faults such as bearing cracks and gear wear, and the threshold strategy is rigid and cannot adapt to dynamic working conditions, resulting in high false positive rates.

[0003] It can be seen that the core problem of the existing technology is the lack of spatiotemporal feature modeling capability and dynamic adaptability. SUMMARY

[0004] The purpose of the present application is to provide an industrial equipment multi-modal data real-time anomaly detection method, device, equipment, medium and product to solve the problems of insufficient spatiotemporal feature modeling capability and lack of dynamic adaptability.

[0005] To achieve the above purpose, the present application provides the following solutions.

[0006] In a first aspect, the present application provides an industrial equipment multi-modal data real-time anomaly detection method, comprising the following steps.

[0007] Fusing multi-modal sensor data and spatiotemporal features to determine enhanced spatiotemporal fusion features; the multi-modal sensor data includes device vibration signals, current signals and temperature signals.

[0008] Performing feature engineering processing on the enhanced spatiotemporal fusion features to determine processed spatiotemporal fusion features.

[0009] Performing anomaly detection on the processed spatiotemporal fusion features based on an isolation forest model to determine an anomaly score.

[0010] The judgment module is configured to construct a dynamic early warning mechanism, determine whether the multi-modal sensor data is abnormal according to the abnormal score and the dynamic early warning mechanism, and determine a judgment result.

[0011] The visualization display module is configured to visually display the multi-modal sensor data and the judgment result through a visual architecture.

[0012] In a second aspect, the present application provides an industrial equipment multi-modal data real-time anomaly detection device, comprising the following modules.

[0013] The fusion module is configured to fuse the multi-modal sensor data and the space-time feature to determine an enhanced space-time fusion feature; the multi-modal sensor data comprises an equipment vibration signal, a current signal and a temperature signal.

[0014] The feature engineering processing module is configured to perform feature engineering processing on the enhanced space-time fusion feature to determine a processed space-time fusion feature.

[0015] The anomaly detection module is configured to perform anomaly detection on the processed space-time fusion feature based on an isolation forest model to determine an abnormal score.

[0016] The judgment module is configured to construct a dynamic early warning mechanism, determine whether the multi-modal sensor data is abnormal according to the abnormal score and the dynamic early warning mechanism, and determine a judgment result.

[0017] The visualization display module is configured to visually display the multi-modal sensor data and the judgment result through a visual architecture.

[0018] In a third aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the industrial equipment multi-modal data real-time anomaly detection method according to any one of the above.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the industrial equipment multi-modal data real-time anomaly detection method according to any one of the above.

[0020] In a fifth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the industrial equipment multi-modal data real-time anomaly detection method according to any one of the above.

[0021] According to the specific embodiments provided in the application, the application discloses the following technical effects: the application determines the enhanced spatio-temporal fusion features by fusing the multi-modal sensor data and the spatio-temporal features, realizes the deep fusion of multi-modal data, improves the spatio-temporal feature modeling capability by combining the spatio-temporal features, solves the defect of the traditional method of spatio-temporal feature fragmentation, and performs feature engineering processing on the enhanced spatio-temporal fusion features, breaks through the precision-speed contradiction by combining the isolation forest model, realizes anomaly detection, determines the anomaly score, and constructs a dynamic early warning mechanism to dynamically adjust the anomaly threshold, realizes dynamic adaptability, and finally accurately judges whether the sensor data is abnormal according to the anomaly score. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 The flow chart of the industrial equipment multi-modal data real-time anomaly detection method in an embodiment of the application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0025] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0026] The industrial equipment multi-modal data real-time anomaly detection method provided in the embodiments of the application is executed by a computer device, which can be executed by a terminal or a server alone or by a terminal and a server together. As shown in Figure 1 , the method comprises the following steps.

[0027] S1: fuse multi-modal sensor data and spatio-temporal features to determine enhanced spatio-temporal fusion features; the multi-modal sensor data includes device vibration signals, current signals and temperature signals.

[0028] S2: perform feature engineering processing on the enhanced spatio-temporal fusion features to determine processed spatio-temporal fusion features.

[0029] S3: Based on the isolated forest model, perform anomaly detection on the processed spatiotemporal fusion features and determine the anomaly score.

[0030] S4: Construct a dynamic early warning mechanism to determine whether the multimodal sensor data is abnormal based on the anomaly score and the dynamic early warning mechanism, and determine the judgment result.

[0031] S5: Visualize the multimodal sensor data and the judgment results using a visualization architecture.

[0032] In an exemplary embodiment, S1 can be replaced by the following steps.

[0033] S11: Perform timestamp normalization processing on the multimodal sensor data to determine the normalized multimodal sensor data.

[0034] S12: Perform missing value processing on the standardized multimodal sensor data to determine the processed multimodal sensor data.

[0035] S13: Based on the spatiotemporal attention architecture, the processed multimodal sensor data is fused with spatiotemporal features to construct enhanced spatiotemporal fusion features.

[0036] In practical applications, a triaxial vibration sensor is used to collect equipment vibration signals in real time, and simultaneously collect current and temperature signals to construct a time-series data stream.

[0037] A dual-path multi-head attention architecture is adopted for spatiotemporal feature fusion. The sliding window structure of the time series data stream provides the input matrix (time dimension × spatial dimension) for the dual-path attention. The dual-path multi-head attention architecture enhances the sensitivity to changes in device state by processing spatiotemporal features in parallel.

[0038] A multi-head self-attention mechanism is used to capture the temporal dependencies of vibration signals; in industrial vibration scenarios, spatiotemporal relationships are strongly coupled.

[0039] Time-series dependence: reflects the evolution of equipment status (e.g., increased wear leads to a gradual increase in vibration amplitude).

[0040] Spatial correlation: physical coupling of triaxial vibration signals (such as abnormal X-axis vibration causing Y / Z axis resonance).

[0041] Temporal dependence and spatial correlation interact: spatial anomalous patterns (such as uniaxial surges) may disrupt normal temporal patterns, and temporal cumulative effects (such as gradual temperature increases) may trigger spatial energy redistribution.

[0042] Example: A fault in the inner ring of a bearing can cause the vibration signal to exhibit triaxial synchronous impact (spatial correlation) at specific time intervals (time-dependent).

[0043] Residual connectivity and layer normalization enhance spatial features by preserving the original spatial features and stabilizing the spatial feature distribution, thereby outputting enhanced spatiotemporal fusion features.

[0044] The enhanced spatiotemporal fusion feature F_final is: F_final = LayerNorm(F_raw + α·TemporalAttn(F_raw) + β·SpatialAttn(F_raw)); where α and β are adaptive weight coefficients, with values ​​ranging from [0.3, 0.7], controlling the fusion ratio of spatiotemporal features; F_raw is the original feature matrix (dimension: time step × number of features), containing the original / normalized values ​​of the vibration triaxial, current, and temperature; TemporalAttn(F_raw): features after temporal attention processing, capturing temporal dependencies; SpatialAttn(F_raw): features after spatial attention processing, modeling the correlation between sensors; LayerNorm: layer normalization operation, stabilizing the feature distribution.

[0045] The enhanced spatiotemporal fusion features include: enhanced vibration triaxial features (reconstructed through spatiotemporal attention); and context-aware representation of current / temperature features (affected by operating conditions).

[0046] The enhanced spatiotemporal fusion features have the same feature dimensions as the original data (time step × 5 dimensions), but contain richer spatiotemporal correlation information.

[0047] In another exemplary embodiment, data preprocessing and spatiotemporal feature fusion specifically include the following steps.

[0048] Timestamp standardization: Consistently align multi-source sensor data to a 1Hz sampling frequency.

[0049] Missing value handling: Missing data points are filled in using linear interpolation.

[0050] Spatiotemporal attention architecture configuration: including temporal attention layer, spatial attention layer, and residual connection ratio coefficient.

[0051] Temporal attention layer: Uses a 4-head self-attention mechanism, with a query vector dimension of 64.

[0052] Spatial attention layer: A channel attention mechanism is used to establish a correlation of triaxial vibration signals.

[0053] The residual connection proportionality coefficients are α1=0.5 and β1=0.5.

[0054] In an exemplary embodiment, S2 can be replaced by the following steps.

[0055] S21: Extract the temporal statistical features of the enhanced spatiotemporal fusion features using a sliding window.

[0056] S22: Construct a spatiotemporal correlation feature map based on the attention weight matrix, and determine the attention features based on the spatiotemporal correlation feature map; the attention weight matrix includes a time dimension weight matrix and a spatial dimension weight matrix.

[0057] S23: Combine the temporal statistical features with the attention features to determine a hybrid feature vector.

[0058] S24: Perform feature standardization processing on the hybrid feature vector to determine the spatiotemporal fusion features after processing.

[0059] In practical applications, time-domain statistical characteristics include mean, standard deviation, RMS value, peak-to-peak value, and 95th percentile within the sliding window.

[0060] The spatiotemporal correlation feature map has a dimension of 64×64, and the temporal statistical features and attention features are concatenated into a 128-dimensional hybrid feature vector.

[0061] In practical applications, feature engineering specifically includes the following steps.

[0062] Temporal feature calculation window configuration: Sliding window length: 200 sample points (corresponding to 3.3 minutes of data); Window overlap rate: 50%.

[0063] Attention weight matrix generation: Time dimension weight matrix: calculates the attention distribution at each time step; Spatial dimension weight matrix: generates a heat map of the correlation between the three-axis sensors.

[0064] Feature standardization: The Z-score standardization method is adopted, with the formula: x'=(xμ) / σ; where μ is the feature mean, σ is the standard deviation, x is the multimodal sensor data, and x' is the standardized multimodal sensor data.

[0065] In one exemplary embodiment, S3 can be replaced by the following steps.

[0066] S31: Construct an isolated forest model and set the model parameters; the model parameters include the number of isolated trees, the maximum number of samples, and the anomaly score threshold.

[0067] S32: Based on the model parameters, perform feature standardization on the processed spatiotemporal fusion features, and determine the anomaly score by traversing multiple isolation trees in parallel through multi-threading.

[0068] In practical applications, anomaly detection based on the isolated forest model involves constructing an ensemble model containing 100 isolated trees (number of base learners n_estimators=100), with 256 sample points randomly sampled from each tree; and calculating anomaly scores in real time: score=2. -E(h(x)) / c(n) Where h(x) is the sample path length, c(n) is the average path length of the binary search tree, and E is the mathematical expectation.

[0069] In practical applications, the model parameter settings include: number of isolation trees: 100; maximum number of samples: 256; anomaly score threshold: initial value set to 0.7.

[0070] Based on the above model parameters, the real-time inference process includes: feature standardization: using online calculated feature mean and standard deviation; parallel tree traversal: achieving millisecond-level response through multi-threading; and anomaly score normalization: mapping the original score to the [0,1] interval.

[0071] In one exemplary embodiment, S4 can be replaced by the following steps.

[0072] S41: Initialize the anomaly threshold based on the 3σ principle of historical data.

[0073] S42: Based on the initialized outlier threshold, the outlier threshold is dynamically adjusted by combining the moving average and standard deviation of historical data sample windows to determine the adjusted dynamic threshold. In practical applications, dynamic adjustment is performed by combining the moving average and standard deviation of 200 sample windows.

[0074] S43: Construct a dynamic early warning mechanism based on the adjusted dynamic threshold; the dynamic early warning mechanism is a multi-level alarm triggering condition, which includes a warning, a yellow warning, and a red warning; the warning is when the abnormal score calculated in real time exceeds the adjusted dynamic threshold, the yellow warning is when a single abnormal score exceeds the adjusted dynamic threshold by 1.5 times, and the red warning is when three consecutive abnormal scores exceed the adjusted dynamic threshold by 2 times.

[0075] S44: Determine the judgment result based on the multi-level alarm triggering conditions; the judgment result includes whether the multi-modal sensor data is an abnormal signal or a normal signal.

[0076] In practical applications, the dynamic threshold adjustment method of S42 specifically includes: initializing the calculation of the abnormal threshold T0: T0 = μ_hist + 3σ_hist; where μ_hist is the mean of historical data and σ_hist is the standard deviation.

[0077] The online adjustment formula is: T_new=ρ·T_prev+(1-ρ)·(μ_window+2σ_window), where T_new is the adjusted dynamic threshold, ρ=0.2 is the forgetting factor, T_prev is the dynamic threshold after the previous adjustment, μ_window is the window mean, and σ_window is the window standard deviation.

[0078] Alarm suppression mechanism: A 3-second delay is applied to confirm momentary noise triggers.

[0079] The dynamic threshold mechanism combines historical benchmarks with real-time adjustments. It provides a stable benchmark by initializing abnormal thresholds to prevent false alarms during the cold start phase; online adjustments adapt to changes in operating conditions (such as sudden changes in equipment load) to avoid threshold rigidity; and a forgetting factor controls the adjustment range, with 0.2 indicating that historical information accounts for 20% and real-time data accounts for 80%.

[0080] In an exemplary embodiment, S5 can be replaced by the following steps.

[0081] The visualization architecture is a double-buffered architecture; the double-buffered architecture includes a front-end display buffer and a back-end processing buffer.

[0082] S51: Based on the aforementioned front-end display buffer, a circular queue is used to store the latest multimodal sensor data and judgment results.

[0083] S52: Based on the back-end processing buffer, a priority queue is used to manage the multimodal sensor data to be processed.

[0084] In practical applications, a dual-buffer architecture is established: a front-end display buffer that retains 200 of the latest samples for waveform drawing; a back-end processing buffer that stores 500 samples for batch calculation; and a parallel processing pipeline that achieves real-time refresh of 60 frames per second.

[0085] The dual-buffer architecture includes a front-end display buffer {data structure: circular queue (capacity 200 samples) function: stores the latest data for real-time waveform drawing, avoiding interface lag}.

[0086] Backend processing buffer {Data structure: priority queue (capacity 500 samples) Function: batch cache data for feature calculation and model inference}.

[0087] The dual-buffered decoupling of data acquisition, calculation, and display processes ensures stable operation of industrial equipment systems with a latency of 18ms.

[0088] The visualization architecture also includes a parallel processing pipeline, as detailed below.

[0089] Data acquisition thread: 1Hz fixed frequency sampling.

[0090] Computation thread: performs feature extraction and model inference asynchronously.

[0091] Rendering thread: Implements 50fps waveform drawing based on OpenGL.

[0092] In practical applications, the real-time performance indicators of the technical solution of this application include: processing latency: average 18ms (±3ms); detection accuracy: ≥98.6% (verified based on PHM2012 dataset); resource consumption: CPU utilization ≤15%; memory consumption ≤512MB; supported device types: CNC machine tools, centrifugal pumps and industrial robots.

[0093] The technical solution of this application is further illustrated below using actual multimodal sensor data from industrial equipment.

[0094] 1. Multimodal data acquisition and preprocessing are achieved using multimodal sensors from industrial equipment and data processing servers.

[0095] 1.1 The data acquisition module obtains device status information in real time through the following sensors.

[0096] Triaxial vibration sensor: acquires X / Y / Z axial vibration acceleration signals (range ±16g, sampling rate 1kHz).

[0097] Current sensor: measures motor drive current with an accuracy of ±0.5% of full scale (FS).

[0098] Temperature sensor: monitors bearing temperature with a range of 0-150℃ and a resolution of 0.1℃.

[0099] 1.2 The data processing server performs preprocessing operations.

[0100] a) Timestamp alignment: Synchronize multi-source data to a unified time base via the NTP protocol.

[0101] b) Missing value handling: Missing data points are filled in using cubic spline interpolation. The interpolation formula is: S(t)=a_i(t-t_i) 3 +b_i(t-t_i) 2 +c_i(t-t_i)+d_i; where S(t) is a piecewise cubic polynomial function used to construct a smooth interpolation curve between known data points; t_i is the timestamp of the known data point, and a_i,b_i,c_i,d_i are the piecewise polynomial coefficients.

[0102] Compared with linear interpolation, cubic spline interpolation improves the signal-to-noise ratio of the reconstructed signal by 12 dB, and the improved data quality increases the F1-score for anomaly detection by 6.3 percentage points.

[0103] c) Data normalization: The vibration signal is normalized by min-max, and the formula is: x_norm=(x-min) / (max-min); where x_norm is the normalized vibration signal, and min / max is the lower limit / upper limit of the sensor range.

[0104] Technical role: To ensure the spatiotemporal consistency of multi-source data and eliminate the impact of dimensional differences on subsequent processing.

[0105] Through data normalization processing, multi-device compatibility is achieved, supporting 5 types of industrial equipment and reducing expansion costs by 60%.

[0106] Dynamic range calibration: Automatically adjusts the min / max parameters according to the equipment type.

[0107] In cross-equipment testing of CNC machine tools and centrifugal pumps, the detection accuracy fluctuation was <2%.

[0108] Attention weight transfer learning: Pre-trained spatiotemporal attention models can be quickly adapted to new devices, reducing the amount of data required for fine-tuning by 80%.

[0109] 2. Spatiotemporal feature fusion is achieved using a feature fusion module.

[0110] 2.1 Temporal Attention Processing: Input: Time-series data with a 200-sample window (dimension 200×5, including triaxial vibration + current + temperature); Using a 4-head self-attention mechanism, calculate the temporal dimension association weight matrix W_t∈R. {200×200} .

[0111] Attention output: TemporalAttn = softmax((QK T ) / )·V; where Q,K,V are the query, key, and value matrices, and d_k=64 is the scaling factor.

[0112] 2.2 Spatial Attention Processing: Constructing the Sensor Spatial Correlation Matrix W_s∈R {5×5} The channel attention mechanism is used to calculate the weight coefficients of each sensor channel: w_i=τ(MLP(AvgPool(F_i))), where τ is the sigmoid function, MLP is a two-layer fully connected network, F_i represents the feature map of the i-th channel in the input feature tensor, and AvgPool is the average pooling operation.

[0113] 2.3 Feature Fusion: Perform residual connections and layer normalization: F_fused=LayerNorm(F_raw+0.5·TemporalAttn(F_raw)+0.5·SpatialAttn(F_raw)); The output dimension remains unchanged at 200×5.

[0114] Technical function: To capture long-term dependencies of device status and cross-sensor correlation characteristics.

[0115] The periodic impact characteristics of equipment vibration signals (such as the periodic impact of bearing failure) are accurately captured by the time attention module (W_t matrix).

[0116] The spatial attention module (W_s matrix) establishes cross-modal correlations between triaxial vibration and operating parameters, and can identify composite fault modes.

[0117] Experimental data: In bearing outer ring fault detection, this module increased recall from 86.2% to 94.7%.

[0118] 3. Use a feature extraction engine to implement hybrid feature engineering.

[0119] 3.1 Calculation of time-domain statistical characteristics.

[0120] |Feature Type|Calculation Formula|Window Parameters| |---------|---------|---------| |mean|μ=(1 / N)Σx_i|N=200| |Standard deviation|σ=√[Σ(x_i-μ)] 2 / (N-1)] | | |Peak-to-peak value|P2P = max(x_i) min(x_i) | | |RMS value| RMS = √[(1 / N)Σx_i 2 ] | | Where x_i is the sensor data value of x at each time i.

[0121] 3.2 Spatiotemporal attention feature extraction.

[0122] Temporal attention features: Extract the principal components of the W_t matrix (the vector corresponding to the first 10 eigenvalues).

[0123] Spatial attention features: Take the upper triangular elements of the W_s matrix (a total of 15 parameters).

[0124] 3.3 Feature splicing.

[0125] The 15-dimensional statistical features and the 25-dimensional attention features are concatenated to form a 40-dimensional hybrid feature vector.

[0126] Technical function: to integrate the shallow physical characteristics and deep correlation characteristics of equipment status.

[0127] Traditional statistical features (mean / peak value, etc.) retain physical interpretability, while attention features (principal components + spatial correlation) enhance the ability to express nonlinear relationships. The feature dimension is optimized from 15 dimensions in traditional methods to 40 dimensions, and the information entropy is increased by 37%.

[0128] 4. Anomaly detection and dynamic threshold adjustment (Execution subject: Isolation Forest Detection Engine).

[0129] 4.1 Construction of the Isolation Forest Model

[0130] The number of trees, n_estimators, is 100.

[0131] The maximum number of samples per tree is max_samples=256.

[0132] The feature subset size max_features = √40 ≈ 6.

[0133] 4.2 Anomaly score calculation: score=2 -E(h(x)) / c(n) Where, h(x): path length of sample x in the tree (unknown); E(h(x)): average path length of all trees (unknown); c(n)=2H(n-1)-2(n-1) / n: average path length of binary search tree (known, n is the number of samples); H(k): harmonic number H(k)=ln(k)+0.5772.

[0134] 4.3 Dynamic threshold update: Initial threshold T0 = Q3 + 1.5IQR (Q3 is the third quartile of historical data, and IQR is the interquartile range).

[0135] The online adjustment formula is: T_new=0.8·T_prev+0.2·(μ_w+2σ_w); where μ_w and σ_w are the mean and standard deviation of the most recent 200 abnormal scores.

[0136] Technical function: To enable real-time quantitative assessment of equipment status anomalies and adaptive alarms.

[0137] Isolation Forest Model: Path length calculation optimization: adopting a pre-generated tree structure reduces traversal time by 75%; Parallel tree inference: utilizing the SIMD instruction set for acceleration, the single sample processing time is reduced from 3.2μs to 0.9μs.

[0138] The dynamic threshold mechanism can significantly reduce the false alarm rate to <0.2%, while the industry average is 1.5-3%.

[0139] Introducing a forgetting factor α2=0.8 effectively suppresses false alarms caused by fluctuations in operating conditions.

[0140] The three-level alarm strategy filters out instantaneous interference, reducing the number of false alarms by 83%.

[0141] 5. Real-time visualization and alarms are provided using a human-computer interaction terminal.

[0142] 5.1 The double buffer architecture is as follows.

[0143] | Buffer type | Data structure | Capacity | Update strategy | |-----------|---------|------|---------| | Display Buffer | Circular Queue | 200 Samples | FIFO Replacement | | Processing Buffer | Priority Queue | 500 Samples | Timestamp Sort | 5.2 Parallel Processing Pipeline.

[0144] Data acquisition thread: 1kHz sampling, 10ms periodic writing to shared memory.

[0145] Feature calculation thread: processes one data window every 200ms.

[0146] Rendering thread: Implements 60fps waveform refresh based on OpenGL.

[0147] 5.3 Multi-level alarm triggering conditions.

[0148] Alarm Level | Triggering Condition | Response Action | |---------|---------|---------| Level 1 Warning | Score > T | Yellow indicator flashing on screen | Level 2 Alarm | Score > 1.5T for 3 consecutive times | Audible and visual alarm + log recording | | Level 3 Emergency Stop | Score Instantaneous > 3T | Triggers Emergency Equipment Shutdown | Where T is the dynamically adjusted real-time alarm threshold.

[0149] Technical role: Provides intuitive equipment status monitoring and tiered emergency response capabilities.

[0150] Breakthrough in real-time performance (latency <20ms, reaching industrial-grade real-time standards); Dual-buffered parallel architecture: decoupling of display buffer and processing buffer to avoid I / O blocking.

[0151] Experimental comparison: Compared to the single-buffered scheme, the 99th percentile latency decreased from 45ms to 21ms.

[0152] The test results on the PHM2012 bearing dataset are as follows.

[0153] Detection accuracy: 98.72% (F1-score).

[0154] Average response latency: 18.3 ± 2.1 ms.

[0155] False alarm rate: <0.2% (after 72 hours of continuous testing).

[0156] Memory usage: ≤427MB (processes 5000 samples / second).

[0157] Based on the same inventive concept, this application also provides an industrial equipment multimodal data real-time anomaly detection device for implementing the above-mentioned industrial equipment multimodal data real-time anomaly detection method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the industrial equipment multimodal data real-time anomaly detection device provided below can be found in the limitations of the industrial equipment multimodal data real-time anomaly detection method described above, and will not be repeated here.

[0158] In one exemplary embodiment, a real-time anomaly detection device for multimodal data of industrial equipment is provided, comprising the following modules.

[0159] The fusion module is used to fuse multimodal sensor data and spatiotemporal features to determine the enhanced spatiotemporal fusion features; the multimodal sensor data includes equipment vibration signals, current signals and temperature signals.

[0160] The feature engineering processing module is used to perform feature engineering processing on the enhanced spatiotemporal fusion features and determine the processed spatiotemporal fusion features.

[0161] An anomaly detection module is used to perform anomaly detection on the processed spatiotemporal fusion features based on the isolated forest model and determine anomaly scores.

[0162] The judgment module is used to construct a dynamic early warning mechanism, determine whether the multimodal sensor data is abnormal based on the anomaly score and the dynamic early warning mechanism, and determine the judgment result.

[0163] The visualization module is used to visualize the multimodal sensor data and the judgment results through a visualization architecture.

[0164] This application, based on spatiotemporal attention and the isolated forest model, overcomes three major technical bottlenecks.

[0165] First, a dual-path multi-head attention mechanism was designed, which solves the problem of the separation of spatiotemporal features in traditional methods by using a time attention module (to capture the periodic impact of vibration signals) and a spatial attention module (to establish a correlation between three-axis sensors).

[0166] Secondly, an isolated forest model was constructed, and a pre-generated tree structure and SIMD instruction set were used to accelerate the process, reducing the single-sample inference time from 3.2μs to 0.9μs. With a latency of 18ms, a detection accuracy of 98.7% was achieved, solving the industry problem of "accuracy and speed being mutually exclusive".

[0167] Finally, an innovative dynamic threshold mechanism was developed, which combines sliding window statistics with a forgetting factor (α2=0.8) to reduce the false alarm rate from 5.2% to 0.18% under fluctuating operating conditions, achieving aerospace-grade reliability standards.

[0168] The breakthrough of this application lies in achieving industrial-grade real-time accurate detection for the first time. The spatiotemporal attention weight matrix (W_t / W_s) quantifies the spatiotemporal coupling relationship of sensor signals, thereby improving the accuracy of composite fault detection to 98.72%. The lightweight hybrid architecture design (model memory footprint of only 427MB) supports deployment on edge devices such as Huawei Atlas 500, reducing the computing power requirement by 85% compared to traditional deep learning solutions.

[0169] This application provides a new generation of infrastructure for intelligent manufacturing. By addressing long-standing pain points such as complex operating condition modeling, real-time constraints, and cross-device adaptation, it not only drives the upgrade of predictive maintenance technology but also significantly lowers the threshold for digital transformation in the manufacturing industry. In the future, as the demand for intelligent industrial equipment continues to grow, the high precision, low latency, and strong adaptability of this technology will reshape the industry ecosystem and inject core driving force into high-quality industrial development.

[0170] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores real-time anomaly detection data for multimodal data of industrial equipment. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for real-time anomaly detection of multimodal data of industrial equipment.

[0171] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0172] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the methods described above.

[0173] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0174] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0175] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0176] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for real-time anomaly detection of multimodal data in industrial equipment, characterized in that, include: By fusing multimodal sensor data and spatiotemporal features, enhanced spatiotemporal fusion features are determined; the multimodal sensor data includes equipment vibration signals, current signals, and temperature signals. The enhanced spatiotemporal fusion features are subjected to feature engineering processing to determine the processed spatiotemporal fusion features; Based on the isolated forest model, anomaly detection is performed on the processed spatiotemporal fusion features to determine anomaly scores; A dynamic early warning mechanism is constructed to determine whether the multimodal sensor data is abnormal based on the anomaly score and the dynamic early warning mechanism, and to determine the judgment result. The multimodal sensor data and the judgment results are visualized using a visualization architecture.

2. The method for real-time anomaly detection of multimodal data in industrial equipment according to claim 1, characterized in that, By fusing multimodal sensor data and spatiotemporal features, a spatiotemporal fusion feature is constructed, specifically including: The multimodal sensor data is timestamped and standardized to determine the standardized multimodal sensor data. Missing values ​​are processed on the standardized multimodal sensor data to determine the processed multimodal sensor data; Based on the spatiotemporal attention architecture, the processed multimodal sensor data is fused with spatiotemporal features to construct enhanced spatiotemporal fusion features.

3. The method for real-time anomaly detection of multimodal data in industrial equipment according to claim 1, characterized in that, The enhanced spatiotemporal fusion features are subjected to feature engineering processing to determine the processed spatiotemporal fusion features, specifically including: Temporal statistical features of the enhanced spatiotemporal fusion features are extracted using a sliding window. Based on the attention weight matrix, a spatiotemporal correlation feature map is constructed, and attention features are determined based on the spatiotemporal correlation feature map; the attention weight matrix includes a time dimension weight matrix and a spatial dimension weight matrix; By fusing the temporal statistical features and the attention features, a hybrid feature vector is determined; The hybrid feature vector is subjected to feature standardization processing to determine the spatiotemporal fusion features after processing.

4. The method for real-time anomaly detection of multimodal data in industrial equipment according to claim 1, characterized in that, Based on the isolated forest model, anomaly detection is performed on the processed spatiotemporal fusion features to determine anomaly scores, specifically including: Construct an isolated forest model and set model parameters; the model parameters include the number of isolated trees, the maximum number of samples, and the anomaly score threshold; Based on the model parameters, the processed spatiotemporal fusion features are standardized, and anomaly scores are determined by traversing multiple isolation trees in parallel using multi-threading.

5. The method for real-time anomaly detection of multimodal data in industrial equipment according to claim 1, characterized in that, A dynamic early warning mechanism is constructed to determine whether the multimodal sensor data is abnormal based on the anomaly score and the dynamic early warning mechanism, and to determine the judgment result, specifically including: Anomaly thresholds are initialized based on the 3σ principle using historical data; σ represents the standard deviation. Based on the initialized anomaly threshold, the anomaly threshold is dynamically adjusted by combining the moving average and standard deviation of historical data sample windows to determine the adjusted dynamic threshold; A dynamic early warning mechanism is constructed based on the adjusted dynamic threshold. The dynamic early warning mechanism is a multi-level alarm triggering condition, which includes a warning, a yellow warning, and a red warning. The warning is when the abnormal score calculated in real time exceeds the adjusted dynamic threshold. The yellow warning is when a single abnormal score exceeds the adjusted dynamic threshold by 1.5 times. The red warning is when three consecutive abnormal scores exceed the adjusted dynamic threshold by 2 times. The judgment result is determined based on the multi-level alarm triggering conditions; the judgment result includes whether the multimodal sensor data is an abnormal signal or a normal signal.

6. The method for real-time anomaly detection of multimodal data in industrial equipment according to claim 1, characterized in that, The multimodal sensor data and the judgment results are visualized through a visualization architecture, specifically including: The visualization architecture is a double-buffered architecture; the double-buffered architecture includes a front-end display buffer and a back-end processing buffer; Based on the aforementioned front-end display buffer, a circular queue is used to store the latest multimodal sensor data and judgment results; Based on the backend processing buffer, a priority queue is used to manage the multimodal sensor data to be processed.

7. A real-time anomaly detection device for multimodal data of industrial equipment, characterized in that, include: The fusion module is used to fuse multimodal sensor data and spatiotemporal features to determine the enhanced spatiotemporal fusion features; the multimodal sensor data includes equipment vibration signals, current signals, and temperature signals; The feature engineering processing module is used to perform feature engineering processing on the enhanced spatiotemporal fusion features and determine the processed spatiotemporal fusion features. An anomaly detection module is used to perform anomaly detection on the processed spatiotemporal fusion features based on the isolated forest model and determine anomaly scores; The judgment module is used to construct a dynamic early warning mechanism, determine whether the multimodal sensor data is abnormal based on the anomaly score and the dynamic early warning mechanism, and determine the judgment result. The visualization module is used to visualize the multimodal sensor data and the judgment results through a visualization architecture.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the real-time anomaly detection method for multimodal data of industrial equipment according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the real-time anomaly detection method for multimodal data of industrial equipment as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the real-time anomaly detection method for multimodal data of industrial equipment as described in any one of claims 1-6.

Images