Multimodal time series anomaly analysis method, apparatus, device, and medium

Abstract

The application relates to the technical field of data analysis, and discloses a multimodal time sequence anomaly analysis method, device, equipment and medium, which comprises the following steps: collecting visual data, audio data and process text data, respectively pre-processing and standardizing different types of data, constructing a timestamp-aligned multimodal data set, then extracting atlas structure features, time-frequency dynamic features and semantic vector features, and using a cross-modal attention mechanism to fuse the features to generate a fusion feature vector, finally performing analysis and processing based on the fusion feature vector and outputting an analysis result. By constructing a time-consistent multimodal data set, extracting structured features of various modes, and introducing a cross-modal attention mechanism to realize deep fusion at the feature level, the application solves the problems of single information utilization and weak feature correlation of existing detection methods, thereby improving the comprehensiveness of defect detection and the accuracy of fault diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a multi-modal time series anomaly analysis method, device, equipment and storage medium. BACKGROUND

[0002] Under the trend of intelligent transformation of automobile manufacturing, stable operation and efficient fault diagnosis capability of production line equipment have become the key to guarantee the quality and capacity of the whole vehicle. However, the existing technology still has many technical bottlenecks that are difficult to overcome in the face of complex state and multi-source data fusion of production line equipment, especially in the accuracy, response efficiency and feature extraction capability of fault detection.

[0003] The current common fault diagnosis methods mainly include traditional signal processing methods, shallow machine learning models and deep learning methods. Traditional methods such as frequency spectrum analysis and wavelet transform, although have certain effect in specific periodic or steady-state signal analysis, but in the face of non-stationary, multi-source coupled signals commonly existing in automobile production line, their feature extraction capability is insufficient, easy to be affected by noise interference, difficult to identify subtle or atypical fault types, and the overall robustness is not strong.

[0004] Shallow machine learning models, such as support vector machine (SVM) and decision tree, have improved the level of diagnosis automation in some scenarios, but they are highly dependent on manual feature design. This feature engineering is difficult to systematically cover the internal structure of high-dimensional and strongly nonlinear data, especially in the case of limited sample size and diversified fault types, the model is prone to overfitting and poor generalization, resulting in high false positive rate and false negative rate, which is difficult to adapt to dynamic changes under complex working conditions.

[0005] In recent years, deep learning methods have gradually expanded in the field of intelligent manufacturing, and some research attempts to use convolutional neural networks (CNN) and recurrent neural networks (RNN) to realize end-to-end fault diagnosis. However, in the face of multi-stage, multi-time sequence feature fusion scenarios such as automobile production lines, existing general deep models often use large-scale network architecture to improve performance, but their structure lacks pertinence, making it difficult to fully model the interaction between multi-dimensional heterogeneous data in the production line, and also unable to effectively capture abnormal performance caused by complex topological structure or time delay. In addition, the high computational overhead of large models also makes them face low inference efficiency and low resource utilization rate when deployed in real-time production monitoring systems.

[0006] In summary, the existing technology is still difficult to effectively support deep fault pattern modeling and cross-modal feature fusion of multi-source, dynamic and non-stationary data, and it is also difficult to achieve accurate identification of sudden or weak anomalies, so optimization and breakthroughs are needed in model structure design, feature learning mechanism and diagnosis reasoning strategy. SUMMARY

[0007] The main purpose of the present application is to provide a multi-modal time series anomaly analysis method, device, equipment and storage medium, aiming at solving the technical problems that the prior art lacks the ability of cooperative modeling of deep features in multi-source non-stationary time series data, and is difficult to realize the fusion reasoning of graph structure, spectral features and inter-channel correlation information.

[0008] To achieve the above-mentioned purpose, the present application provides a multi-modal time series anomaly analysis method, comprising:

[0009] Collecting multi-source sensing signals, and performing preprocessing operation on the multi-source sensing signals to generate preprocessed multi-channel time series data;

[0010] Mapping the preprocessed multi-channel time series data into graph representation, the graph representation being used to reflect the correlation information between monitoring objects;

[0011] Performing multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features;

[0012] Applying channel cooperative attention processing to selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel cooperative attention features;

[0013] Comprehensively reasoning the graph representation, the multi-spectrum fusion features and the channel cooperative attention features through a time series analysis model to generate an analysis result.

[0014] Further, to achieve the above-mentioned purpose, the present application provides a multi-modal time series anomaly analysis device, comprising:

[0015] A data preprocessing module for collecting multi-source sensing signals and performing preprocessing operation on the multi-source sensing signals to generate preprocessed multi-channel time series data;

[0016] A graph construction module for mapping the preprocessed multi-channel time series data into graph representation, the graph representation being used to reflect the correlation information between monitoring objects;

[0017] A spectrum fusion module for performing multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features;

[0018] A channel attention module for applying channel cooperative attention processing to selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel cooperative attention features;

[0019] A time sequence reasoning module is configured to comprehensively reason the graph representation, the multi-spectrum fusion feature and the channel collaborative attention feature by a time sequence analysis model to generate an analysis result.

[0020] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor and a multi-modal time sequence anomaly analysis program stored in the memory and executable on the processor, and the multi-modal time sequence anomaly analysis program realizes the steps of the multi-modal time sequence anomaly analysis method when executed by the processor.

[0021] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a multi-modal time sequence anomaly analysis program, and the multi-modal time sequence anomaly analysis program realizes the steps of the multi-modal time sequence anomaly analysis method when executed by a processor.

[0022] Beneficial effects: The present application relates to the technical field of data analysis, and discloses a multi-modal time sequence anomaly analysis method, device, equipment and medium, which comprises the following steps: collecting multi-source sensing signals and preprocessing to generate multi-channel time sequence data, constructing a graph representation representing the association relationship between devices and sensors, extracting frequency domain features and fusing to generate multi-spectrum fusion features, applying a channel collaborative attention mechanism to specific channel feature data to obtain channel attention features, and jointly reasoning the graph representation, the spectrum fusion features and the channel attention features by a time sequence analysis model to generate an analysis result for fault identification. The present application fuses graph structure features, frequency domain energy features and channel interaction attention features, introduces a time sequence modeling mechanism to realize dynamic joint reasoning of multi-modal features, effectively improves the identification accuracy and response efficiency of line equipment faults under multi-source heterogeneous and strong noise interference, and has strong complex working condition adaptability and model reasoning robustness. BRIEF DESCRIPTION OF DRAWINGS

[0023] The present application will be further described below in combination with the drawings and embodiments, and the drawings are as follows:

[0024] Figure 1 An application environment schematic diagram of the multi-modal time sequence anomaly analysis method in an embodiment of the present application;

[0025] Figure 2 A flowchart of an embodiment of the multi-modal time sequence anomaly analysis method of the present application;

[0026] Figure 3 A functional module schematic diagram of a preferred embodiment of the multi-modal time sequence anomaly analysis device of the present application;

[0027] Figure 4 A structure schematic diagram of a computer device in an embodiment of the present application;

[0028] Figure 5 Fig. 2 is another structural schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0030] The multi-modal time series anomaly analysis method provided by the embodiments of the present application can be applied in an application environment as shown in Fig. 1. Figure 1 In the application environment, a user terminal communicates with a server through a network. The server can collect multi-source sensing signals through the user terminal and pre-process the multi-source sensing signals to generate multi-channel time series data, construct a graph representation representing the association relationship between devices and sensors, extract frequency domain features and fuse to generate multi-spectrum fusion features, apply a channel collaborative attention mechanism to specific channel feature data to obtain channel attention features, and perform joint reasoning on the graph representation, the spectrum fusion features and the channel attention features through a time series analysis model to generate an analysis result for fault identification. The present application fuses graph structure features, frequency domain energy features and channel interaction attention features, introduces a time series modeling mechanism to realize dynamic joint reasoning of multi-modal features, effectively improves the identification accuracy and response efficiency of line equipment faults under multi-source heterogeneous and strong noise interference, and has strong complex working condition adaptability and model reasoning robustness. The user terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0031] Please refer to Figure 2 , Figure 2 Fig. 3 is a flowchart of a multi-modal time series anomaly analysis method according to an embodiment of the present application. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0032] As shown in Fig. 4, the multi-modal time series anomaly analysis method provided by the present application includes the following steps: Figure 2 S10, collect multi-source sensing signals and pre-process the multi-source sensing signals to generate pre-processed multi-channel time series data;

[0033]

[0034] ​In this embodiment, the operation of collecting multi-source sensing signals is based on the goal of comprehensively covering and accurately quantifying information such as equipment state, working condition, and structural response in different links of the production line. The types of sensors involved include temperature sensors, vibration sensors, torque sensors, current sensors, and acoustic sensors. Among them, temperature sensors are mainly used to obtain the thermal state changes of equipment components in high-heat processes such as welding and painting. Vibration sensors are installed on rotating machinery and stamping structures to capture weak structural imbalance or collision characteristics. Torque sensors are connected to driving shafts or couplings to monitor real-time load changes of machinery. Current sensors are embedded in control loops to obtain dynamic current input signals of actuators or drive motors. Acoustic sensors are placed in key positions sensitive to abnormal impact and abnormal noise to assist in identifying low-frequency structural fatigue or high-frequency sudden abnormal events. The collection of these signals is usually transmitted synchronously to edge processing nodes or host computer platforms through industrial Ethernet, fieldbus systems, or wireless acquisition modules. The installation angle, sensitivity parameters, and sampling frequency of each type of sensor should be configured according to the device type and physical scene. The sampling frequency needs to cover more than twice the upper limit of the signal frequency to ensure the integrity of subsequent time domain, frequency domain, and time-frequency domain analysis.

[0035] After the synchronous acquisition of multi-source signals, in order to ensure the consistency and feature alignment accuracy of subsequent analysis, the data collected by heterogeneous sensors needs to be unified in format. The data format unification process includes converting various types of raw sampling data from different protocols (such as MODBUS, CANopen, OPC UA, etc.) to standardized structured time series, and aligning all channels using a unified timestamp mechanism. Time stamp alignment can be achieved by using the NTP time synchronization protocol in industrial networks or by setting a shared trigger signal. Due to the presence of periodic high-frequency interference, low-amplitude noise, or system drift during the acquisition process, multi-dimensional noise reduction operations must be performed. Difference filters are used to remove non-structural disturbances caused by low-frequency trend items or equipment motion. Kalman filtering is suitable for estimating the optimal state between system state and noise, forming a recursive structure of state prediction and observation correction to suppress random noise in the signal and enhance the response capability to sudden changes.

[0036] After noise suppression, in order to further unify the data time window and frequency characteristics, a time alignment mechanism based on global or local trigger conditions is performed. In industrial scenarios, different sensors are affected by communication cycles, trigger logic or control system response differences, which may cause timestamp drift or delay. Therefore, the acquisition data time can be uniformly corrected through a synchronous trigger signal or a self-checking mechanism. After time alignment, in order to avoid structural missing of single-channel data caused by signal interruption or communication packet loss, an interpolation repair algorithm is needed to complete the data. Linear interpolation is suitable for short-time small-amplitude fluctuation signal compensation, and spline interpolation is suitable for data sequence containing change trend or multi-order smoothing requirement, and the interpolation method is flexibly selected according to the motion characteristics of the device and the complexity of the signal change.

[0037] Finally, after the data of each channel is cleaned, compensated and synchronized, the contribution of each channel is evaluated according to the real-time signal-to-noise ratio and numerical stability (such as standard deviation and coefficient of variation in the sliding window) of each signal, and this is used as a weighted parameter input into the fusion model. The fusion process uses an adaptive weighting strategy to form a weight vector through dynamic evaluation of channel quality, performs channel-by-channel weighted summation, and synthesizes a unified scale and dimension multi-channel time series signal set to form the final basic data structure that can be used for time series modeling, mapping and feature extraction.

[0038] The multi-axis vibration sensor can be arranged in the hinged structure between the stamping arm and the fixed support, and the sampling frequency is set to 10 kHz to capture short-period impact response. The torque sensor is integrated into the middle bearing of the robot arm, and the frequency is configured to 1 kHz to balance real-time performance and slow-changing trend. The acoustic sensor can use a directional microphone pointing to a specific impact area and be placed above the stamping die. The signal is transmitted through a wireless acquisition module. The data access module converts the sensor output signal into MQTT or JSON format through an IO card or integrated conversion module supporting multiple industrial protocols for buffering processing, and then pushes it to the edge server in batches according to the fixed sliding window. The three-point central difference formula is used in the difference filter, and the Kalman filter uses a first-order state equation model and sets the system covariance matrix for parameter updating. The reference control signal is used as the synchronization reference for the time alignment operation, and when the sampling point offset of any signal relative to the reference signal exceeds the preset threshold, the time correction module is triggered to adjust the delay. The interpolation operation uses linear or cubic spline function to construct compensation points based on the data density in the current window, and the adaptive fusion weight is automatically generated using the signal-to-noise ratio and stability factor according to the weighting function, and it is updated every 100 window periods.

[0039] Example description: In a certain vehicle assembly workshop, a set of welding robot system is deployed, each robot body is installed with a set of six-axis vibration sensor for monitoring the micro-vibration changes in the mechanical arm action process, at the same time, the current sensor is configured in the motor control unit to collect the driving output state, the system is also equipped with two sets of acoustic sensors for capturing abnormal arc sound. The data is accessed to the centralized control node through the CAN bus, and then received and uniformly time-stamped by the edge computing module. Due to the non-synchronous update problem of some sensors, the system performs dynamic time alignment on the channel data by comparing the main control beat signal. After detecting the breakpoint in the current channel for a period of time, the system automatically calls the cubic spline interpolation function three times to complete the signal reconstruction, and then calculates the real-time weight according to the standard deviation of each channel in the previous period of time, and performs adaptive weighted fusion to form a unified multi-channel feature input stream, providing stable input for the subsequent modeling and anomaly detection module.

[0040] The embodiment introduces signal format standardization, multi-stage noise reduction, time synchronization and interpolation repair mechanism in the preprocessing stage, cooperates with the dynamic fusion strategy driven by signal-to-noise ratio and stability, realizes the unified modeling input of different sources and different dimension sensor signals, reduces the redundancy and offset error between heterogeneous sensor signals, and significantly improves the feature consistency and modeling efficiency of the subsequent fault recognition algorithm. The mechanism effectively solves the misjudgment or missed detection problem caused by data acquisition accuracy fluctuation, communication protocol difference and time asynchrony, and improves the robustness and response ability of the fault diagnosis system under complex working conditions.

[0041] S20, mapping the preprocessed multi-channel time series data into graph representation, the graph representation is used to reflect the association information between monitoring objects;

[0042] In the embodiment, the operation of mapping the preprocessed multi-channel time series data into graph representation is to reveal the potential association structure between monitoring objects in spatial deployment, functional linkage or state propagation. In the implementation process, it is necessary to first construct the data structure framework of the graph, and set the node set and edge set of the graph. The node set includes two basic units: one is the actual existing equipment unit on the production line, such as stamping machine, welding arm, coating nozzle, assembly robot, etc., each equipment unit constitutes an entity node in the graph structure with a unique identifier; the second is the sensor layout point, such as temperature sensor installation site, acoustic pickup site, vibration monitoring point, etc. These monitoring points are associated with specific device positions and exist in the form of virtual observation nodes in the graph.

[0043] After the node definition, the edge set is defined according to the physical installation relationship between devices, the process sequence and the actual data coordination fluctuation trend. The physical adjacent edge represents the actual adjacency between devices in space, for example, two assembly robots are connected through a conveyor belt; the process edge reflects the upstream and downstream dependence on the material flow, control signal driving or energy link, for example, the devices after stamping need to be connected by a directed edge to the welding process. In addition, the high correlation between sensor signals can be combined to dynamically add edges based on data correlation, such as the similarity between two sensor signals through the Pearson correlation coefficient or mutual information metric, if it exceeds the threshold, a temporary edge is added to the graph.

[0044] The defined edge set can be used to construct a weighted adjacency matrix, and the weight value of each edge can be derived from the influence coefficient between devices, signal coordination or expert experience parameters. Among them, the influence coefficient can be set by referring to fault propagation path, control response time delay, etc., and the coordination can be dynamically updated by correlation in the data window. After the adjacency matrix is established, the weighted Laplacian matrix is further calculated, and the adjustment parameter and the high frequency suppression coefficient are introduced to adjust its structure, so that the expression ability of the graph structure in the high frequency abnormal propagation or low frequency stable characteristics can be enhanced in the subsequent graph neural modeling process. Finally, the feature segment of each sensor in the preprocessed time series data corresponds to the node in the graph, and through the time window slicing operation, the multi-dimensional observation value in this time period is mapped to the attribute vector of the node. The node attribute constitutes the feature input matrix of the entire graph, which together with the adjacency matrix and the Laplacian matrix constitutes a complete graph structure representation.

[0045] The control unit, spraying unit and monitoring point of the welding zone device can be respectively included in the graph structure as device nodes and observation nodes, the sequence relationship between welding and spraying is defined as a directed process edge, and the edge weight is set to 0.8. The physical adjacent but non-process sequence devices are set as undirected adjacent edges, and the edge weight is 0.5. For the two groups of temperature sensors, if the Pearson correlation coefficient of their signals in the last 20 time windows exceeds 0.9, a data correlation edge is generated, and the edge weight is set after normalization according to the average mutual information. The graph structure updates the related edge set every 100 sampling points. When calculating the weighted Laplacian matrix, the adjustment parameter is set to 1.0 or 0.4 according to the heat conduction characteristics of adjacent devices, and the high frequency suppression coefficient is adjusted according to the high frequency energy density proportion in the historical abnormal events, if the high frequency proportion is higher than 50%, the suppression coefficient is adjusted from 0.8 to 1.2, to improve the reflection sensitivity to sudden faults. Finally, the five-dimensional features (such as mean, variance, maximum, rising edge density, etc.) collected in the current window of each node are mapped to the node feature vector, and the graph representation construction is completed.

[0046] Example description: In the monitoring system of a certain vehicle body welding area, the graph structure contains 12 robot control nodes and 48 sensor nodes. The system first establishes directed edges between the 12 control nodes according to the process sequence to form a flowchart structure; then establishes adjacency edges according to the equipment space position drawing to form a physical adjacency subgraph; and then constructs correlation edges based on sensor data analysis within three consecutive hours of operation. The graph structure is stored in the edge node in the form of a sparse matrix, and the structure and node features are updated every 20 seconds. The system identifies through the graph structure that certain low-heat efficiency sections between welding units have relevance, providing key structure input for subsequent early warning reasoning.

[0047] By introducing process dependency, physical layout information and data-driven correlation edges into the graph structure, and combining adaptive edge weight adjustment and node feature encoding methods, the embodiment can describe the multiple coupling relationships between monitoring objects on the production line from the structure, not only improving the structural modeling capability of potential fault propagation paths, but also providing more context information constrained feature expressions for subsequent graph neural network based modeling processes. Compared with the traditional independent channel processing method, the graph expression is more easily to capture abnormal correlation features across devices and time periods, and improves the reasoning ability and spatial generalization performance of modeling.

[0048] S30, multi-spectrum information extraction and fusion processing is performed on the pre-processed multi-channel time series data to generate multi-spectrum fusion features;

[0049] In this embodiment, the core purpose of multi-spectrum information extraction and fusion processing for pre-processed multi-channel time series data is to capture potential dynamic features from different frequency dimensions and enhance the perception ability of multi-type fault signals. In the implementation process, the frequency response characteristics and data representation forms of different sensor channels need to be first clarified. For example, structural vibration signals usually have strong periodicity and are significant in the medium and high frequency bands, while acoustic signals may contain sudden high frequency spikes, non-stable amplitude modulation and other phenomena. Therefore, in order to meet the modeling needs of these signal performance differences, frequency spectrum conversion operation needs to be performed on the original signals.

[0050] In the initial stage, discrete Fourier transform is used to transform the time series signal, mapping the sensor signal in each time window from the time domain to the frequency domain to obtain its frequency domain amplitude spectrum. The Fourier transform result only retains the amplitude spectrum in order to enhance the expression of the spectrum information directly related to the energy distribution and avoid the complexity brought by the phase in the modeling process when it does not participate in the modulation processing. After the amplitude spectrum is generated, the spectrum band is divided into low frequency band, medium frequency band and high frequency band according to the characteristic frequency range corresponding to different fault types. For example, the low frequency can cover the 0-100Hz interval, reflecting structural looseness or resonance problems; the medium frequency band (100-1kHz) reflects mechanical contact abnormalities; and the high frequency band (above 1kHz) can correspond to electrical interference and fracture events.

[0051] To suppress background noise interference and enhance the expression of key feature regions, a frequency band mask function is designed. The function is set according to the energy baseline of each frequency band during normal operation. The mask applies weight promotion to frequency bands with significantly deviated energy and weight suppression to redundant frequency bands. After the mask function acts on the corresponding frequency band, the weighted low-frequency band feature spectrum, the medium-frequency band feature spectrum, and the high-frequency band feature spectrum are obtained respectively. Then, the three types of spectral data are cross-band spliced to form a multi-dimensional spectral feature matrix with a unified structure. In the splicing structure, different frequency band channels can be divided by dimension or expanded on the time axis, so that the model can use spatial convolution or attention mechanism to perform feature selection subsequently.

[0052] To further improve the fusion effect of different frequency band features, an attention network is introduced to dynamically weight the spliced spectral feature matrix. The network learns the importance weight of each frequency band under different inputs, dynamically adjusts the channel response, and finally outputs the representation result as the spectral fusion feature. The attention mechanism can include self-attention structure, additive attention, channel attention, etc., to ensure the response adaptability and robustness in information expression of different frequency ranges.

[0053] The data of the vibration channel and the acoustic channel can be intercepted respectively with a sliding window of 0.5 seconds, and each piece of data has a length of 4096 sampling points. Fast Fourier transform is performed on the window data respectively, and the amplitude spectrum of 0-8 kHz is retained. The frequency band is divided into three intervals of 0-100 Hz, 100-2 kHz, and 2-8 kHz. The dynamic baseline of each interval is calculated according to the historical average energy in the window, and the mask weight adjustment is applied to the energy deviated part according to the 3σ criterion. For example, if the low-frequency band abnormally rises by more than the standard deviation, its mask weight is increased to 1.5. The spectral data after the mask are normalized before matrix splicing to ensure the scale consistency of different frequency bands. The channel response of the spliced spectral feature matrix is weighted by a three-layer convolution attention network. The network updates its response preference in each frequency band through gradient backpropagation. The output feature dimension is unified to 64, which is the fused spectral feature.

[0054] Example: When detecting high-frequency welding abnormalities in a vehicle body assembly line, the system extracts data from acoustic sensors deployed near the welding arm and performs Fourier transform to obtain the amplitude spectrum. No obvious abnormalities are found in the 0-100 Hz frequency band, but a strong amplitude rise appears in the 1 kHz-2 kHz frequency band. The mask mechanism increases the mask weight of this frequency band to 2.0. After frequency band splicing, the output feature of the convolution attention network shows a significant weight increase in this frequency band. The subsequent model detects the abnormal prolongation of the welding time and the irregular noise fluctuation, realizing early warning of the uneven mechanical vibration inside the welding equipment.

[0055] The embodiment extracts energy distribution characteristics in different frequency bands through frequency domain analysis, and uses a dynamic mask function to strengthen the expression of key frequency bands. In the process of multi-class sensor data fusion, the recognition ability of non-stationary, burst or low-amplitude anomalies can be significantly improved. Through the attention mechanism, the expression weight of different frequency bands is further adaptively allocated to enhance the discriminability of the fusion features. Compared with single frequency band modeling or unweighted frequency spectrum splicing method, this method can more effectively distinguish the background components and abnormal disturbances in the signal, and improve the response sensitivity and time stability of the model to fine-grained faults.

[0056] S40, applying channel collaborative attention processing to selected multi-channel feature data, the selected multi-channel feature data being derived from the pre-processed multi-channel time series data or node feature data in the graph representation, to generate channel collaborative attention features;

[0057] In the embodiment, for the pre-processed data from multiple source sensors or the feature information extracted in the graph structure node, the purpose of the channel collaborative attention processing is to model the potential mutual relationship between different channels to enhance the expression ability of the feature representation. First, a time series feature tensor containing multiple channel dimensions needs to be constructed, and the shape of the tensor can generally be represented as channel number x time step x feature dimension. Different channels correspond to different sensor sources or different types of graph nodes. When processing high-dimensional time series data, the signals in different channels may exhibit high heterogeneity in time evolution, and separate modeling will ignore the complementarity between channels, so an attention mechanism needs to be introduced to explore the channel collaboration relationship.

[0058] In the implementation process, first, the multi-channel time series tensor is projected into a new space through a linear transformation operation, and query vectors, key vectors and value vectors in the channel dimension are generated respectively. The query vector is used to represent the dependence request of the current channel to other channels, the key vector expresses the feature keyness of each channel, and the value vector represents the original feature content. In the channel dimension, similarity calculation is performed between the query vector and the key vector, and a scaled dot product can be used to generate a channel attention score matrix, which is used to measure the correlation between any two channels.

[0059] In order to avoid excessive concentration or failure of attention scores, a learnable subspace mask is further introduced. The mask defines a sparse structure in the feature space, which can explicitly modify the attention score matrix and only retain part of the important inter-channel connections to suppress the interference of redundant or noisy channels on the output features. The mask weight can be adaptively obtained through training, or in some embodiments, the sparsity of the mask weight can be enhanced through a regular term.

[0060] The modified attention weight matrix and value vector are weighted and summed along the channel dimension to generate an initial channel interaction feature tensor. This tensor fuses features from multiple channels and significantly amplifies the response of high mutual information channels. Subsequently, to further enhance modeling capability, a multi-head attention mechanism is introduced, which divides the initial feature tensor into several subspaces, performs the above collaborative attention process separately, and then splices and fuses to output the final channel collaborative attention feature. The multi-head structure allows the model to model different interaction paths between channels in multiple representation subspaces in parallel, improving generalization ability and structural expression ability.

[0061] The preprocessed 32-channel vibration and acoustic fusion data can be constructed into a tensor with a size of 32x100x64, representing 32 channels, 100 time steps, and 64-dimensional features. This tensor generates query, key, and value matrices through three different linear layers, each with a dimension of 32x100x32. The dot product similarity between the query and the key in the channel dimension is calculated to obtain a 32x32 attention score matrix. An L1 regularization optimization is used to generate a 32x32 sparse mask matrix, which imposes a sparsity constraint on the attention score, retaining the top 4 channel connections with the highest attention values. After modifying the weights, the value matrix is multiplied to generate a channel interaction tensor, which is spliced into multiple heads to form the final attention fusion feature with a shape of 32x100x128, which is used for subsequent time series modeling.

[0062] Example: In the middle of the automobile chassis assembly line, multiple acceleration and sound sensors are deployed to detect component impact mass. During a certain period, an individual acoustic channel detects high-frequency abnormalities, but its energy fails to trigger an alarm independently. Through the channel collaborative attention mechanism, the attention weight of this acoustic channel is highly similar to that of the adjacent vibration channel, and the system automatically identifies the linkage of the two channels in the fault propagation path, enhancing the expression weight of the acoustic channel. The model finally inputs the multi-channel interaction feature into the anomaly detector, successfully locates the asymmetric vibration pattern caused by bolt thread stripping, and prompts the equipment maintenance personnel to intervene, thereby avoiding the occurrence of batch installation errors.

[0063] This embodiment can effectively capture the collaborative evolution pattern between different sensing channels within a certain time window through attention modeling along the channel dimension, thereby incorporating signal channels with structural correlation or functional coupling into the feature enhancement path. Compared to using single-channel input alone, this strategy can enhance the model's expression ability for implicit faults under complex working conditions, avoiding information silos and signal redundancy interference. At the same time, the multi-head mechanism models diverse channel relationships in different subspaces, effectively avoiding the model from being trapped in a single focus path, enhancing the model's adaptability and stability in different scenarios.

[0064] S50, generating an analysis result by comprehensively reasoning the graph representation, the multi-spectrum fusion feature, and the channel collaborative attention feature through a time series analysis model.

[0065] In this embodiment, for the graph representation, the multi-spectrum fusion feature, and the channel collaborative attention feature generated from multiple data processing links, the comprehensive reasoning process relies on a time series analysis model to model, so as to capture the dynamic evolution relationship and potential abnormal pattern of the features in the time dimension. The time series analysis model usually has the ability to process multi-scale and multi-dimensional time series input, and integrates feature modeling, time series prediction and anomaly detection functions. The design of the model usually includes feature input structure, autoregressive prediction module, reconstruction module and anomaly scoring mechanism.

[0066] Firstly, the structural fusion processing of the feature information from different sources is needed. The graph representation is usually represented as a node feature matrix to represent the structural relationship, and the fusion of the frequency domain and the channel interaction feature needs to keep the time series dimension aligned. Therefore, the graph node features are expanded into the same time series structure as the multi-spectrum fusion features and the channel collaborative attention features, and are spliced at the feature level to form a unified input feature vector. The input vector has a high dimensional cross, contains structural dependence, frequency energy response and channel perception information, and can provide a richer signal context for the time series model.

[0067] Inside the model, a multi-level autoregressive mechanism is usually introduced to model short-term and long-term time dependence patterns. The mechanism is based on the joint input features of the current time step to predict the feature evolution trajectory of the future time step, while reconstructing the feature representation of the current time step. The time series result output by the prediction module is compared with the actual observation sequence to calculate the prediction error to capture the potential deviation. The output of the reconstruction module is aligned with the input feature to evaluate the reconstruction error, which is used to identify feature loss or structural mismatch.

[0068] The comprehensive anomaly scoring mechanism is constructed according to the combination of the prediction error and the reconstruction error to measure the state deviation degree of the overall system at the current time. The scoring result does not use a fixed threshold for judgment, but dynamically adjusts the threshold range based on the historical normal state distribution to ensure adaptive recognition ability under different working conditions. Finally, by comparing the score with the dynamic threshold, the state identifier of whether it is abnormal is output.

[0069] To enhance the explainability, a classifier structure can also be integrated to further identify the fault type based on the joint feature vector after detecting the abnormal state, and output the specific fault probability distribution. For example, a Softmax activation function is used to output a probability vector including fault labels such as motor overload, sensor drift, assembly abnormality, etc.

[0070] The time series analysis network can be designed using a dual-path structure with a Transformer encoder+decoder structure, and the input dimension is (time step x feature dimension), where the feature dimension is the concatenation of the graph, frequency, and attention features, and the size is 512. The encoder path is responsible for modeling the context relationship of the current time step and generating the semantic vector of the current feature, and the decoder path predicts the next N time steps. The prediction module outputs the mean square error of the difference between the predicted value and the actual observed value as the prediction score, and the reconstruction module uses the skip connection and residual feedback to generate the reconstruction vector of the current time step, and compares it with the input feature to generate the reconstruction score. The anomaly score is obtained by weighted average of the prediction score and the reconstruction score. In the training stage, the score distribution is established by using historical working condition data, and in the inference stage, the deviation between the anomaly score and the threshold value is calculated in real time, and the abnormal state is output, and the fault classifier is activated to give the label probability.

[0071] Example: In a certain automobile welding production line, the system deploys multiple node sensors to monitor current, voltage, acoustic, and structural stress signals, and forms node time series data based on graph structure. Intermittent welding current abnormal fluctuations occur during operation, and single feature cannot accurately determine whether it is a device failure. After multi-spectrum processing and channel attention fusion, the anomaly presents an enhanced response in the joint features corresponding to the graph nodes. The time series model identifies that the reconstruction error and the prediction error increase synchronously within a continuous time window, the score exceeds the dynamic threshold and is judged as an abnormal state, and the classifier outputs the probability of the welding gun arm elastic structure loosening fault as high as 84%. The analysis result prompts the operation and maintenance personnel to timely locate the problem components, avoids miswelding and product scrap in subsequent processes, and significantly improves the production stability and intelligent response ability.

[0072] The embodiment helps to enhance the model's ability to identify the evolution trend of the system state and improve the recognition rate of complex time series anomalies by jointly modeling on the basis of multi-modal features. The multi-level autoregressive structure realizes feature prediction at different time scales, effectively adapting to rapidly changing and slowly drifting working condition features. The introduction of the reconstruction path builds the ability to reproduce redundant features, making the model have stronger perception ability for hidden anomalies and rare patterns. The dynamic threshold adjustment mechanism can adjust the score judgment in real time according to environmental fluctuations, avoiding the false alarm or missed alarm risk caused by static threshold. The probability distribution output by the classifier provides operational information support for subsequent fault root cause analysis and hierarchical processing.

[0073] The application relates to the technical field of data analysis, and discloses a multi-modal time sequence anomaly analysis method, device, equipment and medium, which comprises the following steps: collecting multi-source sensing signals and preprocessing the same to generate multi-channel time sequence data, constructing a graph representation representing the correlation between devices and sensors, extracting frequency domain features and fusing the same to generate multi-spectrum fusion features, applying a channel collaborative attention mechanism to specific channel feature data to obtain channel attention features, jointly reasoning the graph representation, the spectrum fusion features and the channel attention features through a time sequence analysis model, and generating analysis results for fault identification. The application fuses graph structure features, frequency energy features and channel interaction attention features, introduces a time sequence modeling mechanism to realize dynamic joint reasoning of multi-modal features, effectively improves the identification accuracy and response efficiency of line equipment faults under multi-source heterogeneous and strong noise interference, and has strong complex working condition adaptability and model reasoning robustness.

[0074] In one embodiment, the above step S10 comprises:

[0075] S101, collecting temperature sensor data, multi-axis vibration sensor data, device running torque data, current signals and acoustic sensor data from monitoring devices in the stamping, welding, painting and assembly links of a production line to form initial multi-source sensing signals;

[0076] S102, performing format unification processing on the initial multi-source sensing signals, and converting signals of different protocols into standardized data streams marked with uniform time stamps;

[0077] S103, adopting a combination algorithm of difference filtering and Kalman filtering to perform random noise suppression on the standardized data streams, eliminate power frequency interference and environmental noise, and generate data streams after noise reduction;

[0078] S104, performing time stamp alignment on the data streams after noise reduction based on a network time protocol or a synchronous trigger signal, correcting time deviation caused by transmission delay, and generating time-aligned data streams;

[0079] S105, performing data compensation on missing data in the time-aligned data streams caused by sensor failure or communication interruption by using linear interpolation or spline interpolation of adjacent time points, and generating compensated data streams;

[0080] S106, adjusting adaptive fusion weights according to real-time signal-to-noise ratios and data stationarity evaluation results of each channel sensor signal in the compensated data streams, and generating preprocessed multi-channel time sequence data through weighted summation.

[0081] In this embodiment, the sources of collected sensor signals include typical nodes such as stamping, welding, painting, and assembly in multiple process links of the workshop production line, and multiple types of sensors are deployed in each link. The types of sensors include temperature sensors for sensing the temperature rise of equipment or material surfaces; multi-axis vibration sensors for monitoring the vibration mode and amplitude in mechanism motion; torque sensors for reflecting the force state of equipment execution components during operation; current sensing channels for detecting the load characteristics of power drive systems; and acoustic sensors for collecting audio patterns caused by mechanical collisions, abnormal sounds, or environmental reflections. These data sources constitute an initial multi-source heterogeneous data set, which often has problems such as inconsistent sampling rates, differences in communication protocols, and different data packet structures.

[0082] In order to build a unified data input basis, it is necessary to first perform protocol conversion and time standardization operations to unify the collected data streams of different sources into a standardized structure with global timestamps. The data in a unified format can be encoded in the form of key-value pairs, and each data frame contains a timestamp field, a sensor channel identifier, and an observation value. In terms of implementation, edge protocol converters or data access agents can be deployed at the collection nodes to support industrial protocols such as Modbus, CAN, and Profinet, and to synchronize the transmission in JSON or binary structure format.

[0083] Signals are inevitably affected by industrial site noise, power equipment interference, environmental electromagnetic waves, and other interference factors during the collection process. In order to remove unstructured interference, a multi-stage filtering link needs to be constructed, using differential filtering to eliminate trend drift and low-frequency offset, and then introducing a Kalman filter to dynamically estimate the state of the signal and suppress high-frequency noise. The state prediction function of the Kalman filter here can be parameter-adjusted according to the historical mean and covariance matrix, and the covariance estimate is updated online to improve the real-time convergence ability under changing conditions.

[0084] Since each collection channel may be driven by different equipment and transmitted through asynchronous communication links, there is a time alignment deviation between different signal streams. Using network time protocol (such as NTP) or high-precision synchronous trigger control modules (such as PTP or external TTL trigger) can uniformly correct the collection timestamps, align the data streams to the microsecond level, and solve the problem of time misalignment between signals caused by collection sequence or network delay. The data after synchronization can be referred to as time-aligned data streams.

[0085] Some sensors may be missing due to faults, disconnections, or communication interruptions during the collection process. In order to fill in the missing segments and maintain temporal continuity, a historical neighborhood window-based interpolation algorithm is used. Linear interpolation is suitable for short-term missing of stable signals, and spline interpolation is suitable for smooth reconstruction of periodic or slowly fluctuating signals. Both methods can be dynamically selected according to the signal type and missing length.

[0086] After obtaining the complete time alignment and interpolation compensation data, the reliability of each channel is evaluated based on the signal-to-noise ratio (estimated by the ratio of short-time energy to background fluctuation) and time stability (calculated by moving variance) of each sensor signal. According to the signal quality evaluation results, the fusion weight of each channel is adjusted, and the multi-channel data is synthesized by weighted summation. The weight distribution mechanism should have adaptive ability, automatically adjust the contribution of high noise channel when the influence of high noise channel is enhanced, so as to generate stable and unified data set as the input basis for subsequent modeling analysis.

[0087] The embodiment realizes the structural standardization of different types and multi-time scale sensor signals by constructing a unified preprocessing process, and improves the data fusion capability. The combination mechanism of difference and Kalman filter enhances the resistance to high frequency and low frequency interference, and reduces the error propagation risk. The time alignment and interpolation compensation eliminate the influence of asynchronous signal acquisition and communication loss on subsequent analysis, and enhance the data time sequence continuity. The joint evaluation mechanism of signal-to-noise ratio and stability makes the channel fusion process have adaptive ability, and can continuously output high reliability analysis input data under multiple working conditions.

[0088] In one embodiment, the above step S20 comprises:

[0089] S201, defining a graph node set, the graph node set comprising a production line equipment unit node and a sensor monitoring point node, the equipment unit node corresponding to a stamping robot, a welding workstation, a coating device and an assembly line body, and the sensor monitoring point node being associated with each sensor signal source in the preprocessed multi-channel time series data;

[0090] S202, defining a set of edge relationships between nodes based on physical adjacency or upstream and downstream relationships in the process flow, the physical adjacency edge representing the installation position adjacency between devices, and the process flow edge representing the material transfer or control signal dependency relationship;

[0091] S203, constructing a weighted adjacency matrix according to the set of edge relationships, the element value of the weighted adjacency matrix being determined by the fault propagation strength between devices or the sensor signal correlation;

[0092] S204, calculating a weighted Laplacian matrix containing adjustment parameters and high frequency suppression coefficients based on the weighted adjacency matrix;

[0093] S205, associating the time series signal segment corresponding to each sensor monitoring point node in the preprocessed multi-channel time series data to the node feature vector of the sensor monitoring point node;

[0094] S206, generating a graph representation containing time sequence features and topological association relationships based on the node feature vector, the weighted adjacency matrix and the weighted Laplacian matrix.

[0095] In this embodiment, the pre-processed multi-channel time sequence data has the basic conditions for embedding the structure graph after structure standardization and time alignment. In order to establish a spatial expression mode reflecting the structure relationship between the equipment and the sensor nodes, the whole production line system needs to be modeled in the form of graph construction. The nodes in the graph structure are the core components for expressing the signal source and the physical entity where it is located. The composition of the node set includes two categories. One is the equipment unit node, which specifically corresponds to the stamping robot, welding workstation, painting equipment and assembly line body and other typical production line operation modules. The other is the sensor monitoring point node, which has a one-to-one correspondence with the collected multi-channel sensor signal source, used to represent the physical source location of the signal attachment.

[0096] After the definition of the nodes, the structural relationship between the nodes needs to be further clarified. These relationships not only come from the layout relationship in the physical space, but also may come from the logical order of the process flow. Therefore, the edge relationship set is introduced for description. Each edge has a clear semantic meaning. The physical adjacent edge is used to connect the adjacent equipment nodes in space, indicating that there is a direct structural connection between the equipment, which is often used to evaluate the possibility of vibration propagation, heat conduction or maintenance interference. The process flow edge is used to describe the process logic of the material or control flow from the upstream node to the downstream node, which is used to express the functional path and the influence link of the fault possibility. The edge relationship can be distinguished by label, and the existence of multiple types in the graph is supported.

[0097] According to the established edge set, a complete weighted adjacency matrix can be constructed. The matrix is a two-dimensional sparse matrix, and the numerical value of each element is used to quantify the connection strength between two nodes. The weight in the matrix can be calculated based on multiple ways. For the equipment, the propagation strength can be constructed based on the frequency of the downstream equipment triggering exceptions after the failure of a certain equipment in the historical maintenance record. For sensors, the Pearson correlation coefficient, mutual information or dynamic time warping distance (DTW) of time sequence signals can be used to measure the correlation, so as to assign values to the edge weight.

[0098] After obtaining the weighted adjacency matrix, the Laplace operator can be further introduced to generate a weighted Laplace matrix with a tuning parameter. The Laplace matrix is used to enhance the frequency filtering capability in structure learning, the tuning parameter is used to control the global scaling of the connection strength, and the high-frequency suppression coefficient is used to reduce the sensitivity of high-frequency components in graph signal transmission, preventing noise propagation or local mutations from affecting the overall network representation. The tuning parameter can set the weight priority for different node categories, and the high-frequency suppression coefficient can be adaptively set by the signal energy spectrum structure.

[0099] Subsequently, the existing time series data needs to be assigned to each sensor monitoring point node in the graph structure to form an input tensor that can be used for graph signal processing. The feature vector of each node is derived from the transmission channel associated with the node, and a one-dimensional signal slice or statistical feature (such as mean, variance, and kurtosis) is constructed in a time window to form an attribute vector and is bound to the corresponding node. The node features can be propagated and reconstructed by graph neural networks, spectral convolution, or attention mechanisms in subsequent analysis.

[0100] Finally, by fusing the node feature vector, the weighted adjacency matrix, and the Laplacian matrix, a complete graph representation structure can be constructed. This structure has dual information, reflecting both the local state change of the node and the possible propagation path of the fault or anomaly in the topology structure, and has the basic representation ability that can be used for deep structure learning.

[0101] The present embodiment establishes a structural mapping relationship between sensor signals and production line equipment through graph representation, enabling originally disordered and isolated multi-channel time series data to have a propagable topology semantic, establishing a signal coordination path in space, and enhancing position, structure, and process information in feature expression. By constructing an edge weight association structure based on physical adjacency and process flow, the response capability to fault diffusion and signal linkage in anomaly detection is further improved. After introducing the Laplacian structure, high-frequency fluctuations or local anomalies can be regulated in the frequency domain, reducing the false positive rate and improving the response sensitivity to sudden systemic failures.

[0102] In one embodiment, the above step S204 includes:

[0103] S2041, setting an adjustment parameter according to the strength of the physical adjacency relationship between devices, wherein the adjustment parameter value between physically adjacent devices is set to a first preset value, and the adjustment parameter value between non-adjacent devices is set to a second preset value, the first preset value being higher than the second preset value;

[0104] S2042, obtaining the proportion of signal energy in a high frequency band to total energy in the preprocessed multi-channel time series data, and adjusting the size of the high frequency suppression coefficient according to the proportion;

[0105] S2043, multiplying the adjustment parameter and the original edge weight value of the weighted adjacency matrix to obtain a corrected edge weight value;

[0106] S2044, calculating the degree value of each node according to the corrected edge weight value to generate a degree matrix with an adjustment parameter;

[0107] S2045, multiplying the degree matrix with the adjustment parameter and the high frequency suppression coefficient to generate a weighted Laplacian matrix.

[0108] In the embodiment, the construction process of the weighted Laplacian matrix is based on the edge weight correction of the graph structure and the frequency domain regulation mechanism. First, an adjustment parameter needs to be assigned to different edge connection relationships. The adjustment parameter is a weight coefficient of connection strength, which is used to adjust the contribution difference between different edges in the adjacency matrix. The physical adjacency relationship can be confirmed through a layout diagram or a maintenance archive, and is defined as the case where there is a direct structural connection between devices, such as connected robot arms, shared platforms, or adjacent sensing devices. Such connections are assigned a first preset value, which represents a higher propagation credibility or correlation strength. For device combinations that have no direct physical connection but still have weak coupling paths, a second preset value is assigned, thereby reflecting a weaker signal correlation weight. The first preset value is higher than the second preset value, which is used to highlight the strong connection channels in the adjacency matrix and improve the focusing ability of structure learning.

[0109] Secondly, a high-frequency suppression coefficient needs to be dynamically calculated according to the proportion of high-frequency components in each sensing signal. High-frequency components often contain sudden abnormalities, interference signals or non-structural vibration components. This ratio can be obtained by integrating the spectral energy, for example, by setting a characteristic frequency threshold and dividing the energy above the threshold by the total energy to obtain the proportion. The higher the proportion, the greater the possibility of high-frequency noise on the node or connection edge, so the high-frequency suppression coefficient should be increased accordingly to reduce its interference with the global structure in the construction of the Laplacian matrix. The coefficient can be set as a continuous function, for example, γ = γ o × (1 + α × P), where P is the proportion of high-frequency energy, γ o and α are adjustable parameters.

[0110] After that, the above adjustment parameters are multiplied element by element with the original edge weight values in the weighted adjacency matrix to obtain the modified edge weight matrix. The original edge weight values can come from the cross-correlation coefficient between signals, the historical fault co-occurrence frequency, or the mutual information calculation based on the sampling window. The modification operation improves the adaptability of the graph structure to the physical structure, and at the same time, it weights the connections in the high-frequency signal path and suppresses the influence of its propagation path on the graph signal modeling.

[0111] The degree matrix is calculated based on the modified edge weight matrix, that is, the sum of the connection strengths of each node is calculated. Unlike the traditional degree matrix, the generated degree value here contains the influence of the adjustment parameter, so the generated degree matrix is a degree matrix with adjustment parameters, which retains the adjustment results of the node local structure and high-frequency disturbance. The degree matrix reflects the status of the node in the structure graph and also participates in the overall structure tension calculation as the diagonal part of the Laplacian matrix calculation.

[0112] Finally, the final weighted Laplacian matrix is generated by weighting the above degree matrix and the high-frequency suppression coefficient. This matrix not only retains the global topological information of the graph structure, but also combines the regulation and control capabilities of the edge weight adjustment and spectral suppression in two dimensions. It is a key input structure in subsequent graph signal propagation, spectral filtering or graph convolution operations.

[0113] In the implementation process, the physical adjacency relationship can be automatically parsed through the CAD layout diagram, production process diagram or configuration database, and the preset value setting can be adjusted according to experience by referring to the historical fault influence propagation path. The adjustment parameter generation adopts a conversion mode of logical mapping table or Boolean structure to floating point weight, which is suitable for large-scale node structure.

[0114] The high-frequency energy proportion calculation can be performed by short-time Fourier transform to perform energy statistics on each channel, and then a threshold is set to divide the frequency band, for example, the boundary frequency f o , the ratio of the frequency energy higher than f o to the total energy is input as the proportion value. The spectral energy calculation supports multiple time updates in a sliding window, thereby obtaining time-varying adjustment capability.

[0115] The weighted Laplacian matrix generation process is based on L=D-A structure, where D is the degree matrix with adjustment, and A is the modified adjacency matrix. When introducing the high-frequency suppression coefficient, the coefficient can be multiplied into A, or a factor matrix can be introduced in L to realize the fusion of frequency domain control and structure control.

[0116] Example: In a car door welding production line, if A and B are two adjacent robot units sharing a support platform, the physical adjacency is set to 0.9; while B and C share data but have no structural connection, the adjustment parameter is set to 0.3. If the high-frequency band proportion in the sensor signal of A and B is 0.4, which is higher than the set threshold, the high-frequency suppression coefficient is set to 1.2. The modified edge weight is the original edge weight multiplied by the corresponding adjustment parameter, and then multiplied by 1.2 to get the final adjacency weight. Each node calculates the degree matrix according to the modified edge weight, and generates the weighted Laplacian matrix together with the adjacency matrix. In the process of graph convolution, this matrix greatly limits the feature propagation direction of node B, avoiding the misdirection of features caused by high-frequency interference of node C, thereby improving the stability and accuracy of abnormal response in structural reasoning.

[0117] The embodiment can effectively enhance the response capability of the connection strength difference between devices and the frequency spectrum structure of the sensing signal in the graph structure by constructing a weighted Laplacian matrix containing a regulation parameter and a high-frequency suppression mechanism, so that the structure learning process pays more attention to stable propagation paths and weakens high-frequency abnormal paths, and the selectivity and robustness of the structural feature modeling in the graph neural network are improved. The regulation parameter improves the semantic resolution of the node edge weight, and the high-frequency suppression coefficient realizes dynamic noise prevention and regulation in the frequency domain, and the combined action makes the node information propagation in the graph more stable and the feature convolution process more reliable, thereby improving the discrimination accuracy of the overall fault discrimination model.

[0118] In one embodiment, the above step S30 comprises:

[0119] S301, performing discrete Fourier transform on the vibration signals and acoustic signals in the preprocessed multi-channel time series data to generate corresponding frequency domain amplitude spectra;

[0120] S302, according to the feature frequency range corresponding to the preset fault mode, the frequency domain amplitude spectrum is divided into low frequency band, medium frequency band and high frequency band sub-band;

[0121] S303, based on the energy distribution baseline of each sub-band under normal working condition, a frequency band mask function is designed, which is used for selectively enhancing or suppressing the frequency spectrum components of the specified sub-band;

[0122] S304, the frequency band mask function is respectively applied to the corresponding sub-band to generate weighted low frequency band feature spectrum, medium frequency band feature spectrum and high frequency band feature spectrum;

[0123] S305, cross-band feature splicing is performed on the low frequency band feature spectrum, the medium frequency band feature spectrum and the high frequency band feature spectrum to form a multi-dimensional frequency spectrum feature matrix;

[0124] S306, the weights of each sub-band in the multi-dimensional frequency spectrum feature matrix are dynamically optimized by an attention network to generate a multi-spectrum fusion feature.

[0125] In the embodiment, the key starting point of converting the multi-channel time series data into a frequency spectrum representation is to identify the signal type that contains rich structural features. The vibration signal and the acoustic signal are selected as the main transformation object because of their high sampling rate and dynamic change characteristics, which exhibit obvious frequency domain response in structural impact, device operating state change and transient abnormal event. Discrete Fourier transform can effectively decompose the original time domain signal into different frequency components, and the frequency domain amplitude spectrum is obtained to reveal the energy intensity distribution of each frequency component. Fast Fourier transform algorithm (FFT) can be used for discrete Fourier transform to improve processing efficiency, and sliding calculation is performed in a short time window to preserve the time locality.

[0126] After the generation of the frequency domain amplitude spectrum, the frequency band needs to be divided according to the existing fault knowledge base in the industrial scene. The division of the characteristic frequency range is generally derived from typical fault modes, such as gear meshing faults which are commonly found in the medium frequency band, structural looseness which is often manifested as an increase in low frequency components, and bearing damage which excites high frequency harmonics. Dividing the frequency spectrum into low, medium, and high frequency bands can provide more recognizable feature subspaces for subsequent models. This division can be based on fixed frequency thresholds or dynamically set frequency band boundaries according to device models and process cycles.

[0127] On the basis of existing frequency band division, introducing a frequency band mask function for directional regulation is a key operation to achieve differentiated feature enhancement. The frequency band mask function is based on the energy distribution baseline of each frequency band under normal working conditions, which can be obtained by modeling the statistical mean or quantile of the frequency domain amplitude of historical normal samples. The design of the mask function can take various forms, such as threshold type (setting a certain frequency threshold to retain or suppress), weight type (continuous attenuation or enhancement), and window function type (such as Hamming window modification of specific frequency band edges), which is used to focus on abnormal enhancement areas in the frequency spectrum and reduce the weight of high-noise frequency bands, achieving the purpose of selective enhancement or suppression.

[0128] After applying the mask function to each frequency band, the weighted low, medium, and high frequency band feature spectra are obtained. These feature spectra have consistent dimensions and can be spliced on the time axis or channel axis. The cross-frequency band feature splicing process combines the features of the three frequency bands into a unified structure, forming a multi-dimensional frequency spectrum feature matrix. The splicing method can be a simple channel stacking, or it can use feature-level connection (such as deep direction splicing) or vectorization connection to form a structure expression that is convenient for subsequent feature extraction and modeling.

[0129] Finally, the attention mechanism is introduced to improve the model's ability to distinguish the importance of different frequency band features. The attention network can use channel attention or frequency band attention structure to extract the response strength of each frequency band in the current input sample, and then use weight parameters for normalization to achieve dynamic weighted combination of sub-frequency bands in the multi-dimensional frequency spectrum feature matrix. The network structure can use a lightweight transformer variant, or integrate SE modules, CBAM modules, etc. to realize joint modeling of feature compression and enhancement. The final output of the multi-spectrum fusion feature has high robustness, interpretability, and heterogeneous anomaly coverage capability, providing sufficient information support for subsequent discriminant models.

[0130] By introducing spectral decomposition, band mask modulation and attention optimization mechanisms, this embodiment realizes fine modeling of multi-band coupling characteristics in structural signals. Band division and mask operation strengthen the response of the typical fault excitation area and suppress the noise caused by background interference and structural resonance; the attention mechanism further realizes dynamic weight adjustment between frequency bands, so that the model can adjust the attention area according to the specific input, effectively improving the fusion quality and diagnostic accuracy of multi-source features. Compared with single frequency domain feature extraction method, this method shows stronger stability and discrimination ability in the face of signal disturbance and complex frequency composition in manufacturing scene.

[0131] In one embodiment, the above step S40 comprises:

[0132] S401, constructing a multi-channel time series feature tensor, the channel dimension of the multi-channel time series feature tensor comprising a sensor signal channel in the pre-processed multi-channel time series data or a node feature channel in the graph representation;

[0133] S402, performing linear transformation on the multi-channel time series feature tensor to generate a query matrix, a key matrix and a value matrix in the channel dimension;

[0134] S403, calculating the similarity score of the query matrix and the key matrix to generate an inter-channel attention score matrix;

[0135] S404, applying a learnable subspace mask to sparsify and correct the inter-channel attention score matrix to generate a corrected attention weight matrix;

[0136] S405, weighting and aggregating the corrected attention weight matrix and the value matrix to generate an initial channel interaction feature tensor;

[0137] S406, performing multi-head attention weighted fusion on the initial channel interaction feature tensor to generate a channel collaborative attention feature.

[0138] In this embodiment, before channel collaborative modeling, the multi-dimensional data needs to be unified and arranged into a tensor structure with explicit channel dimension. The multi-channel time series feature tensor refers to a three-dimensional tensor structure, whose dimensions are usually defined as the number of samples, the length of time series and the number of channels. The channel dimension can come from multi-source sensor signals in the original pre-processed data, such as temperature, vibration, acoustics, etc., or it can be a node feature channel in the graph structure, which can represent each node as a feature vector group containing time series information. Mapping different types of data into a unified tensor structure is a prerequisite for subsequent attention mechanism operations.

[0139] On the basis of the tensor, a parameterized linear mapping operation is set to generate a query matrix, a key matrix and a value matrix for the tensor in the channel dimension. This process is implemented by matrix multiplication, that is, three sets of linear mappings are performed on the original feature tensor in the channel dimension to obtain three sets of feature projection matrices of the same dimension or adjustable dimension, which are used to capture the representation mapping relationship between channels. The generated query matrix represents the request of the current channel, the key matrix represents the response condition of other channels, and the value matrix represents the potential feature contribution.

[0140] The attention score matrix between channels is calculated by performing a dot product operation on the query matrix and the key matrix, and optionally scaling (e.g., dividing by the square root of the vector dimension). This matrix depicts the information correlation or synergy degree between channels. Since not all channels have strong correlation, some redundant or noisy channels should be actively suppressed to avoid interfering with the modeling of the main channel. Therefore, a learnable subspace mask is introduced for sparse correction. The mask is a set of differentiable weights obtained through training, which is used to shield the attention links of low correlation areas or high noise channels, achieving sparse connection enhancement of the attention structure.

[0141] The corrected attention score matrix is normalized by softmax and used as the attention weight to perform weighted summation with the value matrix to generate an initial channel interaction feature tensor. This tensor can be regarded as the updated representation of each channel after fusing the contributions from other channels at the current time, and contains cross-channel interaction information. To further improve the modeling capability of the model for complex channel relationships, a multi-head attention mechanism is introduced, that is, the above attention process is repeatedly executed multiple times, each time focusing on a different representation subspace. The channel interaction tensors output by each head are concatenated or weighted and fused to form a channel synergy attention feature.

[0142] This synergy feature not only encodes the information of each channel itself, but also integrates the complementary information of other channels, and has stronger context association expression capability. The mutual influence relationship between different channels is explicitly modeled in this process, enabling the model to dynamically adjust the dependence weight between channels and adapt to the coupling characteristics between sensors in complex industrial scenarios.

[0143] This embodiment enables the model to dynamically identify the dependence structure between different channels by constructing an explicit channel attention mechanism, avoiding the problem of insufficient generalization ability caused by fixed weighted fusion. The subspace mask mechanism improves the robustness of channel modeling, strengthens the representation ability of the main fault signal channel, and suppresses the influence of redundant or low-quality channels. The multi-head mechanism introduces multiple-angle feature aggregation paths, further enhancing the model's ability to fuse heterogeneous information, making the diagnosis system more adaptable and generalizable to complex industrial signals. Compared with the traditional channel concatenation or weighted average strategy, the synergy attention processing has better explainability and abnormal channel mining ability.

[0144] In one embodiment, the step S50 described above comprises:

[0145] S501, constructing a time series analysis model comprising a multi-level autoregressive mechanism;

[0146] S502, performing cross-modal feature splicing on the node feature vector in the graph representation, the multi-spectrum fusion feature, and the channel collaborative attention feature to generate a joint input feature vector;

[0147] S503, modeling the joint input feature vector for time series dependency through the multi-level autoregressive mechanism of the time series analysis model to generate a predicted feature vector for a future time step and a reconstructed feature vector for a current time step;

[0148] S504, calculating the prediction error between the predicted feature vector and the actual observation value, and calculating the reconstruction error between the reconstructed feature vector and the joint input feature vector to generate a comprehensive anomaly score;

[0149] S505, adjusting the anomaly determination threshold based on the distribution of historical normal operating conditions, and generating an anomaly state identifier according to the comparison result of the comprehensive anomaly score and the anomaly determination threshold, taking the anomaly state identifier as the analysis result;

[0150] S506, when the analysis result is an abnormal state, outputting the probability distribution of the specific fault type through the classifier of the time series analysis model.

[0151] In this embodiment, a time series analysis model comprising a multi-level autoregressive mechanism is constructed, aiming to model the multi-scale variation characteristics of time series data. The multi-level autoregressive mechanism refers to the model making predictions and reconstructions on multiple scales of time series. It not only considers the relationship between the current time step and the past time steps, but also includes context modeling across time levels, thereby adapting to the long-term evolutionary trend or short-term mutation pattern that multiple types of fault signals on the production line may have. This mechanism is usually based on a recursive network structure, such as a Gated Recurrent Unit (GRU), a Long Short-Term Memory (LSTM), or a Transformer-based self-attention structure, and combines multiple time window lengths for residual recursive nesting to form a multi-layer time loop.

[0152] The atlas representation, multi-spectrum fusion feature and channel collaborative attention feature are respectively derived from different information modeling paths, each of which encodes the structural dependence between devices, frequency domain distribution characteristics and collaborative representation among multiple channels. Before analysis and reasoning, these heterogeneous features need to be spliced to unify the model input format. The process of cross-modal feature splicing usually merges the features after dimension alignment and normalization processing according to the feature dimension, forming a joint input feature vector that integrates multiple information sources. This vector semantically integrates structural information, frequency spectrum energy changes and temporal signal interconnection, and has strong abnormal perception potential.

[0153] When modeling the temporal dependence of the joint input feature vector, the multi-level autoregressive mechanism simultaneously predicts the state of the next time step and reconstructs the input of the current time step at each time step. In implementation, two parallel sub-networks can be used, one for forward prediction to generate the feature vector of the future time step, and the other for reconstruction to evaluate the model's understanding of the current state by regressing the original input. The results of the two paths reflect the prediction error and reconstruction error, respectively, which are used to measure whether the current system behavior deviates from the normal mode.

[0154] The prediction error is the numerical deviation between the model output and the true future observation, while the reconstruction error is an indicator of the model's ability to restore the current input. By integrating the two in a weighted or nonlinear combination, a comprehensive abnormal score is formed. This score is used to measure the deviation of the current state under the model's cognition. To improve the adaptability of the judgment, the abnormal judgment threshold needs to be dynamically adjusted based on the historical normal operating data distribution. Common strategies include using quantile methods, probability distribution fitting, sliding window statistics, etc. to determine the score boundary, and then determining whether the current score exceeds the normal range.

[0155] The abnormal state identifier is a binary result based on the comparison between the score and the threshold, used to determine whether the current time point is in a potential fault state. When the identifier is an abnormal state, to improve operability and traceability, the fault classifier also needs to output the probability distribution of the specific fault type. The classifier structure can be selected from Softmax fully connected layer or more complex structures based on graph neural networks, Transformers, etc., which combine the multi-source features extracted in the model to generate probability values corresponding to each fault type, supporting subsequent early warning response and maintenance strategy formulation.

[0156] Example description: Deploy multi-source data acquisition and analysis system on tightening robot workstations in automobile assembly lines, aiming to achieve online fault monitoring and early anomaly identification of key execution components. The system first acquires heterogeneous physical signals through a multi-type sensor array deployed on the tightening robot arm, including three-axis vibration, torque, current, and temperature information, with a uniform sampling frequency of 2048 Hz. After data acquisition, the format is unified and time stamp alignment is completed, and a hybrid method combining difference and Kalman filtering is used for denoising of each channel data. Linear and spline interpolation is used to fill in the missing segments for packet loss and failure during sampling.

[0157] According to the signal-to-noise ratio and stability of each channel, the weight is dynamically adjusted to form the fused multi-channel time series data, which is then used to construct the graph. The graph takes the actuators, joint nodes, motors and other physical components in the tightening station as graph nodes, and the edge relationship is established based on the actual physical connection and function transmission. A weighted adjacency matrix is constructed, and combined with the relative structural importance between nodes, the adjustment parameter is set, the high-frequency suppression coefficient is added to suppress rapid signal disturbance, and finally a weighted Laplacian matrix with structural constraint and frequency domain smoothing capability is generated. Each node and its associated sensor data are connected to form a graph representation of the fused time series topology.

[0158] The signals of the vibration and current channels further enter the multi-spectrum processing flow. For the Z-axis vibration signal, its frequency domain amplitude spectrum is divided into 4 frequency bands, each corresponding to a typical frequency interval of different types of faults. Each frequency band applies a learnable mask function to suppress irrelevant frequency bands and emphasize fault-sensitive frequency bands, and the frequency band weight is dynamically adjusted through a weight backpropagation mechanism. Finally, the four frequency band spectral features are spliced into a multi-dimensional spectral matrix, which becomes the input of the next stage.

[0159] The generated multi-channel feature data is modeled cooperatively between channels. A tensor containing 10 feature channels (such as different axial spectra, temperature, current) is constructed, and after query, key, and value mapping, the channel attention distribution is calculated. To prevent irrelevant channels from interfering with the results, a sub-mask matrix is constructed based on the graph structure and statistical correlation to filter weakly correlated connection channels and strengthen the interaction between key channels. The fusion result is a channel cooperative attention feature that emphasizes sensitive feature combinations.

[0160] The graph representation, multi-spectrum fusion feature, and attention optimized feature are spliced in the feature dimension to construct a cross-modal feature input sequence. The input is fed into a Transformer model containing 6 layers of encoders, which performs short-term future prediction and current input reconstruction within each time window. The model outputs a prediction vector and a reconstruction vector, which are compared with the actual values to obtain a comprehensive fault score. The score uses a weighted L2 norm to combine the future prediction bias and the current reconstruction error, reflecting the deviation of the current state.

[0161] The system determines a dynamic threshold based on the score distribution in a historical normal cycle, and if the score of a plurality of consecutive sampling points exceeds the threshold, the system considers it as an anomaly. In this embodiment, the score mechanism can identify the weak vibration signal change caused by bearing wear in advance, and output the current possible fault type probability distribution through the classifier module, indicating that the probability of "end effector structure loosening" or "tightening motor drive fluctuation" is more than 70%. The system sends a warning signal to the production line maintenance system according to the classifier result, recommends to check and maintain the key components, and avoids subsequent more serious fault expansion.

[0162] Through the embedded processing architecture and heterogeneous signal fusion analysis method in the whole process, the system realizes millisecond-level processing response under the premise of ensuring the diagnosis accuracy, and adapts to the operation requirements of high-speed rhythm and complex load change in the automobile production line.

[0163] In this embodiment, the structure correlation, spectral feature distribution and channel synergy information are fused in a unified time sequence modeling framework through the reasoning process, the sensitivity of the model to the system state change is strengthened through the prediction and reconstruction double-path mechanism, and the missing detection problem of small sample faults or weak signal anomalies in traditional methods is avoided. The abnormal score and dynamic threshold judgment mechanism is introduced to effectively solve the problem of inconsistent score scale under different devices and different working conditions, and the generalization ability of abnormal judgment is improved. The classifier outputs the specific fault type to provide clear guidance for subsequent intervention, and improves the fault explainability and diagnosis operability of the overall system.

[0164] In an embodiment, a multi-modal time sequence anomaly analysis device is provided, which corresponds to the multi-modal time sequence anomaly analysis method in the above-mentioned embodiments. Referring to Figure 3 , Figure 3 A functional module schematic diagram of a preferred embodiment of the multi-modal time sequence anomaly analysis device of the present application. The data preprocessing module 10, the atlas construction module 20, the spectrum fusion module 30, the channel attention module 40 and the time sequence reasoning module 50. The detailed description of each functional module is as follows:

[0165] The data preprocessing module 10 is used for collecting multi-source sensing signals, and performing preprocessing operation on the multi-source sensing signals to generate preprocessed multi-channel time sequence data;

[0166] The atlas construction module 20 is used for mapping the preprocessed multi-channel time sequence data into atlas representation, and the atlas representation is used to reflect the correlation information between monitoring objects;

[0167] The spectrum fusion module 30 is used for extracting and fusing multiple spectral information of the preprocessed multi-channel time sequence data to generate multi-spectrum fusion features;

[0168] The channel attention module 40 is configured to apply channel collaborative attention processing on selected multi-channel feature data, to generate channel collaborative attention features, wherein the selected multi-channel feature data is derived from the pre-processed multi-channel time series data or node feature data in the graph representation.

[0169] The time series inference module 50 is configured to comprehensively infer the graph representation, the multi-spectrum fusion features and the channel collaborative attention features by a time series analysis model, to generate an analysis result.

[0170] In an embodiment, the data preprocessing module 10 is specifically configured to:

[0171] The temperature sensor data, multi-axis vibration sensor data, equipment running torque data, current signal and acoustic sensor data are collected from monitoring equipment in the stamping, welding, painting and assembly links of the production line to form initial multi-source sensing signals.

[0172] The initial multi-source sensing signals are subjected to format unification processing, and signals of different protocols are converted into standardized data streams marked with uniform timestamps.

[0173] A combination algorithm of difference filtering and Kalman filtering is used to suppress random noise of the standardized data streams, eliminate power frequency interference and environmental noise, and generate noise-reduced data streams.

[0174] The noise-reduced data streams are subjected to timestamp alignment based on a network time protocol or a synchronous trigger signal, time deviation caused by transmission delay is corrected, and time-aligned data streams are generated.

[0175] Missing data in the time-aligned data streams caused by sensor failure or communication interruption is compensated by linear interpolation or spline interpolation at adjacent time points, and compensated data streams are generated.

[0176] According to real-time signal-to-noise ratio and data stationarity evaluation results of each sensor signal in the compensated data streams, adaptive fusion weights are adjusted, and pre-processed multi-channel time series data are generated by weighted summation.

[0177] In an embodiment, the graph construction module 20 is specifically configured to:

[0178] A graph node set is defined, which includes production line equipment unit nodes and sensor monitoring point nodes, the equipment unit nodes correspond to stamping robots, welding workstations, painting equipment and assembly lines, and the sensor monitoring point nodes are associated with each sensor signal source in the pre-processed multi-channel time series data.

[0179] define a set of edge relations between nodes based on physical adjacency relations or process flow upstream / downstream relations, wherein the physical adjacency edges represent installation location adjacency between devices, and the process flow edges represent material transfer or control signal dependency relations;

[0180] construct a weighted adjacency matrix according to the set of edge relations, wherein element values of the weighted adjacency matrix are determined by fault propagation strength or sensor signal correlation between devices;

[0181] calculate a weighted Laplacian matrix containing an adjustment parameter and a high-frequency suppression coefficient based on the weighted adjacency matrix;

[0182] associate time series signal segments corresponding to each sensor monitoring point node in the preprocessed multi-channel time series data to a node feature vector of the sensor monitoring point node;

[0183] generate a graph representation containing time series features and topological association relations in combination with the node feature vector, the weighted adjacency matrix, and the weighted Laplacian matrix.

[0184] In an embodiment, the graph construction module 20 is specifically configured to:

[0185] set the adjustment parameter according to the strength of the physical adjacency relation between devices, wherein the adjustment parameter value between physically adjacent devices is set to a first preset value, and the adjustment parameter value between non-adjacent devices is set to a second preset value, the first preset value being higher than the second preset value;

[0186] obtain the proportion of signal energy in the high-frequency band to the total energy in the preprocessed multi-channel time series data, and adjust the size of the high-frequency suppression coefficient according to the proportion;

[0187] multiply the adjustment parameter and the original edge weight value of the weighted adjacency matrix to obtain a modified edge weight value;

[0188] calculate the degree value of each node according to the modified edge weight value to generate a degree matrix with an adjustment parameter;

[0189] multiply the degree matrix with the adjustment parameter and the high-frequency suppression coefficient to generate a weighted Laplacian matrix.

[0190] In an embodiment, the spectrum fusion module 30 is specifically configured to:

[0191] perform discrete Fourier transform on the vibration signals and acoustic signals in the preprocessed multi-channel time series data to generate corresponding frequency domain amplitude spectra;

[0192] divide the frequency domain amplitude spectra into low-frequency, medium-frequency, and high-frequency sub-bands according to the characteristic frequency ranges corresponding to the preset fault modes;

[0193] design a band mask function based on the energy distribution baseline of each sub-band under normal working conditions, the band mask function being used for selectively enhancing or suppressing the spectral components of a specified sub-band;

[0194] apply the band mask function to the corresponding sub-band respectively to generate a weighted low-band feature spectrum, a medium-band feature spectrum and a high-band feature spectrum;

[0195] perform cross-band feature splicing on the low-band feature spectrum, the medium-band feature spectrum and the high-band feature spectrum to form a multi-dimensional spectral feature matrix;

[0196] dynamically optimize the weights of each sub-band in the multi-dimensional spectral feature matrix through an attention network to generate a multi-spectrum fusion feature.

[0197] In an embodiment, the channel attention module 40 is specifically configured to:

[0198] construct a multi-channel time series feature tensor, the channel dimension of the multi-channel time series feature tensor including a sensor signal channel in the pre-processed multi-channel time series data or a node feature channel in the graph representation;

[0199] perform linear transformation on the multi-channel time series feature tensor to generate a query matrix, a key matrix and a value matrix in the channel dimension;

[0200] calculate the similarity score of the query matrix and the key matrix to generate an inter-channel attention score matrix;

[0201] apply a learnable subspace mask to sparsify and correct the inter-channel attention score matrix to generate a corrected attention weight matrix;

[0202] perform weighted aggregation on the corrected attention weight matrix and the value matrix to generate an initial channel interaction feature tensor;

[0203] perform multi-head attention weighted fusion on the initial channel interaction feature tensor to generate a channel collaborative attention feature.

[0204] In an embodiment, the time series inference module 50 is specifically configured to:

[0205] construct a time series analysis model including a multi-level autoregressive mechanism;

[0206] perform cross-modal feature splicing on the node feature vector in the graph representation, the multi-spectrum fusion feature and the channel collaborative attention feature to generate a joint input feature vector;

[0207] modeling the time series dependence of the joint input feature vectors through a multi-level autoregressive mechanism of the time series analysis model, to generate a predicted feature vector of a future time step and a reconstructed feature vector of a current time step;

[0208] calculating a prediction error of the predicted feature vector and an actual observation value, and calculating a reconstruction error of the reconstructed feature vector and the joint input feature vector, to generate a comprehensive anomaly score;

[0209] adjusting an anomaly judgment threshold based on a distribution of historical normal working condition data, and generating an anomaly state identifier according to a comparison result of the comprehensive anomaly score and the anomaly judgment threshold, taking the anomaly state identifier as an analysis result;

[0210] when the analysis result is an anomaly state, outputting a probability distribution of a specific fault type through a classifier of the time series analysis model.

[0211] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in FIG. 1. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement the functions or steps of the server side of the multi-modal time series anomaly analysis method. Figure 4 In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in FIG. 2. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of the user terminal side of the multi-modal time series anomaly analysis method.

[0212] Figure 5 In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in FIG. 2. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of the user terminal side of the multi-modal time series anomaly analysis method.

[0213] ​In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:

[0214] Collecting multi-source sensing signals, and performing a preprocessing operation on the multi-source sensing signals to generate preprocessed multi-channel time series data;

[0215] Mapping the preprocessed multi-channel time series data into graph representation, the graph representation being used to embody the association information between monitoring objects;

[0216] Performing multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features;

[0217] Applying channel collaborative attention processing on selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel collaborative attention features;

[0218] Performing comprehensive reasoning on the graph representation, the multi-spectrum fusion features, and the channel collaborative attention features through a time series analysis model to generate an analysis result.

[0219] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0220] Collecting multi-source sensing signals, and performing a preprocessing operation on the multi-source sensing signals to generate preprocessed multi-channel time series data;

[0221] Mapping the preprocessed multi-channel time series data into graph representation, the graph representation being used to embody the association information between monitoring objects;

[0222] Performing multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features;

[0223] Applying channel collaborative attention processing on selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel collaborative attention features;

[0224] Performing comprehensive reasoning on the graph representation, the multi-spectrum fusion features, and the channel collaborative attention features through a time series analysis model to generate an analysis result.

[0225] It should be noted that the functions or steps described above with respect to the computer readable storage medium or the computer device can correspond to the related descriptions of the server side and the user side in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0226] A person of ordinary skill in the art can understand that all or part of the processes in the foregoing method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a nonvolatile computer readable storage medium. When the computer program is executed, the processes of the foregoing embodiments can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include nonvolatile and / or volatile memory. The nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0227] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0228] It should be noted that if a non-company software tool or component appears in the embodiments of the present application, it is only used for example introduction and does not represent actual use. The above described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or equivalently replace some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multi-modal temporal anomaly analysis method, characterized in that, The method comprises the following steps: Collecting multi-source sensing signals and performing preprocessing operations on the multi-source sensing signals to generate preprocessed multi-channel time series data; Mapping the preprocessed multi-channel time series data into a graph representation, which is used to reflect the correlation information between monitoring objects, comprising: defining a graph node set, which includes production line equipment unit nodes and sensor monitoring point nodes, the equipment unit nodes correspond to stamping robots, welding workstations, painting equipment and assembly lines, and the sensor monitoring point nodes are associated with each sensor signal source in the preprocessed multi-channel time series data; based on physical adjacency relationship or process flow upstream and downstream relationship, define a set of edge relationships between nodes, the physical adjacency edge represents the installation position adjacency between devices, and the process flow edge represents the material transfer or control signal dependency relationship; constructing a weighted adjacency matrix according to the edge relationship set, the element value of the weighted adjacency matrix is determined by the fault propagation strength between devices or the sensor signal correlation; based on the weighted adjacency matrix, a weighted Laplace matrix containing adjustment parameters and high-frequency suppression coefficients is calculated; associating the time series signal segment corresponding to each sensor monitoring point node in the preprocessed multi-channel time series data to the node feature vector of the sensor monitoring point node; combining the node feature vector, the weighted adjacency matrix and the weighted Laplace matrix to generate a graph representation containing time series features and topological correlation; Performing multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features, comprising: performing discrete Fourier transform on the vibration signals and acoustic signals in the preprocessed multi-channel time series data to generate corresponding frequency domain amplitude spectra; according to the feature frequency range corresponding to the preset fault mode, the frequency domain amplitude spectrum is divided into low frequency band, medium frequency band and high frequency band sub-band; based on the energy distribution baseline of each sub-band under normal working conditions, a frequency band mask function is designed, which is used to selectively enhance or suppress the spectral components of the specified sub-band; the frequency band mask function is respectively applied to the corresponding sub-band to generate weighted low frequency band feature spectrum, medium frequency band feature spectrum and high frequency band feature spectrum; cross-band feature splicing is performed on the low frequency band feature spectrum, the medium frequency band feature spectrum and the high frequency band feature spectrum to form a multi-dimensional spectrum feature matrix; through attention network, the weight of each sub-band in the multi-dimensional spectrum feature matrix is dynamically optimized to generate multi-spectrum fusion features; Applying channel collaborative attention processing to selected multi-channel feature data, the selected multi-channel feature data is derived from the preprocessed multi-channel time series data or the node feature data in the graph representation, to generate channel collaborative attention features; Through a time series analysis model, the graph representation, the multi-spectrum fusion features and the channel collaborative attention features are comprehensively inferred to generate an analysis result.

2. The multi-modal temporal anomaly analysis method of claim 1, wherein, Collecting multi-source sensing signals and performing preprocessing operations on the multi-source sensing signals to generate preprocessed multi-channel time series data, comprising: Collecting temperature sensor data, multi-axis vibration sensor data, equipment running torque data, current signals and acoustic sensor data from monitoring equipment in stamping, welding, painting and assembly links of the production line to form initial multi-source sensing signals; Performing format unification processing on the initial multi-source sensing signals to convert signals of different protocols into standardized data streams marked with uniform time stamps; Using a combination algorithm of difference filtering and Kalman filtering to perform random noise suppression on the standardized data streams, eliminate power frequency interference and environmental noise, and generate noise-reduced data streams; Aligning time stamps of the noise-reduced data streams based on a network time protocol or a synchronous trigger signal, correcting time deviation caused by transmission delay, and generating time-aligned data streams; Using linear interpolation or spline interpolation at adjacent time points to compensate for missing data in the time-aligned data streams caused by sensor failure or communication interruption, and generating compensated data streams; According to real-time signal-to-noise ratio and data stationarity evaluation results of each sensor signal in the compensated data streams, adjusting adaptive fusion weights, and generating preprocessed multi-channel time series data through weighted summation.

3. The multi-modal temporal anomaly analysis method of claim 1, wherein, Based on the weighted adjacency matrix, a weighted Laplacian matrix containing adjustment parameters and high-frequency suppression coefficients is calculated, including: Setting adjustment parameters according to the strength of the physical adjacency relationship between devices, wherein the adjustment parameter value between physically adjacent devices is set to a first preset value, and the adjustment parameter value between non-adjacent devices is set to a second preset value, the first preset value being higher than the second preset value; Obtaining the proportion of signal energy in the high frequency band in the total energy in the preprocessed multi-channel time series data, and adjusting the size of the high-frequency suppression coefficient according to the proportion; Multiplying the adjustment parameters and the original edge weight values of the weighted adjacency matrix to obtain the modified edge weight values; Calculating the degree values of each node according to the modified edge weight values to generate a degree matrix with adjustment parameters; Multiplying the degree matrix with adjustment parameters and the high-frequency suppression coefficient to generate a weighted Laplacian matrix.

4. The multi-modal temporal anomaly analysis method of claim 1, wherein, Applying channel collaborative attention processing to selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel collaborative attention features, including: Constructing a multi-channel time series feature tensor, the channel dimension of the multi-channel time series feature tensor containing sensor signal channels in the preprocessed multi-channel time series data or node feature channels in the graph representation; Performing linear transformation on the multi-channel time series feature tensor to generate a query matrix, a key matrix and a value matrix in the channel dimension; Calculating the similarity scores of the query matrix and the key matrix to generate an inter-channel attention score matrix; Applying a learnable subspace mask to sparsely correct the inter-channel attention score matrix to generate a modified attention weight matrix; Weighted aggregation of the modified attention weight matrix and the value matrix generates an initial channel interaction feature tensor; Multi-head attention weighted fusion of the initial channel interaction feature tensor generates channel collaborative attention features.

5. The multi-modal temporal anomaly analysis method of claim 1, wherein, The graph representation, the multi-spectrum fusion feature, and the channel collaborative attention feature are comprehensively inferred by a time series analysis model to generate an analysis result, including: A time series analysis model including a multi-level autoregressive mechanism is constructed; The node feature vectors in the graph representation, the multi-spectrum fusion feature, and the channel collaborative attention feature are cross-modality feature spliced to generate a joint input feature vector; The joint input feature vector is time series dependency modeled by the multi-level autoregressive mechanism of the time series analysis model to generate a predicted feature vector of a future time step and a reconstructed feature vector of a current time step; The prediction error of the predicted feature vector and the actual observation value is calculated, and the reconstruction error of the reconstructed feature vector and the joint input feature vector is calculated to generate a comprehensive anomaly score; An abnormality determination threshold is adjusted based on the distribution of historical normal working condition data, and an abnormal state identifier is generated based on the comparison result of the comprehensive anomaly score and the abnormality determination threshold, and the abnormal state identifier is taken as the analysis result; When the analysis result is an abnormal state, the probability distribution of a specific fault type is output by the classifier of the time series analysis model.

6. A multi-modal temporal anomaly analysis apparatus, characterized by, The multi-modal time series anomaly analysis device includes: A data preprocessing module is configured to collect multi-source sensing signals and perform preprocessing operations on the multi-source sensing signals to generate preprocessed multi-channel time series data; A graph construction module is configured to map the preprocessed multi-channel time series data into a graph representation, which is used to represent the association information between monitoring objects, including: defining a graph node set, which includes production line equipment unit nodes and sensor monitoring point nodes, the equipment unit nodes corresponding to stamping robots, welding workstations, painting equipment, and assembly lines, and the sensor monitoring point nodes being associated with each sensor signal source in the preprocessed multi-channel time series data; defining a set of edge relationships between nodes based on physical adjacency relationships or upstream and downstream relationships in the process flow, the physical adjacency edges representing the adjacency of installation positions between devices, and the process flow edges representing material transfer or control signal dependency relationships; constructing a weighted adjacency matrix based on the set of edge relationships, the element values of the weighted adjacency matrix being determined by the fault propagation strength between devices or the sensor signal correlation; calculating a weighted Laplacian matrix including adjustment parameters and high-frequency suppression coefficients based on the weighted adjacency matrix; associating time series signal segments corresponding to each sensor monitoring point node in the preprocessed multi-channel time series data to the node feature vectors of the sensor monitoring point nodes; and generating a graph representation including time series features and topological association relationships by combining the node feature vectors, the weighted adjacency matrix, and the weighted Laplacian matrix. The spectrum fusion module is configured to perform multi-spectrum information extraction and fusion processing on the preprocessed multi-channel time series data to generate multi-spectrum fusion features, including: performing discrete Fourier transform on the vibration signals and acoustic signals in the preprocessed multi-channel time series data to generate corresponding frequency domain amplitude spectra; dividing the frequency domain amplitude spectra into low frequency band, medium frequency band and high frequency band sub-bands according to feature frequency ranges corresponding to preset fault modes; designing a frequency band mask function based on energy distribution baselines of each sub-band under normal working conditions, the frequency band mask function being configured to selectively enhance or suppress spectral components of a specified sub-band; applying the frequency band mask function to the corresponding sub-band to generate weighted low frequency band feature spectrum, medium frequency band feature spectrum and high frequency band feature spectrum; performing cross-band feature splicing on the low frequency band feature spectrum, the medium frequency band feature spectrum and the high frequency band feature spectrum to form a multi-dimensional spectrum feature matrix; and dynamically optimizing the weights of each sub-band in the multi-dimensional spectrum feature matrix through an attention network to generate multi-spectrum fusion features. The channel attention module is configured to apply channel collaborative attention processing to selected multi-channel feature data, the selected multi-channel feature data being derived from the preprocessed multi-channel time series data or node feature data in the graph representation, to generate channel collaborative attention features. The time series reasoning module is configured to comprehensively reason the graph representation, the multi-spectrum fusion features and the channel collaborative attention features through a time series analysis model to generate an analysis result.

7. A computer device, comprising: The computer device includes a memory, a processor, and a multi-modal time series anomaly analysis program stored on the memory and executable on the processor, and the multi-modal time series anomaly analysis program, when executed by the processor, implements the steps of the multi-modal time series anomaly analysis method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a multi-modal time series anomaly analysis program, and the multi-modal time series anomaly analysis program, when executed by the processor, implements the steps of the multi-modal time series anomaly analysis method according to any one of claims 1-5.

Images