Graph neural network driven multi-modal data fusion method

Abstract

The application relates to the field of artificial intelligence and data fusion, and specifically discloses a kind of multi-modal data fusion method driven by graph neural network, which is cut into fixed time window after multi-modal observation data, and the time domain data is projected into frequency domain space by short-time Fourier transform and wavelet transform, so that the graph node represents the energy distribution of a specific frequency band rather than the sampling point of a discrete time point, avoiding the dilemma of inconsistent point-by-point sampling rate. In the frequency domain space, the graph topology structure of the same modal frequency adjacent correlation and the cross-modal resonance coupling correlation is constructed based on the physical priori, and the cross-band spectral graph attention mechanism is introduced to deeply fuse the multi-modal frequency domain features, so that the frequency domain features of different sampling rate modes can be directly interleaved and cooperated. The technical concept shifts from time domain microscopic alignment to frequency domain macroscopic wrapping paradigm transition, effectively eliminating the false features and gradient erasing problems caused by forced interpolation in the time domain, and improving the recognition sensitivity and diagnosis accuracy of early faults of industrial equipment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence and data fusion technology, and more specifically, to a graph neural network-driven multimodal data fusion method. Background Technology

[0002] With the deep development of the Industrial Internet of Things and intelligent manufacturing technologies, the health monitoring and fault diagnosis of rotating machinery increasingly rely on the collaborative sensing capabilities of multi-source heterogeneous sensors. In complex operating scenarios such as power systems, wind farms, and large industrial equipment, equipment status information is often scattered across multiple physical modes, including high-frequency vibration acceleration signals, low-frequency ambient temperature data, and variable frame rate infrared thermal imaging videos. Single-mode data is insufficient to comprehensively depict the actual operating conditions of the equipment. There is an urgent need to construct intelligent analysis methods that can deeply integrate cross-modal heterogeneous information to achieve accurate capture and timely warning of early, subtle fault signs.

[0003] Existing multimodal data fusion technologies mainly rely on traditional feature-level concatenation or decision-level voting mechanisms. While these methods integrate multi-source information to some extent, they essentially remain at a shallow fusion stage and cannot effectively uncover deep semantic relationships and physical coupling patterns between different modalities. In recent years, graph neural networks have been introduced into the field of multimodal fusion due to their powerful relation modeling capabilities. However, the data acquisition environment in industrial settings is extremely complex. The sampling rates of high-frequency and low-frequency sensors often differ by several orders of magnitude, and random data loss is caused by network latency and equipment outages. Traditional graph neural networks, based on a discrete synchronous message passing paradigm, require all nodes to be on a unified logical cross-section at the same time step. When forcibly constructing a graph topology for asynchronous and non-uniform multimodal data, the transient mutation features generated by high-frequency nodes and the stale states lingering in low-frequency nodes are incorrectly and equally weighted, resulting in severe time granularity misalignment and semantic asymmetry during cross-modal message passing. This not only dilutes the sensitivity of high-frequency data to sudden anomalies but also misleadingly assigns outdated information from low-frequency nodes to the current moment's health assessment, ultimately causing a decrease in the identification sensitivity of fault diagnosis models and an increase in false alarm rates, making it difficult to meet the stringent requirements of real-time performance and accuracy in industrial settings.

[0004] Therefore, an optimized graph neural network-driven multimodal data fusion scheme is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a graph neural network-driven multimodal data fusion method, comprising: Step 1: Perform macroscopic time window segmentation on the acquired multimodal observation data stream to obtain multimodal time window data blocks. The multimodal time window data blocks include high-frequency vibration acceleration sampling sequences, low-frequency ambient temperature sampling sequences, and variable frame rate infrared thermal imaging video streams. Step 2: Perform time-frequency domain orthogonal transformation and spectral feature extraction on the multimodal time window data blocks to obtain an initial set of frequency domain latent features; Step 3: Based on the initial set of latent features in the frequency domain, construct a frequency domain graph topology that includes a frequency domain adjacency matrix; Step 4: Using the frequency domain graph topology as the message passing path, the graph attention network is used to perform spectral collaborative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain the multimodal fusion frequency domain feature tensor. Step 5: Perform global graph pooling on the multimodal fusion frequency domain feature tensor to obtain a global graph vector representing the overall operating conditions. Feed the global graph vector into the multilayer perceptual classification network for nonlinear discrimination to output the terminal decision result.

[0006] Compared with existing technologies, this application proposes a graph neural network-driven multimodal data fusion method. First, it performs macroscopic time window segmentation on multi-source data such as high-frequency vibration, low-frequency temperature, and variable frame rate video, preserving the original sampling characteristics of each mode without forced interpolation. Then, it projects the time-domain sequence into a frequency-domain energy spectrum representation through short-time Fourier transform and wavelet transform, transforming graph network nodes from sampling points representing a certain moment into energy beams representing a certain frequency band, completely avoiding temporal conflicts caused by differences in point-by-point sampling rates. Based on this, the method constructs a frequency-domain graph topology, using physical priors to establish adjacent edges of the same modal frequency and cross-modal resonant coupling edges. Then, it uses a graph attention network to perform spectral collaborative aggregation of frequency-domain node features, enabling the harmonic characteristics of high-frequency modes and the DC slow-varying characteristics of low-frequency modes to be directly interleaved and fused in the frequency-domain topology. Finally, through global graph pooling and a multi-layer perception classification network, it outputs accurate judgment results of device health status, effectively improving the sensitivity of identifying early weak fault symptoms and reducing the false alarm rate and missed detection risk in industrial sites. Attached Figure Description

[0007] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 This is a flowchart of a graph neural network-driven multimodal data fusion method according to an embodiment of this application; Figure 2 This is a schematic diagram of data flow in a graph neural network-driven multimodal data fusion method according to an embodiment of this application; Figure 3 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which performs time-frequency domain orthogonal transformation and spectral feature extraction on multimodal time window data blocks to obtain an initial frequency domain latent feature set; Figure 4 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which constructs a frequency domain graph topology structure containing a frequency domain adjacency matrix based on an initial frequency domain latent feature set. Figure 5 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which uses a frequency domain graph topology as a message passing path and utilizes a graph attention network to perform spectral collaborative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain a multimodal fusion frequency domain feature tensor. Figure 6 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which involves aggregating the local neighbor feature buffer set with the neighbor feature space by infiltrating the physical resonance gradient to obtain the multimodal fusion frequency domain feature tensor. Detailed Implementation

[0009] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0010] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0011] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0012] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0013] Existing multimodal data fusion technologies face severe challenges when processing industrial equipment monitoring data. Traditional graph neural networks, based on a discrete synchronous message passing paradigm, suffer from serious temporal granularity misalignment and semantic asymmetry in cross-modal message passing when forcibly constructing a graph topology from asynchronous, non-uniform multimodal data with sampling rates differing by several orders of magnitude. This results in the erroneous equal-weighting of transient abrupt features of high-frequency nodes and stale states of low-frequency nodes, ultimately leading to decreased fault diagnosis sensitivity and increased false alarm rates. Therefore, this application proposes a graph neural network-driven multimodal data fusion method. This method first divides the multimodal observation data stream into a macroscopic time window with a fixed time span. Then, it projects each modality's data into the frequency domain space and uniformly maps it to a latent space of the same dimension through short-time Fourier transform and wavelet transform. This allows graph nodes to represent the energy distribution of specific frequency bands rather than discrete sampling points, thereby utilizing the overall envelopment of the frequency domain to avoid the dilemma of inconsistent point-to-point sampling rates. In the frequency domain, the method constructs a graph topology based on physical priors, establishing adjacency relationships between frequencies of the same mode and cross-modal resonance coupling relationships. It then introduces a cross-band spectral graph attention mechanism to aggregate graph attention features of each target node and its neighboring nodes based on spectral collaboration, enabling direct interleaving and fusion of frequency domain features from different sampling rate modes. This technique effectively eliminates the spurious features and gradient erasure problems caused by forced temporal interpolation, improving the sensitivity and accuracy of early fault identification and diagnosis in industrial equipment.

[0014] Figure 1 This is a flowchart of a graph neural network-driven multimodal data fusion method according to an embodiment of this application. Figure 2 This is a schematic diagram of data flow in a graph neural network-driven multimodal data fusion method according to an embodiment of this application. Figure 1 and Figure 2As shown, a graph neural network-driven multimodal data fusion method according to an embodiment of this application includes: S1, performing macroscopic time window segmentation on the acquired multimodal observation data stream to obtain multimodal time window data blocks, the multimodal time window data blocks including high-frequency vibration acceleration sampling sequences, low-frequency ambient temperature sampling sequences, and variable frame rate infrared thermal imaging video streams; S2, performing time-frequency domain orthogonal transformation and spectral feature extraction on the multimodal time window data blocks to obtain an initial frequency domain latent feature set; S3, constructing a frequency domain graph topology structure containing a frequency domain adjacency matrix based on the initial frequency domain latent feature set; S4, using the frequency domain graph topology structure as a message passing path, using a graph attention network to perform spectral collaborative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain a multimodal fusion frequency domain feature tensor; S5, performing a global graph pooling operation on the multimodal fusion frequency domain feature tensor to obtain a global graph vector representing the overall operating conditions, feeding the global graph vector into a multilayer perceptual classification network for nonlinear discrimination to output the terminal decision result.

[0015] Specifically, in step S1, the acquired multimodal observation data stream is macroscopically segmented into time windows to obtain multimodal time window data blocks. These blocks include high-frequency vibration acceleration sampling sequences, low-frequency ambient temperature sampling sequences, and variable frame rate infrared thermal imaging video streams. It should be noted that, due to the asynchronous and concurrent acquisition characteristics of high-frequency vibration sensors, low-frequency temperature sensors, and variable frame rate infrared thermal imaging equipment in industrial settings, the time progression rate and sampling density of each modal data source differ by orders of magnitude. Therefore, the technical solution of this application first performs macroscopic time window segmentation on the acquired multimodal observation data stream to obtain multimodal time window data blocks. Through this processing, heterogeneous sampling rate data intercepted within the same fixed time span can be uniformly encapsulated into a macroscopically synchronized logical container without performing any forced interpolation or frequency alignment operations, providing a clearly defined spatiotemporal input base for subsequent frequency domain orthogonal transformation.

[0016] More specifically, in a specific example of this application, for a multimodal monitoring scenario of industrial rotating equipment, the underlying listening process is first started to acquire a multimodal observation data stream including a high-frequency vibration acceleration sampling sequence, a low-frequency ambient temperature sampling sequence, and a variable frame rate infrared thermal imaging video stream. To prevent high-speed data loss and solve the microsecond-level communication jitter caused by the transmission of multiple data sources, a first-in-first-out double-ended queue with physical timestamps is opened for each mode. Each arriving discrete sampling point or image frame is marked with the current system's absolute physical timestamp and pushed into the queue of the corresponding mode for temporary storage, thereby generating and maintaining a continuous observation data buffer based on a unified time axis.

[0017] Subsequently, a preset fixed physical time span is set as the sliding slice window. This time span is five seconds in the power equipment monitoring scenario. The system iterates through the modal data queues in the continuous observation data buffer, extracting data points whose actual sampling timestamps fall completely within the current time window interval without any frequency alignment, zero-padding, or interpolation operations. Tens of thousands of sampling points are extracted in high-frequency bands, while only a few sampling points are extracted in low-frequency bands, thus aggregating to generate the original non-fixed-length data fragment set. The mathematical logic of this slice extraction operation is to extract a subset of modal data, where the data fragment extracted from the m-th modality within the k-th window is... satisfy In the formula This represents the m-th mode at its physical absolute timestamp. The original discrete sampled values ​​or image frame data at each moment. This represents the starting absolute physical time of the k-th sliding slice window. This indicates the preset fixed physical time span, i.e., the window width.

[0018] Finally, the trie mapping mechanism is invoked to structurally label the original non-fixed-length data segment set, assigning corresponding data type metadata tag headers to high-frequency vibration slices, low-frequency temperature slices, and video frame slices respectively. These non-equal-length data segments with attached tag headers and physical time span attributes are then packaged and encapsulated in parallel into a unified high-dimensional tensor list structure, forming a multimodal time window data block as a macroscopic spatiotemporal container, which is directly put into the subsequent frequency domain orthogonal transformation processing pipeline.

[0019] Specifically, in step S2, the multimodal time window data block undergoes time-frequency domain orthogonal transformation and spectral feature extraction to obtain an initial frequency domain latent feature set. It should be noted that although the multimodal time window data block generated in the preceding steps has completed the unified encapsulation of macroscopic time boundaries, it still carries essential temporal heterogeneous features: high-frequency vibrations sampled 10,000 times per second, low-frequency temperature samplesd once per second, and infrared video captured at variable frame rates. If a graph neural network topology is directly constructed in the time domain and message passing between nodes is executed, high-frequency modal nodes will carry microsecond-level transient impact information, while low-frequency modal nodes can only provide second-level slowly changing trend information. This order-of-magnitude gap in temporal resolution will lead to a complete physical mismatch in the features transmitted through cross-modal edge connections. Based on this, the technical solution of this application further performs time-frequency domain orthogonal transformation and spectral feature extraction on the multimodal time window data block to obtain an initial frequency domain latent feature set. Through the above processing, the time-domain representation originally anchored to discrete sampling points can be transformed into a frequency-domain representation anchored to the energy distribution of continuous frequency bands. This allows the subsequently constructed graph network nodes to no longer represent a snapshot of data at a certain instant but rather the cumulative energy characteristics of a specific frequency band within the entire time window. By utilizing the natural integral wrapping effect of the frequency domain space on the time dimension, the temporal granularity alignment dilemma caused by inconsistent sampling rates can be completely resolved, laying a frequency-domain isomorphic foundation for the physical rationality of cross-modal graph topology construction.

[0020] Figure 3 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which performs time-frequency domain orthogonal transformation and spectral feature extraction on multimodal time window data blocks to obtain an initial frequency domain latent feature set. Figure 3 As shown, step S2 includes: S21, performing dimensional routing allocation on the multimodal time window data area based on the metadata tag header embedded in the multimodal time window data block to obtain a heterogeneous sequence classification set; S22, extracting the frequency band energy spectrum of the one-dimensional time series component in the heterogeneous sequence classification set to obtain one-dimensional frequency band features, and packaging the one-dimensional frequency band features with the unprocessed two-dimensional visual sequence to obtain a half-frequency domain hybrid set; S23, performing spatiotemporal frequency domain orthogonal decomposition on the two-dimensional infrared video frame sequence in the half-frequency domain hybrid set to obtain a visual frequency domain tensor, and merging the visual frequency domain tensor with the one-dimensional frequency band energy matrix to obtain a full-modal frequency domain pool set; S24, aligning the heterogeneous frequency band features of each modality frequency domain data in the full-modal frequency domain pool set with the same latent space reference to obtain an initial frequency domain latent feature set.

[0021] In step S21, based on the metadata tag header embedded in the multimodal time window data block, dimensional routing is performed on the multimodal time window data area to obtain a heterogeneous sequence classification set. It should be noted that since the high-frequency vibration acceleration sampling sequence and low-frequency ambient temperature sampling sequence encapsulated within the multimodal time window data block are one-dimensional time-varying scalar signals, while the variable frame rate infrared thermal imaging video stream is a two-dimensional spatiotemporal matrix signal, the data structures of different spatial dimensions require distinctly different orthogonal decomposition operators in the subsequent frequency domain transformation stage. Based on this, the technical solution of this application further performs dimensional routing on the multimodal time window data area based on the metadata tag header embedded in the multimodal time window data block to obtain a heterogeneous sequence classification set. Through the above processing, the mixed-encapsulated heterogeneous dimensional data can be accurately decoupled and channel isolated according to spatial topological characteristics, providing structured input preparation for subsequent differentiated calls to short-time Fourier transform for one-dimensional sequences and discrete wavelet transform for two-dimensional sequences.

[0022] More specifically, in a concrete example of this application, for multimodal monitoring data of rotating machinery in power equipment, the parsing engine first reads the metadata tag header embedded in the multimodal time window data block. This tag header has already labeled each modal slice with data type attributes and spatial dimension information during the pre-encapsulation stage. The parsing engine determines the spatial topology of the data based on the dimension identifier field in the tag header, extracting the high-frequency vibration acceleration sampling sequence and the low-frequency ambient temperature sampling sequence, labeled as one-dimensional time-varying signals, from the data block and transferring them to the one-dimensional processing bus. This bus is specifically connected to the subsequent short-time Fourier transform power spectral density calculation module. Simultaneously, the variable frame rate infrared thermal imaging video stream, labeled as two-dimensional spatiotemporal signals, is extracted from the data block and transferred to the multi-dimensional vision processing bus. This bus is specifically connected to the subsequent three-dimensional fast Fourier transform or discrete wavelet transform spatial frequency decomposition module. The unpacked and split data sequences are loaded into a heterogeneous sequence classification set containing type guidance. This set is organized using a trie mapping structure, with dimension identifiers as index keys and corresponding data tensor lists as storage entities. This ensures that subsequent frequency domain transformation steps can automatically match the corresponding orthogonal decomposition operators based on the data dimension characteristics without performing additional type judgment operations.

[0023] In step S22, the one-dimensional time series component in the heterogeneous sequence classification set is subjected to frequency band energy spectrum extraction to obtain one-dimensional frequency band features. These one-dimensional frequency band features are then packaged with the unprocessed two-dimensional visual sequence to obtain a semi-frequency domain hybrid set. It should be noted that since the one-dimensional time series component and the two-dimensional visual sequence component in the heterogeneous sequence classification set require the use of orthogonal decomposition operators of different dimensions during the frequency domain transformation, and the computational complexity of the one-dimensional discrete Fourier transform is much lower than that of the two-dimensional or three-dimensional fast Fourier transform, a staged processing strategy can optimize computational resource allocation. Based on this, the technical solution of this application further performs frequency band energy spectrum extraction on the one-dimensional time series component in the heterogeneous sequence classification set to obtain one-dimensional frequency band features, and packages these features with the unprocessed two-dimensional visual sequence to obtain a semi-frequency domain hybrid set. Through the above processing, the one-dimensional modal features that have undergone frequency domain projection and the two-dimensional modal data to be processed can be organized into a transitional hybrid container, providing structured input for the subsequent unified spatiotemporal frequency domain decomposition of the two-dimensional visual sequence.

[0024] More specifically, in a concrete example of this application, for high-frequency vibration acceleration sampling sequences and low-frequency ambient temperature sampling sequences in a power equipment rotating machinery monitoring scenario, a one-dimensional time series component is first extracted from a heterogeneous sequence classification set. To mitigate the spectral leakage caused by finite time truncation, it is smoothed at time boundaries by multiplying it point-by-point by a Hamming window function. Then, the discrete Fourier transform kernel operator is called to project the discrete time-domain amplitude sequence into a power spectral density distribution in the frequency domain. The core mathematical expression of this transformation process is: in, This represents the power spectral density energy value of the m-th one-dimensional mode in the k-th discrete frequency cluster. This indicates the total number of discrete sampling points actually contained in the one-dimensional modal sequence within the current time window. This represents the nth original discrete sampled value in the time-domain sequence of this modality. This indicates that the Hamming window function applied to the time-domain sequence is used to smooth the time boundaries. The imaginary unit is represented. The power spectral density calculated by this formula transforms the time-domain sequence into a fixed-length frequency-domain feature vector, changing the data representation from depending on the sampling time to depending on the frequency band energy distribution. The transformed one-dimensional frequency band features are then repackaged with the two-dimensional infrared video frame sequence that has not yet undergone frequency domain processing. A nested list structure is used to store the one-dimensional modal spectrum matrix that has completed frequency domain projection and the two-dimensional modal tensor that retains the original spatiotemporal shape side by side, generating a semi-frequency domain hybrid set with transitional properties as the input base for the subsequent spatiotemporal frequency domain decomposition of the two-dimensional visual sequence.

[0025] In step S23, the two-dimensional infrared video frame sequence in the half-frequency domain hybrid set is subjected to spatiotemporal frequency domain orthogonal decomposition to obtain a visual frequency domain tensor. The visual frequency domain tensor is then merged with a one-dimensional frequency band energy matrix to obtain a full-modal frequency domain pool set. It should be noted that since the two-dimensional infrared video frame sequence in the half-frequency domain hybrid set carries the temperature field texture distribution of the device surface in the spatial dimension and the dynamic evolution process of the thermal field in the temporal dimension, a simple two-dimensional spatial Fourier transform cannot capture the frequency characteristics on the time axis. Therefore, a spatiotemporal joint three-dimensional frequency domain decomposition method is required. Based on this, the technical solution of this application further performs spatiotemporal frequency domain orthogonal decomposition on the two-dimensional infrared video frame sequence in the half-frequency domain hybrid set to obtain a visual frequency domain tensor, and then merges the visual frequency domain tensor with a one-dimensional frequency band energy matrix to obtain a full-modal frequency domain pool set. Through the above processing, the spatial texture abrupt features and time-varying features of the two-dimensional visual modality can be simultaneously projected into the frequency domain space, and converged with the one-dimensional modal features that have completed frequency domain conversion to form a frequency domain feature pool with unified dimensions, providing a complete node feature base for subsequent construction of cross-modal frequency domain graph topology.

[0026] More specifically, in a concrete example of this application, for infrared thermal imaging monitoring video of the rotating mechanical surface of power equipment, the two-dimensional infrared video frame sequence that has not yet undergone frequency domain transformation is first extracted from the half-frequency domain mixed set. To simultaneously capture the spatial texture abrupt changes (i.e., the spatial high-frequency components) and the slowly varying dynamics of the temperature field (i.e., the temporal low-frequency components) of the equipment surface during the transformation, a three-dimensional fast Fourier transform is invoked to orthogonally decompose the spatiotemporal voxel data. The core mathematical expression of this decomposition process is: in, This represents the spectral amplitude and phase information of the generated three-dimensional frequency domain complex tensor at the horizontal spatial frequency point u, the vertical spatial frequency point v, and the time frequency point w. and These represent the upper limits of the physical pixel resolution in the horizontal and vertical directions of the visual infrared image plane, respectively. This indicates the total number of infrared video frames fully contained within the current time window. This represents the physical pixel grayscale value at the two-dimensional coordinate position of the t-th frame image. Through this three-dimensional Fourier transform, the video data, originally scattered in the spatiotemporal domain, is converted into a three-dimensional complex frequency feature tensor containing both spatial and temporal frequencies. This tensor can simultaneously characterize the spatial distribution frequency characteristics of local hotspots on the device surface and the temporal evolution frequency characteristics of the overall temperature rise. After the conversion, the visual frequency domain tensor is merged with the one-dimensional frequency band energy matrix generated in the previous step. A modal index dictionary structure is used to organize the frequency domain features from different physical sources into a unified full-modal frequency domain pool set. All data in this set have completed the transition from the time domain to the frequency domain, providing a consistent source of node features for subsequent frequency domain graph topology construction.

[0027] In step S24, the frequency domain data of each modality in the full-modal frequency domain pool set are aligned with the same latent space reference for heterogeneous frequency band features to obtain an initial set of frequency domain latent features. It should be noted that, due to the different modal dimensions, the frequency domain data in the full-modal frequency domain pool set exhibits severely inconsistent absolute scales and tensor shapes. The spectrum of one-dimensional vibration and temperature modes is represented as a one-dimensional array, while the spectrum of two-dimensional infrared video modes is represented as a three-dimensional tensor. Directly using these heterogeneous frequency domain features as node inputs to a graph neural network will lead to dimensionality mismatch errors during message passing. Therefore, the technical solution of this application further aligns the frequency domain data of each modality in the full-modal frequency domain pool set with the same latent space reference for heterogeneous frequency band features to obtain an initial set of frequency domain latent features. Through the above processing, spectral matrices of different shapes can be mapped to a high-dimensional joint latent space with a unified dimension, ensuring that all nodes in the subsequently constructed frequency domain graph topology have comparable and uniformly distributed feature representations, thus removing heterogeneous obstacles for the effective execution of the cross-modal graph attention mechanism.

[0028] More specifically, in a concrete example of this application, for multimodal frequency domain data in a power equipment rotating machinery monitoring scenario, a customized multilayer sensor latent feature mapping channel is assigned to each mode. During processing, the spectral matrices of different shapes are first reduced in dimension and stretched into one-dimensional normal feature vectors through a flattening operation. This operation combines an array of one-dimensional vibration spectra with a length of 512 with a three-dimensional infrared spectrum of size... All tensors are converted into one-dimensional column vectors. Then, an affine transformation is performed using a learnable weight matrix and bias terms, and finally, a nonlinear activation function maps them to a high-dimensional joint latent space with a uniform dimension. The core mathematical expression of this mapping process is: ,in This indicates that the reference feature vector generated in the shared latent space after the m-th modality alignment mapping has a fixed length equal to the preset model dimension value. This represents a nonlinear activation function layer in a feedforward neural network, such as the ReLU or GELU operator. This represents the learnable fully connected mapping weight matrix specifically designed for the m-th modality. The tensor flattening operator flattens and concatenates any high-dimensional matrix into a one-dimensional column vector. This represents the native frequency band feature block of the m-th mode retrieved from the full-modal frequency domain pool set. This represents the bias term constant vector corresponding to the fully connected mapping of the m-th mode. Through this mapping mechanism, the originally different frequency domain features are uniformly projected into a latent space of dimension 512. The feature vectors of all modes have the same length and numerical scale, forming an initial set of frequency domain latent features with perfectly aligned dimensions and comparable numerical distribution. Each feature vector in this set will serve as the attribute representation of a single node in the subsequent frequency domain graph topology.

[0029] Specifically, in step S3, a frequency domain graph topology containing a frequency domain adjacency matrix is ​​constructed based on the initial frequency domain latent feature set. It should be noted that although the initial frequency domain latent feature set generated in the preceding steps has undergone dimensional alignment and numerical scale normalization, its individual frequency domain feature vectors still appear as independent discrete entities, lacking structured connection information capable of characterizing the physical correlation between different frequency bands and cross-modal frequency coupling relationships. The message passing mechanism of graph neural networks relies on the topological connections between nodes to achieve directed information flow and aggregation. Therefore, the technical solution of this application further constructs a frequency domain graph topology containing a frequency domain adjacency matrix based on the initial frequency domain latent feature set. Through the above processing, discrete tensors in the ordinary feature space can be converted into node objects that can be processed by the graph computing framework. Furthermore, through physical prior constraints, frequency-adjacent association edges within the same mode and resonant coupling association edges across modes are established, forming a frequency domain topology network carrying knowledge in the field of rotating machinery fault diagnosis. This provides structured path navigation for the subsequent cross-modal information transmission and deep fusion of the graph attention network.

[0030] Figure 4 This is a flowchart illustrating the construction of a frequency domain graph topology structure, including a frequency domain adjacency matrix, based on an initial frequency domain latent feature set, according to an embodiment of this application's graph neural network-driven multimodal data fusion method. Figure 4 As shown, step S3 includes: S31, traversing each independent frequency band feature vector in the initial frequency domain latent feature set, assigning it a globally unique identifier, and binding its mode type, center frequency value, and latent space vector as attributes to the node object to obtain a frequency domain undirected node set; S32, performing a kernel mapping of intra-modal frequency band adjacency and cross-modal resonance weights on the frequency domain undirected node set to obtain a fully connected topological weight tensor; S33, performing topological sparsification based on confidence truncation on the fully connected topological weight tensor to obtain the frequency domain graph topology structure.

[0031] In step S31, each independent frequency band feature vector in the initial frequency domain latent feature set is traversed, a globally unique identifier is assigned to it, and its mode type, center frequency value, and latent space vector are bound to the node object as attributes to obtain a frequency domain undirected node set. It should be noted that since the feature vectors in the initial frequency domain latent feature set are still in ordinary tensor space, they lack the structured meta-information such as node identification, topological location, and attribute encapsulation required by the graph computing framework, and cannot be directly used as input objects for message passing operations in graph neural networks. Based on this, the technical solution of this application further traverses each independent frequency band feature vector in the initial frequency domain latent feature set, assigns a globally unique identifier to it, and binds its mode type, center frequency value, and latent space vector to the node object as attributes to obtain a frequency domain undirected node set. Through the above processing, tensors in ordinary feature space can be converted into node entities that can be processed by the graph computing framework, providing a structured foundation for subsequently constructing the connection relationships between nodes based on physical priors.

[0032] More specifically, in a concrete example of this application, for an initial frequency domain latent feature set that has been dimensionally aligned in a power equipment rotating machinery monitoring scenario, a graph space instantiation engine is initiated to traverse the set. The engine visits each independent frequency band feature vector in the set one by one. These feature vectors correspond to the 100Hz and 500Hz frequency band features of the high-frequency vibration mode, the 0.1Hz frequency band features of the low-frequency temperature mode, and the spatial high-frequency edge features and temporally low-frequency slowly varying features of the infrared image mode. For each visited feature vector, a globally unique identifier is first generated as the node's identity credential in the graph topology. This identifier uses an incrementing integer encoding method to ensure that there is no duplication throughout the entire graph structure. Subsequently, the source information of the feature vector is extracted, including the physical sensor mode type it belongs to, such as vibration mode, temperature mode, or visual mode, and the value of the center frequency point represented by the feature vector. For a one-dimensional vibration spectrum, the center frequency point is the physical frequency value corresponding to the frequency cluster index after discrete Fourier transform; for a three-dimensional infrared spectrum, the center frequency point is a composite representation of spatial and temporal frequencies. The globally unique identifier, modal type label, center frequency value, and latent space vector data of the feature vector itself are bound as attribute fields to the newly created node object. This node object is organized using a dictionary data structure, with the attribute name as the key and the corresponding data as the storage value. After instantiating a single node object, it is added to the frequency domain undirected node set. This set uses a list structure to maintain all created node objects. After traversal, the frequency domain undirected node set contains node entities that completely correspond to the number of feature vectors in the initial frequency domain latent feature set. Each node carries complete attribute information but has not yet established any connections with each other, presenting a discretely distributed undirected node state.

[0033] In step S32, the set of undirected nodes in the frequency domain is mapped using intramodal band adjacency and cross-modal resonance weights to obtain a fully connected topological weight tensor. It should be noted that although the node entities in the set of undirected nodes in the frequency domain have completed attribute encapsulation, no edge relationships have been established between them. Therefore, there is a lack of topological connection weights that can guide the message passing direction and information flow intensity of the graph neural network. Furthermore, the aggregation operation of the graph attention network relies on the neighborhood range and edge weight allocation defined by the adjacency matrix. Based on this, the technical solution of this application further maps the set of undirected nodes in the frequency domain using intramodal band adjacency and cross-modal resonance weights to obtain a fully connected topological weight tensor. Through the above processing, based on the frequency smooth transition prior and cross-modal energy resonance coupling mechanism in the physics of rotating machinery vibration, node pairs can be assigned physically meaningful connection strength values, forming a dense fully connected weight matrix that provides the original input for subsequent confidence truncation sparsity.

[0034] More specifically, in a concrete example of this application, for a set of undirected nodes in the frequency domain in a power equipment rotating machinery monitoring scenario, an edge weight relationship calculator is activated to initialize the connection strength between all node pairs. This calculator iterates through any two nodes i and j in the node set, first reading the modal type attributes of the two nodes to determine whether they belong to the same physical sensor source. For two nodes within the same physical mode, such as a 100Hz frequency band node and a 200Hz frequency band node both originating from high-frequency vibration modes, the weight of the adjacent frequency association edge is calculated using prior physical common sense. This weight is based on the Euclidean distance between the center frequencies of the two nodes and the Gaussian kernel function to measure the smoothness of their frequency band leakage or natural transition. The physical implication is that adjacent frequency bands exhibit energy diffusion and spectral leakage phenomena in the frequency domain space, thus possessing a natural correlation. For two nodes spanning different sensor modes, such as a 500 Hz frequency band node from a high-frequency vibration mode and a spatial high-frequency edge feature node from an infrared vision mode, the cross-modal resonance coupling correlation edge weight is calculated. This weight measures the potential physical resonance or energy concurrency relationship by calling the absolute Pearson correlation coefficient between the high-dimensional latent spatial feature vectors of the two nodes. The physical implication is that the high-frequency components of the mechanical vibration of rotating equipment often excite the local temperature rise response of the corresponding spatial frequency on the surface of the equipment. The mathematical logic of the above relation edge weight calculation is as follows: in, This represents the initial contiguous edge weight values ​​between node i and node j generated in the fully connected topology weight tensor, and the values ​​are restricted to the range of 0 to 1. and These represent the specific physical center frequency values ​​represented by nodes i and j, respectively. This attribute is only used for Gaussian smoothing when the modes are the same. The prior Gaussian broadband tolerance coefficient, representing the connection between adjacent frequencies of the same mode, is used to control the smoothness of the transition between frequencies within the same mode. This represents the mapping function for obtaining the sensor modality classification label of a node. Let represent the covariance between the latent space feature column vectors of node i and node j. and ...

[0035] In step S33, the fully connected topological weight tensor is subjected to topological sparsification based on confidence truncation to obtain the frequency domain graph topology. It should be noted that since the fully connected topological weight tensor contains the edge weight calculation results for all node pairs within the node set, in the scenario of monitoring rotating machinery in power equipment, if the number of nodes reaches 200, the fully connected graph will contain approximately 20,000 edges. This dense connection not only brings extremely high computational redundancy, but more importantly, a large number of low-weight edges originate from the pseudo-synchronization effect generated by random Gaussian white noise in the industrial field rather than real physical associations. Based on this, the technical solution of this application further performs topological sparsification based on confidence truncation on the fully connected topological weight tensor to obtain the frequency domain graph topology. Through the above processing, noisy edge connections that do not carry physical causal information can be eliminated, while retaining effective edges that truly possess frequency smooth transition characteristics or cross-modal resonant coupling relationships, forming a sparse graph topology that provides a simplified path for efficient message passing in subsequent graph attention networks.

[0036] More specifically, in a concrete example of this application, a hard threshold truncation operator with a masking mechanism is triggered to perform sparsification on the fully connected topological weight tensor generated in the scenario of monitoring rotating machinery of power equipment. This operator first sets an empirical confidence level based on the noise floor level of the industrial site to eliminate a hard threshold. In the scenario of rotating equipment vibration monitoring, this threshold is set to 0.35, which physically means that only edge connections with weights greater than 0.35 are retained to ensure the physical reliability of the connections. Subsequently, the operator iterates through each edge weight in the fully connected topological weight tensor, setting associated edges with weights below 0.35 to 0, and keeping the original weight values ​​unchanged for associated edges with weights greater than or equal to 0.35. The mathematical operation logic of this sparsification truncation and adjacency mapping is represented as follows: in, This represents the value of the graph adjacency correlation matrix in the i-th row and j-th column, which is encapsulated within the final frequency domain graph topology entity. This represents the initial dense weights of the input fully connected topological weight tensor at the corresponding positions. The empirical confidence level, set based on the industrial noise floor level, is used to remove hard thresholds to block weak, sporadic correlations between non-resonant frequencies. This truncation operation reduces the fully connected graph with 20,000 edges to a sparse graph with only about 3,000 effective edges, decreasing the edge density from 100% to 15%. The sparsed node adjacency matrix and its associated non-zero edge weights are then instantiated and encapsulated. A compressed sparse row format is used to store the adjacency matrix to save memory, while an edge weight dictionary is maintained, using node pair indices as keys and corresponding weight values ​​as storage entities. Finally, the frequency domain graph topology is packaged and output as a static road network map for subsequent graph attention network message passing. In this topology, the number of neighbors for each node is reduced from 199 to an average of 15, reducing the computational complexity of subsequent graph convolution operations by approximately 85% while preserving physical relationships.

[0037] Specifically, in step S4, using the frequency domain graph topology as the message passing path, a graph attention network is used to perform spectral-cooperative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain a multimodal fused frequency domain feature tensor. It should be noted that, given that the preceding steps have completed the paradigm shift from the time domain to the frequency domain and constructed a frequency domain graph topology carrying physical priors, the feature representations of each node in the initial frequency domain latent feature set are still independent. Cross-modal information interaction and collaborative enhancement have not yet been achieved through the neighborhood relationships defined by the graph structure. The core advantage of graph neural networks lies in achieving directed flow and aggregation of features between nodes through message passing mechanisms connected along topological edges. Based on this, the technical solution of this application further uses the frequency domain graph topology as the message passing path and a graph attention network to perform spectral-cooperative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain a multimodal fused frequency domain feature tensor. Through the above processing, the 100 Hz harmonic feature nodes of the high-frequency vibration mode, the 0.1 Hz slowly varying feature nodes of the low-frequency temperature mode, and the spatial high-frequency texture feature nodes of the infrared vision mode can achieve deep interaction in the frequency domain topology through attention-weighted message passing. The energy distribution information that was originally scattered in the independent frequency bands of each mode is adaptively fused according to the physical resonance relationship and frequency coupling strength, generating a fused feature representation that simultaneously contains cross-modal cooperative characteristics and overall frequency domain encapsulation. This provides a high-quality multimodal fused frequency domain feature tensor for subsequent global graph pooling and terminal decision-making.

[0038] Figure 5This document describes a graph neural network-driven multimodal data fusion method according to an embodiment of this application. It uses a frequency domain graph topology as the message passing path and a graph attention network to perform spectral collaborative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain a multimodal fused frequency domain feature tensor. (See flowchart for example.) Figure 5 As shown, step S4 includes: S41, performing linear high-dimensional projection of frequency domain features and neighbor path extraction on the initial frequency domain latent feature set and frequency domain graph topology to obtain a set of local projected feature pairs; S42, calculating and normalizing the cross-modal spectral domain attention cross-correlation mask on the set of local projected feature pairs to obtain a normalized spectral graph attention mask and a set of local neighbor feature buffers; S43, based on the normalized spectral graph attention mask, performing neighbor feature space aggregation with infiltration of physical resonance gradient on the set of local neighbor feature buffers to obtain a multimodal fused frequency domain feature tensor.

[0039] In step S41, the initial frequency domain latent feature set and frequency domain graph topology are subjected to linear high-dimensional projection of frequency domain features and neighbor path extraction to obtain a set of local projected feature pairs. It should be noted that since the message passing mechanism of graph attention networks relies on nonlinear transformation of node features to enhance their expressive power in high-dimensional space, it is also necessary to clarify the pairing relationship between each central node and its neighboring nodes to guide subsequent attention weight calculation and feature aggregation operations. Based on this, the technical solution of this application further performs linear high-dimensional projection of frequency domain features and neighbor path extraction on the initial frequency domain latent feature set and frequency domain graph topology to obtain a set of local projected feature pairs. Through the above processing, the original 512-dimensional frequency domain node features can be projected to a higher-dimensional representation space to improve the nonlinear fitting ability of the features, and the effective neighboring nodes of each central node are accurately extracted based on the sparse adjacency matrix in the frequency domain graph topology, forming a center-neighbor feature pairing data stream containing navigation direction to provide structured input for subsequent cross-modal attention weight calculation.

[0040] More specifically, in a concrete example of this application, for the initial frequency domain latent feature set and frequency domain graph topology in the monitoring scenario of rotating machinery in power equipment, after the aggregation engine starts, it first generates a linear translation weight matrix with globally shared characteristics. This matrix is ​​set to a dimension of 512 x 768, used to map the 512-dimensional input feature vector to a 768-dimensional high-dimensional representation space. Then, it iterates through each frequency domain node feature in the initial frequency domain latent feature set, performing a forward linear multiplication operation on it—that is, multiplying the 512-dimensional feature vector by the 512 x 768 weight matrix—to obtain a 768-dimensional high-dimensional projected feature vector. This linear projection operation can expand the feature expression dimension while preserving the semantic information of the features, providing a richer feature space for subsequent nonlinear attention mechanisms.

[0041] After completing the high-dimensional projection of all nodes, the static frequency domain graph topology is parsed to extract the adjacency relationships between nodes. This topology uses a compressed sparse row format to store the adjacency matrix. By reading the positions of non-zero elements in the matrix, the cross-modal effective neighbor nodes connected to each central node can be quickly located. In the power equipment monitoring scenario, for a central node representing the 200 Hz frequency band of high-frequency vibration modes, its neighbor nodes include 100 Hz frequency band nodes and 300 Hz frequency band nodes of the same mode, as well as high-frequency texture feature nodes of the infrared visual modal space across modes. After extracting the neighbor node index list of the central node, the corresponding feature vectors are called from the high-dimensional projection feature set. The 768-dimensional projection features of the central node are combined and paired with the 768-dimensional projection features of each of its neighbor nodes to form a data structure containing a quadruple of central node index, neighbor node index, central node projection feature, and neighbor node projection feature. All center-neighbor pair combinations are loaded into a local projection feature pair set, which is organized using a nested list structure. The outer list is indexed by the center node, and the inner list stores all neighbor pair information of the center node. This ensures that the subsequent attention weight calculation can efficiently access the neighborhood feature pairs of any center node without repeatedly performing graph topology query operations.

[0042] In step S42, a cross-modal spectral domain attention cross-correlation mask is calculated and normalized on the set of local projected feature pairs to obtain a normalized spectral graph attention mask and a local neighbor feature buffer set. It should be noted that although the center-neighbor feature pairs contained in the set of local projected feature pairs have undergone high-dimensional projection, the relative importance of different neighbor nodes to the center node has not yet been quantified. The core mechanism of graph attention networks lies in distinguishing the contribution of different neighbor nodes in the information aggregation process through adaptively learned attention weights. Based on this, the technical solution of this application further calculates and normalizes a cross-modal spectral domain attention cross-correlation mask on the set of local projected feature pairs to obtain a normalized spectral graph attention mask and a local neighbor feature buffer set. Through the above processing, a normalized attention weight scalar can be calculated for each neighbor node of each center node. This weight reflects the contribution strength of the neighbor node features to the center node feature update, while simultaneously stripping and caching the high-dimensional feature vectors of the neighbor nodes as input data for subsequent aggregation operations.

[0043] More specifically, in a concrete example of this application, for a set of local projection feature pairs in a power equipment rotating machinery monitoring scenario, within each pair, the 768-dimensional high-dimensional projection feature of the central node and the 768-dimensional high-dimensional projection features of the neighboring nodes are concatenated along the dimensional lines to generate a 1536-dimensional concatenated feature long vector. This vector is then multiplied by a 1536-dimensional learnable attention parameter vector, and a LeakyReLU activation function is applied to obtain the original attention resonance score. Finally, the Softmax normalization function is used to transform the discrete scores of the target node and all its neighboring nodes into a probability distribution. The core mathematical expression of this normalization process is: in, This represents the normalized attention weight scalar of node i to its neighbor node j, with values ​​ranging from 0 to 1 and having the property of locally accumulating to 1. This represents the transpose of the learnable attention parameter vector. This represents the globally shared feature-upgrading linear mapping weight matrix. and Let represent the frequency domain feature vectors of the center node and its neighboring nodes, respectively. This represents the vector concatenation operator. Let represent the set of all first-order neighbor nodes of node i. The normalized weights are encapsulated in the normalized spectral attention mask, while the 768-dimensional high-dimensional projected feature vectors of the neighbor nodes are stripped and encapsulated in the local neighbor feature buffer set for use in subsequent aggregation steps.

[0044] In step S43, based on the normalized spectral graph attention mask, the local neighbor feature buffer set is aggregated in the neighbor feature space by infiltrating the physical resonance gradient to obtain the multimodal fused frequency domain feature tensor. It should be noted that, given that the previous steps have already completed the calculation of the normalized spectral graph attention mask and cached the high-dimensional feature vectors of neighbor nodes in the local neighbor feature buffer set, but these attention weights and neighbor features have not yet undergone actual weighted aggregation to generate the fused node feature representation, and the essence of information transmission in graph neural networks lies in achieving collaborative updating of node features through attention-weighted neighborhood feature aggregation. Based on this, the technical solution of this application further uses the normalized spectral graph attention mask to aggregate the local neighbor feature buffer set in the neighbor feature space by infiltrating the physical resonance gradient to obtain the multimodal fused frequency domain feature tensor. Through the above processing, the 200 Hz frequency band nodes of the high-frequency vibration mode can absorb the frequency smooth transition information of the 100 Hz and 300 Hz neighboring nodes of the same mode and the resonance coupling information of the cross-modal infrared vision high-frequency texture nodes through the attention weighting mechanism. The frequency domain energy features originally scattered in each neighboring node are adaptively fused according to the physical correlation strength to generate a multimodal fused frequency domain feature tensor that simultaneously contains intramodal frequency continuity and cross-modal resonance synergy, providing a high-quality fused feature basis for subsequent global graph pooling.

[0045] Figure 6 This is a flowchart illustrating a graph neural network-driven multimodal data fusion method according to an embodiment of this application, which involves aggregating the local neighbor feature space by infiltrating the physical resonance gradient into the neighbor feature space of a local neighbor feature buffer set to obtain a multimodal fusion frequency domain feature tensor. For example... Figure 6 As shown, step S43 includes: S431, performing harmonic order coupling relationship detection on the local neighbor feature buffer set to obtain the harmonic order coupling coefficient matrix; S432, performing dual-channel gated attention modulation and re-normalization on the harmonic order coupling coefficient matrix and the normalized spectrum attention mask to obtain the harmonic modulation attention weight; S433, based on the harmonic modulation attention weight, performing harmonic enhancement feature aggregation and nonlinear activation on the local neighbor feature buffer set to obtain the multimodal fusion frequency domain feature tensor.

[0046] In step S431, harmonic order coupling relationship detection is performed on the local neighbor feature buffer set to obtain the harmonic order coupling coefficient matrix. It should be noted that, given that the weights of the normalized spectral attention mask in the first embodiment are entirely generated by data-driven vector similarity learning, it cannot perceive the existence of harmonic order coupling relationships, nor can it distinguish between harmonic coupled neighbors and non-harmonic random neighbors during the aggregation process. However, in the rotating machinery fault diagnosis scenario, when pitting defects occur on the outer ring of a rolling bearing or microcracks occur on the gear meshing surface, the fault impact excitation will, with the fundamental frequency as the energy source, undergo cascaded energy leakage and resonant amplification along strictly integer harmonic directions at multiples of 2, 3, to n. Low-order harmonics carry the main energy of the fault impact, while the energy of higher-order harmonics exhibits an exponential decay at the physical level as the order increases. This harmonic order coupling relationship between frequency components is the most fundamental and diagnostically valuable structural law in the physics of rotating machinery vibration. Based on this, the technical solution of this application further performs harmonic order coupling relationship detection on the local neighbor feature buffer set to obtain the harmonic order coupling coefficient matrix. Through the above processing, the physical structure of harmonic orders, which is hidden in the frequency domain graph space topology and cannot be automatically captured by the data-driven attention mechanism, can be explicitly quantized and extracted. This provides the input basis for the physical channel side of the subsequent dual-channel attention modulation, fundamentally making up for the blind spot of the first embodiment in terms of the perception of harmonic coupling relationship, and enabling the aggregation process to distinguish between harmonic neighbors that carry fault causal information and random noise neighbors that do not carry causal information.

[0047] More specifically, in a concrete example of this application, for a local neighbor feature buffer set in a power equipment rotating machinery monitoring scenario, a learnable frequency surrogate extraction vector is introduced. This vector is set to 768 dimensions, consistent with the dimension of the node's high-dimensional projection features. End-to-end training is used to adaptively capture the frequency location information inherent in the frequency domain node features. The inner product operation is performed on each node's high-dimensional projection feature in the local neighbor feature buffer set, organized and stored in a center-neighbor pairing structure, and the absolute value is taken. This extracts and reduces the frequency indication information implicit in the high-dimensional latent space to a one-dimensional frequency surrogate scalar. For each center-neighbor node pair, using the center node's frequency surrogate scalar as the benchmark and the neighbor node's frequency surrogate scalar as the measured object, the frequency surrogate ratio between the two is calculated. The core mathematical expression of this process is: in, This represents the frequency proxy ratio between the central node i and its neighboring node j. This indicates that the learnable frequency proxy extraction vector is adaptively captured through end-to-end training to capture the frequency location information contained in the features of frequency domain nodes. and These represent the pairing structure where the high-dimensional feature vectors of the central node and its neighboring nodes, after projection onto the globally shared weight matrix, both originate from the local neighbor feature buffer set. This represents a numerical stability constant to prevent the denominator from being zero. If the frequency surrogate ratio is close to a positive integer n, it indicates that the frequency band represented by the neighboring node and the frequency band represented by the center node form an nth-order harmonic overtone relationship. After obtaining the frequency surrogate ratio, a harmonic nearest neighbor kernel function with dual physical constraints is applied to it to evaluate the degree to which the ratio deviates from the nearest positive integer and to jointly consider the order attenuation effect. In the engineering practice of rotating machinery vibration analysis, the energy of the fault impact from the fundamental frequency to higher harmonics exhibits a clear decreasing law, and the energy carried by the second-order harmonic is much higher than that of the fifth or sixth-order harmonics and above. Injecting this physical law into the calculation logic of the coupling coefficient can effectively suppress noise interference introduced by the coincidence of higher-order spurious harmonics. The calculation process of the dual-constraint harmonic coupling coefficient is expressed as follows: in, This represents a scalar value representing the harmonic coupling strength between node i and its neighbor node j in the harmonic order coupling coefficient matrix, with a value between 0 and 1. The rounding operator, which rounds to the nearest positive integer, is used to anchor the target harmonic order corresponding to the frequency ratio. The order attenuation factor represents the physical deceleration rate that controls the coupling strength of higher-order harmonics. The larger the value, the faster the contribution of higher-order harmonics attenuates. This represents the tolerance range of the Gaussian kernel when the harmonic tolerance bandwidth control frequency ratio deviates from a precise integer. In engineering terms, this corresponds to the allowable frequency drift range caused by speed fluctuations or load disturbances in rotating equipment. The first term in the formula is the order attenuation factor, representing the objective law that the physical energy of higher-order harmonics monotonically decreases with increasing order. The second term, the Gaussian exponent kernel, is the harmonic nearest neighbor determination factor, representing the physical fact that the closer the frequency ratio is to a precise integer multiple, the higher the confidence level of harmonic coupling. Multiplying these two terms constitutes a double-constraint gating, ensuring that only low-order and precisely matched harmonic coupling relationships achieve high coupling coefficients.

[0048] In the specific monitoring scenario of rotating machinery in power equipment, for the center node representing the 100 Hz fundamental frequency of the high-frequency vibration mode, the calculated frequency proxy ratio between it and its 200 Hz neighboring node is 1.98. After rounding, this is anchored to the second harmonic relationship, and the order attenuation factor is substituted into the value. The value of 0.5 is related to the harmonic tolerance bandwidth. After setting the value to 0.15, the calculated harmonic coupling coefficient is 0.82, indicating a strong second-order harmonic coupling relationship between the neighboring node and the center node. However, for the same center node and its 650 Hz neighbor node, the frequency surrogate ratio is 6.43. Although this is close to the 6th or 7th order harmonics, the high order leads to an increase in the denominator of the order attenuation factor, and the ratio deviates from an integer, causing Gaussian kernel attenuation. The final calculated harmonic coupling coefficient is only 0.12, indicating that the coupling strength of higher-order harmonics is effectively suppressed by physical constraints. By traversing all center-neighbor node pairs and performing the above calculations, the obtained harmonic coupling coefficients are organized into a matrix, generating a harmonic order coupling coefficient matrix as the physical channel input for subsequent dual-channel gated attention modulation.

[0049] In step S432, the harmonic order coupling coefficient matrix and the normalized spectral attention mask are subjected to dual-channel gated attention modulation and re-normalization to obtain the harmonic modulation attention weights. It should be noted that, given that the attention weights in the first embodiment only encode information in the single dimension of semantic affinity, while the previous step has quantified the physical harmonic coupling relationship, a fusion step is needed to organically integrate the two independent information channels into a unified aggregation decision basis, so that the final attention allocation simultaneously considers data-driven feature similarity and physically driven harmonic structure priors. Based on this, the technical solution of this application further performs dual-channel gated attention modulation and re-normalization on the harmonic order coupling coefficient matrix and the normalized spectral attention mask to obtain the harmonic modulation attention weights. Through the above processing, in the actual scenario of industrial equipment fault diagnosis, for neighboring nodes detected to be in harmonic coupling relationship, the semantic attention weight of the first embodiment will be amplified and enhanced proportionally, so that the harmonic coupling neighbors can obtain higher information transparency in message transmission, and the key frequency band nodes on the fault energy cascade chain can inject more sufficient causal features into the central node; while for random neighboring nodes that do not have harmonic relationship, the multiplicative term degenerates to 1, and the weight remains unchanged at the attention value of the first embodiment, without introducing additional bias.

[0050] More specifically, in a concrete example of this application, for a scenario involving the monitoring of rotating machinery in power equipment, the semantic attention weights in the normalized spectral attention mask and the physical coupling coefficients in the harmonic order coupling coefficient matrix are simultaneously read, and multiplicative gain modulation is performed on both. The mathematical expression of this modulation process is as follows: in, This indicates that the final modulation weight scalar of node i relative to its neighbor node j in the harmonic modulation attention weight generation satisfies the local normalization constraint. This represents the semantic channel attention weights of the first embodiment, passed from the normalized spectral attention mask. This represents the physical channel harmonic coupling coefficients input from the harmonic order coupling coefficient matrix. The learnable harmonic gain adjustment factor controls the amplification strength of the physical harmonic channel on the semantic attention channel. This parameter is adaptively determined through end-to-end backpropagation training to determine the optimal value. This represents the set of all first-order neighbor nodes in the topology that have valid edges connected to node i. This formula possesses strict downward compatibility: when the harmonic gain adjustment factor approaches 0, the modulation weight degenerates into the semantic attention weight of the first embodiment, and the improved mechanism automatically reverts to the single-channel aggregation mode of the first embodiment; when the harmonic gain adjustment factor is greater than 0, the multiplicative modulation term selectively amplifies the harmonic coupled neighbors, constituting a dual-channel collaborative decision-making mechanism based on semantic affinity plus physical harmonic structure.

[0051] To prevent the modulation operation from causing an imbalance in the global weight distribution, the weights of the first embodiment after modulation are re-normalized within a local neighborhood to ensure that the sum of weights strictly converges to a unit value. In the specific scenario of power equipment monitoring, for the center node representing the 100 Hz fundamental frequency of the high-frequency vibration mode, the semantic attention weight of its first embodiment with the 200 Hz second harmonic neighbor node is 0.28, the harmonic coupling coefficient is 0.82, and the harmonic gain adjustment factor is set to 0.5. Substituting these values ​​into the formula, the gain term before modulation is calculated as follows: The semantic weight of this neighboring node is amplified by 1.41 times. However, for the same central node and its 350 Hz non-harmonic neighboring node, the semantic attention weight in the first embodiment is 0.22, the harmonic coupling coefficient is only 0.08, and the calculated gain term is... The semantic weight of the neighbor node was only slightly amplified by 1.04 times. After performing multiplicative modulation on all neighbor nodes and then normalizing them within the local neighborhood, the final modulation weight of the 200 Hz second harmonic neighbor increased from the original 0.28 to 0.35, while the final modulation weight of the 350 Hz non-harmonic neighbor decreased from 0.22 to 0.18. This achieved selective enhancement of harmonic coupled neighbors and relative suppression of random noise neighbors. This step, without destroying the feature association ability already learned by the semantic attention in the first embodiment, injects the prior causal knowledge of harmonic order at the physical level into the data-driven learning process of the graph neural network in a multiplicative gain manner, upgrading the attention allocation from purely feature similarity driven to a joint drive of feature similarity and physical causal structure.

[0052] In step S433, based on harmonic modulation attention weights, harmonic enhancement feature aggregation and nonlinear activation are performed on the local neighbor feature buffer set to obtain a multimodal fused frequency domain feature tensor. It should be noted that, given the dual-channel gated modulation in S432, the attention weights have been upgraded from purely semantically driven in the first embodiment to enhanced weights incorporating harmonic order physical priors. At this point, the final neighborhood feature aggregation operation needs to be performed based on these enhanced weights to gather the cross-modal frequency domain information scattered across each neighbor node to the central node according to a harmonic order-aware weight allocation scheme, completing the directional flow of information under the constraints of physical rules. Based on this, the technical solution of this application further performs harmonic enhancement feature aggregation and nonlinear activation on the local neighbor feature buffer set based on harmonic modulation attention weights to obtain a multimodal fused frequency domain feature tensor. Through the above processing, in the scenario of rotating machinery fault diagnosis, neighboring nodes on the fundamental frequency-harmonic cascade chain will inject more sufficient fault causal correlation information into the central node due to the enhanced high weight, while the interference contribution of noise frequency band nodes deviating from the harmonic chain is effectively suppressed. This ensures that the feature representation of each node in the final output multimodal fusion frequency domain feature tensor has fully absorbed the fault causal correlation information transmitted along the harmonic order cascade chain, thereby providing a fusion feature basis with higher fault identification sensitivity for downstream global graph pooling and terminal decision-making.

[0053] More specifically, in a concrete example of this application, for a scenario involving the monitoring of rotating machinery in power equipment, all node entities in the graph are traversed. For any central node, the information aggregation operator extracts all high-dimensional neighborhood projection features belonging to the node's relation cluster, temporarily stored in the local neighbor feature buffer, as well as the node's own projection features. Self-loop connections retain self-information. The final weight coefficients after dual-channel gated modulation are obtained from the harmonic modulation attention weights. Weighted element-wise multiplication and bitwise summation operations are performed on the above feature vectors. The mathematical expression of this harmonic enhancement feature aggregation is as follows: in, This represents the harmonic enhancement fusion feature representation of the i-th central node generated within the multimodal fusion frequency domain feature tensor after the fusion process is completed. This represents a nonlinear activation mapping operator acting on the terminal, such as an exponential linear unit. The node traversal domain for performing self-loop expanding aggregation indicates that the fusion process must cover the first-order neighbor set and the node itself to preserve self-information. This represents the final attention scalar input from the harmonic modulation attention weights after dual-channel gating modulation. This represents the neighbor node feature representations that have undergone high-dimensional projection and are retrieved from the local neighbor feature buffer set. After aggregation, accumulation, and spatial representation enhancement via a nonlinear activation function, a novel feature representation of the central node is generated, incorporating harmonic cascade physical causal information. After all nodes are updated, the output is binned at the matrix level, forming a multimodal fused frequency domain feature tensor.

[0054] For example, the physical meaning of this weighted aggregation operation is as follows: neighboring nodes on the fundamental frequency-harmonic cascade chain, such as the 200 Hz second harmonic node, will transmit second harmonic energy features to the 100 Hz fundamental frequency center node. Due to the enhanced high weight of 0.35, these features will inject more sufficient fault causal correlation information into the center node. Meanwhile, the interference contribution of nodes in the 350 Hz noise band, which deviates from the harmonic chain, will be effectively suppressed due to the weight being reduced to 0.18. For the center node representing the 100 Hz fundamental frequency, its original 768-dimensional feature vector, after the aggregation operation, integrates harmonic energy features contributed by the 200 Hz second harmonic neighbor with a weight of 0.35, higher-order harmonic features contributed by the 300 Hz third harmonic neighbor with a weight of 0.28, and resonant coupling features contributed by the cross-modal infrared visual high-frequency texture neighbor with a weight of 0.19, while retaining its own fundamental frequency features contributed with a self-loop weight of 0.18. These weighted feature vectors are summed bitwise to obtain a 768-dimensional aggregate vector, which is then nonlinearly transformed using an exponential linear unit activation function to finally generate the harmonic-enhanced fusion feature representation of the central node. In this way, the harmonic order sensing capability constructed in the first two steps is transformed into actual feature fusion behavior. This ensures that the feature representation of each node in the final multimodal fusion frequency domain feature tensor fully absorbs the fault causal correlation information transmitted along the harmonic order cascade chain, thus providing a fusion feature basis with higher fault identification sensitivity for downstream global graph pooling and terminal decision-making.

[0055] Specifically, in step S5, a global graph pooling operation is performed on the multimodal fusion frequency domain feature tensor to obtain a global graph vector representing the overall operating condition. This global graph vector is then fed into a multilayer sensing classification network for nonlinear discrimination to output the terminal decision result. It should be noted that because the multimodal fusion frequency domain feature tensor generated in the preceding steps carries all cross-modal and multi-band local node information but exhibits a geometrically scattered node-level feature shape, it cannot be directly used as the input basis for terminal fault diagnosis decisions. However, industrial equipment health monitoring and fault early warning require outputting a macroscopic assessment conclusion of the overall operating status of the equipment and specific fault type discrimination results. Based on this, the technical solution of this application further performs a global graph pooling operation on the multimodal fusion frequency domain feature tensor to obtain a global graph vector representing the overall operating condition, and feeds this global graph vector into a multilayer sensing classification network for nonlinear discrimination to output the terminal decision result. Through the above processing, the scattered node-level tensors can be compressed and converged into a single global macro-vector that can characterize the overall operational health level of the multimodal system within the current time window. Then, through a multilayer perceptual classification network, the high-dimensional continuous global features are interpreted into a normalized confidence probability distribution of the physical fault hazard classification domain. Finally, a standardized industrial control message containing fault type identification, hazard risk level score, and equipment identification information is output, providing a decision-making basis for predictive maintenance and emergency response of power equipment.

[0056] More specifically, in a concrete example of this application, for the multimodal fused frequency domain feature tensor generated in the monitoring scenario of rotating machinery in power equipment, a graph-level pooling readout engine is first initiated to perform a global dimensionality collapse operation on the tensor. This engine abandons the global average pooling rule that washes away edge abrupt changes and adopts an adaptive attention pooling mechanism. Through a set of independent, learnable weighted pooling vector operations, the explicit contribution of all fused frequency domain nodes to the overall health status of the industrial equipment is evaluated and scored, and then exponentially normalized as a smoothness multiplier. The computational logic of this graph macro-dimensionality collapse based on node-level attention scores is expressed as follows: in, This represents the macroscopic representation vector of the global graph generated after projection collapse. This represents the total number of cross-modal nodes contained in the current frequency domain graph structure. and Let represent the final deep hidden layer feature representations of the i-th and j-th nodes extracted from the input multimodal fusion frequency domain feature tensor after undergoing collaborative interaction, respectively. The transpose of the learnable global pooling projection weight vector, configured separately for the attention readout engine, is used to scalarize a single high-dimensional feature. Through this mapping mechanism, the hash feature, originally containing 200 nodes, is compressed into a single 768-dimensional global graph macroscopic representation vector, which can represent the overall operational health level of the rotating machinery of the power equipment within the current five-second time window.

[0057] Subsequently, the global graph macroscopic representation vector is used as the initial input parameter of the terminal neuron decision network and injected into a multilayer perceptron classifier with hidden layers. The classifier utilizes pre-trained, fixed affine transformation network weight matrices and nonlinear activation barriers to progressively interpret the higher-order anomaly trends inherent in the vector. Finally, a Softmax activation function layer forces a nonlinear folding of the real tensor space. The mathematical derivation of the classifier's nonlinear discrimination boundary delineation and discrete class inversion mapping is expressed as follows: ,in This represents the generated prediction tensor containing probability thresholds for various preset fault categories. This represents the exponentially normalized activation function operator applied to the output of a multi-classification network. This refers to the nonlinear rectifier unit located in the implicit channel of the perceptron, which performs characteristic noise immunity and blocks interference amplification. and Let represent the learnable weight matrix and feature bias vector used by the hidden layers of a multilayer perceptron to perform geometric stretching and translation, respectively. and These represent the decision connection parameter space weight matrix and the final bias term when projecting from the hidden layer to the preset discrete anomaly physical type dimension, respectively. In the power equipment monitoring scenario, the classifier's output layer contains eight neurons corresponding to eight state categories: normal operation, bearing outer ring pitting, bearing inner ring crack, gear meshing surface wear, rotor mass imbalance, coupling eccentricity, base looseness, and combined faults. After Softmax normalization, the output probability distribution is 0.12 for normal operation, 0.68 for bearing outer ring pitting, and a total probability of 0.20 for other categories, indicating a 68% confidence level that the current equipment has a bearing outer ring pitting fault.

[0058] Finally, based on the fault enumeration code and potential risk level score pointed to by the highest confidence level in the discrete discrimination instruction set, the standardized data encoding module is triggered to perform structured data encapsulation. The encoding module converts the fault enumeration code 0x02 and the confidence score of 68% for the pitting corrosion of the bearing outer ring into a potential risk level score of medium risk, adds a unique device identifier such as the device MAC address, and a time scale label of the event trigger time such as 14:32:05 on March 15, 2026, serializes it into a standard byte stream dictionary conforming to the MQTT communication protocol or the industrial Modbus-TCP interaction protocol, forms the terminal decision result, and sends it to the underlying network sending queue pin for external transmission. At this point, the entire process of the multimodal fusion judgment closed-loop algorithm link in this diagram is completed.

[0059] As described above, the graph neural network-driven multimodal data fusion method according to the embodiments of this application can be implemented in various industrial equipment health monitoring and intelligent diagnosis platforms, such as power system rotating equipment monitoring centers, wind farm blade and gearbox condition assessment units, or predictive maintenance terminals for key pumps and valves in petrochemical plants. In one possible implementation, this method can be integrated into an industrial internet equipment health management system or a power monitoring and data acquisition platform as a multimodal feature fusion engine or a frequency domain graph neural network inference component. For example, this method can be an independent fault diagnosis application running on an industrial server in the field, or it can be a multimodal data analysis plug-in and graph attention calculation module of an existing SCADA system, or an intelligent operation and maintenance middleware service deployed in the cloud and distributed through edge collaboration. Of course, the time-frequency domain orthogonal transformation, frequency domain graph topology construction, and spectrum collaboration-based graph attention feature aggregation modules in this method can also run in industrial edge gateways, field acquisition terminals, or distributed vibration monitoring units equipped with limited computing power, serving as the underlying real-time analysis foundation of this multi-level equipment health monitoring system.

[0060] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A graph neural network-driven multimodal data fusion method, characterized in that, include: Step 1: Perform macroscopic time window segmentation on the acquired multimodal observation data stream to obtain multimodal time window data blocks. The multimodal time window data blocks include high-frequency vibration acceleration sampling sequences, low-frequency ambient temperature sampling sequences, and variable frame rate infrared thermal imaging video streams. Step 2: Perform time-frequency domain orthogonal transformation and spectral feature extraction on the multimodal time window data blocks to obtain an initial set of frequency domain latent features; Step 3: Based on the initial set of latent features in the frequency domain, construct a frequency domain graph topology that includes a frequency domain adjacency matrix; Step 4: Using the frequency domain graph topology as the message passing path, the graph attention network is used to perform spectral collaborative graph attention feature aggregation on each target node and its neighboring nodes in the initial frequency domain latent feature set to obtain the multimodal fusion frequency domain feature tensor. Step 5: Perform global graph pooling on the multimodal fusion frequency domain feature tensor to obtain a global graph vector representing the overall operating conditions. Feed the global graph vector into the multilayer perceptual classification network for nonlinear discrimination to output the terminal decision result.

2. The graph neural network-driven multimodal data fusion method according to claim 1, characterized in that, Step one includes: Step 1.1: Stream and buffer the data sources of each modality in the multimodal observation data stream to obtain a continuous observation data buffer. Step 1.2: Using a preset fixed physical time span as a sliding slice window, traverse each modal data queue in the continuous observation data buffer, extract the data point set whose sampling timestamp falls within the current window interval, and truncate and aggregate it to obtain the non-fixed length data segment set of the first embodiment; Step 1.3: Assign data type metadata tag headers to each modal slice in the non-fixed-length data segment set of the first embodiment to obtain multimodal time window data blocks.

3. The graph neural network-driven multimodal data fusion method according to claim 1, characterized in that, Step two includes: Step 2.1: Based on the metadata tag header embedded in the multimodal time window data block, perform dimensional routing allocation on the multimodal time window data area to obtain a heterogeneous sequence classification set; Step 2.2: Extract the frequency band energy spectrum of the one-dimensional time series component in the heterogeneous sequence classification set to obtain one-dimensional frequency band features, and package the one-dimensional frequency band features with the unprocessed two-dimensional visual sequence to obtain a half-frequency domain hybrid set. Step 2.3: Perform spatiotemporal frequency domain orthogonal decomposition on the two-dimensional infrared video frame sequence in the half-frequency domain hybrid set to obtain the visual frequency domain tensor, and merge the visual frequency domain tensor with the one-dimensional frequency band energy matrix to obtain the full-modal frequency domain pool set. Step 2.4: Align the heterogeneous frequency band features of each modal frequency domain data in the full-modal frequency domain pool set with the same latent space reference to obtain the initial frequency domain latent feature set.

4. The graph neural network-driven multimodal data fusion method according to claim 1, characterized in that, Step three includes: Step 3.1: Traverse each independent frequency band feature vector in the initial frequency domain latent feature set, assign it a globally unique identifier, and bind its mode type, center frequency value, and latent space vector as attributes to the node object to obtain a frequency domain undirected node set. Step 3.2: Perform a mapping of intramodal frequency band adjacency and cross-modal resonance weights on the undirected node set in the frequency domain to obtain the fully connected topological weight tensor; Step 3.3: Perform confidence-based topological sparsification on the fully connected topological weight tensor to obtain the frequency domain graph topology.

5. The graph neural network-driven multimodal data fusion method according to claim 1, characterized in that, Step four includes: Step 4.1: Perform linear high-dimensional projection of frequency domain features and neighbor path extraction on the initial frequency domain latent feature set and frequency domain graph topology to obtain a set of local projection feature pairs; Step 4.2: Calculate and normalize the cross-modal spectral domain attention cross-correlation mask for the set of local projected features to obtain the normalized spectral map attention mask and the local neighbor feature buffer set; Step 4.3: Based on the normalized spectral attention mask, the local neighbor feature buffer set is aggregated with the neighbor feature space infiltrated by the physical resonance gradient to obtain the multimodal fused frequency domain feature tensor.

6. The graph neural network-driven multimodal data fusion method according to claim 1, characterized in that, Step five includes: Step 5.1: The multimodal fusion frequency domain feature tensor is dimensionally collapsed using learnable pooling weight vectors to represent the vectors macroscopically in a global graph. Step 5.2: Feed the global graph macroscopic representation vector into the multilayer perceptron classifier to obtain the discrete discriminant instruction set; Step 5.3: Based on the fault enumeration code and hidden danger risk level score pointed to by the highest confidence in the discrete discrimination instruction set, the unique equipment number and event time scale label are added for serialization and encapsulation, and the output is the terminal decision result.

7. The graph neural network-driven multimodal data fusion method according to claim 5, characterized in that, Step 4.3 includes: Harmonic order coupling relationship detection is performed on the local neighbor feature buffer set to obtain the harmonic order coupling coefficient matrix; Dual-channel gated attention modulation and re-normalization are performed on the harmonic order coupling coefficient matrix and the normalized spectrum attention mask to obtain the harmonic modulation attention weights. Based on harmonic modulation attention weights, harmonic enhancement feature aggregation and nonlinear activation are performed on the local neighbor feature buffer set to obtain a multimodal fused frequency domain feature tensor.

Images