A drilling machine multi-modal fault diagnosis system and method

Abstract

The present application relates to the technical field of industrial equipment fault diagnosis, and particularly relates to a drilling machine multi-modal fault diagnosis system and method. The present application significantly improves the accuracy, robustness and practicality of PCB drilling machine fault diagnosis by constructing a lightweight space-time multi-modal fusion diagnosis architecture. The present application adopts a multi-cycle time sequence feature extraction and component alignment visual feature extraction method to achieve fine-grained representation of complex fault modes of the equipment. By introducing a graph convolution network based on physical relations and a directional cross-modal attention fusion mechanism, the space-time propagation law of faults between components and the deep complementary information between heterogeneous data are effectively captured. Finally, combined with the dual calibration strategies of equipment physical constraints and graph structure space smoothing, the diagnosis results conform to the internal logic of the mechanical system, effectively suppressing false positives and unreasonable predictions.

Description

Technical Field

[0001] This invention relates to the field of industrial equipment fault diagnosis technology, and in particular to a multimodal fault diagnosis system and method for drilling machines. Background Technology

[0002] In the field of fault diagnosis for high-precision CNC equipment such as PCB (printed circuit board) drilling machines, ensuring their continuous and stable operation is crucial for guaranteeing production yield. Traditional diagnostic methods mainly rely on single-type sensor data (such as vibration signals) or manual inspection images, which have significant limitations. A single sensor cannot comprehensively capture complex faults caused by the coupling of multiple factors such as mechanical wear, thermal stress accumulation, and electrical anomalies; while manual inspection is inefficient, dependent on experience, and cannot provide real-time early warning. In recent years, although multimodal fusion technology has made it possible to comprehensively perceive the equipment status by combining infrared thermal imaging, RGB images, and data from multiple types of sensors, it still faces serious challenges in actual industrial deployment.

[0003] Existing methods typically involve directly stacking complex deep learning models, resulting in a large number of parameters and high computational costs, making them unsuitable for resource-constrained industrial edge computing environments and data scenarios with limited sample sizes. At the same time, there is significant heterogeneity among different modal data (such as spatial features of images and temporal features of sensors). Simple splicing or shallow fusion strategies cannot fully explore their inherent complementary information and spatiotemporal correlations, resulting in limited diagnostic accuracy and weak generalization ability, which cannot meet the stringent requirements of PCB drilling machines for accurate early warning and real-time diagnosis of faults. Summary of the Invention

[0004] In view of the problems existing in the prior art, the present invention is proposed.

[0005] Therefore, this invention aims to solve the core contradiction of complex multimodal fault diagnosis models and low diagnostic accuracy in PCB drilling machines when the industrial sample size is small.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal fault diagnosis system for a drilling machine, comprising: a feature extraction module, used to extract multi-period temporal features from sensor time-series data and multimodal visual features from infrared thermal imaging images and RGB visible light images; a spatiotemporal modeling module, used to construct a graph structure based on the physical relationships of key components of the drilling machine, map the multi-period temporal features and the multimodal visual features to corresponding nodes of the graph structure, and perform feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features that fuse the spatial relationships of the components; a cross-modal fusion module, used to perform feature fusion using the spatiotemporal features as query vectors and the multimodal visual features as key and value vectors through a cross-modal attention mechanism to obtain multimodal fused features; and a fault prediction and calibration module, used to map the multimodal fused features to preliminary fault prediction results, and calibrate the preliminary fault prediction results sequentially based on equipment physical constraint rules and the node relationships of the graph structure, outputting the final fault type and severity diagnosis results.

[0007] As a preferred embodiment of the multi-modal fault diagnosis system for drilling machines described in this invention, the feature extraction module includes a time-series feature extraction unit, which is used to: preprocess and frequency-domain transform the raw time-series data of the pressure sensor and vibration sensor; select the k key frequency components with the largest amplitude from the frequency domain results and determine the period length corresponding to each key frequency; reshape the one-dimensional time-series data into multiple two-dimensional tensors based on each period length; perform feature deepening processing on each two-dimensional tensor, and aggregate the processed features into the multi-period time-series features based on the amplitude weights of each key frequency component.

[0008] As a preferred embodiment of the multimodal fault diagnosis system for drilling machines described in this invention, the feature extraction module includes a visual feature extraction unit. The visual feature extraction unit employs a visual transformer model and is used to: concatenate the preprocessed infrared temperature image with an RGB image to form a multi-channel input image; divide the multi-channel input image into non-overlapping image blocks corresponding one-to-one with the key components and perform embedding encoding; and extract the multimodal visual features through the multi-head self-attention mechanism and encoding layer of the visual transformer model, wherein each feature vector corresponds to a key component.

[0009] As a preferred embodiment of the drilling machine multimodal fault diagnosis system of the present invention, wherein: in the spatiotemporal modeling module, the graph structure is a spatiotemporal graph, the nodes of which correspond to the key components, and the edges of which include spatial edges and temporal edges; the weight of the spatial edges is calculated based on the physical connection distance and connection type weight between components; the spatiotemporal modeling module generates the spatiotemporal features by fusing the node's own features, the features of spatial neighboring nodes, and the node features of adjacent time steps in a weighted aggregation manner through the graph convolutional network.

[0010] In a preferred embodiment of the drilling machine multimodal fault diagnosis system of the present invention, the calculation process of the cross-modal attention mechanism executed by the cross-modal fusion module is as follows:

[0011]

[0012] in, , , , For the aforementioned spatiotemporal features, For the multimodal visual features, , , For a learnable projection matrix, Let the dimension of the key vector be denoted as ; the output of the attention mechanism is then compared with the spatiotemporal features. The multimodal fusion features are obtained by adding the residual connections. .

[0013] As a preferred embodiment of the drilling machine multimodal fault diagnosis system of the present invention, the fault prediction and calibration module includes: a lightweight mapping unit, used to map the multimodal fusion features into a preliminary fault prediction result containing the probability distribution of fault types and the initial value of fault severity through a two-layer fully connected network; a physical constraint calibration unit, used to correct the preliminary fault prediction result according to a preset rule, the rule including: when the maximum probability of a fault type of a certain component is lower than a threshold θ, the fault severity corresponding to that component is forcibly set to zero; and a spatial smoothing calibration unit, used to reuse the graph structure adjacency matrix in the spatiotemporal modeling module, and perform weighted average smoothing on the prediction results of each node after physical constraint calibration according to the prediction results of its neighboring nodes to obtain the final fault diagnosis result.

[0014] As a preferred embodiment of the multimodal fault diagnosis system for drilling machines described in this invention, the system employs a total loss function during training. Loss from classification regression Domain adaptation loss and modal consistency loss Weighted composition: ,

[0015] Wherein, the classification regression loss This includes classification loss based on visual features, classification loss based on temporal features, and severity regression loss; the domain adaptation loss... The modal consistency loss is used to reduce the difference in feature distribution between the source and target domains. The consistency point prediction results used to constrain the single-modal classification results and the multimodal fusion classification results are weighted and smoothed according to the prediction results of their neighboring nodes to obtain the final fault diagnosis result.

[0016] The second objective of this invention is to provide a multimodal fault diagnosis method for drilling machines, comprising the following steps: S1: Extracting multi-period temporal features from sensor time-series data, and extracting multimodal visual features from infrared thermal imaging images and RGB visible light images; S2: Constructing a graph structure based on the physical relationships of the key components of the drilling machine, mapping the multi-period temporal features and the multimodal visual features to corresponding nodes in the graph structure, and performing feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features of fused component spatial relationships; S3: Using the spatiotemporal features as query vectors and the multimodal visual features as key and value vectors, performing feature fusion through a cross-modal attention mechanism to obtain multimodal fused features; S4: Mapping the multimodal fused features to preliminary fault prediction results, and sequentially calibrating the preliminary fault prediction results based on equipment physical constraint rules and the node relationships of the graph structure, outputting the final fault type and severity diagnosis results.

[0017] As a preferred embodiment of the multi-modal fault diagnosis method for drilling machines described in this invention, step S1, which extracts multi-cycle time-series features, specifically includes: S11: preprocessing and performing fast Fourier transform on the raw time-series data of the pressure sensor and vibration sensor; S12: calculating the amplitude and selecting the previous period with the largest amplitude. Each frequency is used as a key frequency component. And calculate the corresponding period length. S13: Based on the length of each cycle One-dimensional time series data Remodeling into a two-dimensional tensor S14: For each two-dimensional tensor Feature enhancement processing is performed, and weighted aggregation is conducted based on the amplitude weights of each key frequency component to obtain the multi-cycle time-series features. .

[0018] As a preferred embodiment of the multimodal fault diagnosis method for drilling machines described in this invention, the calibration in step S4 specifically includes: S41: applying physical constraints to the preliminary fault prediction results to ensure logical consistency between fault type prediction and severity prediction; S42: utilizing the adjacency matrix of the graph structure. After physical constraint calibration, the first Fault type prediction results for each node Severity prediction results Spatial smoothing is performed using the following formula:

[0019]

[0020]

[0021] in, Represents a node The set of spatial neighbor nodes, and This is the final output result.

[0022] The beneficial effects of this invention are as follows: By constructing a lightweight spatiotemporal multimodal fusion diagnostic architecture, this invention significantly improves the accuracy, robustness, and practicality of PCB drilling machine fault diagnosis. It employs a multi-cycle temporal feature extraction and component-aligned visual feature extraction method to achieve fine-grained characterization of complex equipment fault modes. By introducing a physically-based graph convolutional network and a directional cross-modal attention fusion mechanism, it effectively captures the spatiotemporal propagation patterns of faults among components and the deep complementary information between heterogeneous data. Finally, by combining a dual calibration strategy of equipment physical constraints and graph structure spatial smoothing, it ensures that the diagnostic results conform to the inherent logic of the mechanical system, effectively suppressing false alarms and unreasonable predictions. This scheme has a small overall parameter count and high computational efficiency, making it particularly suitable for industrial edge deployments and training scenarios with limited sample sizes. It can achieve early, accurate, and automated diagnosis of multiple types of faults in PCB drilling machines under resource-constrained conditions, providing reliable technical support for predictive maintenance, thereby effectively reducing unplanned downtime and ensuring production continuity and product quality. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a diagram showing the overall system architecture and data flow of the present invention.

[0025] Figure 2 This is a schematic diagram of the TimesNet time-series feature extraction module of the present invention.

[0026] Figure 3 This is a schematic diagram of the TimesBlock structure of the present invention.

[0027] Figure 4 This is a schematic diagram of the ViT visual feature extraction module of the present invention.

[0028] Figure 5 This is a schematic diagram of the GCN spatiotemporal modeling of the present invention.

[0029] Figure 6 This is a schematic diagram of the cross-modal attention fusion mechanism of the present invention.

[0030] Figure 7 This is the overall flowchart of the method of the present invention. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0033] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is mutually exclusive, either alone or selectively, with other embodiments.

[0034] Example 1, referring to Figures 1-6This is the first embodiment of the present invention, which provides a multimodal fault diagnosis system for a drilling machine, including a feature extraction module for extracting multi-period temporal features from sensor time-series data and multimodal visual features from infrared thermal imaging images and RGB visible light images; a spatiotemporal modeling module for constructing a graph structure based on the physical relationships of key components of the drilling machine, mapping the multi-period temporal features and the multimodal visual features to corresponding nodes of the graph structure, and performing feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features that fuse the spatial relationships of the components; a cross-modal fusion module for using the spatiotemporal features as query vectors and the multimodal visual features as key and value vectors, and performing feature fusion through a cross-modal attention mechanism to obtain multimodal fused features; and a fault prediction and calibration module for mapping the multimodal fused features to preliminary fault prediction results, and calibrating the preliminary fault prediction results sequentially based on equipment physical constraint rules and the node relationships of the graph structure, and outputting the final fault type and severity diagnosis results.

[0035] This embodiment constitutes the overall framework of a highly refined lightweight spatiotemporal multimodal fusion fault diagnosis model (Lightweight ViT-GCN Multi-Modal Fusion Fault Diagnosis, L-VG-MFDD), as follows: Figure 1 As shown, the system targets the drill bit system of a PCB drilling machine. Its data comes from high-precision pressure sensors and piezoelectric vibration sensors deployed on the drill bit base, as well as coaxially mounted infrared thermal imaging cameras and high-definition RGB industrial cameras. All multi-source data undergoes rigorous preprocessing and spatiotemporal alignment to provide high-quality input for subsequent model processing. The feature extraction module works in parallel, processing time-series and image data streams separately. The core of the spatiotemporal modeling module is to construct a graph structure that reflects the mechanical connections and fault propagation paths of the equipment. The nodes of this structure strictly correspond to the component units divided in the feature extraction module. The cross-modal fusion module does not simply stitch features together but designs an attention interaction mechanism dominated by spatiotemporal features. Finally, the fault prediction and calibration module uses a lightweight mapping network combined with a dual calibration strategy to ensure that the output results are both accurate and conform to physical common sense. It is worth noting that the entire model design fully considers the limitations of industrial scenarios with limited sample sizes. Through lightweight module design and efficient fusion strategies, it maximizes diagnostic accuracy while controlling the total number of parameters.

[0036] Specifically, the feature extraction module includes a temporal feature extraction unit, which is used to: preprocess and frequency-domain transform the raw temporal data of the pressure sensor and vibration sensor; select the k key frequency components with the largest amplitude from the frequency domain results and determine the period length corresponding to each key frequency; reshape the one-dimensional temporal data into multiple two-dimensional tensors based on each period length; perform feature deepening processing on each two-dimensional tensor, and aggregate the processed features into the multi-period temporal features based on the amplitude weight of each key frequency component.

[0037] This time-series feature extraction unit is specifically implemented based on the TimesNet model. In the preprocessing stage, vibration sensor data undergoes data synchronization, noise suppression, trend term removal, and preliminary feature extraction sequentially, while pressure sensor data undergoes synchronization, noise suppression, range calibration, and preliminary feature extraction, ultimately forming a standardized time-series sequence. ,in This indicates the timing length. This is represented as the number of sensor variables. To capture the multi-period characteristics of the fault, the element is analyzed in the frequency domain using Fast Fourier Transform (FFT) to calculate the amplitude. And select the previous The frequency with the largest amplitude As a key frequency component, and then according to the formula Calculate the corresponding period length This design specifically addresses three scales of equipment failure evolution: short-term cycles (e.g., 10-second load fluctuations), medium-term cycles (e.g., 8-hour work cycles), and long-term cycles (e.g., 30-day aging trends). Subsequently, based on the length of each cycle... , the original one-dimensional sequence Through fill and reshape operations Convert to two-dimensional tensor This maps the periodic changes in the time dimension to the row and column relationships in a two-dimensional space, facilitating the subsequent capture of local two-dimensional patterns. Each two-dimensional tensor is input into the TimesBlock module for feature refinement. This module is built based on residual connections, as shown in the formula: While extracting fine features, the original structural information is preserved. This refers to the TimesBlock module. It is the TimesNet's... The two-dimensional tensor output by the layer, It is the first The layer outputs a two-dimensional tensor. Finally, all processed two-dimensional representations are flattened back to one dimension and their amplitudes are determined according to their corresponding frequencies. Weights after Softmax normalization Weighted aggregation is performed to obtain global multi-period temporal features. in, Calculation based on amplitude of each period The normalized weights are such that the larger the amplitude of the period, the greater its contribution to fault diagnosis, and the greater its corresponding weight. It is the first A one-dimensional temporal representation obtained by flattening a two-dimensional tensor after processing with TimesBlock; It is the summation operator. This represents the final aggregated global one-dimensional temporal feature.

[0038] Specifically, the feature extraction module includes a visual feature extraction unit, which employs a visual transformer model and is used to: concatenate the preprocessed infrared temperature image with the RGB image to form a multi-channel input image; divide the multi-channel input image into non-overlapping image blocks that correspond one-to-one with the key components and perform embedding encoding; and extract the multimodal visual features through the multi-head self-attention mechanism and encoding layer of the visual transformer model, wherein each feature vector corresponds to a key component.

[0039] The visual feature extraction unit uses Vision Transformer (ViT) as its backbone network, and its structural diagram is shown below. Figure 4 As shown. The input image is formed by stitching together a single-channel infrared temperature thermogram (after registration, temperature calibration, noise suppression, and normalization preprocessing) and a three-channel RGB image, resulting in a four-channel input. Crucially, the image is divided into... Each non-overlapping patch corresponds to a key physical component of the drill bit system (such as the drill tip, bearing housing, or motor housing). This corresponds strictly one-to-one with the subsequent GCN node settings, achieving spatial alignment between visual features and component entities. The embedding process is as follows: in Representing the The pixel vector of each patch For embedding matrix, For position encoding, the encoder consists of multiple stacked Transformer blocks, each layer containing a multi-head self-attention (MSA) and multilayer perceptron (MLP) sublayer, employing layer normalization (LayerNorm) and residual connections. The update formula is as follows:

[0040] in It is the first The output feature tensor of the layer It is the first The final output feature tensor of the layer, It is an index variable of the Encoder layer, representing the iteration round; It is layer normalization. It is a multi-head attention mechanism. It is a multilayer perceptron, containing two layers of linear transformation and The activation function is used to obtain the nonlinear mapping relationship between infrared temperature features and RGB appearance features; the residual connection preserves the basic spatial location and multimodal pixel information of the patch embedding stage, preventing the features of key fault-prone areas such as motor windings and seals from being over-encoded and covered.

[0041] Multi-head self-attention allows the model to globally compute the correlation strength between any two image patches, thus directly modeling the spatial propagation of faults between parts and overcoming the limitation of the limited receptive field in traditional convolutional networks. After L layers of encoding, category embeddings are extracted. Corresponding output features (Where D is the feature dimension) As a global device visual feature, this feature deeply integrates infrared thermal anomaly and RGB appearance defect information.

[0042] Specifically, in the spatiotemporal modeling module, the graph structure is a spatiotemporal graph, whose nodes correspond to the key components, and whose edges include spatial edges and temporal edges; the weight of the spatial edge is calculated based on the physical connection distance and connection type weight between components; the spatiotemporal modeling module uses the graph convolutional network to fuse the node's own features, the features of spatial neighboring nodes, and the node features of adjacent time steps in a weighted aggregation manner to generate the spatiotemporal features.

[0043] The core of this spatiotemporal modeling module is to construct and utilize a spatiotemporal diagram that accurately reflects the physical topology of the equipment and the characteristics of fault propagation. For example... Figure 5 As shown, taking the three core components of a PCB drilling machine—the workpiece, the drill bit, and the drill bit base—as an example, the nodes... , , Representing components, spatial edges (solid lines) connect components that have physical or functional relationships, while temporal edges (dotted-dash lines) connect the states of the same component at different times. Calculating the weights of the spatial edges is crucial; they are determined by the elements of the adjacency matrix A. The formula is in It is the th in the fault propagation adjacency matrix The element in row j represents the device number. Components With the Components The overall fault propagation correlation weight between them; It is the natural exponential function; The topological distance between components. It is the bandwidth parameter of the Gaussian kernel, which controls the rate at which the correlation degree decays with distance; The weighting coefficients for physical connections are set to 1.0 for directly connected components, 0.8 for indirectly connected components, and 0.1 for components without mechanical connections. This weighting quantifies the correlation of fault propagation and weakens ineffective cross-component interference. This design elevates the graph structure beyond a simple physical connection graph to a "fault propagation correlation graph." The input to the spatiotemporal modeling module is the output of the feature extraction module: temporal features. and visual features , where N is the total number of nodes / components. For each component node i, its initial node characteristics Based on the corresponding time series characteristics and visual features It is pieced together, that is The GCN module performs feature propagation based on this, and its optimized core propagation formula is: This formula also aggregates the spatial neighborhood. Information (via spatial edge weights) and weight and temporal neighborhood Information (via time edge weights) and weight The balance coefficient α is dynamically adjusted according to the equipment's operating conditions (α=0.7 during stable operation and α=0.4 during start-up and shutdown) to dynamically balance the contributions of spatial correlation and temporal evolution to the current state. After each layer of propagation, the LeakyReLU activation function is applied and residual connections are added to prevent gradient vanishing and preserve key features. Through multiple layers of such graph convolution operations, the model ultimately outputs spatiotemporal features that integrate component spatial correlation and state temporal evolution. .

[0044] Specifically, the calculation process of the cross-modal attention mechanism executed by the cross-modal fusion module is as follows: in, , , For the aforementioned spatiotemporal features, For the multimodal visual features, , , For a learnable projection matrix, Let the dimension of the key vector be denoted as ; the output of the attention mechanism is then compared with the spatiotemporal features. The multimodal fusion features are obtained by adding the residual connections. .

[0045] The structure of the cross-modal fusion module is as follows: Figure 6As shown, the aim is to achieve precise and efficient interaction between temporal-spatial features and visual features. Prior to this, the model has ensured strict alignment of multi-source data in both temporal and spatial dimensions through data preprocessing (timestamp synchronization based on PLC clock, image patch partitioning based on component coordinates, and sensor binding), laying the foundation for deep fusion. The fusion mechanism uses the spatiotemporal features output by GCN. As a query, the component-level multimodal visual features extracted by ViT are used. As keys and values, this design has a clear physical meaning: It contains information about the component's "operating state" and "affected by its surroundings." As an active querying entity, it searches for anomalies in visual features most relevant to the current spatiotemporal state. In specific calculations, it first uses a learnable projection matrix... , , Will and The data is mapped to a unified feature space, generating Q, K, and V matrices. Attention weights are then calculated. This weight matrix quantifies the correlation between each spatiotemporal feature vector and all visual feature vectors. The attention output is obtained by weighted summation of V using these weights; it is essentially the result of filtering and reorganizing visual features based on spatiotemporal state correlations. To preserve the original core information of fault dynamic evolution in the spatiotemporal features and avoid dilution during fusion, the module employs residual connections, combining the attention output with the original... By directly adding them together, we obtain the final multimodal fusion feature: This mechanism enables the modeling of fine-grained dependencies between heterogeneous modalities, enhancing information complementarity.

[0046] Specifically, the fault prediction and calibration module includes: a lightweight mapping unit, used to map the multimodal fusion features into a preliminary fault prediction result containing the probability distribution of fault types and the initial value of fault severity through a two-layer fully connected network; a physical constraint calibration unit, used to correct the preliminary fault prediction result according to preset rules, the rules including: when the maximum probability of a fault type of a certain component is lower than a threshold θ, the fault severity corresponding to that component is forcibly set to zero; and a spatial smoothing calibration unit, used to reuse the graph structure adjacency matrix in the spatiotemporal modeling module, and perform weighted average smoothing on the prediction results of each node after physical constraint calibration according to the prediction results of its neighboring nodes to obtain the final fault diagnosis result.

[0047] This module is responsible for integrating the high-level features. This is transformed into component-level diagnostic results that can be directly used for maintenance decisions. The dimension is Where N is the total number of component nodes. To fuse feature dimensions, the lightweight mapping unit employs a two-layer MLP to implement lightweight feature mapping. The first layer consists of weights. Bias and Activation function construction and calculation: To preserve weak fault signals. The second layer is divided into two branches: a fault type branch and a weighted branch. Bias and The function outputs the probability distribution of M types of faults; the severity branch is weighted. Bias and The function outputs a preliminary non-negative severity value, which together constitutes the original prediction: The physical constraint calibration unit aims to resolve potential logical inconsistencies in the original predictions. It first applies a fault type probability... After constraining the function, we get: For severity, a consistency constraint is applied: in It is an indicator function that only applies when the maximum probability of a component failure type exceeds a threshold. The severity level is only retained when the condition is met; otherwise, it is set to zero. This avoids the contradictory output of "no fault but high severity," resulting in a different outcome. The spatial smoothing calibration unit further leverages the propagation characteristics of equipment faults to enhance the robustness of the results. It reuses the fault propagation adjacency matrix A from the GCN to perform neighborhood-weighted smoothing on the constrained results for each node. For node i, its final fault type probability... and severity Calculate using the following formulas respectively: This operation essentially diffuses and averages local fault predictions based on the strength of physical correlation between components, effectively correcting isolated and abrupt abnormal predictions, and making the diagnostic results more consistent with the continuous law of fault propagation in space.

[0048] Specifically, the system uses a total loss function during training. Loss from classification regression Domain adaptation loss and modal consistency loss Weighted composition:

[0049]

[0050] Wherein, the classification regression loss This includes classification loss based on visual features, classification loss based on temporal features, and severity regression loss; the domain adaptation loss... The modal consistency loss is used to reduce the difference in feature distribution between the source and target domains. The consistency point prediction results used to constrain the single-modal classification results and the multimodal fusion classification results are weighted and smoothed according to the prediction results of their neighboring nodes to obtain the final fault diagnosis result.

[0051] This multi-objective loss function is a collaborative optimization scheme addressing the three major challenges of scarce annotations, variable operating conditions, and modal heterogeneity in industrial diagnostic scenarios. (Classification / Regression Loss) It is the foundation, consisting of three parts: It is based on the cross-entropy loss of ViT visual features to ensure that the visual model can accurately learn the association between thermal anomalies, appearance defects and fault types; It is based on the cross-entropy loss of TimesNet time series features, which forces the time series model to effectively characterize the relationship between vibration, pressure dynamics and faults; This is the mean squared error (MSE) loss for severity prediction, used to optimize regression accuracy. Domain adaptation loss. (Maximum Mean Difference) aims to improve the model's generalization ability across different operating conditions. It compares the distribution differences of samples from the source and target domains in the feature space (for visual spatiotemporal features respectively). and temporal and spatiotemporal characteristics (Calculation), and minimizing this difference drives the model to learn core fault characteristics that remain unchanged under operating conditions. Modal consistency loss (Kullback-Leibler divergence) is used to align different modes. The visual classification results of ViT are constrained to approach the final fusion classification result. The temporal classification results of TimesNet are constrained to approach the fusion results, thereby forcing single-modal feature learning to align with global consensus and reducing intermodal inconsistencies. Hyperparameters α and β are used to balance the contributions of the three loss parameters. During training, these parameters are minimized using the backpropagation algorithm. By simultaneously optimizing the parameters of all modules, a robust model that is both accurate in diagnosis and adaptable to complex industrial scenarios is finally obtained.

[0052] In use, the system described in this embodiment works as follows: Multi-source sensors and cameras deployed on the PCB drilling machine synchronously acquire vibration and pressure time-series signals, as well as infrared and RGB image data. After rigorous preprocessing and spatiotemporal alignment, these data are input into the feature extraction module. The time-series data is analyzed using TimesNet units, and key cycles are identified through frequency domain transformation. After two-dimensional reconstruction and feature enhancement, multi-cycle time-series features characterizing the multi-scale operating patterns of the equipment are extracted. Simultaneously, the image data is processed by the ViT unit, and through block embedding and global self-attention mechanisms, visual features that are strictly aligned with the physical components and integrate thermal and appearance information are extracted. The spatiotemporal modeling module constructs a spatiotemporal diagram based on the equipment's mechanical topology, and... and Mapping to node features, and using GCN to aggregate information from the node itself, its spatial neighborhood, and its temporal neighborhood, spatiotemporal features of fused component associations and state evolution are generated. The cross-modal fusion module is based on... To guide, through attention mechanisms from Complementary visual evidence is retrieved from the data, and comprehensive multimodal fusion features are obtained through residual connections. The fault prediction and calibration module first uses a lightweight MLP to... The system maps the initial fault type and severity predictions, then applies physical and logical constraints to eliminate contradictions, and performs spatial smoothing based on the component association diagram to correct anomalies. Finally, it outputs accurate and reliable component-level fault diagnosis results that conform to the equipment's operating rules, providing a basis for predictive maintenance decisions.

[0053] Example 2, refer to Figures 1-7 This is the second embodiment of the present invention, which provides a multimodal fault diagnosis method for a drilling machine, including the following steps: S1: Extracting multi-period temporal features from sensor time-series data, and extracting multimodal visual features from infrared thermal imaging images and RGB visible light images; S2: Constructing a graph structure based on the physical association of key components of the drilling machine, mapping the multi-period temporal features and the multimodal visual features to corresponding nodes of the graph structure, and performing feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features of fused component spatial association; S3: Using the spatiotemporal features as query vectors and the multimodal visual features as key and value vectors, performing feature fusion through a cross-modal attention mechanism to obtain multimodal fused features; S4: Mapping the multimodal fused features to preliminary fault prediction results, and calibrating the preliminary fault prediction results sequentially based on equipment physical constraint rules and node association of the graph structure, and outputting the final fault type and severity diagnosis results.

[0054] This embodiment is the method implementation corresponding to the system embodiment, and its execution flow corresponds one-to-one with the functions of the system modules. Step S1 corresponds to the feature extraction process, and its input is the source data after complete preprocessing: vibration and pressure sensor data need to be synchronized, denoised, calibrated, and have basic time and frequency domain features extracted; infrared and RGB images need to be registered, calibrated, noise suppressed, and normalized. Step S2 is the spatiotemporal modeling process, the key of which is to construct a spatiotemporal graph with clear physical meaning. The number of nodes N is consistent with the number of core components of the equipment (such as drill bits, spindles, and bearing seats), and each node is assigned the initial temporal and visual fusion features obtained in step S1. The graph convolutional network performs feature propagation on this graph. Its adjacency matrix A is calculated by incorporating physical distance and connection type weights. The propagation formula is: This achieves the collaborative aggregation of spatial and temporal neighborhood information. Step S3 performs cross-modal fusion, the core of which is to perform a fusion based on spatiotemporal features. Guided, visually-oriented Attention query, calculation And finally passed The deep fusion result is obtained. Step S4 includes prediction and two-level calibration. First, the original prediction is obtained through a two-layer MLP network with simplified parameters. Then, physical logic filtering based on fault probability threshold and neighborhood weighted smoothing based on graph adjacency matrix are applied sequentially to ensure the output result is accurate. This approach is both accurate and conforms to the physical laws governing equipment operation and fault propagation. The entire methodology is designed around the core requirement of balancing accuracy and efficiency in industrial scenarios with limited sample sizes.

[0055] Specifically, step S1 involves extracting multi-period time-series features, including: S11: preprocessing and performing Fast Fourier Transform on the raw time-series data from the pressure sensor and vibration sensor; S12: calculating the amplitude and selecting the preceding period with the largest amplitude. Each frequency is used as a key frequency component. And calculate the corresponding period length. S13: Based on the length of each cycle One-dimensional time series data Remodeling into a two-dimensional tensor S14: For each two-dimensional tensor Feature enhancement processing is performed, and weighted aggregation is conducted based on the amplitude weights of each key frequency component to obtain the multi-cycle time-series features. .

[0056] This step details the temporal feature extraction process in the method described. In the S11 preprocessing stage, for vibration data, this specifically includes data synchronization based on the equipment operation log timestamp, noise suppression using a 5-level decomposition with a db4 wavelet basis, removal of trend terms through linear fitting, and calculation of time-domain indices such as peak value, RMS value, and kurtosis, as well as frequency-domain indices such as peak frequency, to form a basic feature set. For pressure data, this includes synchronization based on the PLC timestamp, denoising using 50Hz low-pass filtering and moving average filtering, and processing according to the calibration formula. in This is the actual pressure value. For digital quantization values, This is the proportionality coefficient. For the offset, range calibration is performed, and time-domain indicators such as mean pressure and peak pressure are calculated. Preprocessed and standardized multivariate time-series data constitute... The S12 frequency domain analysis section... Perform a Fast Fourier Transform (FFT) to calculate the amplitude: in, The calculation of the amplitude value It is a mean calculation. This represents the amplitude vector calculated at each frequency, with the largest amplitude selected first. Frequency And then according to the formula Determine the period length corresponding to each key frequency. S13 Two-dimensional reshaping stage, for each selected cycle Through fill operation and reshaping operation , one-dimensional sequence Convert to a two-dimensional tensor Its size is This operation transforms temporal periodic patterns into spatial structural patterns that can be captured by two-dimensional convolution. The S14 feature enhancement and aggregation stage... The data is fed into a TimesNet module composed of multiple stacked TimesBlocks for feature learning. Each TimesBlock contains operations such as two-dimensional convolution and is connected via residuals. Maintain information flow. After all cycle paths have been processed, the output two-dimensional representation is flattened into one dimension, and the amplitude is determined based on its source frequency. Weights obtained by Softmax normalization Perform weighted summation This integrates multi-scale periodic dynamic information to form the final multi-period time series characteristics.

[0057] Specifically, the calibration in step S4 includes: S41: applying physical constraints to the preliminary fault prediction results to ensure logical consistency between fault type prediction and severity prediction; S42: utilizing the adjacency matrix of the graph structure. After physical constraint calibration, the first Fault type prediction results for each node Severity prediction results Spatial smoothing is performed using the following formula:

[0058]

[0059]

[0060] in, Represents a node The set of spatial neighbor nodes, and This is the final output result.

[0061] This step is a crucial post-processing step to ensure that the diagnostic results conform to physical reality and equipment topology. S41 physical constraint calibration aims to directly correct any inconsistencies in the original model output that violate basic equipment operating logic. Specifically, for any component node i, if its predicted fault type has the highest probability... If the value is below a preset empirical threshold θ (e.g., 0.5), the component is determined to be in the category of "no definite fault" or "normal condition". Based on the physical principle that "no fault means no severity", the predicted original severity is irrelevant at this point. The numerical value is determined by the indicator function. Force it to zero, i.e., apply Furthermore, the fault type probability vector itself can also be constrained, such as by scaling it using the Sigmoid function, to obtain... This constitutes the preliminary calibration results. S42 spatial smoothing calibration goes a step further, leveraging the characteristic that faults in the mechanical structure of equipment typically do not occur in isolation but propagate along physical connection paths. It directly reuses the adjacency matrix A constructed in step S2, which contains the fault propagation intensity. For each node i, its final diagnostic result is not solely determined by its own calibration result. The decision is not made by the nodes of its neighbors. calibration results Perform a weighted average. The weights are the corresponding edge weights in the adjacency matrix. The larger this value, the stronger the correlation between the faults of the two components, and the more similar their diagnostic results should be. The denominator in the smoothing formula ( () is a normalization factor that ensures the output value range is reasonable. This operation can effectively "smooth out" predictions that are seriously inconsistent with the state of surrounding components due to random noise or local sensor errors, so that the diagnostic results of the entire system show continuity and consistency in spatial distribution that conforms to the fault propagation law, greatly improving the robustness and reliability of the diagnostic results.

[0062] The working principle of the method described in this embodiment is as follows: First, a multi-source sensing system synchronously acquires vibration and pressure time-series signals, as well as infrared and RGB images of the PCB drilling machine during operation. These heterogeneous data undergo refined preprocessing and strict spatiotemporal alignment to form the standard input for the model. Next, two paths are executed in parallel during the feature extraction stage: one path transforms the time-series signals to the frequency domain using a Fast Fourier Transform, identifies and extracts multiple periodic components most relevant to the fault, and then mines its deep periodic dynamic patterns through two-dimensional reshaping and the TimesNet module to obtain multi-period time-series features; the other path divides the registered multi-channel images into image blocks aligned with physical components, inputs them into the ViT model, and utilizes its global self-attention mechanism to capture visual anomaly correlations across components, extracting multimodal visual features. Subsequently, the spatiotemporal modeling stage begins. A component-level spatiotemporal graph is constructed based on the equipment's mechanical assembly drawing. The extracted time-series and visual features are assigned to corresponding nodes. A graph convolutional network simultaneously aggregates the spatial neighbor information and the node's own temporal evolution information to generate spatiotemporal features that reflect the spatiotemporal coupling law of the fault. Then, cross-modal fusion is performed, using spatiotemporal features as query vectors to guide an attention mechanism to retrieve the most relevant abnormal appearance information from visual features. These are then deeply fused through residual connections to form a multimodal fusion feature that comprehensively represents the equipment's state. Finally, a lightweight fully connected network maps the fused features to preliminary diagnostic results, and then applies physical logic constraints based on fault probability thresholds and spatial smoothing optimization based on component correlation graphs. This results in a final diagnostic report that accurately detects fault types and severity while fully conforming to the equipment's physical laws and fault propagation characteristics, thereby achieving reliable intelligent diagnosis of complex industrial equipment under limited sample conditions.

[0063] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal fault diagnosis system for drilling machines, characterized in that: include, The feature extraction module is used to extract multi-period time-series features from sensor time-series data, as well as multi-modal visual features from infrared thermal imaging images and RGB visible light images. The spatiotemporal modeling module is used to construct a graph structure based on the physical relationship between the key components of the drilling machine, map the multi-period temporal features and the multimodal visual features to the corresponding nodes of the graph structure, and perform feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features that fuse the spatial relationship between the components. The cross-modal fusion module is used to perform feature fusion using the spatiotemporal features as query vectors and the multimodal visual features as key and value vectors, through a cross-modal attention mechanism, to obtain multimodal fused features. The fault prediction and calibration module is used to map the multimodal fusion features into preliminary fault prediction results, and then calibrate the preliminary fault prediction results based on the equipment physical constraint rules and the node association relationship of the graph structure, and output the final fault type and severity diagnosis results.

2. The drilling machine multi-mode fault diagnosis system as described in claim 1, characterized in that: The feature extraction module includes a temporal feature extraction unit, which is used for: Preprocessing and frequency domain transformation are performed on the raw time-series data of pressure sensors and vibration sensors; Select the k key frequency components with the largest amplitude from the frequency domain results, and determine the period length corresponding to each key frequency; Based on the period lengths mentioned above, the one-dimensional time series data are reshaped into multiple two-dimensional tensors; The two-dimensional tensors are subjected to feature enhancement processing, and the processed features are aggregated into the multi-period time series features based on the amplitude weights of each key frequency component.

3. The drilling machine multi-mode fault diagnosis system as described in claim 1 or 2, characterized in that: The feature extraction module includes a visual feature extraction unit, which employs a visual transformer model and is used for: The preprocessed infrared temperature image is stitched together with the RGB image to form a multi-channel input image. The multi-channel input image is divided into non-overlapping image blocks that correspond one-to-one with the key components and then embedded and encoded. The multimodal visual features are extracted through the multi-head self-attention mechanism and encoding layer of the visual transformer model, where each feature vector corresponds to a key component.

4. The drilling machine multi-mode fault diagnosis system as described in claim 3, characterized in that: In the spatiotemporal modeling module, the graph structure is a spatiotemporal graph, whose nodes correspond to the key components, and whose edges include spatial edges and temporal edges; The weights of the spatial edges are calculated based on the physical connection distance and connection type weights between components; The spatiotemporal modeling module uses the graph convolutional network to fuse the node's own features, the features of spatial neighboring nodes, and the node features of adjacent time steps in a weighted aggregation manner to generate the spatiotemporal features.

5. The drilling machine multi-mode fault diagnosis system as described in claim 1, characterized in that: The calculation process of the cross-modal attention mechanism executed by the cross-modal fusion module is as follows: in, , , , For the aforementioned spatiotemporal features, For the multimodal visual features, , , For a learnable projection matrix, The dimension of the key vector; The output of the attention mechanism is compared with the spatiotemporal features. The multimodal fusion features are obtained by adding the residual connections. .

6. The drilling machine multi-mode fault diagnosis system as described in claim 1, characterized in that: The fault prediction and calibration module includes: A lightweight mapping unit is used to map the multimodal fusion features into a preliminary fault prediction result containing the fault type probability distribution and the initial value of the fault severity through a two-layer fully connected network; A physical constraint calibration unit is used to correct the preliminary fault prediction results according to preset rules, including: when the maximum probability of a fault type of a certain component is lower than a threshold. When this happens, the severity of the fault corresponding to that component is forcibly reset to zero; The spatial smoothing calibration unit is used to reuse the graph structure adjacency matrix in the spatiotemporal modeling module, and to perform weighted average smoothing on the prediction results of each node after physical constraint calibration, according to the prediction results of its neighboring nodes, to obtain the final fault diagnosis result.

7. The drilling machine multi-mode fault diagnosis system as described in claim 1, characterized in that: The system uses a total loss function during training. Loss from classification regression Domain adaptation loss and modal consistency loss Weighted composition: , Wherein, the classification regression loss This includes classification loss based on visual features, classification loss based on temporal features, and severity regression loss; The domain adaptation loss Used to reduce the difference in feature distribution between the source and target domains; The modal consistency loss The consistency point prediction results used to constrain the single-modal classification results and the multimodal fusion classification results are weighted and smoothed according to the prediction results of their neighboring nodes to obtain the final fault diagnosis result.

8. A multi-modal fault diagnosis method for a drilling machine, characterized in that: Includes the following steps: S1: Extract multi-cycle time-series features from sensor time-series data, and extract multi-modal visual features from infrared thermal imaging images and RGB visible light images; S2: Construct a graph structure based on the physical relationship between the key components of the drilling machine, map the multi-period temporal features and the multimodal visual features to the corresponding nodes of the graph structure, and perform feature propagation and aggregation through a graph convolutional network to generate spatiotemporal features that fuse the spatial relationship between the components. S3: Using the spatiotemporal features as the query vector and the multimodal visual features as the key vector and value vector, feature fusion is performed through a cross-modal attention mechanism to obtain multimodal fused features; S4: Map the multimodal fusion features to preliminary fault prediction results, and calibrate the preliminary fault prediction results sequentially based on the equipment physical constraint rules and the node association relationship of the graph structure, and output the final fault type and severity diagnosis results.

9. The multi-modal fault diagnosis method for drilling machines as described in claim 8, characterized in that: The extraction of multi-period time-series features in step S1 specifically includes: S11: Preprocess and perform fast Fourier transform on the raw time-series data of the pressure sensor and vibration sensor; S12: Calculate the amplitude and select the one with the largest amplitude. Each frequency is used as a key frequency component. And calculate the corresponding period length. ; S13: Based on the length of each cycle One-dimensional time series data Remodeling into a two-dimensional tensor ; S14: For each two-dimensional tensor Feature enhancement processing is performed, and weighted aggregation is conducted based on the amplitude weights of each key frequency component to obtain the multi-cycle time-series features. .

10. The multi-modal fault diagnosis method for drilling machines as described in claim 8, characterized in that: The calibration in step S4 specifically includes: S41: Apply physical constraints to the preliminary fault prediction results to ensure logical consistency between fault type prediction and severity prediction; S42: Utilizing the adjacency matrix of the graph structure After physical constraint calibration, the first Fault type prediction results for each node Severity prediction results Spatial smoothing is performed using the following formula: in, Represents a node The set of spatial neighbor nodes, and This is the final output result.

Images