Aviation blade anomaly detection method and device based on multi-modal zero sample learning

Abstract

The invention relates to the technical field of equipment anomaly detection, in particular to an aviation blade anomaly detection method and device based on multi-modal zero sample learning, and the method comprises the steps: firstly carrying out the synchronous alignment of multi-modal time series data in an input link; realizing cross-modal feature depth interaction on the multi-modal alignment features by using a multi-head attention mechanism; meanwhile, a plurality of expert sub-models are introduced to perform semantic enhancement on the features, and weights of the expert sub-models are calculated in combination with metadata such as equipment models, rated rotating speeds and power, so that the models can automatically select optimal feature sub-spaces according to different equipment working conditions; and finally, introducing a CLIP-based large model semantic reasoning mechanism, constructing a fault category semantic prototype by using a natural language template, and carrying out semantic enhancement on key time steps in a time sequence signal through attention time pooling, so that the model can effectively identify unseen faults under a zero sample condition.

Description

Technical Field

[0001] This invention relates to the field of equipment anomaly detection technology, and in particular to a method and device for detecting anomalies in aircraft blades based on multimodal zero-shot learning. Background Technology

[0002] As the core power component of aircraft, the safety and stability of aero-engines directly affect the reliable execution of flight missions and the level of aviation safety. Blades, as the most critical energy conversion and load-bearing units in an engine, have a significant impact on overall engine performance due to their operational status. With the continuous improvement of thrust-to-weight ratio and combustion efficiency, blades, operating under extreme environments such as high temperature, high speed, high pressure, and strong airflow impact for extended periods, are highly susceptible to various types of failures, including cracks, wear, cavitation, icing, corrosion, and fatigue fracture. If these anomalies are not identified early enough, they can lead to serious accidents such as reduced engine thrust, excessive vibration, or even blade detachment. Therefore, blade anomaly detection and health monitoring (BHM) has become an important research direction in the intelligent support system for aero-engines.

[0003] Traditional methods for detecting aircraft blade faults mainly rely on manual experience, periodic shutdown inspections, or rule-based monitoring systems. These methods have significant shortcomings: first, they are poorly adaptable to complex operating conditions, with a significant decrease in identification accuracy under new blade structures or variable operating conditions; second, they have long detection cycles and poor real-time performance, often only discovering problems after obvious damage has occurred on the blade; and third, they require extensive manual annotation and experience-based judgment, making it difficult to support the high-frequency operation and remote health management needs of modern aircraft engines.

[0004] With the development of sensor and data acquisition technologies, multi-source monitoring data of aero-engine blades (including vibration signals, acoustic emission signals, temperature distribution, infrared thermography, and high-frame-rate optical images) are constantly accumulating, providing a solid foundation for data-driven anomaly detection methods. However, how to extract effective features from high-dimensional, heterogeneous, and time-varying data, and achieve multimodal fusion and intelligent decision-making, has become a technical bottleneck in the current field of blade anomaly detection.

[0005] In recent years, breakthroughs in deep learning for tasks such as image recognition, signal processing, and time series prediction have provided new solutions for anomaly detection in aero-engine blades. Models represented by convolutional neural networks (CNNs), recurrent neural networks (RNNs), and time-series convolutional networks (TCNs) can automatically learn features from raw vibration, acoustic, and thermal imaging data, achieving end-to-end modeling from feature extraction to fault identification. By fusing information from multiple sensor sources, deep neural networks can perform high-precision detection of typical anomalies such as blade cracks, wear, aerodynamic stall, and localized overheating, achieving significant results in laboratory environments and some engineering applications.

[0006] Currently, the main technologies in the field of equipment fault and anomaly detection include the following:

[0007] (1) Multimodal fusion technology:

[0008] To address the shortcomings of insufficient single-modal information, multimodal feature fusion has become a mainstream solution. For example, by converting vibration signals into time-frequency spectra through continuous wavelet transform, and combining them with attention mechanisms to fuse one-dimensional vibration time series and two-dimensional time-frequency image features, the diagnostic robustness of screw compressors in refrigeration equipment is significantly improved. In the power sector, multimodal attention fusion models simultaneously process three-phase current time series and time-frequency images generated by synchronous compression transform, using time-series attention and visual attention modules to focus on key fault features, achieving a fault diagnosis accuracy of 98.4% for modular multilevel converter switches. These technologies effectively address the challenges of detecting anomalies caused by noise interference and early, subtle faults through cross-modal complementary information.

[0009] (2) Small sample and zero sample learning:

[0010] To alleviate model overfitting caused by scarce fault samples, generative data augmentation and semantic mapping become crucial. The DiffViT-IBFD method combines a diffusion model with a visual Transformer, utilizing an improved ReC-Unet network to generate high-fidelity time-frequency images, maintaining an accuracy of 74.36% even under extremely imbalanced (30:1) bearing fault data conditions. Meanwhile, sparse representation techniques, through adjustable sparsity fusion coding, dynamically combine orthogonal matching pursuit and compressed sampling matching pursuit, adaptively adjusting sparsity based on the root mean square value of the signal, achieving an accuracy of 99.12% in planetary gearbox diagnostics. Zero-sample directions rely on fault attribute vectors to construct the semantic space, but the sensitivity of existing methods to equipment parameters (such as power and speed) still needs optimization.

[0011] (3) Cross-domain adaptive technology:

[0012] To address feature distribution shifts caused by differences in equipment models and operating conditions, domain adversarial training and transfer learning have become research hotspots. Domain adversarial networks align the feature distributions of the source and target domains through gradient backpropagation, but require pre-training with hundreds of target domain samples. More advanced reconstruction domain adaptive transfer networks introduce reconstruction modules and particle swarm optimization to address the problem of abnormal class interference when the target domain label space is smaller than the source domain in some transfer learning scenarios, thus improving cross-device generalization capabilities. Furthermore, adaptive fusion domain cyclic generative adversarial networks achieve an accuracy of 99.16% in aerospace bearing diagnostics through multi-domain (spatial, time-frequency, frequency domain) feature cyclic transfer, fusing synthetic data with real multi-source data.

[0013] (4) Generative models and self-supervised learning:

[0014] Diffusion models and variational autoencoders are driving the development of unsupervised anomaly detection. Adaptive fusion domain recurrent generative adversarial networks employ a smoothing regularized variational framework to stabilize latent space generation and synthesize highly realistic fault data through a multi-index-guided adaptive fusion strategy, addressing the sample imbalance problem. Self-supervised pre-training utilizes masked temporal reconstruction or subsequent prediction tasks (such as using long short-term memory networks to predict vibration signal trends) to learn general representations from unlabeled data, providing an initial foundation for downstream fault classification.

[0015] However, existing methods for detecting equipment malfunctions have the following main technical problems:

[0016] (1) Lack of zero-sample anomaly detection capability: Existing deep learning methods still rely on a large number of labeled samples, which are insufficient in generalization ability for new working conditions, new materials or new types of blade anomalies, making it difficult to cope with complex anomaly patterns that are varied and unknown under real flight conditions; in addition, the semantic descriptions constructed by hand deviate from the actual features and have weak cross-domain adaptability, resulting in low accuracy of unknown anomaly detection.

[0017] (2) Lack of adaptability: Traditional equipment fault anomaly detection methods usually use fixed models and parameters, which cannot be automatically adjusted according to different equipment or different operating conditions. The operating status and fault modes of equipment are diverse and dynamic. Existing methods perform poorly when facing new fault types or changes in equipment characteristics, and are difficult to meet the needs of complex and ever-changing production environments.

[0018] (3) Insufficient time-series data processing capability: Vibration signals, temperature changes and other data generated during equipment operation usually have strong time-series characteristics. Traditional data processing methods cannot effectively capture the long-term dependencies of equipment operating status, which affects the accuracy of fault and anomaly detection. Summary of the Invention

[0019] Therefore, the technical problem to be solved by the present invention is to overcome the problem that existing equipment fault anomaly detection methods rely on a large number of labeled samples and fixed fault categories during training, resulting in a lack of adaptability and insufficient ability to process time series data.

[0020] To address the aforementioned technical problems, this invention provides a method for detecting anomalies in aircraft blades based on multimodal zero-shot learning, comprising: S1: Collect multimodal timing data and equipment metadata of the aircraft generator blades; the equipment metadata includes the aircraft generator model, rated speed, and power. S2: Input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; Multi-layer perceptrons are used to map the multimodal alignment features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fused features. The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; the device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models; the differentiated features output by multiple expert sub-models are weighted and summed to obtain the semantic enhancement features. Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector; S3: Input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category, including: calculating the similarity score of each fault category based on the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. S4: When an unknown fault category appears, set a text description for the unknown fault category, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to S3.

[0021] Preferably, before S2, the method further includes: training an anomaly detection model composed of the feature extraction module and the fault reasoning module, wherein the method is: setting text descriptions of known fault categories, calculating semantic prototypes of known fault categories based on the text descriptions, and constructing an initial semantic prototype space; and supervising training of the anomaly detection model based on multimodal time-series data and device metadata of known fault categories, as well as the initial semantic prototype space.

[0022] Preferably, when training the anomaly detection model, the loss function includes: mutual information constraint loss calculated based on the unified feature space of each modality; The formula for the mutual information constraint loss is expressed as follows: ; in, For mutual information constraint loss, For the unified feature space of the i-th mode, For the unified feature space of the j-th mode, for and mutual information, Belongs to The sample, Belongs to The sample, for and The joint distribution For joint distribution Expectations For the discriminant function, for Independent distribution, for Independent distribution, For independent distributions and The expected measure.

[0023] Preferably, the multimodal time-series data is synchronized and aligned, and a mapping relationship with device metadata is established to obtain multimodal alignment features, including: Multimodal time series data are mapped to a unified common sampling frequency by a resampling operator, and then the frequency-unified multimodal time series data are aligned based on a common time axis. Missing data points are adaptively interpolated using a sliding window strategy. Based on the mean and standard deviation of the aligned multimodal time series data, the aligned multimodal time series data are standardized to obtain standardized multimodal time series data. Establish a mapping relationship between standardized multimodal time-series data and device metadata to obtain multimodal alignment features; The multimodal time-series data includes vibration signal data, temperature data, and image data.

[0024] Preferably, the unified feature space of all modalities is fused using a multi-head attention mechanism to obtain cross-modal fused features, including: The unified feature space of all modalities is combined pairwise and input into a multi-head attention mechanism to obtain the attention output features of all combinations. Among them, each single-head attention mechanism calculates the attention output features of a combination, including: obtaining the query matrix from the unified feature space of one modality in the current combination, obtaining the key matrix and value matrix from the unified feature space of another modality in the current combination, and calculating the attention output features of the current combination. The attention output features of all combinations are concatenated and then multiplied with the fusion mapping matrix to obtain cross-modal fusion features.

[0025] Preferably, attention-based temporal pooling is performed on the semantic enhancement features to obtain a global temporal semantic vector, including: The feature values ​​at each time step in the semantic enhancement features are projected onto the semantic space through a linear mapping to obtain the mapped feature values ​​at each time step, as shown in the formula: ; in, Let be the mapping feature value at time step t. Let be the feature value at time step t in the semantic enhancement features. The linear mapping weight matrix, For bias terms, This is a linear mapping operation; The temporal attention weights for each time step are calculated based on the mapped feature values ​​at each time step, using the following formula: ; in, Let be the temporal attention weight at time step t, and ; For learning and query purposes, This is the weight matrix. The total number of time steps. For the first The mapping feature values ​​at each time step, For the tanh function; The global temporal semantic vector is calculated based on the mapping feature values ​​and temporal attention weights at each time step, using the following formula: ; in, This is a global temporal semantic vector. This is a normalization operation.

[0026] Preferably, based on the global temporal semantic vector and the semantic prototypes of each fault category in the semantic prototype space, the similarity score of each fault category is calculated, using the following formula: ; in, Let c be the similarity score for the c-th fault category. This is a global temporal semantic vector. This is the semantic prototype of the c-th fault category. For temperature parameters, This represents the total number of fault categories.

[0027] Preferably, when predicting the fault category based on the similarity score of each fault category, hierarchical fault reasoning is performed, and the steps are as follows: All fault categories are divided into multiple major fault categories; Log-sum-exp is used to aggregate the similarity scores of all fault categories in each fault category to obtain the group-level similarity score of each fault category. The overall confidence score for each fault category is calculated based on the group-level similarity score for each fault category, using the following formula: ; in, Let g be the overall confidence score for the g-th fault category. The group-level similarity score for the g-th fault category is... A collection of major categories of faults; Based on the overall confidence score of each fault category, the predicted fault category is obtained using the following formula: ; in, To predict major categories of failures; The fault category with the highest similarity score among the predicted fault categories is used as the predicted fault category.

[0028] Preferably, when predicting fault categories based on similarity scores for each fault category, rejecting fault categories that cannot be determined includes: Softmax normalization is used to convert the group-level similarity score of each fault category into the predicted probability of each fault category, and the predicted probability of the fault category with the highest predicted probability is used as the prediction confidence. ; Obtain the similarity scores of the highest and second-highest value fault categories in the predicted fault category, denoted as . and ;by and The difference is used as the discrimination index. ; If the prediction confidence and discrimination meet the judgment criteria, the fault category with the highest similarity score among the predicted fault categories is output as the predicted fault category; if the prediction confidence and discrimination do not meet the judgment criteria, "unknown fault" is output. The determination criteria are as follows: ,in and All of these are preset judgment thresholds.

[0029] The present invention also provides an aircraft blade anomaly detection device based on multimodal zero-shot learning, comprising: The acquisition module is used to acquire multimodal time-series data and equipment metadata of the aircraft generator blades; the equipment metadata includes the aircraft generator model, rated speed, and power. The feature extraction module is used to input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; Multi-layer perceptrons are used to map the multimodal alignment features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fused features. The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; the device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models; the differentiated features output by multiple expert sub-models are weighted and summed to obtain the semantic enhancement features. Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector; The fault reasoning module is used to input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category. This includes: calculating the similarity score between the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. The fault extension module is used to set a text description of the unknown fault category when an unknown fault category occurs, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to the fault reasoning module.

[0030] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention discloses an anomaly detection method for aircraft blades based on multimodal zero-shot learning. Firstly, it synchronizes and aligns multimodal time-series data at the input stage. Through unified sampling frequency, synchronized timestamps, adaptive interpolation, and standardization, it effectively solves the interference caused by inconsistent sampling rates, missing values, and dimensional differences between different sensor data, thus ensuring the consistency and comparability of cross-modal inputs. Secondly, it utilizes a multi-head attention mechanism to achieve deep cross-modal feature interaction on the aligned features, avoiding the drawbacks of single-modality dominance in the fusion results of traditional methods. Simultaneously, it introduces multiple expert sub-models to semantically enhance the features and calculates the expert sub-models based on metadata such as equipment model, rated speed, and power. The weighting mechanism enables the model to automatically select the optimal feature subspace based on different equipment operating conditions, overcoming the limitation of fixed models being unable to adapt to diverse equipment environments and achieving adaptive diagnostic capabilities across operating conditions and equipment. Finally, this invention introduces a CLIP-based large-model semantic reasoning mechanism, innovatively utilizing natural language templates to construct semantic prototypes of fault categories, and semantically enhancing key time steps in time-series signals through attention-based temporal pooling. This allows the model to effectively identify unseen faults under zero-sample conditions, reducing dependence on large-scale labeled data and significantly improving the scalability and semantic interpretability of fault diagnosis, enabling the model to rapidly expand its diagnostic scope with the semantic description of new categories. This invention, while improving the accuracy of known fault detection, breaks through the challenges of identifying unknown faults and cross-equipment adaptation, providing an efficient, reliable, and widely applicable technical solution for equipment health management in intelligent manufacturing.

[0031] Furthermore, the hierarchical reasoning strategy and the confidence-discrimination dual threshold rejection mechanism proposed in this invention effectively reduce the serious consequences of fine-grained classification errors in engineering practice and improve the reliability and fault tolerance of the system in safety-critical scenarios. Attached Figure Description

[0032] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of an aircraft blade anomaly detection method based on multimodal zero-shot learning according to the present invention; Figure 2 This is a flowchart of zero-sample device unknown fault reasoning based on CLIP; Figure 3 This is a visual comparison of different methods in the embodiments of the present invention, wherein... Figure 3 Column (a) in the table contains the acquired visible light image data. Figure 3 Column (b) in the image is the true mask. Figure 3Column (c) shows the detection results of the AE method. Figure 3 Column (d) in the table shows the detection results of the CutPaste method. Figure 3 Column (e) in the table shows the detection results of the PaDiM method. Figure 3 Column (f) in the table shows the detection results of the DSVDD method. Figure 3 Column (g) in the table shows the detection results of the CLIP-AD method. Figure 3 The (h) column represents the detection results of the method of the present invention; Figure 4 This is a flowchart illustrating the application of a multimodal zero-shot learning-based aircraft blade anomaly detection device in an industrial setting. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0034] Reference Figure 1 As shown, this invention provides a method for detecting anomalies in aircraft blades based on multimodal zero-shot learning, comprising: S1: Collect multimodal timing data and equipment metadata of the aircraft generator blades; the equipment metadata includes the aircraft generator model, rated speed, and power. S2: Input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: S21: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; S22: Multi-layer perceptron is used to map the multi-modal aligned features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are fused through a multi-head attention mechanism to obtain cross-modal fused features; S23: Input the cross-modal fusion features into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; input the device metadata into the gating mechanism to obtain the weight distribution of multiple expert sub-models; and sum the differentiated features output by multiple expert sub-models based on the weight distribution of multiple expert sub-models to obtain the semantic enhancement features. S24: Perform attention-based temporal pooling on the semantic enhancement features to obtain the global temporal semantic vector; S3: Input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category, including: calculating the similarity score of each fault category based on the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. S4: When an unknown fault category appears, set a text description for the unknown fault category, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to S3.

[0035] Specifically, the multimodal time-series data includes: vibration signal data acquired by vibration sensors. Temperature data collected by temperature sensors Visible light images acquired through a vision sensor .

[0036] The feature extraction module and the fault reasoning module constitute the anomaly detection model. The feature extraction module specifically includes a data synchronization module, a data fusion module, a multi-expert gating mechanism, and an attention-based temporal pooling mechanism.

[0037] The device metadata The equipment's model, rated speed, and power are represented as follows: ; in, For the device ID, Rated speed (rpm) Power (kW).

[0038] The input data for the anomaly detection model is .

[0039] Preferably, before S2, the method further includes: training an anomaly detection model composed of the feature extraction module and the fault reasoning module, wherein the method is: setting text descriptions of known fault categories, calculating semantic prototypes of known fault categories based on the text descriptions, and constructing an initial semantic prototype space; and supervising training of the anomaly detection model based on multimodal time-series data and device metadata of known fault categories, as well as the initial semantic prototype space.

[0040] Specifically, during the training phase, this invention constructs an initial semantic prototype space using known fault categories, obtaining a mapping function relationship between the semantic prototypes and the global temporal semantic vectors. During the inference phase, when a new location fault occurs, the model only needs to add a natural language template for that unknown fault category, construct its semantic prototype, and add it to the existing semantic prototype space. There is no need to retrain or adjust existing parameters; based on the mapping function relationship obtained during training, inference for unknown fault categories can be completed within the same semantic space.

[0041] This embodiment addresses the complex operating conditions and diverse appearances of aero-engine blades by remapping the seven major categories of faults in traditional industrial equipment into three main categories: structural, surface, and thermochemical faults. Subcategories are then set up under each main category. During model training and testing, these subcategories are divided into known and unknown fault types. The training phase uses only normal samples and samples from some known fault types for supervised learning. The testing phase introduces unknown fault types, using semantic prototype matching to identify and locate unseen anomaly types, thus verifying the generalization ability of zero-sample anomaly detection. The specific fault categories and their descriptions used in this embodiment are shown in Table 1.

[0042] Table 1. Fault Category Table

[0043] The raw multimodal time series data collected by each sensor are as follows: ; in, For vibration signal data, Let be the sampled value of the vibration signal data at the t-th time step. The sampling length of the vibration signal data; For temperature data, Let be the sampled value of the temperature data at time step t. The sampling length for temperature data; For image data, Let be the sampled value of the image data at time step t. This represents the sampling length of the image data.

[0044] Since the sampling lengths of time series data for different modalities are different, it is necessary to synchronize and align the multimodal time series data and establish a correspondence with the device metadata.

[0045] Preferably, in S21, the multimodal time-series data is synchronized and aligned through the data synchronization module, and a mapping relationship with the device metadata is established to obtain multimodal alignment features, including: S211: Mapping multimodal time series data to a unified common sampling frequency using a resampling operator. The formula is: ; in, This represents the timing data of the m-th mode after frequency unification. This represents the raw time series data for the m-th mode. For common sampling frequency, For resampling operation, , and These represent vibration, temperature, and image modes, respectively, with m being the mode index.

[0046] S212: Based on a common timeline The multimodal time series data after frequency unification are aligned, and missing data points are adaptively interpolated using a sliding window strategy. For the Lth time step, This represents the total number of time steps.

[0047] With the first Taking the sampled values ​​at each time step as an example of missing data points, the missing data point is calculated using the sampled values ​​at the time step before and after the missing data point. The formula is as follows: ; in, For the estimated values ​​of the missing data points, and These are the sampled values ​​at the previous and next time steps, respectively, for the missing data points. These are the interpolation weights.

[0048] S213: Based on the mean and standard deviation of the aligned multimodal time series data, the aligned multimodal time series data is standardized to obtain standardized multimodal time series data.

[0049] With the first Taking the sampled values ​​at each time step as an example, the standardized feature value is: ; ; in, These are the standardized eigenvalues. For the first Sampled values ​​at each time step, and denoted as the mean and standard deviation of the time series data for the m-th mode, respectively.

[0050] S214: Establish the mapping relationship between the standardized multimodal time-series data and device metadata to obtain the multimodal alignment feature, the formula of which is: ; in, For index mapping functions, , , These are the standardized vibration signal data, temperature data, and image data, respectively.

[0051] The obtained multimodal alignment features are represented as follows: ,in For vibration signal data alignment features, For temperature data alignment features, For image data alignment features, For embedding device metadata, , , and These are the dimensions of embedding vibration signal data alignment features, temperature data alignment features, image data alignment features, and device metadata, respectively.

[0052] Step S21 aims to unify the processing of inputs from different modalities, such as vibration signals, temperature data, image data, and device metadata, to ensure that subsequent modeling processes operate within a homogeneous data space. Cross-domain information collaboration is achieved through spatiotemporal joint alignment. This module not only addresses the asynchronous nature of traditional sensor data sampling in terms of time but also introduces spatial geometric correction to ensure consistency between visual modalities, thereby providing structured and unified inputs for downstream zero-sample anomaly detection and feature association modeling. This module first maps multi-source time-series data from sensors such as vibration, temperature, and vision to a unified sampling frequency, and then aligns the timestamps of each mode through interpolation or resampling to eliminate time-series deviations caused by inconsistent sensor sampling rates. For observation points with engine blade anomalies, the module employs an adaptive interpolation strategy based on a sliding window to complete the numerical values, thereby ensuring the continuity and stability of the time-series sequence. Subsequently, it performs normalization operations on each modal signal to make its mean zero and variance one, in order to eliminate dimensional differences between different physical quantities and avoid the unbalanced impact of feature amplitudes on model training. Finally, the module establishes a correspondence between the processed multi-modal time-series features and equipment metadata, that is, it associates static descriptions such as equipment model, rated speed, and power with dynamic signal segments under the same index system to form aligned multi-source joint inputs, providing a consistent and scalable data foundation for cross-modal interactive fusion and subsequent zero-shot fault inference.

[0053] Preferably, in S22, the multimodal alignment features and device metadata input data fusion module uses a multilayer perceptron to map the embedding of the multimodal alignment features and device metadata to the same dimension, obtaining a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fusion features, specifically including:

[0054] S221: Using a multilayer perceptron (MLP), the embedded features of multimodal alignment and device metadata are mapped to the same dimension d, resulting in a unified feature space for each modality. The formula is as follows: ; in, Let m be the unified feature space of the m-th mode. It is a multilayer perceptron.

[0055] S222: The unified feature space of all modalities is fused using a multi-head attention (MHA) mechanism to obtain cross-modal fused features, including: The unified feature space of all modalities is combined pairwise and input into a multi-head attention mechanism to obtain the attention output features of all combinations. Among them, each single-head attention mechanism calculates the attention output features of a combination, including: obtaining the query matrix from the unified feature space of one modality in the current combination, obtaining the key matrix and value matrix from the unified feature space of another modality in the current combination, and calculating the attention output features of the current combination. The attention output features of all combinations are concatenated and then multiplied with the fusion mapping matrix to obtain cross-modal fusion features.

[0056] Multi-head attention mechanisms are used to model the interaction relationships between modalities, enabling dynamic information flow between the feature interactions of the i-th modality and the j-th modality through weighted attention. The core computation is as follows: ; in, The key matrix, The key matrix, For value matrices, and Let i and j be the unified feature spaces of the i-th and j-th modes, respectively. For the parameter matrix, As a dimension, This is the softmax activation function.

[0057] Furthermore, to avoid inter-modal representation collapse and improve cross-modal consistency, during the training of the anomaly detection model, this embodiment calculates the mutual information constraint loss for the unified feature space of each modality in this step, and incorporates the mutual information constraint loss into the total loss function. The formula for the mutual information constraint loss is expressed as: ; in, For mutual information constraint loss, For the unified feature space of the i-th mode, For the unified feature space of the j-th mode, for and mutual information, Belongs to The sample, Belongs to The sample, for and The joint distribution For joint distribution The expectation is used to measure the similarity of the same anomalous fault between two modes, and improve mutual information to enhance the consistency of multi-source datasets; For discriminant functions; for Independent distribution, for Independent distribution, For independent distributions and The expected measure is that the model expects low mutual information between them to avoid erroneous coupling and to distinguish between abnormal and normal.

[0058] In multi-source anomaly detection, semantic consistency between different modalities is crucial for anomaly identification. Therefore, a mutual information constraint term is introduced in S222 during model training to maximize the mutual information of homologous samples. The mutual information is used to minimize the correlation of random sample pairs, achieving consistent alignment across modal feature spaces. Within the mutual information constraint, This represents the expectation of multimodal positive samples from the same location. This represents the expectation of independent random samples. This constraint enhances the model's ability to model multi-source coupling in normal regions and exhibits a significant decrease in mutual information in abnormal regions, thereby achieving anomaly detection. This constraint promotes high correlation and semantic consistency of different modal features in a unified representation space, avoiding the dominance of a single modality in the fusion result.

[0059] Through collaborative optimization using multi-head attention and mutual information constraints, the fused cross-modal features can be formalized as follows: ; in, Cross-modal fusion features serve as the core input for downstream zero-sample fault diagnosis and performance prediction; For splicing operations, As a multi-head attention mechanism, This is the fusion mapping matrix.

[0060] Specifically, in S23, the multi-expert gating mechanism includes multiple expert sub-models and a gating mechanism. Cross-modal fusion features are input into multiple expert sub-models respectively to obtain differentiated features output by each expert sub-model. Device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models. Based on the weight distribution of each expert sub-model, the differentiated features output by each expert sub-model are weighted and summed to obtain semantically enhanced features.

[0061] This embodiment introduces a multi-expert gating mechanism based on equipment metadata. This aims to fully leverage the complementarity between static equipment metadata such as equipment model, power, and speed, and dynamic time-series data such as vibration, temperature, and images. This allows for differentiated modeling and adaptive selection of fault characteristics under different operating conditions, thereby improving the accuracy and generalization ability of diagnosis and prediction. Specifically, this step employs a combined structure of "Mixture-of-Experts (MoE) + Gating Mechanism." The input features are divided into multiple subspaces, and different expert sub-models learn the representation capabilities of their respective specific subspaces. The gating mechanism dynamically allocates weights based on equipment metadata, thereby selecting the optimal feature path to achieve efficient modeling across operating conditions.

[0062] There are K expert sub-models These expert sub-models can be used to model the vibration modes of high-speed rotating equipment, the thermal stability characteristics of low-power equipment, or the stress response of high-pressure equipment in a differentiated manner.

[0063] The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model, as expressed by the formula: ; in, The differential features output by the k-th expert sub-model.

[0064] To achieve adaptive selection, device metadata is input into a gating mechanism to obtain the weight distribution of multiple expert sub-models, including: First, store the device metadata. Embedded to the same dimension as the feature space, the embedded device metadata is obtained. ,in It is a learnable mapping matrix; Gating mechanisms use embedded device metadata Using the input as input, calculate the weight distribution of multiple expert sub-models, using the following formula: ; in, The weight distribution of the expert sub-model represents the dynamic allocation of different expert sub-models under the current equipment operating conditions, and ; This is a mapping matrix used to map input features. The mapping from the d-dimensional input space to K expert channels determines how the gating mechanism weights different experts based on the current device state (or input modality), enabling the model to have task adaptability. The bias vector of the gating mechanism allows certain experts to have a priori advantage in specific situations (e.g., bias towards thermal stability experts under high temperature conditions).

[0065] The semantic enhancement features are obtained by weighted summing of the differential features output by each expert sub-model based on their weight distribution, as shown in the formula: ; in, This is a semantic enhancement feature.

[0066] Semantic enhancement features not only maintain global semantic consistency after cross-modal feature interactions but also enhance cross-device transferability through a device metadata-driven gating mechanism. In engineering implementation, expert sub-models can be trained in parallel, and the gating network only introduces a lightweight MLP structure, thus possessing deployability and real-time performance. When new device models or unseen operating conditions appear, the gating mechanism can automatically adjust expert weights based on metadata, activating the most suitable sub-model combination, thereby achieving local cross-device capabilities. This design effectively supports subsequent CLIP-based zero-shot fault inference, which utilizes alignment with cross-modal text priors to complete fault identification and prediction for new devices and new operating conditions without providing samples, ensuring the system remains robust and feasible under zero-shot conditions.

[0067] This step uses cross-modal feature fusion. As input, combined with static metadata such as equipment model, rated power, and speed, differential modeling and adaptive selection are achieved through a multi-expert structure, thereby giving the model local cross-device capabilities across different devices and providing generalizable feature representations for subsequent CLIP-based zero-shot fault reasoning.

[0068] Specifically, in S24, the semantic enhancement features are input into the attention temporal pooling mechanism for attention temporal pooling to obtain a global temporal semantic vector, including:

[0069] semantic enhancement features eigenvalues ​​at each time step Importance modeling is performed by projecting a linear mapping onto the semantic space, yielding the mapping feature values ​​for each time step, as shown in the formula: ; in, For semantic enhancement features The mapping feature value at time step t. Let be the feature value at time step t in the semantic enhancement features. This is a linear mapping weight matrix used to map temporal features from a multi-expert gating mechanism. From the original dimension Projected onto semantic space dimension This allows subsequent attention mechanisms to operate in a unified semantic feature space, determines how temporal step features are represented in the semantic space, and is a pre-alignment step before temporal attention. As a bias term, the mapped semantic features are shifted to ensure smooth feature mapping across time and prevent mean shift after mapping. This is a linear mapping operation; The temporal attention weights for each time step are calculated based on the mapped feature values ​​at each time step, using the following formula: ; in, Let be the temporal attention weight at time step t, and This reflects the degree to which the model pays attention to different instantaneous features in the time dimension; For learning and query purposes, This is the weight matrix. The total number of time steps. For the first The mapping feature values ​​at each time step, For the tanh function; The global temporal semantic vector is calculated based on the mapping feature values ​​and temporal attention weights at each time step, using the following formula: ; in, This is a global temporal semantic vector. This is a normalization operation.

[0070] Attention-based temporal pooling dynamically weights the semantically enhanced temporal features output by the multi-expert gating mechanism, assigning differentiated importance to different time steps. This highlights critical moments containing fault symptoms while suppressing noise and redundant information. Through this mechanism, long sequences are compressed into global temporal semantic vectors. This approach inherits the cross-device adaptive characteristics of multi-expert gating and provides a stable and alignable input representation for subsequent CLIP-based zero-shot fault inference. The attention-based temporal pooling mechanism can compress an entire time sequence into a single semantic vector and highlight key moments, serving as the basis for aligning the sensor-side input with the text semantics.

[0071] In intelligent diagnostic scenarios for industrial equipment, traditional methods often rely on large-scale labeled samples to train robust fault identification models. However, in actual engineering, many fault types are extremely scarce, or even completely lacking in historical samples, making it difficult to widely apply traditional supervised diagnostic methods. This invention proposes a zero-sample equipment unknown fault reasoning method based on CLIP (Contrastive Language-Image Pretraining), which can automatically identify and reason about new fault categories under conditions of no or few samples, significantly improving the system's practicality and scalability.

[0072] Reference Figure 2 As shown, this invention constructs a CLIP-based fault reasoning module, including a text encoder, a similarity calculation submodule, and a prediction output submodule. The CLIP-based fault reasoning module uses a global temporal semantic vector as input to reason about unknown faults in zero-sample devices. Specific steps are described in S31 to S32.

[0073] S31: Obtain the semantic prototype of the fault category: Pre-set multiple text descriptions for each fault category, and use CLIP's text encoder to convert the text descriptions into semantic vectors; average and normalize the semantic vectors belonging to the same fault category to obtain the semantic prototype of each fault category.

[0074] In zero-shot reasoning, a key challenge is establishing effective semantic representations for categories that have never appeared before. This invention addresses this problem through a natural language prompt mechanism. For each fault category c, the system predefines multiple text descriptions. These templates can be manually written by experts or automatically generated by large language models, covering fault mechanisms, typical signal patterns, and descriptions of operating conditions. For example, for "bearing outer ring fault," text descriptions such as "outer ring defects lead to enhanced periodic impact signals" or "the spectrum shows a significant increase in octave peaks" can be designed.

[0075] Subsequently, CLIP's text encoder The text description is converted into a semantic vector using the following formula: ; in, For the m-th text description of the c-th fault category, For text encoders, For normalization operations, The number of semantic templates, This is the semantic vector corresponding to the m-th text description of the c-th fault category.

[0076] Since the same category may have multiple different templates, it is necessary to average and normalize the semantic vectors belonging to the same fault category to obtain the semantic prototypes of each fault category. The formula is as follows: ; in, Let be the semantic prototype of the c-th fault category.

[0077] Through this process, each fault category c corresponds to a unique semantic vector that can be used for alignment. This mechanism enables the creation of a semantic reference for a fault, even without any fault samples, as long as a linguistic description exists, thus providing the possibility for zero-shot classification.

[0078] S32: Using the similarity calculation submodule, the similarity score of each fault category is calculated based on the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space.

[0079] Obtain the global temporal semantic vector of device operation data The next step is to calculate its semantic prototype with respect to each fault category. The similarity between them. This embodiment uses cosine similarity as the metric, and combines it with temperature parameters for scaling to enhance distinguishability. The formula for calculating the similarity score for each fault category is as follows: ; in, Let c be the similarity score for the c-th fault category. This is a global temporal semantic vector. This is the semantic prototype of the c-th fault category. The temperature parameter is typically set to a small value between 0.01 and 0.07 to avoid all class scores becoming too close, thereby improving the model's sensitivity to fine-grained categories. This represents the total number of fault categories.

[0080] S33: The prediction output submodule converts the similarity scores of each fault category into the predicted probability of each fault category and outputs the predicted fault category.

[0081] This embodiment uses Softmax normalization to convert the similarity score into the predicted probability of each fault category, as shown in the formula: ; in, Let be the predicted probability of the c-th fault category. The fault category with the highest predicted probability is output as the predicted fault category.

[0082] This step is equivalent to aligning visual features with linguistic knowledge in a shared semantic space, enabling the raw data to be directly mapped to category labels defined by the language description. Unlike traditional classification networks, this invention does not rely on a fixed set of labels but can dynamically expand the category set based on new text templates, thus naturally supporting zero-shot learning.

[0083] This invention introduces a CLIP-based semantic reasoning mechanism into the model structure, enabling continuous incremental learning for unknown faults. When a new fault category emerges, the model only needs to add one or more natural language templates for that category, without retraining or adjusting existing parameters, to complete reasoning within the same semantic space. The core of this mechanism lies in the fact that when the model of this invention is trained under supervision using known fault categories, the mapping function learned from the global temporal semantic vector and semantic prototype remains fixed. The expansion of the semantic space is only reflected in the incremental addition of language-side templates, thus ensuring that the relative distance between the original categories and the semantic structure are not disrupted.

[0084] This invention maps global temporal semantic vectors to semantic prototypes constructed from natural language templates in a unified semantic space, achieving compatibility between the scalability and structural stability of the semantic space. This enables the model to have the ability to continuously learn incrementally and dynamically adapt to unknown faults at the semantic level.

[0085] In industrial scenarios, fault systems typically exhibit a hierarchical structure. Preferably, this invention proposes an edge-based hierarchical reasoning strategy to improve robustness and interpretability under a large-scale fault database.

[0086] In the anomaly detection task of aero-engine blades, faults typically exhibit a multi-level semantic structure. Blade anomalies can be divided into three main categories: structural, surface, and thermochemical. Each category contains several fine-grained subcategories, such as "fracture," "ablation," "grooving," and "pitting." To improve the robustness and interpretability of the model under complex multimodal conditions, this invention proposes a hierarchical fault reasoning mechanism.

[0087] Specifically, when predicting fault categories based on similarity scores for each fault category, hierarchical fault reasoning is performed, including: All fault categories are divided into multiple major fault categories.

[0088] The similarity scores of all fault categories within each fault category are aggregated using log-sum-exp to obtain the group-level similarity score for each fault category. The formula is as follows: ; in, The group-level similarity score for the g-th fault category is... A collection of major categories of faults. Let g be the set of the g-th fault category.

[0089] This embodiment assigns scores to fault subcategories belonging to the same major fault category (such as "Electrical" or "Mechanical"). The aggregated scores are the overall score for the major category. Log-sum-exp aggregation preserves the differentiability of subclass scores, avoiding gradient instability caused by only taking the maximum value. "Soft aggregation" is performed among a set of related subclasses, enabling the model to output reliable confidence scores at the major category level even when multiple potentially faulty subclasses exist. By calculating the overall confidence score for the major category, the influence of outliers from a single subclass on the results can be avoided while maintaining numerical stability.

[0090] The overall confidence score for each fault category is calculated based on the group-level similarity score for each fault category, using the following formula: ; in, denoted as the overall confidence score for the g-th fault category.

[0091] Based on the overall confidence score of each fault category, the predicted fault category is obtained using the following formula: ; in, To predict major categories of failures.

[0092] The fault category with the highest similarity score among the predicted fault categories is selected as the predicted fault category, using the following formula: ; in, To predict the fault category with the highest similarity score among the fault categories, This is a set of predicted fault categories.

[0093] The two-stage reasoning approach proposed in this embodiment effectively reduces misjudgments, enabling the model to achieve robust classification under known anomaly types. For unseen anomalies (such as new crack morphologies or unknown hot spots), it can still map them to the corresponding major category through semantic similarity. During reasoning, it simultaneously outputs the confidence scores of both the subclass and the major category, achieving interpretable hierarchical decision-making. Ultimately, the model is based on maximizing... By selecting the most likely fault category and combining it with the local subclass response distribution to generate an anomaly location heatmap, hierarchical anomaly reasoning for different regions of the blade can be achieved.

[0094] In safety-critical industrial applications, the cost of incorrect predictions far outweighs the cost of not predicting at all. Therefore, this embodiment proposes a dual rejection mechanism, controlling the output results through dual thresholds for confidence and discrimination. This rejection mechanism functions by performing dual checks on the confidence and discrimination of the prediction results after CLIP-based similarity inference is completed. This ensures that when the model lacks sufficient confidence in a certain type of fault, it outputs "uncertain / unknown" rather than rashly giving an incorrect conclusion. It builds upon the results of the preceding hierarchical inference, performing safety filtering on the prediction distribution, and providing interfaces for subsequent system alarms, manual review, or expert intervention, thereby improving the engineering reliability and practical value of the overall diagnostic process.

[0095] Preferably, when predicting fault categories based on similarity scores for each fault category, rejecting fault categories that cannot be determined includes: Softmax normalization is used to convert the group-level similarity score of each fault category into the predicted probability of each fault category, and the predicted probability of the fault category with the highest predicted probability is used as the prediction confidence. ; Obtain the similarity scores of the highest and second-highest value fault categories in the predicted fault category, denoted as . and ;by and The difference is used as the discrimination index. ; If the prediction confidence and discrimination meet the judgment criteria, the fault category with the highest similarity score among the predicted fault categories is output as the predicted fault category; if the prediction confidence and discrimination do not meet the judgment criteria, "unknown fault" is output. The determination criteria are as follows: ,in and All of these are preset judgment thresholds. It is usually set to 0.6–0.8. It is usually set to 0.05–0.15 to balance the rejection rate and accuracy.

[0096] The rejection mechanism ensures that when the model cannot make a reliable judgment under certain novel, undefined, or low-quality input conditions, the system can proactively output "uncertainty" or "unknown fault," avoiding serious consequences from misdiagnosis. This mechanism can refuse to output incorrect predictions when the model is uncertain or highly ambiguous, instead returning an "unknown / uncertain fault," ensuring that the model can automatically enter a safe fallback mode under novel faults, low-quality inputs, or semantic ambiguity, thereby significantly improving the reliability and interpretability of the industrial diagnostic process.

[0097] Through the above design, this invention not only reduces the dependence on large-scale labeled data, but also supports the identification of new types of faults in zero-sample scenarios, and completes performance prediction by combining historical data while diagnosing faults. It also has good scalability, and can quickly expand new fault categories by adding semantic description vectors without retraining the overall model.

[0098] This invention discloses an anomaly detection method for aircraft blades based on multimodal zero-shot learning. Firstly, it synchronizes and aligns multimodal time-series data at the input stage. Through unified sampling frequency, synchronized timestamps, adaptive interpolation, and standardization, it effectively solves the interference caused by inconsistent sampling rates, missing values, and dimensional differences between different sensor data, thus ensuring the consistency and comparability of cross-modal inputs. Secondly, it utilizes a multi-head attention mechanism to achieve deep cross-modal feature interaction on the aligned features, avoiding the drawbacks of single-modality dominance in the fusion results of traditional methods. Simultaneously, it introduces multiple expert sub-models to semantically enhance the features and calculates the expert sub-models based on metadata such as equipment model, rated speed, and power. The weighting mechanism enables the model to automatically select the optimal feature subspace based on different equipment operating conditions, overcoming the limitation of fixed models being unable to adapt to diverse equipment environments and achieving adaptive diagnostic capabilities across operating conditions and equipment. Finally, this invention introduces a CLIP-based large-model semantic reasoning mechanism, innovatively utilizing natural language templates to construct semantic prototypes of fault categories, and semantically enhancing key time steps in time-series signals through attention-based temporal pooling. This allows the model to effectively identify unseen faults under zero-sample conditions, reducing dependence on large-scale labeled data and significantly improving the scalability and semantic interpretability of fault diagnosis, enabling the model to rapidly expand its diagnostic scope with the semantic description of new categories. This invention, while improving the accuracy of known fault detection, breaks through the challenges of identifying unknown faults and cross-equipment adaptation, providing an efficient, reliable, and widely applicable technical solution for equipment health management in intelligent manufacturing.

[0099] Furthermore, the hierarchical reasoning strategy and the confidence-discrimination dual threshold rejection mechanism proposed in this invention effectively reduce the serious consequences of fine-grained classification errors in engineering practice and improve the reliability and fault tolerance of the system in safety-critical scenarios.

[0100] To verify the effectiveness of this invention, related experiments were also conducted in this embodiment. This embodiment uses the self-constructed AeBAD (Aero-engine Blade Anomaly Detection) dataset for aero-engine blade anomaly detection as the source of training and testing data. This dataset focuses on collecting and labeling structural damage, surface wear, and thermochemical damage that may occur in aero-engine blades under real service or experimental conditions, and also incorporates normal operating condition samples to support known-type fault detection and zero-sample fault identification research. The AeBAD dataset contains multimodal, multi-condition data from blades of various aero-engine models, including vibration signals, temperature profiles, visible light image sequences, and equipment-related metadata (such as speed, power, equipment number, etc.). The vibration signal sampling frequency is 12kHz, the temperature signal is recorded at 10Hz, the visual image resolution is uniformly 800×800, and images are acquired with fixed camera attitude, angle, and exposure parameters to ensure experimental reproducibility and cross-equipment consistency.

[0101] To adapt to the hierarchical zero-shot inference framework proposed in this invention, AeBAD classifies blade faults into four categories: normal samples, cracks / fractures, ablation / hot spots, pitting / corrosion, and coating peeling. Among them, fracture and coating peeling samples are used for supervised training, while the remaining ablation and pitting samples are used as unseen categories for zero-shot inference verification, forming an experimental system of "seen classes + unseen classes".

[0102] Furthermore, to verify the model's robustness across various equipment scenarios, the dataset further incorporates non-destructive disturbances such as multi-speed, multi-temperature gradient, multi-load condition disturbances, as well as changes in illumination, viewing angle, and material reflection. This ensures that the system can not only identify real-world fault types but also distinguish between appearance changes and equipment anomalies, meeting engineering deployment requirements. The multimodal, cross-condition, and hierarchical labeling design of the AeBAD dataset enables it to realistically reflect the complex anomaly behaviors under aero-engine blade operating conditions, providing a representative and challenging validation benchmark for the method of this invention.

[0103] The proposed method was validated using the AeBAD aero-engine blade anomaly detection dataset and compared with several existing mainstream anomaly detection methods, including autoencoders (AE, Baseline), CutPaste, PaDiM, Deep Support Vector Data Description (DSVDD), and the CLIP-AD method based on contrastive learning. The experiments used the same input modality, model operating environment, and inference conditions to ensure consistency and comparability of the comparison results. Experimental results are shown in Table 2.

[0104] Table 2. Experimental Results

[0105] As can be seen from the anomaly detection accuracy (IoU) evaluation results in Table 2, the method of this invention achieved an accuracy of 0.904 in the known anomaly category recognition task, which is significantly improved compared with the traditional AE method (0.752) and the transfer method DSVDD (0.868). This indicates that the cross-modal feature enhancement and multi-expert gating strategy proposed in this invention can improve the feature representation capability and effectively avoid the feature loss problem caused by single-modal dependence.

[0106] In the more challenging zero-shot class detection scenarios, the method of this invention also demonstrates significant advantages. In the four untrained anomaly categories of Fracture, Ablation, Pitting, and Coating Peeling, the method achieves accuracies of 0.864, 0.882, 0.831, and 0.815, respectively, significantly outperforming unsupervised anomaly detection methods (such as AE, PaDiM, and DSVDD) and the contrastive learning method CLIP-AD. In anomaly types with strong fine-grained texture differences, such as Ablation and Pitting, the method still improves performance by approximately 4.5%–6.2% compared to CLIP-AD, proving that the proposed mechanism of "shared semantic alignment + temporal attention pooling + multimodal unified encoding" can effectively bridge the semantic gap between unseen fault categories and achieve more interpretable anomaly category reasoning.

[0107] Figure 3 The results of the visualization comparison of different methods are presented, among which... Figure 3 Column (a) in the table contains the acquired visible light image data. Figure 3 Column (b) in the image is the true mask. Figure 3 Column (c) shows the detection results of the AE method. Figure 3 Column (d) in the table shows the detection results of the CutPaste method. Figure 3 Column (e) in the table shows the detection results of the PaDiM method. Figure 3Column (f) in the table shows the detection results of the DSVDD method. Figure 3 Column (g) in the table shows the detection results of the CLIP-AD method. Figure 3 Column (h) in the table represents the detection results of the method of this invention. It can be seen that traditional methods often suffer from missed detections or coarse localization in scenarios with weak texture damage (such as minor pitting and initial coating peeling). In contrast, the method of this invention provides a more compact and accurate anomaly heatmap that better reflects the actual damage area, demonstrating that the semantically driven anomaly localization mechanism has better reliability and accuracy. Furthermore, the method of this invention can avoid erroneous triggering for non-fault disturbance samples such as changes in illumination, background changes, and weak reflection interference, exhibiting stronger resistance to environmental interference and better adaptability to engineering deployment.

[0108] Comprehensive comparative results show that this invention not only maintains high accuracy in traditional supervised inspection scenarios, but also achieves reliable anomaly identification and damage localization under zero-sample, cross-device, and weakly labeled conditions, meeting the requirements of safety, generalization ability, and interpretability in intelligent health monitoring scenarios for critical aviation components. Therefore, this invention is not a simple superposition of existing technologies, nor is it an algorithm replacement or network structure adjustment, but rather establishes a multimodal semantic alignment and zero-sample inference system for anomaly detection in critical aviation equipment. This system is irreplaceable in terms of model structure, feature learning methods, and inference mechanisms, and for the first time achieves interpretable identification of unknown fault categories without additional training samples.

[0109] Reference Figure 4 As shown, based on the above-mentioned method for detecting aircraft blade anomalies based on multimodal zero-shot learning, this invention also provides an aircraft blade anomaly detection device based on multimodal zero-shot learning, comprising: The acquisition module is used to acquire multimodal time-series data and equipment metadata of the aircraft generator blades; the equipment metadata includes the aircraft generator model, rated speed, and power. The feature extraction module is used to input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; Multi-layer perceptrons are used to map the multimodal alignment features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fused features. The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; the device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models; the differentiated features output by multiple expert sub-models are weighted and summed to obtain the semantic enhancement features. Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector; The fault reasoning module is used to input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category. This includes: calculating the similarity score between the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. The fault extension module is used to set a text description of the unknown fault category when an unknown fault category occurs, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to the fault reasoning module.

[0110] In summary, this invention achieves systematic improvements in three aspects: multi-source data processing, adaptive modeling, and zero-shot inference, constructing an intelligent fault diagnosis method for industrial equipment that combines high accuracy, strong robustness, and good scalability. This invention not only maintains high accuracy in diagnosing known fault classes but also enables intelligent identification of new fault categories in zero-shot scenarios, overcoming the limitations of traditional methods in terms of sample dependence, cross-device migration, and security assurance. It possesses significant engineering application value and broad prospects for widespread application.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for detecting anomalies in aircraft blades based on multimodal zero-shot learning, characterized in that, include: S1: Collect multimodal timing data and equipment metadata of the aircraft generator blades; the equipment metadata includes the aircraft generator model, rated speed, and power. S2: Input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; Multi-layer perceptrons are used to map the multimodal alignment features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fused features. The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; the device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models; the differentiated features output by multiple expert sub-models are weighted and summed to obtain the semantic enhancement features. Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector; S3: Input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category, including: calculating the similarity score of each fault category based on the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. S4: When an unknown fault category appears, set a text description for the unknown fault category, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to S3.

2. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, Before S2, the method also includes: training an anomaly detection model composed of the feature extraction module and the fault reasoning module, wherein the method is: setting text descriptions of known fault categories, calculating semantic prototypes of known fault categories based on the text descriptions, and constructing an initial semantic prototype space; and supervising training of the anomaly detection model based on multimodal time-series data and device metadata of known fault categories, as well as the initial semantic prototype space.

3. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 2, characterized in that, When training the anomaly detection model, the loss function includes: mutual information constraint loss calculated based on the unified feature space of each modality; The formula for the mutual information constraint loss is expressed as follows: ； in, For mutual information constraint loss, For the unified feature space of the i-th mode, For the unified feature space of the j-th mode, for and mutual information, Belongs to The sample, Belongs to The sample, for and The joint distribution For joint distribution Expectations For the discriminant function, for Independent distribution, for Independent distribution, For independent distributions and The expected measure.

4. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features, including: Multimodal time series data are mapped to a unified common sampling frequency by a resampling operator, and then the frequency-unified multimodal time series data are aligned based on a common time axis. Missing data points are adaptively interpolated using a sliding window strategy. Based on the mean and standard deviation of the aligned multimodal time series data, the aligned multimodal time series data are standardized to obtain standardized multimodal time series data. Establish a mapping relationship between standardized multimodal time-series data and device metadata to obtain multimodal alignment features; The multimodal time-series data includes vibration signal data, temperature data, and image data.

5. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, The unified feature space of all modalities is fused using a multi-head attention mechanism to obtain cross-modal fused features, including: The unified feature space of all modalities is combined pairwise and input into a multi-head attention mechanism to obtain the attention output features of all combinations. Among them, each single-head attention mechanism calculates the attention output features of a combination, including: obtaining the query matrix from the unified feature space of one modality in the current combination, obtaining the key matrix and value matrix from the unified feature space of another modality in the current combination, and calculating the attention output features of the current combination. The attention output features of all combinations are concatenated and then multiplied with the fusion mapping matrix to obtain cross-modal fusion features.

6. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector, including: The feature values ​​at each time step in the semantic enhancement features are projected onto the semantic space through a linear mapping to obtain the mapped feature values ​​at each time step, as shown in the formula: ； in, Let be the mapping feature value at time step t. Let be the feature value at time step t in the semantic enhancement features. The linear mapping weight matrix, For bias terms, This is a linear mapping operation; The temporal attention weights for each time step are calculated based on the mapped feature values ​​at each time step, using the following formula: ； in, Let be the temporal attention weight at time step t, and ; For learning and query purposes, This is the weight matrix. The total number of time steps. For the first The mapping feature values ​​at each time step, For the tanh function; The global temporal semantic vector is calculated based on the mapping feature values ​​and temporal attention weights at each time step, using the following formula: ； in, This is a global temporal semantic vector. This is a normalization operation.

7. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, Based on the global temporal semantic vector and the semantic prototypes of each fault category in the semantic prototype space, the similarity score of each fault category is calculated using the following formula: ； in, Let c be the similarity score for the c-th fault category. This is a global temporal semantic vector. This is the semantic prototype of the c-th fault category. For temperature parameters, This represents the total number of fault categories.

8. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 1, characterized in that, When predicting fault categories based on similarity scores for each fault category, hierarchical fault reasoning is performed, with the following steps: All fault categories are divided into multiple major fault categories; Log-sum-exp is used to aggregate the similarity scores of all fault categories in each fault category to obtain the group-level similarity score of each fault category. The overall confidence score for each fault category is calculated based on the group-level similarity score for each fault category, using the following formula: ； in, Let g be the overall confidence score for the g-th fault category. The group-level similarity score for the g-th fault category is... A collection of major categories of faults; Based on the overall confidence score of each fault category, the predicted fault category is obtained using the following formula: ； in, To predict major categories of failures; The fault category with the highest similarity score among the predicted fault categories is used as the predicted fault category.

9. The method for detecting anomalies in aircraft blades based on multimodal zero-shot learning according to claim 8, characterized in that, When predicting fault categories based on similarity scores for each fault category, fault categories that cannot be determined are rejected. The steps are as follows: Softmax normalization is used to convert the group-level similarity score of each fault category into the predicted probability of each fault category, and the predicted probability of the fault category with the highest predicted probability is used as the prediction confidence. ; Obtain the similarity scores of the highest and second-highest value fault categories in the predicted fault category, denoted as . and ;by and The difference is used as the discrimination index. ; If the prediction confidence and discrimination meet the judgment criteria, the fault category with the highest similarity score among the predicted fault categories is output as the predicted fault category; if the prediction confidence and discrimination do not meet the judgment criteria, "unknown fault" is output. The determination criteria are as follows: ,in and All of these are preset judgment thresholds.

10. An aircraft blade anomaly detection device based on multimodal zero-shot learning, characterized in that, include: The acquisition module is used to acquire multimodal time-series data and equipment metadata of aircraft generator blades; The equipment metadata includes the model, rated speed, and power of the aircraft generator; The feature extraction module is used to input multimodal time-series data and device metadata into the feature extraction module to extract a global temporal semantic vector, including: Synchronize and align multimodal time-series data, and establish a mapping relationship with device metadata to obtain multimodal alignment features; Multi-layer perceptrons are used to map the multimodal alignment features and the embedded device metadata to the same dimension to obtain a unified feature space for each modality; the unified feature spaces of all modalities are then fused through a multi-head attention mechanism to obtain cross-modal fused features. The cross-modal fusion features are input into multiple expert sub-models to obtain the differentiated features output by each expert sub-model; the device metadata is input into the gating mechanism to obtain the weight distribution of multiple expert sub-models; the differentiated features output by multiple expert sub-models are weighted and summed to obtain the semantic enhancement features. Attention-based temporal pooling is applied to the semantic enhancement features to obtain a global temporal semantic vector; The fault reasoning module is used to input the global temporal semantic vector into the fault reasoning module to obtain the predicted fault category. This includes: calculating the similarity score between the global temporal semantic vector and the semantic prototype of each fault category in the semantic prototype space, and outputting the predicted fault category based on the similarity score of each fault category. The fault extension module is used to set a text description of the unknown fault category when an unknown fault category occurs, calculate the semantic prototype of the unknown fault category based on the text description, add the semantic prototype of the unknown fault category to the semantic prototype space, and return to the fault reasoning module.

Images