A smart fault diagnosis system for an optoelectronic pod image acquisition device

Abstract

This invention relates to the field of photoelectric detection and fault diagnosis technology, specifically disclosing an intelligent fault diagnosis system for a photoelectric pod image acquisition device. The system includes a data acquisition and preprocessing module, a cross-domain feature alignment and knowledge distillation module, a few-sample meta-learning diagnosis module, and a diagnosis decision and visualization module. By adapting general fault knowledge to the target domain through cross-domain knowledge transfer and combining it with a few-sample meta-learning mechanism, the system can achieve high-precision diagnosis with very few fault samples and possesses the adaptive capability to quickly adapt to new fault modes.

Description

Technical Field

[0001] This invention belongs to the field of photoelectric detection and fault diagnosis technology, and specifically relates to an intelligent fault diagnosis system for a photoelectric pod image acquisition device. Background Technology

[0002] As a critical payload in the aviation, aerospace, and security fields, the stable operation of the image acquisition device in optoelectronic pods is fundamental to the success of reconnaissance, surveillance, and target identification missions. With increasing mission complexity, real-time and accurate fault diagnosis of the pod system has become a core technical requirement for improving equipment reliability and maintenance efficiency.

[0003] Data-driven intelligent fault diagnosis technology, by analyzing sensor data and image features, can automatically identify abnormal equipment states, making it a key means of predictive maintenance. This technology aims to build a mapping model from data to fault modes to achieve early warning and accurate location of potential faults.

[0004] Existing technologies typically rely on deep learning models for fault diagnosis. However, such methods face bottlenecks: as high-value, precision equipment, optoelectronic pods have extremely scarce fault sample data, causing traditional data-driven models to struggle to converge due to insufficient training samples, severely limiting diagnostic accuracy and generalization ability. Although transfer learning technology offers the possibility of utilizing data from relevant domains and overcoming the bottleneck of small-sample learning, how to effectively achieve cross-domain knowledge transfer and build robust and highly generalizable diagnostic models with limited target fault samples remains a pressing technical challenge. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent fault diagnosis system for an optoelectronic pod image acquisition device, so as to solve the technical contradiction in the prior art where the scarcity of optoelectronic pod fault samples makes it difficult for the data-driven diagnostic model based on deep learning to converge, and the diagnostic accuracy and generalization ability are severely limited.

[0006] To achieve the above objectives, the present invention provides an intelligent fault diagnosis system for an optoelectronic pod image acquisition device, comprising: a data acquisition and preprocessing module, used to synchronously acquire multimodal operating data from the optoelectronic pod image acquisition device and its associated sensors, and to standardize and extract features from the acquired data to obtain multimodal feature representations; The cross-domain feature alignment and knowledge distillation module is used to construct a progressive knowledge transfer framework from the source domain to the target domain based on the multimodal feature representation. This framework uses knowledge distillation to transfer the generalized fault diagnosis knowledge learned by the teacher network corresponding to the source domain to the student network corresponding to the target domain, thus overcoming the problem of scarce fault samples in the target domain. The small sample meta-learning diagnostic module is used to perform cloud learning training on the student network, enabling the student network to quickly adapt to new fault modes. The diagnostic decision and visualization module is used to generate the final fault diagnosis conclusions and present them intuitively.

[0007] Preferably, the cross-domain feature alignment and knowledge distillation module includes a source domain pre-trained model library, a dynamic feature distribution alignment unit, and a hierarchical knowledge distillation unit; The source domain pre-trained model library pre-stores multiple deep convolutional neural network models trained on a general mechanical equipment fault dataset; The dynamic feature distribution alignment unit is used to receive the normal state data features of the target domain optoelectronic pod from the data acquisition and preprocessing module, and calculate the maximum mean difference between the features and the intermediate layer feature output of the source domain pre-trained model. The dynamic feature distribution alignment unit is used to dynamically adjust the shallow network weights of the source domain model through the backpropagation algorithm, so as to minimize the maximum mean difference between the feature distributions of the source domain and the target domain. The hierarchical knowledge distillation unit is used to perform knowledge transfer based on feature alignment, including: First, the deep network weights of the source domain pre-trained model are frozen as the teacher network, and the student network is constructed. Then, the following two-stage knowledge distillation process is performed: In the first stage, normal state samples of the target domain are used, and the feature map output by the deep layer of the teacher network is used as a soft target to guide the student network to reconstruct high-level semantic features. The loss function is mean squared error. In the second stage, target domain fault samples are introduced. The output probability distribution of the teacher network's final classification layer is used as a soft label. Combined with the student network's own hard label classification loss for fault samples, the student network is fine-tuned using a weighted cross-entropy loss function. The weight of the soft label loss decreases linearly from 0.9 to 0.1. The number of target domain fault samples is less than the number of normal state samples.

[0008] Preferably, the small sample meta-learning diagnostic module includes a support set and query set construction unit, a prototype network computing unit, and a meta-training optimizer; The support set and query set construction unit is used to randomly construct multiple meta-tasks from a dataset containing labeled target domain fault samples according to the N-way K-shot sampling rule; The prototype network computing unit is used to map all samples in the support set to a high-dimensional feature space through the student network for each meta-task, and to calculate the prototype vector of each fault category in the feature space. The prototype vector is the mean of the feature vectors of all support set samples of that category. For each sample in the query set, calculate the Euclidean distance from its feature vector to the prototype vectors of the N categories, and use the softmax function to transform these distances into probability distributions belonging to each fault category; the meta-training optimizer adopts a gradient-based meta-learning algorithm, and the optimization objective is to minimize the expected loss of the model on a large number of randomly sampled meta-tasks. In each iteration, the meta-training optimizer uses the support set data to calculate the loss of each meta-task and obtain the gradient of the model parameters in the inner loop, and uses the query set data to calculate the loss in the outer loop, and updates the basic parameters of the student network in a way that makes the model perform better on the new task.

[0009] Preferably, the diagnostic decision and visualization module integrates a real-time inference engine, a confidence assessment unit, and a comprehensive visualization interface; The real-time inference engine is used to load the trained student network model, receive the feature vectors output in real time by the data acquisition and preprocessing module, perform forward propagation calculations, and output the preliminary prediction results of the fault category and its probability. The confidence assessment unit is used to calculate the normalized distance between the input feature vector and the prototype vector of the predicted fault category, and to determine the confidence of the preliminary prediction result based on this distance and a preset dynamic threshold. If the confidence is greater than the preset threshold, the diagnosis result is confirmed and the feature vector of the sample is stored in the memory bank of the corresponding category. If the confidence is less than the threshold, it is determined to be an unknown fault or an abnormal state, and an early warning signal is triggered. The integrated visualization interface is used to graphically display the real-time status curves, fault diagnosis results, confidence levels, historical fault records, and remaining service life estimation trend charts based on fault prediction for each subsystem of the optoelectronic pod.

[0010] Preferably, the data acquisition and preprocessing module includes a multi-source data synchronous acquisition unit, a data cleaning and standardization unit, and a primary feature extraction unit; The multi-source data synchronous acquisition unit is used to synchronously acquire, with millisecond-level timestamps, the original image sequence output by the image sensor of the optoelectronic pod, the attitude and position data output by the gyroscope and encoder, the internal ambient temperature data output by the temperature sensor, and the voltage and current data output by the power management unit through a hardware interface and data bus. The data cleaning and standardization unit is used to remove outliers and impute missing values ​​in the collected raw data, normalize all numerical data to the range of 0 to 1, and perform denoising and grayscale normalization on the original image sequence. The primary feature extraction unit is used to extract gradient-based texture features and statistical histogram features from the preprocessed image data, calculate the first-order and second-order difference features of the pose and position data, and calculate the mean, variance, and trend features within the sliding window of the ambient temperature and electrical parameter data.

[0011] Preferably, calculating the maximum mean difference between the dynamic feature distribution alignment unit and the intermediate layer feature output of the source domain pre-trained model specifically includes: calculating the maximum mean difference using a multi-kernel combination method with Gaussian kernel functions; The alignment process of the dynamic feature distribution alignment unit adopts an alternating optimization strategy. After every 5 gradient updates with the maximum mean difference as the loss function, a mini-batch gradient update based on the classification task is performed.

[0012] Preferably, the prototype network computing unit employs an attention mechanism when computing the category prototype vector; The attention mechanism assigns different weights to each sample based on the cosine similarity between the feature vectors of the support set samples, with samples that have high similarity receiving higher weights. The final prototype vector is a weighted average of the feature vectors of all samples.

[0013] Preferably, the dynamic threshold in the confidence assessment unit is adaptively adjusted based on the accuracy of historical diagnostic data; The specific adjustment logic is as follows: for every 100 diagnoses completed by the system, the accuracy rate of the confidence confirmation results in the past 100 diagnoses is calculated; if the accuracy rate is greater than 95%, the dynamic threshold is increased by 5%; if the accuracy rate is less than 85%, the dynamic threshold is decreased by 5%.

[0014] Preferably, the diagnostic decision and visualization module is also connected to a digital twin simulation unit; The digital twin simulation unit constructs a virtual mapping model of the photoelectric pod based on its physical model and historical operation data. When the real-time diagnostic system detects a potential fault or performance degradation trend, the digital twin simulation unit receives fault characteristic parameters, performs fault injection and evolution simulation in the virtual model, predicts the fault development path and its impact on imaging quality, and displays the simulation results in a 3D animation overlaid on the visualization interface.

[0015] Preferably, the multi-source data synchronous acquisition unit adopts a hardware synchronization scheme based on the IEEE 1588 precision time protocol. All sensors are equipped with precision time protocol slave clocks, and the central master clock provides unified time synchronization, achieving a synchronization accuracy of 100 microseconds. The primary feature extraction unit oversamples the angular velocity signal output by the gyroscope and extracts the power spectral density features in the frequency domain within the range of 0 to 500 Hz. The primary feature extraction unit introduces inter-frame motion vector field calculation based on optical flow method into the image sequence, and extracts its root mean square amplitude and direction entropy as features.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention creatively solves the core bottleneck of scarce fault samples in optoelectronic pods by constructing a cross-domain feature alignment and knowledge distillation module. This module utilizes rich, general fault knowledge from the source domain, adapts the source domain model to the target domain feature space through dynamic feature distribution alignment, and then transfers deep semantic knowledge to a lightweight student network through hierarchical knowledge distillation. Essentially, this process transforms the prior knowledge of fault mechanisms contained in large-scale source domain data into transferable feature extraction capabilities suitable for small-sample scenarios in the target domain. This fundamentally breaks through the dependence of traditional models on large amounts of labeled target domain data, enabling the establishment of a high-precision diagnostic foundation even with extremely limited samples.

[0017] 2. This invention integrates a few-sample meta-learning diagnostic module, enabling the system to rapidly adapt to unknown or newly added fault modes. This module optimizes model parameters by simulating a meta-training process of "learning in the task," making it easy to quickly adapt to new tasks. When a new fault mode appears, only a few samples are needed for the system to accurately identify the new fault by calculating the prototype vector of the new category and quickly adjusting the decision boundary. This mechanism upgrades the fault diagnosis system from a static, closed model to a dynamic, open, lifelong learning system, improving the system's long-term applicability and generalization robustness in the face of equipment upgrades or changes in the task environment.

[0018] 3. This invention, through the confidence assessment and digital twin simulation unit in the diagnostic decision-making and visualization module, achieves the quantification of diagnostic results' reliability and the visualization of fault impact. The confidence assessment, based on a dynamic threshold mechanism using feature space distance, effectively distinguishes between confident diagnoses, boundary cases, and unknown anomalies, preventing arbitrary judgments by the model when confidence is low and improving the reliability of system decisions. The digital twin simulation unit transforms abstract fault codes into concrete, visualized physical process simulations, enabling maintenance personnel to intuitively understand fault mechanisms, assess severity, and predict development trends. This elevates intelligent diagnosis from status alarms to decision support, enhancing the system's engineering practicality and maintenance efficiency. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention; Figure 2 This is a schematic diagram of the core principle framework of the cross-domain feature alignment and knowledge distillation module in this invention; Figure 3This is a logical flowchart of the small sample element learning and diagnosis module in this invention; Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow of the diagnostic decision and visualization module in this invention. Detailed Implementation

[0020] Example 1: The intelligent fault diagnosis system for the photoelectric pod image acquisition device of the present invention has the following overall technical architecture: Figure 1 As shown, it consists of four core functional units: a data acquisition and preprocessing module, a cross-domain feature alignment and knowledge distillation module, a small sample meta-learning and diagnostic module, and a diagnostic decision-making and visualization module. These modules are interconnected through standardized data interfaces, forming a complete closed-loop process from front-end perception to back-end decision-making. The following will combine the attached diagram... Figure 1 To be continued Figure 4 This paper will describe the specific implementation of this system in a real deployment environment, step by step.

[0021] The data acquisition and preprocessing module, serving as the sensing front-end of the entire system, is responsible for the synchronous acquisition and preliminary structured processing of multi-source heterogeneous operational data. This module synchronously acquires multimodal operational data from the photoelectric pod image acquisition device and its associated sensors, and performs standardization and feature extraction on the acquired data to obtain multimodal feature representations. The specific implementation of this module is described below.

[0022] First, this module can contain three sub-functional units: a multi-source data synchronous acquisition unit, a data cleaning and standardization unit, and a primary feature extraction unit.

[0023] Among them, the multi-source data synchronous acquisition unit, through a hardware-level timestamp synchronization mechanism, can acquire four types of key operational data in parallel from the photoelectric pod image acquisition device and its associated sensor array with a minimum time granularity of 1 millisecond: The first category consists of raw image sequences output by the image sensor, with a resolution of 1920×1080 and a frame rate of 30 Hz; The second category consists of attitude angles (pitch angle, yaw angle, roll angle) and position coordinates (longitude, latitude, altitude) data jointly output by the gyroscope and encoder, with a sampling frequency of 100 Hz; The third category consists of ambient temperature data collected by temperature sensors located at key thermal nodes inside the pod, with a sampling frequency of 10 Hz. The fourth category consists of power supply voltage and current data monitored in real time by the power management unit, with a sampling frequency of 50 Hz.

[0024] All data streams are transmitted to the central processing unit via a high-speed data bus (such as PCIe 3.0 or Gigabit Ethernet) and are precisely timestamped according to a unified clock source to ensure strict alignment of multimodal data in the time dimension.

[0025] After receiving the timestamped raw data stream, the data cleaning and standardization unit first performs outlier detection and removal. For numerical data (attitude, temperature, electrical parameters), outliers are identified and removed using a statistical method based on the three-standard-deviation principle. For image sequences, burst noise or all-black / all-white frames caused by sensor failure are detected and discarded using the inter-frame differencing method. Subsequently, missing data points are filled in using linear interpolation or spatiotemporal interpolation strategies based on neighboring frames. After cleaning, all numerical data... The normalization formula is used to normalize the closed interval between 0 and 1. ; For the normalized result value, and These are the minimum and maximum values ​​of the parameter in historical operating data, respectively, and the range can be dynamically updated with the system's operating cycle. The image data is processed by a non-local mean denoising algorithm and converted into an 8-bit grayscale image. Its pixel values ​​are also linearly mapped to the range of 0 to 255 and then normalized to 0 to 1.

[0026] The primary feature extraction unit performs feature engineering on the standardized data. For image data, it extracts two types of primary visual features: One is the gradient magnitude histogram calculated based on the Sobel operator, which is used to characterize the texture complexity of an image. Secondly, it is a statistical histogram feature calculated based on the global gray-scale distribution, including four statistical measures: mean, variance, skewness, and kurtosis.

[0027] For attitude and position data, the first-order difference (velocity) and second-order difference (acceleration) of their time series are calculated to capture the instantaneous changes in motion state.

[0028] For temperature and electrical parameter data, calculate the mean, standard deviation, and linear fit slope (trend characteristics) within a sliding window of 10 seconds, with a window sliding step of 1 second.

[0029] All extracted feature vectors are concatenated into a high-dimensional feature vector with 128 dimensions, and a corresponding timestamp label is attached as the above multimodal feature representation. This multimodal feature representation is used as input to subsequent modules, namely the cross-domain feature alignment and knowledge distillation module.

[0030] This cross-domain feature alignment and knowledge distillation module constructs a progressive knowledge transfer framework from the source domain to the target domain based on the multimodal feature representation, overcoming the problem of scarce fault samples in the target domain. It is the core mechanism for solving the problem of scarce fault samples in the target domain, and its internal structure is shown in the attached figure. Figure 2 As shown, the system includes a source domain pre-trained model library, a dynamic feature distribution alignment unit, and a hierarchical knowledge distillation unit. The source domain pre-trained model library pre-stores three deep convolutional neural network models trained on a large-scale public dataset of general mechanical equipment faults. These models all use the ResNet-34 architecture and achieve a classification accuracy of over 98% on the source domain task, demonstrating powerful general fault feature extraction capabilities. The outputs of the intermediate layers (layers 10, 20, and 30) of each model are saved as feature representations.

[0031] The Dynamic Feature Distribution Alignment Unit receives the target domain normal state feature vector (i.e., the multimodal feature representation extracted from the operational data of the photoelectric pod under fault-free conditions) from the primary feature extraction unit and inputs it into the shallow network (first 10 layers) of the source domain pre-trained model. This unit calculates the maximum mean difference (MMD) between the intermediate layer feature outputs of the source domain model on the source domain data and the intermediate layer feature outputs on the target domain normal data. The MMD is calculated using a combination of multi-kernel Gaussian kernel functions with a kernel bandwidth set of {0.1, 1, 10} to accommodate feature distribution differences at different scales.

[0032] When using a single core bandwidth of 0.1, it can only adapt to small-scale feature alignment, with an MMD convergence value of 0.08 and low diagnostic accuracy after cross-domain migration. When using a single core bandwidth of 10, it can only adapt to large-scale feature alignment, with an MMD convergence value of 0.09 and low accuracy. However, when using a multi-core combination of {0.1, 1, 10}, the MMD convergence value drops below 0.05, and the accuracy is improved. This proves that the core bandwidth set can fully cover the scale distribution of multimodal features of the optoelectronic pod and solve the problem of insufficient feature alignment caused by a single core bandwidth.

[0033] Specifically, the MMD value is defined as the squared distance between the source domain feature mean and the target domain feature mean in the Hilbert space of the regenerating kernel. This unit updates only the weights of the first 10 layers of the source domain model using the backpropagation algorithm to minimize the MMD value. The alignment process employs an alternating optimization strategy: after every 5 gradient descent updates with MMD as the loss, a mini-batch gradient update with the cross-entropy of the source domain classification task as the loss is performed once to prevent the loss of discriminative information during feature alignment. This process continues iterating until the MMD value converges to below a preset threshold of 0.05.

[0034] The hierarchical knowledge distillation unit is activated after feature alignment is complete. This unit freezes the deep parts (layers 11 to 34) of the aligned source domain model (at which point the shallow layers have been adapted to the target domain) as the teacher network.

[0035] Simultaneously, a streamlined student network is constructed, with its backbone adopting the MobileNetV2 architecture. The input dimension matches the aligned target domain feature dimension (128 dimensions), and the output layer dimension equals the number of fault categories in the target domain (e.g., 5 categories). Knowledge distillation is divided into two stages.

[0036] In the first stage, a large number of normal state samples (typically greater than 10,000) from the target domain are used. The student network needs to match not only the soft target (i.e., the probability distribution of the softmax output) of the teacher network's final classification layer, but also the feature maps of the three intermediate supervision layers located at 1 / 4, 1 / 2, and 3 / 4 of the network depth. To achieve this, the student network adds additional 1×1 convolutional layers and upsampling layers at the corresponding depths to align the spatial size and number of channels of its output feature maps with those of the teacher network. The total loss function in this stage is a weighted sum of the mean squared error losses of each layer, with the weights allocated as follows: 0.4 for the final layer and 0.2 for each of the three intermediate layers.

[0037] In the second stage, a small number of target domain fault samples are introduced (only 5 to 10 per class), at which point the loss function... Change to weighted cross-entropy form: ; The KL divergence between student network outputs and teacher network soft labels. The cross-entropy between the student network output and the real hard labels, and the weighting coefficients. Starting from 0.9, it linearly decays to 0.1 with each training round.

[0038] When the soft label weight is fixed at 0.9, the student network relies excessively on the generalized knowledge of the teacher network, resulting in low accuracy in fitting the real faults in the target domain. When the weight is fixed at 0.1, the knowledge transfer effect fails, and the student network degenerates into a traditional model that only relies on a small number of fault samples, leading to a decrease in accuracy. However, when a linear decay strategy of 0.9→0.1 is adopted, the accuracy is improved. This strategy not only makes full use of the generalized knowledge of the teacher network to compensate for the lack of samples in the target domain, but also strengthens the fitting of the real fault labels through the later weight decay, effectively solving the contradiction between knowledge transfer and real label adaptation under the scarcity of fault samples in the target domain.

[0039] This design allows the student network to initially mimic the generalization ability of the teacher network, while gradually strengthening its ability to fit real labels in the later stages.

[0040] The small-sample meta-learning diagnostic module is built on the student network obtained through knowledge distillation, aiming to give the system the ability to quickly adapt to new failure modes. Its logical flow is shown in the attached figure. Figure 3 As shown, the few-shot meta-learning diagnostic module consists of three parts: a support set and query set construction unit, a prototype network computation unit, and a meta-training optimizer. The support set and query set construction unit constructs meta-tasks from the labeled target domain fault dataset according to the N-way K-shot rule.

[0041] When N=3, only the three basic fault types of the optoelectronic pod are covered, resulting in a high rate of missed detection for new faults. When N=5, most single fault types of the pod are covered, reducing the missed detection rate. When N=7, the prototype vector's discriminative power decreases due to the large number of categories, leading to an increased diagnostic confusion rate. When K=1, the prototype vector representation has a large bias, resulting in low diagnostic accuracy for small samples. When K=3, the accuracy improves. When K=5, the accuracy only improves, but the required number of fault samples doubles, contradicting the core technology of scarce fault samples in the target domain.

[0042] Therefore, in this embodiment, The value is 5 (meaning each meta-task contains 5 different fault categories). The value is set to 3 (i.e., 3 support samples per category). This addresses the balance between the completeness of fault category coverage and the accuracy of prototype vector representation in small sample scenarios. Each meta-task's support set contains 15 samples (5 categories × 3), while the query set randomly selects 20 samples from the remaining samples (approximately 4 per category). All samples undergo the aforementioned data preprocessing and feature extraction processes. The prototype network computation unit processes each meta-task independently. First, all samples in the support set are input into the student network to obtain their feature vectors in the embedding space, which has a 64-dimensional dimension. Subsequently, for each fault category... Calculate its prototype vector Traditional methods use a simple mean, but this embodiment introduces an attention mechanism for weighted averaging. Specifically, for any two supporting samples in the fault category... and Calculate its eigenvector and cosine similarity .

[0043] sample weight Defined as the sum of its similarity to all other samples of the same type: ; Ultimately, prototype vectors .

[0044] This mechanism allows samples with more concentrated and representative features to account for a larger proportion in prototype computation, thereby improving the robustness of the prototype.

[0045] For any sample in the query set Its eigenvectors After being calculated, with all Calculate the Euclidean distance from each prototype vector. Then, the distance is transformed into a probability distribution using the softmax function: ; This probability distribution This is the prediction result for that sample. It is an iterable variable for fault categories; yes and The squared Euclidean distance between the prototype vectors.

[0046] The meta-training optimizer is optimized using a variant of the MAML (Model-Agnostic Meta-Learning) algorithm.

[0047] In the inner loop, for the current meta-task, the loss (i.e., the cross-entropy between the query set prediction and the true label) is calculated using the support set data, and the gradient of the student network parameters is calculated accordingly to generate task-specific temporary parameters.

[0048] In the outer loop, the loss under temporary parameters is calculated using the query set data, and the underlying parameters of the original student network are updated using this loss.

[0049] The optimization objective is to minimize the model's expected loss on a large number of random meta-tasks. The meta-training process lasts for 5000 epochs, with 100 meta-tasks sampled per epoch. After training, the student network acquires a "learning-to-learn" ability: when faced with a completely new fault category, it only needs to provide... By constructing a support set from a single sample, the query sample can be immediately classified by calculating a new prototype vector, without retraining the entire network.

[0050] The diagnostic decision and visualization module is responsible for integrating all the aforementioned processing results to generate the final, actionable diagnostic output. The interaction relationships are shown in the attached diagram. Figure 4As shown, this module includes a real-time inference engine, a confidence assessment unit, and a comprehensive visualization interface. The real-time inference engine loads a student network model that has been trained using both cross-domain knowledge distillation and few-shot meta-learning. During system operation, it continuously receives 128-dimensional feature vectors output from the data acquisition and preprocessing module, performs forward propagation, and outputs a probability distribution containing N+1 categories (N+1 known fault categories plus 1 "normal" category).

[0051] The confidence assessment unit verifies the reliability of the inference engine's output. The core mechanism is the computation of the input feature vector. With the prototype vector of c Normalized Euclidean distance between This distance is related to the dynamic threshold. Compare. If If so, it is determined to be a high-confidence diagnosis, the fault category is confirmed as c, and then... Stored in the memory of c, it is used for online fine-tuning of the prototype vector (e.g., updating using a moving average). ; This is the updated prototype vector. This is the prototype vector before the update.

[0052] like If the event is identified as a low-confidence event, it may be an unknown fault, a compound fault, or a serious abnormal state. The system will immediately trigger a three-level warning signal (audio-visual alarm, log recording, and data snapshot saving).

[0053] Dynamic threshold It is not fixed, but rather adaptively adjusted based on the system's historical performance. The specific strategy is as follows: For every 100 diagnostic decisions completed by the system, the actual accuracy rate of the diagnostic results that were identified as having high confidence by the confidence assessment unit in the past 100 decisions is calculated (obtained through post-event manual review or cross-validation by redundant sensors).

[0054] If the accuracy rate is greater than 95%, it indicates that the current threshold is too lenient and the system is overconfident, therefore... Increase by 5% (e.g., from 0.3 to 0.315); If the accuracy rate is less than 85%, it indicates that the threshold is too strict and the system is too conservative, therefore... Reduced by 5%.

[0055] When the statistical sample size is 50, threshold adjustments are highly susceptible to random errors, leading to significant fluctuations in diagnostic accuracy. With a statistical sample size of 100, the fluctuation range decreases, balancing real-time performance and stability. Setting the accuracy threshold to 90% is overly lenient, resulting in a high false positive rate; setting it to 95% reduces the false positive rate. Setting the accuracy threshold to 80% is overly strict, leading to a high false negative rate; setting it to 85% reduces the false negative rate. A 5% adjustment range avoids diagnostic fluctuations caused by sudden threshold changes, ensuring stable system operation.

[0056] This mechanism automatically balances the sensitivity and specificity of diagnosis during long-term operation.

[0057] The integrated visualization interface adopts a multi-window layout and integrates the following information: the left area displays the real-time status curves of each subsystem of the optoelectronic pod (gimbal motor, image sensor, temperature control system, power system), including historical trends of parameters such as temperature, voltage, current, and attitude angle; The central area displays the current diagnostic conclusion (such as "normal", "gyroscope drift", "image sensor overheating" etc.) and the corresponding confidence percentage in a prominent color block; The upper right area lists the historical fault records for the past 7 days, including the time of occurrence, fault type, and handling suggestions; The lower right area displays the estimated Remaining Useful Life (RUL) curve based on the current fault state. This curve is generated by an independent LSTM prediction model, whose inputs are the current fault characteristics and historical degradation trajectories.

[0058] In addition, the diagnostic decision-making and visualization module is connected to a digital twin simulation unit. This unit constructs a high-fidelity virtual mapping based on the multi-physics coupling model of the optoelectronic pod (including sub-models such as mechanical dynamics, heat conduction, and optical imaging) and historical operational big data.

[0059] When the real-time diagnostic system detects a potential fault (such as a confidence assessment triggering an early warning) or a performance degradation trend (such as a RUL prediction of less than 30 days), the digital twin simulation unit receives fault characteristic parameters (such as gyroscope deviation and sensor hotspot temperature), injects the corresponding fault into the virtual model, and runs multi-step simulations to predict the evolution path of the fault in the next 24 hours (such as whether the deviation will worsen or whether it will trigger a chain reaction) and its quantitative impact on the final imaging quality (such as image blur and jitter amplitude).

[0060] Simulation results are overlaid in a 3D animation in the central area of ​​the visualization interface. For example, overheated components are highlighted in red, vibration transmission paths are indicated by dynamic arrows, and affected areas of view are covered with semi-transparent blocks. This feature allows maintenance personnel to intuitively understand the physical nature and potential consequences of faults, enabling them to make more informed decisions regarding maintenance scheduling and resource allocation.

[0061] Those skilled in the art should know that the cross-domain feature alignment and knowledge distillation module and the few-sample meta-learning diagnostic module mainly play a role in the training process of the intelligent fault diagnosis system. In some cases, after the intelligent fault diagnosis system is trained, the output of the data acquisition and preprocessing module can be directly input into the student network. After the output of the student network is given to the diagnostic decision and visualization module, the diagnostic decision and visualization module can generate the final fault diagnosis conclusion and present it intuitively.

[0062] In summary, this embodiment achieves accurate synchronization and structuring of multi-source data through a data acquisition and preprocessing module; effectively transfers source domain knowledge through cross-domain feature alignment and knowledge distillation modules to overcome the scarcity of samples in the target domain (i.e., transferring fault diagnosis knowledge from various large-scale general-purpose mechanical equipment to the optoelectronic pod domain); empowers the system with the ability to quickly adapt to new faults through a small-sample meta-learning diagnostic module; and finally achieves reliable, intuitive, and forward-looking fault management through a diagnostic decision-making and visualization module. The entire system forms a complete intelligent closed loop from perception and cognition to decision-making, improving the reliability and maintainability of the optoelectronic pod image acquisition device in complex battlefields or harsh environments.

[0063] Example 2: Based on Example 1, this example optimizes the system specifically for a particular application scenario—the challenges of strong vibration and rapid temperature change faced by the optoelectronic pod when deployed on a high-dynamic flight platform (such as a high-speed UAV). In particular, it strengthens the robust design of the data acquisition and preprocessing module and the small sample learning and diagnostic module.

[0064] In the data acquisition and preprocessing module, the synchronization accuracy of the multi-source data synchronous acquisition unit has been improved to the 100 microsecond level to cope with the drastic changes in attitude data under high-speed maneuvering.

[0065] When the synchronization accuracy is 1 millisecond, the time misalignment between gyroscope attitude data and image sensor data in highly dynamic flight reaches 0.5-1 milliseconds, causing the MMD value of the dynamic feature distribution alignment unit to fail to converge to below 0.05, resulting in low diagnostic accuracy after cross-domain knowledge transfer. When the synchronization accuracy is improved to 100 microseconds, the time misalignment is controlled within 50 microseconds, the MMD value stably converges to 0.03, and the diagnostic accuracy improves. When further improved to 50 microseconds, the accuracy only improves, but the hardware cost increases by 3 times and exceeds the computing power capacity of airborne equipment. Therefore, a synchronization accuracy of 100 microseconds effectively solves the problem of cross-domain knowledge transfer failure caused by the time misalignment of multimodal data in highly dynamic scenarios.

[0066] To address this, the system employs a hardware synchronization scheme based on the IEEE 1588 Precision Time Protocol (PTP). All sensors are equipped with PTP slave clocks, and time is uniformly synchronized by a central master clock. The primary feature extraction unit incorporates dedicated features for vibration interference.

[0067] Specifically, the angular velocity signal output by the gyroscope is oversampled by an additional 2000 Hz on the basis of 100 Hz sampling, and the power spectral density characteristics in the range of 0 to 500 Hz are extracted in the frequency domain, with a focus on the energy concentration in typical mechanical resonance frequency bands such as 100 Hz and 200 Hz.

[0068] When the frequency range is 0-300 Hz, bearing fault characteristics in the 250-400 Hz band are missed, resulting in a low fault identification rate. When the frequency range is 0-500 Hz, the typical fault frequency bands of the pod's gyroscope and gimbal motor are fully covered, improving the fault identification rate. When the frequency range is extended to 0-1000 Hz, a large amount of high-frequency noise is introduced, leading to feature redundancy, decreased diagnostic speed, and no improvement in accuracy. Therefore, the 0-500 Hz frequency range is a creative value that strikes a balance between fault identification rate and diagnostic efficiency, based on the fault characteristic distribution of key components of the optoelectronic pod.

[0069] Meanwhile, for the image sequence, an inter-frame motion vector field calculation based on optical flow is introduced, and its root mean square amplitude and orientation entropy are extracted as features characterizing the impact of platform jitter on imaging.

[0070] In the cross-domain feature alignment and knowledge distillation module, the source domain pre-trained model library has added a model specifically trained on the aero-engine vibration dataset to better match fault modes in highly dynamic scenarios. In the MMD calculation of the dynamic feature distribution alignment unit, the bandwidth set of the Gaussian kernel function is expanded to {0.01, 0.1, 1, 10, 100} to cover a wider range of feature scale variations.

[0071] This embodiment also adjusts the student network architecture of the hierarchical knowledge distillation unit, replacing the depthwise separable convolution in MobileNetV2 with ordinary convolution to enhance the ability to extract high-frequency vibration features. Although this will slightly increase the amount of computation, it is within the limits of the computing power of the onboard GPU.

[0072] This embodiment improves the meta-task construction strategy of the few-sample meta-learning diagnostic module. Considering that faults in highly dynamic environments are often complex (e.g., vibration causing connector loosening, which in turn leads to power fluctuations), the support set and query set construction units no longer sample only according to a single fault category, but introduce a "composite fault" category. In N-way K-shot sampling, N is expanded to 6, of which 5 are single fault categories and 1 is a composite fault category. The support samples for composite faults are generated by linearly mixing the feature vectors of two single fault samples in a certain ratio (e.g., 0.6:0.4) and assigned a new composite label.

[0073] When processing composite fault categories, the prototype network computing unit calculates attention weights not only by considering the similarity between samples of the same type, but also by considering the distance between the composite prototype and the prototype vectors of the two basic fault categories that constitute it, ensuring that the composite prototype can effectively reflect its constituent components.

[0074] The digital twin simulation unit of the diagnostic decision-making and visualization module has also been upgraded. A flexible body dynamics module has been added to its physical model, enabling more accurate simulation of the pod support deformation under strong vibrations. This deformation is then used as a disturbance input to the optical imaging sub-model, thus more realistically predicting the degree of image blurring and defocusing. A new "Vibration Health Index" dashboard has been added to the visualization interface. This index integrates multiple indicators such as gyroscope high-frequency energy, image optical flow amplitude, and motor current harmonics, using a value from 0 to 100 to intuitively reflect the comprehensive impact of the current vibration environment on the pod's imaging performance.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent fault diagnosis system for an optoelectronic pod image acquisition device, characterized in that, include: The data acquisition and preprocessing module is used to synchronously acquire multimodal operating data from the photoelectric pod image acquisition device and its associated sensors, and to standardize and extract features from the acquired data to obtain multimodal feature representations. The cross-domain feature alignment and knowledge distillation module is used to construct a progressive knowledge transfer framework from the source domain to the target domain based on the multimodal feature representation. This framework uses knowledge distillation to transfer the generalized fault diagnosis knowledge learned by the teacher network corresponding to the source domain to the student network corresponding to the target domain, thus overcoming the problem of scarce fault samples in the target domain. The small sample meta-learning diagnostic module is used to perform cloud learning training on the student network, enabling the student network to quickly adapt to new fault modes. The diagnostic decision and visualization module is used to generate the final fault diagnosis conclusions and present them intuitively.

2. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 1, characterized in that, The cross-domain feature alignment and knowledge distillation module includes a source domain pre-trained model library, a dynamic feature distribution alignment unit, and a hierarchical knowledge distillation unit. The source domain pre-trained model library pre-stores multiple deep convolutional neural network models trained on a general mechanical equipment fault dataset; The dynamic feature distribution alignment unit is used to receive the normal state data features of the target domain optoelectronic pod from the data acquisition and preprocessing module, and calculate the maximum mean difference between the features and the intermediate layer feature output of the source domain pre-trained model. The dynamic feature distribution alignment unit is used to dynamically adjust the shallow network weights of the source domain model through the backpropagation algorithm, so as to minimize the maximum mean difference between the feature distributions of the source domain and the target domain. The hierarchical knowledge distillation unit is used to perform knowledge transfer based on feature alignment, including: First, the deep network weights of the source domain pre-trained model are frozen as the teacher network, and the student network is constructed. Then, the following two-stage knowledge distillation process is performed: In the first stage, normal state samples of the target domain are used, and the feature map output by the deep layer of the teacher network is used as a soft target to guide the student network to reconstruct high-level semantic features. The loss function is mean squared error. In the second stage, target domain fault samples are introduced. The output probability distribution of the teacher network's final classification layer is used as a soft label. Combined with the student network's own hard label classification loss for fault samples, the student network is fine-tuned using a weighted cross-entropy loss function. The weight of the soft label loss decreases linearly from 0.9 to 0.

1. The number of target domain fault samples is less than the number of normal state samples.

3. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 2, characterized in that, The small sample meta-learning diagnostic module includes a support set and query set construction unit, a prototype network computing unit, and a meta-training optimizer. The support set and query set construction unit is used to randomly construct multiple meta-tasks from a dataset containing labeled target domain fault samples according to the N-way K-shot sampling rule; The prototype network computing unit is used to map all samples in the support set to a high-dimensional feature space through the student network for each meta-task, and to calculate the prototype vector of each fault category in the feature space. The prototype vector is the mean of the feature vectors of all support set samples of that category. For each sample in the query set, calculate the Euclidean distance from its feature vector to the prototype vectors of the N categories, and use the softmax function to transform these distances into probability distributions belonging to each fault category; the meta-training optimizer adopts a gradient-based meta-learning algorithm, and the optimization objective is to minimize the expected loss of the model on a large number of randomly sampled meta-tasks. In each iteration, the meta-training optimizer uses the support set data to calculate the loss of each meta-task and obtain the gradient of the model parameters in the inner loop, and uses the query set data to calculate the loss in the outer loop, and updates the basic parameters of the student network in a way that makes the model perform better on the new task.

4. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 1, characterized in that, The diagnostic decision and visualization module integrates a real-time inference engine, a confidence assessment unit, and a comprehensive visualization interface. The real-time inference engine is used to load the trained student network model, receive the feature vectors output in real time by the data acquisition and preprocessing module, perform forward propagation calculations, and output the preliminary prediction results of the fault category and its probability. The confidence assessment unit is used to calculate the normalized distance between the input feature vector and the prototype vector of the predicted fault category, and to determine the confidence of the preliminary prediction result based on this distance and a preset dynamic threshold; if the confidence is greater than the preset threshold, the diagnosis result is confirmed and the feature vector of the sample is stored in the memory bank of the corresponding category. If the confidence level is less than the threshold, it is determined to be an unknown fault or abnormal state, and an early warning signal is triggered. The integrated visualization interface is used to graphically display the real-time status curves, fault diagnosis results, confidence levels, historical fault records, and remaining service life estimation trend charts based on fault prediction for each subsystem of the optoelectronic pod.

5. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 1, characterized in that, The data acquisition and preprocessing module includes a multi-source data synchronous acquisition unit, a data cleaning and standardization unit, and a primary feature extraction unit. The multi-source data synchronous acquisition unit is used to synchronously acquire, with millisecond-level timestamps, the original image sequence output by the image sensor of the optoelectronic pod, the attitude and position data output by the gyroscope and encoder, the internal ambient temperature data output by the temperature sensor, and the voltage and current data output by the power management unit through a hardware interface and data bus. The data cleaning and standardization unit is used to remove outliers and impute missing values ​​in the collected raw data, normalize all numerical data to the range of 0 to 1, and perform denoising and grayscale normalization on the original image sequence. The primary feature extraction unit is used to extract gradient-based texture features and statistical histogram features from the preprocessed image data, calculate the first-order and second-order difference features of the pose and position data, and calculate the mean, variance, and trend features within the sliding window of the ambient temperature and electrical parameter data.

6. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 2, characterized in that, Calculating the maximum mean difference between the dynamic feature distribution alignment unit and the intermediate layer feature output of the source domain pre-trained model specifically includes: calculating the maximum mean difference using a multi-kernel combination method with Gaussian kernel functions; The alignment process of the dynamic feature distribution alignment unit adopts an alternating optimization strategy. After every 5 gradient updates with the maximum mean difference as the loss function, a mini-batch gradient update based on the classification task is performed.

7. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 3, characterized in that, The prototype network computing unit employs an attention mechanism when computing the category prototype vector; The attention mechanism assigns different weights to each sample based on the cosine similarity between the feature vectors of the support set samples, with samples that have high similarity receiving higher weights. The final prototype vector is a weighted average of the feature vectors of all samples.

8. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 4, characterized in that, The dynamic threshold in the confidence assessment unit is adaptively adjusted based on the accuracy of historical diagnostic data. The specific adjustment logic is as follows: for every 100 diagnoses completed by the system, the accuracy rate of the confidence confirmation results in the past 100 diagnoses is calculated; if the accuracy rate is greater than 95%, the dynamic threshold is increased by 5%; if the accuracy rate is less than 85%, the dynamic threshold is decreased by 5%.

9. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 1, characterized in that, The diagnostic decision and visualization module is also connected to a digital twin simulation unit; The digital twin simulation unit constructs a virtual mapping model of the photoelectric pod based on its physical model and historical operation data. When the real-time diagnostic system detects a potential fault or performance degradation trend, the digital twin simulation unit receives fault characteristic parameters, performs fault injection and evolution simulation in the virtual model, predicts the fault development path and its impact on imaging quality, and displays the simulation results in a 3D animation overlaid on the visualization interface.

10. The intelligent fault diagnosis system for an optoelectronic pod image acquisition device according to claim 5, characterized in that, The multi-source data synchronous acquisition unit adopts a hardware synchronization scheme based on the IEEE 1588 precision time protocol. All sensors are equipped with precision time protocol slave clocks, which are uniformly synchronized by the central master clock, and the synchronization accuracy reaches the level of 100 microseconds. The primary feature extraction unit oversamples the angular velocity signal output by the gyroscope and extracts the power spectral density features in the frequency domain within the range of 0 to 500 Hz. The primary feature extraction unit introduces inter-frame motion vector field calculation based on optical flow method into the image sequence, and extracts its root mean square amplitude and direction entropy as features.

Images