Avionics fault diagnosis method based on multi-mode distillation and related equipment

Abstract

The invention relates to the technical field of avionics fault diagnosis, and discloses an avionics fault diagnosis method based on multi-modal distillation and related equipment. Through a double-stage architecture design of a double-encoder teacher model and a lightweight student model, low-rank adapter injection and a multi-level knowledge distillation strategy are combined; the problems that an existing efficient parameter fine adjustment method is insufficient in cross-modal semantic alignment in a multi-modal diagnosis task, and cross-modal interaction knowledge is difficult to transmit in traditional knowledge distillation are effectively solved. Due to the introduction of a low-rank adapter, parameter efficiency is kept, and meanwhile, the model is assisted to accurately learn specific features of a multi-modal task; the multi-level knowledge distillation not only aligns model output, but also more fully migrates key knowledge such as cross-modal attention association and a multi-level feature fusion mode in the teacher model, and significantly improves the cross-modal joint reasoning ability of the student model.

Description

Technical Field

[0001] This invention belongs to the field of avionics fault diagnosis technology, specifically a method and related equipment for avionics fault diagnosis based on multimodal distillation. Background Technology

[0002] Intelligent fault diagnosis of modern aviation equipment relies on a deep understanding and fusion of multimodal information such as equipment images and maintenance texts. To achieve localized deployment of diagnostic models on handheld maintenance terminals with strictly limited computing power, power consumption, and latency, it is necessary to effectively compress and adapt large-scale pre-trained models.

[0003] In lightweight model techniques, efficient parameter fine-tuning achieves rapid adaptation to specific tasks by introducing a small number of trainable parameters into the pre-trained model; knowledge distillation transfers knowledge from a large teacher model to a lightweight student model to improve inference efficiency. Specifically, efficient parameter fine-tuning enables the model to learn task-specific features with almost no increase in parameters by injecting low-rank matrices or adapter modules; while knowledge distillation achieves effective knowledge transfer by matching output distributions, intermediate layer features, or attention maps. These techniques constitute the basic technical path for deploying intelligent models on resource-constrained devices and provide a direct technical starting point for constructing the multimodal lightweight diagnostic framework in this application.

[0004] However, existing parameter-efficient fine-tuning methods are mostly geared towards single-modal designs. When dealing with multimodal diagnostic tasks involving both images and text, they struggle to effectively establish and maintain cross-modal semantic alignment, resulting in limited model performance when fusing visual and textual information for joint reasoning. Furthermore, traditional knowledge distillation methods typically only align the model's final output or single-modal features, failing to effectively transfer cross-modal interaction knowledge learned in the teacher model, such as attentional associations between text and images and multi-level feature fusion patterns. This leads to significant deficiencies in the student model's cross-modal reasoning capabilities. Therefore, achieving complete and efficient transfer of multimodal knowledge while maintaining parameter efficiency has become a crucial issue in the lightweight deployment of current avionics fault diagnosis models. Summary of the Invention

[0005] This invention provides a method and related equipment for avionics fault diagnosis based on multimodal distillation, which solves the problem of diagnostic accuracy caused by insufficient cross-modal knowledge transfer in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for avionics fault diagnosis based on multimodal distillation includes: Acquire and preprocess avionics fault image and text data; The preprocessed avionics fault image data and text data are input into the trained avionics fault diagnosis model, and the diagnosis results are output. The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

[0007] Preferably, the training method for the avionics fault diagnosis model is as follows: A dual-encoder teacher model is constructed, which includes a text encoder and a visual encoder to extract the high-level text representation and high-level visual representation of the input avionics data, respectively, and output classification logits, selected intermediate layer features and attention maps. In the self-attention modules of each Transformer layer of the teacher model and the student model to be built, low-rank adaptation modules are injected into the query, key and value projection matrices to update the parameters through low-rank decomposition and keep the original pre-trained weights frozen. A lightweight student model is constructed, which includes a lightweight text encoder and a lightweight visual encoder, and self-supervised pre-training is performed on the student model. A multi-level knowledge distillation strategy is adopted to perform supervised fine-tuning of the pre-trained student model based on the classification logits, intermediate layer features, attention graph, and low-rank adaptation module residual information output by the teacher model. The text and visual features output by the student model are adaptively fused through a dynamic multimodal fusion mechanism to generate a fused representation. The avionics fault diagnosis result is then output through a classification head to complete the model training.

[0008] Preferably, the parameter update of the low-rank adaptation module is specifically as follows: For each pre-trained weight matrix Low-rank decomposition is adopted Perform parameter updates, where , The low-rank adapter is only trained. , The matrix retains the original pre-trained weights frozen, where For rank.

[0009] Preferably, performing self-supervised pre-training on the student model specifically involves: The student model text side uses a masking language for modeling, with a masking rate of [missing information]. The student model image is reconstructed using masked image patches, with a mask rate of [missing information]. Pre-training uses the AdamW optimizer with a learning rate of Batch size is The training period is One epoch, training uses linear warm-up before... The step is followed by a cosine-decreasing learning rate, and the weights decay by... The maximum norm of gradient clipping is Save checkpoints by epoch and stop early when the validation set metrics no longer improve.

[0010] Preferably, the multi-level knowledge distillation strategy specifically involves constructing a hierarchical distillation objective function containing five loss terms, the objective function being:

[0011]

[0012] in, For cross-entropy loss, The softened output distribution of the teacher-student model is aligned using KL divergence. Calculate the L2 distance of the low-rank adapter residual output in the teacher-student model. Feature alignment is performed in the selected intermediate layer. Matching self-attention distribution, , , , and These are the weighting coefficients.

[0013] Preferably, the dynamic multimodal fusion mechanism is as follows: Text features from the text encoder and image features from the visual encoder are mapped to a shared feature space using a learnable linear projection matrix. A learnable modality scaling factor is introduced to dynamically adjust the contribution weights of each modality based on the input content. A gated fusion mechanism is employed, with the gate value calculated using the sigmoid function. The projected features are weighted and fused, and finally the original spliced ​​features and the fused features are added together through residual connections.

[0014] A multimodal distillation-based avionics fault diagnosis system includes: Data acquisition and processing module: used to acquire and preprocess avionics fault image data and text data; Diagnostic module: Used to input preprocessed avionics fault image data and text data into the trained avionics fault diagnosis model and output the diagnosis results; The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of an avionics fault diagnosis method based on multimodal distillation.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an avionics fault diagnosis method based on multimodal distillation.

[0017] A computer program product includes a computer program that, when executed by a processor, implements the steps of an avionics fault diagnosis method based on multimodal distillation.

[0018] Compared with existing technologies, this invention has the following advantages: This invention provides an avionics fault diagnosis method based on multimodal distillation. Through a two-stage architecture design of a dual-encoder teacher model and a lightweight student model, combined with low-rank adapter injection and multi-level knowledge distillation strategies, it effectively solves the problems of insufficient cross-modal semantic alignment in existing efficient parameter fine-tuning methods for multimodal diagnosis tasks and the difficulty of traditional knowledge distillation in transferring cross-modal interactive knowledge. The introduction of the low-rank adapter, while maintaining parameter efficiency, helps the model accurately learn specific features of multimodal tasks; multi-level knowledge distillation not only aligns the model output but also fully transfers key knowledge such as cross-modal attention associations and multi-level feature fusion patterns from the teacher model, significantly improving the cross-modal joint reasoning ability of the student model. This method achieves efficient and lightweight adaptation of large-scale pre-trained models to handheld maintenance terminals, ensuring the accuracy and reliability of avionics fault diagnosis while strictly controlling computational power, power consumption, and latency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an avionics fault diagnosis method based on multimodal distillation according to an embodiment of the present invention; Figure 2This is a training and workflow diagram of an avionics fault diagnosis method based on multimodal distillation according to an embodiment of the present invention; Figure 3 This is a block diagram of an avionics fault diagnosis system based on multimodal distillation, according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, this embodiment of the invention provides an avionics fault diagnosis method based on multimodal distillation, including: S1: Acquire and preprocess avionics fault image and text data; S2: Input the preprocessed avionics fault image data and text data into the trained avionics fault diagnosis model and output the diagnosis results; The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

[0026] By employing a two-stage architecture design with a dual-encoder teacher model and a lightweight student model, combined with low-rank adapter injection and multi-level knowledge distillation strategies, this approach effectively addresses the shortcomings of existing parameter-efficient fine-tuning methods in cross-modal semantic alignment during multimodal diagnostic tasks, and the difficulty of traditional knowledge distillation in transferring cross-modal interaction knowledge. The introduction of the low-rank adapter maintains parameter efficiency while helping the model accurately learn specific features of multimodal tasks. Multi-level knowledge distillation not only aligns the model output but also fully transfers key knowledge from the teacher model, such as cross-modal attention associations and multi-level feature fusion patterns, significantly improving the student model's cross-modal joint reasoning ability. This method achieves efficient and lightweight adaptation of large-scale pre-trained models to handheld maintenance terminals, ensuring the accuracy and reliability of avionics fault diagnosis while strictly controlling computational power, power consumption, and latency.

[0027] like Figure 2 As shown, the detailed training steps for the avionics fault diagnosis model are as follows: S1. Constructing a dual-encoder teacher model: The text encoder adopts XLM-RoBERTa-Large, containing a 24-layer Transformer encoder with a hidden dimension of... 024, the intermediate dimension of the feedforward network is The number of attention heads is 6; The text is taken from the pooled representation of the last layer, then subjected to a linear projection and normalized to the specified level. Dimensions are used as the high-level representation of text; the visual encoder uses ViT-Base-16, and the input is... RGB image and by The split patch contains a twelve-layer Transformer encoder, with hidden dimensions. The number of attention heads is 2; The visual end retrieves the class token, outputs it, performs a linear projection layer, and then normalizes it to... The dimension serves as the high-level visual representation; the teacher model exposes intermediate representations and attention maps hierarchically, specifically from the text's first dimension. , No. , No. 8. With the first Layers and visual layers , No. , No. , No. The intermediate features and corresponding attention maps are extracted for knowledge transfer. The high-level text representation and the high-level visual representation are fused element-wise through linear projection, learnable modality scaling factor and gating mechanism and residual connections are retained to generate a fused representation. The fused representation is output as softmax logits by the classification head. The teacher model outputs classification logits, selected intermediate layer features and attention maps for subsequent distillation and alignment.

[0028] S2. Injecting Low-Rank Adaptor Modules: In the self-attention module of each Transformer layer in both the teacher and student models, inject low-rank adapters into the query, key, and value projection matrices. For each pre-trained weight matrix... Low-rank decomposition is adopted Perform parameter updates, where , ,rank Based on the experiment, it was determined to be... scaling factor Set as The low-rank adapter is only trained. , The matrix maintains the original pre-trained weights frozen, significantly reducing the number of trainable parameters and achieving efficient domain adaptation; By injecting the low-rank adapter module in step S2, only the following is added: It achieves efficient domain adaptation with a limited number of parameters, effectively overcoming the high computational cost of traditional full-parameter fine-tuning, and enabling the student model to adapt to a smaller number of parameters. can still maintain The performance of the teacher model.

[0029] S3. Constructing a lightweight student model: The text encoder uses XLM-R-DistilRoBERTa-Base, containing a six-layer Transformer encoder with a hidden dimension of [missing information]. The visual encoder uses DeiT-Tiny-Distilled, with an embedded dimension of The depth is twelve layers, and the visual output is linearly projected onto... The dimension and text representation share the same vector space. On the student side, a projection head is set after both text and visual outputs to output logits and intermediate features for distillation. The training process first performs self-supervised pre-training on avionics domain data. On the text side, a masking language is used for modeling, with a masking rate of [missing value]. The image side is reconstructed using masked image blocks, with a mask rate of [missing information]. Pre-training uses the AdamW optimizer with a learning rate of Batch size is The training period is One epoch, training uses linear warm-up before... The step is followed by a cosine-decreasing learning rate, and the weights decay by... The maximum norm of gradient clipping is The checkpoints are saved by epoch and stopped early when the validation set metrics no longer improve. After completing the self-supervised pre-training, the student model is supervisedly fine-tuned in the distillation stage with multi-level distillation objectives such as logits alignment, feature alignment, and attention graph alignment to obtain the final deployable lightweight model.

[0030] The lightweight student model constructed in step S3 employs a distilled text and visual encoder architecture, combined with a self-supervised pre-training strategy. This significantly reduces the number of parameters while enhancing the model's ability to represent avionics features, enabling the student model to achieve high performance using only... It can still effectively learn the knowledge representation of the teacher model even with a small number of parameters.

[0031] S4. Design a multi-level knowledge distillation strategy: Construct a hierarchical distillation objective function containing five loss terms:

[0032]

[0033] in, For cross-entropy loss, The softened output distribution of the teacher-student model was aligned using KL divergence, with temperature parameters... ; Calculate the L2 distance of the low-rank adapter residual output of the teacher-student model; Perform feature alignment in the selected intermediate layer; Matching the self-attention distribution. The weight coefficients were determined via grid search. ; By employing the multi-level knowledge distillation strategy designed in step S4, the output distribution, adapter residuals, intermediate layer features, and attention distribution of both teacher and student models are aligned simultaneously, thus fully conveying the cross-modal reasoning knowledge of the teacher model. This enables the student model to exhibit stronger robustness and generalization ability under noisy data and incomplete information conditions.

[0034] S5. Achieve dynamic multimodal fusion: integrate the text features output by the text encoder. Image features output by the visual encoder Through a learnable linear projection matrix and Mapped to A shared feature space is established. A learnable modality scaling factor is introduced. and The contribution weights of each modality are dynamically adjusted based on the input content. A gating fusion mechanism is employed, with the gating value calculated using the sigmoid function. Weighted fusion of the projected features:

[0035] Finally, the original spliced ​​features and the fused features are added together through residual connections to preserve the unique information of each modality; The dynamic multimodal fusion mechanism implemented in step S5 employs a learnable modality scaling factor and a gated fusion strategy to dynamically adjust the contribution weights of text and image modalities based on the input content. Combined with residual connections to preserve the original feature information, it achieves accurate cross-modal semantic fusion under limited computing resources, significantly improving diagnostic accuracy.

[0036] S6. Overall Model Training Process: The overall training process of this invention includes four stages: teacher model construction and LoRA domain adaptation, teacher model freezing, student model initialization, self-supervised pre-training, and multimodal distillation training. First, a teacher model composed of a text encoder and a visual encoder, along with a corresponding lightweight student model, are constructed. During the construction of both models, a low-rank LoRA adapter is injected into the Q, K, and V projection matrices in the self-attention modules of each Transformer layer. After adding LoRA, the teacher model first undergoes domain adaptation training, updating only the LoRA parameters to learn residual information specific to the avionics domain. After domain adaptation, the teacher model, including the backbone weights and LoRA weights, remains completely frozen in the subsequent distillation stage, outputting only high-order features of each layer, attention maps, LoRA residuals, and final classification logits, serving as a stable cross-modal knowledge provider. Subsequently, self-supervised pre-training is performed on the student model, with masked language modeling on the text side and masked image patch reconstruction on the image side to obtain a general representation capability for the avionics domain. At this stage, both the backbone parameters and student LoRA parameters of the student model are trainable. This completes the self-supervised pre-training. After training, the model enters the multimodal distillation stage. Guided by the frozen text features, visual features, intermediate layer hidden states, attention maps, and LoRA residual signals provided by the teacher model, the student model undergoes end-to-end training through multi-level distillation objectives such as logits distribution alignment, intermediate layer feature alignment, attention map alignment, and LoRA residual alignment. During distillation training, all trainable parameters of the student model, including the backbone and LoRA, are continuously updated, while the teacher model remains completely frozen. In the distillation stage, both the text output and visual output of the student model are mapped to a shared feature space and dynamically fused across modalities through a learnable modal scaling factor and gating mechanism. The fusion result is then input into the classification head to calculate the distillation loss. The student model trained through the above process can fully inherit the cross-modal semantic information of the teacher model and the domain residual knowledge learned by LoRA, maintaining high accuracy while possessing lightweight inference capabilities suitable for deployment on resource-constrained devices. S7. Optimize model deployment process: Convert the weights of the trained student model to half-precision floating-point format, significantly reducing memory usage. Use TensorRT for inference optimization, including operator fusion, automatic kernel tuning, and inter-layer tensor memory sharing. Based on testing and verification on a high-performance GPU server, this optimization scheme supports dynamic batch processing, with a maximum batch size of [missing information]. Test results show that the optimized model, while maintaining diagnostic accuracy, is expected to have its inference latency on embedded platforms controlled within a certain range. Within milliseconds, peak memory usage is expected to be MB can meet the real-time requirements of avionics field diagnostics; Through the model deployment process optimized in step S7, and by employing hybrid precision conversion and TensorRT inference optimization, tests based on high-performance GPU servers show that the expected inference latency on embedded platforms can be controlled within 75 milliseconds, with an estimated memory footprint of 302MB. This provides a feasible technical path for deployment on resource-constrained devices and can meet the real-time requirements of avionics field diagnostics.

[0037] S8. Implement noise-robust training: Systematically inject various types of noise during training to improve model robustness. Add Gaussian blur (kernel size...) to the visual side. , ) and random occlusion (occlusion area) Simulated image quality degradation; random text discarding. The nouns and verbs are vaguely described and incomplete. Using the AdamW optimizer, the initial learning rate... Linear preheating (preheating) is adopted Training steps and cosine annealing strategy, weight decay Gradient clipping threshold The early stopping mechanism during training is based on the validation set F1 score, with a patience value of [value missing]. One cycle.

[0038] Through the noise-robust training implemented in step S8, various noises such as image blurring, occlusion, and missing text are systematically introduced. Combined with optimized training strategies, the stability of the model in real noise environments is significantly improved, ensuring the reliability of diagnostic results under complex on-site conditions.

[0039] Experiments on an avionics fault diagnosis dataset demonstrate that the proposed method maintains 95.8% of the teacher model performance and achieves a diagnostic accuracy of 88.2% despite a 55.2% reduction in parameters. Simulation results on a high-performance GPU server show that the expected inference latency on an embedded platform can be controlled within 75 milliseconds, meeting real-time diagnostic requirements. The accuracy remains above 85.2% even under noisy conditions, validating the method's comprehensive advantages in model compression and robustness.

[0040] In avionics fault diagnosis benchmark tests, the method of this invention was comprehensively compared with four baseline methods: full parameter fine-tuning, low-rank adaptation only, knowledge distillation only, and low-rank adaptation combined with knowledge distillation. Experimental results show that the method of this invention, when using only... With M parameters, the following was achieved Diagnostic accuracy and The F1 score maintained the teacher model. The performance level of the teacher model is [missing information]. In comparison, the accuracy of the fully parameter-tuned teacher model is [missing information]. The number of parameters reached M; The accuracy of the low-rank fitting combined with knowledge distillation method is... The number of parameters is M; the accuracy rate using only the low-rank adaptation method is... The accuracy rate using only knowledge distillation is... These data fully demonstrate the superior balance between model compression and performance preservation achieved by the method of this invention.

[0041] In terms of robustness testing, the method of this invention achieves an accuracy of [percentage missing] under image noise conditions (including blurring and occlusion). Accuracy reached under text noise (keyword dropping) conditions The average accuracy rate is Compared to other methods, the low-rank fitting combined with knowledge distillation method achieves a higher accuracy rate under the same conditions. (Image noise) and (Text noise), average The accuracy rate using only knowledge distillation is... (Image noise) and (Text noise), average This result fully demonstrates the significant effect of the hierarchical knowledge distillation strategy in improving model robustness.

[0042] In terms of model building and training, the optimal configuration was determined through systematic parameter optimization: the low-rank adapter rank was tested for r= , , Three possible values, among which In terms of accuracy ( ) and model efficiency ( MB / Achieving the optimal balance between ms; learning rates were compared. , , Three gradients, final selection get Highest accuracy; batch size tested , , Three specifications, determined This is the optimal choice. In the validation of the distillation strategy, only the output distillation... Accuracy starts with gradually adding adapter residual alignment. ), intermediate layer feature alignment ( The final complete strategy reaches... The accuracy rate.

[0043] Regarding deployment verification, simulation tests based on high-performance GPU servers show that the model size at FP16 accuracy is... MB, Expected inference delay ms; The model is compressed to INT8 after quantization. MB, expected latency reduced to ms, accuracy loss controlled within Within [a certain range]. Test results demonstrate the feasibility of deploying this method on embedded platforms; ultimately, it achieved [a certain level of success] on the test set. Accuracy and F1 score, number of parameters M, compared to teacher model compression At the same time, it maintained The performance of the invention fully verifies its practical value.

[0044] like Figure 3 As shown, another embodiment of the present invention provides an avionics fault diagnosis system based on multimodal distillation, comprising: Data acquisition and processing module: used to acquire and preprocess avionics fault image data and text data; Diagnostic module: Used to input preprocessed avionics fault image data and text data into the trained avionics fault diagnosis model and output the diagnosis results; The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

[0045] A computer device is provided according to an embodiment of the present invention. This computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0046] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0047] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0048] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0049] The memory can be used to store the computer program and / or module, and the processor implements various functions of the computer device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0050] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0051] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0052] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for avionics fault diagnosis based on multi-modal distillation, characterized in that, include: Acquire and preprocess avionics fault image and text data; The preprocessed avionics fault image data and text data are input into the trained avionics fault diagnosis model, and the diagnosis results are output. The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

2. The method of claim 1, wherein, The specific training method for the avionics fault diagnosis model is as follows: A dual-encoder teacher model is constructed, which includes a text encoder and a visual encoder to extract the high-level text representation and high-level visual representation of the input avionics data, respectively, and output classification logits, selected intermediate layer features and attention maps. In the self-attention modules of each Transformer layer of the teacher model and the student model to be built, low-rank adaptation modules are injected into the query, key and value projection matrices to update the parameters through low-rank decomposition and keep the original pre-trained weights frozen. A lightweight student model is constructed, which includes a lightweight text encoder and a lightweight visual encoder, and self-supervised pre-training is performed on the student model. A multi-level knowledge distillation strategy is adopted to perform supervised fine-tuning of the pre-trained student model based on the classification logits, intermediate layer features, attention graph, and low-rank adaptation module residual information output by the teacher model. The text and visual features output by the student model are adaptively fused through a dynamic multimodal fusion mechanism to generate a fused representation. The avionics fault diagnosis result is then output through a classification head to complete the model training.

3. The method of claim 2, wherein, The parameter updates for the low-rank adaptation module are as follows: For each pre-trained weight matrix , a low-rank decomposition is performed for parameter update, where , The low-rank adapter only trains , the matrices, keeping the original pre-trained weights frozen, where is the rank.

4. The method of claim 2, wherein the method is characterized by, The self-supervised pre-training of the student model specifically involves: The student model text side uses mask language modeling with a mask rate of , the student model image side uses mask image block reconstruction with a mask rate of , pre-training uses an AdamW optimizer with a learning rate of , a batch size of , a training period of epochs, and linear warm-up for the first steps followed by cosine decay learning rate, weight decay of , gradient clipping maximum norm of , and saving checkpoints by epoch and early stopping when the validation set metric no longer improves.

5. The method of claim 2, wherein the method is based on multi-modal distillation. The multi-level knowledge distillation strategy specifically involves constructing a hierarchical distillation objective function containing five loss terms. The objective function is as follows: wherein, is a cross-entropy loss, aligning the softened output distribution of the teacher-student model using the KL divergence, computing the L2 distance of the teacher-student model low-rank adapter residual output, performing feature alignment at selected intermediate layers, matching the self-attention distribution, , , , and are each a weight coefficient.

6. The method of claim 2, wherein the method is based on multi-modal distillation. The dynamic multimodal fusion mechanism is as follows: The text features output by the text encoder and the image features output by the visual encoder are mapped to a shared feature space through a learnable linear projection matrix, a learnable modal scaling factor is introduced, the contribution weights of each modal are dynamically adjusted according to the input content, a gating fusion mechanism is adopted, and the gating value is calculated through a sigmoid function The weighted fusion is performed on the projected features, and finally the original splicing features and the fused features are added through a residual connection.

7. A multi-modal distillation based avionics fault diagnosis system, characterized in that, include: Data acquisition and processing module: used to acquire and preprocess avionics fault image data and text data; Diagnostic module: Used to input preprocessed avionics fault image data and text data into the trained avionics fault diagnosis model and output the diagnosis results; The trained avionics fault diagnosis model adopts a two-stage architecture of a dual-encoder teacher model and a lightweight student model. Both the teacher model and the student model are injected with low-rank adapters. A multi-level knowledge distillation strategy is adopted. Based on the data output by the teacher model and the residual information of the low-rank adapter module, the student model is adjusted in a supervised manner.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-7. The processor executes the computer program to implement the steps of the avionics fault diagnosis method based on multimodal distillation as described in any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the avionics fault diagnosis method based on multimodal distillation as described in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the avionics fault diagnosis method based on multimodal distillation as described in any one of claims 1-6.

Images