Fault diagnosis method and system based on multi-modal feature fusion and lightweight fine tuning

Abstract

The invention relates to the technical field of industrial intelligent diagnosis and large language model crossing, and provides a fault diagnosis method and system based on multi-modal feature fusion and lightweight fine tuning. According to the method, multi-source heterogeneous data such as a time sequence, an image and a text of industrial equipment are collected, after feature extraction and cross-modal fusion are conducted, fusion features are adapted to a large language model input space through a projection embedding layer, and then intelligent diagnosis is achieved in combination with a large language model subjected to low-rank self-adaptive fine adjustment. According to the method, the problems of semantic gaps and dimension mismatch of multi-modal industrial data are effectively solved, the diagnosis precision and the model generalization ability are remarkably improved, meanwhile, the requirement for computing resources is greatly reduced, the interpretability of diagnosis results is enhanced through structured output, and an innovative technical path is provided for intelligent operation and maintenance of industrial equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial intelligent diagnosis and large language model cross, more particularly, to a fault diagnosis method and system based on multi-modal feature fusion and light fine-tuning. BACKGROUND

[0002] Industrial equipment fault diagnosis is a key technology to ensure production safety and efficiency. Traditional methods mainly rely on manual experience and professional knowledge, which have low diagnosis efficiency and poor accuracy. With the development of artificial intelligence technology, deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN) have been widely used in fault diagnosis, such as analyzing vibration signals, acoustic vibration data or multi-modal information to improve diagnosis accuracy. In recent years, large language models (such as GPT, BERT) have shown excellent performance in natural language processing, and their introduction into industrial fault diagnosis can achieve semantic understanding of heterogeneous data such as text description and maintenance records, which is expected to further improve the intelligent level of diagnosis. However, general large language models lack professional knowledge in the industrial field, and direct application is difficult to adapt to the specific scene of equipment failure. In addition, existing fine-tuning methods usually require full parameter updating, resulting in large computation and long training time, and are prone to overfitting problems. Furthermore, industrial equipment involves multiple fields such as machinery, electrical, hydraulic, etc., and a single fine-tuning strategy cannot fully utilize cross-domain expert knowledge, limiting the model's generalization ability and practical value.

[0003] Although existing fault diagnosis methods have made progress in specific fields, they still have obvious limitations. For example, a certain patent document discloses a power equipment fault diagnosis method based on time series-spectrum-discrete data multi-modal fusion, which fuses acoustic vibration, local discharge spectrum and other data through cross-modal attention mechanism, alleviating the problem of insufficient single-modal data. However, it relies on traditional deep learning models (such as VGG-Net, ResNet) and does not involve the application of large language models, which cannot handle text-based fault descriptions, and the model structure is fixed, making it difficult to adapt to multi-field requirements. Another patent document discloses a fault diagnosis method based on adaptive multi-modal feature fusion with attention mechanism, which uses attention mechanism and residual layers to improve feature representation for mechanical equipment vibration signals, solving the problem of diagnosis accuracy under variable working conditions. However, this method is limited to time series signal processing and lacks compatibility for multi-source heterogeneous data, and the deep learning model's hyperparameter adjustment is complex and susceptible to sample imbalance. Overall, existing technologies focus on single modal or specific fields, with low computational efficiency in the fine-tuning process, and cannot dynamically integrate multi-expert knowledge, resulting in insufficient diagnosis robustness in complex industrial scenarios.

[0004] In view of the above deficiencies, it is necessary to develop a new scheme to solve the core problems of large computation, overfitting and insufficient utilization of multi-domain knowledge in industrial equipment fault diagnosis. SUMMARY

[0005] The present application aims at the technical problems existing in the prior art, and provides a fault diagnosis method and system based on multi-modal feature fusion and light fine-tuning, which overcomes the defects of existing fine-tuning methods through multi-modal information fusion and evaluation, and provides an efficient and scalable solution for intelligent operation and maintenance of industrial equipment.

[0006] According to a first aspect of the present application, a fault diagnosis method based on multi-modal feature fusion and light fine-tuning is provided, comprising: S1, acquiring real-time multi-modal data of a diagnosed device, the multi-modal data at least including time series data, image data and text data; S2, extracting features of the multi-modal data and performing cross-modal feature fusion to generate a fusion feature vector; based on a pre-trained projection embedding layer, projecting the fusion feature vector into a projection embedding vector matching the input dimension of a large language model; S3, combining the projection embedding vector with a text instruction to form a multi-modal prompt, inputting the multi-modal prompt into a large language model fine-tuned by light, and outputting a structured fault diagnosis result; wherein the large language model is obtained by injecting trainable parameters by a low-rank adaptive method and fine-tuning, and the large language model only activates the low-rank adaptive parameters during diagnosis.

[0007] On the basis of the above technical solution, the present application can also be improved as follows.

[0008] Optionally, in step S1, the preprocessing includes: Wavelet transform denoising is used for time series data; Normalization processing is used for image data, and pixel temperature values are mapped to the [0, 1] interval to highlight high temperature abnormal areas; The text data is extracted to include at least key information of fault type and fault severity, and is subjected to semantic standardization processing.

[0009] Optionally, in step S2, the extraction of the features of the multi-modal data and the cross-modal feature fusion to generate the fusion feature vector include: S201, using a pre-trained feature extraction network to extract a time series feature vector, an image feature vector and a text feature vector from the time series data, the image data and the text data, respectively; Taking the text feature vector as a semantic reference, generating semantically aligned time series features and semantically aligned image features; S202, respectively unify the semantic-aligned time sequence features and the semantic-aligned image features to the same dimension as the text feature vector through linear mapping, respectively perform residual connection between the mapped time sequence features and the image features and the text feature vector, and generate enhanced time sequence final features and image final features; S203, after performing feature enhancement processing on the time sequence final features, the image final features and the text feature vector, generate the fusion feature vector through weighted fusion and splicing operation; wherein the feature enhancement processing at least includes adaptive dropout noise suppression and layer normalization.

[0010] Optionally, in step S201, the text feature vector is taken as a semantic reference to generate the semantic-aligned time sequence features and the semantic-aligned image features, including: Taking the text feature vector as a semantic reference, calculating the attention weights between the time sequence feature vector and the text feature vector and the attention weights between the image feature vector and the text feature vector through a cross-modal attention mechanism; Based on the attention weights between the time sequence feature vector and the text feature vector, performing weighted fusion on the time sequence feature vector to generate the semantic-aligned time sequence features; Based on the attention weights between the image feature vector and the text feature vector, performing weighted fusion on the image feature vector to generate the semantic-aligned image features.

[0011] Optionally, in step S2, the projection embedding layer is pre-trained, and the fusion feature vector is projected into a projection embedding vector matching the input dimension of the large language model, including: S204, the projection embedding layer performs linear transformation on the fusion feature vector based on a projection weight matrix and a bias vector, to map the fusion feature vector from a first preset dimension to a second preset dimension of the projection embedding vector, wherein the second preset dimension is consistent with the input embedding dimension of the target large language model.

[0012] Optionally, the step of obtaining the large language model by injecting trainable parameters and fine-tuning through a low-rank adaptive method includes: Inject a low-rank matrix pair (A, B) into the Transformer self-attention layer of the large language model, the rank of the low-rank matrix pair is set to a preset low value; the low-rank matrix A is initialized with a normal distribution, the low-rank matrix B is initialized with zero, and a scaling factor is configured to match the output amplitude of the low-rank matrix pair (A, B) and the original weight; A fine-tuning framework is constructed for the cooperation of main tasks and auxiliary tasks, wherein the main tasks are fault type classification and diagnosis result generation, and the auxiliary tasks include industrial term matching and modal feature consistency verification; a gating network is used to dynamically calculate the weights of the main and auxiliary tasks, and a weighted joint loss function is generated to optimize the low-rank matrix parameters; During the fine-tuning process, the gradient signal is back-propagated through the weighted joint loss function, and only the low-rank matrix parameters are updated, while the base model weights are frozen; After the performance is stable, the optimized low-rank matrix parameters are fused with the base model parameters to generate a final diagnosis model.

[0013] Optionally, in step S3, the projection embedding vector and the text instruction are combined to form a multi-modal prompt, and the multi-modal prompt is input into a large language model that is fine-tuned to output a structured fault diagnosis result, including: S301, the projection embedding vector and the text instruction are combined to form a multi-modal prompt, and the text instruction is a natural language description containing diagnosis task requirements; S302, the multi-modal prompt is input into a large language model that is fine-tuned, and the large language model generates a structured diagnosis result based on the multi-modal prompt, including fault type, fault severity, associated multi-modal diagnosis basis, and maintenance suggestions.

[0014] According to the second aspect of the present application, a fault diagnosis system based on multi-modal feature fusion and light fine-tuning is provided, including: A data acquisition module is configured to acquire real-time multi-modal data of a diagnosed device and perform preprocessing, and the multi-modal data at least includes time series data, image data and text data; A feature fusion module is connected to the data acquisition and preprocessing module, configured to extract features of the multi-modal data and perform cross-modal feature fusion to generate a fusion feature vector; the feature fusion module further includes a pre-trained projection embedding layer, configured to project the fusion feature vector into a projection embedding vector matching the input dimension of a large language model; A light diagnosis module is connected to the multi-modal feature fusion module, configured to combine the projection embedding vector and a text instruction to form a multi-modal prompt, and the multi-modal prompt is input into a large language model that is fine-tuned, wherein the large language model is obtained by injecting trainable parameters through a low-rank adaptive method and fine-tuning, and only the low-rank adaptive parameters are activated during diagnosis; the light diagnosis module outputs a structured fault diagnosis result.

[0015] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor configured to implement the steps of the above-mentioned fault diagnosis method based on multi-modal feature fusion and light-weight fine-tuning when executing a computer management program stored in the memory.

[0016] According to a fourth aspect of the present application, a computer readable storage medium is provided, having a computer management program stored thereon, the computer management program being configured to implement the steps of the above-mentioned fault diagnosis method based on multi-modal feature fusion and light-weight fine-tuning when executed by a processor.

[0017] The present application provides a fault diagnosis method, system, electronic device and storage medium based on multi-modal feature fusion and light-weight fine-tuning, which constructs a set of collaborative technical solutions of multi-modal data fusion→ feature space projection→ light-weight diagnosis reasoning. The scheme first collects multi-modal data of industrial equipment, such as time series sensor data, infrared image data and operation and maintenance text records, and performs preprocessing operations such as wavelet denoising, temperature normalization and semantic standardization, to provide standardized input for subsequent analysis. Then, a cross-modal attention mechanism based on text features as semantic benchmarks is used to realize semantic alignment and weighted fusion of time series, image and text features, and generate a unified fusion feature vector. The fusion features are accurately mapped to the input embedding space of a large language model through linear transformation of the projection embedding layer, solving the problems of dimension mismatch and semantic gap. Finally, the projected features and text instructions are combined into multi-modal prompts, which are input into a large language model fine-tuned by low-rank adaptive (LoRA), and efficient reasoning is realized by updating only a small number of parameters, and the structured results containing fault types, diagnosis basis and maintenance suggestions are directly output.

[0018] The present application effectively solves the semantic gap problem of heterogeneous data through cross-modal feature alignment and projection embedding technology, enabling the diagnosis model to fully exploit the complementary features in multi-source information, thereby significantly improving fault recognition accuracy and reliability. The low-rank adaptive fine-tuning strategy significantly reduces the consumption of computing resources while maintaining the integrity of the pre-trained model knowledge, making the system feasible for efficient deployment. Its dynamic routing and multi-task learning mechanism endow the model with strong cross-domain adaptability, enabling it to flexibly cope with complex working condition changes in different industrial scenarios. The finally generated structured diagnosis results have both accuracy and explainability, providing a complete technical closed loop for intelligent operation and maintenance of industrial equipment. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of the fault diagnosis method based on multi-modal feature fusion and light-weight fine-tuning provided by the present application is provided. Figure 2 A flowchart of the fault diagnosis method for wind power equipment fault diagnosis scenarios is provided. Figure 3 A fault diagnosis system block diagram based on multi-modal feature fusion and light fine-tuning is provided for the present application. Figure 4 A possible hardware structure schematic diagram of an electronic device is provided for the present application. Figure 5 A possible hardware structure schematic diagram of a computer readable storage medium is provided for the present application. DETAILED DESCRIPTION

[0020] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0021] Figure 1 A fault diagnosis method flowchart based on multi-modal feature fusion and light fine-tuning is provided for the present application, Figure 2 The method of the present application is exemplified in the context of wind power equipment fault diagnosis.

[0022] In conjunction with Figure 1 and Figure 2 shown, the fault diagnosis method based on multi-modal feature fusion and light fine-tuning provided by the embodiments of the present application includes steps S1-S3: S1, acquiring real-time multi-modal data of the diagnosed equipment and preprocessing, the multi-modal data at least including time series data, image data and text data.

[0023] This step forms a multi-source heterogeneous data set by collecting real-time time series data (such as vibration sensor signals), image data (such as infrared thermal images) and text data (such as maintenance logs) of the diagnosed industrial equipment. In the preprocessing stage, targeted processing methods are used, for example, using time series signal processing methods (such as wavelet transform) to filter environmental noise and retain key frequency band features reflecting the state of the equipment; normalizing the image data to enhance the contrast of the abnormal area; performing entity recognition and semantic standardization on the text information to extract key fields such as fault type and severity. The preprocessed multi-modal data is used for subsequent time series feature extraction, image feature extraction and text feature extraction. This step ensures the integrity, cleanliness and alignability of the multi-modal data, providing standardized input for subsequent feature fusion.

[0024] S2, extracting features of the multi-modal data and performing cross-modal feature fusion to generate a fusion feature vector; based on a pre-trained projection embedding layer, projecting the fusion feature vector into a projection embedding vector matching the input dimension of a large language model.

[0025] The step realizes intelligent mapping of multi-modal data to a unified semantic space. For example, first, the deep features of each modality are extracted by a pre-trained feature extraction network (such as CNN, ResNet, BERT), then the text features are used as a semantic reference, and the cross-modal attention mechanism is used to calculate the correlation weight of the time sequence and image features with the text to generate a semantically aligned feature representation. Then, dimension unification and feature enhancement are realized through linear mapping and residual connection, and finally a fused feature vector is generated through weighted fusion. The projection embedding layer precisely projects the fused feature vector to the input dimension space of the large language model through linear transformation, solving the problem of dimension mismatch between heterogeneous data and model input.

[0026] S3, combining the projection embedding vector with the text instruction to form a multi-modal prompt, the multi-modal prompt being input to the large language model that is lightened and fine-tuned to output a structured fault diagnosis result; wherein the large language model is obtained by injecting trainable parameters and fine-tuning through a low-rank adaptive method.

[0027] This step forms a multi-modal prompt rich in semantic information by combining the projection embedding vector with the text instruction (such as "please analyze the equipment fault type and provide diagnostic evidence"). The prompt is input to the large language model that is specially optimized. The large language model uses the low-rank adaptive (LoRA) fine-tuning technology to inject trainable low-rank matrix parameters (less than 5% of the total model parameters) only in the model attention layer, thereby maintaining the pre-trained knowledge while realizing efficient adaptation. The model performs inference based on the multi-modal prompt and outputs a structured result containing fault type, diagnostic evidence and maintenance suggestion, significantly improving the explainability and practicality of the diagnosis result.

[0028] It can be understood that based on the defects in the background art, the embodiments of the present application propose a fault diagnosis method based on multi-modal feature fusion and light fine-tuning. The method first collects multi-source heterogeneous data such as time series, images and text of industrial equipment, and performs standardization preprocessing to eliminate noise and unify the data scale, providing clean and aligned input basis for subsequent analysis, improving the expression ability and discrimination of fault features, and enabling the diagnosis model to capture richer fault patterns from complementary multi-source information; then, using cross-modal feature fusion technology, taking text features as the semantic reference, the semantic alignment and weighted fusion of multi-modal features are realized through the attention mechanism to generate a unified fusion feature vector; then, the projection embedding layer is used to accurately map the fusion features to the input space of the large language model, solving the problem of dimension mismatch and semantic inconsistency; then, the projected features and text instructions are combined into multi-modal prompts, which are input into the large language model fine-tuned by low-rank self-adaptive. The model based on the light fine-tuning strategy of low-rank self-adaptation greatly reduces the model calculation complexity while ensuring the diagnosis accuracy, so that the system can be efficiently deployed in resource-constrained industrial environments; the model only updates a small number of parameters to adapt to the industrial scene, realizes accurate classification of fault types and structured result generation, and thus completes the end-to-end intelligent inference from data to diagnosis. Through the combination of structured result output and multi-modal prompt mechanism, the explainability and practicality of the diagnosis process are enhanced, providing intuitive and reliable decision support for operation and maintenance personnel.

[0029] In a possible embodiment, in step S1, the time series data is vibration signal time series data collected by a vibration sensor, the image data is image data obtained by an infrared thermal imager, and the text data is equipment maintenance log exported by a wind power operation and maintenance system. The three types of data are preprocessed, including: Wavelet transform is used for denoising the time series data to remove interference signals such as wind speed and environmental vibration; Normalization processing is used for image data to map pixel temperature values to the [0, 1] interval, highlighting high-temperature abnormal areas; The entity recognition technology is used to extract key information including at least fault type and fault severity from the text data, and perform semantic standardization processing.

[0030] The obtained "denoised time series data", "normalized image data" and "labeled text data" are used as three input data sources for multi-modal feature fusion in step S2, which are used for subsequent time series feature extraction, image feature extraction and text feature extraction.

[0031] It can be understood that the data preprocessing step adopts wavelet transform denoising for time series data, effectively filtering out high-frequency interference and random noise in the device operating environment, and retaining key signal components reflecting the health status of the device; temperature value normalization processing is performed on image data, the contrast of abnormal temperature zones is enhanced through linear mapping, and the fault features in the infrared thermal image are more clearly visualized; for text data, key information extraction and semantic standardization are performed to eliminate description ambiguity and construct structured knowledge representation. The cooperative effect of the preprocessing operation not only solves the problems of dimensional difference, noise interference and semantic inconsistency existing in multi-source heterogeneous data, but also improves the efficiency and accuracy of subsequent feature fusion, thereby enhancing the robustness and reliability of the fault diagnosis system in complex industrial scenarios as a whole.

[0032] In a possible implementation manner, in step S2, the feature of the multi-modal data is extracted and cross-modal feature fusion is performed to generate a fusion feature vector, including sub-steps S201-S203.

[0033] S201, using a pre-trained feature extraction network, respectively extracting a time series feature vector, an image feature vector and a text feature vector from the time series data, the image data and the text data.

[0034] In combination Figure 2 As shown in the figure, for example, time series feature extraction: using a time series feature extraction network (such as CNN, Transformer, etc.), performing convolution and pooling operations on the denoised vibration signal, extracting core features such as time domain peak value and frequency domain power spectrum, and outputting a 256-dimensional time series feature vector ; Image feature extraction: using an image feature extraction network (such as ResNet-18, Vision Transformer, etc.), performing feature encoding on the infrared thermal image, extracting core features such as temperature gradient and high-temperature area, and outputting a 512-dimensional image feature vector ; Text feature extraction: using a pre-trained language model (such as an industrial fine-tuned BERT-base, Roberta-base) or a text encoder, performing sentence-level encoding on the maintenance log text, and outputting a 768-dimensional text feature vector .

[0035] Then, taking the text feature vector as a semantic reference, the semantic-aligned time series feature and the semantic-aligned image feature are generated. Specifically, it includes: Taking the text feature vector as a semantic reference, calculating the attention weight between the time series feature vector and the text feature vector, and the attention weight between the image feature vector and the text feature vector through a cross-modal attention mechanism; The time sequence feature vector is weighted and fused based on the attention weight between the time sequence feature vector and the text feature vector to generate a semantic alignment time sequence feature. The image feature vector is weighted and fused based on the attention weight between the image feature vector and the text feature vector to generate a semantic alignment image feature.

[0036] In combination Figure 2 As shown, for example, to reduce the semantic difference between time sequence, image and text, the semantic association of the above three kinds of modal feature vectors is realized through attention weight calculation, wherein: The time sequence-text attention weight is used to measure the matching degree of the time sequence feature and the text semantics. The image-text attention weight is used to measure the matching degree of the image feature and the text semantics.

[0037] In this embodiment, the higher the weight is, the stronger the association of the corresponding modal feature and the text fault description is, and the specific calculation method is as follows: (1) Time sequence-text attention weight:

[0038] (2) Image-text attention weight:

[0039] Wherein, is the time sequence-text attention weight (value range [0, 1]), is the image-text attention weight (value range [0, 1]), is the time sequence feature vector, is the image feature vector, is the text feature vector, = 256 (time sequence feature dimension), = 512 (image feature dimension), and is a scaling factor, which can avoid the problem of gradient disappearance caused by too high feature dimension, ensure the stability of the attention weight, and the Softmax function normalizes the feature similarity matrix to a weight distribution in the interval [0, 1], which is convenient for subsequent weighted fusion.

[0040] Then, based on the attention weight, the time sequence, image features and text sub-features (linearly mapped to match the dimension) are weighted and fused to generate a semantic alignment feature: (1) Time sequence alignment feature:

[0041] (2) Image alignment feature:

[0042] wherein, is the semantic aligned time sequence feature vector, is the semantic aligned image feature vector, is the time sequence-text attention weight (value range [0, 1]), is the image-text attention weight (value range [0, 1]), is is the 256-dimensional sub-feature after linear mapping (time sequence adaptation sub-feature of text feature, matching dimension), is is the 512-dimensional sub-feature after linear mapping (image adaptation sub-feature of text feature, matching dimension), through the weighted fusion of “modal feature + text sub-feature”, the time sequence and image features carry the text semantic information, and the cross-modal semantic gap is solved.

[0043] It can be understood that, in order to realize semantic unification of multi-modal data, the core fault features of each modal are extracted respectively, and then the semantic association of time sequence, image and text is established through the attention mechanism by taking the text feature as the semantic reference.

[0044] S202, the semantic aligned time sequence feature and the semantic aligned image feature are respectively unified to the same dimension as the text feature vector through linear mapping, and the mapped time sequence feature and image feature are respectively connected with the text feature vector through residual connection to generate enhanced time sequence final feature and image final feature.

[0045] In this step, in order to realize the dimension unification of multi-modal features, for example, the semantic aligned time sequence feature and the semantic aligned image feature are unified to 768 dimensions, which is consistent with the dimension of the text feature, and at the same time, the feature information loss in the mapping process is avoided, and dynamic dimension mapping and residual connection are designed. Now, in combination with Figure 2 , the process of dynamic dimension mapping and residual connection is exemplarily illustrated.

[0046] 1. Dynamic dimension mapping.

[0047] As shown in Figure 2 , the time sequence feature dimension is 256, the image feature dimension is 512, and the text feature dimension is 768. The semantic aligned time sequence feature and image feature are converted to 768 through linear mapping. When the weight matrix of linear mapping is initialized, the covariance matrix of time sequence-text and image-text features is referred to, so as to ensure that the semantic distribution of the mapped feature is consistent with the text feature. The specific formula is as follows: (1) Time sequence feature mapping:

[0048] (2) Image feature mapping:

[0049] in, This is the mapped temporal feature vector (768 dimensions). This is the mapped image feature vector (768 dimensions). The time-series mapping weight matrix is ​​768×256. The image mapping weight matrix is ​​768×512. This is the corresponding bias vector (768 dimensions).

[0050] 2. Perform residual connection.

[0051] To avoid feature distortion caused by linear mapping, such as Figure 2 As shown, the mapped temporal features and image features are added to the text features respectively, preserving the original text semantic information. The specific formula is as follows: (1) Final characteristics of the time series:

[0052] (2) Final image features:

[0053] (Final temporal feature after residual connection, dimension 768) (Final image features after residual connection, dimension 768) and original text features (Original text features, dimension 768) These three elements serve as input to the feature enhancement module, ensuring that the multimodal features undergo noise suppression and distribution unification before fusion.

[0054] S203, after performing feature enhancement processing on the final temporal features, final image features, and text feature vectors, the fused feature vector is generated through weighted fusion and concatenation operations; wherein, the feature enhancement processing includes at least adaptive dropout noise suppression and layer normalization.

[0055] In this step, to improve the robustness and distribution consistency of multimodal features, a three-step enhancement operation is designed: adaptive dropout (regularization), layer normalization, and multimodal fusion.

[0056] 1. Adaptive dropout noise suppression Industrial data often contains environmental interference, requiring dynamic adjustment of the dropout rate based on the noise level of each modality: higher noise levels necessitate a higher dropout rate to filter redundant noise features, while lower noise levels result in a lower dropout rate to retain valid features. First, define the noise metrics for each modality, such as time-series noise. = 1 / signal-to-noise ratio, image noise = pixel gray level variance, text noise = number of out-of-vocabulary words / total number of words, and the dropout rate is calculated by the following formula:

[0057] wherein, , respectively corresponding to the time series, image, and text features; wherein 0.1 is the basic dropout rate, used to ensure that there is a regularization effect even at low noise, avoiding overfitting; wherein 0.8 is the noise adjustment coefficient, used to linearly increase the dropout rate with the noise intensity, the stronger the noise, the higher the dropout rate; The maximum dropout rate is limited to 0.5 to avoid excessive dropout resulting in loss of effective features.

[0058] The dropout processing is performed on the multi-modal features, and the denoised time series features are:

[0059] wherein, is the denoised time series feature vector, is the dropout operation (randomly setting part of the elements in the feature vector to 0 with a probability P), is the time series final feature after residual connection, is the dynamic dropout rate of the time series features.

[0060] The denoised image features are:

[0061] wherein, is the denoised image feature vector, is the image final feature after residual connection, is the dynamic dropout rate of the image features.

[0062] The denoised text features are:

[0063] wherein, is the denoised text feature vector, is the original text feature, is the dynamic dropout rate of the text features.

[0064] 2. Layer normalization To unify the distribution of each modal feature and avoid affecting the subsequent fusion effect due to differences in feature value ranges, the following formula is used: Figure 2As shown, the features after dynamic mapping are subjected to layer normalization, and the specific formula is as follows: (1) Normalized time series characteristics:

[0065] in, This is the normalized time series feature vector. For denoising time series features, Each batch The mean and variance, To prevent tiny values ​​with a denominator of 0; Trainable parameters (scaling / translation parameters) specific to time-series features are used to adaptively adjust the normalized feature distribution.

[0066] (2) Normalized image features:

[0067] in, For denoising image features, These are trainable parameters (scaling / translation parameters) specific to image features, and their logic is consistent with temporal normalization.

[0068] (3) Normalized text features:

[0069] in, For denoising text features, Trainable parameters (scaling / translation parameters) specific to text features ensure that the distribution scale of text features is consistent with that of temporal and image features.

[0070] 3. Multimodal fusion To preserve the core information of each modality, the normalized temporal, image, and text features are first weighted and fused using adaptive weights to generate comprehensive features. Then, it is concatenated with the normalized features of each modality to generate fused features. The adaptive weights satisfy the following: The specific formula is as follows: (1) Weighted fusion:

[0071] (2) Final fusion:

[0072] in, These are the normalized results for time-series, image, and text features, respectively. This is the weighted and fused composite feature vector. This is the final multimodal fusion feature vector; injecting a Transformer encoder, outputting 768-dimensional unified encoding features .

[0073] It can be understood that the embodiment builds a hierarchical multi-modal feature processing mechanism through the three-order architecture of "feature extraction-semantic alignment-enhanced fusion", realizing the deep integration of heterogeneous data. The embodiment first extracts independent feature vectors of each modality through a pre-training network, then takes the text feature as a semantic reference, calculates the correlation weight of the time sequence and image features with the text using a cross-modal attention mechanism, realizes the accurate alignment of multi-source features in a unified semantic space, and then completes the dimension unification and feature enhancement through linear mapping and residual connection technology, and finally generates a fusion feature vector with rich semantic information using an adaptive weighted fusion strategy.

[0074] The embodiment effectively solves the semantic gap problem between heterogeneous data through the cross-modal attention mechanism with the text feature as the semantic reference, realizes the accurate alignment of multi-source features in a unified semantic space. The feature enhancement strategy based on residual connection and adaptive weighting not only retains the fine-grained information of the original features, but also enhances the robustness of the model to industrial scene noise. The hierarchical processing process enables the system to adapt to the distribution characteristics of different modal data, providing a high-discrimination feature representation for subsequent fault diagnosis tasks. The embodiment provides an innovative solution for intelligent processing of multi-modal industrial data through an end-to-end feature optimization mechanism.

[0075] In a possible implementation manner, in step S2, the projection embedding layer based on the pre-trained projection embedding layer is used to project the fusion feature vector into a projection embedding vector matching the input dimension of the large language model, including: S204, the projection embedding layer performs linear transformation on the fusion feature vector based on a projection weight matrix and a bias vector, to map the fusion feature vector from a first preset dimension to a projection embedding vector of a second preset dimension, wherein the second preset dimension is consistent with the input embedding dimension of the target large language model.

[0076] It can be understood that this step builds a projection embedding layer (PEL) and completes industrialized collaborative initialization. The projection embedding layer (PEL) is used to solve the incompatibility problem between the unified encoding features output after step S203 fusion and the input embedding space of the large language input model (LLM), which requires that the unified encoding features after the fusion of multi-modal data The dimension conversion is converted into an input dimension acceptable by a large language input model (LLM), such as the input embedding dimension of ChatGLM3 being 4096, and it is also necessary to ensure that the converted features (PE) are semantically aligned with the input space of the large language input model (LLM) to avoid reasoning bias caused by semantic differences.

[0077] In combination Figure 2 As shown in the figure, the specific implementation process of step S204 in the embodiment is as follows: 1. PEL structure design The linear mapping module is used to realize dimension conversion and semantic alignment. The input of linear mapping is 768-dimensional unified coding features , and the output is a 4096-dimensional projection embedding vector PE. The specific formula is as follows:

[0078] , wherein, is the projection embedding vector (4096 dimensions); is a 4096x768 projection weight matrix, which is responsible for dimension conversion and semantic mapping; is a 4096-dimensional bias vector, which is used to adjust the feature offset after projection.

[0079] The advantage of this formula is that it can be adapted through simple linear operations, the parameter size is small, and it will not increase the reasoning burden, and at the same time, semantic alignment can be ensured through subsequent initialization optimization.

[0080] 2. Collaborative initialization In order to make PE have both fault feature representation ability and LLM input compatibility, PE and multi-modal feature fusion layer need to be collaboratively initialized, for example, 20,000 pieces of lightweight industrial data (including fault labels) are input into the two modules, and the parameters are optimized through a joint loss function. The specific loss design is as follows: (1) Cross-entropy loss (optimize fault feature representation) used to ensure that PE can accurately distinguish different fault types. The input is PE, and the output is the fault type prediction probability. The loss formula is as follows:

[0081] , wherein, is the cross-entropy loss, whose value range is [0, +∞], the smaller the loss is, the stronger the fault representation ability of PE is; is the number of fault types, is the one-hot encoding of the real fault label (one-hot encoding), such as bearing wear corresponding to [1, 0, …, 0], only the position corresponding to the fault type is 1, and the rest is 0; The cross-entropy loss is used to output the failure prediction probability of the simple classification head (2-layer fully connected network). The smaller the cross-entropy loss is, the more accurate the representation of the failure feature is. (2) Cosine similarity loss (optimize semantic alignment) The cosine similarity loss is used to ensure that the semantic distribution of the PE is consistent with the LLM input embedding space, and the input is the text embedding vector of the PE and the LLM, for example, generated by the “failure description text”. The loss formula is as follows:

[0082] wherein, is the vector dot product of the projected embedding vector and the text embedding vector of the LLM, which is used to measure the consistency of the direction of the two vectors. is the L2 norm of and , and more specifically, is the 4096-dimensional embedding vector generated by the LLM for the “wind turbine failure description text”; is the cosine similarity between and , which has a value range of [-1, 1], and the higher the similarity is, the more consistent the semantics is. By using “1-similarity”, the cosine similarity loss is converted into “the higher the similarity is, the smaller the loss is”, so as to force the projected embedding vector

[0083] to align with the LLM input space.

[0084] wherein, is the joint initialization loss, wherein 0.8 is the cross-entropy loss weight, which is used to prioritize the accuracy of the failure representation of ; and wherein 0.2 is the cosine similarity loss weight, which is used to consider the semantic alignment.

[0085] In one possible embodiment, in order to enable the general LLM to have the wind turbine failure diagnosis capability and avoid high computing power consumption of full-scale fine-tuning, a light fine-tuning scheme based on LoRA is designed, that is, only a low-rank matrix is injected into the Transformer self-attention layer of the LLM, and a small number of parameters are updated to realize task adaptation; at the same time, the main task and the auxiliary task are designed to be trained cooperatively, so that the model learns the failure diagnosis while accumulating industrial knowledge.

[0086] In combination with​​​​Figure 2 As shown, the step of injecting trainable parameters and fine-tuning the large language model by the low-rank adaptive method includes: 1. LoRA (Low-Rank Adaptation) module design.

[0087] The core of LoRA is to approximate the full parameter update by a low-rank matrix, reducing the size of the training parameters. A low-rank matrix pair (A, B) is injected in the Transformer self-attention layer (Query, Key matrix) of the large language model LLM, including: a low-rank matrix A (dimension ) and a low-rank matrix B (dimension ), the rank of the low-rank matrix pair (A, B) is set to a preset low value; wherein is the input embedding dimension of LLM; is the low-rank dimension, taking into account adaptability and parameter size.

[0088] The output of the self-attention layer is composed of "original weight output + low-rank matrix output", and the specific formula is as follows:

[0089] Where W is the original weight (4096x4096) of the self-attention layer Query / Key matrix, and X is the input feature of the self-attention layer; is the low-rank matrix product (dimension 4096x4096), used to approximate the update amount of W; the advantage of this design is that only A and B need to be trained (the parameter size is less than 5% of the original weight), which can achieve an adaptation effect similar to full fine-tuning, greatly reducing the computing power requirement.

[0090] To avoid the low-rank matrix output being too large at the beginning of fine-tuning and damaging the pre-training knowledge of LLM, a specific initialization strategy is adopted for the low-rank matrix A and the low-rank matrix B: The low-rank matrix A is initialized with a normal distribution to ensure that the initial update amount is small; the low-rank matrix B is initialized to zero, so that the LoRA module output is 0 at the beginning of fine-tuning, and the model relies on pre-training knowledge for inference; at the same time, a scaling factor ( such as ) is set to ensure that the output amplitude of the low-rank matrix product matches the output amplitude of the original weight W, maintaining the stability of the model inference.

[0091] 2. Fine-tuning task design (based on multi-modal prompt input).

[0092] To let the LLM clearly diagnose the task target and obtain multi-modal fault features, a multi-modal prompt input of "PE + text instruction" is designed in this step - PE carries fine-grained fault features, and the text instruction clearly diagnoses the requirements (such as output fault type, confidence, and maintenance suggestion), and the two cooperate to ensure the accuracy of the model reasoning direction.

[0093] Therefore, a fine-tuning framework is constructed for the cooperation of main tasks and auxiliary tasks, wherein the main task is fault type classification and diagnosis result generation, and the auxiliary task includes industrial term matching and modal feature consistency verification; a weighted joint loss function is generated by dynamically calculating the main and auxiliary task weights through a gating network to optimize the low-rank matrix parameters. Combined with Figure 2 As shown in the following example: (1) Main task (fault type classification and diagnosis result generation) The goal of the main task is to let the model learn the mapping of "multi-modal features → fault diagnosis result", and the instance input is The multi-modal prompt is, for example: = [PE] + "Please judge the fault type based on XXX (the specific content of "XXX" can be selected), and give maintenance suggestions based on multi-modal basis and fault severity"; The output is a JSON format diagnosis result. To optimize this task, cross-entropy loss is used to calculate the prediction error, and the formula is as follows:

[0094] Wherein, C = number of fault types, is the one-hot encoding (one-hot encoding) of the true label, is the fault type prediction probability output by the LLM.

[0095] (2) Auxiliary task 1 (industrial term - modal feature matching) The auxiliary task 1 is to let the model learn the association between industrial terms and multi-modal features (such as "vibration peak frequency" corresponding to the specific features of the vibration signal), and the instance input is The multi-modal prompt is: = [PE] + "Please complete the masked term: The [mask] of the wind turbine vibration signal is 1000 Hz, and the corresponding fault type is?" The output is the masked term and the fault type. To optimize this task, cross-entropy loss is used to calculate the term prediction error, and the formula is as follows:

[0096] Wherein, K = size of industrial term dictionary, is the one-hot encoding (one-hot encoding) of the true label of the masked term, The term prediction probability output by the LLM.

[0097] (3) Auxiliary task 2 (modal feature consistency verification) Auxiliary task 2 is to let the model learn the logical consistency between multi-modal features (such as “vibration peak 1000 Hz + bearing temperature 38°C” should correspond to the same fault), and the instance input is a multi-modal prompt: = [PE] + “Does the vibration peak, infrared temperature and maintenance log of this wind turbine correspond to the same fault? (Yes / No)” The output is the consistency judgment result; to optimize this task, the cross-entropy loss is used to calculate the judgment error, and the formula is as follows:

[0098] where M = 2, including “yes” and “no”, is the one-hot encoding (one-hot encoding) of the true label of the consistency judgment, is the judgment probability output by the LLM; To balance the priority of the main task and auxiliary tasks 1 and 2, the losses of the three are fused according to the weights to form a multi-task joint loss:

[0099] where 0.7 is the main task weight to prioritize fault diagnosis accuracy; 0.15 is the auxiliary task weight to balance the needs of industrial knowledge learning and consistency verification; only the A and B matrix parameters of LoRA are trained in the fine-tuning process, and the training parameter ratio is <5%, which adapts to the computing power demand of the industrial scene.

[0100] 3. During the fine-tuning process, the gradient signal is backpropagated through the weighted joint loss function, and only the low-rank matrix parameters are updated, while the base model weights are frozen.

[0101] Specifically, only the core parameters (such as attention weights, linear mapping matrices) of the multi-modal feature fusion layer and the of the PEL are updated during the backpropagation of the training process; when the validation set loss does not decrease for 3 consecutive rounds, the majority of the parameters of the fusion layer and the PEL parameters are frozen, and only the feature enhancement sublayer (dropout, layer normalization) of the fusion layer is retained for subsequent fine-tuning, reducing the parameter size during the fine-tuning phase.

[0102] 4. After the performance is stable, the optimized low-rank matrix parameters and the base model parameters are fused to generate the final diagnosis model.

[0103] The output final diagnosis model is used to provide an industrial equipment fault diagnosis service. For example, receiving multi-modal industrial data, outputting a JSON format diagnosis result containing a fault type and an associated basis. In combination with the fault type, a hierarchical maintenance suggestion is generated to provide an auxiliary decision basis for formulating a maintenance plan for workers.

[0104] In one possible implementation manner, based on the final diagnosis model obtained through lightweight fine-tuning, in step S3, the projection embedding vector is combined with a text instruction to form a multi-modal prompt, and the multi-modal prompt is input into the lightweight fine-tuned large language model to output a structured fault diagnosis result, including sub-steps S301-S302: S301, the projection embedding vector is combined with a text instruction to form a multi-modal prompt, and the text instruction is a natural language description containing diagnosis task requirements.

[0105] The specific implementation process of this step can refer to the description of forming a multi-modal prompt in the main task, auxiliary task 1 and auxiliary task 2 in the foregoing embodiments, which will not be described here.

[0106] S302, the multi-modal prompt is input into the lightweight fine-tuned large language model, and the large language model generates a structured diagnosis result containing a fault type, a fault severity, an associated multi-modal diagnosis basis and a maintenance suggestion based on the multi-modal prompt.

[0107] In this step, the multi-modal prompt is formed by combining the projection embedding vector (representing the fused time sequence, image and text features) with the text instruction (such as "please analyze the equipment fault features and output the diagnosis type, basis and maintenance suggestion"). After inputting into the low-rank self-adaptive fine-tuned large language model, the model first analyzes the multi-modal information: for example, decoding the abnormal frequency band features of the vibration signal, the temperature distribution pattern in the infrared image and the historical maintenance clues in the text record from the projection embedding vector; then associating these modalities through attention mechanism, such as associating the vibration anomaly with the "poor lubrication" described in the text, verifying the consistency of the fault; finally generating a structured diagnosis result, including the main fault type (such as bearing wear), the severity (such as moderate), the multi-modal basis (such as matching the wear pattern in the vibration spectrum, locating the fault point by the temperature hotspot) and the hierarchical maintenance suggestion (such as emergency lubrication inspection, regular replacement of parts). The fault diagnosis process improves the diagnosis reliability through multi-modal evidence cross-validation, and enhances the explainability and decision support capability of the result through structured output.

[0108] Figure 3 A fault diagnosis system structure diagram based on multi-modal feature fusion and lightweight fine-tuning is provided for the embodiments of the present application, as shown in Figure 3As shown, a fault diagnosis system based on multi-modal feature fusion and light-weight fine-tuning includes a data acquisition module, a feature fusion module, and a light-weight diagnosis module, wherein: The data acquisition module is used to acquire real-time multi-modal data of the diagnosed equipment and pre-process, and the multi-modal data at least includes time series data, image data, and text data. The feature fusion module is connected with the data acquisition and pre-processing module, used to extract features of the multi-modal data and perform cross-modal feature fusion to generate a fusion feature vector; the feature fusion module further includes a pre-trained projection embedding layer, used to project the fusion feature vector into a projection embedding vector matching the input dimension of a large language model; The light-weight diagnosis module is connected with the multi-modal feature fusion module, used to combine the projection embedding vector with a text instruction to form a multi-modal prompt, and the multi-modal prompt is input into a large language model fine-tuned by light-weight; wherein the large language model is obtained by injecting trainable parameters by a low-rank adaptive method and fine-tuning, and only the low-rank adaptive parameters are activated during diagnosis; the light-weight diagnosis module outputs a structured fault diagnosis result.

[0109] It can be understood that the fault diagnosis system based on multi-modal feature fusion and light-weight fine-tuning provided by the present application corresponds to the fault diagnosis method based on multi-modal feature fusion and light-weight fine-tuning provided by the foregoing embodiments, and the related technical features of the fault diagnosis system based on multi-modal feature fusion and light-weight fine-tuning can refer to the related technical features of the fault diagnosis method based on multi-modal feature fusion and light-weight fine-tuning, which will not be repeated here.

[0110] Please refer to Figure 4 , Figure 4 The embodiment of the electronic device provided by the present application is shown in the embodiment schematic diagram of the electronic device. As shown in the figure, Figure 4 The embodiment of the present application provides an electronic device 400, which includes a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420, and the processor 420 implements the following steps when executing the computer program 411: S1, acquiring real-time multi-modal data of the diagnosed equipment and pre-processing, and the multi-modal data at least includes time series data, image data, and text data; S2, extracting features of the multi-modal data and performing cross-modal feature fusion to generate a fusion feature vector; based on a pre-trained projection embedding layer, projecting the fusion feature vector into a projection embedding vector matching the input dimension of a large language model; S3, combine the projection embedding vector with the text instruction to form a multi-modal prompt, and input the multi-modal prompt into the large language model after light-weight fine-tuning to output a structured fault diagnosis result; wherein the large language model is obtained by injecting trainable parameters and fine-tuning through a low-rank adaptive method.

[0111] Please refer to Figure 5 , Figure 5 An embodiment of a computer readable storage medium provided by the present application is shown in the figure. As shown in Figure 5 , the embodiment provides a computer readable storage medium 500, which stores a computer program 411, and the computer program 411 is executed by a processor to implement the following steps: S1, obtaining real-time multi-modal data of the diagnosed equipment and preprocessing, the multi-modal data at least including time series data, image data and text data; S2, extracting features of the multi-modal data and performing cross-modal feature fusion to generate a fusion feature vector; based on a pre-trained projection embedding layer, projecting the fusion feature vector into a projection embedding vector matching the input dimension of a large language model; S3, combining the projection embedding vector with the text instruction to form a multi-modal prompt, and inputting the multi-modal prompt into the large language model after light-weight fine-tuning to output a structured fault diagnosis result; wherein the large language model is obtained by injecting trainable parameters and fine-tuning through a low-rank adaptive method.

[0112] The embodiment of the present application provides a fault diagnosis method, system and storage medium based on multi-modal feature fusion and light-weight fine-tuning, based on a three-level collaborative architecture of "multi-modal feature fusion-projection adaptation-lightweight diagnosis", through constructing an end-to-end intelligent diagnosis pipeline to realize accurate identification and decision support of industrial equipment fault. The scheme first maps heterogeneous data such as time series, images and texts into a unified semantic space through multi-source data acquisition and cross-modal feature fusion technology, solving the semantic gap problem of multi-modal data; then, the projection embedding layer is used to realize the accurate adaptation of the feature vector and the input space of the large language model, ensuring the efficient transmission of multi-modal information; finally, the low-rank adaptive fine-tuning technology is used to optimize the parameters of the large language model, so that it can quickly adapt to the industrial diagnosis scene while keeping the pre-training knowledge, and form a structured and interpretable diagnosis decision.

[0113] The application realizes comprehensive improvement of industrial fault diagnosis efficiency through the synergistic innovation of multi-modal feature fusion and light fine-tuning. The scheme effectively solves the three core problems of poor compatibility of multi-source heterogeneous data, high demand for computing resources, and weak interpretability of diagnosis results in traditional methods. In terms of diagnosis accuracy, through cross-modal feature alignment and deep fusion mechanism, the discrimination ability and recognition reliability of fault features are significantly improved; in terms of computing efficiency, the parameter-efficient light fine-tuning strategy is adopted, which greatly reduces the resource consumption of model training and deployment; in terms of system practicability, the structured output and multi-modal basis traceability function enhance the interpretability of the diagnosis process, providing reliable decision support for industrial sites. This technical path provides an innovative solution for intelligent operation and maintenance of industrial equipment.

[0114] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0115] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0116] The application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0117] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0118] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operations steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide the function of realizing the processes specified in the flowcharts Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0119] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be construed to include all such modifications and variations as fall within the scope of the application.

[0120] Obviously, various modifications and changes are possible in the present application without departing from the spirit and scope of the application. It is to be understood that the application includes any such modifications and changes only insofar as they come within the scope of the appended claims and their equivalents.

Claims

1. A fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning, characterized in that, include: S1, acquire and preprocess real-time multimodal data of the device under diagnosis, wherein the multimodal data includes at least time-series data, image data and text data; S2, extract the features of the multimodal data and perform cross-modal feature fusion to generate a fused feature vector; based on the pre-trained projection embedding layer, project the fused feature vector into a projection embedding vector that matches the input dimension of the large language model; S3, the projection embedding vector is combined with the text instruction to form a multimodal prompt, and the multimodal prompt is input into a large language model that has been lightly fine-tuned to output a structured fault diagnosis result; wherein, the large language model is obtained by injecting trainable parameters and fine-tuning them through a low-rank adaptive method.

2. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 1, characterized in that, In step S1, the preprocessing includes: Wavelet transform is used to denoise the time-series data; The image data is normalized and the pixel temperature values ​​are mapped to the [0,1] interval to highlight the high temperature abnormal area; Extract key information from the text data, including at least the fault type and the severity of the fault, and perform semantic standardization processing.

3. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 1, characterized in that, In step S2, extracting features from the multimodal data and performing cross-modal feature fusion to generate a fused feature vector includes: S201, using a pre-trained feature extraction network, extract time-series feature vectors, image feature vectors, and text feature vectors from the time-series data, image data, and text data, respectively; Using text feature vectors as semantic benchmarks, semantically aligned temporal features and semantically aligned image features are generated; S202, the semantically aligned temporal features and semantically aligned image features are unified to the same dimension as the text feature vector through linear mapping, and the mapped temporal features and image features are residually connected to the text feature vector to generate the enhanced final temporal features and final image features; S203, after performing feature enhancement processing on the final temporal features, final image features, and text feature vectors, the fused feature vector is generated through weighted fusion and concatenation operations; wherein, the feature enhancement processing includes at least adaptive dropout noise suppression and layer normalization.

4. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 3, characterized in that, In step S201, generating semantically aligned temporal features and semantically aligned image features based on text feature vectors includes: Using text feature vectors as semantic benchmarks, attention weights between temporal feature vectors and text feature vectors, as well as between image feature vectors and text feature vectors, are calculated through a cross-modal attention mechanism. Based on the attention weights between the temporal feature vector and the text feature vector, the temporal feature vector is weighted and fused to generate semantically aligned temporal features; Based on the attention weights between the image feature vector and the text feature vector, the image feature vector is weighted and fused to generate semantically aligned image features.

5. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 1, characterized in that, In step S2, projecting the fused feature vector into a projection embedding vector that matches the input dimension of the large language model based on the pre-trained projection embedding layer includes: S204, the projection embedding layer performs a linear transformation on the fused feature vector based on the projection weight matrix and the bias vector, so as to map the fused feature vector from the first preset dimension to the projection embedding vector of the second preset dimension, wherein the second preset dimension is consistent with the input embedding dimension of the target large language model.

6. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 1, characterized in that, The steps for obtaining the large language model by injecting trainable parameters and fine-tuning them using a low-rank adaptive method include: A pair of low-rank matrices (A, B) is injected into the Transformer self-attention layer of the large language model, and the rank of the pair of low-rank matrices is set to a preset low value. The low-rank matrix A is initialized with a normal distribution, the low-rank matrix B is initialized with zero, and a scaling factor is configured to match the output amplitude of the pair of low-rank matrices (A, B) with the original weights. A fine-tuning framework for the collaboration between the main task and auxiliary tasks is constructed. The main task is fault type classification and diagnosis result generation, and the auxiliary tasks include industrial terminology matching and modal feature consistency verification. The weights of the main and auxiliary tasks are dynamically calculated through a gating network to generate a weighted joint loss function to optimize the low-rank matrix parameters. During fine-tuning, the gradient signal is back-propagated through the weighted joint loss function, updating only the low-rank matrix parameters and freezing the weights of the base model. After verifying that the performance is stable, the optimized low-rank matrix parameters will be fused with the basic model parameters to generate the final diagnostic model.

7. The fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning according to claim 1, characterized in that, In step S3, combining the projection embedding vector with the text instruction to form a multimodal prompt, and inputting the multimodal prompt into a lightweight, fine-tuned large language model to output a structured fault diagnosis result, includes: S301, the projection embedding vector is combined with the text instruction to form a multimodal prompt, wherein the text instruction is a natural language description containing the requirements of the diagnostic task; S302, the multimodal prompts are input into a large language model that has been lightly optimized and finely tuned. The large language model performs reasoning based on the multimodal prompts to generate a structured diagnostic result that includes fault type, fault severity, associated multimodal diagnostic criteria, and maintenance suggestions.

8. A fault diagnosis system based on multimodal feature fusion and lightweight fine-tuning, characterized in that, include: The data acquisition module is used to acquire and preprocess real-time multimodal data of the device under diagnosis. The multimodal data includes at least time-series data, image data, and text data. The feature fusion module, connected to the data acquisition and preprocessing module, is used to extract features from the multimodal data and perform cross-modal feature fusion to generate a fused feature vector. The feature fusion module also includes a pre-trained projection embedding layer, which projects the fused feature vector into a projection embedding vector that matches the input dimension of the large language model. A lightweight diagnostic module, connected to the multimodal feature fusion module, is used to combine the projection embedding vector with text instructions to form a multimodal prompt. The multimodal prompt is input to a large language model that has been fine-tuned by lightweighting. The large language model is obtained by injecting trainable parameters and fine-tuning them through a low-rank adaptive method. Only the low-rank adaptive parameters are activated during diagnosis. The lightweight diagnostic module outputs structured fault diagnosis results.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the processor is used to execute computer management programs stored in the memory to implement the steps of the fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer management program, which, when executed by a processor, implements the steps of the fault diagnosis method based on multimodal feature fusion and lightweight fine-tuning as described in any one of claims 1-7.

Images