Intelligent part damage identification and quantitative analysis based on multi-modal fusion

Abstract

The invention discloses intelligent part damage identification and quantitative analysis based on multi-modal fusion, and particularly relates to the technical field of intelligent part damage identification. According to the method, synchronous or asynchronous real-time data acquisition is carried out on a target part, multi-modal features are extracted in combination with a heterogeneous feature extraction network, confidence scores of all modals are calculated, mutual information between the modals is fused, and an attention weighted fusion process is guided; when it is detected that the modality is abnormally suppressed, feature enhancement and dynamic weight adjustment are implemented, key weak signals are prevented from being ignored, fusion features are input into a damage identification model, and a damage identification result, modal weight visualization and early damage risk scoring are output; the technology effectively improves the recognition capability of the model for early and hidden damage, is especially suitable for sensitive capture and fusion judgment of weak modal signals in high-safety scenes such as wind power and aviation, and significantly enhances the early warning accuracy and maintenance foresight of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent identification of component damage, and in particular to intelligent identification and quantitative analysis of component damage based on multimodal fusion. Background Art

[0002] Intelligent identification and quantitative analysis of component damage based on multimodal fusion involves fusing and processing multiple different types of data (such as images, sounds, vibration signals, and temperature) and using artificial intelligence technologies (such as deep learning and machine learning) to automatically identify damage to mechanical or structural components and quantitatively analyze the extent, location, and type of damage. This approach improves detection accuracy and reliability and is widely used in equipment maintenance, structural health monitoring, and other fields.

[0003] The existing technology has the following shortcomings:

[0004] When using trained machine learning or deep learning models (such as CNN, RNN, Transformer, etc.) to classify and identify the fused features, when the information of a certain modality (such as image) is too prominent, the model may tend to rely on this modality and ignore the weak but critical feature information in other modalities (such as acoustic emission signals), resulting in a decrease in the ability to identify early and hidden damage. In addition, this problem may cause the model to be unable to identify potential faults in a timely manner in actual applications, missing the best maintenance window. For example, in the early development stage of microcracks in wind turbine blades, there is no obvious abnormality in the image, but the acoustic signal has already shown damage characteristics. If the model ignores this signal, it is easy to misjudge it as a normal state, which will bury operational risks. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent identification and quantitative analysis of component damage based on multimodal fusion to address the shortcomings of the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: intelligent identification and quantitative analysis of component damage based on multimodal fusion, including:

[0007] Perform synchronous or asynchronous real-time data collection on target components;

[0008] The collected multimodal data is processed using the corresponding feature extraction network;

[0009] For each modal feature, its confidence score is calculated based on the modal feature validity index;

[0010] A heterogeneous attention mechanism is used to perform weighted fusion of each modality feature, where the fusion weight is jointly determined by the modality confidence score and the mutual information between modalities;

[0011] Detect modal suppression during the fusion process, perform feature enhancement on abnormal modal suppression, and dynamically adjust its fusion weight;

[0012] The fused features are input into the damage identification model, and the output includes component damage identification results, modal weight visualization results, and early damage probability scores.

[0013] Preferably, the confidence score is calculated based on the modal feature validity index, specifically including: setting a weighting factor for each modal index, and calculating the weighted total score as the preliminary confidence score of the modality: Where: C i is the confidence score of mode i; w ij is the weight of the jth effectiveness indicator; f ij is the normalized value of the jth indicator, and n is the total number of indicators.

[0014] Preferably, the received embedding feature representations of each modality are: {X1,X2,...,X n}, and the confidence scores corresponding to each modality {C1,C2,...,C n}, for any two modal eigenvectors X i With X j , calculate its mutual information MI ij ; For each modality i, calculate its fusion weight W i , the formula is: in: is the normalized confidence score; α, β are adjustable fusion coefficients, which perform weighted aggregation on the eigenvectors of each modality: Output the fused multimodal unified feature vector X fusion .

[0015] Preferably, detecting the modal suppression during the fusion process specifically includes: analyzing the signal activity of the current modal input in time / space to obtain the modal content activity abnormality index, and the acquisition method is:

[0016] Input the image modality into the pre-trained classification model and perform standard forward reasoning; obtain the predicted category y c ; Select the last convolution layer from the model as the analysis layer, recorded as the feature map set Where k represents the channel number; for the target category y c The score pair feature map To find the gradient, the expression is: in: is the weight of channel k for category c; Z is the number of pixels in the feature map, that is, Z = H × W; Represents the gradient of the category score with respect to each pixel; generates a category activation map in a weighted linear combination manner: in: Represents the final category response heat map; ReLU represents the linear rectification function, normalizes the heat map, analyzes its maximum response area and high response mean, defines the high response ratio, that is, calculates the value of the number of pixels in the category response heat map that is greater than the response intensity threshold divided by the number of pixels in the feature map; and defines the high response mean, that is, in the Grad-CAM heat map, the average value of the response value of the area where the response value is higher than the response threshold; the obtained high response ratio and high response mean are weightedly averaged and calculated to obtain the modal content activity anomaly index.

[0017] Preferably, the modality history credibility deviation index is obtained after analyzing the deviation degree between the current modality and its historical performance, and the acquisition method is:

[0018] The modal confidence score at the current time t is recorded as C t , set the initial value μ0, and calculate the historical confidence trend value in the current time window The expression is: Where: λ is the smoothing factor, the value range is 0<λ≤1, is the historical confidence trend value at the previous moment. By calculating the difference between the current confidence and the historical confidence trend value, the modal historical credibility deviation index is calculated. The expression is: Where CDI is the modal history credibility deviation index.

[0019] Preferably, the modal content activity anomaly index and the modal history credibility deviation index are converted into a comprehensive feature vector, and the comprehensive feature vector is used as the input of the machine learning model. The machine learning model uses the prediction of the anomaly suppression risk score value label for each group of comprehensive feature vectors as the prediction target, and takes minimizing the sum of the prediction errors of all anomaly suppression risk score value labels as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the model training is stopped. The anomaly suppression risk score value is determined according to the model output result, wherein the machine learning model is a polynomial regression model.

[0020] Preferably, the obtained abnormal suppression risk score value is compared with a preset threshold. If the abnormal suppression risk score value is greater than or equal to the preset threshold, it indicates that there is a high risk probability of a modality being mistakenly suppressed in the current multimodal fusion process, and it is determined to be an abnormal modal suppression situation; if the abnormal suppression risk score value is less than the preset threshold, it indicates that there is a low risk probability of a modality being mistakenly suppressed in the current multimodal fusion process, and it is determined to be a normal modal suppression situation.

[0021] Preferably, when abnormal suppression is detected in a certain mode, the following processing steps are performed:

[0022] For mode M i The original eigenvector X i Perform feature enhancement to obtain the enhanced feature vector X′ i , the expression is: X′ i =X i +η·ΔX i ; Among them: X i is the original modal eigenvector, ΔX i is the feature enhancement amount, which is based on the local high response features extracted from the abnormal area inside the modality; η is the feature enhancement coefficient; the original fusion weight is W i After determining that the mode is abnormally suppressed, dynamic adjustment

[0023] The new fusion weight W′ i , the calculation formula is: W′ i =W i +λ·(CEI i +CDI i )·R t ; Among them, CEI i CDI is the abnormal activity index of the current modal content. i is the historical credibility deviation index of the current modal, R t is the modal anomaly suppression risk score, and λ is the weight adjustment coefficient.

[0024] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0025] 1. This invention introduces a modal confidence assessment mechanism and jointly models the mutual information between modalities to construct a multimodal fusion method with dynamic adjustment capabilities, which can effectively solve the problem of important feature omission caused by "strong modal suppressing weak modal" in the existing technology. By analyzing the modal content activity anomaly index and the modal history credibility offset index, a comprehensive feature vector is constructed and input into the polynomial regression model to achieve a score prediction of the abnormal modal suppression risk. Based on the score, feature enhancement and fusion weight adjustment are dynamically performed, thereby enhancing the system's response to weak signal modalities (such as acoustic emission and vibration signals) and improving the model's recognition accuracy for early and hidden damage.

[0026] 2. This invention has excellent adaptability and scalability, making it suitable for multimodal monitoring scenarios such as wind power, aviation, and rail transit. It can maintain high recognition accuracy even under conditions such as asynchronous data acquisition, uneven modal quality, or weak early damage signals. Through a risk-aware, fusion optimization strategy, it not only improves the system's ability to warn of potential faults but also reduces the likelihood of misjudgments and missed detections, facilitating intelligent and highly interpretable health management and predictive maintenance of key components. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0028] Figure 1 A mind map for intelligent identification and quantitative analysis of component damage based on multimodal fusion. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] For examples, see Figure 1 As shown, the intelligent identification and quantitative analysis of component damage based on multimodal fusion described in this embodiment includes:

[0031] Perform synchronous or asynchronous real-time data collection on target components;

[0032] The collected multimodal data is processed using the corresponding feature extraction network;

[0033] For each modal feature, its confidence score is calculated based on the modal feature validity index;

[0034] A heterogeneous attention mechanism is used to perform weighted fusion of each modality feature, where the fusion weight is jointly determined by the modality confidence score and the mutual information between modalities;

[0035] Detect modal suppression during the fusion process, perform feature enhancement on abnormal modal suppression, and dynamically adjust its fusion weight;

[0036] The fused features are input into the damage identification model, and the output includes component damage identification results, modal weight visualization results, and early damage probability scores.

[0037] This embodiment combines synchronous and asynchronous acquisition mechanisms to efficiently acquire multimodal data of wind turbine blades in operation for subsequent intelligent damage identification and quantitative analysis.

[0038] The monitoring target is the composite blade structure in the wind turbine, with key monitoring objects including the blade root, middle reinforcement ribs, blade tip and other areas prone to fatigue, cracks or peeling.

[0039] Visible light industrial cameras and infrared thermal imaging equipment are installed on the outside of the blade and the cabin monitoring platform to regularly capture images of the blade surface; the shooting frequency is 0.5Hz to 1Hz, and data collection is asynchronous (compared to other modes) because the visual changes are slow.

[0040] Several acoustic emission (AE) sensors are attached to key stress points inside the blade to collect acoustic signals from the material. Real-time data stream acquisition is performed with a sampling rate of 250kHz and a continuous synchronous acquisition method.

[0041] A three-axis acceleration sensor is installed at the root of the blade to detect abnormal dynamic responses caused by cracks or structural loosening; it is collected synchronously with the acoustic emission, with a sampling rate of 10kHz and a synchronous acquisition mechanism.

[0042] Thermistors for ambient temperature and humidity and blade surfaces are installed to monitor the thermal expansion and contraction effects of materials caused by temperature fluctuations. Sampling is done once per minute, using low-frequency asynchronous acquisition.

[0043] The main control unit is used for timestamp annotation mechanism to uniformly mark the absolute time label for each modal data; the synchronous modes (acoustic emission and vibration) adopt real-time parallel processing queue; the asynchronous modes (image, temperature) are delayed aligned through the modal buffer pool to ensure time correlation in subsequent fusion analysis.

[0044] The multimodal data collected in this embodiment includes the following modalities:

[0045] Visible light image data (RGB format); infrared thermal imaging data (grayscale image); acoustic emission signal (continuous waveform data); vibration signal (acceleration time series data).

[0046] According to the structural characteristics and physical semantics of different modal data, the following deep neural network structure is selected for feature extraction:

[0047] Image modality (visible light / infrared), using the pre-trained convolutional neural network ResNet50; input size: 224×224×3 (RGB image) or 224×224×1 (infrared image); processing steps: data normalization and channel standardization; using convolutional layers to extract image features such as edges, textures, and crack contours; obtaining fixed-length image feature vectors (e.g., 1024 dimensions) through global average pooling.

[0048] Acoustic emission (AE) signals are processed using a 1D-CNN + Transformer encoder architecture. The input data format is a 1D time series (with a fixed sampling length, such as 4096 points). The processing steps are as follows: 1D-CNN extracts local frequency patterns (such as sudden events and high-frequency envelopes); the Transformer module models long-range dependencies and explores the sequential patterns in microcrack propagation; and the final output is an embedded feature vector (e.g., 512 dimensions).

[0049] Vibration signal, using a multi-scale 1D-CNN network; input format: multi-channel acceleration data (e.g., 3 channels, sampling window length 2048); processing steps: separate channel input, separate convolution to extract time domain features; use multi-scale convolution kernels to extract signals in different frequency bands in parallel; after pooling, output a fused feature vector (e.g., 384 dimensions)

[0050] After all modal features are output, they are uniformly converted to a fixed dimension (through a fully connected layer or an embedding transformation module);

[0051] The alignment methods include: length unification: each modality ultimately outputs an embedding vector of the same length; timing alignment: window synchronization or dynamic time warping (DTW) through timestamps; and finally forming a multimodal feature tensor as the input of the subsequent fusion module.

[0052] The above-mentioned feature extraction network is deployed in an edge server or cloud GPU environment. The feature extraction processes of each modality can be processed in parallel to improve the overall response efficiency. It supports online model fine-tuning and incremental learning to adapt to dynamic adjustments in scenarios such as blade aging and environmental changes.

[0053] For each modal data (such as image, acoustic emission, vibration, etc.), a dedicated feature extraction network (such as CNN, 1D-CNN, Transformer) is used to extract the original high-dimensional feature vector.

[0054] For different modal characteristics, a set of indicators is defined to evaluate the correlation between the modal information quality and damage, such as image modal indicators (image clarity and abnormality): edge strength statistics (such as Laplacian variance); abnormal area ratio (such as based on hot spots or contours); image signal-to-noise ratio (SNR);

[0055] Acoustic emission / vibration modal indicators (signal activity and discernibility): signal energy density; ratio of instantaneous spectrum peak to mean; signal entropy or spectrum entropy; statistical characteristics such as kurtosis and skewness.

[0056] Standardize or normalize the various indicators of each mode, converting physical quantities of different scales into a unified scoring range (such as 0 to 1); use methods such as Z-score standardization, Min-Max normalization, or empirical threshold mapping.

[0057] Set a weighting factor for each modal indicator (which can be set based on historical training data or engineering experience) and calculate the weighted total score as the preliminary confidence score of the mode: Where: C i is the confidence score of mode i; w ij is the weight of the jth effectiveness indicator; f ij is the normalized value of the jth indicator, and n is the total number of indicators.

[0058] The confidence scores of each modality are combined into a confidence vector, which serves as the basis for modal weighting in the subsequent fusion layer. It can be used in conjunction with the modal attention mechanism in the subsequent fusion module to achieve dynamic modal weight control.

[0059] In the multimodal component damage identification process, weighted fusion of different modal features is performed to ensure that key damage features in weaker modes are not obscured. This method combines modal confidence scores with the calculation of mutual information between modalities to improve fusion accuracy.

[0060] Receive the embedded feature representation of each modality output from the previous feature extraction module: {X1,X2,...,X n}; Receive the confidence scores corresponding to each modality {C1, C2, ..., C n}, calculated by the modal validity evaluation module;

[0061] The confidence scores are normalized (such as Softmax) and used as a priori basis for the initial fusion weights.

[0062] For any two modal eigenvectors X i With X j , calculate its mutual information MI ij ;

[0063] You can use one of the following methods:

[0064] Mutual information approximation based on Kullback–Leibler divergence estimation; or the maximum information coefficient (MIC) is used as an alternative; mutual information is used to measure the information redundancy and complementarity between modalities.

[0065] For each modality i, calculate its fusion weight W i , the formula is: in: is the normalized confidence score; represents the sum of mutual information between mode i and other modes; α and β are adjustable fusion coefficients, which are used to control the contribution ratio of confidence and mutual information in fusion.

[0066] The fusion weights {W1,W2,...,W n The input is sent to the heterogeneous attention fusion module to perform weighted aggregation on the feature vectors of each modality: Before fusion, each modal feature is adaptively transformed through the attention sub-network to enhance feature saliency.

[0067] Output the fused multimodal unified feature vector X fusion , used for subsequent damage identification, classification and quantitative analysis tasks; while retaining the fusion weight vector {W i}As a basis for system visualization or modal contribution analysis.

[0068] Detect the modal suppression during the fusion process, analyze the temporal / spatial activity of the current modal input signal, and obtain the modal content activity anomaly index. The acquisition method is:

[0069] Input the image modality into a pre-trained classification model (such as ResNet50) and perform standard forward inference; obtain the predicted category y c (Usually "damage" class or other categorical output).

[0070] Select the last convolutional layer (or other high-level semantic layer) from the model as the analysis layer, and record it as the feature map set Where k represents the channel number.

[0071] For the target category y c The score pair feature map To find the gradient, the expression is: in: is the weight of channel k for category c; Z is the number of pixels in the feature map, that is, Z = H × W; represents the gradient of the class score with respect to each pixel.

[0072] Generate category activation maps using weighted linear combinations: in: Represents the final category response heat map; ReLU represents the linear rectification function, which suppresses negative responses.

[0073] The heatmap was normalized (0-1), and its maximum response area and high response mean were analyzed. The high response ratio was defined as the ratio of the number of pixels in the category response heatmap with a response intensity threshold greater than the response intensity threshold to the number of pixels in the feature map. The high response mean was also defined as the average response value of regions in the Grad-CAM heatmap with response values ​​above the response threshold. The response threshold represents the threshold for determining "high response" regions, with common values ​​such as 0.7 or set based on statistical distribution (such as the 95th percentile). This is used to eliminate background or non-target areas. The obtained high response ratio and high response mean were weighted and summed to obtain the modal content activity anomaly index, which ranges from 0 to 1. Higher values ​​indicate that the image modality currently contains more intense lesion activation areas.

[0074] The modal history credibility deviation index is obtained by analyzing the deviation between the current mode and its historical performance (the confidence mean in the sliding window). The method for obtaining it is:

[0075] The modal confidence score at the current time t is recorded as C t , which is output by the modal feature validity evaluation module. Set the initial value μ0, which can be the confidence score at the initial moment or the historical average value: Calculate the historical confidence trend value within the current time window The expression is: Where: λ is a smoothing factor, ranging from 0<λ≤1. A larger value makes the EMA more sensitive to recent changes (e.g., a common value is 0.1 to 0.3). is the historical confidence trend value at the previous moment. By calculating the difference between the current confidence and the historical confidence trend value, the modal historical credibility deviation index is calculated. The expression is: Where CDI is the modal historical credibility deviation index. The larger the CDI, the more serious the deviation of the current modal confidence from the historical trend.

[0076] The modal content activity anomaly index and the modal history credibility deviation index are converted into comprehensive feature vectors, which are used as inputs of the machine learning model. The machine learning model uses the prediction of the anomaly suppression risk score value label for each group of comprehensive feature vectors as the prediction target, and minimizes the sum of the prediction errors of all anomaly suppression risk score value labels as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the model training is stopped. The anomaly suppression risk score value is determined based on the model output results. The machine learning model is a polynomial regression model.

[0077] The obtained abnormal suppression risk score value is compared with the preset threshold. If the abnormal suppression risk score value is greater than or equal to the preset threshold, it means that there is a high risk probability of modal erroneous suppression in the current multimodal fusion process, and it is determined to be an abnormal modal suppression situation; if the abnormal suppression risk score value is less than the preset threshold, it means that there is a low risk probability of modal erroneous suppression in the current multimodal fusion process, and it is determined to be a normal modal suppression situation.

[0078] When abnormal suppression is detected in a certain mode, the following processing steps are performed:

[0079] For mode M i The original eigenvector X i Perform feature enhancement to obtain the enhanced feature vector X′ i , the expression is: X′ i =X i +η·ΔX i ; Among them: X i is the original modal eigenvector, ΔX i is the feature enhancement amount, which is based on the local high-response features extracted from the abnormal area inside the modal (such as high-frequency bands, local hot spots, strong edge textures, etc.); η is the feature enhancement coefficient, which controls the enhancement amplitude. The value range is generally 0.1 to 1, which is set according to the system sensitivity.

[0080] The original fusion weight is W i After determining that the mode is abnormally suppressed, it is dynamically adjusted to the new fusion weight W′ i , the calculation formula is: W′ i =W i +λ·(CEI i +CDI i )·R t ; Among them, CEI i CDI is the abnormal activity index of the current modal content. i is the current modal history credibility deviation index, R t is the modal anomaly suppression risk score, ranging from [0, 1]. λ is the weight adjustment coefficient, which controls the compensation strength, and the recommended value range is 0.1 to 0.5.

[0081] By enhancing the features of the local activation areas of abnormal modes, the system's response sensitivity to weak signals is improved; by combining CEI, CDI and risk scores, data-driven fusion regulation is achieved to avoid information shielding caused by fixed strategies; and feedback correction capabilities are introduced into the fusion mechanism to enhance the accuracy of identifying sudden or atypical injuries.

[0082] After feature enhancement and weight adjustment, the fused multimodal feature vector is input into the damage identification model. The model classifies and scores the input data, and the output results include the following three items:

[0083] Component damage identification results: The model outputs whether damage exists and the type of damage (such as cracks, corrosion, spalling, etc.); it can be further refined into multi-class classification results or multi-label recognition.

[0084] Modal weight visualization results: Outputs the final fusion weights of the current modes, reflecting the system's dependence on different modes in the decision-making process; it facilitates engineers to analyze the judgment basis and fusion bias of the model.

[0085] Early damage probability score: Outputs a numerical score (e.g., 0 to 1) indicating that the current input may contain early damage risk that has not yet developed significantly but has already shown signs. This score is derived based on the activation of weak modal features and the abnormal sensitivity mechanism within the model.

[0086] The output results are used comprehensively in the intelligent diagnosis decision support system, which not only determines the current damage status, but also assists in predicting potential risks and improving the foresight and reliability of structural health management.

[0087] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0088] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0089] It should be understood that the term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In addition, the character " / " herein generally indicates that the objects associated with each other are in an "or" relationship, but it may also indicate an "and / or" relationship, which can be understood by referring to the context. A person of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. Intelligent identification and quantitative analysis of component damage based on multimodal fusion, characterized by: include: Perform synchronous or asynchronous real-time data collection on target components; The collected multimodal data is processed using the corresponding feature extraction network; For each modal feature, its confidence score is calculated based on the modal feature validity index; A heterogeneous attention mechanism is used to perform weighted fusion of each modality feature, where the fusion weight is jointly determined by the modality confidence score and the mutual information between modalities; Detect modal suppression during the fusion process, perform feature enhancement on abnormal modal suppression, and dynamically adjust its fusion weight; The fused features are input into the damage identification model, and the output includes component damage identification results, modal weight visualization results, and early damage probability scores.

2. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 1 is characterized by: Calculate the confidence score based on the modal feature validity index, specifically including: setting a weighting factor for each modal index and calculating the weighted total score as the preliminary confidence score of the modality: Where: C i is the confidence score of mode i; w ij is the weight of the jth effectiveness indicator; f ij is the normalized value of the jth indicator, and n is the total number of indicators.

3. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 2 is characterized by: Receive the embedded feature representation of each modality: {X1, X2, ..., X n }, and the confidence scores corresponding to each modality {C1, C2, ..., C n }, for any two modal eigenvectors X i With X j , calculate its mutual information MI ij ; For each modality i, calculate its fusion weight W i , the formula is: in: is the normalized confidence score; α, β are adjustable fusion coefficients, which perform weighted aggregation on the eigenvectors of each modality: Output the fused multimodal unified feature vector X fusion .

4. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 1 is characterized by: Detecting modal suppression during the fusion process involves analyzing the temporal and spatial activity of the modal input signal to obtain the modal content activity anomaly index. The acquisition method is: Input the image modality into the pre-trained classification model and perform standard forward reasoning; obtain the predicted category y c ; Select the last convolution layer from the model as the analysis layer, recorded as the feature map set Where k represents the channel number; for the target category y c The score pair feature map To find the gradient, the expression is: in: is the weight of channel k for category c; Z is the number of pixels in the feature map, that is, Z = H × W; Represents the gradient of the category score with respect to each pixel; generates a category activation map in a weighted linear combination manner: in: Represents the final category response heat map; ReLU represents the linear rectification function, normalizes the heat map, analyzes its maximum response area and high response mean, defines the high response ratio, that is, calculates the value of the number of pixels in the category response heat map that is greater than the response intensity threshold divided by the number of pixels in the feature map; and defines the high response mean, that is, in the Grad-CAM heat map, the average value of the response value of the area where the response value is higher than the response threshold; the obtained high response ratio and high response mean are weightedly averaged and calculated to obtain the modal content activity anomaly index.

5. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 4 is characterized by: After analyzing the deviation between the current mode and its historical performance, the modal historical credibility deviation index is obtained. The acquisition method is: The modal confidence score at the current time t is recorded as C t , set the initial value μ0, and calculate the historical confidence trend value in the current time window The expression is: Where: λ is the smoothing factor, the value range is 0<λ≤1, is the historical confidence trend value at the previous moment. By calculating the difference between the current confidence and the historical confidence trend value, the modal historical credibility deviation index is calculated. The expression is: Where CDI is the modal history credibility deviation index.

6. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 5 is characterized by: The modal content activity anomaly index and the modal history credibility deviation index are converted into comprehensive feature vectors, which are used as inputs of the machine learning model. The machine learning model uses the prediction of the anomaly suppression risk score value label for each group of comprehensive feature vectors as the prediction target, and minimizes the sum of the prediction errors of all anomaly suppression risk score value labels as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the model training is stopped. The anomaly suppression risk score value is determined based on the model output results. The machine learning model is a polynomial regression model.

7. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 6 is characterized by: Compare the obtained abnormal suppression risk score with the preset threshold. If the abnormal suppression risk score is greater than or equal to the preset threshold, it indicates that there is a high probability of modal suppression being mistakenly performed in the current multimodal fusion process, and it is determined to be an abnormal modal suppression situation. If the abnormal suppression risk score value is less than the preset threshold, it means that the risk probability of modal erroneous suppression in the current multimodal fusion process is low, and it is judged to be a normal modal suppression situation.

8. The intelligent identification and quantitative analysis of component damage based on multimodal fusion according to claim 7 is characterized by: When abnormal suppression is detected in a certain mode, the following processing steps are performed: For mode M i The original eigenvector X i Perform feature enhancement to obtain the enhanced feature vector X′ i , the expression is: X′ i =X i +η·ΔX i ; Where: X i is the original modal eigenvector, ΔX i is the feature enhancement amount, which is based on the local high response features extracted from the abnormal area inside the modality; η is the feature enhancement coefficient; the original fusion weight is W i , after determining that the mode is abnormally suppressed, dynamically adjust to the new fusion weight W′ i , the calculation formula is: W′ i =W i +λ·(CEI i +CDI i )·R t ; Among them, CEI i CDI is the abnormal activity index of the current modal content. i is the historical credibility deviation index of the current modal, R t is the modal anomaly suppression risk score, and λ is the weight adjustment coefficient.

Images