Aircraft defect intelligent evaluation system and method based on multi-modal fusion

Abstract

The invention relates to the technical field of aircraft intelligent detection and maintenance systems, and discloses an aircraft defect intelligent evaluation system and method based on multi-modal fusion, and the system comprises a multi-modal data collection module, a tensor construction and decomposition module, a meta-prototype relation network module, a multi-target game optimization module, and a closed-loop feedback module. The method comprises the steps of constructing a five-order feature tensor through multi-modal data synchronous acquisition and space-time alignment, extracting low-rank features through hypergraph block item decomposition, dynamically generating a defect prototype set in combination with meta-learning, generating a maintenance decision by adopting a Nash equilibrium strategy and fusing multi-constraint conditions, and optimizing system parameters through closed-loop feedback. The whole-process intelligentization of aircraft defect detection and maintenance is realized; according to the method, high-precision defect detection is realized through multi-modal data fusion and hypergraph modeling, an intelligent decision is generated in combination with dynamic prototype learning and multi-target game optimization, and continuous self-optimization is performed by means of a closed-loop feedback mechanism, so that the operation and maintenance efficiency and safety of the aircraft are improved automatically in the whole process.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft intelligent detection and maintenance systems, and in particular to an aircraft defect intelligent assessment system and method based on multimodal fusion. Background Art

[0002] Aircraft defect detection is a core component of flight safety. Traditional methods rely primarily on manual visual inspection or automated analysis based on threshold segmentation using single-modal sensor data. However, aircraft defects are complex and diverse, involving various types such as surface cracks, internal debonding, and corrosion. Single-modal data struggles to fully capture cross-scale and cross-dimensional defect characteristics. In existing technologies, multimodal data often face difficulties in information fusion due to inconsistent temporal and spatial benchmarks. The feature extraction process is susceptible to noise interference, and the physical correlation between modalities is not effectively modeled, resulting in the risk of misjudgment and missed detection in detection results.

[0003] During the defect classification and decision-making phase, most systems rely on static pre-trained models or fixed rule bases, lacking the ability to adapt to new defect patterns. This significantly degrades performance, especially when data distribution changes dynamically. Furthermore, maintenance decisions are often based on single-objective optimization, ignoring the collaborative game relationships among multiple constraints. This results in actual implementation results deviating from theoretical expectations.

[0004] Existing technical systems generally lack a closed-loop feedback mechanism and are unable to dynamically optimize model parameters and decision-making logic based on actual maintenance results. During long-term operation, errors are easily accumulated due to factors such as sensor drift and environmental disturbances, making it difficult to meet the high-reliability operation and maintenance requirements of aircraft.

[0005] Therefore, the present invention proposes an aircraft defect intelligent assessment system and method based on multimodal fusion to address the deficiencies of the existing technology. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides an intelligent aircraft defect assessment system and method based on multimodal fusion, which solves the problems of aircraft defect detection such as difficulty in multimodal data fusion, poor adaptability to dynamic environments, insufficient multi-constraint decision-making optimization capabilities, and long-term deployment performance degradation.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent aircraft defect assessment system based on multimodal fusion, comprising:

[0008] a multimodal data acquisition module configured to synchronously acquire visible light images, infrared thermal images, ultrasonic signals, and X-ray projection data, and perform spatiotemporal alignment preprocessing to obtain multimodal data;

[0009] A tensor construction and decomposition module is used to construct the multimodal data into a fifth-order feature tensor in a unified spatiotemporal coordinate system, and extract low-rank feature components through a block decomposition algorithm guided by hypergraph modeling;

[0010] The meta-prototype relationship network module dynamically constructs an incrementally updateable set of defect category prototypes based on the decomposed feature components, and calculates the relationship matching degree between the inspected area and each prototype;

[0011] The multi-objective game optimization module receives the relationship matching calculation results, combines the preset maintenance cost parameters and real-time environmental conditions, and generates the optimal maintenance decision through the Nash equilibrium strategy;

[0012] The closed-loop feedback module feeds back the actual execution effect of the maintenance decision to the meta-prototype relationship network module and the tensor decomposition module to complete the dynamic adjustment of the system parameters.

[0013] Preferably, the multimodal data acquisition module includes:

[0014] Visible light camera, used to obtain high-definition image data of the aircraft surface;

[0015] Infrared thermal imager, used to capture temperature anomaly distribution data;

[0016] Ultrasonic array probe, used to collect internal defect reflection waveform data;

[0017] X-ray detectors for generating projection data of the internal structure of materials;

[0018] Among them, each sensor realizes data acquisition timing alignment through a time synchronization protocol to ensure that the timestamp deviation of each modal data is less than the set threshold.

[0019] Preferably, the tensor construction and decomposition module performs:

[0020] Feature extraction is performed on multimodal data respectively, where:

[0021] Visible light data V t Apply the pre-trained ResNet-34 network to generate the feature vector ψ(V t );

[0022] Infrared Data I t Through the temperature gradient operator Extract the feature vector φ(I t );

[0023] Ultrasonic Data U t Use short-time Fourier transform to extract the first d f =128-dimensional frequency domain components

[0024] X-ray data t Statistical histogram feature χ(X t );

[0025] The four modal eigenvectors are stacked along the time dimension T to construct a fifth-order eigentensor Its slice satisfies in, is the vector outer product operation;

[0026] Extracting low-rank feature components based on block decomposition guided by hypergraph modeling:

[0027] Construct the hypergraph adjacency matrix W, and the hyperedge weight is calculated as:

[0028]

[0029] in, is the characteristic distribution entropy of the mth mode at the kth time slice; is the characteristic distribution of the mth mode in the kth time slice;

[0030] Solve an optimization problem with a hypergraph regularization term:

[0031]

[0032] in, is the core tensor; is the factor matrix of the mth mode; D is the hypergraph degree matrix; is the tensor modal product, λ is the coefficient of the hypergraph regularization term, and R is the decomposition rank.

[0033] Preferably, the algorithm for block item decomposition satisfies:

[0034] Decompose the rank constraint into [rank(A (1) ),rank(A (2) ),rank(A (3) ),ran(A (4) )]=[8,8,6,3];

[0035] Among them, A (m) is the factor matrix of the mth mode; rank(·) is the matrix rank constraint, which is used to control the dimensional compression rate of the modal features;

[0036] The hypergraph Laplacian matrix L = DW, where the degree matrix D ii =∑ j W ij .

[0037] Preferably, the meta-prototype relationship network module performs the following steps:

[0038] Low-rank feature components output from block decomposition Extract the embedding vector from:

[0039]

[0040] Among them, ResNet 34 Contains three residual blocks, each containing two layers of 3×3 convolution, with output channels of 64, 128, and 256 respectively;

[0041] Calculate the missing category prototype vector:

[0042]

[0043] Among them, K≥5 is the number of supporting samples for each category, and C is the total number of missing categories;

[0044] Incrementally update the prototype set, when new samples are added When updating the prototype:

[0045]

[0046] Calculate the area to be inspected With prototype P c Matching degree:

[0047]

[0048] in, is the global average pooling; W1 and W2 are the weights of the fully connected layer; g(·) is the global average pooling operation; σ is the Sigmoid function; || is the vector concatenation operation.

[0049] Preferably, the multi-objective game optimization module performs the following steps:

[0050] Based on the input relationship matching R qc ∈[0,1] to construct the defective state space Among them, s i The severity level meets And N S ≤10; It is a floor operation;

[0051] Calculate the three-dimensional return tensor The calculation satisfies:

[0052]

[0053] Among them, a is the maintenance action; e is the real-time environmental parameter, which is dynamically collected by the onboard sensor; α1, α2, α3 are adjustable weight coefficients, α1+α2+α3=1 and α i≥0; S(a) is the safety benefit function; C(a) is the maintenance cost function; E(e) is the environmental adaptability function;

[0054] Define the maintenance action space And the environmental parameter space E={e j =(Temperature j ,humidity j )}, where N A ≥5;e j Real-time data collection from onboard sensors;

[0055] Solve the Nash equilibrium strategy through iteration:

[0056] Initialize strategy distribution

[0057] Update regret value:

[0058]

[0059] in, R qc (s) is the mean value of the relationship matching degree corresponding to the defect state s;

[0060] Generate strategy:

[0061]

[0062] Output optimal maintenance decision Among them, T max is the number of iterations.

[0063] Preferably, the closed-loop feedback module performs the following steps:

[0064] Calculate feedback signals based on the actual execution effect of maintenance decisions:

[0065]

[0066] Among them, t 实际 The actual maintenance time; 预佔 Estimate the time required for decision making; C 实际 is the actual cost; C 预算 is the preset budget threshold; N 违规 The number of safety regulations violations;

[0067] Meta-prototype network support set sample number update:

[0068]

[0069] in, It is a floor operation;

[0070] Tensor decomposition hypergraph regularization coefficient update:

[0071]

[0072] Where sgn(·) is the sign function;

[0073] When δ∈(0.9,1.1] is satisfied for 5 consecutive times, the system parameters are frozen.

[0074] The present invention also provides an aircraft defect intelligent assessment method based on multimodal fusion, comprising the following steps:

[0075] S1. Use visible light cameras, infrared thermal imagers, ultrasonic probes, and X-ray detectors to synchronously collect aircraft surface inspection data. Use timestamp matching and spatial coordinate calibration to achieve spatiotemporal alignment of the four-modal data and generate a raw data set with a unified spatiotemporal reference.

[0076] S2. Based on the aligned data, the physical eigenvectors of each mode are extracted respectively, stacked along the time dimension to construct a fifth-order eigenvector. The tensor is then decomposed into blocks using hypergraph modeling technology to obtain low-rank eigenvalues ​​and modal correlations.

[0077] S3. Based on the low-rank feature components, dynamically construct a set of category prototypes reflecting different defect types through a meta-learning mechanism, and calculate the multidimensional relationship matching between the features of the area to be detected and each category prototype;

[0078] S4. Based on the relationship matching degree, combined with preset game weight parameters, including maintenance cost weight, safety priority weight, environmental adaptability weight, real-time environmental monitoring parameters, and safety specification requirements, a Nash equilibrium strategy is used to search for the optimal maintenance decision solution in the action space;

[0079] S5. Based on the actual execution effect data of the maintenance plan, the hypergraph model parameters, category prototype set and game weights are jointly and dynamically adjusted to form a closed-loop iterative process of detection-decision-optimization.

[0080] The present invention provides an intelligent aircraft defect assessment system and method based on multimodal fusion. It has the following beneficial effects:

[0081] 1. This invention builds a joint perception system with a unified spatiotemporal benchmark by synchronously collecting and spatially aligning heterogeneous multi-source data from visible light, infrared, ultrasound, and X-rays. This system addresses the limited viewing angle and susceptibility to environmental interference associated with single-modality detection. Block decomposition technology guided by hypergraph modeling enables low-rank joint representation of cross-modal features, significantly improving the recognition accuracy and noise immunity of complex defects (such as internal debonding and surface cracks).

[0082] 2. This invention dynamically constructs and incrementally updates a set of defect category prototypes through a meta-learning mechanism, overcoming the limitations of traditional fixed prototype models and enabling the system to adapt to new defect patterns and data distribution drift. Combined with a nonlinear relationship matching network, it effectively quantifies the similarity between the inspected area and historical defects, enhancing the generalized recognition capability for small sample defects and edge cases.

[0083] 3. This invention utilizes a multi-dimensional game optimization strategy based on Nash equilibrium, modeling multiple constraints such as maintenance costs, safety regulations, and environmental adaptability as a payoff function. This strategy generates a globally optimal decision through an iterative equilibrium solution. Compared to traditional single-objective optimization methods, this significantly improves the balance between the economic and safety aspects of maintenance solutions and reduces the risk of subjective errors in human decision-making.

[0084] 4. This invention's closed-loop feedback loop, from decision execution to parameter adjustment, dynamically modifies feature extraction, prototype learning, and game weight configuration based on actual maintenance results, enabling continuous self-optimization of the system in a dynamic environment. This mechanism effectively mitigates the problem of model performance degradation over time, ensuring robustness and reliability for long-term deployment.

[0085] 5. This invention automates the entire process, from multimodal data acquisition, feature fusion, defect identification, to maintenance decision-making, reducing the need for manual intervention. By integrating core technologies such as spatiotemporal alignment, tensor decomposition, and game optimization, it provides a high-precision, high-efficiency intelligent solution for aircraft maintenance, significantly reducing operational costs and improving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a system architecture diagram of the present invention;

[0087] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0088] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0089] See also Figure 1 The embodiment of the present invention provides an aircraft defect intelligent assessment system and method based on multimodal fusion, including:

[0090] a multimodal data acquisition module configured to synchronously acquire visible light images, infrared thermal images, ultrasonic signals, and X-ray projection data, and perform spatiotemporal alignment preprocessing to obtain multimodal data;

[0091] In this embodiment, the multimodal data acquisition module realizes the synchronous acquisition and preprocessing of aircraft defect data through multi-sensor fusion and spatiotemporal alignment technology. The specific implementation method is as follows:

[0092] The multimodal data acquisition module integrates a visible light imaging unit, an infrared thermal imaging unit, an ultrasonic detection unit and an X-ray detection unit.

[0093] The visible light imaging unit uses a high-resolution optical camera to capture high-definition image data of the aircraft surface. Its imaging data contains RGB three-channel color information and spatial texture features.

[0094] The infrared thermal imaging unit is equipped with an uncooled microbolometer array, which generates thermal image data representing temperature abnormality areas by measuring the infrared radiation intensity distribution on the target surface.

[0095] The ultrasonic detection unit is composed of a multi-channel piezoelectric transducer array, which transmits ultrasonic pulse signals into the interior of the aircraft, receives the echo waveform reflected from the defect interface, and obtains internal structural integrity information through time domain signal analysis.

[0096] The X-ray detection unit uses a digital flat-panel detector and generates projection data reflecting the density differences inside the material based on the principle of penetrating imaging.

[0097] To achieve spatiotemporal consistency of multimodal data, the modules adopt the following collaborative processing approach:

[0098] For time synchronization, a precision clock synchronization protocol (preferably PTP) is used to establish a time base for each sensor. Hardware trigger signals are used to control data acquisition timing, ensuring that the timestamp deviation of visible light image frames, thermal image frames, ultrasonic waveform sampling periods, and X-ray projection data is less than a set threshold. Specifically, each sensor is equipped with a GPS-synchronized clock module, and timestamp interpolation compensation technology is used to eliminate transmission delays, achieving microsecond-level time alignment across modal data.

[0099] In terms of spatial alignment, a unified coordinate system is established based on a multi-sensor joint calibration method. Preferably, a checkerboard calibration plate is used as a spatial reference benchmark, and spatial registration is achieved through the following steps:

[0100] Place a checkerboard calibration plate of known size in the aircraft inspection area so that it covers the field of view of the visible light camera, infrared thermal imager, and X-ray detector simultaneously;

[0101] Collect the observation data of each sensor on the chessboard and calculate the internal parameter matrix of each modal imaging system and external parameter matrix where R m is the rotation matrix, tm is the translation vector;

[0102] Through the spatial transformation matrix T m→w =[R m |t m ;0 1×3 |1] Map the sensor data to the world coordinate system to satisfy the coordinate transformation relationship:

[0103] X w =T m→w ·X m ;

[0104] Among them, X m is the coordinate in the local coordinate system of the sensor, X w The coordinates are in the world coordinate system.

[0105] The module performs the following preprocessing operations on the raw multimodal data to remove noise and distortion:

[0106] Visible light images: A dark channel prior dehazing algorithm is applied to enhance surface details, and the Zhang Zhengyou calibration method is used to correct lens distortion to output a distortion-free RGB image.

[0107] Infrared thermal image: Non-uniformity correction is performed based on the blackbody radiation calibration curve, and the original radiation intensity is converted into temperature distribution data through the temperature-grayscale mapping table;

[0108] Ultrasonic signal: Adaptive threshold filtering is used to eliminate environmental noise, the effective echo interval is intercepted by the time domain window function, and the time domain waveform envelope characteristics are extracted;

[0109] X-ray projection: A flat-panel detector gain correction algorithm is applied to compensate for pixel response differences, and the FDK algorithm is used to reconstruct tomographic images and generate orthogonal projection data.

[0110] The module verifies the spatiotemporal alignment accuracy through feature matching and reprojection error evaluation:

[0111] Time alignment verification: During the synchronous acquisition period, the high-frequency LED flasher is controlled to generate a light pulse signal. By analyzing the pulse trigger time deviation in each sensor signal, the time alignment residual is calculated. Ensure that Δt < τ (τ is the preset threshold);

[0112] Spatial alignment verification: Calculating reprojection error at the corners of the checkerboard calibration plate Among them, x j is the measured corner coordinate, is the theoretical projection coordinate, and the space alignment is qualified when E<∈.

[0113] The preprocessed multimodal data is packaged in the following format:

[0114] Timestamp: ISO-8601 standard time format is uniformly used to record the data collection time;

[0115] Spatial coordinates: additional position tags and sensor attitude parameters in the world coordinate system;

[0116] Data channel: Visible light RGB image, infrared temperature matrix, ultrasonic time domain waveform, and X-ray projection matrix are stored as multidimensional arrays in time series.

[0117] A tensor construction and decomposition module is used to construct the multimodal data into a fifth-order feature tensor in a unified spatiotemporal coordinate system, and extract low-rank feature components through a block decomposition algorithm guided by hypergraph modeling;

[0118] In this embodiment, the tensor construction and decomposition module realizes the joint representation of aircraft defect features and the extraction of low-rank components through multimodal feature fusion and hypergraph-guided block decomposition technology. The specific implementation method is as follows:

[0119] The module first performs modality-specific feature extraction on the spatiotemporally aligned multimodal data:

[0120] Visible light data V t Feature extraction: The pre-trained ResNet-34 deep residual network is used as the basic feature extractor. After removing its fully connected layer, the first three residual block structures are retained. After the visible light image is input into the network, the global average pooling layer of the third residual block outputs a 512-dimensional feature vector. Capture deep semantic features of aircraft surface textures, cracks, and corrosion areas.

[0121] Infrared thermal image feature extraction: Calculate the horizontal gradient of the thermal image based on the temperature gradient operator With vertical gradient Generate a 256-dimensional feature vector by element-by-element addition and maximum pooling operation Characterize the morphological characteristics and heat conduction properties of the temperature anomaly area, I t For infrared data.

[0122] Ultrasonic signal feature extraction: After adding a Hamming window to the original time domain waveform, a short-time Fourier transform (STFT) is performed, and the energy spectrum density values ​​of the first 128 frequency points are intercepted to form the frequency domain feature vector Reflection characteristics and structural integrity information reflecting internal defects of materials, U t Ultrasonic data.

[0123] X-ray projection feature extraction: Statistical projection data 16-bin histogram distribution, after L2 normalization processing to generate 64-dimensional histogram feature vector Quantify the spatial distribution of density differences and potential defects within a material.

[0124] The four modal feature vectors are stacked along the time dimension T to construct a fifth-order feature tensor Its time slice satisfies:

[0125]

[0126] in, is the vector outer product operation, that is This operation encodes the nonlinear interaction relationship between modal features into a high-order tensor structure, realizing multidimensional correlation modeling of cross-modal features.

[0127] To characterize the complex relationships between multimodal features, the module constructs a hypergraph model The vertex set Corresponding to the four types of sensor modalities, the hyperedge set ε describes the consistency of cross-modal feature distribution. Construct the hypergraph adjacency matrix W, and the hyperedge weight is calculated as:

[0128]

[0129] in, is the feature distribution entropy value of the mth mode at the kth time slice, reflecting the degree of discreteness of the feature distribution; is the feature distribution of the mth mode in the kth time slice; p(x) is the normalized probability distribution of the feature histogram.

[0130] The module extracts low-rank feature components through a block decomposition algorithm with a hypergraph regularization term to solve the following optimization problem:

[0131]

[0132] in, is the core tensor; is the factor matrix of the mth mode; D is the hypergraph degree matrix; is the tensor modal product, λ = 0.1 is the hypergraph regularization coefficient, and R = 5 is the decomposition rank.

[0133] The block term decomposition algorithm imposes an explicit rank constraint on the factor matrix of each mode, which can be formalized mathematically as follows:

[0134] Decompose the rank constraint into [rank(A (1) ),rank(A (2) ),rank(A (3) ),rank(A(4) )]=[8,8,6,3];

[0135] Among them, A (m) is the factor matrix of the mth mode; rank(·) is the matrix rank constraint, which is used to control the dimensional compression rate of the modal features.

[0136] To strengthen the semantic association between multimodal features, the algorithm introduces a hypergraph Laplace regularization term into the objective function:

[0137]

[0138] in, is the hypergraph adjacency matrix; D is the hypergraph degree matrix in diagonal matrix form, satisfying D ii =∑ j W ij ; It is the horizontal concatenation of all factor matrices of the mth mode; the hypergraph Laplacian matrix L = DW.

[0139] This regularization term forces the decomposed low-rank features to satisfy the following properties by penalizing the discontinuity of the factor matrix on the hypergraph correlation mode:

[0140] Cross-modal consistency: If two modalities have a high-weight connection in the hyperedge (i.e., the feature distribution entropy is similar), the projection directions of their factor matrices in the latent space tend to be aligned;

[0141] Intra-class compactness: modal features within the same hyperedge have low variance distribution in the column space of the factor matrix;

[0142] Noise robustness: Abnormal modal correlation caused by sensor noise is suppressed by the Laplacian operator of DW.

[0143] The algorithm solves the constrained optimization problem using the alternating direction method of multipliers (ADMM), iteratively updating the following variables:

[0144] Core Tensor Update: Fixed factor matrix Solve by least squares method:

[0145]

[0146] Factor Matrix Update: Fixed core tensor With other modal factor matrices, update by gradient descent:

[0147]

[0148] Where η is the learning rate, The reconstructed tensor for the current iteration;

[0149] Lagrange multiplier update: adjust the multiplier matrix according to the constraint violation amount to ensure that the decomposition result satisfies the rank constraint.

[0150] The above optimization problem is solved iteratively by the alternating direction multiplier method (ADMM) to obtain the decomposed low-rank feature components Per-core tensor Encodes specific types of cross-modal defect feature patterns, such as surface cracks, internal disbonds, or material corrosion. The factor matrix This reveals the contribution weight of each modal feature in the corresponding missing pattern, providing interpretable feature association information for the subsequent meta-prototype relationship network.

[0151] The meta-prototype relationship network module dynamically constructs an incrementally updateable set of defect category prototypes based on the decomposed feature components, and calculates the relationship matching degree between the inspected area and each prototype;

[0152] In this embodiment, the meta-prototype relationship network module realizes adaptive representation and similarity measurement of aircraft defect categories through dynamic prototype learning and relationship matching mechanism. The specific technical solution is as follows:

[0153] The low-rank feature components output from the block decomposition module The process of extracting high-dimensional embedding vectors is described as follows:

[0154]

[0155] Among them, ResNet 34 It contains three residual blocks, each containing two layers of 3×3 convolution, with output channels of 64, 128, and 256 respectively. The last layer is connected to a global average pooling layer to compress the feature vector to 512 dimensions.

[0156] The residual connection is implemented through cross-layer identity mapping to alleviate the gradient disappearance problem. The formula is:

[0157]

[0158] in, is the residual function; x is the input feature; W i are the convolutional layer parameters.

[0159] Constructing prototype vector sets reflecting different defect categories based on embedded vectors

[0160]

[0161] Among them, K≥5 is the number of supporting samples for each category, ensuring the statistical significance of the prototype representation; C is the total number of missing categories, the initial value is set by prior knowledge and dynamically expanded with incremental learning; the prototype vector The essence is the centroid of the corresponding category feature space, which aggregates the common features of similar samples through average pooling.

[0162] When a new missing sample is added When , the sliding average strategy is used to update the prototype vector:

[0163]

[0164] This update rule adjusts the proportional relationship between the prototype vector and the number of supported samples, gradually incorporating new sample characteristics while retaining historical feature information, and avoiding prototype offset caused by sudden noise samples. Preferably, set the new sample screening threshold Only samples whose distance from the prototype is less than ∈ are allowed to participate in the update, thereby suppressing outlier interference.

[0165] Characteristics of the area to be inspected With prototype P c Nonlinear mapping of the matching degree:

[0166]

[0167] in, is global average pooling, which compresses the spatial dimension and retains the channel response; W1 and W2 are the weights of the fully connected layer, which mine cross-modal correlation features through nonlinear transformation; g(·) is the global average pooling operation; σ is the Sigmoid function; || is the vector concatenation operation;

[0168] The module optimizes the prototype representation and matching network parameters end-to-end through the cross entropy loss function:

[0169]

[0170] Among them, y c ∈{0, 1} is the true label of the missing class. During the backpropagation process, the gradient signal simultaneously updates the convolutional layer parameters θ and the fully connected layer weights W1 and W2 of ResNet-34, prompting the prototype vector to converge towards the characteristics of similar samples and away from heterogeneous samples.

[0171] The multi-objective game optimization module receives the relationship matching calculation results, combines the preset maintenance cost parameters and real-time environmental conditions, and generates the optimal maintenance decision through the Nash equilibrium strategy;

[0172] In this embodiment, the multi-objective game optimization module realizes the collaborative optimization of maintenance decisions through multi-constraint fusion and game equilibrium strategy. The specific technical solution is as follows:

[0173] Relationship matching degree R based on the output of meta-prototype relation network qc ∈[0,1], construct the discretized defect state space:

[0174]

[0175] Among them, s i The severity level meets And N S ≤10; To round down, the continuous matching degree is mapped to a 10-level defect severity scale, where s = 1 represents a minor surface scratch and s = 10 represents penetrating structural damage. This discretization process compresses the decision space dimension through quantitative indicators, providing structured input for solving game strategies.

[0176] The benefit function of maintenance decision is defined as a weighted combination of safety benefit, cost-effectiveness and environmental adaptability, and a three-dimensional benefit tensor is constructed.

[0177]

[0178] Among them, a is the maintenance action, including emergency repair, routine maintenance, delayed processing, component replacement and shutdown maintenance; e is the real-time environmental parameter, which is dynamically collected by onboard sensors; α1, α2, α3 are adjustable weight coefficients, α1+α2+α3=1 and α i ≥0; S(a) is the safety benefit function; C(a) is the maintenance cost function; E(e) is the environmental adaptability function.

[0179] Define the maintenance action space And the environmental parameter space E={e j =(Temperature j ,humidity j )}, where N A ≥5;e j Real-time data collection from onboard sensors;

[0180] The Nash equilibrium point is approached by the regret minimization algorithm. The specific process is as follows:

[0181] Define the initial strategy distribution of each maintenance action as uniform distribution:

[0182]

[0183] In the tth iteration, calculate the action a i Utility difference relative to other actions:

[0184]

[0185] The utility function Ui (a i ) is calculated as:

[0186]

[0187] Among them, R qc (s) is the mean value of the relationship matching degree corresponding to the defect state s;

[0188] Use exponential smoothing mapping to convert regret values ​​into probability distributions:

[0189]

[0190] After T max After iterations, the action with the highest cumulative probability is selected as the optimal maintenance decision:

[0191]

[0192] Among them, T max is the number of iterations.

[0193] A closed-loop feedback module feeds back the actual execution effect of the maintenance decision to the meta-prototype relationship network module and the tensor decomposition module to complete the dynamic adjustment of system parameters;

[0194] In this embodiment, the closed-loop feedback module realizes dynamic self-adjustment and optimization of system parameters through back propagation of actual maintenance results. The specific technical solution is as follows:

[0195] Based on the actual execution effect of the maintenance decision, the normalized feedback signal δ is calculated to quantify the decision quality:

[0196]

[0197] Among them, t 实际 The actual maintenance time; 预佔 Estimate the time required for decision making; C 实际 is the actual cost; C 预算 is the preset budget threshold; N 违规 The number of safety regulations violations;

[0198] The feedback signal Comprehensively reflect time deviation, cost control and safety compliance:

[0199] When δ>1, it indicates that the actual execution effect is worse than expected (such as overtime, overspending or violation);

[0200] When δ<1, it indicates that the execution effect is better than expected;

[0201] When δ≈1, it indicates that the system decision matches the actual situation well.

[0202] Adjust the number of support set samples K of the meta-prototype relationship network according to the feedback signal δ:

[0203]

[0204] in, It is a floor operation;

[0205] Preferably, the upper and lower limits of K are set to K min =5, K max =50 to avoid the support set being too small or too large due to extreme feedback.

[0206] Update the hypergraph regularization coefficient λ of the tensor decomposition module based on δ:

[0207]

[0208] Where sgn(·) is the sign function;

[0209] When δ∈[0.9,1.1] is satisfied for 5 consecutive iterations, the system is determined to be stable and parameter freezing is triggered:

[0210] The number of samples K in the support set of the meta-prototype network stops updating;

[0211] The hypergraph regularization coefficient λ of the tensor decomposition module is fixed to the current value;

[0212] The system enters read-only mode and only performs decision reasoning without any parameter adjustment.

[0213] This mechanism avoids parameter oscillation caused by noise interference after the system converges by monitoring the stability of the feedback signal.

[0214] See also Figure 2 The present invention also provides an aircraft defect intelligent assessment method based on multimodal fusion, comprising the following steps:

[0215] S1. This step collects multi-dimensional data on the surface and internal structure of the aircraft through a sensor network composed of visible light cameras, infrared thermal imagers, ultrasonic array probes and X-ray detectors. The visible light camera captures high-resolution surface images, the infrared thermal imager records the temperature distribution field, the ultrasonic probe collects the internal reflection waveform of the material, and the X-ray detector generates penetrating projection data. In order to achieve spatiotemporal consistency of cross-modal data, a precise clock synchronization protocol is used to establish a unified time base to ensure the microsecond alignment of the sampling moments of each sensor; at the same time, based on the multi-sensor joint calibration technology, a unified spatial coordinate system is established through a checkerboard calibration plate, and each modal data is mapped to the same physical space base to eliminate the difference in perspective and distortion. After the original data is denoised, distortion corrected and format standardized, a spatiotemporally aligned multimodal data set is generated;

[0216] S2. Modal-specific features are extracted from the multimodal data after time-space alignment: deep semantic features are extracted from visible light images through pre-trained deep networks; temperature gradient field features are calculated from infrared thermal images; frequency domain energy distribution is extracted from ultrasonic signals through time-frequency transformation; and histogram features are statistically analyzed for X-ray projection data. The four types of feature vectors are stacked along the time axis by outer products to construct a fifth-order feature tensor, whose high-order dimensions represent the nonlinear interaction relationship between cross-modal features. Hypergraph modeling technology is further introduced, with sensor modalities as hypergraph vertices, and a hyperedge weight matrix is ​​constructed based on the feature distribution entropy. The hypergraph regularization term is designed to embed the block term decomposition objective function. Under the low-rank constraint, the core components reflecting the defect mode and the modal correlation factor matrix are separated to achieve joint dimensionality reduction and correlation mining of cross-modal features.

[0217] S3. Based on the low-rank feature components obtained by decomposition, a meta-learning framework is used to dynamically construct a prototype set of defect categories. The low-rank components are mapped to high-dimensional embedding vectors through an improved residual network, and the centroid of the embedding vectors of similar defective samples is calculated as the prototype representation. When new defective samples are added, the prototype vector is updated using a sliding average strategy to ensure that the prototype set evolves dynamically with the data distribution. For the features of the detection area, after compressing the spatial dimension through global pooling, they are spliced ​​with the prototype vector and input into the fully connected network. The degree of matching between the nonlinear relationship and each prototype is calculated to quantify the similarity and severity of the defect type;

[0218] S4. Map the relationship matching degree to a discrete defect state level. Combined with the preset maintenance cost model (including labor cost and time coefficient), safety priority weights, and real-time environmental parameters (temperature, humidity, and wind speed), a three-dimensional benefit tensor is constructed to quantify the comprehensive benefits of different maintenance actions under specific conditions and environments. Based on Nash equilibrium theory, an iterative regret value matching algorithm is used to search for the optimal strategy in the maintenance action space: initialize a uniform strategy distribution, and through multiple rounds of utility calculation and strategy update, gradually converge to an equilibrium solution, outputting a maintenance decision plan that comprehensively balances safety, economy, and environmental adaptability.

[0219] S5. Based on the actual execution results of maintenance decisions (including time deviation, cost overruns, and the number of safety violations), a normalized feedback signal is calculated and the size of the meta-prototype network support set, the tensor decomposition hypergraph regularization coefficient, and the game weight parameters are dynamically adjusted. When the feedback signal remains continuously in the stable range, the system parameter freezing mechanism is triggered, terminating the adaptive adjustment process. This closed-loop mechanism continuously integrates actual operating condition data to optimize feature representation, decision logic, and multi-objective weight configuration, ensuring that the system maintains high accuracy and strong robustness in a dynamic environment, forming a complete iterative optimization chain from data acquisition and decision generation to parameter self-correction.

[0220] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The intelligent aircraft defect assessment system based on multimodal fusion is characterized by: include: a multimodal data acquisition module configured to synchronously acquire visible light images, infrared thermal images, ultrasonic signals, and X-ray projection data, and perform spatiotemporal alignment preprocessing to obtain multimodal data; A tensor construction and decomposition module is used to construct the multimodal data into a fifth-order feature tensor in a unified spatiotemporal coordinate system, and extract low-rank feature components through a block decomposition algorithm guided by hypergraph modeling; The meta-prototype relationship network module dynamically constructs an incrementally updateable set of defect category prototypes based on the decomposed feature components, and calculates the relationship matching degree between the inspected area and each prototype; The multi-objective game optimization module receives the relationship matching calculation results, combines the preset maintenance cost parameters and real-time environmental conditions, and generates the optimal maintenance decision through the Nash equilibrium strategy; The closed-loop feedback module feeds back the actual execution effect of the maintenance decision to the meta-prototype relationship network module and the tensor decomposition module to complete the dynamic adjustment of the system parameters.

2. The aircraft defect intelligent assessment system based on multimodal fusion according to claim 1, characterized in that: The multimodal data acquisition module includes: Visible light camera, used to obtain high-definition image data of the aircraft surface; Infrared thermal imager, used to capture temperature anomaly distribution data; Ultrasonic array probe, used to collect internal defect reflection waveform data; X-ray detectors for generating projection data of the internal structure of materials; Among them, each sensor realizes data acquisition timing alignment through a time synchronization protocol to ensure that the timestamp deviation of each modal data is less than the set threshold.

3. The aircraft defect intelligent assessment system based on multimodal fusion according to claim 1, characterized in that: The tensor construction and decomposition module performs: Feature extraction is performed on multimodal data respectively, where: Visible light data V t Apply the pre-trained ResNet-34 network to generate the feature vector ψ(V t ); Infrared Data I t Through the temperature gradient operator Extract the feature vector φ(I t ); Ultrasonic Data U t Use short-time Fourier transform to extract the first d f =128-dimensional frequency domain components X-ray data t Statistical histogram feature χ(X t ); The four modal eigenvectors are stacked along the time dimension T to construct a fifth-order eigentensor Its slice satisfies in, is the vector outer product operation; Extracting low-rank feature components based on block decomposition guided by hypergraph modeling: Construct the hypergraph adjacency matrix W, and the hyperedge weight is calculated as: in, is the characteristic distribution entropy of the mth mode at the kth time slice; is the characteristic distribution of the mth mode in the kth time slice; Solve an optimization problem with a hypergraph regularization term: in, is the core tensor; is the factor matrix of the mth mode; D is the hypergraph degree matrix; is the tensor modal product, λ is the coefficient of the hypergraph regularization term, and R is the decomposition rank.

4. The aircraft defect intelligent assessment system based on multimodal fusion according to claim 3, characterized in that: The block item decomposition algorithm satisfies: Decompose the rank constraint into [rank(A (1) ),rank(A (2) ),rank(A (3) ),rank(A (4) )]=[8,8,6,3]; Among them, A (m) is the factor matrix of the mth mode; rank(·) is the matrix rank constraint, which is used to control the dimensional compression rate of the modal features; The hypergraph Laplacian matrix L = DW, where the degree matrix D ii =∑ j W ij .

5. The aircraft defect intelligent assessment system and method based on multimodal fusion according to claim 1, characterized in that: The meta-prototype relationship network module performs the following steps: Low-rank feature components output from block decomposition Extract the embedding vector from: Among them, ResNet 34 Contains three residual blocks, each containing two layers of 3×3 convolution, with output channels of 64, 128, and 256 respectively; Calculate the missing category prototype vector: Among them, K≥5 is the number of supporting samples for each category, and C is the total number of missing categories; Incrementally update the prototype set, when new samples are added When updating the prototype: Calculate the area to be inspected With prototype P c Matching degree: in, is the global average pooling; W1 and W2 are the weights of the fully connected layer; g(·) is the global average pooling operation; σ is the Sigmoid function; || is the vector concatenation operation.

6. The aircraft defect intelligent assessment system based on multimodal fusion according to claim 1, characterized in that: The multi-objective game optimization module performs the following steps: Based on the input relationship matching R qc ∈[0,1] to construct the defective state space Among them, s i The severity level meets And N S ≤10; It is a floor operation; Calculate the three-dimensional return tensor The calculation satisfies: Among them, a is the maintenance action; e is the real-time environmental parameter, which is dynamically collected by the onboard sensor; α1, α2, α3 are adjustable weight coefficients, α1+α2+α3=1 and α i ≥0; S(a) is the safety benefit function; C(a) is the maintenance cost function; E(e) is the environmental adaptability function; Define the maintenance action space And the environmental parameter space E={e j =(Temperature j ,humidity j )}, where N A ≥5;e j Real-time data collection from onboard sensors; Solve the Nash equilibrium strategy through iteration: Initialize strategy distribution Update regret value: in, R qc (s) is the mean value of the relationship matching degree corresponding to the defect state s; Generate strategy: Output optimal maintenance decision Among them, T max is the number of iterations.

7. The aircraft defect intelligent assessment system based on multimodal fusion according to claim 1, characterized in that: The closed-loop feedback module performs the following steps: Calculate feedback signals based on the actual execution effect of maintenance decisions: Among them, t 实际 The actual maintenance time; 预佔 Estimate the time required for decision making; C 实际 is the actual cost; C 预算 is the preset budget threshold; N 违规 The number of safety regulations violations; Meta-prototype network support set sample number update: in, It is a floor operation; Tensor decomposition hypergraph regularization coefficient update: Where sgn(·) is the sign function; When δ∈[0.9,1.1] is satisfied for 5 consecutive times, the system parameters are frozen.

8. An intelligent aircraft defect assessment method based on multimodal fusion, applied to the system according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1. Use visible light cameras, infrared thermal imagers, ultrasonic probes, and X-ray detectors to synchronously collect aircraft surface inspection data. Use timestamp matching and spatial coordinate calibration to achieve spatiotemporal alignment of the four-modal data and generate a raw data set with a unified spatiotemporal reference. S2. Based on the aligned data, the physical eigenvectors of each mode are extracted respectively, stacked along the time dimension to construct a fifth-order eigenvector. The tensor is then decomposed into blocks using hypergraph modeling technology to obtain low-rank eigenvalues ​​and modal correlations. S3. Based on the low-rank feature components, dynamically construct a set of category prototypes reflecting different defect types through a meta-learning mechanism, and calculate the multidimensional relationship matching between the features of the area to be detected and each category prototype; S4. Based on the relationship matching degree, combined with preset game weight parameters, including maintenance cost weight, safety priority weight, environmental adaptability weight, real-time environmental monitoring parameters, and safety specification requirements, a Nash equilibrium strategy is used to search for the optimal maintenance decision solution in the action space; S5. Based on the actual execution effect data of the maintenance plan, the hypergraph model parameters, category prototype set and game weights are jointly and dynamically adjusted to form a closed-loop iterative process of detection-decision-optimization.

Images