Digital twinning-based all-weather aircraft external defect intelligent detection system

Abstract

The invention relates to the field of aircraft external detection, and discloses an all-weather aircraft external defect intelligent detection system based on digital twinning, and the system comprises a multi-modal data collection module which is used for collecting aircraft external multi-modal data, and the multi-modal data comprises aircraft external image data, ultrasonic wave, X-ray and thermal infrared data; the fusion module is used for enabling the multi-modal data to form a unified fusion feature; the defect identification module is used for carrying out aircraft external defect identification based on a defect identification model and the fusion features; and the spatial positioning module is used for mapping the identification result. The unmanned aerial vehicle and the mechanical arm unit are combined, and external images, ultrasonic waves, X-rays and thermal infrared data of the aircraft are rapidly and efficiently collected. The unmanned aerial vehicle unit can cover surface images of a wide area, the mechanical arm unit detects external defects through ultrasonic and X-ray technologies, the defect recognition speed is increased, the system adapts to the complex environment, and the all-weather detection requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of aircraft external detection, and specifically to an all-weather aircraft external defect intelligent detection system based on digital twins. Background Art

[0002] Aircraft external defect detection is crucial to ensuring aircraft safety and reliability. Aircraft may experience various external damages during long-term flight, including cracks, corrosion, and deformation. These defects not only affect flight performance but can also threaten flight safety. Timely and effective defect detection can identify potential problems early, preventing more serious failures caused by untimely repairs. Therefore, rapid and efficient external defect detection is crucial for aircraft maintenance and repair.

[0003] However, traditional manual inspection methods often struggle in complex environments. Manual inspections require inspectors to perform detailed inspections close to the aircraft, which often poses safety risks. For example, in inclement weather or unstable environments, manual inspections not only pose significant safety risks but are also inefficient, making it difficult to conduct comprehensive inspections of large aircraft in a short period of time. Furthermore, manual inspections rely on human experience and judgment, which can easily lead to omissions or misjudgments, making them unable to meet the stringent requirements for rapid aircraft inspection and repair. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides an all-weather intelligent detection system for aircraft external defects based on digital twins, which solves the problems of low efficiency of traditional manual inspection, difficulty in conducting comprehensive inspections of large aircraft in a short period of time, and the risk of omissions or misjudgments.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an all-weather aircraft external defect intelligent detection system based on digital twins, comprising:

[0006] A multimodal data acquisition module, configured to acquire multimodal data of the aircraft's exterior, including external image data, ultrasonic data, X-ray data, and thermal infrared data of the aircraft;

[0007] The fusion module is used to preprocess multimodal data to form a unified fusion feature;

[0008] The defect recognition module identifies defects on the aircraft's exterior based on the defect recognition model and fusion features, and outputs the recognition results.

[0009] The spatial positioning module is used to map the recognition results into spatial coordinates in the overall structure of the aircraft.

[0010] Preferably, the multimodal data acquisition module includes:

[0011] UAV unit: used to collect external image data of the aircraft;

[0012] Robotic arm unit: used to collect ultrasonic, X-ray and thermal infrared data from the outside of the aircraft.

[0013] Preferably, the pretreatment includes:

[0014] For image data: perform image grayscale histogram equalization, image denoising, enhancement and feature extraction;

[0015] For thermal infrared data: perform infrared temperature linear conversion;

[0016] For ultrasonic data: perform signal filtering, envelope extraction, and characteristic waveform analysis to identify echo features and estimate defect location and size;

[0017] For X-ray data: perform image enhancement, edge detection, and contrast adjustment to highlight abnormal areas of the material's internal structure and extract high-density abnormal features.

[0018] Preferably, the image data, thermal infrared data, ultrasonic data and X-ray data are pre-processed and then normalized and weighted fused in sequence to generate fusion features.

[0019] Preferably, the spatial positioning module is based on the digital twin model of the aircraft, maps the recognition results into spatial coordinates in the overall structure of the aircraft, and displays the defect location on the digital twin model, providing real-time visual information of the defect location.

[0020] Preferably, the defect recognition model includes:

[0021] Defect sample library: This stores samples of actual defects, including those generated under different flight environments, temperature and humidity conditions, as well as defect types of fuselage materials and fuselage structural components. The fuselage structural components include wings, fuselage, tail, and engine. Defect types include cracks, corrosion, and deformation. A defect recognition model is established based on the defect types.

[0022] Defect judgment functional unit: Based on the deep learning model and the defect recognition model, the fusion feature is analyzed, and the defect features are identified and output. The defect features include the defect location and defect type of the aircraft.

[0023] Preferably, the defect locations include: wings, fuselage, tail and engine; the defect types include: cracks, dents, corrosion and deformation.

[0024] Preferably, the defect recognition model further includes an updating unit, which uses machine learning and deep learning algorithms to train and update the defect recognition model.

[0025] Preferably, the machine learning and deep learning algorithms include a multi-scale residual network that integrates a convolutional neural network with an attention mechanism, and the network is used to extract multi-level defect features from the aircraft external image and perform classification and positioning tasks, and the optimization objective function is as follows:

[0026] L total =λ1·L cls +λ2·L loc +λ3·L att +λ4·∥θ∥ 2 ;

[0027] Among them, L total is the optimization objective function; L cls Represents the classification loss of defect type, using cross entropy loss function; L loc Represents the regression loss of the defect position, using a smooth loss function; L att Represents the difference loss between the feature weight map generated by the attention mechanism and the actual defect area; ∥θ∥ 2 is a model regularization term used to prevent overfitting; λ1, λ2, λ3 and λ4 are hyperparameters used to control the weight ratio of the loss.

[0028] Preferably, the network structure further includes:

[0029] The basic feature extraction layer uses multiple convolutional layers to obtain local image features;

[0030] Multi-scale residual layer, used to extract defect edge and texture features at different scales;

[0031] Channel attention and spatial attention modules dynamically focus on potential defect areas;

[0032] The dual-branch output layer outputs the defect type classification results and location coordinates respectively.

[0033] The present invention provides an all-weather intelligent detection system for aircraft exterior defects based on digital twins. It has the following beneficial effects:

[0034] 1. By combining a drone unit with a robotic arm unit, the present invention can efficiently collect external images, ultrasonic, X-ray, and thermal infrared data of an aircraft in a short period of time. The drone unit can collect images of the aircraft surface over a wide area, while the robotic arm unit can detect defects on the exterior using ultrasonic, X-ray, and other technologies. This improves the speed of identifying defects on the aircraft's exterior, adapts to complex and changing inspection environments, and can meet all-weather and all-time inspection needs.

[0035] 2. The present invention can realize intelligent and efficient identification of external defects of aircraft by analyzing the fusion characteristics of multimodal data based on a deep learning model. The defect recognition module can accurately identify various types of defects such as cracks, corrosion, and dents by analyzing the fusion characteristics of different modal data, and locate their specific positions. It can greatly reduce the errors and time costs in manual inspection and significantly improve the accuracy and reliability of detection.

[0036] 3. The present invention trains and updates the defect recognition model through an update unit. As new defect data is continuously added, the system can self-learn and optimize recognition performance, improving recognition accuracy and generalization capabilities. Especially in complex environments and changing conditions, the update unit ensures that the system can continue to respond to various challenges, further improving the safety and maintenance management efficiency of the aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the system architecture of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the specification of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] In order to better understand the present invention, the above contents are described in detail below in conjunction with specific embodiments.

[0040] Please see the attached Figure 1 The embodiment of the present invention provides an all-weather aircraft external defect intelligent detection system based on digital twins, including:

[0041] A multimodal data acquisition module, configured to acquire multimodal data of the aircraft's exterior, including external image data, ultrasonic data, X-ray data, and thermal infrared data of the aircraft;

[0042] In this embodiment, the multimodal data acquisition module is used to collect multiple data types from the aircraft's exterior to support subsequent defect identification, analysis, and assessment. The module's drone unit and robotic arm unit work together to capture comprehensive information about the aircraft's exterior. Multimodal data includes, but is not limited to, aircraft external image data, ultrasonic data, X-ray data, and thermal infrared data. Each data type corresponds to a different sensor device, enabling inspection and monitoring of the aircraft's surface and key components from multiple dimensions.

[0043] Specifically, the drone unit is used to collect image data of the aircraft's exterior, including visible light images and high-resolution photographic images. The use of the drone platform is highly flexible, enabling comprehensive scanning of the aircraft's surface at various altitudes and angles.

[0044] Specifically, image data is collected by a high-definition visible light camera onboard the drone. With a resolution of tens of millions of pixels, this camera can capture detailed images of cracks, corrosion, and other structural defects on the aircraft's surface. In some embodiments, a high-definition infrared camera can also be installed to capture the aircraft's surface temperature distribution, thereby identifying thermal anomalies caused by material aging or structural defects.

[0045] The robotic arm unit is another key component of the multimodal data acquisition module. This unit typically includes a motion system with 6-7 degrees of freedom, enabling flexible manipulation and positioning. The robotic arm's design enables precise control of sensor equipment, allowing it to collect data from hard-to-reach areas outside the aircraft. Specifically, the robotic arm is primarily used to collect the following types of data:

[0046] Ultrasonic data

[0047] In some embodiments, ultrasonic sensors precisely scan aircraft surfaces using a robotic arm. Ultrasonic sensors can detect surface and interior defects such as cracks, pores, and corrosion by emitting and receiving high-frequency sound waves. The reflections and echoes of the ultrasonic signals reveal the location and size of these defects.

[0048] During processing, ultrasonic data often requires signal filtering, envelope extraction, and waveform analysis. By analyzing the echo signal, the specific location and size of the defect can be determined.

[0049] X-ray data

[0050] Mounted on a robotic arm, the X-ray imaging system acquires high-resolution X-ray images without damaging the aircraft structure, enabling inspection of the aircraft's internal structure. X-ray sensors can penetrate materials, revealing hidden defects such as cracks and corrosion. This process involves image enhancement, contrast adjustment, and edge detection, enabling the clear display of defects.

[0051] Thermal infrared data

[0052] The robotic arm unit can also be equipped with a thermal infrared sensor to capture thermal infrared images of the aircraft surface. This data reveals the temperature distribution across the aircraft surface, enabling the identification of thermal anomalies caused by internal defects. During processing, the temperature information in the infrared image is converted to a temperature value through a linear conversion and compared with the background ambient temperature, highlighting thermal anomalies caused by damage.

[0053] Collaborative working mechanism for data collection

[0054] In practice, the drone and robotic arm units work together to form a highly integrated data acquisition platform. The drone provides rapid, global image acquisition of the aircraft's exterior, while the robotic arm performs more detailed inspections of specific areas, particularly those beyond the reach of the drone (such as the engine compartment and underwing surfaces). This collaborative approach ensures comprehensive inspection of the aircraft's exterior in a variety of environmental conditions.

[0055] Through this approach, the multimodal data acquisition module accurately and rapidly acquires a wide range of critical data from the aircraft's exterior, providing reliable data support for subsequent data preprocessing, defect identification, and assessment. Furthermore, the highly flexible nature of the drone and robotic arm ensures efficient operation in adverse weather and complex environments, ensuring all-weather adaptability.

[0056] In a typical implementation, after a drone captures image data of an aircraft's surface, a robotic arm then performs detailed ultrasonic scanning of specific areas, or uses X-ray and thermal infrared sensors to further obtain internal defect information. This combination of sensor data collection and subsequent processing enables comprehensive, efficient, and accurate aircraft exterior defect detection.

[0057] The fusion module is used to preprocess multimodal data to form a unified fusion feature;

[0058] In this embodiment, the fusion module is used to generate unified fusion features from multimodal data through a series of preprocessing steps. This module is a core component of the integration system, responsible for fusing data from different sensors to improve the accuracy and reliability of defect detection. By fusing multiple data types, the system can conduct comprehensive aircraft inspections from multiple dimensions, providing accurate data support for subsequent defect identification and risk assessment.

[0059] In the aforementioned multimodal data acquisition module, aircraft exterior image data, ultrasonic data, X-ray data, and thermal infrared data are already collected. The processing methods and technical implementations for each data type are described in detail in the relevant sections. These collected data must undergo appropriate preprocessing to ensure they can be fused under a unified standard and generate high-quality fused features.

[0060] Data preprocessing process

[0061] In this embodiment, the preprocessing process includes specific processing methods for different types of data:

[0062] For image data:

[0063] In the process of image data processing, the image grayscale histogram equalization is first performed. This process makes the details in the image clearer and more visible by enhancing the contrast of the image. The histogram equalization method uses the following formula:

[0064]

[0065] Where s is the grayscale value of the output image; T(r) is the grayscale transformation function, that is, the new grayscale value after the original grayscale r is mapped; p r (w) is the grayscale distribution function of the input image; L is the total number of grayscale levels; is the integral part from 0 to r.

[0066] In addition, image denoising, enhancement, and feature extraction are also widely used. These processes can effectively remove noise from images and improve the accuracy of subsequent defect recognition.

[0067] For thermal infrared data:

[0068] Thermal infrared data preprocessing involves performing a linear temperature conversion on the image, converting the grayscale values ​​in the thermal infrared image to actual temperature values. Specifically, the temperature value of each pixel in the infrared image is converted according to the following linear relationship:

[0069] T = a × G + b;

[0070] Where T is the actual temperature; G is the grayscale value in the infrared image; a and b are coefficients obtained by device calibration.

[0071] For ultrasonic data:

[0072] Ultrasonic data preprocessing includes signal filtering, envelope extraction, and signature waveform analysis. Filtering the echo signal removes high-frequency noise and enhances the signal's signature. Envelope extraction is used to extract the ultrasonic signal's envelope, which is crucial for identifying cracks, corrosion, and other surface defects. Signature waveform analysis helps estimate the specific location and size of defects.

[0073] For X-ray data:

[0074] X-ray image preprocessing includes image enhancement, edge detection, and contrast adjustment. Image enhancement improves overall image quality, making details clearer. Edge detection methods, such as the Sobel operator or Canny edge detection, can effectively identify structural changes in images and help highlight defects within materials. Contrast adjustment is used to improve image clarity, especially in low-contrast areas.

[0075] Data fusion process

[0076] After the aforementioned preprocessing, the image data, thermal infrared data, ultrasound data, and X-ray data are uniformly normalized. This process unifies the scale and range of the different modal data, enabling them to be weighted and fused under the same standard. In some embodiments, the normalization process includes linear scaling or Z-score normalization to ensure balanced weighting of different data sources.

[0077] Specifically, in the fusion module, the fusion of various types of data is usually performed through weighted summation. In this process, each data type is assigned a weight coefficient to reflect its importance in the entire fusion feature. The fusion process is performed using the following formula:

[0078]

[0079] Among them, F represents the fused feature vector; x i is the feature representation of the i-th type of detection data; n represents the total number of input features; w i For the corresponding x i The weight coefficient of , and satisfy:

[0080]

[0081] This weighted fusion method can flexibly adjust the contribution ratio of each type of data in the final fusion features according to the characteristics of different data types.

[0082] Output of fused features

[0083] After preprocessing and weighted fusion, the resulting fused features contain comprehensive information from various sensors. These features are used in subsequent defect identification and risk assessment processes, providing more accurate and comprehensive defect detection results.

[0084] In some embodiments, the fused features can be further passed to other modules, such as the defect recognition module, for deep learning training and reasoning. The fused features not only enhance information collaboration between different data sources, but also improve the robustness and reliability of the entire system in different environments.

[0085] Through the above methods, the fusion module effectively integrates multidimensional data from different modalities into a unified feature representation. This process not only improves data availability but also provides a reliable guarantee for defect detection accuracy. During implementation, the fusion module can also flexibly adjust the weight coefficients based on actual application requirements to cope with different defect detection tasks.

[0086] In general, the fusion module of the present invention provides an innovative technical solution in multimodal data processing and feature extraction, and significantly improves the overall performance of the defect detection system through precise preprocessing and reasonable weighted fusion.

[0087] The defect recognition module identifies defects on the aircraft's exterior based on the defect recognition model and fusion features, and outputs the recognition results.

[0088] In this embodiment, the defect recognition module, based on an advanced defect recognition model, identifies defects on the aircraft's exterior by analyzing preprocessed fusion features. This module utilizes deep learning technology to extract and analyze key features from multimodal data, enabling precise identification and location of defect types and locations. The module's output is a comprehensive identification of aircraft defects, including defect type and location, providing crucial insights for subsequent assessment and repair.

[0089] Structure and working principle of defect recognition model

[0090] The defect sample library, serving as the foundation for the defect recognition model, stores a large number of actual defect samples. These samples cover defects occurring under different flight environments, temperature and humidity conditions, and include defect types for fuselage materials and various structural components. These structural components include wings, fuselage, tail, and engine, while defect types include cracks, corrosion, deformation, and other possible defect types. Based on these samples, a defect recognition model was established, enabling it to accurately identify external defects on aircraft.

[0091] In some embodiments, the defect recognition model includes the following core functional units:

[0092] Defect Detection Unit: This unit uses a deep learning model and a defect recognition model to analyze the fusion features output by the aforementioned fusion module, identifying and outputting defect signatures for the aircraft. These signatures include defect location and defect type. Specifically, defect locations can include wings, fuselage, tail, and engine, while defect types include cracks, dents, corrosion, and deformation.

[0093] In one possible implementation, the defect determination unit uses a convolutional neural network (CNN) to perform deep learning analysis on the fused features, extracting features related to the defect location and type. This process not only efficiently identifies defects but also accurately locates their location.

[0094] Specifically, the defect determination unit utilizes a deep learning model to analyze and identify defect features on the aircraft's exterior based on preprocessed and fused feature data. The unit's output includes the defect's location (spatial coordinates) and type (cracks, corrosion, deformation, etc.). By analyzing the fused features, the deep learning model accurately identifies different types of defects and locates their specific locations.

[0095] To achieve this function, the defect determination unit uses a convolutional neural network (CNN) model, which processes the input fusion features through the following formula:

[0096] Convolutional neural network processing

[0097] In deep learning models, input features (fused features) are extracted and transformed through multiple layers of convolutional layers, activation functions, pooling layers, and other structures. Assume that the feature map after convolution processing can be expressed as:

[0098] F i =f(W i *X+b i );

[0099] Among them, F i is the feature map after convolution, indicating the features after the i-th layer of convolution;

[0100] W i is the convolution kernel (weight matrix) of the i-th layer;

[0101] b i is the bias term of the i-th layer;

[0102] * indicates a convolution operation.

[0103] f(.) is the activation function. Common activation functions include ReLU (Rectified-Linear-Unit), whose formula is:

[0104] f(x)=max(0,x);

[0105] Feature map fusion and fully connected layer

[0106] The convolutional neural network extracts features through multiple convolutional layers, and then reduces the dimension of the extracted feature map through the pooling layer to obtain a fused feature map. The fused feature map is then sent to the fully connected layer for feature fusion. Assuming that the output of the fully connected layer is Y, the formula is:

[0107] Y=σ(W f ·F fusion +b f );

[0108] Among them, Wf is the weight matrix of the fully connected layer; b f is the bias term of the fully connected layer, F fusion is the fused feature map obtained after multi-layer convolution and pooling operations; σ(.) is the activation function (usually Sigmoid or Softmax function, depending on the output task)

[0109] Defect type identification and regression positioning

[0110] The defect type classification and location regression tasks are usually assigned to two output branches. Assume that the defect type classification task uses the Softmax function and the defect location regression task uses the L2 loss for optimization.

[0111] Classification loss L of defect type class (Cross Entropy Loss):

[0112]

[0113] Where C is the number of defect types (such as cracks, corrosion, deformation, etc.); y c is the indicator function of the true label (1 means it belongs to the category, 0 means it does not belong); is the predicted probability of belonging to category C.

[0114] The regression loss L of the defect location reg (Smooth L1 loss):

[0115]

[0116] in, represents the predicted defect location; p i is the actual defect position; Smooth1 is the smooth L1 loss function, the formula is as follows:

[0117]

[0118] Comprehensive loss function

[0119] The final total loss function is the weighted sum of classification loss, regression loss and regularization term, and the goal is to optimize both classification accuracy and position regression accuracy. Specifically expressed as:

[0120] L=λ a ·L class +λ b ·L reg +L reg ;

[0121] Where L represents the total loss function (or objective function); λ a is the weight for controlling the classification loss;b To control the weight of regression loss; L class is the classification loss of defect type; L reg is the regression loss of the defect location.

[0122] Model output

[0123] After optimizing the deep learning model, the output will be two parts:

[0124] Defect type: The probability distribution of the defect type output by the Softmax layer. The category corresponding to the maximum probability is the defect type.

[0125] Defect location: The coordinate value output by the regression network indicates the specific location of the defect in the aircraft image.

[0126] Therefore, the defect detection unit uses a convolutional neural network (CNN) to deeply analyze the preprocessed fusion features and optimizes them using a combination of cross-entropy loss and smoothed L1 regression loss. Ultimately, the model outputs the defect type and location. Using this formula and deep learning network structure, it can accurately identify and locate various defects on the aircraft's exterior.

[0127] To improve the accuracy and robustness of defect recognition, the defect recognition model further includes an update unit that trains and updates the model using machine learning and deep learning algorithms. These algorithms employ a multi-scale residual network that integrates a convolutional neural network (CNN) with an attention mechanism. This network structure is capable of extracting multi-level defect features from aircraft external images and effectively classifying and localizing defects.

[0128] Specifically, the optimization objective function is as follows:

[0129] L total =λ1·L cls +λ2·L loc +λ3·L att +λ4·∥θ∥ 2 ;

[0130] Among them, L total is the optimization objective function; L cls Represents the classification loss of defect type, using cross entropy loss function; L loc Represents the regression loss of the defect position, using a smooth loss function; L att Represents the difference loss between the feature weight map generated by the attention mechanism and the actual defect area; ∥θ∥ 2 is a model regularization term used to prevent overfitting; λ1, λ2, λ3 and λ4 are hyperparameters used to control the weight ratio of the loss.

[0131] Detailed description of the network structure

[0132] The multi-scale residual network structure further includes the following key layers:

[0133] Basic feature extraction layer: This layer uses multiple convolutional layers to extract local features in the image, such as edges and textures. The function of the convolutional layer is to extract low-level features from the image, which provide the basis for subsequent complex feature analysis.

[0134] Multi-scale residual layer: This layer extracts defect edge and texture features at different scales. Defects may vary in size and shape in different parts of an aircraft, so multi-scale feature extraction effectively captures defect information at different scales, enhancing the robustness of the network.

[0135] Channel attention and spatial attention modules: These modules improve the network's attention allocation ability by dynamically focusing on potential defective areas. The channel attention module emphasizes the importance of key channels, while the spatial attention module focuses on defective areas in the spatial dimension, avoiding wasting computational resources on irrelevant areas.

[0136] Dual-branch output layer: This layer has two output branches, one for classifying the defect type and the other for the defect location. The defect type branch uses a softmax activation function to output the probability of each class, while the defect location branch uses regression to output the coordinates of the defect in the image.

[0137] Training and updating of defect recognition models

[0138] The update unit of the defect recognition model trains the model by combining machine learning and deep learning algorithms, so that the model can gradually improve the recognition accuracy and generalization ability based on the continuously accumulated defect sample data.

[0139] Typically, the update unit incrementally trains the model based on new defect samples, continuously optimizing the loss function to improve the overall system's defect recognition capabilities. During each update, the model adjusts weights and parameters based on the latest data to ensure recognition accuracy under varying environmental conditions.

[0140] Through the aforementioned deep learning methods, the defect recognition module achieves high-precision defect detection and location. This module not only identifies common defect types (such as cracks, corrosion, and deformation), but also accurately identifies defects in various locations (such as wings, fuselage, tail, and engine) even in complex flight environments. This recognition capability provides a strong guarantee for aircraft safety.

[0141] In some embodiments, the defect identification module can further improve the identification accuracy by integrating more flight data (such as flight speed, flight altitude, etc.) and environmental data (such as temperature, humidity, etc.), so that the system can adapt to more diversified application scenarios.

[0142] The spatial positioning module is used to map the recognition results into spatial coordinates in the overall structure of the aircraft;

[0143] In this embodiment, the spatial positioning module is used to map the defect location and type information output by the defect identification module to the overall aircraft structure and visualize it in real time through the digital twin model. This module maps the identification results to spatial coordinates based on the aircraft's digital twin model, displaying the specific location of the defect. This function allows users to view defect locations in real time and obtain accurate defect information, providing reliable data support for aircraft maintenance and safety assessments.

[0144] In some embodiments, the spatial positioning module implements spatial mapping and display of defects through the following steps:

[0145] First, a digital twin model of the aircraft is created based on the external image data of the aircraft collected in the previous step. A digital twin model is a three-dimensional virtual model constructed based on the aircraft's geometry, structural characteristics, and material information. This model closely reproduces the aircraft's actual structure, including the spatial layout and physical properties of individual components such as the wings, fuselage, tail, and engine. Specifically, the digital twin model not only reflects the aircraft's external appearance but also includes the precise dimensions, structural form, and performance parameters of each component, providing detailed spatial information for subsequent defect identification, location, and analysis.

[0146] Next, based on the digital twin model, the spatial positioning module maps the recognition results to spatial coordinates. The defect determination unit uses a deep learning algorithm to analyze and fuse features and identify the type and location of aircraft defects (such as cracks, corrosion, and deformation). This defect information includes the specific location of the defect, the type of defect, and the possible impact range. After passing through the spatial positioning module, this information is accurately mapped to the corresponding parts of the digital twin model.

[0147] Specifically, the defect's spatial coordinate mapping process involves associating the image coordinates of the defect location with the 3D coordinate system in the digital twin model. This process can be performed as follows: First, image processing techniques are used to match the defect location coordinates in the aircraft's external image with the 3D coordinate system in the digital twin model. Typically, this process requires geometric transformations, image registration techniques, and spatial calibration based on sensor data. After this processing, the defect's spatial coordinates are accurately mapped to the corresponding part of the digital twin model.

[0148] In some embodiments, the digital twin model can display not only the location of the defect but also the type and status of the defect. Specifically, the corresponding component in the digital twin model will use different colors, markers, or highlights to display the defect area depending on the defect type. For crack-type defects, red may be used; for corrosion-type defects, yellow or green gradients may be used. This display method can help personnel clearly identify the defect type and severity.

[0149] As an option, the digital twin model can also provide repair or assessment recommendations based on the actual defect. For example, for a severe crack defect, the system may automatically recommend a repair plan or inspection requirements based on the aircraft's historical data and maintenance manual. This feature can further enhance the intelligence and automation of aircraft maintenance.

[0150] Specifically, in certain embodiments, the spatial positioning module updates defect information in the digital twin model in real time, allowing aircraft maintenance personnel to view the defect status of various parts in real time. This real-time visualization capability greatly improves the efficiency of defect detection and maintenance, thereby further ensuring aircraft safety.

[0151] In some embodiments, the digital twin model can also be integrated with other systems (such as flight data recorders and environmental monitoring systems) to further enhance the system's intelligence and collaborative capabilities. By integrating external data sources, the digital twin model can be dynamically updated to promptly reflect changes in the aircraft's status and defects under different environmental conditions.

[0152] Overall, the spatial positioning module, based on the digital twin model, visualizes defect information through precise spatial coordinate mapping. This not only provides real-time display of defect location and type, but also provides strong support for subsequent aircraft inspection, assessment, and repair. The implementation of this module significantly improves the accuracy and efficiency of defect identification and repair, further enhancing aircraft safety and maintenance management.

[0153] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The all-weather aircraft external defect intelligent detection system based on digital twin is characterized by: include: A multimodal data acquisition module, configured to acquire multimodal data of the aircraft's exterior, including external image data, ultrasonic data, X-ray data, and thermal infrared data of the aircraft; The fusion module is used to preprocess multimodal data to form a unified fusion feature; The defect recognition module identifies defects on the aircraft's exterior based on the defect recognition model and fusion features, and outputs the recognition results. The spatial positioning module is used to map the recognition results into spatial coordinates in the overall structure of the aircraft.

2. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 1 is characterized in that: The multimodal data acquisition module includes: UAV unit: used to collect external image data of the aircraft; Robotic arm unit: used to collect ultrasonic, X-ray and thermal infrared data from the outside of the aircraft.

3. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 1, characterized in that: The pretreatment includes: For image data: perform image grayscale histogram equalization, image denoising, enhancement and feature extraction; For thermal infrared data: perform infrared temperature linear conversion; For ultrasonic data: perform signal filtering, envelope extraction, and characteristic waveform analysis to identify echo features and estimate defect location and size; For X-ray data: perform image enhancement, edge detection, and contrast adjustment to highlight abnormal areas of the material's internal structure and extract high-density abnormal features.

4. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 3 is characterized in that: The image data, thermal infrared data, ultrasonic data and X-ray data are pre-processed and then normalized and weighted fused in sequence to generate fusion features.

5. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 1, characterized in that: The spatial positioning module is based on the digital twin model of the aircraft, maps the recognition results into spatial coordinates in the overall structure of the aircraft, and displays the defect location on the digital twin model, providing real-time visual information of the defect location.

6. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 1, characterized in that: The defect recognition model includes: Defect sample library: This stores samples of actual defects, including those generated under different flight environments, temperature and humidity conditions, as well as defect types of fuselage materials and fuselage structural components. The fuselage structural components include wings, fuselage, tail, and engine. Defect types include cracks, corrosion, and deformation. A defect recognition model is established based on the defect types. Defect judgment functional unit: Based on the deep learning model and the defect recognition model, the fusion feature is analyzed, and the defect features are identified and output. The defect features include the defect location and defect type of the aircraft.

7. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 6, characterized in that: The defect locations include: wings, fuselage, tail and engine; the defect types include: cracks, dents, corrosion and deformation.

8. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 6, characterized in that: The defect recognition model further includes an updating unit, which uses machine learning and deep learning algorithms to train and update the defect recognition model.

9. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 8, characterized in that: The machine learning and deep learning algorithms include a multi-scale residual network that integrates a convolutional neural network with an attention mechanism. The network is used to extract multi-level defect features from aircraft external images and perform classification and localization tasks. The optimization objective function is as follows: L total =λ1·L cls +λ2·L loc +λ3·L att +λ4·∥θ∥ 2 ; Among them, L total is the optimization objective function; cls Indicates the classification loss of defect type, using cross entropy loss function; L loc Represents the regression loss of the defect position, using a smooth loss function; L att Represents the difference loss between the feature weight map generated by the attention mechanism and the actual defect area; ∥θ∥ 2 is a model regularization term used to prevent overfitting; λ1, λ2, λ3 and λ4 are hyperparameters used to control the weight ratio of the loss.

10. The all-weather aircraft external defect intelligent detection system based on digital twin according to claim 9, characterized in that: The network structure further comprises: The basic feature extraction layer uses multiple convolutional layers to obtain local image features; Multi-scale residual layer, used to extract defect edge and texture features at different scales; Channel attention and spatial attention modules dynamically focus on potential defect areas; The dual-branch output layer outputs the defect type classification results and location coordinates respectively.

Images