Cross-modal aircraft fuselage surface fault detection method based on multi-task learning

Abstract

The invention relates to the technical field of aircraft detection, in particular to a cross-modal aircraft fuselage surface fault detection method based on multi-task learning, which comprises the following steps: collecting a visible light image and an infrared image of an aircraft fuselage surface and carrying out image preprocessing; feature extraction is carried out for target identification, and infrared and visible light image fusion is realized; carrying out fault detection on the extracted features by adopting a multi-task learning framework; parameters of the model are continuously adjusted, the trained model is applied to actual aircraft fuselage surface fault detection, and when a new image is input, the model can quickly provide accurate decision support; visible light and infrared images are creatively fused, and an efficient and accurate aircraft fuselage surface fault detection model is constructed by using a multi-task learning strategy; an image fusion algorithm, a feature extraction method and a task division and cooperation mode in a multi-task learning framework are key technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft detection, and specifically provides a cross-modal aircraft fuselage surface fault detection method based on multi-task learning. Background Art

[0002] As an important tool for modern transportation, the safety of an aircraft is of crucial importance. The condition of the aircraft fuselage surface directly affects flight safety. Traditional aircraft fuselage surface fault detection mainly relies on manual visual inspection. This method is not only inefficient but also easily affected by the experience and subjective factors of the inspectors, resulting in inaccurate detection results. In addition, for some minor and imperceptible faults, such as minor scratches, early corrosion, and unobvious pits, manual detection is prone to omission.

[0003] With the development of technology, some detection methods based on single-modal images have been proposed, such as detecting only using visible light images. However, visible light images have limitations in some cases. For example, in low light or complex backgrounds, fault features may not be obvious. There are also some methods that rely solely on infrared images for detection, but infrared images may not be sensitive enough to certain specific types of faults.

[0004] At the same time, existing detection technologies perform poorly in dealing with faults of different sizes and categories. For larger-sized faults, their boundaries and details may not be accurately identified due to insufficient image resolution; for smaller-sized faults, they may be ignored due to weak signal intensity. For different categories of faults, such as scratches, corrosion, and pits, due to their large feature differences, existing detection methods are often not universal, and specialized detection algorithms need to be developed for each type of fault, which increases the complexity and cost of detection.

[0005] Traditional aircraft fuselage surface fault detection methods have many disadvantages. To overcome these problems, the present invention proposes a cross-modal aircraft fuselage surface fault detection method based on multi-task learning. Summary of the Invention

[0006] The purpose of the present invention is to provide a cross-modal aircraft fuselage surface fault detection method based on multi-task learning to solve the problems raised in the above background art.

[0007] To solve the above technical problems, the present invention provides the following technical solutions:

[0008] A cross-modal aircraft fuselage surface fault detection method based on multi-task learning, the method includes:

[0009] S100. During the maintenance and inspection of an aircraft, use specialized image acquisition equipment to obtain visible light images and infrared images of the aircraft fuselage surface; and perform preprocessing operations such as image denoising, enhancement, cropping, and normalization on the acquired visible light images and infrared images;

[0010] S200. Use a fusion algorithm double-layer optimization model to extract features for target recognition, and construct a generative adversarial network based on a dual discriminator to achieve infrared and visible light image fusion;

[0011] S300. Construct multiple sub-task models, and use a multi-task learning framework to detect faults in the extracted features;

[0012] S400. During the model training process, use the labeled aircraft fuselage surface fault image data for supervised learning, continuously adjust the model parameters, and apply the trained model to the actual aircraft fuselage surface fault detection. When new images are input, the model can quickly give information about the location, type, and severity of the faults, providing accurate decision-making support for maintenance personnel.

[0013] Preferably, S200 includes:

[0014] S201. Use a double-layer optimization model to analyze the optimal solutions for image fusion and subsequent target detection;

[0015] S202. Construct a generative adversarial network based on a dual discriminator to achieve infrared and visible light image fusion, and learn the inherent and common features during the fusion process;

[0016] S203. Derive a cooperative training strategy to obtain the best network parameters and output accurate detection results.

[0017] Preferably, S201 includes:

[0018] A1. Assume that the infrared, visible light, and fused images are all grayscale images with the same picture size of m×n, represented as column vectors x, y respectively, and Then the fusion algorithm double-layer optimization model is:

[0019]

[0020] Among them, Λ d represents the loss function for target detection, Ψ represents a common neural network for target detection, ω d represents the learnable parameters of the target detection network, f(·) represents an energy-based fidelity term to achieve feature retention of the input image, which includes the fused image u, the source images x and y, while g T (·) and g D(·) represent the feasibility constraints on the intrinsic mode features defined based on infrared and visible light images respectively;

[0021] A2. Introduce an infrared and visible light image fusion network Φ to solve this energy function, and convert the double-layer optimization model into a single-layer optimization model:

[0022]

[0023] Among them, ω f represents the learnable parameters of the fusion network Φ.

[0024] Preferably, the network structure of the generative adversarial network in S202 includes: a generator, a target discriminator D T and a detail discriminator D D ;

[0025] The generator includes two parts: a feature extraction part and a feature aggregation part; the feature extraction part uses 5 dense connection blocks to extract the common features between different image modalities; the feature aggregation part uses three cascaded convolutional layers to fully aggregate the features; among them, each convolutional layer consists of a convolutional operation, a normalization layer, and a ReLU activation function, and the fused image u generated by the generator has the same size as the source image;

[0026] The target discriminator D T and the detail discriminator D D share the same network structure and use four convolutional layers and a fully connected layer.

[0027] Preferably, S202 includes:

[0028] B1. The goal of the generator is to generate a fused image whose overall structural information retains a pixel intensity distribution similar to that of the source image. Therefore, when training the generator, the structural similarity metric is selected as the loss function:

[0029]

[0030] Among them, Λ SSIM represents the structural similarity loss;

[0031] B2. The target discriminator D T is used to discriminate and distinguish the foreground object of the fusion result from the foreground salient object of the infrared image; the detail discriminator D D is used to discriminate and distinguish the background texture details of the fusion result from the visible light background texture details;

[0032] First, use the pre-trained saliency detection network to calculate the saliency map m of the target from the infrared image. Under the constraint of m, the two discriminators can better confront and discriminate the foreground target and background texture details in their respective salient regions. Then, the adversarial loss function Λ of the discriminator is defined as:

[0033]

[0034]

[0035] where P = x ⊙ m and ⊙ represents the per-pixel multiplication operation, denotes extracting gradient information using the Laplacian operator; the point multiplication of the saliency map mask / inverse mask and the source image pair aims to distinguish the target from the foreground / background;

[0036] Next, the adversarial loss functions in the target discriminator and background discriminator are calculated in the optimal transport manner to discriminate the foreground target and background texture details in the fused image from the input image, defined as:

[0037]

[0038] where denotes a sample space similar to ; k and p both represent weight coefficients, which are set to 2 and 6 respectively;

[0039] B3. Combine the loss functions of the generator and the two discriminators respectively to obtain the generation fusion regularizer Λ f , to achieve the learning of inherent and common features:

[0040]

[0041] where α and β represent balance parameters.

[0042] Preferably, S203 includes:

[0043] The double-layer optimization naturally derives a cooperative training strategy to obtain the optimal network parameters ω d =(ω d ,ω f ), then introduce a fusion regularization term Λ f , and transform the detection optimization process under the fusion constraint into a mutual optimization process:

[0044]

[0045] s.t. u * =Φ(x,y;ω f );

[0046] Among them, λ is a trade-off parameter;

[0047] For the learnable parameter ω in the detection network d and the learnable parameter ω in the fusion network f the gradient update method is as follows:

[0048]

[0049] Then it can be obtained that the gradient update of the detection loss with respect to the detection network parameters and the gradient update with respect to the fusion parameters are both backpropagated by the chain rule of differentiation; at this time, accurate detection results are output according to the optimal network parameters.

[0050] Preferably, S300 includes:

[0051] S301. After outputting the detection result and performing feature extraction, a multi-task learning framework is used to fuse and analyze the extracted features;

[0052] Construct multiple sub-task models, share the underlying feature representation, and at the same time have their own specific high-level features and classifiers, which are used to achieve accurate detection and classification of faults of different sizes and types;

[0053] S302. Introduce an attention mechanism, and dynamically adjust the attention to different regions and features, so that the model can pay more attention to the important information related to faults and ignore irrelevant or interfering information.

[0054] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps in the above-mentioned cross-modal aircraft fuselage surface fault detection method based on multi-task learning are implemented.

[0055] A computer device, including a memory, a processor, and a computer program stored on the memory and running on the processor, and when the processor executes the program, the steps in the above-mentioned cross-modal aircraft fuselage surface fault detection method based on multi-task learning are implemented.

[0056] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0057] The present invention brings many remarkable beneficial effects. First of all, by fusing visible light and infrared images, the accuracy of fault detection is greatly improved. It can comprehensively capture the characteristics of different types of faults. Whether it is a tiny scratch, early corrosion or an inconspicuous pit, it can be accurately identified, reducing the situation of missed detection and false detection.

[0058] Secondly, the multi-task learning strategy makes the fault detection for different sizes and categories of faults universal. Multi-task learning can handle multiple related but different tasks simultaneously. For example, in the fault detection on the surface of an aircraft fuselage, large-size faults, small-size faults, and different types of faults (such as scratches, corrosion, and pits) can be detected and classified at the same time. This approach better copes with the complex and diverse fault situations in actual detection.

[0059] Furthermore, it reduces the workload and human errors. The automated detection process avoids the subjectivity and fatigue of manual detection, improves the detection efficiency and reliability, and reduces the misjudgment caused by human factors.

[0060] In addition, the present invention can detect potential faults in a timely manner, which helps to take maintenance measures in advance, avoid the further deterioration of faults, and thus reduces the maintenance cost and flight risk.

[0061] Finally, the advanced algorithms and models can quickly process a large amount of image data, adapt to different detection scenarios and conditions, and improve the flexibility and adaptability of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, but do not constitute a limitation to the present invention. In the drawings:

[0063] Figure 1 is a schematic diagram of common fault types (paint peeling, impact, corrosion, pit) of different materials of the aircraft fuselage of the present invention;

[0064] Figure 2 is a flowchart of a cross-modal aircraft fuselage surface fault detection method based on multi-task learning of the present invention;

[0065] Figure 3 is a framework diagram of an RGB-T image fusion method for aircraft structure fault detection of the present invention;

[0066] Figure 4 is a schematic structural diagram of a generative adversarial network in the fusion algorithm of the present invention;

[0067] Figure 5 is an architecture diagram of a hierarchical multi-task learning model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0069] Please refer to Figures 1-5 , the present invention provides a technical solution:

[0070] Embodiment 1:

[0071] A cross-modal aircraft fuselage surface fault detection method based on multi-task learning, the method includes:

[0072] S100. During the maintenance and detection of the aircraft, use special image acquisition devices to obtain visible light images and infrared images of the aircraft fuselage surface. These image acquisition devices have the characteristics of high resolution and high precision, and can capture minute fault details;

[0073] Preprocess the collected visible light and infrared images: including operations such as image denoising, enhancement, cropping, and normalization, etc., to improve the quality and consistency of the images and prepare for subsequent processing;

[0074] S200. Use a fusion algorithm double-layer optimization model to perform feature extraction for target recognition, and construct a generative adversarial network based on a dual discriminator to achieve infrared and visible light image fusion;

[0075] Preferably, S200 includes:

[0076] S201. Use the double-layer optimization model to analyze the optimal solutions for image fusion and subsequent target detection;

[0077] S202. Construct a generative adversarial network based on a dual discriminator to achieve infrared and visible light image fusion, and learn the inherent and common features during the fusion process;

[0078] S203. Derive a cooperative training strategy to obtain the best network parameters and output accurate detection results.

[0079] Preferably, S201 includes:

[0080] A1. Assume that the infrared, visible light, and fused images are all grayscale images, with the same picture size of m×n, and are respectively represented as column vectors x, y, and The framework of the fusion algorithm for target detection described in the present invention is as Figure 3As shown in the figure, the upper-layer sub-problem objective is to achieve the optimal solution of the target detection task on the fused image, and the lower-layer sub-problem aims to generate a fused image for the detection task for the upper-layer sub-problem;

[0081] Then the double-layer optimization model of the fusion algorithm is:

[0082]

[0083] Among them, Λ d represents the loss function of target detection, Ψ represents a common neural network for target detection, ω d represents the learnable parameters of the target detection network, f(·) represents an energy-based fidelity term to achieve feature retention of the input image, which includes the fused image u, the source images x and y, while g T (·) and g D (·) respectively represent the feasibility constraints on the intrinsic mode features defined based on the infrared and visible light images;

[0084] A2. Introduce an infrared and visible light image fusion network Φ to solve this energy function, and convert the double-layer optimization model into a single-layer optimization model:

[0085]

[0086] Among them, ω f represents the learnable parameters of the fusion network Φ;

[0087] The image fusion network designed in the present invention first seeks the commonalities of the two modalities and further learns from the differences containing the complementary features of these two modalities.

[0088] Preferably, the network structure of the generative adversarial network in S202 includes: a generator, a target discriminator D T and a detail discriminator D D ;

[0089] The generator includes two parts: a feature extraction part and a feature aggregation part; the feature extraction part uses 5 dense connection blocks to extract the common features between different image modalities; the feature aggregation part uses three cascaded convolutional layers to fully aggregate the features; among them, each convolutional layer consists of a convolutional operation, a normalization layer, and a ReLU activation function, and the fused image u generated by the generator has the same size as the source images;

[0090] The target discriminator D T and the detail discriminator D D share the same network structure, and use four convolutional layers and one fully connected layer.

[0091] Preferably, S202 includes:

[0092] B1. The goal of the generator is to generate a fused image whose overall structural information retains a pixel intensity distribution similar to that of the source image. Therefore, when training the generator, the structural similarity metric is selected as the loss function:

[0093]

[0094] where Λ SSIM represents the structural similarity loss;

[0095] The generator G aims to learn the common features of the two modalities to generate an informative fused image that can pass the discrimination of the two discriminators;

[0096] B2. The target discriminator D T is used to discriminate and distinguish the foreground object in the fusion result from the foreground salient object in the infrared image; the detail discriminator D D is used to discriminate and distinguish the background texture details in the fusion result from the visible light background texture details;

[0097] First, use the pre-trained saliency detection network to calculate the saliency map m of the object from the infrared image. Under the constraint of m, the two discriminators can better perform adversarial discrimination on the foreground object and background texture details in their respective salient regions. Then, the adversarial loss function Λ of the discriminator is defined as:

[0098]

[0099] where P = x ⊙ m and ⊙ represents the per-pixel multiplication operation, represents using the Laplacian operator to extract gradient information; Multiplying the saliency map mask / inverse mask and the source image pair pixel by pixel aims to distinguish the object from the foreground / background;

[0100] Next, the adversarial loss functions in the target discriminator and the background discriminator are calculated in the optimal transport manner to discriminate the foreground object and background texture details in the fused image from the input image, defined as:

[0101]

[0102] where represents the sample space similar to k and p both represent weight coefficients, which are set to 2 and 6 respectively;

[0103] The role of the target discriminator DT is to determine whether the intensity of the target information in the foreground of the infrared image is consistent with that of the target in the foreground of the fused image generated by G; the role of the detail discriminator DD is to determine whether the distribution of the background texture details of the visible light image is consistent with that of the background texture details of the fused image.

[0104] The two discriminators perform corresponding discrimination on the image in different domain spaces. That is, the target discriminator DT performs discrimination on the foreground target in the pixel domain space of the image, while the detail discriminator DD performs discrimination on the background texture details in the gradient domain space of the image.

[0105] B3. Combine the loss functions of the generator and the two discriminators respectively to obtain the generation fusion regularizer Λ f , to achieve the learning of intrinsic and common features:

[0106]

[0107] Among them, α and β represent balance parameters.

[0108] Preferably, S203 includes:

[0109] The double-layer optimization naturally derives a cooperative training strategy to obtain the optimal network parameters ω d =(ω d , ω f ), then introduce a fusion regularization term Λ f , and transform the detection optimization process subject to fusion constraints into a mutual optimization process:

[0110]

[0111] s.t. u * =Φ(x, y; ω f );

[0112] Among them, λ is a compromise parameter;

[0113] Compared with designing a weighted regularizer, this regularizer can better balance the relationship between image fusion and target detection. Figure 3 The right lower part shows the gradient propagation process of the collaborative training of the fusion and detection networks;

[0114] For the learnable parameters ω d in the detection network and the learnable parameters ω f in the fusion network, the gradient update methods are as follows:

[0115]

[0116] Then, it is obtained that the gradient updates of the detection loss with respect to the detection network parameters and the gradient updates with respect to the fusion parameters are both backpropagated using the chain rule of differentiation; at this time, accurate detection results are output according to the optimal network parameters.

[0117] S300. Construct multiple sub-task models and use the framework of multi-task learning to perform fault detection on the extracted features;

[0118] Preferably, S300 includes:

[0119] S301. After outputting the detection results and performing feature extraction, use the framework of multi-task learning to fuse and analyze the extracted features;

[0120] Construct multiple sub-task models, such as a detection model for large-size faults, a detection model for small-size faults, and a classification model for different types of faults (such as scratches, corrosion, pits). These sub-task models share the underlying feature representations and at the same time have their own specific high-level features and classifiers, which are used to achieve accurate detection and classification of different sizes and types of faults;

[0121] S302. In order to improve the accuracy and robustness of detection, introduce an attention mechanism, dynamically adjust the attention to different regions and features, so that the model can pay more attention to the important information related to faults and ignore irrelevant or interfering information.

[0122] S400. During the model training process, use the labeled aircraft fuselage surface fault image data for supervised learning, continuously adjust the model parameters, and apply the trained model to the actual aircraft fuselage surface fault detection. When new images are input, the model can quickly give information about the location, type, and severity of the faults, providing accurate decision-making support for maintenance personnel.

[0123] The present invention fully integrates the advantages of visible light and infrared images. Visible light images can provide clear surface texture and detail information, while infrared images can reflect the temperature distribution of objects. The information contained in RGB images and thermal infrared images has a certain degree of complementarity and has better detection ability for potential faults such as early corrosion. By combining these two modalities of images for salient object detection, the fault features on the aircraft fuselage surface can be captured more comprehensively and accurately.

[0124] Secondly, for faults of different sizes and categories, the present invention adopts a multi-task learning strategy. By constructing multiple sub-task models, learning and training are respectively carried out for large-size, small-size, and different types of faults, so that the model can adaptively handle various situations and improve the generality and accuracy of detection.

[0125] In addition, to reduce the workload and minimize human errors, the present invention introduces an automated feature extraction and analysis algorithm. Instead of relying on manual and cumbersome analysis and judgment of images, the machine automatically completes the extraction and identification of fault features, greatly improving the detection efficiency and reliability.

[0126] Finally, during the model training process, a large number of real fault image data of the aircraft fuselage surface are used, and combined with advanced deep learning technologies, the parameters of the model are continuously optimized to enable it to better adapt to various changes and challenges in the actual detection scenario.

[0127] Embodiment 2:

[0128] The computer-readable storage medium of this embodiment stores a computer program, and when the program is executed by a processor, it implements the steps in a cross-modal aircraft fuselage surface fault detection method based on multi-task learning in Embodiment 1.

[0129] The computer-readable storage medium of this embodiment can be an internal storage unit of the terminal, such as the hard disk or memory of the terminal; the computer-readable storage medium of this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash card, etc. equipped on the terminal; further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the terminal.

[0130] The computer-readable storage medium of this embodiment is used to store the computer program and other programs and data required by the terminal, and the computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.

[0131] Embodiment 3:

[0132] The computer device of this embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a cross-modal aircraft fuselage surface fault detection method based on multi-task learning in Embodiment 1.

[0133] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc.; the memory can include a read-only memory and a random access memory, and provides instructions and data to the processor. A part of the memory can also include a non-volatile random access memory. For example, the memory can also store information about the device type.

[0134] Those skilled in the art should understand that the content disclosed in the embodiments can be provided as a method, a system, or a computer program product. Therefore, the present solution can be implemented in the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present solution can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0135] The present solution is described with reference to the flowcharts and / or block diagrams of methods and computer program products according to the embodiments of the present solution. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 flow or multiple flows and / or block Figure 1 diagram or multiple blocks.

[0136] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 flow or multiple flows and / or block Figure 1 diagram or multiple blocks.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 flow or multiple flows and / or block Figure 1 diagram or multiple blocks.

[0138] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM), etc.

[0139] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning, characterized in that: The method comprises: S100, during the maintenance and inspection process of the aircraft, using a special image acquisition device to acquire a visible light image and an infrared image of the aircraft fuselage surface; and performing image denoising, enhancement, cropping and normalization preprocessing operations on the acquired visible light image and infrared image; S200, using the dual-layer optimization model of the fusion algorithm to extract features for target recognition, and constructing a generative adversarial network based on dual discriminators to achieve infrared and visible light image fusion; S300, construct multiple sub-task models, and use a multi-task learning framework to perform fault detection on the extracted features; S400. During the model training process, the labeled aircraft fuselage surface fault image data is used for supervised learning, the model parameters are continuously adjusted, and the trained model is applied to the actual aircraft fuselage surface fault detection. When a new image is input, the model can quickly give the location, type and severity information of the fault, providing accurate decision support for maintenance personnel.

2. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 1, characterized in that: The S200 includes: S201, analyzing the optimal solution for image fusion and subsequent target detection using a two-layer optimization model; S202, constructing a generative adversarial network based on a dual discriminator to achieve infrared and visible light image fusion, and learning inherent and common features in the fusion process; S203, deriving a cooperative training strategy to obtain optimal network parameters and output accurate detection results.

3. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 2, characterized in that: The S201 includes: A1. Assume that the infrared, visible light and fused images are all grayscale images with the same size of m×n, represented by column vectors x and y respectively, and Then the two-layer optimization model of the fusion algorithm is: Among them, Λ d represents the loss function of target detection, Ψ represents a commonly used neural network for target detection, ω d represents the learnable parameters of the target detection network, f(·) represents an energy-based fidelity term to preserve the features of the input image, which contains the fused image u, the source images x and y, and g T (·) and g D (·) represents the feasibility constraints on the intrinsic modal features defined based on infrared and visible light images, respectively; A2. Introduce an infrared and visible light image fusion network Φ to solve the energy function and convert the two-layer optimization model into a single-layer optimization model: Among them, ω f represents the learnable parameters of the fusion network Φ.

4. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 2, characterized in that: The network structure of the generated adversarial network in S202 includes: a generator, a target discriminator D T and detail discriminator D D ; The generator includes two parts: a feature extraction part and a feature aggregation part; the feature extraction part uses a 5-layer dense connection block to extract the common features between different image modalities; the feature aggregation part uses three cascaded convolutional layers to fully aggregate the features; each convolutional layer consists of a convolution operation, a normalization layer and a ReLU activation function, and the fused image u generated by the generator has the same size as the source image; The target discriminator D T and detail discriminator D D They share the same network structure, using four convolutional layers and one fully connected layer.

5. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 4, characterized in that: The S202 includes: B1. The goal of the generator is to generate a fused image whose overall structural information retains a pixel intensity distribution similar to that of the source image. When training the generator, the structural similarity metric is used as the loss function: Among them, Λ SSIM represents the structural similarity loss; B2, the target discriminator D T It is used to distinguish and differentiate the foreground target of the fusion result from the foreground salient target of the infrared image; the detail discriminator D D Used to distinguish and differentiate the background texture details of the fusion result from the visible light background texture details; First, the pre-trained saliency detection network is used to calculate the saliency map m of the target from the infrared image. Under the constraint of m, the two discriminators can better confront and discriminate the foreground target and background texture details in their respective salient regions. The adversarial loss function Λ of the discriminator is defined as: Where P = x⊙m and ⊙ represents pixel-by-pixel multiplication operation, Indicates the use of Laplacian operator to extract gradient information; Next, the adversarial loss functions in the target discriminator and the background discriminator are calculated in an optimal transmission manner to discriminate the foreground target and background texture details in the fused image from the input image, which is defined as: in, Representation and For similar sample spaces, k and p both represent weight coefficients, which are set to 2 and 6, respectively; B3. Combine the loss functions of the generator and the two discriminators to obtain the generated fusion regularizer Λ f , to achieve learning of inherent and shared features: Among them, α and β represent equilibrium parameters.

6. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 2, characterized in that: The S203 includes: The two-level optimization naturally leads to a cooperative training strategy to obtain the optimal network parameters ω d =(ω d ,ω f ), then a fusion regularization term Λ is introduced f , and transform the detection optimization process subject to fusion constraints into a mutual optimization process: s.t.u * =Φ(x,y;ω f ); Among them, λ is the trade-off parameter; For the learnable parameter ω in the detection network d and the learnable parameter ω in the fusion network f The gradient update method is as follows: It can be concluded that the gradient update of the detection loss relative to the detection network parameters and the gradient update relative to the fusion parameters are both back-propagated using the chain derivative rule; at this time, accurate detection results are output based on the optimal network parameters.

7. A cross-modal aircraft fuselage surface fault detection method based on multi-task learning as claimed in claim 1, characterized in that: The S300 includes: S301, after outputting the detection results and extracting features, a multi-task learning framework is used to fuse and analyze the extracted features; Build multiple subtask models that share the underlying feature representation while having their own specific high-level features and classifiers to accurately detect and classify faults of different sizes and types; S302. Introduce the attention mechanism to dynamically adjust the attention of different regions and features, so that the model can pay more attention to important information related to the fault and ignore irrelevant or interfering information.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a cross-modal aircraft fuselage surface fault detection method based on multi-task learning as described in any one of claims 1 to 7 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps in the cross-modal aircraft fuselage surface fault detection method based on multi-task learning as described in any one of claims 1-7 are implemented.

Images