Defect detection model training method and device

Abstract

The invention provides a defect detection model training method and device, and relates to the technical field of defect detection. The training method of the defect detection model comprises the steps that original aircraft panel images in multiple scene states in a preset assembly environment are acquired, and the states of at least one dimension in shooting time, shooting angles and assembly sites in different scene states are different; according to the original aircraft panel images in the multiple scenes, a sample data set is constructed, and the sample data set comprises multiple sample aircraft panel images marked with defect parts and defect description information corresponding to the sample aircraft panel images; according to the sample data set, training the initial defect detection model to obtain a defect detection model; wherein the defect detection model is used for processing the image of the to-be-detected aircraft panel to obtain a defect detection result of the to-be-detected aircraft panel. Based on the defect detection model, the defect detection effect on the aircraft panel can be improved, and the detection efficiency can also be improved.

Description

Technical Field

[0001] The present application relates to the technical field of defect detection, and in particular to a method and device for training a defect detection model. Background Art

[0002] With the continuous progress of the aviation industry and the frequent replacement of aircraft equipment, the demand for the quality and safety of aircraft structures and parts is also increasing. As an important component of the aircraft structure, the integrity of the aircraft panel directly affects flight safety. However, during the manufacturing and assembly process of the aircraft panel, defects such as scratches, cracks, and deformation may occur. These defects may cause damage to the aircraft structure and seriously affect flight safety. Therefore, defect detection of aircraft panels is crucial to ensure the safe flight of the aircraft.

[0003] In the related art, ultrasonic detection technology is used to detect defects in aircraft wall panels. Ultrasonic detection technology is limited by the accuracy and detection depth of the sensor, and is not effective in detecting defects in aircraft wall panels, and has low detection efficiency. Summary of the invention

[0004] The purpose of this application is to provide a method and device for training a defect detection model to address the above-mentioned deficiencies in the prior art, so as to solve the above-mentioned technical problems existing in the related technology.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for training a defect detection model, the method comprising:

[0007] Acquire original aircraft panel images in multiple scene states in a preset assembly environment, where the states of at least one dimension of shooting time, shooting angle, and assembly location are different in different scene states;

[0008] Constructing a sample data set according to the original aircraft panel images in the multiple scenes, wherein the sample data set includes: multiple sample aircraft panel images with defective parts marked, and defect description information corresponding to the sample aircraft panel images;

[0009] Training an initial defect detection model according to the sample data set to obtain the defect detection model;

[0010] The defect detection model is used to process the image of the aircraft panel to be inspected under the preset assembly environment to obtain the defect detection result of the aircraft panel to be inspected.

[0011] Optionally, the initial defect detection model includes: an initial backbone network module, an initial pyramid network module and an initial detection module; the initial defect detection model is trained according to the sample data set to obtain the defect detection model, including:

[0012] Using the initial backbone network module to process the sample aircraft siding image in the sample data set to obtain a first feature map;

[0013] Using the initial pyramid network module to process the first feature map to obtain a second feature map;

[0014] Outputting a sample detection result for the sample aircraft siding image using the initial detection module according to the second feature map;

[0015] According to the sample detection results and the defect description information corresponding to the sample aircraft panel image, the initial backbone network module, the initial pyramid network module and the initial detection module are trained to obtain the defect detection model including the target backbone network module, the target pyramid network module and the target detection model.

[0016] Optionally, there are multiple sample detection results; the initial backbone network module, the initial pyramid network module and the initial detection module are trained according to the sample detection results and the defect description information corresponding to the sample aircraft panel image to obtain the defect detection model including a target backbone network module, a target pyramid network module and a target detection model, including:

[0017] Determine a target sample test result from the plurality of sample test results;

[0018] Calculating a first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft siding image;

[0019] According to the first loss function corresponding to the target sample detection result, the initial backbone network module, the initial pyramid network module and the initial detection module are trained until a preset termination condition is met, thereby obtaining the defect detection model including the target backbone network module, the target pyramid network module and the target detection model.

[0020] Optionally, calculating a first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft siding image includes:

[0021] Using a first loss function, according to the detection defect position in the target sample detection result and the sample defect position in the defect description information, calculate a first regression loss function value;

[0022] Using a second loss function, according to the detection defect type in the target sample detection result and the sample defect type in the defect description information, calculate the first classification loss function value;

[0023] The first loss function is calculated according to the first regression loss function value and the first classification loss function value.

[0024] Optionally, the initial backbone network module includes: a plurality of convolution modules and a target convolution module connected in sequence; the target convolution module includes: a first convolution layer, a deep attention deformable convolution module and a second convolution layer connected in sequence;

[0025] The initial backbone network module is used to process the sample aircraft panel image in the sample data set to obtain a first feature map, including:

[0026] The sample aircraft panel image is processed using the multiple convolution modules to obtain a first sub-feature map output by a last preset first number of convolution modules among the multiple convolution modules; wherein the output of a previous convolution module among the multiple convolution modules is used as the output of a subsequent convolution module;

[0027] Using the first convolution layer to process the target first sub-feature map output by the last convolution module among the multiple convolution modules to obtain a first intermediate map;

[0028] Processing the first intermediate image using the deep attention deformable convolution module to obtain a second intermediate image, wherein the second intermediate image includes offset information;

[0029] Processing the second intermediate image using the second convolutional layer to obtain a second sub-feature image;

[0030] The first feature map includes: the first sub-feature map and the second sub-feature map.

[0031] Optionally, the deep attention deformable convolution module includes: a plurality of third convolution layers and a spatial attention module connected in sequence;

[0032] The step of processing the first intermediate image by using the deep attention deformable convolution module to obtain a second intermediate image includes:

[0033] Processing the first intermediate image using the multiple third convolutional layers to obtain a fourth intermediate image output by the last third convolutional layer among the multiple third convolutional layers; wherein the output of the previous third convolutional layer is used as the input of the next third convolutional layer;

[0034] The fourth intermediate image is processed using the spatial attention module to obtain the second intermediate image.

[0035] Optionally, determining the target sample detection result from the plurality of sample detection results comprises:

[0036] Calculating a plurality of second loss function values ​​corresponding to the plurality of sample detection results according to the sample detection results and the defect description information;

[0037] Sort the plurality of second loss function values ​​from small to large to obtain a sorting result;

[0038] According to the sorting result, the target sample detection result is determined from the multiple sample detection results.

[0039] Optionally, determining the target sample detection result from the plurality of sample detection results according to the sorting result includes:

[0040] Determine the target quantity according to the defect description information;

[0041] According to the sorting result, a target number of target sample detection results are selected from the multiple sample detection results.

[0042] Optionally, constructing a sample data set according to the original aircraft panel images in the multiple scenes includes:

[0043] Annotating each of the original aircraft wall panel images to obtain each annotated aircraft wall panel image;

[0044] Generating defect description information corresponding to each of the annotated aircraft panel images;

[0045] Performing an image enhancement operation on each of the labeled aircraft siding images to obtain a plurality of sample aircraft siding images;

[0046] The sample data set is constructed according to the plurality of sample aircraft panel images and the defect description information corresponding to each of the labeled aircraft panel images.

[0047] In a second aspect, an embodiment of the present application further provides a defect detection model training device, comprising: a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the defect detection model training method described in any one of the first aspects above is implemented.

[0048] The beneficial effects of the present application are as follows: the present application embodiment provides a method for training a defect detection model, the method may include: obtaining original aircraft siding images under multiple scene states in a preset assembly environment, the shooting time, shooting angle and assembly location under different scene states being different in at least one dimension; constructing a sample data set based on the original aircraft siding images under multiple scenes, the sample data set including: multiple sample aircraft siding images with defective parts annotated, and defect description information corresponding to the sample aircraft siding images; training an initial defect detection model based on the sample data set to obtain a defect detection model; wherein the defect detection model is used to process the image of the aircraft siding to be detected under the preset assembly environment to obtain a defect detection result of the aircraft siding to be detected. Model training is performed based on multiple sample aircraft siding images with defective parts annotated and defect description information to obtain a defect detection model; performing defect detection on the image of the aircraft siding to be detected based on the defect detection model obtained by the training can improve the defect detection effect of the aircraft siding and can also improve the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 1 ;

[0051] Figure 2 A schematic diagram of a network structure of an initial defect detection model provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 2 ;

[0053] Figure 4 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 3 ;

[0054] Figure 5 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 4 ;

[0055] Figure 6A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 5 ;

[0056] Figure 7 A schematic diagram of the network structure of a deep attention deformable convolution module provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 6 ;

[0058] Fig. 9 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 7 ;

[0059] Fig.10 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 8 ;

[0060] Fig.11 A schematic diagram of the structure of a defect detection model training device provided in an embodiment of the present application;

[0061] Fig.12 A schematic diagram of the structure of a defect detection model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0063] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0064] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0065] In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0067] An embodiment of the present application provides a method for training a defect detection model, which is applied to a training device for a defect detection model. The training device for the defect detection model may be a terminal device or a server. The terminal device may be any one of a computer, a laptop computer, a tablet computer, and a smart phone.

[0068] The following is an explanation of a method for training a defect detection model provided in an embodiment of the present application.

[0069] Figure 1 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method may include:

[0070] S101, obtaining original aircraft panel images in multiple scene states in a preset assembly environment.

[0071] Among them, the state of at least one dimension of shooting time, shooting angle and assembly location under different scene states is different.

[0072] It should be noted that during the manufacturing and assembly process of aircraft panels, defects such as scratches, cracks, and deformation may occur. Due to the complex environment of the aircraft assembly workshop, the original aircraft panel images in multiple scene states also have a relatively complex background, and some defect targets of the aircraft panel in the original aircraft panel images are relatively small. In this way, the defect detection model trained based on the sample data set constructed based on the original aircraft panel images can be applied to defect detection of images of aircraft panels to be detected with complex scenes and small defect parts.

[0073] In addition, the original aircraft panel images collected at different shooting times, different shooting angles, and different assembly locations contain more complex scenes. This allows the defect detection model trained based on the sample data set constructed based on the original aircraft panel images to be applicable to defect detection of images of aircraft panels to be inspected in a variety of complex scenes.

[0074] S102: construct a sample data set based on original aircraft panel images in multiple scenes.

[0075] The sample data set includes: a plurality of sample aircraft panel images with defective parts marked, and defect description information corresponding to the sample aircraft panel images.

[0076] In the embodiment of the present application, the multiple sample aircraft panel images with defective parts marked refer to selecting the defective parts of the sample aircraft panel in the sample aircraft panel images. The defect description information corresponding to the sample aircraft panel images includes: defect location information and defect type information corresponding to the sample aircraft panel images, and may also include the size of the sample aircraft panel images. The defect location information may be the defect location coordinates.

[0077] S103: Train the initial defect detection model according to the sample data set to obtain a defect detection model.

[0078] The defect detection model is used to process the image of the aircraft panel to be inspected under a preset assembly environment to obtain the defect detection result of the aircraft panel to be inspected.

[0079] In some embodiments, the initial defect detection model is trained according to the sample data set, and the model parameters in the initial defect detection model are updated until the preset termination condition is met to obtain the defect detection model. Wherein, when the number of iterations is greater than or equal to the preset number of iterations, it is determined that the preset termination condition is met, or when the preset loss function converges, it is determined that the preset termination condition is met.

[0080] It is worth noting that the defect detection result is the first detection result or the second detection result, wherein the first detection result is used to indicate that there are no defects in the aircraft wall panel to be detected, and the second detection result is used to indicate that there are defects in the aircraft wall panel to be detected, and the defect description information corresponding to the defect, and the defect description information may include the defect location and defect type of the aircraft wall panel to be detected. Of course, the second detection result can also mark and frame the defect location in the image of the aircraft wall panel to be detected.

[0081] In summary, the embodiment of the present application provides a method for training a defect detection model, which may include: obtaining original aircraft siding images under multiple scene states in a preset assembly environment, wherein the states of at least one dimension of shooting time, shooting angle, and assembly location under different scene states are different; constructing a sample data set based on the original aircraft siding images under multiple scenes, wherein the sample data set includes: multiple sample aircraft siding images with defective parts annotated, and defect description information corresponding to the sample aircraft siding images; training an initial defect detection model based on the sample data set to obtain a defect detection model; wherein the defect detection model is used to process the image of the aircraft siding to be detected under the preset assembly environment to obtain a defect detection result of the aircraft siding to be detected. Model training is performed based on multiple sample aircraft siding images with defective parts annotated and defect description information to obtain a defect detection model; performing defect detection on the image of the aircraft siding to be detected based on the defect detection model obtained by the training can improve the defect detection effect of the aircraft siding and can also improve the detection efficiency.

[0082] Optionally, Figure 2 A schematic diagram of a network structure of an initial defect detection model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the initial detection module may include: an initial backbone network module, an initial pyramid network module and an initial detection module; wherein the initial backbone network module is connected to the initial pyramid network module, and the initial pyramid network module is connected to the initial detection module.

[0083] Figure 3 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 2 ,like Figure 3 As shown, the process of training the initial defect detection model according to the sample data set in the above S103 to obtain the defect detection model may include:

[0084] S201. Using an initial backbone network module to process a sample aircraft siding image in a sample data set to obtain a first feature map.

[0085] Optionally, the initial backbone network module can use Resnet-50 (a deep learning model) as a basis.

[0086] S202: Use an initial pyramid network module to process the first feature map to obtain a second feature map.

[0087] S203 , using an initial detection module to output a sample detection result for the sample aircraft siding image according to the second feature map.

[0088] In some embodiments, a sample aircraft siding image in a sample data set is input into an initial backbone network module, the initial backbone network module processes the sample aircraft siding image, and the initial backbone network module outputs a first feature map; the first feature map is input into an initial pyramid network model, the initial pyramid network model is used to process the first feature map, and a second feature map output by the initial pyramid network model is obtained; the second feature map is input into an initial detection module, the initial detection module processes the second feature map, and a sample detection result of the sample aircraft siding image is obtained output by the initial detection module.

[0089] S204, according to the sample detection results and the defect description information corresponding to the sample aircraft siding image, the initial backbone network module, the initial pyramid network module and the initial detection module are trained to obtain a defect detection model including a target backbone network module, a target pyramid network module and a target detection model.

[0090] It should be noted that, according to the sample detection results and the defect description information corresponding to the sample aircraft siding image, the initial backbone network module, the initial pyramid network module and the initial detection module are trained, and the model parameters in the initial backbone network module, the initial pyramid network module and the initial detection module are updated until the preset termination conditions are met, and a defect detection model including a target backbone network module, a target pyramid network module and a target detection model is obtained.

[0091] Optionally, the number of sample test results is multiple; Figure 4 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, the process of training the initial backbone network module, the initial pyramid network module, and the initial detection module according to the sample detection results and the defect description information corresponding to the sample aircraft panel image in the above S204 to obtain a defect detection model including a target backbone network module, a target pyramid network module, and a target detection model may include:

[0092] S301. Determine a target sample detection result from multiple sample detection results.

[0093] Among them, a preset sample allocation strategy can be adopted to determine the target sample detection result from multiple sample detection results. Optionally, the preset sample allocation strategy can be called a simOTA sample allocation strategy. The target sample detection result can be a positive sample, and the remaining sample detection results except the target sample detection result in the multiple sample detection results are negative samples.

[0094] S302: Calculate a first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft siding image.

[0095] In some embodiments, a preset loss function formula is used to calculate a first loss function corresponding to the target sample detection result based on the target sample detection result and defect description information corresponding to the sample aircraft siding image.

[0096] S303. According to the first loss function corresponding to the target sample detection result, the initial backbone network module, the initial pyramid network module and the initial detection module are trained until the preset termination condition is met, and a defect detection model including the target backbone network module, the target pyramid network module and the target detection model is obtained.

[0097] Among them, according to the first loss function corresponding to the target sample detection result, the initial backbone network module, the initial pyramid network module and the initial detection module are trained, and the model parameters in the initial backbone network module, the initial pyramid network module and the initial detection module are updated until the preset termination condition is met, and a defect detection model including the target backbone network module, the target pyramid network module and the target detection model is obtained.

[0098] Optionally, Figure 5 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 4 ,like Figure 5 As shown, the process of calculating the first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft wall panel image in the above S302 may include:

[0099] S401. Using a first loss function, calculate a first regression loss function value according to a detection defect position in a target sample detection result and a sample defect position in defect description information.

[0100] It should be noted that the first loss function may be a Smooth L1 loss function.

[0101] In some embodiments, the Smooth L1 loss function can be expressed as:

[0102]

[0103] Wherein, the x value is the difference between the detected defect position and the sample defect position. Both the detected defect position and the sample defect position can be the coordinate information of the defect position.

[0104] S402: Using the second loss function, calculate the first classification loss function value according to the detection defect type in the target sample detection result and the sample defect type in the defect description information.

[0105] It should be noted that the second loss function may be a Focal Loss loss function which is a loss function specially designed to deal with the problem of class imbalance.

[0106] In some embodiments, the Focal Loss function can be expressed as:

[0107]

[0108] Among them, p represents the target probability of detecting defect types, y represents the sample defect type, α is a factor used to control the weights of positive and negative samples, and γ is a weight factor used to control difficult and easy samples.

[0109] S403: Calculate a first loss function according to the first regression loss function value and the first classification loss function value.

[0110] Among them, a preset loss function formula is used to calculate the first loss function according to the first regression loss function value and the first classification loss function value.

[0111] In some embodiments, the preset loss function formula can be expressed as: L = L reg +L cls , L cls Represents the first regression loss function value, L reg Represents the first classification loss function value.

[0112] Alternatively, if Figure 2 As shown, the initial backbone network module includes: a plurality of convolution modules and a target convolution module connected in sequence; the target convolution module includes: a first convolution layer, a deep attention deformable convolution module and a second convolution layer connected in sequence;

[0113] Figure 6 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 5 ,like Figure 6 As shown, the process of using the initial backbone network module in the above S201 to process the sample aircraft panel image in the sample data set to obtain the first feature map may include:

[0114] S501, using multiple convolution modules to process a sample aircraft siding image to obtain a first sub-feature map output by a last preset first number of convolution modules among the multiple convolution modules.

[0115] Among them, the output of the previous convolution module among multiple convolution modules is used as the output of the next convolution module.

[0116] In some embodiments, the number of multiple convolution modules connected in sequence can be 4, the convolution kernel size of the convolution layer in the first convolution module among the multiple convolution modules can be 7*7, the number of channels can be 64, and the step size can be 2; the second convolution module can include 3 first sub-modules connected in sequence, each first sub-module includes a maximum pooling layer and three convolution layers connected in sequence, the maximum pooling layer kernel size of the maximum pooling layer can be 3*3, the channel can be 2, the convolution kernel size of the first convolution layer can be 1*1, the number of channels can be 64, the convolution kernel size of the second convolution layer can be 3*3, the number of channels can be 64, and the convolution kernel size of the third convolution layer can be 1*1, and the number of channels can be 256.

[0117] Among them, the second convolution module may include 4 second submodules connected in sequence, each second submodule may include three convolution layers connected in sequence, the convolution kernel size of the first convolution layer may be 1*1, the number of channels may be 128, the convolution kernel size of the second convolution layer may be 3*3, the number of channels may be 128, and the convolution kernel size of the third convolution layer may be 1*1, and the number of channels may be 512. The fourth convolution module may include 6 third submodules connected in sequence, each third submodule may include three convolution layers connected in sequence, the convolution kernel size of the first convolution layer may be 1*1, the number of channels may be 256, the convolution kernel size of the second convolution layer may be 3*3, the number of channels may be 256, and the convolution kernel size of the third convolution layer may be 1*1, and the number of channels may be 1024.

[0118] S502: Using a first convolutional layer to process a target first sub-feature map output by a last convolutional module among multiple convolutional modules, to obtain a first intermediate map.

[0119] In some embodiments, the initial backbone network module includes four fourth submodules connected in sequence, and each fourth submodule may include a first convolutional layer, a deep attention deformable convolutional module, and a second convolutional layer connected in sequence. Among them, the convolution kernel size of the first convolutional layer can be 1*1, the number of channels can be 512, the deep attention deformable convolutional module can be 3*3, the number of channels can be 512, and the convolution kernel size of the second convolutional layer can be 1*1, and the number of channels can be 2048.

[0120] In an embodiment of the present application, the first convolution layer, the deep attention deformable convolution module, and the second convolution layer are three Bottlenecks in the initial defect detection model (an important component introduced in the deep residual network to reduce the computational complexity of the model and improve the feature extraction capability).

[0121] S503: Use a deep attention deformable convolution module to process the first intermediate image to obtain a second intermediate image.

[0122] The second intermediate image includes offset information.

[0123] S504: Use a second convolutional layer to process the second intermediate image to obtain a second sub-feature image.

[0124] The first feature map includes: a first sub-feature map and a second sub-feature map.

[0125] In an embodiment of the present application, the last convolution module among multiple convolution modules outputs the target first sub-feature map to the first convolution layer, the first convolution layer processes the target first sub-feature map to obtain a first intermediate map, and the first convolution layer outputs the first intermediate map to the deep attention deformable convolution module; the deep attention deformable convolution module processes the first intermediate map to obtain a second intermediate map, and the deep attention deformable convolution module outputs the second intermediate map to the second convolution layer; the second convolution layer processes the second intermediate map to obtain a second sub-feature map.

[0126] It should be noted that if Figure 2 As shown, the last two convolution modules in the multiple convolution modules are connected to the fourth convolution layer and the fifth convolution layer in the initial pyramid network module, and the target convolution module is connected to the sixth convolution layer in the original pyramid network model; wherein, the last two convolution modules in the multiple convolution modules output the first sub-feature map to the fourth convolution layer and the fifth convolution layer respectively, and the target convolution module outputs the second sub-feature map to the sixth convolution layer.

[0127] The result output by the sixth convolutional layer is upsampled to obtain a first upsampling result; the result output by the fifth convolutional layer is fused with the first upsampling result to obtain a first fusion result; the first fusion result is upsampled for the second time to obtain a second upsampling result; the result output by the fourth convolutional layer is fused with the second upsampling result to obtain a second fusion result, and the second fusion result, the first fusion result, and the result output by the sixth convolutional layer are respectively input to the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer, as shown in FIG. Figure 2 As shown, the ninth convolutional layer is connected to the eleventh convolutional layer through the tenth convolutional layer.

[0128] In the embodiment of the present application, the initial pyramid network module is used to fuse the first sub-feature map and the second sub-feature map output by the initial backbone network module. The structure fuses low-level high-resolution features with high-level semantic features by establishing lateral connections between different feature maps. This connection helps to transmit more refined information and provide a more comprehensive feature representation.

[0129] like Figure 2 As shown, the initial detection model includes: multiple regression and classification modules, the seventh convolution layer, the eighth convolution layer, the ninth convolution layer, the tenth convolution layer, and the eleventh convolution layer are respectively connected to each regression and classification module to output a second feature map to each regression and classification module, and each regression and classification module outputs multiple sample detection results according to the second feature map.

[0130] In the embodiment of the present application, the initial detection model is divided into a classification branch and a regression branch. The classification branch is used to output the detection defect category, and the regression branch is used to output the detection defect location. The classification branch is a small full convolutional network structure connected to each feature pyramid layer, and the regression branch attaches another small full convolutional network to each pyramid level to restore the offset of each anchor box to the nearby real object.

[0131] Optionally, Figure 7 A schematic diagram of the network structure of a deep attention deformable convolution module provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the deep attention deformable convolution module includes: a plurality of third convolution layers and a spatial attention module connected in sequence;

[0132] The process of using the deep attention deformable convolution module to process the first intermediate image in S503 to obtain the second intermediate image may include:

[0133] The first intermediate image is processed by multiple third convolutional layers to obtain a fourth intermediate image output by the last third convolutional layer among the multiple third convolutional layers; wherein the output of the previous third convolutional layer is used as the input of the next third convolutional layer; and the fourth intermediate image is processed by a spatial attention module to obtain a second intermediate image.

[0134] Among them, Figure 7 As shown, the first intermediate image can be input into the first third convolution layer of multiple third convolution layers, and processed in sequence by the second third convolution layer, the third third convolution layer, and the spatial attention module. The spatial attention module outputs an offset region with an offset, and the second intermediate image can be output.

[0135] In an embodiment of the present application, the deep attention deformable convolution module is a hybrid convolution attention module that combines multiple third convolution layers and a spatial attention module. Multiple third convolution layers can be called deformable convolutions. Deformable convolution introduces additional learnable offsets at each convolution kernel position, so that the convolution kernel can dynamically adjust the sampling position according to the deformation of the input feature map. However, the offset of the deformable convolution uses a simple convolution to obtain the offset. When processing complex backgrounds, a simple convolution layer may not be able to fully learn the offset of the complex deformation. Therefore, the present application improves the offset generation process by adding convolution layers and introducing an attention mechanism. There is an offset in the feature map.

[0136] Specifically, the deep attention deformable convolution module is a combination of multiple third convolutional layers and spatial attention modules. The superposition of multiple third convolutional layers forms a deeper network, which can better capture the information of the target. The addition of the spatial attention module enables the network to adaptively focus on important feature areas and improve the generation quality of the offset, thereby improving the detection performance of the model under complex backgrounds.

[0137] It should be noted that the initial defect detection model can be an improved Retina Net (a single-stage target detector) model. In an embodiment of the present application, the pre-trained weights on the COCO dataset (a large-scale image recognition dataset used for object detection, segmentation and subtitle tasks) can be used to initialize the initial defect detection model, and the obtained model weights are loaded into the initial defect detection model. Among them, the sample data set can be divided into a training set and a validation set, and the initial defect detection model is then trained and fine-tuned using the training set to obtain a defect detection module. Compared with training the model from scratch, fine-tuning can save a lot of computing resources and computing time, and improve computing efficiency and accuracy. The defect detection model is tested using a validation set. According to the test results, the performance indicators of the defect detection model, such as accuracy, precision, recall, etc., are calculated.

[0138] In summary, the deep attention deformable convolution module is introduced into Resnet-50 in the embodiment of the present application, so that the network can adaptively focus on important feature areas and improve the generation quality of offsets, thereby improving the defect detection performance of the defect detection model under complex backgrounds.

[0139] Optionally, Figure 8 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 6 ,like Figure 8 As shown, the process of determining the target sample detection result from the multiple sample detection results in S301 may include:

[0140] S601. Calculate multiple second loss function values ​​corresponding to multiple sample detection results according to each sample detection result and defect description information.

[0141] In some embodiments, a plurality of preset candidate anchor frames in the sample aircraft panel image are determined; and the intersection-and-union ratio between each candidate anchor frame and the annotation frame in the sample aircraft panel image is calculated. b1 and b2 represent the candidate anchor box and the annotation box, respectively. The sample defect position in the defect description information can represent the position of the annotation box in the sample aircraft panel image, and the sample defect type in the defect description information can represent the defect type selected in the annotation box in the sample aircraft panel image.

[0142] Among them, according to the intersection-and-union ratio between each candidate anchor frame and the labeled frame in the sample aircraft siding image, the anchor frames with the top L intersection-and-union ratios are selected from multiple candidate anchor frames as multiple sample detection results.

[0143] In an embodiment of the present application, a second regression loss function value is calculated based on the detection defect positions and sample defect positions in multiple sample detection results; a second classification loss function value is calculated based on the detection defect types in multiple sample detection results and the sample defect types in the defect description information; and based on the second regression loss function value and the second classification loss function value, multiple second loss function values ​​corresponding to the multiple sample detection results are calculated.

[0144] S602. Sort the multiple second loss function values ​​from small to large to obtain a sorting result.

[0145] S603: Determine a target sample detection result from multiple sample detection results according to the sorting result.

[0146] According to the sorting result, the first target number of sample detection results are selected from the multiple sample detection results as the target sample detection results. The first target number of sample detection results refers to the target number of sample detection results with the smallest second loss function value in the sorting result.

[0147] Optionally, Fig. 9 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 7 ,like Fig. 9 As shown, the process of determining the target sample detection result from the multiple sample detection results according to the sorting result in S603 may include:

[0148] S701. Determine the target quantity according to the defect description information.

[0149] In some embodiments, based on the intersection-and-union ratio between each candidate anchor frame and the labeled frame in the sample aircraft siding image, the top K anchor frames with the intersection-and-union ratio are selected from multiple candidate anchor frames as multiple sample detection results, the K values ​​are summed and rounded, and the summed and rounded value is recorded as M, and M is guaranteed to be at least 1.

[0150] S702. Select a target number of target sample detection results from multiple sample detection results according to the sorting result.

[0151] Among them, the number of targets can be expressed as M.

[0152] In an embodiment of the present application, the sample defect position in the defect description information, that is, the annotation box in the sample aircraft siding image, the first M sample detection results selected from the L sample detection results are used as the target sample detection results, and the remaining sample detection results are used as negative samples.

[0153] Among them, the anchor boxes in the detection results of the first M samples, that is, the detected defect locations are positive samples.

[0154] It should be noted that in this example, the simOTA sample allocation strategy is used to allocate positive and negative samples. Small targets are usually small in area and have unclear features in the image. The traditional maximum intersection-over-union (IoU) sample allocation strategy uses the IoU between the anchor box and the true box. If the IoU is greater than the positive threshold, it is a positive sample, and if it is less than the negative threshold, it is a negative sample. However, since small targets occupy fewer pixels in the image, the IoU between the predicted box and the small target is usually low. In this case, it is difficult for small targets to obtain positive samples that match them, resulting in low detection accuracy for small targets. The simOTA sample allocation strategy combines factors such as the target's confidence and classification loss, so that small targets can obtain more positive samples, thereby increasing the network's attention to small targets.

[0155] In summary, the simOTA sample allocation strategy is adopted, which combines factors such as target confidence and classification loss, so that small targets can obtain more positive samples to increase the network's attention to small targets, thereby improving the network's detection accuracy for small targets.

[0156] Optionally, Fig.10 A schematic diagram of a defect detection model training method provided in an embodiment of the present application Figure 8 ,like Fig.10 As shown, the process of constructing a sample data set according to the original aircraft panel images in multiple scenes in S102 may include:

[0157] S801. Label each original aircraft panel image to obtain each labeled aircraft panel image.

[0158] Among them, an optical camera is used to collect original aircraft wall panel images, and each original aircraft wall panel image can be called the most initial unlabeled fault data.

[0159] S802: Generate defect description information corresponding to each annotated aircraft panel image.

[0160] In some embodiments, each original aircraft panel image is annotated using a preset software to obtain each annotated aircraft panel image, and defect description information corresponding to each annotated aircraft panel image is obtained. The defect description information may be a ".json" format file.

[0161] Optionally, the preset software may be Label Me software (an image annotation tool).

[0162] S803: Perform image enhancement operations on each labeled aircraft siding image to obtain a plurality of sample aircraft siding images.

[0163] Image enhancement includes operations such as scaling, cropping, horizontal flipping, and contrast enhancement. By performing image enhancement operations on the labeled aircraft panel images, the sample size can be increased. This operation can enhance the distribution diversity of the data set and improve the generalization ability of the model.

[0164] S804: construct a sample data set based on a plurality of sample aircraft panel images and defect description information corresponding to each labeled aircraft panel image.

[0165] In summary, the embodiment of the present application provides a method for training a defect detection model, which uses deep learning to detect defects in aircraft sidings, greatly saving manpower and material resources; the introduction of the deep attention deformable convolution module enables the network to adaptively focus on important feature areas, improve the generation quality of offsets, and thus improve the detection performance of the model under complex backgrounds. The simOTA sample allocation strategy is adopted, which combines factors such as the confidence and classification loss of the target, so that small targets can obtain more positive samples to increase the network's attention to small targets, thereby improving the network's detection accuracy for small targets.

[0166] The following describes the defect detection model training device, equipment, storage medium, etc. used to execute the defect detection model training method provided in this application. Its specific implementation process and technical effects refer to the relevant content of the above-mentioned defect detection model training method, which will not be repeated below.

[0167] Fig.11 A structural diagram of a defect detection model training device provided in an embodiment of the present application, such as Fig.11 As shown, the device comprises:

[0168] An acquisition module 101 is used to acquire original aircraft panel images in multiple scene states in a preset assembly environment, where the states of at least one dimension of shooting time, shooting angle and assembly location are different in different scene states;

[0169] A construction module 102 is used to construct a sample data set based on the original aircraft panel images in the multiple scenes, wherein the sample data set includes: multiple sample aircraft panel images with defective parts marked, and defect description information corresponding to the sample aircraft panel images;

[0170] A training module 103 is used to train the initial defect detection model according to the sample data set to obtain the defect detection model;

[0171] The defect detection model is used to process the image of the aircraft panel to be inspected under the preset assembly environment to obtain the defect detection result of the aircraft panel to be inspected.

[0172] Optionally, the initial defect detection model includes: an initial backbone network module, an initial pyramid network module and an initial detection module; the training module 103 is specifically used to use the initial backbone network module to process the sample aircraft siding image in the sample data set to obtain a first feature map; use the initial pyramid network module to process the first feature map to obtain a second feature map; use the initial detection module to output a sample detection result for the sample aircraft siding image based on the second feature map; according to the sample detection result and the defect description information corresponding to the sample aircraft siding image, the initial backbone network module, the initial pyramid network module and the initial detection module are trained to obtain the defect detection model including a target backbone network module, a target pyramid network module and a target detection model.

[0173] Optionally, the number of the sample detection results is multiple; the training module 103 is specifically used to determine the target sample detection result from the multiple sample detection results; calculate the first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft siding image; according to the first loss function corresponding to the target sample detection result, train the initial backbone network module, the initial pyramid network module and the initial detection module until a preset termination condition is met, so as to obtain the defect detection model including the target backbone network module, the target pyramid network module and the target detection model.

[0174] Optionally, the training module 103 is specifically used to use a first loss function to calculate a first regression loss function value according to a detection defect position in the target sample detection result and a sample defect position in the defect description information; use a second loss function to calculate a first classification loss function value according to a detection defect type in the target sample detection result and a sample defect type in the defect description information; and calculate the first loss function according to the first regression loss function value and the first classification loss function value.

[0175] Optionally, the initial backbone network module includes: a plurality of convolution modules and a target convolution module connected in sequence; the target convolution module includes: a first convolution layer, a deep attention deformable convolution module and a second convolution layer connected in sequence;

[0176] The training module 103 is specifically used to use the multiple convolution modules to process the sample aircraft siding image to obtain a first sub-feature map output by the last preset first number of convolution modules in the multiple convolution modules; wherein the output of the previous convolution module in the multiple convolution modules is used as the output of the next convolution module; using the first convolution layer to process the target first sub-feature map output by the last convolution module in the multiple convolution modules to obtain a first intermediate map; using the deep attention deformable convolution module to process the first intermediate map to obtain a second intermediate map, wherein the second intermediate map includes offset information; using the second convolution layer to process the second intermediate map to obtain a second sub-feature map; wherein the first feature map includes: the first sub-feature map and the second sub-feature map.

[0177] Optionally, the deep attention deformable convolution module includes: a plurality of third convolution layers and a spatial attention module connected in sequence;

[0178] The training module 103 is specifically used to use the multiple third convolutional layers to process the first intermediate image to obtain a fourth intermediate image output by the last third convolutional layer among the multiple third convolutional layers; wherein the output of the previous third convolutional layer is used as the input of the next third convolutional layer; and the spatial attention module is used to process the fourth intermediate image to obtain the second intermediate image.

[0179] Optionally, the training module 103 is specifically used to calculate multiple second loss function values ​​corresponding to multiple sample detection results based on each of the sample detection results and the defect description information; sort the multiple second loss function values ​​from small to large to obtain a sorting result; and determine the target sample detection result from the multiple sample detection results based on the sorting result.

[0180] Optionally, the training module 103 is specifically configured to determine a target quantity according to the defect description information; and select a target quantity of target sample detection results from a plurality of the sample detection results according to the sorting result.

[0181] Optionally, the construction module 102 is specifically used to annotate each of the original aircraft panel images to obtain each annotated aircraft panel image; generate defect description information corresponding to each of the annotated aircraft panel images; perform image enhancement operations based on each of the annotated aircraft panel images to obtain a plurality of sample aircraft panel images; and construct the sample data set based on the plurality of sample aircraft panel images and the defect description information corresponding to each of the annotated aircraft panel images.

[0182] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0183] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0184] Fig.12 A structural diagram of a defect detection model training device provided in an embodiment of the present application, such as Fig.12 As shown, the device includes: a processor 201 and a memory 202.

[0185] The memory 202 is used to store programs, and the processor 201 calls the programs stored in the memory 202 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0186] Optionally, the present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute the above method embodiment when executed by a processor.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0190] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0191] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a defect detection model, characterized in that: The method comprises: Acquire original aircraft panel images in multiple scene states in a preset assembly environment, where the states of at least one dimension of shooting time, shooting angle, and assembly location are different in different scene states; Constructing a sample data set according to the original aircraft panel images in the multiple scenes, wherein the sample data set includes: multiple sample aircraft panel images with defective parts marked, and defect description information corresponding to the sample aircraft panel images; Training an initial defect detection model according to the sample data set to obtain the defect detection model; The defect detection model is used to process the image of the aircraft panel to be inspected under the preset assembly environment to obtain the defect detection result of the aircraft panel to be inspected.

2. The method according to claim 1, characterized in that The initial defect detection model includes: an initial backbone network module, an initial pyramid network module and an initial detection module; the initial defect detection model is trained according to the sample data set to obtain the defect detection model, including: Using the initial backbone network module to process the sample aircraft siding image in the sample data set to obtain a first feature map; Using the initial pyramid network module to process the first feature map to obtain a second feature map; Outputting a sample detection result for the sample aircraft siding image using the initial detection module according to the second feature map; According to the sample detection results and the defect description information corresponding to the sample aircraft panel image, the initial backbone network module, the initial pyramid network module and the initial detection module are trained to obtain the defect detection model including the target backbone network module, the target pyramid network module and the target detection model.

3. The method according to claim 2, characterized in that The number of the sample detection results is multiple; the initial backbone network module, the initial pyramid network module and the initial detection module are trained according to the sample detection results and the defect description information corresponding to the sample aircraft wall panel image to obtain the defect detection model including the target backbone network module, the target pyramid network module and the target detection model, including: Determine a target sample test result from the plurality of sample test results; Calculating a first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft siding image; According to the first loss function corresponding to the target sample detection result, the initial backbone network module, the initial pyramid network module and the initial detection module are trained until a preset termination condition is met, thereby obtaining the defect detection model including the target backbone network module, the target pyramid network module and the target detection model.

4. The method according to claim 3, characterized in that The step of calculating a first loss function corresponding to the target sample detection result according to the target sample detection result and the defect description information corresponding to the sample aircraft panel image comprises: Using a first loss function, according to the detection defect position in the target sample detection result and the sample defect position in the defect description information, calculate a first regression loss function value; Using a second loss function, according to the detection defect type in the target sample detection result and the sample defect type in the defect description information, calculate the first classification loss function value; The first loss function is calculated according to the first regression loss function value and the first classification loss function value.

5. The method according to claim 2, characterized in that: The initial backbone network module includes: a plurality of convolution modules and a target convolution module connected in sequence; the target convolution module includes: a first convolution layer, a deep attention deformable convolution module and a second convolution layer connected in sequence; The initial backbone network module is used to process the sample aircraft panel image in the sample data set to obtain a first feature map, including: The sample aircraft panel image is processed using the multiple convolution modules to obtain a first sub-feature map output by a last preset first number of convolution modules among the multiple convolution modules; wherein the output of a previous convolution module among the multiple convolution modules is used as the output of a subsequent convolution module; Using the first convolution layer to process the target first sub-feature map output by the last convolution module among the multiple convolution modules to obtain a first intermediate map; Processing the first intermediate image using the deep attention deformable convolution module to obtain a second intermediate image, wherein the second intermediate image includes offset information; Processing the second intermediate image using the second convolutional layer to obtain a second sub-feature image; The first feature map includes: the first sub-feature map and the second sub-feature map.

6. The method according to claim 5, characterized in that The deep attention deformable convolution module includes: a plurality of third convolution layers and a spatial attention module connected in sequence; The step of processing the first intermediate image by using the deep attention deformable convolution module to obtain a second intermediate image includes: Processing the first intermediate image using the multiple third convolutional layers to obtain a fourth intermediate image output by the last third convolutional layer among the multiple third convolutional layers; wherein the output of the previous third convolutional layer is used as the input of the next third convolutional layer; The fourth intermediate image is processed using the spatial attention module to obtain the second intermediate image.

7. The method according to claim 3, characterized in that Determining a target sample detection result from the plurality of sample detection results comprises: Calculating a plurality of second loss function values ​​corresponding to the plurality of sample detection results according to the sample detection results and the defect description information; Sort the plurality of second loss function values ​​from small to large to obtain a sorting result; According to the sorting result, the target sample detection result is determined from the multiple sample detection results.

8. The method according to claim 7, characterized in that Determining the target sample detection result from the plurality of sample detection results according to the sorting result includes: Determine the target quantity according to the defect description information; According to the sorting result, a target number of target sample detection results are selected from the multiple sample detection results.

9. The method according to claim 1, characterized in that: The constructing of the sample data set according to the original aircraft wall panel images in the plurality of scenes comprises: Annotating each of the original aircraft wall panel images to obtain each annotated aircraft wall panel image; Generating defect description information corresponding to each of the annotated aircraft panel images; Performing an image enhancement operation on each of the labeled aircraft siding images to obtain a plurality of sample aircraft siding images; The sample data set is constructed according to the plurality of sample aircraft panel images and the defect description information corresponding to each of the labeled aircraft panel images.

10. A defect detection model training device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the training method of the defect detection model described in any one of claims 1 to 9 is implemented.

Images