Aircraft riveting assembly mistake and omission automatic detection method based on multi-task learning

Abstract

The application provides an aviation riveting assembly mistake and omission automatic detection method based on multi-task learning, comprising the following steps: 1) introducing a dedicated expert module in a multi-task learning framework, sharing the expert module and the dedicated expert of each task to extract image features, and being capable of fusing two different image features, finally outputting the fused feature images extracted by the dedicated expert module of each task and the shared expert module to a visual detection model and a depth estimation model respectively; 2) taking the output of the multi-task learning framework as total input image information, and embedding an adaptive spatial feature fusion (ASFF) module in the connection channel between a pyramid pooling feature fusion module and a decoder in the depth estimation model; 3) adding two fully connected layers (FC) after the channel attention network channel of the visual detection model. The application has better target recognition accuracy and depth estimation performance.

Description

Technical Field

[0001] This invention relates to the field of visual inspection and depth assessment technology, and in particular to an automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning. Background Technology

[0002] Statistics show that the number of rivets on a large passenger aircraft can reach millions. Since rivet connections directly affect the structural strength of the aircraft, the riveting of the aircraft fuselage and the inspection of any missing rivets are particularly important and complex. However, given the sheer number of rivets, traditional manual riveting and inspection are not only inefficient, but also struggle to guarantee the uniformity of riveting and the accuracy of inspection. Existing automated inspection systems mostly focus on single tasks (such as appearance inspection or depth estimation), making it difficult to meet the comprehensive needs of complex aerospace manufacturing scenarios for efficient, high-precision, and multi-dimensional monitoring of missing rivets. For example:

[0003] Question 1: In multi-task learning, because the underlying Expert module of traditional multi-task learning models is shared, increasing the weight of the visual detection task on the shared Expert module will improve its detection accuracy. However, this will inevitably take away weight from the depth estimation task, causing its estimation performance to decline. This is the "seesaw" problem inherent in traditional multi-task learning: an improvement in the accuracy of one task often leads to a decrease in the accuracy of another. In the engineering applications of automated riveting and missing rivet detection, the goal is to simultaneously improve the accuracy of both types of tasks, balance their weights, and reduce the constraints between them.

[0004] Question 2: In multi-task learning-based depth estimation tasks, a pyramid pooling module (PPM), or multi-scale feature fusion module, is often introduced to improve the estimation performance of depth estimation models. The PPM uses four encoders to obtain four different sized feature images from the input image. These feature images contain different feature information: smaller feature images tend to retain detailed information of the input image, while larger feature images retain more contextual information, i.e., global information. By fusing these different sized feature images, the PPM can preserve both the detailed and global features of the input image in its fused output feature image, thus improving the model's performance in estimating the depth of objects in the input image. However, the pyramid pooling feature fusion method also has many drawbacks. The most significant drawback is that since larger feature images mainly extract global information, while smaller feature images mainly extract small targets or detailed information, inconsistencies arise between the different sized feature images. When the size difference between the feature images is large, direct fusion can lead to significant feature extraction errors. These feature extraction errors can lead to problems such as target loss or depth estimation deviation when inspecting rivets or rivet holes, resulting in production accidents such as missed rivets or missed inspections. Therefore, this problem urgently needs to be solved in the engineering application of automated riveting and missed rivet detection.

[0005] Question 3: In multi-task learning-based visual detection tasks, the SELayer channel attention mechanism is often introduced to improve detection accuracy. The SELayer channel attention mechanism introduces a new architectural unit—the "Squeeze-and-Excitation" module, or SE module for short. For a given arbitrary transformation F... tr X1→X2, The SELayer architecture constructs a corresponding SE module to perform feature recalibration. Feature U first passes through F... sqThe squeezing operation yields a linear feature vector 1×1×C. This squeezing operation aggregates feature maps across spatial dimensions H×W to generate channel descriptors. These descriptors are embedded in two fully connected layers (FC), allowing information from the network's global scope to be utilized by its lower layers. Then, an activation operation activates specific samples within the descriptors, and a channel-dependent self-gating mechanism is used to learn and control the activation of each channel. A sigmoid non-linear activation function is then used to constrain the vector values ​​to be between 0 and 1. Finally, the resulting feature vector is reweighted and fed back into the feature map U to generate the output of the SE block, which can be directly fed into subsequent layers. This results in a reweighted feature set where desired features are highlighted and unwanted features are suppressed. However, real-world detection experiments have revealed several shortcomings of the SELayer channel attention mechanism. Because SE attention compresses spatial dimensional information through global average pooling (GAP), if key features only occupy a local area of ​​the image, GAP weakens their importance, potentially leading to the loss of some local details in the image (such as small objects or textures). Furthermore, while the SE module generates fixed weights for each channel of the image, the importance of features at different locations in real-world scenarios changes dynamically. Therefore, the target capture and tracking ability is relatively poor during dynamic recognition, potentially leading to target loss in the visual detection model. Since rivets are small targets in images, and rivet position detection and missed rivet detection are often dynamic and real-time, the problems of small target loss and poor dynamic target capture ability in the SELayer channel attention mechanism embedded model urgently need to be addressed. Summary of the Invention

[0006] The purpose of this invention is to provide an automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning. It solves the technical problems of feature extraction errors caused by the direct fusion of feature images of different sizes in the pyramid pooling feature fusion module, the easy loss of small targets in the SELayer channel attention mechanism embedded model, and the poor dynamic target capture capability. It improves the learning accuracy in both depth estimation and visual detection tasks, so as to better adapt to the dynamic real-time detection of small rivets and rivet holes by robotic arm sensors in actual working scenarios.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] An automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning includes the following steps:

[0009] 1) Under natural light conditions, images of rivets or rivet holes in industrial scenes are captured by an industrial camera. The captured images are then input into a multi-task model. A dedicated expert module is introduced into the multi-task learning framework. When the input image enters the model, the shared expert module and the dedicated expert for each task extract the image features simultaneously. The two different image features are then fused using a fully connected layer (FC) and a Softmax activation function. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the visual detection model and the depth estimation model, respectively.

[0010] 2) The output G of the multi-task learning framework D (x) As the total input image information, the pyramid pooling feature fusion module in the depth estimation model has four different encoders. An adaptive spatial feature fusion (ASFF) module is embedded in the connection channel between the pyramid pooling feature fusion module and the decoder in the depth estimation model. The adaptive spatial feature fusion (ASFF) module is used to divide the pixel ratio output by the pyramid pooling feature fusion module into four feature images of different sizes for feature fusion.

[0011] 3) Two fully connected layers (FCs) are added after the channel attention network channel of the visual detection model, and the two fully connected layers (FCs) are connected to the subsequent layers through the Softmax activation function.

[0012] Furthermore, the formula for expressing the input feature information is as follows:

[0013] ;

[0014] in, It is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features extracted from the dedicated expert module .

[0015] Furthermore, the output G of the multi-task learning framework D The expression for (x) is:

[0016] ;

[0017] Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

[0018] Furthermore, the feature image fusion expressions of the four adaptive spatial feature fusion ASFF modules are as follows:

[0019] First layer: ;

[0020] Second layer: ;

[0021] Third layer: ;

[0022] Fourth layer: ;

[0023] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; χ represents the feature image spatial weight when layer 4 is mapped to other layers. By fusing features of different sizes through four ASFF modules, four sets of fused features of the same size, Y1, Y2, Y3, and Y4, are obtained. Then, these four sets of features are fused through the decoder to obtain the final depth image output.

[0024] In summary, the beneficial effects of this invention are as follows: The automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning of this invention solves the "seesaw" problem commonly found in traditional multi-task learning frameworks, the feature extraction error caused by the direct fusion of feature images of different sizes in the pyramid pooling feature fusion module, and the problems of easy loss of small targets and poor dynamic target capture ability in the SELayer channel attention mechanism embedded model by designing and embedding dedicated expert modules for depth estimation and visual detection in the multi-task learning framework, introducing an adaptive spatial feature fusion module (ASFF) in the depth estimation model, and adding two additional fully connected layers (FC) after the SELayer channel attention network channel. The automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning of this invention has better target recognition accuracy and depth estimation performance. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This invention introduces a multi-task learning network framework with dedicated expert modules.

[0027] Figure 2 This invention introduces an adaptive spatial feature fusion module (ASFF) for the depth estimation model structure.

[0028] Figure 3This is the enhanced SELayer channel attention network structure in this invention;

[0029] Figure 4 This is the overall network structure of the multi-task learning model proposed in this invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0031] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" or "linked" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a mechanical connection or an electrical connection; it can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0032] Please see Figures 1-4 This invention discloses an automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning, comprising the following steps:

[0033] 1) Under natural light conditions, images of rivets or rivet holes in industrial scenes are captured using an industrial camera. These images are then input into a multi-task model. The multi-task learning framework introduces the concept of a specific expert module. When the input image enters the model, the shared expert module and the specific expert for each task simultaneously extract image features. These two different image features are then fused using a fully connected layer (FC) and the softmax activation function. The input feature information can be expressed by the following formula:

[0034] ;

[0035] Wherein, SD(x) is a selection matrix consisting of all selected vectors, including the features ET(S,a) extracted by the shared expert module and the features ET(D,a) extracted by the dedicated expert module.

[0036] In this way, each task can obtain a weighted average of multiple expert modules. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the visual detection model and the depth estimation model, respectively. The grid structure is as follows: Figure 1 As shown. Its output expression is:

[0037] ;

[0038] in, and represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

[0039] Since each task's dedicated expert module is only used to extract the features required for that task, its extraction accuracy is not affected by other tasks, thus effectively solving the "seesaw" problem in multi-task learning models.

[0040] 2) The output G of the multi-task learning framework D (x) serves as the total input image information. The Pyramid Pooling Module (PPM) in the depth estimation model has four different encoders (Encoder1, Encoder2, Encoder3, and Encoder4). An Adaptive Spatial Feature Fusion Module (ASFF) is embedded in the connection channel between the PPM and the decoder in the depth estimation model. The ASFF module is used to fuse the feature images of four different sizes (4×4, 8×8, 16×16, and 32×32) output by the PPM module. The feature image fusion expressions of the four ASFF modules are as follows:

[0041] First layer: ;

[0042] Second layer: ;

[0043] Third layer: ;

[0044] Fourth layer: ;

[0045] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b Y represents the feature value mapped from the sub-feature image of layer a to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers. Four ASFF modules are used to fuse features of different sizes to obtain four sets of fused features of the same size: Y1, Y2, Y3, and Y4. These four sets of features are then fused through a decoder to obtain the final depth image output.

[0046] The ASFF module fuses four sets of feature maps of different sizes, enabling information exchange between feature maps of different scales and obtaining four sets of feature information with different extracted content but the same size, which are then fused. That is, four levels of Y are obtained through the above formula, and the sizes of these four Ys are equal. This aligns the four different sizes of feature images extracted by the pyramid pooling feature fusion module, and its grid structure is as follows: Figure 2 As shown. This effectively reduces the error generated when fusing features with large size differences, thus solving the problem of feature extraction error caused by direct fusion due to different feature image sizes, and thereby improving the detection performance of the depth estimation model.

[0047] 3) To address the weakness of SELayer channel attention in the visual detection model, two additional fully connected (FC) layers are added after the network channels. These FC layers are then connected to subsequent layers using the Softmax activation function. The mesh structure is as follows: Figure 3 As shown. The introduction of the last two fully connected layers integrates global features, introduces nonlinear transformations, and optimizes classification decisions in the visual detection model. This enhances the SELayer channel attention mechanism's ability to detect small and localized targets and makes it more adaptable to dynamic detection in real-world scenarios. This improves the visual detection model's accuracy in recognizing small rivets and rivet holes in actual dynamic scenes.

[0048] Specifically, for visual inspection tasks, the output G of the multi-task learning framework will be... V (x) serves as the total input image information. The visual detection model incorporates an enhanced SELayer channel attention network, G, as described in this invention. V (x) The input image enters the enhanced SELayer channel attention network through the visual detection input port. The SE module then performs an Fsq compression operation on the input image to obtain a 1×1×C linear feature channel descriptor. Convolutional layers then extract image feature information from the feature channel descriptor, generating multiple feature maps, each corresponding to different visual features. Two fully connected layers (FC) subsequently flatten these feature maps into one-dimensional vectors, integrating all feature information to provide a global perspective for subsequent classification. These FC layers, combined with the ReLU activation function, introduce non-linear transformations, enhancing the model's expressive power and enabling it to capture complex feature combinations. The extracted features are then converted into a probability distribution using the Softmax function, determining the category to which the image features belong and thus determining their weights. Finally, these feature weights are input into subsequent layers of the visual detection model to output the final object detection image.

[0049] The present invention provides an automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning. This method addresses several issues through innovative code design: it designs and embeds dedicated expert modules for depth estimation and visual detection within a multi-task learning framework; it introduces an adaptive spatial feature fusion module (ASFF) into the depth estimation model; and it adds two additional fully connected layers (FC) after the SELayer channel attention network channel. These innovations solve the common "seesaw" problem in traditional multi-task learning frameworks, the feature extraction error caused by the direct fusion of feature images of different sizes in the pyramid pooling feature fusion module, and the issues of small target loss and poor dynamic target capture capability in the SELayer channel attention mechanism embedded model. The resulting automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning exhibits superior target recognition accuracy and depth estimation performance.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may utilize the disclosed technical content to make changes or equivalent variations to other fields. However, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, shall still fall within the protection scope of the present invention. In the description of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood through specific circumstances.

Claims

1. An automated detection method for errors and omissions in aerospace riveting assembly based on multi-task learning, characterized in that: Includes the following steps: 1) Under natural light conditions, images of rivets or rivet holes in industrial scenes are captured by an industrial camera. The captured images are then input into a multi-task model. A dedicated expert module is introduced into the multi-task learning framework. When the input image enters the model, the shared expert module and the dedicated expert for each task extract the image features simultaneously. The two different image features are then fused using a fully connected layer (FC) and a Softmax activation function. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the visual detection model and the depth estimation model, respectively. 2) The output G of the multi-task learning framework D (x) As the total input image information, the pyramid pooling feature fusion module in the depth estimation model has four different encoders. An adaptive spatial feature fusion (ASFF) module is embedded in the connection channel between the pyramid pooling feature fusion module and the decoder in the depth estimation model. The adaptive spatial feature fusion (ASFF) module is used to divide the pixel ratio output by the pyramid pooling feature fusion module into four feature images of different sizes for feature fusion. 3) Two fully connected layers (FCs) are added after the channel attention network channel of the visual detection model, and the two fully connected layers (FCs) are connected to the subsequent layers through the Softmax activation function; The formula for expressing the input feature information is: ； Among them, S D (x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features E extracted by the dedicated expert module T (D,a) ; The output G of the multi-task learning framework D The expression for (x) is: ； Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features; The four feature image fusion expressions of the adaptive spatial feature fusion ASFF module are as follows: First layer: ; Second layer: ; Third layer: ; Fourth layer: ; Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; χ represents the feature image spatial weight when layer 4 is mapped to other layers. By fusing features of different sizes through four ASFF modules, four sets of fused features of the same size, Y1, Y2, Y3, and Y4, are obtained. Then, these four sets of features are fused through the decoder to obtain the final depth image output.

Images