Detection method for identifying quality of airplane riveting pier head based on convolutional neural network

Abstract

The invention discloses a detection method for identifying the quality of an aircraft riveting pier head based on a convolutional neural network, and the method comprises the steps: obtaining an aircraft rivet pier head height foreground point cloud image and a rivet pier head round surface foreground point cloud image through a collection device, and building a training image data set; respectively sorting the foreground point cloud images, labeling the images, and obtaining a mask image after the labeling is completed; establishing an improved Deep Lab V3 + convolutional neural network model, and training the mask image; carrying out image segmentation on the foreground point cloud image through an improved Deep Lab V3 + convolutional neural network model, and obtaining a rivet upset head height and a rivet upset head circular surface image; skeleton extraction is conducted on the obtained height of the rivet upset head, and the protruding amount of the riveting upset head is obtained; according to the rivet upset head circular surface image, the circle center of the upset head surface is determined through least square method fitting circle, the longest diameter and the shortest diameter of the upset head surface are obtained, and the roundness of the riveting upset head is obtained. The method solves the problem of manual detection errors, is high in reliability, is high in image acquisition precision, is convenient, and is good in detection effect.

Description

Technical Field

[0001] The invention relates to a method for detecting aircraft riveted butt quality based on convolutional neural network recognition, and belongs to the technical field of aircraft skin riveting quality detection. Background Art

[0002] The wing of an aircraft is an important component of the aircraft. The skin forms the surface of the wing, and its function is to maintain the shape of the wing and directly bear the aerodynamic load. The main connection method of the skin is riveting, and the quality of the riveting operation directly affects the function of the aircraft. At present, in the practical teaching process, the quality inspection of aircraft skin riveting mainly relies on the experience of teachers to use naked eye inspection, which has a strong subjective judgment consciousness factor, low inspection efficiency, and affects the inspection results, and cannot meet the curriculum standards for improving students' practical training skills.

[0003] The patent with publication number CN113129268A discloses a method for quality inspection of aircraft riveted pier heads, which uses Gaussian blur, double threshold detection, etc. to perform image processing on the edge of the riveted pier head, uses the pixel measurement principle to indirectly obtain the rivet pier head area, and then obtains the pier head protrusion height through the volume invariance principle. This method uses Gaussian blur, double threshold detection, etc. to perform image processing quality inspection on the edge of the rivet pier head, and uses the pixel measurement principle to indirectly obtain the pier head area through an intermediate medium. The errors in calculating the number of pixels and other intermediate substances involved in edge image processing will affect the accuracy of rivet pier head area recognition; after the rivet is in contact with the rivet gun and the working surface to complete the riveting, the rivet volume will also produce loss error changes. At the same time, it is necessary to measure the roundness and pier head height of each riveted pier head area separately, which will affect the results of the teacher's riveted workpiece quality inspection of students during the teaching process, and the inspection efficiency is low, affecting the inspection accuracy. Summary of the invention

[0004] Purpose of the invention: In view of the shortcomings of the prior art, the present invention provides a method for detecting the quality of aircraft riveted pier heads based on a convolutional neural network. The present invention obtains a foreground point cloud image of the height of the aircraft rivet pier head and a foreground point cloud image of the circular surface of the rivet pier head, and after processing and training the images, obtains the protrusion amount of the riveted pier head and the roundness of the pier head, thereby improving the detection efficiency and accuracy.

[0005] Technical solution: A method for detecting the quality of aircraft riveted pier heads based on convolutional neural networks includes the following steps:

[0006] Step 1: Obtain the foreground cloud image of the height of the rivet head of a standard model aircraft and the foreground cloud image of the circular surface of the rivet head through a collection device to establish a training image data set;

[0007] Step 2: sort the aircraft rivet head height foreground point cloud image and the rivet head circular surface foreground point cloud image obtained in step 1 respectively, annotate the images, and obtain a mask image after the annotation is completed;

[0008] Step 3: Establish an improved Deep Lab V3+ convolutional neural network model and train the mask image in step 2;

[0009] Step 4: Use the improved Deep Lab V3+ convolutional neural network model trained and fitted in step 3 to perform image segmentation on the foreground point cloud image of the rivet head height and the foreground point cloud image of the rivet head circular surface of the riveted sample to be inspected, and obtain the rivet head height and rivet head circular surface images;

[0010] Step 5: Perform skeleton extraction on the rivet head height obtained in step 4 to obtain the convex amount of the riveted pier head;

[0011] Step 6: The circular surface image of the rivet head obtained after image segmentation in step 4 is fitted with a circle using the least squares method to determine the center of the head surface, and then the longest and shortest diameters of the head surface are found, and finally the roundness of the riveted head is determined.

[0012] Furthermore, the step 1 is specifically as follows:

[0013] Take photos perpendicular to the rivet direction, adjust the distance threshold between the acquisition device and the rivet, and obtain the foreground cloud image at the height of the rivet head;

[0014] Take photos parallel to the rivet direction, adjust the distance threshold between the acquisition device and the rivet, and obtain the foreground cloud image of the circular surface of the rivet head.

[0015] Furthermore, the step 2 is specifically as follows:

[0016] After sorting and renaming the sample images of aircraft rivet heads, the sample images are labeled in turn. According to actual needs, the image labels are divided into rivet head and rivet circle surface categories. The unlabeled areas are regarded as irrelevant backgrounds. The .json file generated after annotation is converted into a PASCAL VOC format mask image, and the PASCAL VOC format dataset production task is finally completed.

[0017] Furthermore, the step 3 is specifically as follows:

[0018] RestNet50 with ImageNet pre-trained weights is used as the backbone feature extraction network to accelerate the convergence of the network model training and optimize the feature extraction network;

[0019] The standard convolution in the atrous spatial convolution pooling pyramid ASPP is replaced with a depthwise separable convolution to improve the training and application efficiency of the model, thereby improving the atrous spatial convolution pooling pyramid ASPP.

[0020] In the atrous spatial convolutional pooling pyramid ASPP, the attention mechanism CBAM is used to add two positions to enhance the extraction of main features; the improved Deep Lab V3+ convolutional neural network model is established, the mask image is input into the improved Deep Lab V3+ convolutional neural network model for training, and the improved Deep Lab V3+ convolutional neural network model is fitted.

[0021] Furthermore, in step 3, the attention mechanism CBAM is used to add two positions in the dilated spatial convolutional pooling pyramid ASPP to enhance the main feature extraction, specifically:

[0022] The first attention mechanism CBAM is placed after the multi-scale feature fusion of the atrous spatial pooling pyramid ASPP, and the second attention mechanism CBAM is placed at the end of the atrous spatial pooling pyramid ASPP.

[0023] Furthermore, the step 4 is specifically as follows:

[0024] A standard height target is used as a reference object, and then the reference object and the riveted sample to be tested are segmented by the improved DeepLabV3+ convolutional neural network model after training and fitting, and finally the rivet head height and the rivet head circular surface image are obtained, where the rivet head height is obtained by measuring the reference object.

[0025] Furthermore, the step 5 is specifically as follows:

[0026] The transparency and topological structure of the object contour in the rivet head height image after image segmentation are expressed by a single curve, specifically a skeleton graph with a width of 1 pixel. According to the actual height a of the reference object and the pixel value b occupied by the skeleton contour, the actual length of the riveted pier head is solved h=a / b×d, where d is the pixel value occupied by the rivet pier head, that is, the protrusion of the rivet pier head.

[0027] Furthermore, the step 6 is specifically as follows:

[0028] According to the circle formula (xA) 2 +(yB) 2 =R 2 Available,

[0029] R 2 =x 2 -2Ax+A 2 +y 2 -2By+B 2

[0030] make:

[0031] a=-2A, b=-2B, c=A 2 +B 2 -R 2

[0032] This gives another form of the equation of the circle:

[0033] x 2 +y 2 +ax+by+c=0 (1)

[0034] According to formula (1), we only need to find a, b, and c to get the parameters of the center and radius:

[0035]

[0036] Specifically, let the sample set on the edge of the rivet head contour in aircraft riveting be [(x i ,y i ), i∈1, 2, 3, .....n], the center of the fitting circle is (A, B), and the distance from the point to the center is d i：

[0037] d i 2 =(X i -A) 2 +(Y i -B) 2

[0038] Let point (X i , Y i ) to the edge of the circle and the square of the radius is:

[0039] δ=d i 2 -R 2 =(X i -A) 2 +(Y i -B) 2 -R 2 =X i 2 +Y i 2 +aX i +bY i +c

[0040] Let Q(a, b, c) be δ 2 The sum of squares:

[0041]

[0042] Since the square difference Q(a, b, c) is greater than or equal to 0, the function has a minimum value greater than or equal to 0, and its maximum value is infinity; Q(a, b, c) with respect to (a, b, c), let the partial derivative equal to 0, then we get the extreme point, and by comparing all the extreme point functions, we can get the minimum value;

[0043]

[0044] Let, constant D = (X i 2 +Y i 2 +bY i +c), the above formula can be transformed into:

[0045]

[0046] but:

[0047]

[0048] Substituting D into the above formula, we get:

[0049]

[0050] Similarly, we can get the partial derivatives of b and c:

[0051]

[0052]

[0053] Rewrite the above equation into an equation form:

[0054]

[0055] According to the ∑ operation rule, the constant is taken out, and because the equation is equal to 0, it can be directly omitted, and the following formula is obtained:

[0056]

[0057] Expand the above formula into the following form:

[0058]

[0059] According to the summation formula, we get Substituting into the equation we get:

[0060]

[0061] So solve the system of equations, eliminate c, equation (2) have to

[0062]

[0063] (3) have to

[0064]

[0065] make

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] Then, we get the equation:

[0072]

[0073] therefore:

[0074]

[0075]

[0076]

[0077] Then solve the least squares circle center and radius:

[0078]

[0079] After the center of the fitting circle is determined, all points on the rivet head contour are obtained, and the distance from any pixel point of the contour through the center of the fitting circle to any pixel point at the other end of the contour is solved to obtain the riveted head diameter information set, sort the information in the set, and filter out the maximum diameter D max and the shortest diameter D min , by collecting information and formula Solve for the average diameter D ave ,

[0080] Therefore, the circular deviation value δ oor for

[0081]

[0082] The corresponding roundness △R is

[0083]

[0084] Beneficial effect: The present invention is based on an improved Deep Lab V3+ convolutional neural network model, combined with two shooting angles perpendicular to the rivet surface and parallel to the rivet surface, to form a foreground spot cloud image of the machine rivet pier head height and a foreground spot cloud image of the rivet pier head circular surface, respectively, to achieve rivet pier head height measurement and pier head surface roundness measurement, solve the errors caused by human detection, and has higher reliability. Compared with the traditional RGB threshold segmentation method and rivet pier head edge detection method, the image acquisition accuracy is higher, it is more convenient, and the detection effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0086] Figure 1 This is a flow chart of a method for detecting the quality of aircraft riveted pier heads based on a convolutional neural network according to the present invention;

[0087] Figure 2 The RGB diagram of the plate rivet head height for teaching provided by the embodiment of the present invention;

[0088] Figure 3 RGB image of the circular surface of a plate rivet provided for teaching in an embodiment of the present invention;

[0089] Figure 4 A point cloud foreground image of the plate rivet head height provided for teaching purposes in an embodiment of the present invention;

[0090] Figure 5 A point cloud foreground image of a plate rivet circle surface for teaching provided by an embodiment of the present invention;

[0091] Figure 6 The improved network model provided in the embodiment of the present invention has a ResNet residual network architecture for feature extraction;

[0092] Figure 7 In the improved network model provided by the embodiment of the present invention Figure 7 (a) Standard convolution and Figure 7 (b) Schematic diagram of the computational complexity of depthwise separable convolution;

[0093] Figure 8 A structural diagram of the improved Deep Lab V3+ convolutional neural network model provided in an embodiment of the present invention;

[0094] Fig. 9 A skeleton extraction diagram of the plate rivet head height provided for teaching purposes in an embodiment of the present invention;

[0095] Fig.10 A schematic diagram of the circular surface fitting of a plate rivet head for teaching purposes provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] In the description of the present invention, it is necessary to understand that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0098] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0099] like Figure 1-Figure 3 As shown, the method for detecting the quality of aircraft riveted pier heads based on convolutional neural network includes the following steps:

[0100] Step 1: Use an RGB-D camera to obtain a standard model aircraft rivet head height foreground cloud image and a rivet head circular surface foreground cloud image to establish a training image dataset;

[0101] The quality of the foreground point cloud image at the height of the aircraft rivet head and the foreground point cloud image on the circular surface of the rivet head directly affects the robustness of the network model training, and is prone to poor segmentation effect. Therefore, the RGB-D camera TOF ranging principle is adopted to change the distance threshold between the camera and the object to be measured, so as to obtain the foreground point cloud image at the height of the rivet head and the foreground point cloud image on the circular surface of the rivet head; specifically, the laser projection component in the camera continuously radiates modulated light to the outline of the target, the TOF sensor receives the modulated light returned from the surface of the object to be measured, and the color camera obtains color information. The embedded computing vision processor processes the information obtained by the TOF sensor and the color camera, solves the depth information, and completes the consistency with the RGB information at the same time, and realizes the acquisition of three-dimensional point cloud images, transmits the acquired information to the host, receives the signal from the upper host, and finally displays the RGB image and three-dimensional point cloud information.

[0102] Use the RGB-D camera to shoot perpendicular to the rivet, adjust the distance between the camera and the rivet to be detected, solve the optimal distance threshold, and obtain the foreground cloud image at the height of the rivet head, such as Figure 4 As shown in the figure, the RGB-D camera is used to shoot in parallel with the rivet, the distance between the camera and the detected rivet is adjusted, the optimal distance threshold is solved, the foreground cloud image of the circular surface of the rivet head is obtained, and the data set for training the network model is established, as shown in the figure. Figure 5 As shown, this embodiment is acquired under the RGB-D camera FM850-E2.

[0103] Step 2: sort the aircraft rivet head height foreground point cloud image and the rivet head circular surface foreground point cloud image obtained in step 1 respectively, annotate the images, and obtain a mask image after the annotation is completed;

[0104] The aircraft rivet head height foreground point cloud images and rivet head circular surface foreground point cloud images in the training image dataset are sorted and renamed in the format of "01.jpg", "02.jpg", "03.jpg"....., and all the images to be annotated are placed in a folder. Secondly, the folder is opened by labelme software, and the aircraft rivet head image samples are sorted and renamed using the "Createpolygons" tool. The sample images are annotated in turn and saved in the folder. According to actual needs, the image labels are divided into rivet head class and rivet circular surface class. The unannotated areas are regarded as irrelevant backgrounds. The .json file generated after annotation is converted into a PASCAL VOC format mask image, and finally the PASCAL VOC format dataset production task is completed.

[0105] Step 3: Establish an improved Deep Lab V3+ convolutional neural network model and train the mask image in step 2;

[0106] The original Deeplab V3+ model uses the Xception series as the feature extraction network. In order to improve the accuracy of feature extraction of the network model, this method uses ResNet 50 with ImageNet pre-trained weights as the backbone feature extraction network. The network model with pre-trained weights can accelerate the convergence of the model during training. In the field of image recognition, the effective extraction of high-level information in the input image is the basis for improving the robustness of image segmentation. In order to meet higher accuracy, more network models tend to have deeper layers, but as the network deepens, the network weight update is unstable during back propagation, resulting in phenomena such as gradient explosion or gradient disappearance. Figure 6 , ResNet network effectively solves this kind of problem. Its residual block adopts skip connection and identity mapping layer, that is, the information features extracted by the previous layer jump to the next layer for feature processing. While the depth of the network model increases, the problem of gradient disappearance is solved, so that the accuracy of feature extraction of the network model is effectively guaranteed. The residual unit can be expressed as follows:

[0107] y z =h(x z )+F(x z , w z )

[0108] x z+1 =f(y1)

[0109] Each residual network structure is composed of a multi-layer structure. Usually, xz is used to represent the input information of the residual unit, and xz+1 represents the output information of the residual unit. The function of obtaining the residual network structure information is expressed by F, where f is the activation function Relu, and the related mapping is expressed by h(xz)=xz;

[0110] The standard convolution in the ASPP module of the original Deeplab V3+ network model is replaced by the depthwise separable convolution. The depthwise separable convolution consists of pointwise convolution and depthwise convolution. Through the depthwise separable convolution operation, the number of parameters and the amount of computation of the network model can be greatly reduced on the basis of less loss of accuracy, thereby improving the training and application efficiency of the model. Specifically, for the standard convolution-related calculation parameter quantity, when the input image size is 12×12×3, a 5×5×3 convolution kernel is used to carry out the convolution to produce an 8×8×1 output. When 256 5×5×3 convolution kernels are used for the convolution operation, an output of 8×8×256 is obtained, and the parameter quantity is: 256×5×5×3=19200. When calculating the parameters related to the depthwise separable convolution, in the depthwise convolution, a 5×5×1 convolution kernel is used, which is the same as and corresponds to the number of channels of the input image. Since the input image has three channels, three 8×8×1 outputs are obtained, and the outputs are spliced ​​to obtain a result of 8×8×3. In the pointwise convolution, in order to obtain an output of 8×8×256 by comparison with the standard convolution, 256 1×1×3 convolutions are used to perform pointwise convolution operations on the depthwise convolution output features, and finally obtain an output of 8×8×256. The depthwise convolution parameters can be calculated by using a 5×5×1 convolution to perform convolution operations on the three channels of the image: 5×5×3=75. After obtaining the depthwise convolution output features, the pointwise convolution parameters can be obtained by performing convolution operations on each point: 256×1×1×3=768. The total parameters of the depthwise separable convolution are: 75+768=843. It can be seen that the number of parameters of depthwise separable convolution is much smaller than that of standard convolution. The schematic diagram of the calculation amount of standard convolution and depthwise separable convolution is as follows: Figure 7 (a) and Figure 7 (b)

[0111] In order to make the network model have better performance, this method adopts the attention mechanism CBAM and adds it to the network model, thereby strengthening the main feature extraction, suppressing invalid features, and greatly improving the accuracy and reducing the number of parameters. When the attention mechanism is in different positions of the network model, the network model will have different performance. Usually, the attention mechanism is placed at the end of the entire network model. This method places the attention mechanism in two different positions. The first one is placed after the 1×1 convolution immediately after the multi-scale feature fusion of the void space pooling pyramid ASPP, that is, before the 4x upsampling. The second attention mechanism is placed at the end of the model, that is, before the 4x upsampling.

[0112] After completing the establishment of the improved Deep Lab V3+ convolutional neural network model, the mask image is input into the improved Deep Lab V3+ convolutional neural network model for training, and the improved Deep Lab V3+ convolutional neural network model is fitted.

[0113] Step 4: Use the improved Deep Lab V3+ convolutional neural network model trained and fitted in step 3 to perform image segmentation on the foreground point cloud image of the rivet head height and the foreground point cloud image of the rivet head circular surface of the riveted sample to be inspected, and obtain the rivet head height and rivet head circular surface images;

[0114] like Figure 8 As shown, a countersunk rivet of standard height is used as a reference object, whose height is constant and the outline is clear. Then, the reference object and the riveted template to be tested are subjected to image segmentation through the improved Deep Lab V3+ convolutional neural network model that is fitted after training, and finally the rivet head height and the rivet head circular surface image are obtained, wherein the rivet head height is obtained by measuring the reference object. In this embodiment, a 120° countersunk rivet specially used for aviation is used as a reference object, and its standard height is 12 mm.

[0115] Step 5: Perform skeleton extraction on the rivet head height obtained in step 4 to obtain the convex amount of the riveted pier head;

[0116] like Fig. 9 As shown in the figure, the transparency of the object contour and its topological structure in the rivet head height image after image segmentation are expressed by a single curve, specifically a skeleton image with a width of 1 pixel. According to the actual height a of the reference object and the pixel value b occupied by the skeleton contour, the actual length of the riveted pier head h = a / b×d is solved, where d is the pixel value occupied by the rivet pier head, that is, the protrusion of the rivet pier head.

[0117] Step 6: The quality of the rivet head can be reflected by detecting the roundness of the rivet head, and then the quality of the riveted parts can be judged. If the force is uneven during the riveting process, the circle of the head projection surface will be irregular and the roundness will be poor. The rivet head circular surface image obtained after image segmentation in step 4 is fitted with a circle using the least squares method, as shown in Fig.10 As shown in the figure, the center of the pier head surface is determined, and then the longest and shortest diameters of the pier head surface are found, and finally the roundness of the riveted pier head is determined. Among them, the determination of the roundness of the rivet pier head adopts the least square circle fitting of the edge points to evaluate the roundness. The least square circle fitting process of the circle is to use the coordinate value of the extracted circle contour to determine the position and radius of the circle center. Find any point in the actual contour so that the sum of the squares of the radial distances from each point on the contour to the circle with the point as the center is the smallest. This point is the center of the least square circle:

[0118] According to the circle formula (xA) 2 +(yB) 2 =R 2 Available,

[0119] R 2 =x 2 -2Ax+A2 +y 2 -2By+B 2

[0120] make:

[0121] a=-2A, b=-2B, c=A 2 +B 2 -R 2

[0122] This gives another form of the equation of the circle:

[0123] x 2 +y 2 +ax+by+c=0 (1)

[0124] According to formula (1), we only need to find a, b, and c to get the parameters of the center and radius:

[0125]

[0126] Specifically, let the sample set on the edge of the rivet head contour in aircraft riveting be [(x i ,y i ), i∈1, 2, 3, .....n], the center of the fitting circle is (A, B), and the distance from the point to the center is d i：

[0127] d i 2 =(X i -A) 2 +(Y i -B) 2

[0128] Let point (X i , Y i ) to the edge of the circle and the square of the radius is:

[0129] δ=d i 2 -R 2 =(X i -A) 2 +(Y i -B) 2 -R 2 =X i 2 +Y i 2 +aX i +bY i +c

[0130] Let Q(a, b, c) be δ 2 The sum of squares:

[0131]

[0132] Since the square difference Q(a, b, c) is greater than or equal to 0, the function has a minimum value greater than or equal to 0, and its maximum value is infinity; Q(a, b, c) with respect to (a, b, c), let the partial derivative equal to 0, then we get the extreme point, and by comparing all the extreme point functions, we can get the minimum value;

[0133]

[0134] Let, constant D = (X i 2 +Y i 2 +bY i +c), the above formula can be transformed into:

[0135]

[0136] but:

[0137]

[0138] Substituting D into the above formula, we get:

[0139]

[0140] Similarly, we can get the partial derivatives of b and c:

[0141]

[0142]

[0143] Rewrite the above equation into an equation form:

[0144]

[0145] According to the ∑ operation rule, the constant is taken out, and because the equation is equal to 0, it can be directly omitted, and the following formula is obtained:

[0146]

[0147] Expand the above formula into the following form:

[0148]

[0149] According to the summation formula, we get Substituting into the equation we get:

[0150]

[0151] So solve the system of equations, eliminate c, equation (2) have to

[0152]

[0153] have to

[0154]

[0155] make

[0156]

[0157]

[0158]

[0159]

[0160]

[0161] Then, we get the equation:

[0162]

[0163] therefore:

[0164]

[0165]

[0166]

[0167] Then solve the least squares circle center and radius:

[0168]

[0169] After the center of the fitting circle is determined, all points on the rivet head contour are obtained, and the distance from any pixel point of the contour through the center of the fitting circle to any pixel point at the other end of the contour is solved to obtain the riveted head diameter information set, sort the information in the set, and filter out the maximum diameter D max and the shortest diameter D min , by collecting information and formula Solve for the average diameter D ave ,

[0170] Therefore, the circular deviation value δ oor for

[0171]

[0172] The corresponding roundness △R is

[0173]

[0174] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0175] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting the quality of aircraft riveted pier heads based on convolutional neural networks, characterized in that: The following steps are involved: Step 1: Obtain the height foreground cloud image of the aircraft rivet pier head and the foreground cloud image of the circular surface of the rivet pier head of the standard sample through the acquisition device to establish a training image data set; Step 2: sort the aircraft rivet head height foreground point cloud image and the rivet head circular surface foreground point cloud image obtained in step 1 respectively, annotate the images, and obtain a mask image after the annotation is completed; Step 3: Establish an improved Deep Lab V3+ convolutional neural network model and train the mask image in step 2; Step 4: Use the improved Deep Lab V3+ convolutional neural network model trained and fitted in step 3 to perform image segmentation on the foreground point cloud image of the rivet head height and the foreground point cloud image of the rivet head circular surface of the riveted sample to be inspected, and obtain the rivet head height and rivet head circular surface images; Step 5: Perform skeleton extraction on the rivet head height obtained in step 4 to obtain the convex amount of the riveted pier head; Step 6: The circular surface image of the rivet head obtained after image segmentation in step 4 is fitted with a circle using the least squares method to determine the center of the head surface, and then the longest and shortest diameters of the head surface are found, and finally the roundness of the riveted head is determined.

2. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 1 is characterized in that: The step 1 is specifically as follows: Take photos perpendicular to the rivet direction, adjust the distance threshold between the acquisition device and the rivet, and obtain the foreground cloud image at the height of the rivet head; Take photos parallel to the rivet direction, adjust the distance threshold between the acquisition device and the rivet, and obtain the foreground cloud image of the circular surface of the rivet head.

3. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 1 is characterized in that: The step 2 is specifically as follows: After sorting and renaming the aircraft rivet head image samples, the sample images are annotated in turn. According to actual needs, the image labels are divided into rivet head class and rivet circle surface class, and the unlabeled areas are regarded as irrelevant background; the .json file generated after annotation is converted into a PASCAL VOC format mask image, and finally the PASCAL VOC format dataset production task is completed.

4. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 3 is characterized in that: The step 3 is specifically as follows: RestNet50 with ImageNet pre-trained weights is used as the backbone feature extraction network to accelerate the convergence of the network model training and optimize the feature extraction network; The standard convolution in the atrous spatial convolution pooling pyramid ASPP is replaced with a depthwise separable convolution to improve the training and application efficiency of the model, thereby improving the atrous spatial convolution pooling pyramid ASPP. In the atrous spatial convolutional pooling pyramid ASPP, the attention mechanism CBAM is used to add two positions to enhance the extraction of main features; the improved DeepLabV3+ convolutional neural network model is established, the mask image is input into the improved DeepLabV3+ convolutional neural network model for training, and the improved DeepLabV3+ convolutional neural network model is fitted.

5. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 4 is characterized in that: In step 3, the attention mechanism CBAM is used to add two positions in the dilated spatial convolutional pooling pyramid ASPP to enhance the main feature extraction, specifically: The first attention mechanism CBAM is placed after the multi-scale feature fusion of the atrous spatial pooling pyramid ASPP, and the second attention mechanism CBAM is placed at the end of the atrous spatial pooling pyramid ASPP.

6. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 1 is characterized in that: The step 4 is specifically as follows: A target of standard height is used as a reference object, and then the reference object and the riveted template to be tested are subjected to image segmentation through the improved DeepLabV3+ convolutional neural network model that is fitted after training. Finally, the rivet head height and the rivet head circular surface image are obtained, where the rivet head height is obtained by measuring the reference object.

7. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 1 is characterized in that: The step 5 is specifically as follows: The transparency and topological structure of the object contour in the rivet head height image after image segmentation are expressed by a single curve, specifically a skeleton graph with a width of 1 pixel. According to the actual height a of the reference object and the pixel value b occupied by the skeleton contour, the actual length of the riveted pier head is solved h=a / b×d, where d is the pixel value occupied by the rivet pier head, that is, the protrusion of the rivet pier head.

8. The method for detecting the quality of aircraft riveted pier heads based on convolutional neural network according to claim 1 is characterized in that: The step 6 is specifically as follows: According to the circle formula (xA) 2 +(yB) 2 =R 2 Available, R 2 =x 2 -2Ax+A 2 +y 2 -2By+B 2 make: a=-2A,b=-2B,c=A 2 +B 2 -R 2 This gives another form of the equation of the circle: x 2 +y 2 +ax+by+c=0 (1) According to formula (1), we only need to find a, b, and c to get the parameters of the center and radius: Specifically, let the sample set on the edge of the rivet head contour in aircraft riveting be [(x i ,y i ), i∈1, 2, 3, ........n], the center of the fitting circle is (A, B), and the distance from the point to the center is d i： d i 2 =(X i -A) 2 +(Y i -B) 2 Let point (X i , Y i ) to the edge of the circle and the square of the radius is: δ=d i 2 -R 2 =(X i -A) 2 +(Y i -B) 2 -R 2 =X i 2 +Y i 2 +aX i +bY i +c Let Q(a, b, c) be δ 2 The sum of squares: Since the square difference Q(a, b, c) is greater than or equal to 0, the function has a minimum value greater than or equal to 0, and its maximum value is infinity; Q(a, b, c) with respect to (a, b, c), let the partial derivative equal to 0, then we get the extreme point, and by comparing all the extreme point functions, we can get the minimum value; Let, constant D = (X i 2 +Y i 2 +bY i +c), the above formula can be transformed into: but: Substituting D into the above formula, we get: Similarly, we can get the partial derivatives of b and c: Rewrite the above equation into an equation form: According to the ∑ operation rule, the constant is taken out, and because the equation is equal to 0, it can be directly omitted, and the following formula is obtained: Expand the above formula into the following form: According to the summation formula, we get Substituting into the equation we get: So solve the system of equations, eliminate c, equation (2) have to have to make Then, we get the equation: therefore: Then solve the least squares circle center and radius: After the center of the fitting circle is determined, all points on the rivet head contour are obtained, and the distance from any pixel point of the contour through the center of the fitting circle to any pixel point at the other end of the contour is solved to obtain the riveted head diameter information set, sort the information in the set, and filter out the maximum diameter D max and the shortest diameter D min , by collecting information and formula Solve for the average diameter D ave , Therefore, the circular deviation value δ oor for The corresponding roundness △R is

Images