Method and system for rapidly identifying and counting surface fasteners of aviation parts

Abstract

The present application belongs to the field of aviation parts measurements. Provided are a method for rapidly identifying and counting surface fasteners of aviation parts. The method comprises: first, collecting parts surface images by using a high-definition industrial camera, and labeling the parts surface images, and then establishing a fastener and hole location image database; building a deep neural network model for fastener and hole location identification; then training the deep neural network model by using the fastener and hole location image database, so as to obtain a fastener and hole location identification model; and performing lightweight processing on the fastener and hole location identification model, and then obtaining a final fastener and hole location identification model. During an operating process, automatic marking and counting of various types of fasteners and hole locations can be completed by means of collecting parts surface images in real time and by using a fastener and hole location identification model; and rapid batch analysis and processing can be performed on the parts surface images, thereby significantly improving the efficiency and accuracy of detecting the number of fasteners and the number of hole locations.

Description

A method and system for rapid identification and statistical analysis of surface fasteners of aviation parts Technical Field

[0001] The present application relates to the field of aviation component measurement, and in particular to a method and system for quickly identifying and counting fasteners on the surface of aviation components. Background Art

[0002] The rapid development of the aviation manufacturing industry has placed higher demands on the efficiency and quality of aircraft assembly. Aircraft are composed of numerous parts connected by fasteners such as rivets and bolts. Fastener connection is a critical process step in the aircraft manufacturing and assembly process, directly impacting the aircraft's safety and reliability. The number and variety of fasteners on aviation parts are numerous, and the inspection of the number of fasteners installed and the number of holes drilled still relies on manual visual counting, which is subject to problems such as low efficiency, frequent interference, and large errors. To achieve efficient identification and detection of fastener quantities, it is urgent to adopt digital and information-based measurement and inspection technologies for improvement and innovation.

[0003] Image recognition and deep learning technologies offer a new solution for the rapid and reliable detection of fastener and hole counts on aviation components. With the rapid development of deep learning technology, deep neural networks are being used to learn and extract features from key targets, such as fasteners and holes, enabling rapid identification and counting of fasteners and holes. By capturing images of the surfaces of aviation components with high-definition industrial cameras, a database of fastener and hole images of different types of fasteners and holes is established. For the recognition of small targets, such as fasteners, a deep neural network model with a multi-scale feature extraction module is designed and constructed. By utilizing lightweight neural networks, the system ultimately achieves automated and digital measurement of the assembly process, from component surface image input to rapid feedback of fastener and hole count statistics.

[0004] Inspection of the number of fasteners installed and holes drilled on the surfaces of aviation parts still relies on manual visual counting, which is inefficient and prone to errors. To meet the stringent requirements for aircraft assembly efficiency and quality, next-generation image recognition technology is being used to quickly and accurately verify the number of fasteners and holes drilled on parts on-site, thereby improving product quality.

[0005] Summary of the Invention

[0006] The purpose of this application is to provide a method and system for quickly identifying and counting fasteners on the surface of aviation parts, so as to solve the problem that the number of fasteners installed and the number of holes made in aviation parts still need to be checked manually by visual methods.

[0007] The technical solution of this application is: a method for rapid identification and statistical analysis of surface fasteners of aviation parts, including:

[0008] Use a high-definition industrial camera to capture a certain amount of component surface images until the acquisition range covers the entire surface of the aviation component. Classify and locate the various types of fasteners and holes in the captured component surface images, and establish a fastener and hole image database based on the classification and location annotation information.

[0009] A deep neural network model for fastener and hole recognition is built based on a database of fastener and hole images. The deep neural network model can identify the classification and location feature information of fasteners and holes in component surface images.

[0010] The deep neural network model is trained using a database of fastener and hole images until the training loss of the deep neural network model converges and the test accuracy is stable, thus obtaining a fastener and hole recognition model.

[0011] The neural network model is lightweighted for the fastener and hole position recognition model, and redundant parameters in the fastener and hole position recognition model are eliminated to obtain the final fastener and hole position recognition model;

[0012] Real-time acquisition of component surface images. Utilizing fastener and hole recognition models, rapid classification and location identification of fasteners and holes in the component surface images. Marking each type of fastener and hole with a different color box on the component surface image to obtain a marked component surface image.

[0013] Count the number of each type of fasteners and holes, and feed the statistical results and the marked component surface images back to the data management system for storage, management and display.

[0014] Preferably, the fastener and hole position recognition model adopts the convolutional neural network architecture YOLOv5, and the design method for lightweighting the fastener and hole position recognition model is:

[0015] For the features and weights of the nth layer, set the L1 norm, filter the channels with smaller L1 norms of features and weights, and calculate the L1 norm of the features and weight channels of the nth layer as follows:

[0016] Where, Represents the total number of feature channels filtered from the nth layer of the model, Represents the nth layer of the model The L1 norm of the output feature channels, Represents the total number of weight channels filtered from the nth layer of the model, Represents the nth layer of the model L1 norm of weight channels;

[0017] Then, the L1 norms of the output feature channels and weight channels are sorted from large to small, and a certain amount of output feature channels and weight channels with smaller L1 norms are removed.

[0018] Preferably, the fastener and hole position recognition model adopts the convolutional neural network architecture YOLOv5, and a multi-scale feature extraction module is provided in the fastener and hole position recognition model. The design method of the multi-scale feature extraction module is:

[0019] Given input features After a convolution operation, the features are obtained After that, the feature operations of different scales are performed in four ways. The feature calculation formula extracted in each way is as follows: F2=F*W2 F4=F*W4

[0020] In the formula, * represents the convolution operation, represents point-by-point element addition, u is the upsampling function, W1, W2, W3, W4 are the weights of each path, and the four-path features are fused to obtain the output feature F out , calculated as follows:

[0021] Where W5 is the weight of the path, and Concat represents the cascade function.

[0022] Preferably, when classifying and positioning each type of fastener and hole in each component surface image, a corresponding label document in XML format is generated, and the fastener and hole image database includes the component surface image and the corresponding label document.

[0023] Preferably, a data enhancement module is provided in the fastener and hole position recognition model, and the data enhancement module includes image flipping, scaling, translation and random cropping.

[0024] Preferably, before photographing the surface of an aviation component, the aircraft surface is manually grid-marked; when shooting, a high-definition industrial camera captures the component surface image within a grid each time; before image analysis, a statistical limit range of the component surface image is set according to the physical grid, and after identifying the fasteners and hole positions in the component surface image, a polygonal mesh is formed in combination with the boundary values ​​of the statistical limit range, and the fastener and hole position information exceeding the statistical limit boundary values ​​is removed.

[0025] As a specific embodiment, a rapid identification and statistics system for fasteners on the surface of aviation parts adopts the above-mentioned method, including a high-definition industrial camera, a computing power server and a data management system. The high-definition industrial camera can collect component surface images in real time and transmit the component surface images to the computing power server via a wired network. The high-definition industrial camera is correspondingly provided with an auxiliary light source. The computing power server includes a data enhancement module and a fastener and hole position recognition model. The data enhancement module can increase the number of component surface images in the fastener and hole position image database. The fastener and hole position recognition model can respectively identify various types of fastener and hole position information.

[0026] The present application discloses a method for rapid identification and statistics of fasteners on the surface of aviation parts. The method first uses a high-definition industrial camera to capture part surface images, and after annotating the part surface images, establishes a fastener and hole image database, builds a deep neural network model for fastener and hole identification, and then uses the fastener and hole image database to train the deep neural network model until the training loss value of the deep neural network model converges and the test accuracy is stable, thereby obtaining a fastener and hole identification model; after lightweighting the fastener and hole identification model, a final fastener and hole identification model is obtained; during the working process, by real-time acquisition of part surface images, the fastener and hole identification model can be used to automatically mark and count various types of fasteners and holes; thereby, rapid batch analysis and processing of part surface images can be achieved, greatly improving the efficiency and accuracy of fastener and hole quantity detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions provided by this application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0028] Figure 1 is a schematic diagram of the overall process of this application;

[0029] FIG2 is a schematic diagram of the overall process of the multi-scale feature extraction module of this application;

[0030] FIG3 is a schematic diagram of the overall structure of the system corresponding to the method of the present application. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] A method for quickly identifying and counting fasteners on the surface of aviation parts includes the following steps:

[0033] As shown in FIG1 , step S100 , collecting images of the component surface

[0034] A certain amount of component surface images are collected using high-definition industrial cameras until the collection range covers the entire surface of the aviation component to ensure the comprehensiveness of the data; various types of fasteners and holes in the collected component surface images are manually classified and located and annotated. The annotated information includes rivets, bolts, and hole locations, etc. A fastener and hole image database is established based on the classification and location annotation information.

[0035] The types of fasteners commonly included in aircraft parts are rivets and bolts. This embodiment marks the three types of information of rivets, bolts, and hole positions respectively.

[0036] When classifying and labeling each type of fastener and hole in each component surface image, a corresponding label document in XML format is generated. This document records the type and coordinate location of the fastener and hole in the image. This coordinate location information is the relative coordinates of the fastener or hole within the captured image area, not the coordinates of the entire machine. The fastener and hole image database includes component surface images and corresponding label documents. The label document format is not limited to XML and can be adjusted according to actual needs.

[0037] In this way, both the component surface image and label document can be input into the system for reading, facilitating subsequent training.

[0038] Step S200: Deep neural network model establishment

[0039] A deep neural network model for fastener and hole recognition is built based on the fastener and hole image database. The deep neural network model can identify the classification and positioning feature information of fasteners and holes in the surface image of components.

[0040] The deep neural network model uses the convolutional neural network architecture YOLOv5. The convolutional neural network architecture YOLOv5 is characterized by fast speed and high accuracy. It can detect and identify multiple objects in an image in a short time, and is more suitable for the situation of this application.

[0041] To improve the diversity and scale of input data, a data augmentation module is set up in the fastener and hole recognition model. The data augmentation module includes operations such as image flipping, scaling, translation, and random cropping. It can effectively increase the number of images in the fastener and hole recognition model, enabling the fastener and hole recognition model to achieve better detection performance.

[0042] Step S300: Deep neural network model training

[0043] The fastener and hole image database is input into the deep neural network model for forward calculation to obtain the training loss value between the predicted value and the true value of the label. The training loss value is then backpropagated to the deep neural network weights and the corrected weights are updated until the training loss value of the deep neural network model converges and the test accuracy is stable, thus realizing deep neural network training.

[0044] Since the image clarity and distance may vary during actual shooting by high-definition industrial cameras, how to accurately identify the characteristics of fasteners and holes is the key to quickly counting fasteners and holes in component surface images.

[0045] To address this issue, a multi-scale feature extraction module is set up in the fastener and hole position recognition model. The design method of the multi-scale feature extraction module is as follows:

[0046] As shown in Figure 2, given the input features After a convolution operation, the features are obtained After that, the feature operations of different scales are performed in four ways. The feature calculation formula extracted in each way is as follows: F2=F*W2 F4=F*W4

[0047] In the formula, * represents the convolution operation, represents point-by-point element addition, u is the upsampling function, W1, W2, W3, W4 are the weights of each path, and the four-path features are fused to obtain the output feature F out , calculated as follows:

[0048] Where W5 is the weight of the path, and Concat represents the cascade function, which is used to stack multiple layers of feature channels.

[0049] The multi-scale feature extraction module achieves multi-scale feature extraction by fusing each set of features with convolution kernels of different sizes, thereby enabling accurate extraction of specific features at different spatial scales and improving the deep neural network model's recognition and detection accuracy for fasteners and holes.

[0050] Step S400: Lightweighting the neural network model

[0051] The neural network model of the fastener and hole recognition model is lightweighted, and redundant parameters in the fastener and hole recognition model are eliminated to obtain the final fastener and hole recognition model.

[0052] When identifying fasteners in component surface images, since the component surface images may contain other features, such as beams, ribs, etc., these features are invalid features and will affect the recognition of fasteners and holes. Therefore, lightweight training of the fastener and hole recognition model is required.

[0053] The design method for lightweighting the fastener and hole identification model is as follows:

[0054] For the features and weights of the nth convolutional layer, set the L1 norm, filter the channels with smaller L1 norms of features and weights, and calculate the L1 norms of the nth layer features and weight channels as follows:

[0055] Where, Represents the total number of feature channels filtered from the nth layer of the model, Represents the nth layer of the model The L1 norm of the output feature channels, Represents the total number of weight channels filtered from the nth layer of the model, Represents the nth layer of the model The L1 norm of the weight channels.

[0056] The smaller the L1 norm, the greater the probability that the feature channel or weight channel represents other features. Therefore, the L1 norms of the output feature channels and weight channels are sorted from large to small, and a certain number of output feature channels and weight channels with small L1 norms are removed. The specific number can be 10%, 15%, etc., depending on actual needs. This design effectively removes redundant feature parameters other than fasteners, thereby reducing the model's computational complexity and improving the detection speed of fasteners and holes.

[0057] Step S500: Real-time recognition of component surface images

[0058] Real-time acquisition of component surface images. High-definition industrial cameras will continuously capture multiple sets of component surface images until the entire aviation component surface image is captured. Fastener and hole recognition models are used to quickly classify and locate fasteners and holes in the component surface image. The results of classification and location recognition can quickly locate and accurately count fasteners and holes at different locations on the component surface. Different types of fasteners and holes are marked on the component surface image with different color boxes to obtain the marked component surface image.

[0059] When high-definition industrial cameras photograph the surface of aircraft parts, there is generally a certain overlap between adjacent photos, and rivets, bolts, and hole features may appear in the overlapping areas, which can easily lead to misjudgment of the number when counting the features.

[0060] To address this issue, the surfaces of aircraft parts are manually gridded and marked. Fastener features are avoided from being drawn on the grid lines when drawing the grid. High-definition industrial cameras capture the surface image of the part within a grid each time. The camera tries to focus on the center of the grid and capture an image slightly larger than the grid to form an image of the part surface. At this point, only features outside the grid need to be removed. This effectively avoids duplication when counting the number of fasteners and holes.

[0061] The method for removing fastener and hole information outside the grid is as follows: before image analysis, the statistical limit range of the component surface image is set according to the physical grid. After the fasteners and holes in the component surface image are identified, the boundary values ​​of the statistical limit range are combined to form a polygonal grid. The fastener and hole information exceeding the statistical limit boundary values ​​are removed, and the desired component surface image is obtained. Of course, when high-definition industrial cameras shoot at different rectangular grid positions, the distance difference is relatively small. For example, when shooting a fastener at a certain position at a distance of 1m, the distance for shooting a fastener at another position may be 1.1m or 0.9m, rather than 2m or 3m, to ensure shooting accuracy. Existing high-definition industrial cameras all have a focus function, and the shooting scale generally does not differ much.

[0062] This design can effectively avoid feature duplication of fasteners and holes, ensuring accurate statistics of the number of fasteners and holes.

[0063] Step S600: Count the number of fasteners and holes

[0064] Count the number of each type of fasteners and holes, and feed the statistical results and the marked component surface images back to the data management system for storage, management and real-time display, thereby improving the level of digital management of the assembly process and reducing the labor intensity of operators.

[0065] This application first uses a high-definition industrial camera to capture component surface images, and after annotating the component surface images, establishes a fastener and hole image database, builds a deep neural network model for fastener and hole identification, and then uses the fastener and hole image database to train the deep neural network model until the training loss value of the deep neural network model converges and the test accuracy is stable, thereby obtaining a fastener and hole identification model; after lightweighting the fastener and hole identification model, the final fastener and hole identification model is obtained; during the working process, by real-time acquisition of component surface images, the fastener and hole identification model can be used to automatically mark and count various types of fasteners and holes. This enables rapid batch analysis and processing of component surface images, greatly improving the efficiency and accuracy of fastener and hole quantity detection, overcoming the problems of low efficiency, high interference, and large errors caused by manual visual counting, greatly reducing the manpower and time investment in the fastener quantity detection process, ensuring the stability and consistency of fastener installation quantity detection on aviation component surfaces, and improving the quality of digital assembly.

[0066] As a specific embodiment, a system for rapidly identifying and counting fasteners on the surfaces of aviation parts is also included. As shown in Figure 3, it includes a high-definition industrial camera, a computing server, and a data management system. The high-definition industrial camera can capture component surface images in real time and transmit them to the computing server via a wired network. The high-definition industrial camera is equipped with an auxiliary light source to ensure brightness. The computing server includes a data enhancement module and a fastener and hole position recognition model. The data enhancement module can increase the number of component surface images in the fastener and hole position image database, and the fastener and hole position recognition model can separately identify various types of fastener and hole position information. By setting up this system, accurate statistics of fasteners and holes on the surfaces of aircraft parts can be achieved without manual visual inspection, thereby effectively improving work efficiency and product quality.

[0067] Finally, a few points should be explained: First, in the description of this application, it should be noted that, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense, and may refer to mechanical or electrical connections, internal communication between two components, or direct connection. "Up," "down," "left," and "right" are only used to indicate relative positional relationships. When the absolute positions of the objects being described change, the relative positional relationships may also change.

[0068] Secondly: The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to conventional designs. The same embodiment and different embodiments of the present invention may be combined with each other without conflict.

[0069] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for rapid identification and statistics of surface fasteners of aviation parts, characterized in that: include: Use high-definition industrial cameras to collect a certain amount of component surface images until the collection range covers the entire surface of aviation components, classify and locate various types of fasteners and holes in the collected component surface images, and establish a fastener and hole image database based on the classification and location annotation information; A deep neural network model for fastener and hole recognition is built based on the fastener and hole image database. The deep neural network model can identify the classification and positioning feature information of fasteners and holes in the surface image of parts. The deep neural network model is trained using the fastener and hole image database until the training loss value of the deep neural network model converges and the test accuracy is stable, thereby obtaining a fastener and hole recognition model; Lightweight the neural network model of the fastener and hole position recognition model, remove redundant parameters in the fastener and hole position recognition model, and obtain the final fastener and hole position recognition model; Collect component surface images in real time, use fastener and hole position recognition models to quickly classify and locate fasteners and holes in component surface images, and use different color boxes to mark different types of fasteners and holes on the component surface images to obtain marked component surface images; Count the number of fasteners and holes of each type, and feed the statistical results and the marked surface images of components to the data management system for storage, management and display; The fastener and hole position recognition model adopts the convolutional neural network architecture YOLOv5, and the design method for lightweighting the fastener and hole position recognition model is as follows: For the features and weights of the nth layer, set the L1 norm, select the channels with smaller L1 norms of features and weights, and calculate the L1 norm R(F) of the feature channels of the nth layer. n ) and the weight channel L1 norm R(W n )for: In the formula, Represents the total number of feature channels filtered from the nth layer of the model, Represents the nth layer of the model The L1 norm of the output feature channels, Represents the total number of weight channels filtered from the nth layer of the model, Represents the nth layer of the model The L1 norm of the weight channels; Then, the L1 norms of the output feature channels and weight channels are sorted from large to small, and a certain amount of output feature channels and weight channels with smaller L1 norms are removed; The fastener and hole position recognition model adopts the convolutional neural network architecture YOLOv5, and a multi-scale feature extraction module is provided in the fastener and hole position recognition model. The design method of the multi-scale feature extraction module is as follows: Given input feature F in ∈R C×H×W , where ∈ is a symbol, R is a real number, C is the feature depth, H is the feature height, and W is the feature width; after one convolution operation, the feature F∈R is obtained. C×H×W Afterwards, the feature operations of different scales are performed in four paths, and the feature calculation formula extracted in each path is as follows: <h2 style=";text-align:left;direction:ltr">F2=F*W2<h2 style=";text-align:left;direction:ltr"> F4=F*W4 In the formula, * represents the convolution operation, represents point-by-point element addition, u is the upsampling function, Avgpool is the average pooling function, F1 is the first feature, F2 is the second feature, F3 is the third feature, F4 is the fourth feature, W1, W2, W3, W4 are the weights of each channel, and the obtained four-way features are fused to obtain the output feature F out , calculated as follows: Where W5 is the weight of the path, and Concat represents the cascade function.

2. The method for rapid identification and counting of fasteners on the surface of aviation parts according to claim 1, characterized in that: When various types of fasteners and holes in each component surface image are classified and positioned, a corresponding label document in XML format is generated. The fastener and hole image database includes the component surface image and the corresponding label document.

3. The method for rapid identification and counting of fasteners on the surface of aviation parts according to claim 1, characterized in that: A data enhancement module is provided in the fastener and hole position recognition model, and the data enhancement module includes image flipping, scaling, translation and random cropping.

4. The method for rapid identification and counting of fasteners on the surface of aviation parts according to claim 1, characterized in that: Before photographing the surface of aviation parts, the aircraft surface is manually gridded and marked; When shooting, the high-definition industrial camera takes a picture of the component surface within a grid each time; before image analysis, the statistical limit range of the component surface image is set according to the physical grid. After the fasteners and holes in the component surface image are identified, a polygonal mesh is formed in combination with the boundary values ​​of the statistical limit range, and the fastener and hole information that exceeds the statistical limit boundary values ​​is removed.

5. A system for rapid identification and statistics of fasteners on the surface of aviation parts, using the method as described in any one of claims 1 to 4, characterized in that: It includes a high-definition industrial camera, a computing power server and a data management system. The high-definition industrial camera can collect component surface images in real time and transmit the component surface images to the computing power server through a wired network. The high-definition industrial camera is correspondingly provided with an auxiliary light source. The computing power server includes a data enhancement module and a fastener and hole position recognition model. The data enhancement module can increase the number of component surface images in the fastener and hole position image database. The fastener and hole position recognition model can respectively recognize various types of fasteners and hole position information to obtain fastener and hole position recognition results. The data management system can record and manage the fastener and hole position recognition results.

Images