A visual detection method for cracks in aviation industry structural parts

Abstract

The present invention belongs to the field of photoelectric detection technology for temporary security, and relates to a method for visual detection of cracks in aviation industry structural parts. A method for visual detection of cracks in aviation industry structural parts is proposed for the rivets and bolts at the connection parts of aviation industry structural parts, which may cause fatigue cracks during use due to increased bearing capacity and complex stress-strain states. A method for visual detection of cracks in aviation industry structural parts is proposed, and a two-stage method combining machine learning semantic segmentation and small sample segmentation is used to effectively solve the problem of small samples caused by the low probability of cracks in aircraft in the aviation field. The semantic segmentation method is used to identify the basic characteristics of the cracks, and the small sample method is used to avoid the overfitting problem that may be caused by a large number of parameters, thereby enhancing the robustness and generalization of the model. In order to avoid the problem of difficulty in convergence caused by the mutual influence of two-stage network parameter learning, a corresponding two-stage training mode is proposed, and different objective functions are used to enable the model to converge quickly, which can be used in various industrial crack automation detection processes.

Description

Technical Field

[0001] The invention belongs to the technical field of on-site security photoelectric detection, and in particular relates to a method for visually detecting cracks in aviation industry structural parts. Background Art

[0002] In recent years, the importance of aviation technology has become increasingly prominent, and there is an urgent need for a diversified, cross-domain, three-dimensional, coordinated, and intelligent ground security technology system for the needs of defense, protection, production, safety, and rescue in the ground space. Aircraft are the main means of transportation in low-altitude airspace (an important part of ground security), and their performance and safety are crucial. In recent years, while the structure of aircraft has become more complex and the functions have become more complete, the working environment of aircraft has become more severe, the load of the structure has increased, and the stress-strain state has become more complex, which has put higher requirements on the integrity of the aircraft structure. Fatigue strength, as an important part of the structural integrity of aircraft, has also received more widespread attention and attention in recent years, and flight accidents caused by structural fatigue damage have also occurred from time to time. Among the fatigue of all aircraft structures, fatigue of connection parts plays an extremely important role, such as the long life of riveted and bolted connections.

[0003] Faced with many catastrophic accidents caused by structural fatigue problems, in order to determine the durability / damage tolerance performance of aircraft structures and ensure that the structures have sufficient fatigue strength, aircraft structural durability / damage tolerance tests are required. By detecting and monitoring possible fatigue cracks on aircraft structures, and observing and studying the formation and expansion process of cracks, and then understanding the fatigue damage mechanism, this is of great significance for verifying the correctness of durability / damage tolerance design methods, discovering weak parts of aircraft structures, improving structural design and manufacturing processes, and formulating inspection and maintenance outlines.

[0004] Cracks generated during fatigue testing of aviation industry structural parts can be detected using various non-destructive testing technologies (NDT). However, NDT often cannot flexibly cope with the diversity of defects and requires a lot of manual interpretation. In recent years, machine learning methods have been widely used in crack detection in various environments due to their accuracy and robustness. In the existing technology, an enhanced RCNN network is used to simultaneously detect, segment and classify in-situ damage of aircraft engine blades; there is also a transfer learning-based X-ray image of aviation composite materials for defect detection; and there is also a UNet network used to explore the crack propagation law in fatigue testing of rolled thin plates.

[0005] However, the above methods often have the following problems during detection: 1) They cannot cope with the problem of small samples caused by the difficulty in obtaining crack samples. The number of rivets and bolts on a large aircraft may reach more than one million. The image backgrounds of rivets and bolts on different parts of the aircraft may be quite different. The number of training samples available is limited, and the number of samples is unbalanced and insufficient. At present, deep neural networks with good detection effects are trained under the premise that there are sufficient training samples and the difference between training images and test images is not large. 2) Model universality and domain adaptation issues. While deep neural networks have high detection accuracy, they also have problems with poor transferability and universality. When the structure and semantic information of the test image are similar to those of the training image, the detection effect is often better. However, when there is a large difference in the image structure and semantic information, the detection effect will be greatly affected. In the scenarios of different models of aircraft and different types of aircraft, how the network maintains accuracy involves the problem of domain adaptation of the model.

[0006] Therefore, it is necessary to provide a visual detection method for cracks in aviation industry structural parts in order to solve the above problems. Summary of the invention

[0007] The technical problems to be solved by the present invention are:

[0008] In order to better solve the problem of intelligent detection of cracks in aviation industry structural parts and ensure the fatigue reliability of aircraft riveted structures and the safety of the entire machine structure, the present invention provides a visual detection method for cracks in aviation industry structural parts. The method consists of three parts: a basic learner UNet++, a prototype extraction module, and a meta-learner. First, the semantic segmentation method is used to train the basic learner; second, the prototype is extracted using the prediction results of the basic learner and the corresponding real labels; third, the small sample learning method is used to train the meta-learner, which uses the prototype and the intermediate features of the basic learner as input. Specifically, the purpose of the present invention is to improve the following aspects:

[0009] A technical solution provided by the present invention is:

[0010] A method for visually detecting cracks in aviation industry structural parts, the method steps are as follows:

[0011] Step 1: Using the base learner, given an image x, use the base learner to learn from X 4,0 The layer extracts the intermediate image features, expressed as:

[0012] f=ε(x),

[0013] Where ε represents the base learner encoder and f is X 4,0 Layer image features.

[0014] Step 2: Use the small sample proxy prototype extraction method to extract the prototype v using the support set image mask m and the support set image feature f:

[0015] v=F pool (f⊙I(m)),

[0016] where ⊙ denotes the Hadamard product, I denotes the scale, and F pool represents mask average pooling;

[0017] Step 3: Take the intermediate image feature f obtained from the base learner encoder ε of image x as the input of the decoder φ to obtain the prediction result p:

[0018] p=φ(f),

[0019] Compare the prediction result p with the support set image mask m to obtain the refined mask M α ,M β :

[0020]

[0021] M β =mM α ,

[0022] Where (x, y) represents the two-dimensional coordinates of the mask;

[0023] Step 4: Use the refined mask M α ,M β And image features f get the proxy prototype v l ,l={α,β}:v l =F pool (f⊙I(M l )),l={α,β};

[0024] Step 5: Expand the spatial scale of the prototype v and convert the query set image x q Input the base learner encoder ε in step 1 to obtain the query set image features f q , the extended prototype v and f q Splicing, denoted as F guide , and obtain the comprehensive feature f s,q :

[0025] f q =ε(x q ),

[0026] f s,q =F guide (v,f q );

[0027] Step 6: Use the query set image feature fq The activation graph A is calculated with two proxy prototypes α ,A β ;

[0028] Step 7: Combine comprehensive features f s,q With the activation feature map A α ,A β Get the final feature f final , input the final feature into the decoder D, and obtain the final meta-learner prediction result p meta :

[0029]

[0030] p meta =D(f final );

[0031] Step 8: Input the query set image into the base learner encoder ε to obtain f q Then the decoder is used to obtain the query set image prediction result p base , p base And the meta-learning prediction result p meta Perform logical operations to obtain the final prediction result p final .

[0032] A further technical solution of the present invention is: in step 1, the basic learner is trained using the UNet++ deep supervision method:

[0033]

[0034] Where L i is the prediction graph loss obtained by the i-th layer feature of UNet++, and the coefficient η i ≡1, feature predictions of all sizes are treated equally.

[0035] A further technical solution of the present invention is: in step 6, the query set feature f is calculated q The cosine distance between the two agent prototypes is the activation map A. α ,A β :

[0036]

[0037] A further technical solution of the present invention is: in step 7, the decoder D includes a dilated convolutional pooling pyramid ASPP and two residual modules.

[0038] A further technical solution of the present invention is: in step 8, the logical operation is a logical AND operation:

[0039] p final=p base ∪p meta .

[0040] A further technical solution of the present invention is: using an industrial camera to collect time series crack images under different stress states.

[0041] Beneficial Effects

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] 1. The present invention provides a method for visual detection of cracks in aviation industry structural parts. It uses a two-stage method combining machine learning semantic segmentation and small sample segmentation to effectively alleviate the small sample problem caused by the low probability of cracks in aircraft in the aviation field.

[0044] 2. The present invention provides a visual crack detection method for aviation industry structural parts. Compared with the existing crack detection technology research, which mostly stays on the more traditional image processing algorithm, has low robustness and overfitting problems due to insufficient data. The semantic segmentation method is used to identify the basic characteristics of the crack, and then the small sample method is used to avoid the overfitting problem that may be caused by a large number of parameters, while enhancing the robustness and generalization of the model.

[0045] 3. The present invention provides a visual crack detection method for aviation industry structural parts. Compared with the existing deep learning crack detection method, which only targets a single scene and has poor transferability to scenes with complex and changeable backgrounds, a corresponding two-stage training mode is proposed to avoid the problem of mutual influence of two-stage network parameter learning leading to difficulty in convergence. Different objective functions are used to enable the model to converge quickly and the algorithm segmentation accuracy is higher. The invention can be used in the automated detection process of various industrial cracks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Attached Figure 1 The present invention is a flow chart of a method for visually detecting cracks in aviation industry structural parts. DETAILED DESCRIPTION

[0047] In order to more clearly illustrate the technical solution implemented by the present invention, the various modules required in the embodiment description are briefly introduced below. Obviously, the accompanying drawings described below are only flowcharts of the present invention. For ordinary technicians in this field, they can also be expanded based on this accompanying drawing without paying creative labor. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but cannot be understood as limiting the present invention.

[0048] A technical solution provided by this embodiment is:

[0049] A method for visually detecting cracks in aviation industry structural parts, the method steps are as follows:

[0050] Step 1: Use UNet++ as the base learner. Due to the changes in depth of field and illumination during the shooting process, there are large differences in the cracks on different aviation industry components. The deep supervision mechanism proposed in UNet++ comprehensively considers the characteristics of cracks at different levels. The dense jump connections aggregate features of different semantic scales, so it has good adaptability to cracks. Given an image x, use the base learner to extract the image from x. 4,0 The layer extracts the intermediate image features, which can be expressed as:

[0051] f=ε(x),

[0052] Where ε represents the encoder of the base learner and f is X 4,0 Layer image features.

[0053] Step 2: Use the small sample proxy prototype extraction method to extract the prototype v using the support set image mask m and the support set image feature f:

[0054] v=F pool (f⊙I(m)),

[0055] where ⊙ denotes the Hadamard product, I denotes the scale, and F pool Represents MaskedAverage Pooling, MAP;

[0056] Step 3: Take the intermediate image feature f obtained from the base learner encoder ε of image x as the input of the decoder φ to obtain the prediction result p:

[0057] p=φ(f),

[0058] Compare the prediction result p with the support set image mask m to obtain the refined mask M α ,M β :

[0059]

[0060] M β =mM α ,

[0061] Where (x, y) represents the two-dimensional coordinates of the mask;

[0062] Step 4: Use the refined mask M α ,M β And image features f can get the proxy prototype v l ,l={α,β}:v l =F pool (f⊙I(Ml )),l={α,β}.

[0063] Step 5: Expand the spatial scale of the prototype v and convert the query set image x q Input the base learner encoder ε in step 1 to obtain the query set image features f q , the extended prototype v and f q Splicing, denoted as F guide , and obtain the comprehensive feature f s,q :

[0064] f q =ε(x q ),

[0065] f s,q =F guide (v,f q ).

[0066] Step 6: Calculate the query set feature f q The cosine distance between the two agent prototypes is the activation map A. α ,A β :

[0067]

[0068] Step 7: Combine comprehensive features f s,q With the activation feature map A α ,A β Get the final feature f final , the final feature is input into the decoder D composed of atrous spatial pyramid pooling (ASPP) and two residual modules, and the final meta-learner prediction result p is obtained. meta :

[0069]

[0070] p meta =D(f final ).

[0071] Step 8: Input the query set image into the base learner encoder ε to obtain f q Then the decoder is used to obtain the query set image prediction result p base , p base And the meta-learning prediction result p meta Perform logical operations to obtain the final prediction result p final :

[0072] p final =p base ∪p meta .

[0073] The basic learner is trained using the UNet++ deep supervision method:

[0074]

[0075] in The coefficient η is the prediction graph loss obtained by the i-th layer feature of UNet++. i ≡1, which means that feature predictions of all scales are treated equally.

[0076] The effect of the present invention can be further illustrated by the following experiments.

[0077] 1. Experimental conditions

[0078] This embodiment is an experiment conducted on NVIDIA GeForce RTX 3090 and Ubuntu 18.04 operating system using the Python language Pytorch 1.6.0 framework.

[0079] 2. Experimental Data

[0080] A compact servo-hydraulic fatigue test system was used to perform fatigue tests on aviation industry riveted parts, and the stress value increased with time. Hikvision's industrial camera CE200-10GM was used to collect time series crack images under different stress states, with more than 16,000 images collected.

[0081] 3. Experimental content

[0082] First, different UNet series segmentation networks were tested on the dataset. The mIoU results and parameter quantity indicators led to the final selection of UNet++ as the base learner. Then, in order to prove the effectiveness of the small sample crack segmentation method combining the base learner and the meta-learner proposed in this application, it was compared with the best small sample segmentation network in the past three years. The results are shown in Table 1.

[0083] Table 1 Comparison between the proposed method and the SOTA in the past three years

[0084]

[0085] In the above table, the three best methods in the past three years are selected: PFENet, HSNet, and DCP.

[0086] Among them, Year represents the year when the algorithm was proposed, 1-shot and 5-shot represent the experimental results when the number of labeled samples is 1 and 5 respectively. It can be seen that the results of the algorithm proposed in this application are 0.513 and 0.529, which has higher segmentation accuracy than other small sample methods.

[0087] Among them, PFENet is in the literature "Zhuotao Tian, ​​Hengshuang Zhao, Michelle Shu, ZhichengYang, Ruiyu Li, and Jiaya Jia. Prior guided

[0088] feature enrichment network for few-shot segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.” HSNet was proposed in the paper “Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 6941–6952.” DCP was proposed in the paper “Chunbo Lang, Binfei Tu, Gong Cheng, and Junwei Han. Beyond the prototype: Divide-and-conquer proxies for few-shot segmentation. arXiv preprint arXiv:2204.09903, 2022”.

[0089] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A visual detection method for cracks in aviation industry structural parts, Features: The detection method steps are as follows: Step 1: Using the base learner, given an image x, use the base learner to learn from X 4,0 The layer extracts the intermediate image features, expressed as: f=ε(x), Where ε represents the base learner encoder and f is X 4,0 Layer image features; The basic learner is trained using the UNet++ deep supervision method: Where L i Set the coefficient η to be the prediction graph loss obtained by the i-th layer feature of UNet++ i ≡1, feature predictions of all sizes are treated equally; Step 2: Use the small sample proxy prototype extraction method to extract the prototype v using the support set image mask m and the support set image feature f: v=F pool (f⊙I(m)), where ⊙ denotes the Hadamard product, I denotes the scale, and F pool represents mask average pooling; Step 3: Take the intermediate image feature f obtained from the base learner encoder ε of image x as the input of the decoder φ to obtain the prediction result p: p=φ(f), Compare the prediction result p with the support set image mask m to obtain the refined mask M α ,M β : M β =m-M α , Where (x, y) represents the two-dimensional coordinates of the mask; Step 4: Use the refined mask M α ,M β And image features f get the proxy prototype v l ,l={α,β}: v l =F pool (f⊙I(M l )),l={a,b}; Step 5: Expand the spatial scale of the prototype v and convert the query set image x q Input the base learner encoder ε in step 1 to obtain the query set image features f q , the extended prototype v and f q Splicing, denoted as F guide , and obtain the comprehensive feature f s,q : f q =ε(x q ), f s,q =F guide (v,f q ); Step 6: Using query set feature f q The activation graph A is calculated with two proxy prototypes α ,A β ; Step 7: Combine comprehensive features f s,q With the activation feature map A α ,A β Get the final feature f final , input the final feature into the decoder D, and obtain the final meta-learner prediction result p meta : p meta =D(f final ); Step 8: Input the query set image into the base learner encoder ε to obtain f q Then the decoder is used to obtain the query set image prediction result p base , p base And the meta-learning prediction result p meta Perform logical operations to obtain the final prediction result p final .

2. A method for visually detecting cracks in aviation industry structural parts according to claim 1, Features: In step 6, the query set feature f is calculated q The cosine distance between the two agent prototypes is obtained by the activation map A α ,A β :

3. A method for visually detecting cracks in aviation industry structural parts according to claim 1, Features: In step 7, the decoder D includes a dilated convolutional pooling pyramid ASPP and two residual modules.

4. A method for visually detecting cracks in aviation industry structural parts according to claim 1, Features: In step 8, the logic operation is to perform a logical AND operation: p final =p base ∪p meta 。 5. A method for visually detecting cracks in aviation industry structural parts according to claim 1, Features: The industrial camera CE200-10GM was used to collect time series crack images under different stress states.

Images