A method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network

Abstract

The present invention belongs to the field of machine vision, specifically, a method for automatically identifying standard parts on the surface of airborne finished products based on a graph neural network, comprising the following steps: obtaining an image of the surface of an airborne finished product to be tested, constructing an image data set of an airborne finished product surface; annotating the image to be tested; obtaining a scaled standard part image; inputting the standard part image into a feature extraction neural network to obtain a corresponding feature vector; performing normalization processing; dissimilarity between standard parts in the image to be tested; constructing a standard part relationship diagram in the image to be tested; outputting a relationship diagram containing local information; selecting the category with the highest probability as the predicted category of the standard part; constructing a loss function, and training the neural network using a training set; selecting the neural network with the best performance on the validation set as the final model, and testing it on a test set. The present invention can significantly improve the accuracy of standard part recognition and reduce the risk of false or missed detection.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision, and specifically is a method for automatically identifying standard parts on the surface of airborne finished products based on graph neural networks. Background Art

[0002] In order to fix and connect various components on the onboard finished product box, a large number of standard parts of different specifications and types, such as screws, gaskets, spring washers, etc., are installed on the surface of the finished product box. These standard parts are installed independently on the surface of the finished product or combined in a fixed order at a certain position. If wrong installation and missing installation occur, it is easy to cause the finished product to loosen during the flight and landing of the aircraft, affecting the performance of the finished product and endangering the flight safety of the aircraft. Therefore, it is necessary to identify the standard parts installed on the surface of the finished product. At present, the wrong and missing installation of standard parts on the surface of the finished product mainly relies on manual visual inspection. However, due to the large number of standard parts on the surface of the finished product, small size, small differences, and scattered distribution, the manual inspection work intensity is high, and visual fatigue is prone to occur, resulting in wrong and missed inspections.

[0003] Deep learning-based visual inspection is a technology that uses deep learning algorithms to analyze and process images to achieve automated inspection. It has the characteristics of high efficiency and long working time. With the vigorous development of deep learning, deep learning-based visual inspection has gradually begun to be applied to industries such as electronics, medical care, and manufacturing. With the support of deep learning algorithms, visual inspection systems can learn and recognize various features and patterns in images. Through the training process, the system can gradually improve its ability to identify anomalies in images. Compared with traditional visual inspection methods, deep learning-based visual inspection has higher accuracy and efficiency.

[0004] Currently, most deep learning-based visual inspection technologies usually use convolutional neural networks to extract deep feature maps of images, and then use the deep feature maps to classify and regress the target category and location. However, a large number of standard parts have small differences and are blocked from each other, resulting in poor algorithm classification results, making it difficult to distinguish between standard parts with similar appearances and easily classifying standard parts with dissimilar appearances into the same category. Summary of the invention

[0005] In order to solve the above-mentioned problems existing in the prior art, the present invention proposes a method for automatic identification of standard parts on the surface of airborne finished products based on graph neural networks. The installation relationship of standard parts is utilized to construct a standard parts relationship diagram, and the types of standard parts are jointly inferred in the diagram, thereby improving the accuracy of automatic identification of standard parts types on the surface of airborne finished products.

[0006] In order to achieve the above-mentioned invention object, the technical solution provided by this application is as follows:

[0007] A method for automatically identifying standard parts on the surface of airborne finished products based on graph neural networks comprises the following steps:

[0008] S1. Obtain the surface image of the airborne finished product to be tested, construct an airborne finished product surface image dataset, and divide the dataset into a training set, a validation set, and a test set according to a certain ratio;

[0009] S2. Annotate the images to be tested in the training set, validation set, and test set, and record the position and category information of the standard parts in the images;

[0010] S3. According to the marked position, from the image to be tested Cut out the corresponding standard part image , and uniformly scale it to a fixed size to obtain the scaled standard part image ;

[0011] S4. The scaled standard part image Input feature extraction neural network, output The corresponding feature map ,right Perform average pooling and stretching to obtain The corresponding eigenvector ;

[0012] S5. For the standard part in the image to be tested The position in is normalized to obtain The corresponding position vector ;

[0013] S6. Calculate the dissimilarity between the standard parts in the image to be tested according to the feature vector and the position vector;

[0014] S7. construct a standard component relationship diagram in the image to be tested based on the characteristic vector, the position vector and the dissimilarity, wherein each node in the diagram corresponds to a standard component in the image to be tested, and each node is connected to a certain number of other nodes through edges based on the dissimilarity;

[0015] S8. Input the relationship graph into the graph neural network and output the relationship graph containing local information, wherein each node continuously aggregates the information of neighboring nodes and updates the feature representation of the node to obtain a feature vector containing local information;

[0016] S9. Input the feature vector of each node in the relationship graph into the recognition neural network, output the probability of the standard part represented by the node in each category, and select the category with the largest probability as the predicted category of the standard part;

[0017] S10. Construct a loss function and use the training set to train the neural network. After completing a round of training, evaluate the network performance on the validation set.

[0018] S11. Select the neural network with the best performance on the validation set as the final model and test it on the test set.

[0019] Furthermore, the position information in S2 mainly refers to the coordinates of the upper left corner of the standard part annotation box. and the length and width of the annotation box , expressed as .

[0020] Furthermore, in S3, the standard part image Scale uniformly to a fixed size to obtain the scaled standard part image , which includes sampling using bilinear interpolation to achieve image size scaling, where the fixed size is set to .

[0021] Furthermore, the feature extraction neural network in S4 refers to ResNet50.

[0022] Furthermore, the output in S4 is The corresponding feature map It refers to the network obtained after passing through more than 50 convolutional layers, pooling layers, and batch normalization layers of ResNet. feature map.

[0023] Furthermore, in S4 Perform average pooling stretching to obtain The corresponding eigenvector It means Feature map of dimension conduct The average pooling of The pooled features of the dimension are then flattened to obtain a 2048-dimensional feature vector .

[0024] Furthermore, the normalization process in S5 refers to the coordinates of the upper left corner of the annotation box. and the length and width of the annotation box Normalize.

[0025] Furthermore, in S5 The corresponding position vector Represented as a 4-dimensional vector ,in and are the length and width of the original image respectively.

[0026] Furthermore, the dissimilarity in S6 It is based on the feature similarity of two standard part images. Distance from center Calculation, the formula is:

[0027]

[0028] in, is the cosine similarity of the feature vectors of the two standard parts images, is the Euclidean distance between the center points of two different standard parts, and the center point coordinates The calculation formula is:

[0029]

[0030] .

[0031] Furthermore, constructing the standard parts relationship graph in the image to be tested in S7 refers to taking the standard parts in the image to be tested as nodes of the graph, calculating the dissimilarity between the nodes and sorting them from low to high, and selecting the top nodes as its nearest neighbor nodes, and add connecting edges between each node and its nearest neighbor nodes, where the initial node feature vector of each node is the connection of its corresponding feature vector and position vector. Set to 5.

[0032] Furthermore, the graph neural network in S8 is a GCN model, which obtains the updated features of each node by propagating the features of the neighboring nodes of each node. After obtaining the updated features, these features are aggregated using a shared convolution kernel to obtain the final feature vector of each node.

[0033] Furthermore, the identification neural network in S9 is a multi-layer perceptron, and the final feature vector of the node passes through multiple fully connected layers, batch normalization layers, etc. in succession to obtain the predicted probability of the node in each category, among which the last fully connected layer is followed by a softmax function, and the remaining fully connected layers are followed by batch normalization layers and ReLU functions.

[0034] The beneficial effects of the present invention are:

[0035] 1. Compared with other recognition algorithms, the automatic recognition method of airborne finished product surface standard parts based on graph neural network proposed in the present invention can significantly improve the recognition accuracy of standard parts and reduce the risk of false or missed detection.

[0036] 2. The present invention constructs a relationship diagram of standard parts on the surface of airborne finished products and uses a graph neural network to update features, thereby mining the hidden relationship information between standard parts and helping the algorithm identify the categories of standard parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is to explain the present invention rather than to limit the present invention. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] The specific implementation method of the present invention is described below with reference to the accompanying drawings and examples, but the present invention is not limited to this embodiment.

[0040] Example 1

[0041] like Figure 1 As shown, a method for automatically identifying standard parts on the surface of airborne finished products based on graph neural networks includes the following steps:

[0042] S1. Obtain the surface image of the airborne finished product to be tested, construct an airborne finished product surface image dataset, and divide the dataset into a training set, a validation set, and a test set according to a certain ratio; the ratio here can be set manually.

[0043] S2. Annotate the images to be tested in the training set, validation set, and test set, and record the position and category information of the standard parts in the images;

[0044] S3. According to the marked position, from the image to be tested Cut out the corresponding standard part image , and uniformly scale it to a fixed size to obtain the scaled standard part image ;

[0045] S4. The scaled standard part image Input feature extraction neural network, output The corresponding feature map ,right Perform average pooling and stretching to obtain The corresponding eigenvector ;

[0046] S5. For the standard part in the image to be tested The position in is normalized to obtain The corresponding position vector ;

[0047] S6. Calculate the dissimilarity between the standard parts in the image to be tested according to the feature vector and the position vector;

[0048] S7. construct a standard component relationship diagram in the image to be tested based on the characteristic vector, the position vector and the dissimilarity, wherein each node in the diagram corresponds to a standard component in the image to be tested, and each node is connected to a certain number of other nodes through edges based on the dissimilarity;

[0049] S8. Input the relationship graph into the graph neural network and output the relationship graph containing local information, wherein each node continuously aggregates the information of neighboring nodes and updates the feature representation of the node to obtain a feature vector containing local information;

[0050] S9. Input the feature vector of each node in the relationship graph into the recognition neural network, output the probability of the standard part represented by the node in each category, and select the category with the largest probability as the predicted category of the standard part;

[0051] S10. Construct a loss function and use the training set to train the neural network. After completing a round of training, evaluate the network performance on the validation set.

[0052] S11. Select the neural network with the best performance on the validation set as the final model and test it on the test set.

[0053] Example 2

[0054] An automatic identification method for airborne finished product surface standard parts based on a graph neural network in an embodiment of the present application includes the following steps:

[0055] S1 obtains the surface image of the airborne finished product to be tested, constructs an airborne finished product surface image dataset, and divides the dataset into a training set, a validation set, and a test set according to a certain ratio;

[0056] S2 annotates the images to be tested in the training set, validation set, and test set, and records the position and category information of the standard parts in the images;

[0057] S3 According to the marked position, Cut out the corresponding standard part image , and uniformly scale it to a fixed size to obtain the scaled standard part image ;

[0058] S4 will scale the standard part image Input feature extraction neural network, output The corresponding feature map ,right Perform average pooling and stretching to obtain The corresponding eigenvector ;

[0059] S5: Standard parts in the image to be tested The position in is normalized to obtain The corresponding position vector ;

[0060] S6 calculates the dissimilarity between each standard component in the image to be tested according to the feature vector and the position vector;

[0061] S7 constructs a standard component relationship graph in the image to be tested according to the characteristic vector, the position vector and the dissimilarity, wherein each node in the graph corresponds to a standard component in the image to be tested, and each node is connected to a certain number of other nodes through edges according to the dissimilarity;

[0062] S8 inputs the relationship graph into the graph neural network and outputs a relationship graph containing local information, wherein each node continuously aggregates the information of neighboring nodes and updates the feature representation of the node to obtain a feature vector containing local information;

[0063] S9 inputs the feature vector of each node in the relationship graph into the recognition neural network, outputs the probability of the standard part represented by the node in each category, and selects the category with the largest probability as the predicted category of the standard part;

[0064] S10 constructs a loss function and uses the training set to train the neural network. After completing a round of training, the network performance is evaluated on the validation set.

[0065] S11 selects the neural network with the best performance on the validation set as the final model and tests it on the test set.

[0066] In S2, the position and category information of the standard part in the image are recorded, where the position information mainly refers to the coordinates of the upper left corner of the standard part annotation box. and the length and width of the annotation box , expressed as .

[0067] In S3, the standard part image Scale uniformly to a fixed size to obtain the scaled standard part image , which mainly includes the use of bilinear interpolation for sampling to achieve image size scaling.

[0068] Preferably, in S3, the fixed size is set to .

[0069] Preferably, in S4, the feature extraction neural network mainly refers to ResNet50.

[0070] In S4, the output The corresponding feature map , mainly refers to the ResNet50 obtained after more than 50 convolutional layers, pooling layers, and batch normalization layers feature map.

[0071] In S4, Perform average pooling stretching to obtain The corresponding eigenvector , mainly refers to Feature map of dimension conduct The average pooling of The pooled features of the dimension are then flattened to obtain a 2048-dimensional feature vector .

[0072] In S5, normalization mainly refers to the coordinates of the upper left corner of the annotation box. and the length and width of the annotation box Normalize.

[0073] Preferably, in S5, The corresponding position vector , mainly represented as a 4-dimensional vector ,in and are the length and width of the original image respectively.

[0074] In S6, the dissimilarity , mainly based on the feature similarity of two standard part images Distance from center Calculation, the formula is:

[0075]

[0076] in, is the cosine similarity of the feature vectors of the two standard parts images, is the Euclidean distance between the center points of two different standard parts, and the center point coordinates The calculation formula is:

[0077]

[0078]

[0079] In S7, a standard part relationship graph in the image to be tested is constructed, which mainly refers to taking the standard parts in the image to be tested as nodes of the graph, calculating the dissimilarity between the nodes and sorting them from low to high, and selecting the top nodes as its nearest neighbor nodes, and add connecting edges between each node and its nearest neighbor nodes, where the initial node feature vector of each node is the connection of its corresponding feature vector and position vector. Set to 5.

[0080] In S8, the graph neural network is a GCN model, which obtains the updated features of each node by propagating the features of the neighboring nodes of each node. After obtaining the updated features, these features are aggregated using a shared convolution kernel to obtain the final feature vector of each node.

[0081] In S9, the recognition neural network is a multi-layer perceptron, and the final feature vector of the node passes through multiple fully connected layers, batch normalization layers, etc. in succession to obtain the predicted probability of the node in each category. Among them, the last fully connected layer is followed by a softmax function, and the remaining fully connected layers are followed by batch normalization layers and ReLU functions.

Claims

1. A method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network, characterized in that: The steps include: S1. Obtain the surface image of the airborne finished product to be tested, construct an airborne finished product surface image dataset, and divide the dataset into a training set, a validation set, and a test set according to the ratio; S2. Annotate the images to be tested in the training set, validation set, and test set, and record the position and category information of the standard parts in the images; S3. According to the marked position, from the image to be tested Cut out the corresponding standard part image , and uniformly scale it to a fixed size to obtain the scaled standard part image ; S4. The scaled standard part image Input feature extraction neural network, output The corresponding feature map ,right Perform average pooling and stretching to obtain The corresponding eigenvector ; S5. For the standard part in the image to be tested The position in is normalized to obtain The corresponding position vector ; S6. Calculate the dissimilarity between the standard parts in the image to be tested according to the feature vector and the position vector; S7. construct a standard component relationship diagram in the image to be tested according to the characteristic vector, the position vector and the dissimilarity, wherein each node in the diagram corresponds to a standard component in the image to be tested, and each node is connected to multiple other nodes through edges according to the dissimilarity; S8. Input the relationship graph into the graph neural network and output the relationship graph containing local information, wherein each node continuously aggregates the information of neighboring nodes and updates the feature representation of the node to obtain a feature vector containing local information; S9. Input the feature vector of each node in the relationship graph into the recognition neural network, output the probability of the standard part represented by the node in each category, and select the category with the largest probability as the predicted category of the standard part; S10. Construct a loss function and use the training set to train the neural network. After completing a round of training, evaluate the network performance on the validation set. S11. Select the neural network with the best performance on the validation set as the final model and test it on the test set.

2. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: The position information in S2 refers to the coordinates of the upper left corner of the standard part annotation box. and the length and width of the annotation box , expressed as .

3. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: S3 will be the standard part image Scale uniformly to a fixed size to obtain the scaled standard part image , which includes sampling using bilinear interpolation to achieve image size scaling, where the fixed size is set to .

4. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: The feature extraction neural network in S4 refers to ResNet50.

5. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 4, characterized in that: Output in S4 The corresponding feature map It refers to the network obtained after passing through more than 50 convolutional layers, pooling layers, and batch normalization layers of ResNet. feature map.

6. The method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network according to claim 5, characterized in that: S4 Middle Pair Perform average pooling stretching to obtain The corresponding eigenvector It means Feature map of dimension conduct The average pooling of The pooled features of the dimension are then flattened to obtain a 2048-dimensional feature vector .

7. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: Normalization in S5 refers to the coordinates of the upper left corner of the annotation box. and the length and width of the annotation box Normalize.

8. The method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network according to claim 7, characterized in that: S5 The corresponding position vector Represented as a 4-dimensional vector ,in and are the length and width of the original image respectively.

9. The method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: Dissimilarity in S6 It is based on the feature similarity of two standard part images. Distance from center Calculation, the formula is: ； in, is the cosine similarity of the feature vectors of the two standard parts images, is the Euclidean distance between the center points of two different standard parts, and the center point coordinates The calculation formula is: ； 。 10. The method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: In S7, constructing the standard parts relationship graph in the image to be tested refers to taking the standard parts in the image to be tested as nodes of the graph, calculating the dissimilarity between the nodes and sorting them from low to high, and selecting the top nodes as its nearest neighbor nodes, and add connecting edges between each node and its nearest neighbor nodes, where the initial node feature vector of each node is the connection of its corresponding feature vector and position vector. Set to 5.

11. The method for automatic recognition of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: The graph neural network in S8 is a GCN model, which obtains the updated features of each node by propagating the features of the neighboring nodes of each node. After obtaining the updated features, these features are aggregated using a shared convolution kernel to obtain the final feature vector of each node.

12. The method for automatic identification of standard parts on the surface of airborne finished products based on graph neural network according to claim 1, characterized in that: The recognition neural network in S9 is a multi-layer perceptron. The final feature vector of the node passes through multiple fully connected layers and batch normalization layers in succession to obtain the predicted probability of the node in each category. Among them, the last fully connected layer is followed by a softmax function, and the remaining fully connected layers are followed by batch normalization layers and ReLU functions.

Images