A multi-scale target perception classification method and system based on a twin network

Abstract

The application discloses a kind of multi-scale target perception classification method and system based on twin network, comprising: obtaining the point image of each item of positioned support tube clamp and pre-processing, obtain the pre-processed point image of each item;According to the pre-processed point image of each item, combined with improved main network resnet50 and twin neural network, construct multi-scale target perception classification model based on twin network;The model training is carried out to the multi-scale target perception classification model based on twin network, and the trained multi-scale target perception classification model based on twin network is obtained;Using the trained multi-scale target perception classification model based on twin network, the defect classification is carried out to the point image of each item of support tube clamp to be detected positioning, and classification result is obtained.The method of the application has high accuracy in recognizing the defects of catenary 4C positioning support tube clamp area, and has high recognition efficiency.

Description

A Multi-Scale Target Perception Classification Method and System Based on Siamese Networks Technical Field

[0001] This invention relates to the field of defect classification technology for the 4C positioning support pipe clamp area of ​​the overhead contact line, specifically to a multi-scale target perception and classification method and system based on twin networks. Background Technology

[0002] The existing twin neural network defect detection technology solutions mainly include: the existing technology with application number "201911394231.4" and title "Defect Detection System and Method Based on Template Alignment Using Twin Neural Network". This technology uses a twin neural network to input the image to be detected and the template image into two ResNet backbone networks to extract feature information and share weights. Finally, softmax is used to calculate the concatenated vector to obtain the defect detection result.

[0003] However, given the limited number of defects in the overhead contact system and the significant differences between similar defects, coupled with numerous interferences in the natural shooting environment, including occlusion, lighting variations, motion blur, and rotation, directly applying the above methods would result in low accuracy and efficiency in classifying and identifying defects in the positioning support pipe clamp area. Therefore, the existing methods described above are not suitable for classifying defects in the 4C positioning support pipe clamp area of ​​the overhead contact system. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-scale target perception and classification method and system based on Siamese neural networks. This invention applies Siamese neural networks to the field of railway 4C overhead contact lines. Taking into account the characteristics of the small number of defects in the overhead contact line and the numerous interferences in the natural shooting environment, including occlusion, lighting changes, motion blur, and rotation, the Siamese neural network structure is optimized and improved to design a reasonable and effective algorithm, achieving real-time, high efficiency, and strong generalization. The detection results are obtained by inputting the images of various points of the positioning support pipe clamp to be detected and the template image into the trained network. The method of this invention has high accuracy and high efficiency in classifying defects in the 4C positioning support pipe clamp area of ​​the overhead contact line.

[0005] This invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a multi-scale target perception classification method based on Siamese networks, the method comprising:

[0007] Acquire the images of each point of the located support pipe clamp and perform preprocessing to obtain the preprocessed images of each point;

[0008] Based on the preprocessed point images, and combining the improved backbone network ResNet50 and Siamese neural network, a multi-scale target perception classification model based on Siamese network is constructed.

[0009] The multi-scale target perception classification model based on Siamese network is trained to obtain a trained multi-scale target perception classification model based on Siamese network.

[0010] Using a trained multi-scale target perception and classification model based on Siamese networks, defects are classified for each point image of the support pipe clamp to be detected and located, and the classification results are obtained.

[0011] Furthermore, the preprocessing includes:

[0012] The acquired image data of each point of the located support pipe clamp (this is small sample image data) is classified to form different defect sample sets and qualified sample sets;

[0013] By combining data augmentation image flipping with weighted addition, the sample set with few located support pipe clip sub-points is expanded to obtain the corresponding expanded sample set.

[0014] Based on the expanded corresponding sample sets, each sample set is divided into a training set, a validation set, and a test set. The training set samples are used to train and optimize the model, the test set samples are used to predict and evaluate the model's performance, and the validation set samples are used to evaluate the model's generalization ability and adjust the selected network parameters.

[0015] Furthermore, data augmentation image flipping combined with weighted addition includes mean filtering and image flipping combined with weighted addition;

[0016] First, the sample set with few located support pipe clamp sub-items is filtered by the mean filtering method to obtain the filtered sample set.

[0017] Secondly, the filtered sample set is rotated, mirrored, and contrast adjusted by combining image flipping with weighted addition to obtain the flipped sample set, which is then used as the expanded corresponding sample set.

[0018] Furthermore, the formula for image flipping combined with weighted addition is:

[0019]

[0020] Among them, dst ij represents the pixel value of the element in the i-th row and j-th column after the weighted average is flipped, and src represents the filtered sample set image. The weights of the filtered sample set images are represented by γ, which represents the correction offset and adjusts the contrast.

[0021] Furthermore, the different defect sample sets include sample sets with the bolt area facing forward and sample sets with the bolt area facing backward;

[0022] The bolt area facing the sample set includes missing nuts, missing bolts, loose nuts, and missing cotter pins;

[0023] The bolt area facing away from the sample set includes bolt loosening.

[0024] Furthermore, the multi-scale object perception classification model based on Siamese networks includes a feature aggregation module, an attention module, and a multi-size object perception module;

[0025] The feature aggregation module is used to extract feature information between template blocks and test blocks through the improved backbone network ResNet50.

[0026] The attention module is used to fine-tune relevant feature information by combining key local information from template features and test features respectively;

[0027] A multi-scale object perception module is used to obtain multi-scale spatial information through a fixed receptive field.

[0028] Furthermore, the steps for feature extraction in the multi-scale target perception classification model based on Siamese networks are as follows:

[0029] Two input images are received, including an item template image and an item test image.

[0030] After three-layer feature extraction using the improved backbone network ResNet50, item template image features and item test image features are obtained.

[0031] The item template image features and item test image features are correlated pixel by pixel and then concatenated by channel, and then feature fusion is performed to generate a higher-dimensional fused feature.

[0032] The fused features are input into the attention module, and each feature map of the fused features is compressed into a compressed vector through a global average pooling layer.

[0033] The compressed vector is input into a two-layer multilayer perceptron, and the channel attention weights are mapped to the range of 0-1 using the sigmoid function.

[0034] The channel attention weights are multiplied element-wise with the fusion features, and then the fusion result is added to the fusion features to complete the nonlocal attention optimization, resulting in the optimized relevant features.

[0035] The relevant features are input into three similar convolutional modules in parallel to obtain three features; then the three features are linearly added proportionally to obtain the final feature after aggregating multi-scale object information.

[0036] Furthermore, when using a multi-scale target perception classification model based on Siamese networks for classification, the Euclidean distance is calculated based on the final features; and the classification prediction is performed using the Euclidean distance result.

[0037] Furthermore, feature information between the template block and the test block is extracted using the improved backbone network ResNet50, including:

[0038] The preprocessed point images are input into two parallel residual blocks through an activation function, and the feature extraction is enhanced by using two parallel residual blocks.

[0039] The image output from two parallel residual blocks is then fed into three serial residual blocks and a deconvolution block to enhance super-resolution and obtain a super-resolution image.

[0040] Furthermore, the multi-scale target perception classification model based on Siamese networks is trained. During training, the stochastic gradient descent algorithm is used for iterative training several times, and the training results are saved for each iteration. The specific steps for each training iteration are as follows:

[0041] The item template image and item test image are respectively input into the multi-scale target perception classification model based on Siamese network to extract the final template feature and final test feature after aggregating multi-scale object information;

[0042] Calculate the Euclidean distance between the final template features and the final test features;

[0043] The Euclidean distance is fully connected twice, with the second connection to a single neuron. The result of this neuron is then activated using a sigmoid function to obtain the similarity between the two input images, which is then used as the training result.

[0044] Secondly, the present invention provides a multi-scale target perception and classification system based on Siamese networks, which uses the aforementioned multi-scale target perception and classification method based on Siamese networks; the system includes:

[0045] The acquisition unit is used to acquire images of various points of the located support pipe clamp;

[0046] The preprocessing unit is used to preprocess the point images to obtain the preprocessed point images;

[0047] The classification model building unit is used to construct a multi-scale target perception classification model based on Siamese network by combining the improved backbone network ResNet50 and Siamese neural network with the preprocessed point images.

[0048] The model training unit is used to train the multi-scale target perception classification model based on Siamese networks to obtain the trained multi-scale target perception classification model based on Siamese networks.

[0049] The defect classification unit is used to classify defects in the image of each point of the support pipe clamp to be detected and located using a trained multi-scale target perception classification model based on Siamese network, and obtain the classification result.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] This invention discloses a multi-scale target perception and classification method and system based on Siamese neural networks. The invention applies Siamese neural networks to the field of railway 4C overhead contact lines. Considering the small number of defects in the contact line field and the numerous interferences in the natural shooting environment, including occlusion, lighting changes, motion blur, and rotation, the Siamese neural network structure is optimized and improved to design a reasonable and effective algorithm, achieving real-time, high efficiency, and strong generalization. The detection results are obtained by inputting the images of various points of the positioning support pipe clamp to be detected and the template image into the trained network. This invention's method has high accuracy and high efficiency in classifying defects in the 4C positioning support pipe clamp area of ​​the overhead contact line. Specifically:

[0052] (1) To address the characteristics of the dataset being affected by changes in lighting and spatial motion states due to environmental changes, the sample set is expanded by combining data augmentation image flipping with weighted addition.

[0053] (2) To address the problem of a small number of defective samples and large differences among similar samples, this invention adopts a feature fusion method in the SiamMOP network. By extracting the output features of the three convolutional layers of ResNet and combining them with pixel-level related channels, the robustness of the algorithm is improved.

[0054] (3) To address the problem of motion blur and inconspicuous defects, an attention mechanism is proposed to combine fusion features and attention weights to complete non-local attention optimization.

[0055] (4) Improve the backbone network by changing the serial connection in the residual module to multiple parallel connection networks and adding a multi-scale perception module in the SiamMOP network to obtain more comprehensive and accurate feature information.

[0056] (5) To address the time efficiency problem of deep structure training of the SiamMOP network of the present invention, a gradient descent fast iteration optimization algorithm is introduced.

[0057] (6) In the case of insufficient sample resolution to effectively extract features, this invention introduces a super-resolution module to effectively improve the surface feature extraction effect of the algorithm. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0059] Figure 1 is a flowchart of a multi-scale target perception and classification method based on Siamese networks according to the present invention;

[0060] Figure 2 is a flowchart of the improved backbone network resnet50 of the present invention;

[0061] Figure 3 is a diagram of the improved residual module of the present invention;

[0062] Figure 4 is a structural diagram of the super-resolution module of the present invention;

[0063] Figure 5 is a comparison between the original image and the super-resolution effect image of the present invention;

[0064] Figure 6 is a diagram of the improved SiamMOP network structure of the present invention;

[0065] Figure 7 is a schematic diagram of the front of the present invention facing the sample set;

[0066] Figure 8 is a schematic diagram of the sample set with the reverse side facing outwards in this invention;

[0067] Figure 9 is a schematic diagram of the rotation contrast enhancement of missing cotter pin data according to the present invention;

[0068] Figure 10 is a schematic diagram comparing the similarity between defective samples and normal samples of the present invention;

[0069] Figure 11 is a schematic diagram of the similarity comparison of normal samples in this invention;

[0070] Figure 12 is a schematic diagram of the structure of a multi-scale target perception and classification system based on Siamese networks according to the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0072] Given the limited number of defects in the overhead contact system and the significant differences between similar defects, coupled with numerous interferences in the natural shooting environment, including occlusion, lighting variations, motion blur, and rotation, directly applying existing defect classification methods would result in low accuracy and efficiency in classifying and identifying defects in the positioning support pipe clamp area. Therefore, existing defect classification methods are not suitable for classifying defects in the 4C positioning support pipe clamp area of ​​the overhead contact system.

[0073] This invention designs a multi-scale target perception and classification method and system based on Siamese neural networks. Applying Siamese neural networks to the 4C overhead contact system of railways, and considering the characteristics of a small number of defects in the contact network field and numerous interferences in the natural shooting environment, including occlusion, lighting changes, motion blur, and rotation, the invention optimizes and improves the Siamese neural network structure to design a reasonable and effective algorithm, achieving real-time performance, high efficiency, and strong generalization. The detection results are obtained by inputting the images of various points of the positioning support pipe clamp to be detected and the template image into the trained network. This invention's method has high accuracy and high efficiency in classifying defects in the 4C positioning support pipe clamp area of ​​the overhead contact system.

[0074] Specifically, by combining the improved backbone network ResNet50 and the Siamese neural network, a multi-scale target perception classification model based on the Siamese network is constructed as follows:

[0075] First, the backbone network design is as follows:

[0076] The backbone network of this invention will use the ResNet50 residual network, which has better performance in image classification tasks. The increase in the number of network layers can extract more abstract features. For traditional CNN networks, simply increasing the network depth can easily lead to gradient vanishing and exploding. The degradation problem is solved by constructing an identity mapping.

[0077] H(x) = F(x) + x

[0078] Where F(x) is the network mapping before summation, and H(x) is the network mapping from input to summation. The mapping heap with residuals has a more obvious output change, a greater effect on weight adjustment, and a better overall effect.

[0079] The residual block (Bottleneck) structure introduces 1*1 convolutions to increase and decrease the number of channels, realizing a linear combination of multiple feature maps while preserving the original feature map size. At the same time, each convolutional layer adds a ReLU activation function, introducing more non-linear mappings to reduce computational complexity.

[0080] To address the issue of large differences in the distribution of similar samples, which makes it difficult for classifiers to correctly classify and identify them, this invention improves the backbone network ResNet50 to obtain the improved backbone network ResNet50. The flowchart of the improved backbone network ResNet50 is shown in Figure 2.

[0081] The three improvements of this invention are shown in Figure 2. For layer 5, the downsampling sequence step in the first Bottlneck is improved and connected in parallel with the second Bottlneck, and max pooling is added. At the same time, two serial Bottlnecks and a deconvolution block are introduced to form a super-resolution structure. Layers 3 and 4 are similar, but with one less downsampling improvement step compared to layer 5.

[0082] The three improvements are as follows:

[0083] (1) Improve the downsampling part

[0084] As shown in Table 1, after the input image enters the last convolutional layer, it first undergoes a 1*1 convolution with stride=2, which will cause the information of the final input feature map 3 / 4 to be unused. In order to avoid a large loss of feature information, the downsampling is moved to the later 3*3 convolution, and the global average pooling layer and the following part are removed.

[0085] Table 1. RestNet50 Feature Extraction Network Structure

[0086]

[0087] (2) Improved residual block

[0088] The original serial connection network in the residual block is transformed into multiple parallel connection networks, enabling simultaneous convolution and aggregation at multiple scales to extract features at different scales, resulting in a sparse, computationally efficient network structure. Furthermore, by directly routing the input information, the integrity of the information is preserved. The improved residual module is shown in Figure 3.

[0089] (3) Add super-resolution module

[0090] In real-world scenarios, datasets often have excessively low resolution, hindering effective feature extraction. Low-resolution images negatively impact surface feature learning; therefore, this invention designs a super-resolution module. It primarily utilizes multiple residual blocks as the main structure for feature extraction, and finally employs a deconvolution module to increase the feature map size. As shown in Figure 4, the super-resolution module consists of three residual blocks and one deconvolution block. To balance the feature extraction performance, this invention performs super-resolution on target templates with an original resolution of less than 64 (a×a), resulting in a 2a×2a super-resolution target template, as shown in Figure 5.

[0091] Second, a multi-scale target perception classification model based on Siamese networks is designed as follows:

[0092] (1) SiamMOP network based on Siam network for multi-scale target perception classification model

[0093] Because the sample dataset is significantly affected by changing external environmental features and exhibits diverse spatial morphology and ambiguity, the similarity between samples of the same type varies considerably. To better obtain global features and improve the robustness of the algorithm, this invention combines the ResNet network and Siamese network frameworks to improve the SiamMOP network structure, as shown in Figure 6.

[0094] This invention abandons the anchorless prediction module of the previous SiamMOP. Instead, it introduces an attention module and a multi-scale perception module after fine feature aggregation for template input and test input branches, and then uses Euclidean distance to classify and predict the final features.

[0095] To compare the similarity of two inputs, Siam networks are an excellent architecture, capable of real-time feature extraction and weight sharing. The improved SiamMOP network consists of three sub-modules: a fine-grained feature aggregation module, an attention module, and a multi-scale perception module. SiamMOP starts with a shared feature extraction network, ResNet, and aggregates feature information between template and test blocks through the fine-grained feature aggregation module. Then, the attention module fine-tunes relevant features by combining key local information from template and test features respectively. The multi-scale perception module obtains multi-scale spatial information by fixing the receptive field.

[0096] (2) Design of SiamMOP Feature Extraction Method

[0097] Two input images are received, including an item template image and an item test image.

[0098] After three-layer feature extraction using the improved backbone network ResNet50, item template image features and item test image features are obtained.

[0099] The item template image features and item test image features are correlated pixel-wise in pairs, then concatenated by channel, and feature fusion is performed to generate a higher-dimensional fused feature.

[0100] The fused features are input into the attention module, and each feature map of the fused features is compressed into a compressed vector through a global average pooling layer.

[0101] The compressed vector is input into a two-layer multilayer perceptron, and the channel attention weights are mapped to the range of 0-1 using the sigmoid function.

[0102] The channel attention weights are multiplied element-wise with the fusion features, and then the fusion result is added to the fusion features to complete the nonlocal attention optimization, resulting in the optimized relevant features.

[0103] The relevant features are input into three similar convolutional modules in parallel to obtain three features; and the three features are linearly added in proportion to obtain the final feature after aggregating multi-scale object information.

[0104] Furthermore, when using a multi-scale target perception classification model based on Siamese networks for classification, the Euclidean distance is calculated based on the final features; and the classification prediction is performed using the Euclidean distance result.

[0105] In addition, considering that the dataset is affected by changes in lighting and has different spatial motion states due to environmental changes, this invention also proposes a method to expand the sample set by combining data augmentation image flipping with weighted addition.

[0106] Example 1

[0107] As shown in Figure 1, the present invention provides a multi-scale target perception and classification method based on Siamese networks, the method comprising:

[0108] Acquire the images of each point of the located support pipe clamp and perform preprocessing to obtain the preprocessed images of each point;

[0109] Based on the preprocessed point images, and combining the improved backbone network ResNet50 and Siamese neural network, a multi-scale target perception classification model based on Siamese network is constructed.

[0110] The multi-scale target perception classification model based on Siamese network is trained to obtain a trained multi-scale target perception classification model based on Siamese network.

[0111] Using a trained multi-scale target perception and classification model based on Siamese networks, defects are classified for each point image of the support pipe clamp to be detected and located, and the classification results are obtained.

[0112] As a further implementation, the multi-scale object perception classification model based on Siamese networks includes a feature aggregation module, an attention module, and a multi-size object perception module;

[0113] The feature aggregation module is used to extract feature information between template blocks and test blocks through the improved backbone network ResNet50.

[0114] The attention module is used to fine-tune relevant feature information by combining key local information from template features and test features respectively;

[0115] A multi-scale object perception module is used to obtain multi-scale spatial information through a fixed receptive field.

[0116] As a further implementation, the steps for feature extraction using a multi-scale target perception classification model based on Siamese networks are as follows:

[0117] The system receives two inputs: a point template image x. temp With the item test image x test After feature extraction through three layers of the backbone network (CONV3, CONV4, CONV5), the item template image features are obtained. Item test image features Taking the item template image as input as an example, the features of the item template image are... Decomposed into h×w sub-convolutional kernels k i ∈R c ,i∈{1,…,h*w}, connect each sub-convolution kernel with After performing basic correlation, connections are made according to channels, where w and h are features. Width and height, c is the number of channels, R k The mathematical formula for obtaining the features is as follows:

[0118]

[0119] The obtained features and After performing pixel-wise correlation on each pair of features, concatenate them by channel, and then perform feature fusion to generate a higher-dimensional fused feature R. c :

[0120]

[0121] Fusion feature R c In the input channel attention module of SENet, the fused features R c Each feature map layer is compressed into a vector using a global average pooling layer. The formula is as follows:

[0122]

[0123] H and W are fusion features R c The height and width are then calculated, and the vector is input into a two-layer multilayer perception. The sigmoid function is used to map the channel attention weights to the range of 0-1, as shown in the following mathematical expression:

[0124] AC =σ(L2RELU(L1S))

[0125] in and There are two fully connected layers, where σ represents the activation function, and A... C It is the obtained channel attention weights, which are then combined with the fused feature R. c Element-wise multiplication is performed channel by channel, where ⊙ denotes element-wise multiplication:

[0126]

[0127] The final fusion feature R c Addition completes the nonlocal attention optimization, where The formula for summing elements is as follows:

[0128]

[0129] Relevant features R A Parallel input of three similar convolutional modules φ a,b ,(a,b)∈{(7,7),(7,13),(13,7),the three features φ obtained are a,b Added linearly in proportion:

[0130]

[0131] Where a and b represent the total convolutional kernel size of the convolutional module, X mop It is a feature that aggregates multi-scale object information, a a,b These are the hyperparameters trained together by the network, using three concatenated small convolutional layers φ. 7,7 φ 7,13 φ 13,7 The kernel sizes are (3,3), (3,5), and (5,3), the stride is 1, and the padding is (0,0), (0,1), and (1,1). The three convolutional kernels have different convolutional receptive fields.

[0132] As a further implementation, as shown in Figure 2, the improved backbone network ResNet50 is used to extract feature information between the template block and the test block, including:

[0133] The preprocessed point images are input into two parallel residual blocks through an activation function, and the feature extraction is enhanced by using two parallel residual blocks.

[0134] The image output from two parallel residual blocks is then fed into three serial residual blocks and a deconvolution block to enhance super-resolution and obtain a super-resolution image.

[0135] The specific implementation is as follows:

[0136] 1. Image preprocessing

[0137] Step A involves classifying the acquired image data of various points of the located support pipe clamps (this is a small sample image data) to form different defect sample sets and qualified sample sets. In this experiment, the located bolt areas have different orientations. The sample set facing the front is divided into (normal, missing nut, missing bolt, loose nut, missing cotter pin), and the sample set facing the back is divided into (normal, loose bolt), as shown in Figures 7 and 8. The number of sample sets is shown in Table 2.

[0138] Table 2 Sample Quantity for Each Point of Positioning Support Pipe

[0139]

[0140] Step B addresses the characteristics of the dataset being affected by environmental changes, such as lighting and different spatial motion states. It employs data augmentation, image flipping, and weighted addition to expand the sample set in Table 2 for some items with fewer points, resulting in the expanded sample set.

[0141] As shown in Figure 9, the original image is shown. By combining data augmentation with image flipping and weighted addition operations, different angles and contrasts are achieved.

[0142] Specifically, data augmentation image flipping combined with weighted addition includes mean filtering and image flipping combined with weighted addition;

[0143] First, the sample set with few located support pipe clamp sub-items is filtered by the mean filtering method to obtain the filtered sample set.

[0144] Secondly, the filtered sample set is rotated, mirrored, and contrast adjusted by combining image flipping with weighted addition to obtain the flipped sample set, which is then used as the expanded corresponding sample set.

[0145] The mean filtering formula is as follows:

[0146]

[0147] s xyThe center point is at (x,y), K represents a filter window of size m×n, K=(2m+1)(2n+1), g(s,t) represents the original image, and f(x,y) represents the image obtained after mean filtering.

[0148] The formula for image flipping combined with weighted addition is as follows:

[0149]

[0150] Among them, dst ij represents the pixel value of the element in the i-th row and j-th column after the weighted average is flipped, and src represents the filtered sample set image. The weights of the filtered sample set images are represented by γ, which represents the correction offset and adjusts the contrast.

[0151] Step C: Based on the expanded corresponding sample sets, each sample set is divided into a training set, a validation set, and a test set. The training set accounts for 60% of the samples and is used to train and optimize the model. The test set accounts for 20% of the samples and is used to predict and evaluate the model's performance. The validation set accounts for 20% of the samples and is used to evaluate the model's generalization ability and adjust the selected network parameters.

[0152] 2. Model training based on a multi-scale target perception classification model using Siamese networks.

[0153] A multi-scale target perception classification model based on Siamese networks was trained. During training, the stochastic gradient descent algorithm was used for iterative training several times, and the training results were saved for each iteration. The specific steps for each training iteration were as follows:

[0154] The item template image and item test image are respectively input into the multi-scale target perception classification model based on Siamese network to extract the final template feature and final test feature after aggregating multi-scale object information;

[0155] Calculate the Euclidean distance between the final template features and the final test features;

[0156] The Euclidean distance is fully connected twice, with the second connection to a single neuron. The result of this neuron is then activated using a sigmoid function to obtain the similarity between the two input images, which is then used as the training result.

[0157] In practice, a template image and a test image of size 124*96 are input respectively. Features are extracted through a multi-scale object perception classification model based on Siamese networks. Finally, X values ​​with multi-scale object perception features are obtained. mop-temp and X mop-test Calculate their Euclidean distance:

[0158] Eω(xtemp ,x test )=|X mop-temp -X mop-test |

[0159] This distance is processed through two fully connected layers. The second fully connected layer is connected to a single neuron. The result of this neuron is then processed using a sigmoid function, which represents the similarity between the two input images.

[0160] p=σ(RELU(Eω(x temp ,x test )))

[0161] The final loss can be calculated by performing cross-entropy calculation on the network's output and the ground truth labels, as shown in the following formula:

[0162]

[0163] Here, y is the binary label 0 or 1, and p(y) is the probability that the output belongs to the label y. As a loss function, binary cross-entropy is used to evaluate the quality of a binary classification model's prediction results.

[0164] During training, the Stochastic Gradient Descent (SGD) algorithm was used for 30 iterations, and the results of each iteration were saved. The parameters of ResNet were frozen for the first ten iterations, and the last ten iterations were trained with one-tenth of the overall network learning rate.

[0165] 3. Model validation of a multi-scale target perception classification model based on Siamese networks.

[0166] To verify the performance of the Siamese neural networks trained on the two different training sets, this experiment input defect datasets with different orientations and normal datasets into the two models for verification. A label of 1 was used when two inputs pointed to the same type, and 0 when they pointed to different types. Each normal sample was compared with samples of different defect types for similarity verification. The verification results are shown in Figures 10 and 11, where Similarity is the similarity calculated using Loss.

[0167] The number of samples with missing cotter pins, missing bolts, and missing nuts are represented by n1, n2, n3, n4, and n5, respectively, and the number of correctly identified samples is represented by k. The formulas for the recognition rates p1, p2, p3, p4, and p5 for each item are as follows:

[0168]

[0169] The training set was trained 150 times, and the results of the validation on 20% of the dataset are shown in Table 3. A total of 5 sets of experiments were conducted, and the accuracy of three of the experiments reached over 90%. This demonstrates the superiority of this method in defect identification when the training samples are small and the defects are obvious. However, this method still has shortcomings when the defects are not obvious and the sample images are uneven.

[0170] Table 3. Number of points and accuracy of each item in the validation set

[0171]

[0172] 4. Using the trained and validated multi-scale target perception classification model based on Siamese networks, perform defect classification on the images of various points of the support pipe clamp to be detected and located, and obtain the classification results.

[0173] The advantage of this application is that the SiamMOP network transforms the classification task into a template similarity matching problem instead of a common binary classification problem. (1) To address the characteristics of the dataset being affected by environmental changes, such as lighting and spatial motion state distribution, the sample set is expanded by combining data augmentation, image flipping, and weighted addition. (2) To address the problem of few defect samples and large differences among similar samples, this invention adopts a feature fusion method in the SiamMOP network, which improves the robustness of the algorithm by extracting the output features of the three convolutional layers of ResNet and combining them with pixel-level related channels. (3) To address the problem of motion blur and indistinct defects, an attention mechanism is proposed, which combines fused features and attention weights to complete non-local attention optimization. (4) The backbone network is improved by changing the serial connection in the residual module to multiple parallel connection networks, and a multi-scale perception module is added to the SiamMOP network to obtain more comprehensive and accurate feature information. (5) To address the problem of the training efficiency of the deep structure of the SiamMOP network in this invention, a gradient descent fast iteration optimization algorithm is introduced. (6) In the case of insufficient sample resolution to effectively extract features, this invention introduces a super-resolution module to effectively improve the surface feature extraction effect of the algorithm.

[0174] Example 2

[0175] As shown in Figure 12, the difference between this embodiment and Embodiment 1 is that this embodiment provides a multi-scale target perception classification system based on Siamese networks. This system uses a multi-scale target perception classification method based on Siamese networks from Embodiment 1. The system includes:

[0176] The acquisition unit is used to acquire images of various points of the located support pipe clamp;

[0177] The preprocessing unit is used to preprocess the point images to obtain the preprocessed point images;

[0178] The classification model building unit is used to construct a multi-scale target perception classification model based on Siamese network by combining the improved backbone network ResNet50 and Siamese neural network with the preprocessed point images.

[0179] The model training unit is used to train the multi-scale target perception classification model based on Siamese networks to obtain the trained multi-scale target perception classification model based on Siamese networks.

[0180] The defect classification unit is used to classify defects in the image of each point of the support pipe clamp to be detected and located using a trained multi-scale target perception classification model based on Siamese network, and obtain the classification result.

[0181] The execution process of each unit can be carried out according to the steps of the multi-scale target perception classification method based on Siamese network in Embodiment 1, and will not be described in detail in this embodiment.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0186] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-scale target perception and classification method based on Siamese networks, characterized in that, The method includes: acquiring and preprocessing point images of the located support pipe clamps to obtain preprocessed point images; constructing a multi-scale target perception classification model based on the Siamese network by combining the improved backbone network ResNet50 and the Siamese neural network based on the preprocessed point images; training the Siamese network-based multi-scale target perception classification model to obtain a trained Siamese network-based multi-scale target perception classification model; and using the trained Siamese network-based multi-scale target perception classification model to classify defects in the point images of the support pipe clamps to be detected and located, obtaining classification results. The Siamese network-based multi-scale target perception classification model includes a feature aggregation module, an attention module, and a multi-size object perception module. The feature aggregation module is used to extract feature information between template blocks and test blocks through the improved backbone network ResNet50. The attention module is used to fine-tune relevant feature information by combining key local information of template features and test features respectively. The multi-size object perception module is used to obtain multi-scale spatial information by fixing the receptive field. The improved backbone network ResNet50 is used to extract feature information between template blocks and test blocks. The et50 algorithm extracts feature information between the template block and the test block, including: inputting the preprocessed point images into two parallel residual blocks through an activation function to enhance feature extraction; outputting the image through the two parallel residual blocks to three serial residual blocks and a deconvolution block to enhance super-resolution and obtain a super-resolution image; the improved backbone network resnet50 is obtained by improving the backbone network resnet50; the improvements include: improvements to layer 5: improving the downsampling order step in the first Bottlneck and performing parallel connection between the first and second Bottlnecks and adding max pooling operations, while introducing two serial Bottlnecks and a deconvolution block to form a super-resolution structure; improvements to layer 3 and layer 4: performing parallel connection between the first and second Bottlnecks and adding max pooling operations, while introducing two serial Bottlnecks and a deconvolution block to form a super-resolution structure.

2. The multi-scale target perception and classification method based on Siamese networks according to claim 1, characterized in that, The preprocessing includes: classifying the acquired image data of each point of the located support pipe clamp to form different defect sample sets and qualified sample sets; using data augmentation, image flipping, and weighted addition, expanding the sample sets with fewer points of the located support pipe clamp to obtain expanded corresponding sample sets; and dividing each sample set into a training set, a validation set, and a test set according to the expanded corresponding sample sets. The training set samples are used to train and optimize the model, the test set samples are used to predict and evaluate the model performance, and the validation set samples are used to evaluate the model's generalization ability and adjust the selected network parameters.

3. The multi-scale target perception and classification method based on Siamese networks according to claim 2, characterized in that, The data augmentation image flipping combined with weighted addition includes mean filtering and image flipping combined with weighted addition. The mean filtering method is used to filter the sample set with few located support pipe clips to obtain the filtered sample set. The image flipping combined with weighted addition is used to rotate, mirror and adjust the contrast of the filtered sample set to obtain the flipped sample set, which is used as the corresponding expanded sample set.

4. The multi-scale target perception and classification method based on Siamese networks according to claim 3, characterized in that, The formula for image flipping combined with weighted addition is: ;in, Indicates the weighted average after flipping. OK List the pixel values ​​of each element. This represents the filtered sample set image. The weights represent the weights of the filtered sample set images. 。 5. The multi-scale target perception and classification method based on Siamese networks according to claim 2, characterized in that, Different defect sample sets include bolt area front-facing sample sets and bolt area back-facing sample sets; the bolt area front-facing sample set includes missing nuts, missing bolts, loose nuts, and missing cotter pins; the bolt area back-facing sample set includes loose bolts.

6. The multi-scale target perception and classification method based on Siamese networks according to claim 1, characterized in that, The feature extraction steps of the multi-scale target perception classification model based on Siamese networks are as follows: Two input images are received, including an item template image and an item test image; after three layers of feature extraction using the improved backbone network ResNet50, item template image features and item test image features are obtained; the item template image features and item test image features are correlated pixel by pixel and concatenated by channel, and then fused to generate a higher-dimensional fused feature; the fused feature is input into an attention module, and each layer of the fused feature's feature map is compressed into a compressed vector through a global average pooling layer; the compressed vector is input into a two-layer multilayer perceptron, and the sigmoid function is used to map the channel attention weights to between 0 and 1; the channel attention weights are multiplied element-wise with the fused feature channel by channel, and the fusion result is added to the fused feature to complete non-local attention optimization, obtaining optimized relevant features; the relevant features are input in parallel into three convolutional modules to obtain three features. The three features are then linearly added proportionally to obtain the final features after aggregating multi-scale object information.

7. The multi-scale target perception and classification method based on Siamese networks according to claim 6, characterized in that, The multi-scale target perception classification model based on Siamese network is trained by using stochastic gradient descent algorithm for iterative training several times and saving the training results each time. The specific steps of each training are as follows: input the item template image and item test image into the multi-scale target perception classification model based on Siamese network respectively, and extract the final template features and final test features after aggregating multi-scale object information. Calculate the Euclidean distance between the final template features and the final test features; perform two fully connected operations on the Euclidean distance, with the second fully connected to a single neuron, and apply sigmoid activation to the result of this neuron to obtain the similarity between the two input images and use it as the training result.

8. A multi-scale target perception and classification system based on Siamese networks, characterized in that, The system includes: an acquisition unit for acquiring images of various points of the located support pipe clamp; a preprocessing unit for preprocessing the images of various points to obtain preprocessed images of various points; a classification model construction unit for constructing a multi-scale target perception classification model based on a Siamese network, combining the preprocessed images of various points with an improved backbone network ResNet50 and a Siamese neural network; a model training unit for training the multi-scale target perception classification model based on a Siamese network to obtain a trained multi-scale target perception classification model based on a Siamese network; and a defect classification unit for classifying defects in the images of various points of the support pipe clamp to be detected and located using the trained multi-scale target perception classification model based on a Siamese network to obtain classification results. The multi-scale target perception classification model based on a Siamese network includes a feature aggregation module, an attention module, and a multi-size object perception module. The feature aggregation module is used to extract feature information between template blocks and test blocks using the improved backbone network ResNet50; the attention module is used to fine-tune relevant feature information by combining key local information of template features and test features respectively; and the multi-size object perception module is used to obtain... Multi-scale spatial information; the extraction of feature information between template blocks and test blocks through the improved backbone network ResNet50 includes: inputting the preprocessed point images into two parallel residual blocks through an activation function, enhancing feature extraction through the two parallel residual blocks; outputting the image through the two parallel residual blocks to three serial residual blocks and a deconvolution block to enhance super-resolution and obtain a super-resolution image; the improved backbone network ResNet50 is obtained by improving the backbone network ResNet50; the improvements include: improvements to layer 5: improving the downsampling order step in the first Bottlneck and performing parallel connection between the first and second Bottlnecks and adding max pooling operations, while introducing two serial Bottlnecks and a deconvolution block to form a super-resolution structure; improvements to layer 3 and layer 4: performing parallel connection between the first and second Bottlnecks and adding max pooling operations, while introducing two serial Bottlnecks and a deconvolution block to form a super-resolution structure.

Images