Contact network parts defect detection method and device

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for detecting defects in contact network components. The method comprises: obtaining image data of the contact network components; intercepting the image data to obtain a target template image; processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model comprises a convolutional autoencoder model; and performing similarity matching between the target template image and the image of the component to be detected according to the feature extraction model to output a detection result. The use of this method can effectively improve the accuracy of the defect detection results of contact network components.

Description

Technical Field

[0001] The present application relates to the technical field of image detection and processing, and in particular to a method and device for detecting defects in contact network components. Background Art

[0002] The detection of nuts, cotter pins, insulators and other related parts by the overhead contact network suspension status detection and monitoring device (also known as the 4C system) in the railway system needs to be automatically identified and analyzed in high-resolution images to form maintenance recommendations and guide overhead contact network maintenance. Currently, defect identification almost entirely relies on manually annotated data sets, and then uses deep learning methods of neural networks to locate the target region (ROI) and detect and identify parts and defects.

[0003] However, manual labeling of large-scale images is time-consuming and laborious, and the effectiveness of traditional methods is highly dependent on the quality of data labeling, resulting in inaccurate detection results. Summary of the invention

[0004] Based on this, it is necessary to provide a contact network component defect detection method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a method for detecting defects in contact network components. The method comprises:

[0006] Acquiring image data of the contact network components;

[0007] intercepting the image data to obtain a target template image;

[0008] Processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model includes a convolutional autoencoder model;

[0009] According to the feature extraction model, similarity matching is performed on the target template image and the image of the component to be detected to output a detection result.

[0010] In one embodiment, the step of processing the image data based on a deep convolutional neural network model to obtain a feature extraction model includes:

[0011] Performing sliding window cutting of a preset size on the image data to obtain image block data; the preset size includes a large size;

[0012] Performing pre-detection processing on the image block data to obtain target image block data;

[0013] Cutting the target image block data into blocks to obtain sub-image block data;

[0014] The sub-image block data is input into the convolutional autoencoder model for training to obtain the feature extraction model.

[0015] In one embodiment, the step of performing pre-detection processing on the image block data to obtain target image block data includes:

[0016] Performing grayscale processing on the image block data to obtain grayscale image block data;

[0017] Based on the image segmentation model, binarizing the grayscale image block data to obtain binary image block data;

[0018] Performing morphological change processing on the binary image block data to obtain corresponding contour area data of each binary image block;

[0019] The binary image block whose contour area data is greater than or equal to the contour area threshold is determined as the target image block data.

[0020] In one embodiment, the convolutional autoencoder model includes an encoding model and a decoding model; the step of inputting the sub-image block data into the convolutional autoencoder model for training to obtain the feature extraction model includes:

[0021] The encoding model and the decoding model are used in sequence to train the sub-image block data for a preset number of times; when the preset number of trainings is completed, the decoding model is eliminated, and the retained encoding model is determined as the feature extraction model.

[0022] In one embodiment, the step of performing similarity matching between the target template image and the image of the component to be detected based on the feature extraction model to output the detection result includes:

[0023] The image of the component to be inspected is sequentially subjected to block cutting, pre-inspection and block cropping processing to obtain a sub-image block to be inspected;

[0024] Inputting the target template image and the sub-image block to be detected into the feature extraction model for forward propagation respectively, so as to extract the template intermediate layer feature map and the intermediate layer feature map to be detected;

[0025] The template intermediate layer feature map and the intermediate layer feature map to be detected are processed to obtain the detection result.

[0026] In one of the embodiments, the template intermediate layer feature map includes a template content feature map and a template style feature map; the intermediate layer feature map to be detected includes a content feature map to be detected and a style feature map to be detected; the detection result includes a defect classification result and a defect location result;

[0027] The step of processing the template intermediate layer feature map and the intermediate layer feature map to be detected to obtain the detection result comprises:

[0028] Determining content distance data of each to-be-detected sub-image block with respect to each target template image based on the template content feature map and the to-be-detected content feature map;

[0029] Determining style distance data of each of the sub-image blocks to be detected with respect to each of the target template images according to the template style feature map and the style feature map to be detected;

[0030] Based on the content distance data and the style distance data, the defect classification result and the defect location result are obtained in sequence.

[0031] In a second aspect, the present application also provides a contact network component defect detection device. The device comprises:

[0032] A data acquisition module, used to acquire image data of the contact network components;

[0033] An interception module, used for intercepting the image data to obtain a target template image;

[0034] A feature extraction module, used to process the image data based on a deep convolutional neural network model to obtain a feature extraction model;

[0035] The detection processing module is used to perform similarity matching on the target template image and the image of the component to be detected according to the feature extraction model to output the detection result.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Acquiring image data of the contact network components;

[0038] intercepting the image data to obtain a target template image;

[0039] Processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model includes a convolutional autoencoder model;

[0040] According to the feature extraction model, similarity matching is performed on the target template image and the image of the component to be detected to output a detection result.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0042] Acquiring image data of the contact network components;

[0043] intercepting the image data to obtain a target template image;

[0044] Processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model includes a convolutional autoencoder model;

[0045] According to the feature extraction model, similarity matching is performed on the target template image and the image of the component to be detected to output a detection result.

[0046] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Acquiring image data of the contact network components;

[0048] intercepting the image data to obtain a target template image;

[0049] Processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model includes a convolutional autoencoder model;

[0050] According to the feature extraction model, similarity matching is performed on the target template image and the image of the component to be detected to output a detection result.

[0051] The above-mentioned contact network component defect detection method, device, computer equipment, storage medium and computer program product obtain a target template image by intercepting the acquired image data of the contact network components, and uses a deep convolutional neural network model to process the image data to obtain a feature extraction model, and then performs similarity matching on the target template image and the image of the component to be detected according to the feature extraction model, thereby outputting the detection result; the present application can obtain a highly generalized feature extraction network through unsupervised deep learning model training. Compared with deep learning models that require manually labeled data sets for target detection tasks and image classification tasks, the present application does not need to rely on manually labeled data sets, which can greatly reduce the model experience risk and effectively improve the accuracy of the contact network component defect detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a process of a method for detecting defects in contact network components in one embodiment;

[0053] Figure 2 A schematic flow chart of steps for processing image data based on a deep convolutional neural network model to obtain a feature extraction model in one embodiment;

[0054] Figure 3 A schematic diagram of a sliding window cutout in one embodiment;

[0055] Figure 4 It is a flowchart of the steps of performing pre-detection processing on the image block data to obtain the target image block data in one embodiment;

[0056] Figure 5 Schematic diagram of the specific structure of a convolutional autoencoder model in one embodiment;

[0057] Figure 6 It is a schematic diagram of the structure of a contact network component defect detection device in one embodiment. DETAILED DESCRIPTION

[0058] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0060] In the paper “A fast and robust convolutional neural network-based defect detection model in product quality control”, Tian Wang, Yang Chen et al. proposed to slice the input image by sliding a window from the upper left corner of the target image, moving from left to right and from top to bottom in units of the upper left corner pixel of a specified size, and then send the sliced ​​image blocks to the deep network for learning. During inference, position-by-position recognition is also performed by sliding window detection. A similar template matching algorithm is to find out whether the template image exists in the target image based on the specified template image. Every time a pixel is reached, an image of the same size as the template is cut out from the source image with this pixel as the upper left corner vertex, and a pixel-wise two-dimensional comparison operation is performed with the template to find the area with the highest similarity as the matching result.

[0061] Junwen Chen et al. proposed a three-stage defect localization method in the paper "Automatic Defect Detection of Fasteners on the Catenary Support Device Using Deep Convolutional Neural Network". First, the target detection technology is used to use the SSD (Single Shot MultiBox Detector, SSD) algorithm as the detection network to locate the position of the main structural parts; then YOLO (You Only LookOnce, YOLO) is used to quickly detect the fasteners identified in the first stage; finally, the detection image is cropped and sent to the classification network for classification to analyze whether there are any missing fasteners.

[0062] Xian Tao et al. designed a cascaded autoencoder (CASAE) structure in the paper "Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks" for defect segmentation and localization. The cascade network converts the input defect image into a pixel-level prediction mask based on semantic segmentation. The compressed convolutional neural network (CNN) is used to divide the defect area of ​​the segmentation result into specific classes.

[0063] The above literatures all pre-train image classification models based on deep learning, using image categories as labels, inputting image features into a feature extraction structure consisting of convolutional layers, pooling layers, etc. to obtain high-dimensional feature vectors, and then mapping them to the category vector space at the output layer to obtain the predicted probability of each category, and then training the classification model by optimizing the cross entropy loss function. However, the effectiveness of these methods is highly dependent on the quality and scale of data annotation.

[0064] In terms of rapid detection of image geometric features based on image pixel information, the contact network suspension defect recognition designed by Zhou Wen et al. in "High-speed rail contact network suspension fastener defect recognition method based on two-stage cascaded lightweight convolutional neural network" consists of two parts: fastener detection and defect recognition. The fastener detection part contains three modules: lightweight feature extraction network, global attention module, mutually enhanced classifier and detector. The classifier is used to determine whether the image contains fasteners, and the detection is used to detect the position and category of the fasteners. The mutual enhancement of the two is achieved through the multi-task loss function. Through these three modules, the fasteners can be located and classified. The fastener defect recognition part can identify fastener defects for a given fastener image. The overall process is to cut the input image into blocks and then input it into the fastener detection network to locate the fastener. Then the output fastener image is scaled to 72*72 pixels as the input of the fastener defect recognition network to complete the final defect recognition.

[0065] Wang Ling et al. proposed a fastener location algorithm based on the statistics of pixel point scanning in a specific area in the article "A Rail Fastener Nut Missing Detection System Based on Computer Vision". The principal component analysis (PCA) method was used to extract the nut image features, and the minimum distance classifier was used for classification. The location of the fastener is to binarize the image with a pixel value of 75 as the threshold, and then locate the sleeper area according to the number of pixel values ​​1 in the area, and then locate the position of the fastener. After locating the fastener, the PCA algorithm is used to reduce the dimension of the fastener, and then the image is classified according to the Euclidean distance between the features.

[0066] Dai Peng et al. proposed a fast fastener sub-graph localization algorithm based on confidence map and a fastener defect image recognition method based on semi-supervised deep learning in the article "Ballastless track fastener defect image recognition method based on semi-supervised deep learning". The fast fastener sub-graph localization algorithm based on confidence map means that firstly, a fastener sub-graph neighborhood is formulated as a guide map in the track image based on prior information, and the texture map of the area and the confidence map of the fastener sub-graph in the area are constructed. Then, according to the approximate interval between adjacent fasteners in the track image space, the next fastener sub-graph neighborhood is extracted and its texture map and confidence map are constructed, and then the fastener sub-graph in the new area is located by calculating the maximum value point of the correlation function between the confidence map and the guide map. Fastener defect image recognition based on semi-supervised deep learning means that unsupervised learning is first performed on a large data set to obtain representation features, and then supervised learning is performed on a relatively small labeled data set to obtain classification capabilities.

[0067] A paper with the patent number 2020104102387 and the patent name "High-speed rail contact network fastener identification and positioning method based on attention mechanism" proposes a deep network model, including two modules: attention module and region recommendation. First, the data is input into the attention network of ResNet (Residual Neural Network, ResNet) 50, and then repeatedly input into the attention module, and finally a deep network feature map with the feature representation of the contact network fasteners is obtained. Then this deep network feature map is input into the region recommendation module, and the border regression is realized according to a pixel classification network. Finally, the location and category of different fasteners can be obtained.

[0068] Another paper with patent number 2017108643563 and patent name "A method for identifying and detecting fasteners of wrist-arm connectors of a high-speed railway contact network support device" proposes to first mark the wrist-arm connectors in the image, and then use the Faster R-CNN (Region-Convolutional Neural Network, R-CNN) network model to locate the wrist-arm connectors, and then mark the fasteners of the contact network into 6 categories according to their orientation and whether they are abnormal, and then use the Faster R-CNN network model based on the wrist-arm connectors to identify and locate the fasteners.

[0069] Another paper with patent number 2020105497232 and titled "A method for detecting abnormalities of high-speed rail contact network insulator fasteners" proposes to first label the components, and then use the Faster R-CNN network model to identify and locate the fasteners. The identified contact network components are labeled into 6 categories according to their orientation and whether they are abnormal, and then the fastener images are detected for abnormalities using a distance metric deep network.

[0070] From the perspective of the goal of 4C system fastener defect identification, the existing technology generally has the following shortcomings:

[0071] Directly using a small-sized sliding window to cut the original image will result in a slow execution speed and will detect a large amount of semantically non-semantic background information in the 4C image. However, directly using the template matching algorithm is very sensitive to the feature distribution of the template image. This means that when the content of the template image actually exists in the target image, but image deformations such as aspect ratio scaling occur, the calculated best matching area often cannot be accurately positioned. This has great limitations on the accurate positioning of the target and is difficult to meet the matching and positioning needs of complex images in actual scenes.

[0072] The pre-trained image classification models based on deep learning often have high model complexity and a large number of parameters. If fine-tuning is performed directly at the output classification end, it is difficult to ensure a very high accuracy in the identification and classification of defects. If all parameters are updated to obtain high precision, huge gradient calculation costs are required.

[0073] Image processing algorithms can extract satisfactory ROI targets from images with prominent geometric features through binary image transformation algorithms and contour extraction algorithms. Among them, image binarization algorithms, such as threshold-based binarization, are sensitive to local grayscale information. In practical applications, if the contrast between the foreground target and the background is not large, it will be difficult to effectively segment the image to obtain a binary image with clear features, which is not conducive to subsequent feature contour characterization.

[0074] In addition to the common defects of being relatively dependent on annotated data, relatively complex algorithm models, and high costs, the above-mentioned papers and patent documents also have some other limitations. For example, the methods proposed in the papers "A rail fastener nut missing detection system based on computer vision" and "A ballastless track fastener defect image recognition method based on semi-supervised deep learning" are relatively special. The positioning of fasteners cannot be applied in other situations, and the high-speed rail contact network fasteners cannot be effectively positioned, and the classification effect of other data defects is not good. The deep network models proposed in the three patents have poor processing effect and efficiency for data with extremely high signal-to-noise ratios. Among them, the patents of "High-speed rail contact network fastener identification and positioning method based on attention mechanism" and "A high-speed rail contact network support device arm connector fastener identification and detection method" only realize the classification and positioning of different fasteners, and do not realize the abnormal detection of fasteners.

[0075] The contact network component defect detection method provided in the present application does not need to rely on manually annotated data sets, and can obtain a feature extraction network with extremely strong generalization. Compared with deep learning models that require manual organization and annotation for target detection tasks and image classification tasks, it greatly reduces the annotation burden and reduces the model's empirical risk, thereby effectively improving the accuracy of contact network component defect detection results.

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] In one embodiment, Figure 1 As shown, a method for detecting defects in contact network components is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0078] Step S110, acquiring image data of contact network components;

[0079] Specifically, image data of the status of suspended components (or fasteners) of heavy-duty railway contact networks are obtained. In some examples, the image data can be obtained by taking images from a global or local angle by an imaging device on the roof of an inspection vehicle. The image size can be 5120px×5120px, where the inspection vehicle starts from a designated line and takes high-definition images of all contact networks along the line. Furthermore, the image data can include image data of nuts, cotter pins, and insulators. In some examples, the inspection line can be used as the root directory for storage, the second-level directory is for each quarter, and the third-level directory is for the name of each component. The image data of each component collected is saved, and the image data we have captured is stored in this file structure.

[0080] Step S120, intercepting the image data to obtain a target template image;

[0081] Specifically, various parts with clear features are extracted from the acquired image data as target template images. In some examples, it is possible to observe whether the original image contains parts of various categories and sort out images of parts containing the detection targets. Further, the images of parts to be detected are cropped (cropped), and the cropped size is uniformly 200px×200px. For example, target template images of normal nuts and nuts that are missing, loose, or inserted in the wrong direction, target template images of normal cotter pins and cotter pins that are missing, have insufficient angles, or are inserted in the wrong direction, and target template images of normal insulators and damaged insulators are cropped.

[0082] Step S130, processing the image data based on a deep convolutional neural network model to obtain a feature extraction model; the deep convolutional neural network model includes a convolutional autoencoder model;

[0083] Specifically, the deep convolutional neural network model is trained using the image data so that it can extract features of the target area image, thereby obtaining a feature extraction model; in some examples, a convolutional autoencoder model can be selected as a training model;

[0084] In one embodiment, if Figure 2 As shown, the step S130 of processing the image data based on the deep convolutional neural network model to obtain the feature extraction model may include:

[0085] Step S210, performing sliding window cutting of a preset size on the image data to obtain image block data; the preset size includes a large size;

[0086] Specifically, all image data are cut into blocks with a sliding window of a preset size. Furthermore, a large-size sliding window can be used for cutting to obtain image block data, wherein each image can be cut into several small image blocks by sliding window cutting. In some examples, since the size of the collected image data is uniformly 5120px×5120px, and the size of the target template image is uniformly 200px×200px, the preset size (block height and width) of the initial sliding window cutting can be set to 512px×512px, and 50% overlap can be retained during cropping.

[0087] In a specific example, the schematic diagram of sliding window slicing of image data is as follows: Figure 3 As shown, the size of the image data is 5120px×5120px, the preset size of the initial sliding window block is 512px×512px, 50% overlap is retained during cropping, and the stride is 256px.

[0088] The above method obtains the positioning target through efficient sliding window cutting and realizes adaptive image cropping.

[0089] Step S220, performing pre-detection processing on the image block data to obtain target image block data;

[0090] Specifically, in some examples, an image processing algorithm can be used to perform pre-detection processing on image block data, eliminate background image blocks, retain image blocks that may contain detection targets, and obtain target image block data. The image processing algorithm can also perform adaptive threshold segmentation on the pixel information of a single image block to achieve faster image feature detection using multi-channel convolution operations at multiple scales.

[0091] In one embodiment, if Figure 4 As shown, the step S220 of performing pre-detection processing on the image block data to obtain the target image block data may include:

[0092] Step S410, grayscale processing is performed on the image block data to obtain grayscale image block data;

[0093] Step S420, based on the image segmentation model, binarizing the grayscale image block data to obtain binary image block data;

[0094] Step S440, performing morphological change processing on the binary image block data to obtain corresponding contour area data of each binary image block;

[0095] Step S440: Determine the binary image block whose contour area data is greater than or equal to the contour area threshold as the target image block data.

[0096] Specifically, the pixels of each channel of the image block data of the three color channels are averaged to obtain the gray image block data of a single channel, and then the gray image block data is denoised by Gaussian smoothing with a kernel size of 3×3, median smoothing with a kernel size of 5×5, and Sobel operator with a kernel size of 3×3 to make the feature edges of the gray image clearer; then the gray image block data is binarized using the image segmentation model. In some examples, a local adaptive threshold segmentation algorithm can be used to obtain binary image block data;

[0097] Further, the binary image block data is subjected to morphological change processing. In some examples, a 9×1 rectangular template and a 9×7 rectangular template can be constructed for the binary image block data, respectively, wherein the former rectangular template is used for dilation operation, and the latter rectangular template is used for erosion operation; then, one dilation, one erosion, and three dilations are performed in sequence, so that local holes in each binary image in the binary image block data are filled and local isolated points are eliminated, thereby forming a complete feature area for facilitating contour search and extraction;

[0098] Further, contour extraction is performed on the binary image blocks after the morphological change processing to obtain contour area data corresponding to each binary image block, so that the binary image blocks whose contour area data is greater than or equal to the contour area threshold are determined as target image block data;

[0099] In some examples, since the size of the target template image is 200px×200px, the contour area threshold can be set to 10000; the Suzuki algorithm (contour tracking algorithm) can be executed on the binary image block data after morphological change processing to find the largest circumscribed contour and obtain the contour area. If the contour area is larger than the contour area threshold, the corresponding binary image block is retained and determined as the target image block for further block cutting and positioning.

[0100] In the above-mentioned process of positioning the detection target, a large-size sliding window is used to obtain an image block, and then the image block is pre-detected using an image processing algorithm. This will discard background information that is not required for detection and greatly reduce the high time overhead of generating small image blocks and then classifying them in sequence in the prior art. At the same time, it effectively avoids the global search of the image by the target detection model, and also avoids the stringent requirements on the target features of the original image by directly using the image processing algorithm to obtain the target, such as requiring the original image to have good geometric features or a high signal-to-noise ratio, etc., which are prerequisites for easy image segmentation. The present application only needs to use a simple transformation to determine whether there is an ROI in the image block, which effectively overcomes the defects of the image segmentation algorithm having high requirements on image quality during the binarization process and difficulty in finding the target after segmentation. This image processing algorithm pre-detection method can quickly filter out a large amount of background information that is not required for detection in the 4C system image.

[0101] Step S230, cutting the target image block data into blocks to obtain sub-image block data;

[0102] Specifically, each target image block in the target image block data is further cropped to obtain sub-image block data of smaller size; in some examples, since the current target image block is 512px×512px in height and width, the block size is set to 256px×256px, and 50% overlap is retained during cropping.

[0103] The above-mentioned step-by-step slicing after pre-detection processing using an image processing algorithm can locate the ROI more quickly and accurately, and provide reliable data preprocessing for training the convolutional autoencoder model.

[0104] Step S240: input the sub-image block data into a convolutional autoencoder model for training to obtain a feature extraction model.

[0105] Specifically, the bidirectional convolutional autoencoder model can process the information of the sub-image block data and obtain a feature extraction model through unsupervised learning training.

[0106] The above-mentioned method of training the convolutional autoencoder model through unlabeled sub-image block data belongs to unsupervised learning. Theoretically, it can be trained on an infinite amount of data to obtain a feature extraction model with extremely strong generalization ability. Moreover, this feature extraction model does not have to be used for classification, which also reduces the labeling overhead. Compared with deep learning models that require manual organization and labeling for target detection tasks and image classification tasks, it greatly reduces the labeling burden and reduces the model's empirical risk.

[0107] In one embodiment, the convolutional autoencoder model includes an encoding model and a decoding model; the sub-image block data is input into the convolutional autoencoder model for training, and the step S240 of obtaining the feature extraction model includes:

[0108] The encoding model and the decoding model are used in turn to train the sub-image block data for a preset number of times; when the preset number of trainings is completed, the decoding model is eliminated and the retained encoding model is determined as the feature extraction model.

[0109] Specifically, the sub-image block data is first input into the encoding model, which can perform compression according to the information of the sub-image block data to achieve feature extraction. The feature map data extracted from the sub-image block data is then input into the decoding model for a preset number of trainings, so that the output of the decoding model (decompression model) is close to the original sub-image block data. When the training is completed, the decoding model is eliminated, and the retained encoding model is the feature extraction model.

[0110] In order to better illustrate this application, a specific example is given below:

[0111] The encoding model and the decoding model can be constructed using an encoding network and a decoding network respectively. Assuming that the input sub-image block is X, the encoding network is represented by function f, and the decoding network is represented by function g. Then the output of the encoding network encoded can be expressed as the following formula (1), and the output of the decoding network decoded can be expressed as the following formula (2):

[0112] encoded=f(X) (1)

[0113] decoded=g(encoded)=g(f(X)) (2)

[0114] The loss function is shown in the following formula (3):

[0115] loss(X, decoded) = ∑ i,j (X i,j-decoded i,j ) 2 (3)

[0116] Among them, i and j represent the coordinate position; the smaller the loss function is, the closer the output of the decoding network is to the original image. Finally, the decoding network, i.e., function g, is eliminated, and only the encoding network shown in formula (1) is retained as the feature extraction model, i.e., the feature extraction network.

[0117] In some examples, the encoding network includes convolutional layers and maxpooling layers, where maxpooling is responsible for spatial downsampling, and specifically adopts the VGG-19 (Visual Geometry Group, VGG) neural network model. The decoding network includes convolutional layers and upsampling, and specifically adopts a structure similar to VGG-19, but opposite to the encoding network. In one example, the specific structural diagram of the convolutional autoencoder model is as follows Figure 5 As shown, in detail, the process of encoding network training may include the following steps:

[0118] (1) Assume that the data dimension of the input sub-image block is (batch size, 3, h, w), where batch size indicates the data size of one iteration, 3 indicates the number of initial input channels, h indicates the height of the sub-image block, and w indicates the width of the sub-image block; the sub-image block data is input into two convolutional layers with a convolution kernel of 3×3, and then passes through a maxpooling layer. Then, the height and width (hereinafter referred to as height and width) of the sub-image block data dimensions are halved for the first time, and the number of channels becomes 64. Figure 5 In the above code, conv3-64 represents the convolution layer parameters of the process, where 3 represents the size of the convolution kernel and 64 represents the number of channels.

[0119] (2) The output of step (1) is input into two convolutional layers with 3×3 convolution kernels, and then passes through a maxpooling pooling layer. The height and width are halved for the second time, and the number of channels becomes 128. Figure 5 In the above code, conv3-128 represents the convolution layer parameters of this process;

[0120] (3) The output in step (2) is passed through four convolutional layers with 3×3 kernels and one pooling layer. The height and width are halved for the third time, and the number of channels becomes 256. Figure 5 In the above code, conv3-256 represents the convolution layer parameters of this process;

[0121] (4) Similarly, the output of step (3) is passed through four convolutional layers with a 3×3 kernel and a maxpooling layer. The height and width are halved for the fourth time, and the number of channels becomes 512. Figure 5Where conv3-512 represents the convolution layer parameters of this process;

[0122] (5) Finally, the output of step (4) is passed through four more convolutional layers with a 3×3 convolution kernel and a maxpooling layer. The height and width are halved for the fifth time, and the number of channels remains unchanged at 512. Figure 5 In the above figure, conv3-512 represents the convolution layer parameters of this process; the output at this time is encoded.

[0123] Accordingly, the process of decoding network training may include the following steps:

[0124] ① Input the output of step (5) in the encoding network training, i.e., encoded, into four convolutional layers and one upsampling layer. The height and width are doubled for the first time, and the number of channels remains unchanged, which is still 512.

[0125] ② The output of step ① is passed through four convolutional layers and one upsampling layer, so the height and width are doubled for the second time, and the number of channels remains unchanged, which is still 512;

[0126] ③ The output in step ② is passed through four convolutional layers and one upsampling layer in the same way, so the height and width are doubled for the third time, and the number of channels becomes 256;

[0127] ④ Input the output of step ③ into two convolutional layers and one upsampling layer, then the height and width are doubled for the fourth time, and the number of channels becomes 128;

[0128] ⑤ The output in step ④ is passed through two more convolutional layers, the number of channels becomes 64, and through an upsampling layer, the height and width are doubled for the fifth time;

[0129] ⑥Finally, the output in step ⑤ passes through another convolution layer, and the number of channels returns to 3. At this time, the output is consistent with the data dimensions of the initial sub-image block in terms of both the number of channels and the height and width.

[0130] As mentioned above, the preset number of training processes of the encoding network and the decoding network can be represented by t. Specifically, the t parameter can be set according to the tolerable computing time, that is, assuming that the encoding network and the decoding network are calculated for 10 seconds and the tolerable time is 600 seconds, then t=60.

[0131] When the preset number of trainings are completed, the neural network part in the decoding network is removed, and only the encoding network part is retained. The retained part is the convolution part of the VGG neural network, which is used as the feature extraction network. Its output encoded is the feature map data extracted from the sub-image block data.

[0132] Step S140: performing similarity matching on the target template image and the image of the component to be inspected according to the feature extraction model to output the inspection result.

[0133] Specifically, using the feature extraction model obtained above, the image of the component to be detected and the target template image are respectively input into the feature extraction model for similarity matching, thereby obtaining a detection result.

[0134] The feature extraction model obtained by training the convolutional autoencoder model is used for feature similarity calculation, and its network calculation complexity is much lower than the calculation complexity of the target detection model in the prior art.

[0135] In one embodiment, the step of performing similarity matching on the target template image and the image of the component to be detected according to the feature extraction model to output the detection result includes:

[0136] The image of the component to be inspected is sequentially cut into blocks, pre-inspected, and cut into blocks to obtain a sub-image block to be inspected;

[0137] The target template image and the sub-image block to be detected are respectively input into the feature extraction model for forward propagation to extract the template intermediate layer feature map and the intermediate layer feature map to be detected;

[0138] The template intermediate layer feature map and the intermediate layer feature map to be detected are processed to obtain the detection result.

[0139] Specifically, first, the image of the component to be detected is subjected to the same sliding window slicing operation of the preset size as in step S210. In some examples, the preset size of 512px×512px can be selected for sliding window slicing to obtain an image block to be detected. A single image of the component to be detected can obtain a number of image blocks to be detected. Further, the same pre-detection processing as in step S220 is performed. Specifically, the image block to be detected is subjected to pre-detection processing using an image processing algorithm. Further, the image block to be detected that has undergone the pre-detection processing is subjected to the same block cropping processing as in step S230. In some examples, the size of the block cropping is also set to 256px×256px to obtain a number of sub-image blocks to be detected.

[0140] Furthermore, the obtained sub-image block to be detected and the target template image are respectively input into the feature extraction model for forward propagation. The target template image can extract the corresponding template intermediate layer feature map through the feature extraction model, and the sub-image block to be detected can extract the corresponding intermediate layer feature map to be detected through the feature extraction model. Then, the template intermediate layer feature map and the intermediate layer feature map to be detected are similarly matched to obtain the detection result.

[0141] In one embodiment, the template intermediate layer feature map may include a template content feature map and a template style feature map; the intermediate layer feature map to be detected may include a content feature map to be detected and a style feature map to be detected; the detection result includes a defect classification result and a defect location result;

[0142] The steps of processing the template intermediate layer feature map and the intermediate layer feature map to be detected and obtaining the detection result include:

[0143] Determining content distance data of each sub-image block to be detected with respect to each target template image based on the template content feature map and the content feature map to be detected;

[0144] Determining style distance data of each sub-image block to be detected with respect to each target template image according to the template style feature map and the style feature map to be detected;

[0145] Based on the content distance data and the style distance data, the defect classification results and the defect location results are obtained in turn.

[0146] Specifically, in some examples, since the feature extraction network is a VGG-19 architecture, the middle layer at the top of the feature extraction network, the

[25] th layer, can be used as a content layer to calculate the large-scale feature similarity between the sub-image block and the target template image, and determine the content distance data of each sub-image block to be detected with respect to each target template image based on the template content feature map extracted by the content layer and the content feature map to be detected;

[0147] In addition, the middle layer at the bottom of the feature extraction network, the [0, 5, 10, 19, 28]th layer, can be used as a style layer to calculate the local texture feature similarity between the sub-image block and the target template image, and the style distance data of each sub-image block to be detected with respect to each target template image can be determined according to the template style feature map extracted by the style layer and the style feature map to be detected;

[0148] Furthermore, based on the obtained content distance data and style distance data, defect classification results and defect location results of the component to be inspected are obtained.

[0149] In a specific example, the target template image can be converted into a 4-dimensional mini-batch tensor represented by a value from 0 to 1 through image preprocessing, and each dimension is represented as (number of templates, 3, height, width), where 3 represents the number of initial input channels, and then input into the feature extraction network for forward propagation, and the template content feature map extracted from the content layer is saved, and each template style feature map extracted from the style layer is respectively calculated to obtain each Gram matrix, and the Gram matrix of each style layer is saved. The specific calculation method is as follows:

[0150] Since the feature map dimension of the current style layer is (number of templates, c, h, w), where c represents the number of channels, h and w represent the height and width of a single feature map respectively, the last two dimensions are flattened to obtain (number of templates, c, h*w), and then the Gram matrix of the dot product between the feature vectors represented by each channel in the last two dimensions is calculated. Specifically, the Gram matrix can be calculated by the following formula (4):

[0151] Gram i,j = <c i , c j > (4)

[0152] Among them, c i represents the i-th channel, c j represents the jth channel; accordingly, according to the above extraction method, the content feature map to be detected under the content layer of the sub-image block to be detected and each style feature map to be detected under each style layer are obtained, and each Gram matrix is ​​calculated according to each style feature map to be detected;

[0153] Furthermore, L2_dist is calculated for each target template image and each content feature set of the sub-image block to be detected. content The distance is calculated by the following formula (5):

[0154]

[0155] And calculate L2_dist based on the Gram matrix of the target template image and the sub-image block to be detected style The distance is calculated by the following formula (6):

[0156]

[0157] Among them, in the above formula (5) and the above formula (6), l represents the number of layers of the neural network model VGG-19, "-" represents element-by-element subtraction of the matrix, square represents element-by-element square of the matrix, and C in formula (5) represents the feature map matrix under each channel.

[0158] The content distance dimension of each sub-image block to be detected with respect to each target template image obtained above is expressed as (number of sub-image blocks * number of templates, C, h, w), and the style distance dimension is expressed as (number of sub-image blocks * number of templates, Gram matrix); further, the last three digits in the content distance dimension are averaged to obtain the content distance of each sub-image block to be detected with respect to each template; the element mean of the Gram matrix in the style distance dimension is taken to obtain the style distance of each sub-image block to be detected with respect to each template, where the smaller the distance, the greater the similarity.

[0159] Further, the category of the target template image with the maximum similarity to the sub-image block to be detected is selected as the defect classification result of the current input component image to be detected. Specifically, the maximum similarity judgment process is to select the target template image with the minimum style distance to the sub-image block to be detected as the defect category of the sub-image block to be detected. If there are multiple target template images with the same style distance, then the target template image with the minimum content distance to the sub-image block to be detected is selected from the multiple target template images as the defect category of the sub-image block to be detected.

[0160] Furthermore, defect classification results of several sub-image blocks to be detected can be obtained. Among the sub-image blocks to be detected whose defect categories have been determined, similar to the above-mentioned maximum similarity judgment process, the sub-image block with the greatest similarity to the target template image under this defect category is selected as the defect positioning result of the current input image of the component to be detected. Since the number of block cutting times and the number of valid blocks (including the image blocks of the detection target) are small, the positioning of the image of the component to be detected is faster.

[0161] The above-mentioned contact network component defect detection method uses an unlabeled data set to train a convolutional autoencoder model to obtain a feature extraction model for unsupervised extraction of features in contact network related images, and calculates the similarity between images through image style differences. Based on this, the target component position and defect category of the input image are predicted, thereby greatly improving the accuracy of the contact network component defect detection results.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a contact network component defect detection device for implementing the contact network component defect detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of a contact network component defect detection device provided below can refer to the limitations of the contact network component defect detection method above, and will not be repeated here.

[0164] In one embodiment, Figure 6 As shown, a contact network component defect detection device is provided, including: a data acquisition module 610, an interception module 620, a feature extraction module 630 and a detection processing module 640, wherein:

[0165] A data acquisition module 610 is used to acquire image data of contact network components;

[0166] The interception module 620 is used to intercept the image data to obtain a target template image;

[0167] A feature extraction module 630 is used to process the image data based on a deep convolutional neural network model to obtain a feature extraction model;

[0168] The detection processing module 640 is used to perform similarity matching between the target template image and the image of the component to be detected according to the feature extraction model to output the detection result.

[0169] Each module in the above-mentioned contact network component defect detection device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0170] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting defects in contact network components is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0171] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0172] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0173] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0175] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting defects in contact network components, characterized in that: The method comprises: Acquiring image data of the contact network components; intercepting the image data to obtain a target template image; Performing sliding window cutting of a preset size on the image data to obtain image block data; the preset size includes a large size; Performing pre-detection processing on the image block data to obtain target image block data; The target image block data is cut into blocks to obtain sub-image block data, and the sub-image block data is input into a convolutional autoencoder model for training to obtain a feature extraction model; The image of the component to be inspected is sequentially cut into blocks, pre-inspected, and cut into blocks to obtain a sub-image block to be inspected; The target template image and the sub-image block to be detected are respectively input into the feature extraction model for forward propagation to extract the template intermediate layer feature map and the intermediate layer feature map to be detected; the template intermediate layer feature map includes a template content feature map and a template style feature map; the intermediate layer feature map to be detected includes a content feature map to be detected and a style feature map to be detected; Determining content distance data of each sub-image block to be detected with respect to each target template image based on the template content feature map and the content feature map to be detected, and determining style distance data of each sub-image block to be detected with respect to each target template image based on the template style feature map and the style feature map to be detected; Based on the content distance data and the style distance data, a defect classification result and a defect location result are obtained in turn.

2. The method according to claim 1, characterized in that The step of performing pre-detection processing on the image block data to obtain target image block data comprises: Performing grayscale processing on the image block data to obtain grayscale image block data; Based on the image segmentation model, binarizing the grayscale image block data to obtain binary image block data; Performing morphological change processing on the binary image block data to obtain corresponding contour area data of each binary image block; The binary image block whose contour area data is greater than or equal to the contour area threshold is determined as the target image block data.

3. The method according to claim 1, characterized in that The convolutional autoencoder model includes an encoding model and a decoding model; the step of inputting the sub-image block data into the convolutional autoencoder model for training to obtain a feature extraction model includes: The encoding model and the decoding model are used in sequence to train the sub-image block data for a preset number of times; when the preset number of trainings is completed, the decoding model is eliminated, and the retained encoding model is determined as the feature extraction model.

4. The method according to claim 3, characterized in that The sequentially using the encoding model and the decoding model to train the sub-image block data for a preset number of times includes: Inputting the sub-image block data into the encoding model to obtain feature map data corresponding to the sub-image block data; The feature map data is input into the decoding model for a preset number of trainings.

5. A contact network component defect detection device, characterized in that: The device comprises: A data acquisition module, used to acquire image data of the contact network components; An interception module, used for intercepting the image data to obtain a target template image; A feature extraction module is used to perform sliding window slicing of the image data with a preset size to obtain image block data; the preset size includes a large size; perform pre-detection processing on the image block data to obtain target image block data; perform block cropping on the target image block data to obtain sub-image block data, and input the sub-image block data into a convolutional autoencoder model for training to obtain a feature extraction model; The detection processing module is used to perform block slicing, pre-detection and block cropping processing on the image of the component to be detected in sequence to obtain the sub-image block to be detected; the target template image and the sub-image block to be detected are respectively input into the feature extraction model for forward propagation to extract the template intermediate layer feature map and the intermediate layer feature map to be detected; the template intermediate layer feature map includes a template content feature map and a template style feature map; the intermediate layer feature map to be detected includes a content feature map to be detected and a style feature map to be detected; based on the template content feature map and the content feature map to be detected, determine the content distance data of each sub-image block to be detected with respect to each target template image, and, based on the template style feature map and the style feature map to be detected, determine the style distance data of each sub-image block to be detected with respect to each target template image; based on the content distance data and the style distance data, obtain the defect classification result and the defect location result in sequence.

6. The device according to claim 5, characterized in that When the feature extraction module is used to perform pre-detection processing on the image block data to obtain target image block data, it includes: Performing grayscale processing on the image block data to obtain grayscale image block data; Based on the image segmentation model, binarizing the grayscale image block data to obtain binary image block data; Performing morphological change processing on the binary image block data to obtain corresponding contour area data of each binary image block; The binary image block whose contour area data is greater than or equal to the contour area threshold is determined as the target image block data.

7. The device according to claim 5, characterized in that The convolutional autoencoder model includes an encoding model and a decoding model; when the feature extraction module is used to input the sub-image block data into the convolutional autoencoder model for training to obtain a feature extraction model, it includes: The encoding model and the decoding model are used in sequence to train the sub-image block data for a preset number of times; when the preset number of trainings is completed, the decoding model is eliminated, and the retained encoding model is determined as the feature extraction model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images