A self-supervised spring defect detection method based on data augmentation strategy

Abstract

The present invention discloses a self-supervised spring defect detection method based on a data enhancement strategy, comprising: obtaining a spring image to be detected; inputting the spring image to be detected into a trained spring defect detection model to obtain a spring defect detection result. In the process of training the spring defect detection model, a data enhancement strategy of randomly "cut-and-pasting" original image blocks is used to provide a supervisory signal for self-supervised learning, so that the network is able to learn the potential representation of the real pattern from a large amount of unlabeled data, and provide a useful visual representation for subsequent spring defect detection. Thereby, it is possible to quickly and accurately determine whether there is a defect in the spring part of a TVDS test image, exclude a large number of vehicle spring normal operating state images, screen out images with a high probability of abnormalities, and locate and display defective parts, greatly reducing the burden on staff, while improving the accuracy and timeliness of abnormality detection.

Description

Technical Field

[0001] The invention relates to the field of spring defect detection, and in particular to a self-supervised spring defect detection method based on a data enhancement strategy. Background Art

[0002] With the rapid development of social economy, the railway passenger volume has increased dramatically. As a railway passenger car safety technical equipment, TVDS (Train Coach Machine Vision Detection System) plays an increasingly important role in the railway passenger car safety prevention system. Due to the requirements of the existing TVDS system's technical characteristics and human resource issues, facing a large amount of real-time image data, the staff cannot obtain the abnormal information and fault location of vehicle parts in time, which seriously threatens the safety of passenger car operation. Therefore, establishing an effective real-time vehicle fault detection method and quickly discovering the fault information in vehicle operation is a problem that must be solved in the field of TVDS image anomaly detection. As the most common vehicle component, springs often suffer from wear and aging, loose joints and broken faults, which are representative of all vehicle abnormal conditions. Therefore, we designed an abnormal detection method for spring components in TVDS images. Summary of the invention

[0003] The present invention provides a self-supervised spring defect detection method based on a data enhancement strategy to solve the problem of low abnormality detection efficiency of springs in the existing TVDS system.

[0004] In order to achieve the above object, the present invention adopts the following technical solution.

[0005] A self-supervised spring defect detection method based on data enhancement strategy, comprising:

[0006] Acquire the image of the spring to be tested;

[0007] Input the image of the spring to be detected into the trained spring defect detection model to obtain the spring defect detection result;

[0008] Among them, the spring defect detection model is obtained by the following method:

[0009] Acquire several normal spring images;

[0010] For the normal spring image, use it as the original image, copy an original image as the base image, and make a pure black image with the same size as the original image as the mask image;

[0011] The original image, the base plate image, and the mask image are respectively divided into a plurality of regions;

[0012] Randomly select an area from each patch of the original image and crop an image block, paste it to a random position of any different patch of the mask image in sequence, and record the vertex coordinates of each image block after pasting;

[0013] According to the recorded vertex coordinates of each image block at the corresponding position in the base plate image, the pixel value in the area is erased, and then the processed base plate image is superimposed with the mask image to obtain the defective spring image;

[0014] A training sample set with category labels is constructed based on a number of normal spring images and the obtained spring images with defects;

[0015] The neural network model is trained based on the training sample set to obtain a spring defect detection model.

[0016] Furthermore, the original image, the base image, and the mask image are divided into a plurality of regions respectively, including:

[0017] Establish rectangular coordinate systems in the original image, base image, and mask image respectively, and divide the image into four quadrants;

[0018] The image block is a polygonal irregular image block.

[0019] Furthermore, before obtaining the image of the spring to be detected, the following steps are also included:

[0020] Acquire the captured spring image;

[0021] Input the spring image into the trained spring target detection model to detect the location of the spring;

[0022] The individual spring images are segmented according to the locations of the detected springs, and are cropped into preset sizes to obtain the spring images to be detected.

[0023] Furthermore, the spring target detection model is obtained by the following method:

[0024] Acquire a number of captured spring images;

[0025] Use the annotation tool to select the spring in the spring image and label it;

[0026] The processed spring images are used as labeled samples and input into the Detection Transformer model for training to obtain the spring object detection model.

[0027] Furthermore, the neural network model is trained based on the training sample set to obtain a spring defect detection model, including:

[0028] Constructing a neural network model including a first feature extractor and a binary classifier, taking each spring image in the training sample set as input and its corresponding category label as output, training the neural network model including the first feature extractor and the binary classifier to obtain a spring defect binary classification model;

[0029] Constructing a spring defect detection model including a second feature extractor and a single classifier; wherein the second feature extractor is obtained based on the migration of the first feature extractor, and the single classifier outputs the probability that the input spring image has defects;

[0030] The obtained probability is compared with the preset defect threshold. When the probability is not greater than the defect threshold, it is considered that there is no defect; when the probability is greater than the defect threshold, it is considered that there is a defect.

[0031] Furthermore, in the process of training the spring defect binary classification model, the following loss function is used:

[0032]

[0033] Where N is the number of spring images in the training sample set, CEL is the cross entropy loss function, and x i represents the positive samples composed of normal spring images in the training sample set, which are replaced by the label "0", A(*) represents the simulated negative samples composed of defective spring images in the training sample set, which are replaced by the label "1"; c(*) represents the constructed binary classifier.

[0034] Furthermore, the probability that the spring image input by the single classifier has defects includes:

[0035] Get the spring image deep visual representation obtained by the second feature extractor, and calculate the Mahalanobis distance D between it and the preset potential real representation Mahalanobis As the probability of a spring image being defective;

[0036]

[0037] Where f(θ) represents the deep visual representation of the spring image obtained by the second feature extractor; X m represents the preset potential true representation, which is the mean of the deep visual feature vectors obtained by the second feature extractor of several normal spring images; X conv Represents the covariance matrix of the deep visual feature vectors obtained by the second feature extractor for several normal spring images.

[0038] Furthermore, it also includes:

[0039] For spring images identified as having defects, the defect position is located and displayed through a bounding box or heat map.

[0040] Furthermore, the method of displaying through the bounding box is as follows:

[0041] The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized;

[0042] The value of each pixel position in the normalized feature map is used as the abnormal probability estimate of the pixel and compared with the preset threshold. When the abnormal probability of the pixel is greater than the preset threshold, the horizontal and vertical coordinates of the pixel are recorded;

[0043] The maximum and minimum values ​​of the horizontal and vertical coordinates of all abnormal pixels are selected as the boundary vertices of the defect area to draw a predicted rectangular boundary box in the spring image to be tested;

[0044] The method of displaying through heat map is as follows:

[0045] The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized;

[0046] Assign the corresponding color to each pixel according to the value of the position of each pixel in the normalized feature map;

[0047] The processed feature map is superimposed with the original spring image to be detected to obtain a thermal map.

[0048] Furthermore, the spring image to be detected is named according to the shooting position and the train number and carriage information of the shooting;

[0049] When it is identified that the image of the spring to be detected has defects, the shooting position and train number and carriage information corresponding to the spring to be detected are reversely located according to the name of the image to be detected.

[0050] Beneficial Effects

[0051] The present invention proposes a self-supervised spring defect detection method based on data enhancement strategy, which provides supervision signals for self-supervised learning through the data enhancement strategy of randomly "cutting-pasting" irregular original image blocks, so that the neural network has the ability to learn the potential representation of the real pattern from a large amount of unlabeled data, and provides useful visual representation for subsequent spring defect detection. It can quickly and accurately determine whether there are defects in the spring part of a TVDS test image, exclude a large number of vehicle spring normal operation status images, screen out images with a high probability of abnormalities, and locate and display defective parts, greatly reducing the burden on staff, while improving the accuracy and timeliness of abnormality detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a flow chart of a self-supervised spring defect detection method based on a data enhancement strategy provided by an embodiment of the present invention;

[0054] Figure 2 It is a schematic diagram of some normal spring images provided by an embodiment of the present invention;

[0055] Figure 3 : The original image (left) and the defect image after data enhancement (right) provided by an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the display effect of the heat map (left) and bounding box (right) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0058] The main purpose of the present invention is to detect whether there are defects in the spring part in the TVDS high-resolution image, so as to provide intelligent real-time abnormal warning of the image, and realize real-time detection of whether there are faults in the spring part in the TVDS image under the premise of massive data. When the abnormal probability of the spring part in the spring image to be detected is calculated to be high, the abnormal information is automatically reported, and the potential abnormal (defective) area is located and displayed, reminding the user to confirm and manage the abnormal situation. The potential fault images of vehicles that need to be manually processed are greatly reduced, and the time cost and labor cost required for TVDS image detection are significantly reduced, while ensuring the accuracy and timeliness of the detection.

[0059] Based on the above purpose, the present invention provides a self-supervised spring defect detection method based on data enhancement strategy. Before implementing the detection method, it is necessary to first construct a spring target detection model and a spring defect detection model. The specific construction method is as follows:

[0060] S1: Building a spring image dataset

[0061] S11: obtaining a spring image of a vehicle in real-time running state taken by a TVDS high-speed shooting device deployed along the railway, and naming the spring image according to the shooting position and the train number and carriage information;

[0062] S12: Select part of the spring image, use the annotation tool to select the spring in the spring image, and mark the label;

[0063] S13: The processed spring image with the spring bounding box is input as a labeled sample into the DetectionTransformer model for training to obtain a spring object detection model;

[0064] Among them, the Detection Transformer model treats target detection as a set prediction problem. It adopts an "encoder-decoder" architecture based on the transformer structure, including a convolutional neural network backbone network that extracts image feature representations and a simple feedforward network for final detection prediction. It realizes image target detection from large to small and from global to local, and achieves a one-to-one match between the prediction results and the true values.

[0065] S14: inputting the remaining unlabeled spring images into the trained spring target detection model, and segmenting the individual spring images according to the spring locations predicted by the model;

[0066] S15: All spring images segmented according to the spring locations are cropped into a preset size to obtain a spring image dataset. Figure 2 Shown is a schematic diagram of a spring image partially cut to a preset size.

[0067] S2: Construct self-supervisory information by expanding training samples through data enhancement

[0068] S21: Acquire several normal spring images from the spring image dataset;

[0069] S22: for each normal spring image, use it as the original image, copy an original image as the base image, and make a pure black image with the same size as the original image as the mask image;

[0070] S23: respectively dividing the original image, the base image, and the mask image into a plurality of regions; specifically, respectively establishing a rectangular coordinate system in the original image, the base image, and the mask image, and dividing the image into four quadrants with the center point of the image as the origin;

[0071] S24: randomly selecting an area from each quadrant of the original image and cutting out a polygonal irregular image block, pasting the area to a random position of any different quadrant of the mask image in sequence, and recording the vertex coordinates of each polygonal irregular image block after pasting;

[0072] S25: according to the recorded vertex coordinates of each polygonal irregular image block at the corresponding position in the base image, the pixel value in the area is erased and filled with RGB value (0,0,0), and then the processed base image is superimposed with the mask image, so as to achieve the translation of the polygonal irregular image block in the original image with the help of the mask, and obtain a defective image with a local irregular pattern compared with the original image after data enhancement, that is, a defective spring image; some original images and corresponding defective images are shown in FIG. Figure 3 As shown, the left image is the original image, and the right image is the image with local defects after the original image is augmented by the "crop-paste" data augmentation strategy;

[0073] S26: Construct a training sample set with category labels based on all normal spring images and the obtained defective spring images.

[0074] S3: Building a spring defect detection model

[0075] A two-stage framework for self-supervised learning is constructed. In the first stage, the proxy task is preset, and the self-supervised network is trained to learn the deep visual representation of spring images of different categories by guiding the binary classifier to classify normal spring images and spring images with artificial defects; in the second stage, the pre-trained model obtained in the first stage is transferred to construct a TVDS spring image anomaly detection single-class classifier, and the anomaly probability of the spring image is calculated. When the probability is higher than the preset threshold, it is determined that there is a defect in the spring part of the spring image, and the anomaly detection log report is output to the inspection personnel, otherwise the spring image is determined to be a normal image.

[0076] S31: In the first stage, a neural network model including a first feature extractor and a binary classifier is constructed, each spring image in the training sample set is used as input, and its corresponding category label is used as output. The neural network model including the first feature extractor and the binary classifier is trained to obtain a spring defect binary classification model. Among them, the first feature extractor is a convolutional neural network, which is used to extract high-dimensional deep visual representations, and the binary classifier classifies deep visual representations of different categories. During the training process, the optimization of the network is guided by the supervision information and the loss function shown in the following formula:

[0077]

[0078] Where N is the number of spring images in the training sample set, CEL is the cross entropy loss function, and x irepresents the positive class samples composed of normal spring images in the training sample set, which are replaced by the label "0", and A(*) represents the simulated negative class samples composed of defective spring images in the training sample set, which are replaced by the label "1"; c(*) represents the constructed binary classifier. When the loss function converges after a certain number of training rounds, the spring defect binary classification model can be obtained.

[0079] S32: In the second stage, a spring defect detection model including a second feature extractor and a single classifier is constructed; wherein the second feature extractor is obtained based on the migration of the first feature extractor, and the second feature extractor and the first feature extractor have the same neural network structure and parameters; the single classifier outputs the probability that the input spring image has defects, which includes a multi-layer fully connected neural network layer, the single classifier obtains the deep visual representation encoded by the second feature extractor, and finally outputs a decimal in the range of 0 to 1 as the probability predicted by the single classifier that the spring image has defects.

[0080] In specific implementation, a single classifier obtains the deep visual representation of the spring image obtained by the second feature extractor and calculates the Mahalanobis distance D between it and the preset potential true representation. Mahalanobis As the probability of a spring image being defective;

[0081]

[0082] Where f(θ) represents the deep visual representation of the spring image obtained by the second feature extractor; X m represents the preset potential true representation, which is the average of the deep visual representations of several normal spring images obtained by the second feature extractor; X conv Covariance matrix of deep visual representation obtained by the second feature extractor for several normal spring images.

[0083] Among them, X m and X conv Obtained by the following method:

[0084] The first feature extractor or the second feature extractor is used to extract the high-dimensional deep visual representation of the normal spring image, and the mean value X of the deep visual representation of all normal spring images is calculated by the following two formulas respectively: m and the covariance matrix X conv :

[0085] X m =E x∈X (f(x))

[0086]

[0087] Where X represents the set of all normal spring image samples, M represents the number of samples in X, f(x) represents the high-dimensional deep visual representation obtained after feature extraction of the normal spring image sample x, and E (expectation) calculates the potential true representation of the spring image in TVDS.

[0088] S33: Compare the obtained probability with the preset spring image defect threshold. When the probability is not greater than the threshold, it is considered that the spring part of the spring image has no defect; when the probability is greater than the threshold, it is considered that the spring part of the spring image has a defect. When it is determined that there is a potential abnormality in the spring part of a test image, it is necessary to locate the defect position and report the abnormality log.

[0089] After obtaining the above spring target detection model and spring defect detection model, spring defect detection can be performed. The process includes:

[0090] A1: Get the captured spring image;

[0091] A2: Input the spring image into the trained spring object detection model to predict the location of the spring;

[0092] A3: Segment the individual spring images according to the predicted spring locations, and crop them into a preset size to obtain the spring image to be tested;

[0093] A4: Input the image of the spring to be tested into the spring defect detection model to obtain the spring defect detection result;

[0094] A5: For spring images identified as having defects, locate the defect location and display it through a bounding box or heat map. Figure 4 As shown, the left picture is a heat map and the right picture is a bounding box map.

[0095] Specifically, the method of displaying through the bounding box is as follows:

[0096] The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized;

[0097] The value of each pixel position in the normalized feature map is used as the abnormal probability estimate of the pixel and compared with the preset threshold. When the abnormal probability of the pixel is greater than the preset threshold, the horizontal and vertical coordinates of the pixel are recorded;

[0098] The maximum and minimum values ​​of the horizontal and vertical coordinates of all abnormal pixels are selected as the boundary vertices of the defect area to draw a predicted rectangular boundary box in the spring image to be tested;

[0099] The method of displaying through heat map is as follows:

[0100] The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized;

[0101] Assign the corresponding color to each pixel according to the value of the position of each pixel in the normalized feature map;

[0102] The processed feature map is superimposed with the original spring image to be detected to obtain a thermal map.

[0103] Since the value of each pixel position in the feature map reflects the probability of defects at that position, by pre-setting the correspondence between the pixel position value and different colors, each pixel can be directly assigned a corresponding color for display. Therefore, after the feature map is superimposed with the original spring image to be tested, the probability of defects at that position can be reflected through different colors.

[0104] In this embodiment, the spring image is named according to the shooting position and the train number and carriage information; when it is identified that the spring image to be tested has defects, the shooting position and train number and carriage information corresponding to the spring to be tested are reversely located according to the name of the image to be tested.

[0105] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0106] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A self-supervised spring defect detection method based on data enhancement strategy, characterized in that: include: Acquire the image of the spring to be tested; Input the image of the spring to be detected into the trained spring defect detection model to obtain the spring defect detection result; Among them, the spring defect detection model is obtained by the following method: Acquire several normal spring images; For the normal spring image, use it as the original image, copy an original image as the base image, and make a pure black image with the same size as the original image as the mask image; The original image, the base plate image, and the mask image are respectively divided into a plurality of regions; Randomly select an area from each patch of the original image and crop an image block, paste it to a random position of any different patch of the mask image in sequence, and record the vertex coordinates of each image block after pasting; According to the recorded vertex coordinates of each image block at the corresponding position in the base plate image, the pixel value in the area is erased, and then the processed base plate image is superimposed with the mask image to obtain the defective spring image; A training sample set with category labels is constructed based on a number of normal spring images and the obtained spring images with defects; The neural network model is trained based on the training sample set to obtain a spring defect detection model; The neural network model is trained based on the training sample set to obtain a spring defect detection model, including: Constructing a neural network model including a first feature extractor and a binary classifier, taking each spring image in the training sample set as input and its corresponding category label as output, training the neural network model including the first feature extractor and the binary classifier to obtain a spring defect binary classification model; Constructing a spring defect detection model including a second feature extractor and a single classifier; wherein the second feature extractor is obtained based on the migration of the first feature extractor, and the single classifier outputs the probability that the input spring image has defects; The obtained probability is compared with the preset defect threshold. When the probability is not greater than the defect threshold, it is considered that there is no defect; when the probability is greater than the defect threshold, it is considered that there is a defect; In the process of training the spring defect binary classification model, the following loss function is used: Where N is the number of spring images in the training sample set, CEL is the cross entropy loss function, and x i represents the positive class samples composed of normal spring images in the training sample set, which are replaced by the label "0", A(*) represents the simulated negative class samples composed of defective spring images in the training sample set, which are replaced by the label "1"; c(*) represents the constructed binary classifier; The single classifier outputs the probability that the input spring image has defects, including: Get the spring image deep visual representation obtained by the second feature extractor, and calculate the Mahalanobis distance D between it and the preset potential real representation Mahalanobis As the probability of a spring image being defective; Where f(θ) represents the deep visual representation of the spring image obtained by the second feature extractor; X m represents the preset potential true representation, which is the average of the deep visual representations of several normal spring images obtained by the second feature extractor; X conv Covariance matrix of deep visual representation obtained by the second feature extractor for several normal spring images.

2. The self-supervised spring defect detection method based on data enhancement strategy according to claim 1 is characterized in that: The original image, the base plate image, and the mask image are divided into a plurality of regions respectively, including: Establish rectangular coordinate systems in the original image, base image, and mask image respectively, and divide the image into four quadrants; The image block is a polygonal irregular image block.

3. The self-supervised spring defect detection method based on data enhancement strategy according to claim 1 is characterized in that: Before acquiring the image of the spring to be tested, the following steps are also included: Acquire the captured spring image; Input the spring image into the trained spring target detection model to detect the location of the spring; The individual spring images are segmented according to the locations of the detected springs, and are cropped into preset sizes to obtain the spring images to be detected.

4. The self-supervised spring defect detection method based on data enhancement strategy according to claim 3 is characterized in that: The spring target detection model is obtained by the following method: Acquire a number of captured spring images; Use the annotation tool to select the spring in the spring image and label it; The processed spring images are used as labeled samples and input into the Detection Transformer model for training to obtain the spring object detection model.

5. The self-supervised spring defect detection method based on data enhancement strategy according to any one of claims 1 to 4, characterized in that: Also includes: For spring images identified as having defects, the defect position is located and displayed through a bounding box or heat map.

6. The self-supervised spring defect detection method based on data enhancement strategy according to claim 5 is characterized in that: The method of displaying through the bounding box is as follows: The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized; The value of each pixel position in the normalized feature map is used as the abnormal probability estimate of the pixel and compared with the preset threshold. When the abnormal probability of the pixel is greater than the preset threshold, the horizontal and vertical coordinates of the pixel are recorded; The maximum and minimum values ​​of the horizontal and vertical coordinates of all abnormal pixels are selected as the boundary vertices of the defect area to draw a predicted rectangular boundary box in the spring image to be tested; The method of displaying through heat map is as follows: The feature map of the last layer of the feature extraction network in the spring defect detection model is intercepted and normalized; Assign the corresponding color to each pixel according to the value of the position of each pixel in the normalized feature map; The processed feature map is superimposed with the original spring image to be detected to obtain a thermal map.

7. The self-supervised spring defect detection method based on data enhancement strategy according to claim 1 is characterized in that: The spring image to be tested is named according to the shooting position and the train number and carriage information; When it is identified that the image of the spring to be detected has defects, the shooting position and train number and carriage information corresponding to the spring to be detected are reversely located according to the name of the image to be detected.

Images