Rail fastener image generation method and model training method

Abstract

The present invention discloses a rail fastener image generation method and a model training method, the method comprising: obtaining an original image containing at least one target object; calculating a first spring bar image and a first background image of the target object; generating a second spring bar image based on the first spring bar image; obtaining a second background image based on the first background image; generating a second fastener image based on the second spring bar image and the second background image; training an image classification model based on the second fastener image to obtain a fastener image classification model for the target object. The present invention generates images of fasteners with and without defects based on real images of defect-free fasteners. The generated images have rich diversity and high authenticity, solve the sample imbalance problem of images of defect-free fasteners and images of defective fasteners, and greatly improve the accuracy of image detection of defective fasteners.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a rail fastener image generation method and a model training method. Background Art

[0002] In the railway track system, the task of the fastener is to connect the rails and sleepers together and keep the two rails in the correct relative position for a long time. However, the open-air working environment and the impact caused by the train running process make it inevitable that the fasteners will become loose, damaged, lost, etc. Therefore, it is necessary to regularly inspect the track to find and replace the fasteners with problems in time.

[0003] In the early stages of railway inspection, the detection of defective fasteners was done manually. This method is time-consuming and inefficient. In the past few decades, some universities and railway research institutes have devoted themselves to using computer vision technology to detect railway defects to overcome the shortcomings of manual inspection. Although existing methods have achieved some results in the detection of defective fasteners, they encounter the problem of data imbalance, that is, the number of images of defective fasteners is much less than the number of images of non-defective fasteners. In particular, in recent years, deep learning technology has been widely used in railway track defect detection, making the importance of data balance more and more prominent. Summary of the invention

[0004] To solve the above problems, the present invention discloses a rail fastener image generation method and a model training method, which generate defective and non-defective fastener images based on existing real non-defective fastener images; the defective fastener images include fastener broken images and fastener lost images; and a balance is achieved between the number of non-defective fastener images and the number of defective fastener images, thereby improving the detection accuracy of defective fastener images.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A rail fastener image generation method and a model training method, comprising:

[0007] Step 1: obtaining an original image containing at least one target object;

[0008] Step 2: Calculate the first strip image and the first background image of the target object;

[0009] Step 3: generating a second spring strip image based on the first spring strip image;

[0010] Step 4: obtaining a second background image based on the first background image;

[0011] Step 5: generating a second fastener image based on the second spring bar image and the second background image;

[0012] Step six: training the image classification model based on the second fastener image to obtain a fastener image classification model for the target object.

[0013] Furthermore, the target object in step 1 includes fasteners on railway tracks such as high-speed railways or subways, such as WJ-7B fasteners, WJ-8B fasteners, WJ-8C fasteners, etc. The original image is a track fastener image containing at least one target object, and the acquisition method includes but is not limited to: track inspection vehicles, track inspection robots, cameras and other devices that can capture images on the track.

[0014] Furthermore, in step 2, the elastic bar region and the background region are separated from the target object. Specifically, the first mask image of the elastic bar region of the target object is obtained by image segmentation technology, and then the original image is intersected with the first mask image to obtain the first elastic bar image, and the rest is the first background image. The first elastic bar image is an image corresponding to the separated elastic bar region, which is a defect-free elastic bar. The first background image is an image corresponding to the background region in the target object, that is, the part of the target object without the elastic bar region. The first mask image is a mask image of a defect-free fastener obtained by image segmentation technology.

[0015] Optionally, if the original image contains multiple target objects, the original image containing only one target object is extracted through object detection.

[0016] Furthermore, in step 3, the first mask image is preprocessed to remove some interference and noise. Then, the skeleton line of the first mask image is extracted. Finally, the second elastic strip image is generated by constraining the geometric shape change of the skeleton line.

[0017] The preprocessing operation includes calculating the distance field M(y) of the first mask image according to the following equation: i )

[0018]

[0019] Here, y i represents the pixel value in the first mask image, ζ={y i =1, i∈Ω} represents the foreground pixel set of the first mask image, represents the boundary pixel of the foreground, ξ(·) represents the sampling function, and d(,) represents the Euclidean distance between two pixels

[0020]

[0021] The above formula represents the shortest distance from a pixel in the first mask image to its boundary pixel. Then, a binary image is obtained based on the distance field image and the image is expanded and eroded.

[0022] Furthermore, in step 3, the skeleton line of the first mask image is extracted according to the following formula:

[0023]

[0024] Here, ! represents the corrosion operation, ⊕ represents the expansion operation, Q represents the structural unit,

[0025]

[0026] T is the last iteration step before X completes the corrosion

[0027]

[0028] Furthermore, in step 3, the second elastic strip image is generated by constraining the geometric shape change of the skeleton line by the following method. First, the skeleton line is simplified according to the following formula:

[0029]

[0030] In the formula, S represents the skeleton line, and B(p,r(p)) represents the skeleton circle with radius r and center point p that is tangent to B. As shown in the following formula, at each iteration step k, the weight w of each skeleton line is updated t

[0031] w t (k) =1-[area(S R (S (k) -S t (k) ))] / area(S R (S (k) ))

[0032] Here, area(·) is the area function. Next, the length of each skeleton line is changed to control the change of the elastic bar geometry.

[0033] Specifically, assume that the skeleton line L = {ξ1,ξ2,…,ξ m} contains m points. Define a truncation factor Γ = μ for each skeleton line c / μ, where μ c represents the cutoff length of L, and μ represents the total length of L. During the algorithm, each point in the set L is traversed and the length between the starting point and the current point is calculated until it exceeds μ. c . Assume that the traversal stops at ξ n point, then record the traversed points L c ={ξ1,ξ2,…ξ n}. After calculating L cAfter each point in is a circle with a center radius of γ, the areas in the first mask image that intersect with these circles are deleted to obtain a series of second mask images. Further, by finding the intersection of the second mask image and the first elastic strip image, a series of second elastic strip images, that is, the generated second elastic strip images with or without defects, can be obtained. In general, according to the different value ranges of the truncation factor Γ, the generated second elastic strip images can be divided into three categories. As shown in the following formula, let G represent the generated second elastic strip image, then

[0034]

[0035] From the above formula, it can be seen that this method can generate three types of second elastic strip images: disease-free, missing and damaged, where Γ1, Γ2 and Γ3 control the length changes of L1, L2 and L3 respectively.

[0036] Furthermore, in step 4, an image restoration method is used to restore the damaged area in the first background image to obtain a second background image. It should be noted that the image restoration method can be an image restoration method based on a deep learning network, or any other method capable of restoring damaged areas in an image, such as a traditional machine learning method.

[0037] Furthermore, in step 5, the second fastener image is obtained by fusing the second spring bar image and the second background image. Specifically, the second spring bar image is first directly superimposed on the second background image to obtain the first fastener image. Then, the fusion effect of the edge area of ​​the spring bar in the first fastener image is improved in the following manner to obtain the second fastener image: u is the pixel set of the first spring bar image, C represents the spring bar area, C0 represents the boundary optimization area, C+C0 represents the fusion area, represents the boundary of the spring clip area, Indicates the outer boundary of the fusion area. Define a weight factor E u To optimize the vector field v. In the fusion area C+C0, the weight factor is calculated as follows:

[0038] ΔE u =0over C0, where

[0039] That is, E u The value range of is [0,1]. u The value is 1, and the E of other areas not in C u The value is 0. The calculation method of the fused pixel value f in the area C+C0 is to minimize the following energy function:

[0040]

[0041]

[0042] where f * Represents the pixel value of the background image before fusion. The following boundary conditions and the above equation are solved

[0043]

[0044] Then, a fused image is obtained. The pixel values ​​in the C0 region of the first fastener image are replaced with the pixel values ​​in the corresponding region of the fused image to obtain a second fastener image.

[0045] Furthermore, in step 6, a data set for training a fastener image classification model is first created. Specifically, the data set includes images of healthy fasteners, images of damaged fasteners, and images of lost fasteners. Then, an image classification model is trained based on the created data set for detecting and classifying defective fasteners.

[0046] Step six also includes adding the repaired background image into the dataset as a fastener image of the missing type.

[0047] Step 6 also includes that part of the defect-free fastener images in the data set are the generated second fastener images, and the other part are real defect-free fastener images. The defect-free fastener images may all be the generated second fastener images.

[0048] The beneficial effects of the present invention are:

[0049] The rail fastener image generation method and model training method described in the present invention generate images of defective and healthy fasteners based on real healthy fastener images. The generated images have rich diversity and high authenticity, solve the sample imbalance problem of healthy fastener images and defective fastener images, and greatly improve the accuracy of defective fastener image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a processing flow chart of a rail fastener image generation and model training method provided by an embodiment of this specification;

[0051] Figure 2 It is a schematic diagram of separating target objects of a track fastener image provided by an embodiment of this specification;

[0052] Figure 3 It is a schematic diagram of generating an image of a second spring bar of a rail fastener provided by an embodiment of this specification;

[0053] Figure 4 It is a schematic diagram of a second fastener image generation principle of a track fastener provided by an embodiment of this specification;

[0054] Figure 5It is a schematic diagram of generating a second background image of a track fastener provided by an embodiment of this specification. DETAILED DESCRIPTION

[0055] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0056] Figure 1 The processing flow of the rail fastener image generation method and model training method provided in this specification is shown:

[0057] S101: Acquire an original image containing at least one target object

[0058] Specifically, the target objects include fasteners on railway tracks such as high-speed railways or subways, such as WJ-7B fasteners, WJ-8B fasteners, WJ-8C fasteners, etc. The original image is a track fastener image containing at least one target object, and the acquisition method includes but is not limited to: track inspection vehicles, track inspection robots, cameras and other devices that can collect images on the track.

[0059] S102: Calculate the first elastic image and the first background image of the target object

[0060] like Figure 2 As shown, the elastic bar region and the background region are separated from the target object. Specifically, the first mask image of the elastic bar region of the target object is obtained by image segmentation technology, and then the original image is intersected with the first mask image to obtain the first elastic bar image, and the rest is the first background image. The first elastic bar image is an image corresponding to the separated elastic bar region, which is a defect-free elastic bar. The first background image is an image corresponding to the background region in the target object, that is, the part of the target object without the elastic bar region. The first mask image is a mask image of a defect-free fastener obtained by image segmentation technology.

[0061] Optionally, if the original image contains multiple target objects, an image containing only one target object is extracted through object detection.

[0062] S103: Generate a second spring strip image based on the first spring strip image

[0063] 1) Preprocessing. The first mask image is preprocessed to remove some interference and noise. The distance field M(y i )

[0064]

[0065] Here, y i represents the pixel value in the first mask image, ζ={yi =1, i∈Ω} represents the foreground pixel set of the first mask image, represents the boundary pixel of the foreground, ξ(·) represents the sampling function, and d(,) represents the Euclidean distance between two pixels

[0066]

[0067] Formula (2) represents the shortest distance from a pixel in the first mask image to its boundary pixel. Then, a binary image is obtained based on the distance field image and some morphological operations are performed.

[0068] 2) Skeleton extraction. Figure 3 As shown in the figure, for the first mask image after preprocessing, its central skeleton line is extracted according to formula (3):

[0069]

[0070] Here, ! represents the corrosion operation, ⊕ represents the expansion operation, Q represents the structural unit,

[0071]

[0072] T is the last iteration step before X completes the corrosion

[0073]

[0074] 3) Generate a strip image. Generate a second strip image by constraining the geometric shape change of the skeleton line. First, perform structural division according to the geometric features of the strip area in the first mask image. For example, Figure 4 As shown in (a), the double solid line in the figure is used as the boundary to divide the elastic bar area into three parts: P1, P2, and P3. For each part, Figure 4 As shown in (c), there is only one skeleton line to control the geometric shape change of its corresponding part. In order to achieve this one-to-one mapping relationship between the skeleton line and the structural area, the following method is used to simplify the skeleton line

[0075]

[0076] As shown in formula (6), its basic principle is to reconstruct according to the skeleton line of the target object, and then delete the skeleton line that has the least impact on the reconstruction (such as Figure 4 (b) is the skeleton line between 1 and 2). In the formula, S represents the skeleton line, and B(p,r(p)) represents the skeleton circle with radius r and center point p that is tangent to B. As shown in formula (7), at each iteration step k, the weight w of each skeleton line is updated t

[0077] w t (k)=1-[area(S R (S (k) -S t (k) ))] / area(S R (S (k) )) (7)

[0078] Here, area(·) is the area function. This skeleton simplification method removes the skeleton line with the lowest weight because it is irrelevant to the main structure of the target object and is therefore considered a redundant branch. Figure 4 (c) is a typical example of a fastener after skeleton simplification, where skeleton lines L1, L2, and L3 represent the three main parts P1, P2, and P3, respectively.

[0079] Next, the length of each skeleton line is changed to control the change of the geometric shape of the spring bar. Specifically, assume that the skeleton line L = {ξ1,ξ2,…,ξ m} contains m points. Define a truncation factor Γ = μ for each skeleton line c / μ (e.g. Γ1, Γ2, Γ3), where μ c represents the cutoff length of L, and μ represents the total length of L. During the algorithm, each point in the set L is traversed and the length between the starting point and the current point is calculated until it exceeds μ. c . Assume that the traversal stops at ξ n point, then record the traversed points L c ={ξ1,ξ2,…ξ n}.like Figure 3 As shown, after calculating L c After each point in is a circle with a center radius of γ, the areas in the first mask image that intersect with these circles are deleted to obtain a series of second mask images. Further, by finding the intersection of the second mask image and the first elastic strip image, a series of second elastic strip images, that is, generated elastic strip images with or without defects, can be obtained. In general, according to the different value ranges of the truncation factor Γ, the generated second elastic strip images can be divided into three categories. As shown in formula (8), let G represent the generated second elastic strip image, then

[0080]

[0081] From formula (8), we can see that this method can generate three types of second elastic strip images: non-defective, missing and damaged, where Γ1, Γ2 and Γ3 control the length changes of L1, L2 and L3 respectively.

[0082] S104: Calculate the second background image from the first background image

[0083] like Figure 5As shown, the first background image is an image separated from the target object, and has a damaged area. The present invention proposes to use an image restoration method to restore the damaged area in the first background image to obtain a second background image. It should be noted that the image restoration method can be an image restoration method based on a deep learning network, such as the DeepFill v2 image restoration deep network, or any other method capable of restoring damaged areas in an image.

[0084] S105: Generate a second fastener image based on the second spring bar image and the second background image

[0085] The second fastener image is obtained by fusing the second spring bar image and the second background image. Specifically, the second spring bar image is first directly superimposed on the second background image to obtain the first fastener image. Then, the fusion effect of the edge area of ​​the spring bar in the first fastener image is improved in the following manner to obtain the second fastener image:

[0086] like Figure 4 As shown in (d), u is the pixel set of the first elastic strip image, C represents the elastic strip area, C0 represents the boundary optimization area, and C+C0 represents the fusion area. represents the boundary of the spring clip area, Indicates the outer boundary of the fusion area. Define a weight factor E u To optimize the vector field v. In the fusion area C+C0, the weight factor is calculated as follows:

[0087] ΔE u =0over C0, where

[0088] That is, E u The value range of is [0,1]. u The value is 1, and the E of other areas not in C u The value is 0. The calculation method of the fused pixel value f in the area C+C0 is to minimize the following energy function:

[0089]

[0090] where f * Represents the pixel value of the background image before fusion.

[0091]

[0092] Formula (10) can be solved by formula (11) to obtain the fused image. The pixel values ​​in the C0 region of the first fastener image are replaced with the pixel values ​​in the corresponding region of the fused image to obtain the second fastener image. At this point, the generated second fastener image includes images of fasteners with and without defects, which can be used to train the defective fastener detection model, thereby solving the problem of low detection accuracy caused by the imbalance in the number of images of defective fasteners and images of non-defective fasteners in actual scenes.

[0093] S106: training an image classification model based on the second fastener image to obtain an image classification model for the target object

[0094] First, a data set is created for training a fastener image classification model. Specifically, the data set includes images of healthy fasteners, images of damaged fasteners, and images of lost fasteners. A portion of the images of healthy fasteners in the data set are generated second fastener images, and another portion are real images of healthy fasteners. The images of defective fasteners may all be generated second fastener images. It should be noted that the present invention adds the repaired background image as a lost type fastener image to the data set. Then, an image classification model is trained based on the created data set for the detection and classification of defective fasteners.

[0095] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A rail fastener image generation method and a model training method, characterized in that: Methods include: Step 1: obtaining an original image containing at least one target object; Step 2: Calculate the first strip image and the first background image of the target object; Specifically, the first mask image of the spring bar region of the target object is obtained by image segmentation technology, and then the original image and the first mask image are intersected to obtain the first spring bar image, and the rest is the first background image; the first spring bar image is an image corresponding to the separated spring bar region, which is a spring bar without defects; the first background image is an image corresponding to the background region in the target object, that is, the part of the target object without the spring bar region; the first mask image is a mask image of a fastener without defects obtained by image segmentation technology; If the original image contains multiple target objects, the original image containing only one target object is extracted through target detection; Step 3, generating a second elastic strip image based on the first elastic strip image; firstly, preprocessing the obtained first mask image to remove some interference and noise; then, extracting the skeleton line of the first mask image; finally, generating the second elastic strip image by constraining the geometric shape change of the skeleton line; Step 4: obtaining a second background image based on the first background image; specifically, using an image restoration method to restore the damaged area in the first background image to obtain the second background image; Step 5: generating a second fastener image based on the second spring bar image and the second background image; Specifically, firstly, the second spring bar image is directly superimposed on the second background image to obtain the first fastener image; then, the fusion effect of the edge area of ​​the spring bar in the first fastener image is improved by using the following formula to obtain the second fastener image: u is the pixel set of the first spring bar image, C represents the spring bar area, C0 represents the boundary optimization area, C+C0 represents the fusion area, represents the boundary of the spring clip area, Indicates the outer boundary of the fusion area; defines a weight factor E u To optimize the vector field v; in the fusion area C+C0, the weight factor is calculated as follows: ΔE u =0over C0, where That is, E u The value range of is [0,1]; E in area C u The value is 1, and the E of other areas not in C u The value is 0; the calculation method of the fused pixel value f in the area C+C0 is to minimize the following energy function: where f * Represents the pixel value of the background image before fusion. The following boundary conditions and the above equation are solved Then, a fused image is obtained; the pixel values ​​in the C0 region of the first fastener image are replaced with the pixel values ​​in the corresponding region of the fused image to obtain a second fastener image; Step six: training the image classification model based on the second fastener image to obtain a fastener image classification model for the target object.

2. The rail fastener image generation method and model training method according to claim 1, characterized in that: In step one, the target object is a fastener on a railway track, and the original image is a track fastener image containing at least one target object, and its acquisition method includes but is not limited to: track inspection vehicles, track inspection robots, and cameras, which are devices that collect images on the track.

3. The rail fastener image generation method and model training method according to claim 1, characterized in that: The preprocessing operation in step 3 includes calculating the distance field M(y) of the first mask image according to the following equation: i ) Here, y i represents the pixel value in the first mask image, ζ={y i =1, i∈Ω} represents the foreground pixel set of the first mask image, represents the boundary pixel of the foreground, ξ(·) represents the sampling function, and d(,) represents the Euclidean distance between two pixels The above formula represents the shortest distance from a pixel in the first mask image to its boundary pixel; then, a binary image is obtained based on the distance field image and the image is expanded and eroded.

4. The rail fastener image generation method and model training method according to claim 1, characterized in that: In step 3, the skeleton line of the first mask image is extracted according to the following formula: here represents the corrosion operation, represents the expansion operation, Q represents the structural unit, T is the last iteration step before X completes the corrosion 5. The rail fastener image generation method and model training method according to claim 1, characterized in that: In step 3, the second elastic strip image is generated by constraining the geometric shape change of the skeleton line by the following method: First, the skeleton line is simplified according to the following formula In the formula, S represents the skeleton line, B(p,r(p)) represents the skeleton circle tangent to B with radius r and center point p; as shown in the following formula, at each iteration step k, the weight w of each skeleton line is updated t w t (k) =1-[area(S R (S (k) -S t (k) ))] / area(S R (S (k) )) Here, area(·) is the area function; Next, change the length of each skeleton line to control the change of the elastic bar geometry; Specifically, assume that the skeleton line L = {ξ1,ξ2,…,ξ m } contains m points; define a truncation factor Γ = μ for each skeleton line c / μ, where μ c represents the cutoff length of L, and μ represents the total length of L; During the algorithm, we traverse each point in the set L and calculate the length between the starting point and the current point until it exceeds μ c ; Assume that the traversal stops at ξ n point, then record the traversed points L c ={ξ1,ξ2,…ξ n }; After calculating L c After each point in is a circle with a center radius of γ, the areas in the first mask image that intersect with these circles are deleted to obtain a series of second mask images; further, by obtaining the intersection of the second mask image and the first elastic strip image, a series of second elastic strip images, that is, the generated second elastic strip images with or without defects, are obtained; according to the different value ranges of the truncation factor Γ, the generated second elastic strip images are divided into three categories; as shown in the following formula, let G represent the generated second elastic strip image, then From the above formula, we know that this method can generate three types of second elastic strip images: disease-free, missing and damaged, where Γ1, Γ2 and Γ3 control the length changes of L1, L2 and L3 respectively.

6. The rail fastener image generation method and model training method according to claim 1, characterized in that: In step six, firstly, a data set for training a fastener image classification model is created; specifically, the data set includes images of healthy fasteners, images of damaged fasteners, and images of lost fasteners; the repaired background image is added to the data set as a fastener image of the lost type; part of the fastener images of the healthy type in the data set are the generated second fastener images, and the other part are the real healthy fastener images; the fastener images of the damaged type are all the generated second fastener images, or part of them are the generated second fastener images and part of them are the real damaged fastener images; Then, an image classification model is trained based on the created dataset for the detection and classification of defective fasteners.

Images