A train component positioning method based on three-dimensional axis information

Abstract

The application discloses a train part positioning method based on three-dimensional axis information, which comprises three parts of axis information acquisition, template information calibration and train part positioning, and specifically comprises the following steps: acquiring a three-dimensional image and a two-dimensional image of a train, converting the three-dimensional image into a two-dimensional depth image, and calculating the coordinates of an axle in the two-dimensional depth image; taking a reference train image as a template to acquire position information of a part to be positioned and a template image; acquiring a rough area image containing the part to be positioned, performing feature matching on the image of the part to be positioned in the rough area image and the template image; calculating a transformation matrix; and obtaining the accurate position of the part to be positioned through the transformation matrix and the position information of the template image. In view of the problems of tedious labeling and easy interference of similar parts in the prior art, the axle is positioned through three-dimensional information, the template information is calibrated, the search range of the part is reduced, and the part positioning is realized based on target feature registration.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, relates to the rapid positioning of train components, and in particular to a train component positioning method based on three-dimensional axis information. Background Art

[0002] With the rapid development of the transportation rail industry in recent years, passenger traffic has continued to rise. While subways provide passengers with a comfortable and safe riding environment, they are also subject to the slow erosion of various natural and unnatural factors such as wind, sun, rain, and so on. Long-term operation and service put increasing pressure on the exterior of the carriages, and various damages will occur to the body of the train, especially the roof and undercarriage. To improve the quality of train maintenance and inspection, many subway lines have introduced dynamic vehicle image monitoring systems to assist maintenance personnel. This system scans the vehicle body image using line scan cameras and 3D cameras installed at the trackside, and uses image processing, deep learning, and other methods to detect abnormal areas on the vehicle body. To facilitate maintenance personnel's inspection, the system needs to provide the relatively precise location and component name of the abnormal area, and the system detection time must be as short as possible. Therefore, a method for quickly locating train components is urgently needed.

[0003] Chinese patent CN104751190A discloses a method: first, the component samples are labeled and HOG features are extracted, and then the SVM classifier is used for training to obtain a model of detectable components. In addition, there are some deep learning methods for target detection and positioning, such as Yolo, Faster RCNN, etc. This type of method first requires labeling and classifying a large number of component images, and then inputs the neural network for backpropagation to solve the model parameters, and finally obtains a model of detectable components. The above method is relatively fast and effective for a small number of detection targets, but there are dozens of components on the bottom and side of the subway train, and the labeling of the detection components requires a lot of time and manpower. The trained model is not universal, and changing the vehicle model or adding detection components requires additional labeling and retraining. In addition, there are many boxes on the train that look similar, which can easily lead to misidentification problems. Summary of the Invention

[0004] The purpose of the present invention is to propose a method for positioning train components based on three-dimensional information of axles, so as to achieve accurate positioning of key components.

[0005] The technical solutions of the present invention are as follows:

[0006] A train component positioning method based on three-dimensional axle information, comprising axle information acquisition, template information calibration and train component positioning;

[0007] Among them, axle information acquisition: obtain the three-dimensional image and two-dimensional image of the train, convert the three-dimensional image into a two-dimensional depth image, and calculate the coordinates of the axle in the two-dimensional depth image;

[0008] Template information calibration: Using the reference train image as a template, the position information and template image of the components to be located are obtained;

[0009] Train component positioning: Obtain a rough area image containing the component to be positioned, perform feature matching between the image of the component to be positioned in the rough area image and the template image; calculate the transformation matrix; and then obtain the precise position of the component to be positioned through the transformation matrix and the position information of the template image.

[0010] Furthermore, the coordinates of the axle in the two-dimensional depth image are calculated as follows: the vertical coordinate of the track in the two-dimensional depth image is set to y0, the distance from the axle to the 3D camera surface is set to d0, the tolerance range of the axle depth value is set to (d0-w1, d0+w1), w1 represents the maximum error of all axle depth values; a small distance h1 is set, the y0+h1 row of pixels in the two-dimensional depth image is selected for processing, and the y0+h1 row of pixels is traversed from the beginning; the depth value of the current traversed coordinate is compared with the depth value of the previous coordinate. Mark the absolute value of the depth difference △d, △d = |d(x1,y0+30)-d(x2,y0+30)|; when the difference △d is within the tolerance range, record it as the current horizontal coordinate L1, that is, the front endpoint of the axle is found; continue to traverse the image, when the difference △d is within the tolerance range again, mark the horizontal coordinate L2, that is, the rear endpoint of the axle is found; at this time, the horizontal coordinate of the axle center in the image can be calculated as L = (L1+L2) / 2; repeat the above steps to find the coordinates of all axles in the image.

[0011] Furthermore, the two-dimensional depth image is smoothed and the image is subjected to median filtering to remove interference from stray noise points.

[0012] Furthermore, the template information calibration is as follows: a line scan image of a reference train is obtained, and multiple line scan images are connected end to end to form a whole vehicle image; the position information Pos (z, zp, x, y, w, h) of the components to be located is marked, where z represents the axle number of the train, zp represents the position of the axle center in the image, x represents the horizontal coordinate of the upper left point of the component with the axle as the starting point, y represents the vertical coordinate of the upper left point of the component, w represents the width of the component in the image, and h represents the height of the component in the image; the position information and names of the components to be located are stored in the database in order, and the marked component images are captured as template images.

[0013] Furthermore, the train component positioning: Step 1: read the position information Pos0 (z0, zp0, x0, y0, w0, h0) of the component to be positioned in the database and the template image img0; obtain the horizontal coordinate zp1 of the z0th axle of the current train in the image, and calculate the rough horizontal coordinate of the position of the component to be positioned as Loc = zp1-zp0+x0; set the bandwidth r, the value of r is related to the degree of lateral distortion of the train image, then the horizontal coordinate of the area containing the train component is (Loc-r, Loc+r), intercept the image of this area, that is, the image img1 containing the component to be positioned; Step 2: The edge and corner features of the image img1 and the template image img0 are detected, and the feature point descriptors of the two images are calculated; Step three: the feature point description matrix of the template image is used as a training set, the Euclidean distance between the feature points of the target image and all the feature points of the template image is calculated, the two closest feature point descriptors are obtained, and the ratio of the closest distance to the second closest distance is calculated. When the ratio is less than the threshold, a pair of excellent matching points is obtained; Step four: through the matching points of the above two images, and solve the perspective transformation matrix H; according to the transformation matrix H and the position parameter Pos0 of the template image, the position Pos1 of the component to be located in the current train image is calculated.

[0014] Furthermore, the transformation matrix H: extracts the coordinates of the four or more corresponding matching points, establishes a superlinear equation system, and estimates the transformation matrix H by the least square method. Furthermore, the position Pos1 of the component to be located in the current train image is calculated:

[0015] (1) Obtain the axle number z1 of the current train component and the position zp1 of the axle center in the image through the axle information and template information;

[0016] (2) The coordinates of the upper left point of the current train component position (x1, y1) are calculated by the following formula, where the right term represents the transformation matrix H (a 11 ,a 12 ....) and the upper left point coordinates (x, y) of the template component position, that is, the x0, y0 information in Pos0;

[0017]

[0018] (3) Based on the height h0 and width w0 of the template component, the coordinates of the lower right point of the template component position are obtained (x0+w0, y0+h0); similarly, the coordinates of the lower right point of the current train component position are calculated according to the formula in (2);

[0019] (4) Then, by subtracting the coordinates of the lower right point and the lower left point of the current train component, the height h1 and width w1 of the current component are obtained; in summary, the position information Pos1 (z1, zp1, x1, y1, w1, h1) of the current train component can be obtained.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] 1. Using axle information can quickly locate the rough area of ​​the component, greatly reducing the component search time and the interference caused by similar components.

[0022] 2. The technical method adopted by the present invention only requires marking template information once, without the need for a large number of repeated markings, and it is relatively easy to subsequently add the required detection components.

[0023] 3. Compared with the machine learning method for detecting axles, the method of the present invention using the difference in three-dimensional depth information is faster and can obtain the axle position at a pixel level. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the train component positioning method based on three-dimensional axle information of the present invention.

[0025] Figure 2 Depth image of the bogie axle area.

[0026] Figure 3 An image of the gateway valve components on the train.

[0027] Figure 4 The registration map of the gateway valve template and the target area. DETAILED DESCRIPTION

[0028] In order to better understand the implementation process of the present invention, the specific implementation methods of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0029] Example 1:

[0030] The present invention takes the scanning image of the subway car body dynamic image monitoring system as an example to demonstrate the process of train component positioning. Figure 1 This is the overall flow chart of the train component positioning method. It can be seen that the technical solution of the present invention includes three parts: axle information acquisition, template information calibration and train component positioning.

[0031] 1. Regarding the axis information acquisition part, it includes the following steps:

[0032] Step 1: Use a 3D camera and a line array camera (the 3D camera and the line array camera are set close to each other and located on the side of the track) to shoot the train body and obtain the train bogie image, which includes a line scan image and a 3D image; pre-process the 3D point cloud information of the bogie channel and convert it into a 2D depth image, in which the grayscale value represents the depth information, such as Figure 2 shown.

[0033] Step 2: Smooth the two-dimensional depth map of the bogie and perform median filtering on the image with a window size of 5*5 to remove interference from some stray noise points.

[0034] Step 3: Figure 2 As shown in the figure, the track's height in the depth image is fixed, and the calibrated track's vertical coordinate in the depth image is y0. In the area immediately above the track, only the axle information is visible, without interference from other components. This difference in depth information can be used to effectively detect the axle.

[0035] On this basis, the distance from the axle to the 3D camera surface is recorded as d0, and a tolerance range of the axle depth value is set (d0-w1, d0+w1), where w1 represents the maximum error of all axle depth values.

[0036] Set a smaller distance h1, select the pixels in the y0+h1 row in the depth image for processing, and traverse the pixels in the y0+h1 row from the beginning.

[0037] Calculate the absolute value △d of the difference between the depth value of the current traversed coordinate and the depth value of the previous coordinate, △d = |d(x1,y0+30)-d(x2,y0+30)|; when the difference △d is within the tolerance range, record it as the current horizontal coordinate L1, that is, the front endpoint of the axle is found; continue to traverse the image, when the difference △d is within the tolerance range again, mark the horizontal coordinate L2, that is, the rear endpoint of the axle is found; at this time, the horizontal coordinate of the axle center in the image can be calculated as L = (L1+L2) / 2.

[0038] Step 4: According to step 3 above, find the coordinates of all axles in the image in turn and record them in a text file or database.

[0039] 2. Regarding the template information calibration part, it includes the following steps:

[0040] Step 1: Select a normal train and mark it as the reference train using the 3D camera and line scan camera mentioned above. A line scan image of the reference train is acquired. Because the line scans are performed line by line, the acquired line scan images can be connected end to end to create a complete train image.

[0041] Step 2: The train components of this embodiment take the gateway valve as an example, Figure 3As shown, the position information Pos(z,zp,x,y,w,h) of the gateway valve is marked, where z represents the axle number of the train, zp represents the position of the axle center in the image, x represents the horizontal coordinate of the upper left point of the component with the axle as the starting point, y represents the vertical coordinate of the upper left point of the component, w represents the width of the component in the image, and h represents the height of the component in the image; the gateway valve position information and name are stored in the database in order, and the marked gateway valve image is cut out as a template image.

[0042] 3. Regarding the positioning of train components, the following steps are included:

[0043] Step 1: Read the gateway valve position information Pos0 (z0, zp0, x0, y0, w0, h0) and the template image img0 in the database, obtain the horizontal coordinate zp1 of the z0th axle of the current train in the image, and calculate the rough horizontal coordinate of the component to be located as Loc = zp1 - zp0 + x0, where x0 is the horizontal coordinate of the upper left point of the gateway valve;

[0044] Because the unevenness of train speed causes image stretching and contraction, it's necessary to search for the component within the expanded range of the aforementioned rough location. A bandwidth, r, is set, whose value is related to the degree of lateral distortion in the train image. The region containing the gateway valve component has a horizontal coordinate of (Loc-r, Loc+r). The image of this region is captured, i.e., image img1 containing the gateway valve component.

[0045] Step 2: Use the SURF algorithm to detect the edge and corner features of the image containing the gateway valve and the gateway valve template image, and use the SURF algorithm to calculate the feature point descriptors of the two images.

[0046] Step 3: Use the feature point description matrix of the template image as the training set, calculate the Euclidean distance between the target image feature points and all feature points of the template image, obtain the two closest feature point descriptors, calculate the ratio of their closest distance to the next closest distance, and when the ratio is less than the threshold of 0.6, a pair of excellent matching points is obtained. The matching results are as follows: Figure 4 shown.

[0047] Step 4: Obtain the matching points of the above two images, solve the perspective transformation matrix H (extract the coordinates of more than 4 groups of corresponding matching points, establish a superlinear equation system, and estimate the transformation matrix H through the least squares method), and eliminate the incorrect matching points through the random sampling consistency method.

[0048] According to the obtained transformation matrix H and the template image position parameter Pos0, the position Pos1 of the component in the current train image can be calculated.

[0049] Specifically:

[0050] (1) Through the axle information and template information, the axle number z1 where the current train component is located and the position zp1 of the axle center in the image are obtained.

[0051] (2) The coordinates of the upper left point of the current train component position (x1, y1) are calculated by the following formula, where the right term represents the transformation matrix H (a 11 ,a 12 ....) and the upper left point coordinates (x, y) of the template component position, that is, the x0, y0 information in Pos0.

[0052]

[0053] (3) Based on the height h0 and width w0 of the template component, the coordinates of the lower right point of the template component position are obtained (x0+w0, y0+h0); similarly, the coordinates of the lower right point of the current train component position are calculated according to the above formula;

[0054] Then, the height h1 and width w1 of the current component are obtained by subtracting the coordinates of the lower right point and the lower left point of the current train component.

[0055] According to the above steps, the current train component position information Pos1 (z1, zp1, x1, y1, w1, h1) can be obtained.

[0056] The train component positioning method of the present invention addresses the problems of cumbersome labeling and easy interference between similar components in the prior art. By locating the axle through three-dimensional information and calibrating the template information, the search range of the components is narrowed, and alignment is performed based on target features, thereby realizing component positioning.

[0057] The above description is only the best embodiment of the present invention, which is only used to illustrate the technical solution of the present invention rather than to limit it. These modifications or replacements do not cause the essence of the corresponding technical solution to deviate from the spirit and scope of the technical solution of each embodiment of the present invention.

Claims

1. A train component positioning method based on three-dimensional axis information, characterized in that: The method includes axle information acquisition, template information calibration and train component positioning; Axle information acquisition: Obtain 3D and 2D images of the train, convert the 3D image into a 2D depth image, and calculate the coordinates of the axle in the 2D depth image: Set the vertical coordinate of the track in the two-dimensional depth image to y0, the distance from the axle to the 3D camera surface to d0, and the tolerance range of the axle depth value to (d0-w1, d0+w1), where w1 represents the maximum error of all axle depth values; set a small distance h1, select the y0+h1 row of pixels in the two-dimensional depth image for processing, and traverse the y0+h1 row of pixels from the beginning; calculate the absolute value △d of the difference between the depth value of the current traversed coordinate and the depth value of the previous coordinate, △d=|d(x1,y0+30)-d(x2,y0+30)|; when the difference △d is within the tolerance range, record it as the current horizontal coordinate L1, where L1 is the front end point of the axle; Continue traversing the image. When the difference △d is within the tolerance again, mark the horizontal coordinate L2, where L2 is the rear endpoint of the axle. Calculate the horizontal coordinate of the axle center in the image as L = (L1 + L2) / 2. Repeat the above steps to find the coordinates of all axles in the image. Template information calibration: Using a reference train image as a template, the position information and template image of the components to be located are obtained. Template information calibration: obtaining a line scan image of the reference train, connecting multiple line scan images end to end to form a complete vehicle image; annotating the position information Pos (z, zp, x, y, w, h) of the components to be located, where z represents the axle number of the train, zp represents the position of the axle center in the image, x represents the horizontal coordinate of the upper left point of the component with the axle as the starting point, y represents the vertical coordinate of the upper left point of the component, w represents the width of the component in the image, and h represents the height of the component in the image; the position information and names of the components to be located are stored in a database in order, and the annotated component images are captured as template images; Train component positioning: Obtain a rough area image containing the component to be positioned, and perform feature matching between the image of the component to be positioned in the rough area image and the template image; Calculate the transformation matrix; Then, the precise position of the component to be positioned is obtained through the transformation matrix and the position information of the template image.

2. A train component positioning method based on three-dimensional axis information according to claim 1, characterized in that: The two-dimensional depth image is smoothed and the image is subjected to median filtering to remove interference from stray noise points.

3. The train component positioning method based on three-dimensional axis information according to claim 2, characterized in that: Positioning of the train components: Step 1: Read the position information Pos0 (z0, zp0, x0, y0, w0, h0) of the component to be located in the database and the template image img0; obtain the horizontal coordinate zp1 of the z0th axle of the current train in the image, and calculate the rough horizontal coordinate of the position of the component to be located as Loc=zp1-zp0+x0; set the bandwidth r, the value of r is related to the degree of lateral distortion of the train image, then the horizontal coordinate of the area containing the train component is (Loc-r, Loc+r), intercept the image of this area, that is, the image img1 containing the component to be located; Step 2: Detect the edge and corner features of the image img1 containing the component to be located and the template image img0, and calculate the feature point descriptors of the two images; Step 3: Use the feature point description matrix of the template image as the training set, calculate the Euclidean distance between the feature points of the target image and all the feature points of the template image, obtain the two closest feature point descriptors, and calculate the ratio of their closest distance to the next closest distance. When the ratio is less than the threshold, a pair of excellent matching points is obtained. Step 4: Use the matching points of the two images above and solve the perspective transformation matrix H; calculate the position Pos1 of the component to be located in the current train image based on the transformation matrix H and the position parameter Pos0 of the template image.

4. A train component positioning method based on three-dimensional axis information according to claim 3, characterized in that: The transformation matrix H: extract the coordinates of the four or more corresponding matching points, establish a superlinear equation group, and estimate the transformation matrix H by the least squares method.

5. The train component positioning method based on three-dimensional axis information according to claim 3, characterized in that: The position Pos1 of the component to be located in the current train image is calculated: (1) Obtain the axle number z1 of the current train component and the position zp1 of the axle center in the image through the axle information and template information; (2) The coordinates of the upper left point of the current train component position (x1, y1) are calculated by the following formula, where the right term represents the transformation matrix H (a 11 ,a 12 ....) and the upper left point coordinates (x, y) of the template component position, that is, the x0, y0 information in Pos0, ； (3) Then, based on the height h0 and width w0 of the template component, the coordinates of the lower right point of the template component position (x0+w0, y0+h0) are obtained; similarly, the coordinates of the lower right point of the current train component position are calculated according to the formula in (2) above; (4) Then, by subtracting the coordinates of the lower right point and the lower left point of the current train component, the height h1 and width w1 of the current component are obtained; in summary, the position information Pos1 (z1, zp1, x1, y1, w1, h1) of the current train component can be obtained.

Images